JP5853719B2 - Image processing system and vehicle equipped with the same - Google Patents

Description

  The present invention relates to an image processing system that detects a road surface state (dry state or frozen state) and uses it for the purpose of alerting a driver, and a vehicle equipped with the image processing system.

  Conventionally, a method for detecting whether or not the road surface state is a dry state or a frozen state based on a polarized image obtained by imaging the road surface has been proposed (see, for example, Patent Document 1).

  The road surface state detection device disclosed in Patent Document 1 includes an imaging unit that images a road surface, and a polarization unit that is disposed in front of the imaging unit and whose polarization plane can be varied in the horizontal and vertical directions, and images the road surface. A power spectrum image is generated by performing Fourier transform on the horizontally polarized image and the vertically polarized image, and a feature is calculated from the distribution of frequency components in the power spectrum image, and the calculated feature value is larger than the set value. Whether or not the road surface is frozen, wet and dry is determined and output.

  However, the road surface state detection device disclosed in Patent Document 1 is configured to sequentially set a vertical polarization filter and a horizontal polarization filter by a filter conversion unit when capturing a horizontal polarization image and a vertical polarization image, and provides a real-time image. Not suitable for imaging.

  Although details of the filter conversion unit are not described in Patent Document 1, it is assumed that the filter conversion unit is a mechanical mechanism such as a general motor or a switching mechanism such as a liquid crystal. When these mechanisms are mounted on a vehicle such as an automobile, there are problems in terms of heat resistance and vibration resistance.

  In view of this, an imaging apparatus that can cope with such a problem has been proposed (see, for example, Patent Document 2). According to Patent Document 2, it is possible to capture a horizontally polarized image and a vertically polarized image with one image sensor.

  In recent years, there has been a demand for an image pickup apparatus that picks up a driver's attention by picking up images of road signs and the like. The road signs provide various information, and the attention level is expressed in colors such as red, yellow, and blue.

  However, the imaging device disclosed in Patent Document 2 acquires polarization information and luminance information, but is not configured to acquire color information. For this reason, there has been a problem that the attention level of the sign expressed in color cannot be recognized.

  The present invention has been made to solve such a conventional problem, and is capable of detecting a road surface state using a polarization image and generating a color image for detecting a sign. An object is to provide a system and a vehicle including the system.

  In order to solve the above-described problem, an image processing system according to the present invention includes a Bayer array image sensor in which R pixels, G pixels, and B pixels are two-dimensionally arrayed, and an optical element disposed in front of the image sensor. An imaging means having a filter; and an image analysis means for analyzing an imaging result of the imaging means, wherein the optical filter includes a substrate that transmits incident light, and four pixels adjacent to each other in the vertical and horizontal directions of the imaging element. A polarizing filter layer on which a polarizer for making any one of the horizontal polarization component and the vertical polarization component of the incident light enter only one of the two G pixels included in the pixel group, and The image analysis means generates a horizontal polarization component image, a vertical polarization component image, and a polarization degree image using the pixel value of the one G pixel and the pixel value of the other G pixel of the two G pixels. When The pixel values of the other G pixel, the pixel values of the R pixels, and, and generating a color image using the pixel values of the B pixels.

  With this configuration, a polarizer is disposed in front of the color sensor, so that it is possible to detect a road surface state using a polarization image and also generate a color image for detecting a sign.

  The present invention provides an image processing system capable of detecting a road surface state using a polarization image and generating a color image for detecting a sign, and a vehicle including the image processing system.

The schematic diagram which shows schematic structure of an on-vehicle apparatus control system provided with the image processing system which concerns on this invention 1 is a schematic diagram showing a schematic configuration of an imaging unit included in an image processing system according to the present invention. Cross-sectional schematic diagram along the light transmission direction of the optical filter, imaging device, and sensor substrate Front schematic view of the optical filter viewed from the sensor substrate side, and front schematic view of the image sensor viewed from the imaging lens side Schematic diagram illustrating correspondence between polarization filter layer of optical filter and pixel of image sensor Explanatory drawing showing the pattern of the polarizer Explanatory drawing for demonstrating the change of reflected light when a road surface state is a wet state, and when it is a dry state The graph which shows the incident angle dependence of the horizontal polarization component of the reflected light with respect to the incident light of the light intensity I, and a vertical polarization component Explanatory drawing which shows the luminance image and polarization degree image which imaged the concrete surface where the ice plate was put Explanatory drawing which shows the brightness | luminance image and polarization degree image which imaged the obstacle in a shadow Schematic diagram showing how the sign detection is performed by the image processing system mounted inside the vehicle Schematic diagram showing the installation position of the imaging unit Flow chart showing the procedure of exposure control Graph showing the spectral characteristics of the first spectral filter layer Enlarged view of the wire grid structure constituting the polarizing filter layer Schematic which shows the pattern of the polarizer in 2nd Embodiment. Schematic which shows the pattern of the polarizer in 3rd Embodiment. Schematic diagram illustrating the correspondence between the polarizing filter layer of the optical filter and the pixels of the image sensor in the fourth embodiment The schematic diagram which shows the pattern of the polarizer in 5th Embodiment Sectional schematic drawing along the light transmission direction showing the structure of the optical filter in the sixth embodiment Schematic front view of the optical filter according to the sixth embodiment viewed from the sensor substrate side Graph showing spectral characteristics of color filter Graph showing the spectral characteristics of the second spectral filter layer

  Hereinafter, an image processing system according to the present invention and a vehicle including the image processing system will be described with reference to the drawings. In addition, the dimensional ratio of each structure on each drawing does not necessarily correspond with the actual dimensional ratio.

(First embodiment)
FIG. 1 is a schematic diagram showing a schematic configuration of an in-vehicle device control system including an image processing system 110 according to the present invention. The in-vehicle device control system uses the captured image data of the front area in the traveling direction or the rear area in the traveling direction of the vehicle 100 captured by the imaging unit 101 mounted on the vehicle 100 such as an automobile to distribute the light of the headlamp 104. Control, drive control of the wiper 107 for removing foreign matter adhering to the windshield (transparent member) 105, and control of other in-vehicle devices are performed.

  The in-vehicle device control system shown in FIG. 1 mainly includes an imaging unit 101, an image analysis unit (image analysis means) 102, a headlamp control unit 103, a wiper control unit 106, and a vehicle travel control unit 108. Prepare.

  The image processing system 110 according to the present embodiment includes an imaging unit 101 and an image analysis unit 102. The image analysis unit 102 has a function of controlling the imaging unit 101 and a function of analyzing captured image data transmitted from the imaging unit 101.

  The image analysis unit 102 analyzes the captured image data transmitted from the imaging unit 101 to detect foreign matters such as raindrops adhering to the windshield 105, or to detect other vehicles existing in front of the vehicle 100 in the captured image data. The position, the direction, and the distance are calculated, and detection objects such as white lines (division lines) and signs on the road surface existing in the imaging region are detected.

  Hereinafter, information such as the position, direction, distance, white line (division line) on the road surface, road surface state, and signs of other vehicles in front or behind the vehicle 100 is also referred to as vehicle peripheral information. In the detection of another vehicle, the image analysis unit 102 detects a preceding vehicle that travels in the same traveling direction as the vehicle 100 by identifying the tail lamp of the other vehicle from captured image data obtained by imaging the surrounding information of the preceding vehicle. By identifying the headlamp of the vehicle, an oncoming vehicle traveling in the opposite direction to the vehicle 100 is detected.

  The calculation result of the image analysis unit 102 is sent to the headlamp control unit 103. For example, the headlamp control unit 103 generates a control signal for controlling the headlamp 104 that is an in-vehicle device of the vehicle 100 from the distance data calculated by the image analysis unit 102. Specifically, for example, the driver of the vehicle 100 can avoid the dazzling of the driver of the other vehicle while avoiding the strong light of the headlamp of the vehicle 100 entering the eyes of the driver of the preceding vehicle or the oncoming vehicle. Therefore, the switching of the high beam and the low beam of the headlamp 104 is controlled, or partial light shielding control of the headlamp 104 is performed.

  Further, as will be described later, the headlamp control unit 103 may perform lighting control of a fog lamp (not shown) based on information on the degree of polarization recognized by the image analysis unit 102.

  The calculation result of the image analysis unit 102 is also sent to the wiper control unit 106. The wiper control unit 106 controls the wiper 107 to remove deposits such as raindrops attached to the windshield 105 of the vehicle 100. The wiper control unit 106 receives a foreign object detection result detected by the image analysis unit 102 and generates a control signal for controlling the wiper 107. When the control signal generated by the wiper control unit 106 is sent to the wiper 107, the wiper 107 is operated to ensure the visibility of the driver of the vehicle 100.

  The calculation result of the image analysis unit 102 is also sent to the vehicle travel control unit 108. Based on the white line detection result detected by the image analysis unit 102, the vehicle travel control unit 108 notifies the driver of the vehicle 100 of a warning when the vehicle 100 is out of the lane area defined by the white line. Or driving support control such as controlling the steering wheel and brake of the host vehicle.

  In addition, the vehicle travel control unit 108 is based on the difference between the sign detection result (described later) detected by the image analysis unit 102 and the vehicle travel state, for example, when the vehicle 100 is traveling at a speed close to the speed limit. The driver is alerted, or the speed of the vehicle 100 is controlled when the vehicle 100 is traveling beyond the speed limit.

  FIG. 2 is a schematic diagram illustrating a schematic configuration of the imaging unit 101. The imaging unit 101 includes an imaging lens 204 that collects light transmitted through the windshield 105 from the outside of the vehicle 100, and an imaging element 206 that performs imaging by photoelectrically converting the light collected by the imaging lens 204 for each pixel. , An optical filter 205 disposed between the imaging lens 204 and the imaging device 206, a sensor substrate 207 on which the imaging device 206 is mounted, and an analog electrical signal output from the sensor substrate 207 (each light reception on the imaging device 206). A signal processing unit 208 that outputs captured image data obtained by converting a received light amount received by the element into a digital electric signal.

  The imaging lens 204, the optical filter 205, the imaging element 206, and the sensor substrate 207 are arranged in this order from the windshield 105 side. The signal processing unit 208 is electrically connected to the image analysis unit 102. FIG. 2 shows an example in which the image sensor 206 and the signal processing unit 208 are provided independently, but the configuration of the imaging unit 101 is not limited to this. For example, when an image sensor 206 having an A / D converter for each pixel is used, the A / D converter becomes the signal processor 208. That is, in this case, the signal processing unit 208 is built in the image sensor 206.

  The imaging lens 204 is composed of, for example, a plurality of lenses, and the focal position is set at infinity or between the infinity and the outer wall surface of the windshield 105.

  Incident light from the imaging region including the subject (detection target) passes through the imaging lens 204, passes through the optical filter 205, and is photoelectrically converted into an electrical signal corresponding to the light intensity by the imaging element 206. An electrical signal (analog signal) output from the image sensor 206 via the sensor substrate 207 is input to the signal processing unit 208. The signal processing unit 208 then converts the digital signal (captured image data) including the brightness (luminance information) and color information of each pixel on the image sensor 206 together with the horizontal / vertical synchronization signal of the image into the subsequent image analysis unit 102. Output to.

  As already described, in this embodiment, the focal position of the imaging lens 204 is set to infinity or between infinity and the outer wall surface of the windshield 105. Thereby, when detecting a preceding vehicle or an oncoming vehicle, or detecting a white line, appropriate information can be acquired from the captured image data of the imaging unit 101.

  However, when focusing at infinity, when identifying the tail lamp of a preceding vehicle traveling far, there may be about one light receiving element that receives the light of the tail lamp on the image sensor 206. In this case, the light from the tail lamp may not be received by the red light receiving element that receives the tail lamp color (red). In this case, the tail lamp cannot be recognized and the preceding vehicle cannot be detected. In order to avoid such a problem, it is preferable that the imaging lens 204 is focused before infinity. As a result, the tail lamp of the preceding vehicle traveling far is out of focus, so that the number of light receiving elements that receive the light of the tail lamp can be increased, the recognition accuracy of the tail lamp is increased, and the detection accuracy of the preceding vehicle is improved.

  FIG. 3 is a schematic cross-sectional view along the light transmission direction of the optical filter 205, the image sensor 206, and the sensor substrate 207. 4A is a schematic front view of the optical filter 205 viewed from the sensor substrate 207 side, and FIG. 4B is a schematic front view of the image sensor 206 viewed from the imaging lens 204 side. 3 and 4, each pixel of the image sensor 206 is illustrated in a simplified manner, but actually, the image sensor 206 is configured by about several hundreds of thousands of pixels arranged two-dimensionally.

  As shown in FIG. 3, the optical filter 205 includes a substrate 220 that is transparent to incident light in a use band (in this embodiment, a visible light region and an infrared light region), and a surface on the imaging lens 204 side of the substrate 220. It is formed on the entire surface of the effective imaging region (the region corresponding to all the pixels constituting the imaging device 206), and emits light having a wavelength component in the range of wavelengths λ1 to λ2 and λ3 to λ4 (λ1 <λ2 <λ3 <λ4). A selectively transmitting spectral filter layer (first spectral filter layer) 221, a polarizing filter layer 223 formed on the surface of the substrate 220 on the image sensor 206 side, and a filler 224 filled on the polarizing filter layer 223. And the surface of the filler 224 on the image sensor 206 side is disposed close to the image sensor 206.

  The image sensor 206 is an image sensor using a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor) or the like, and a color filter 206b is formed on the surface of a pixel array 206a composed of a plurality of pixels arranged two-dimensionally. Color sensor.

  The color filter 206b includes a filter that mainly transmits light in the red wavelength band (λ = 550 to 650 nm), a filter that mainly transmits light in the green wavelength band (λ = 500 to 550 nm), and a blue wavelength band (λ (= 450 to 500 nm) has a Bayer array configuration in which the filter that mainly transmits light is disposed corresponding to each pixel of the pixel array 206a.

  Each pixel of the pixel array 206a is provided with a light receiving element such as a photodiode for imaging light incident from the optical filter 205 side and transmitted through the color filter 206b. In order to increase the light collection efficiency of the light receiving element, a microlens (not shown) may be provided on the incident side of the pixel array 206a corresponding to each pixel. The imaging element 206 configured in this manner is bonded to a PWB (Printed Wiring Board) by a technique such as wire bonding and mounted on the sensor substrate 207.

  FIG. 5 is a schematic view illustrating the correspondence between the polarization filter layer 223 of the optical filter 205 and the pixels of the image sensor 206. Hereinafter, the pixel of the image sensor 206 corresponding to the filter for the red wavelength band is an R pixel, the pixel of the image sensor 206 corresponding to the filter for the green wavelength band is the G pixel, and the image sensor corresponding to the filter for the blue wavelength band. The pixel 206 is referred to as a B pixel. Further, four pixels (one R pixel, two G pixels, and one B pixel) adjacent in the vertical and horizontal directions are referred to as a pixel group. In the effective imaging area, an area in which R pixels are arranged in a pixel group is denoted as r, an area in which G pixels are disposed is denoted as g1 and g2, and an area in which B pixels are disposed is denoted as b.

  In the polarizing filter layer 223, a polarizer corresponding to each pixel of the image sensor 206 is divided into regions over the entire effective imaging region. In the present embodiment, a polarizer for causing one of the horizontal polarization component P and the vertical polarization component S of incident light to enter only one of the two G pixels included in each pixel group is applied to the polarization filter layer 223. Is formed.

  FIG. 5 shows an example in which a polarizer that transmits the vertical polarization component S of incident light is formed so as to correspond to the G pixel disposed in the region g1. Polarizers are not formed in the regions g2, r, and b, and incident light to these regions is not subjected to polarization limitation. That is, the area where the polarizer is formed occupies 1/4 of the effective imaging area.

  As a result, from each pixel group, the pixel value Ir of the R pixel in the region r where polarization is not limited, the pixel value Ig1 of the G pixel in the region g1 through which only the vertical polarization component S of the incident light is transmitted, and the region g2 where polarization is not limited The pixel value Ig2 of the G pixel and the pixel value Ib of the B pixel in the region b where the polarization is not limited are output.

  Here, the pixel value Ig1 includes information on only the vertical polarization component S, and the pixel value Ig2 on which polarization is not limited includes information corresponding to the sum of the horizontal polarization component P and the vertical polarization component S. Yes. Therefore, the horizontal polarization component P of the G pixel can be detected by taking the difference between Ig2 and Ig1 adjacent in the diagonal direction within the pixel group.

  As described above, the image analysis unit 102 acquires the horizontal polarization component P and the vertical polarization component S of the G pixel from each pixel group, and performs a known image interpolation process, thereby performing the horizontal polarization component image and the vertical polarization component. Two types of images are generated. For example, in the case where a vertical polarization component image is formed from the pixel value Ig1 including only the information of the vertical polarization component S, the pixel values of the regions g2, r, and b are set to the same pixel group and adjacent pixel groups. What is necessary is just to set it as the average value of the pixel value of the area | region g1 which has.

  More specifically, as shown in FIG. 6, for example, the pixel value of the region g2 of the pixel group A is, for example, the region g1 adjacent in the diagonal direction and the regions g11, g12, and g13 of other pixel groups. The average value of the pixel values. The pixel value of the region r of the pixel group A is, for example, the average value of the pixel values of the region g1 adjacent in the horizontal direction and the region g11 of another pixel group. The pixel value of the region b of the pixel group A is, for example, the average value of the pixel values of the region g1 adjacent in the vertical direction and the region g13 of another pixel group.

Further, the image analysis unit 102 obtains the degree of polarization for each pixel from the following equation (1), where I (P) is the pixel value of the horizontal polarization component image and I (S) is the pixel value of the vertical polarization component image. Thus, a polarization degree image that does not depend on luminance information is generated.
Polarization degree = (I (P) −I (S)) / (I (P) + I (S)) (1)

  In the example shown in FIG. 5, the degree of polarization for each pixel group corresponds to (Ig2-2 × Ig1) / Ig2.

  Further, the image analysis unit 102 generates a color image by using the image interpolation technique as described above. That is, since the R image, the G image, and the B image are obtained from the pixel values Ir, Ig2, and Ib that are not subjected to the polarization limitation, a color image is generated by superimposing these. That is, the color information of each pixel group corresponds to the pixel value Ir, twice the pixel value Ig2, and the overlap of the pixel value Ib.

  The polarization degree image is used to detect road surface conditions such as dry, wet, and frozen road surfaces in order to prevent the vehicle 100 from skidding. Here, the change of the reflected light according to the road surface state will be described.

  In the road surface state determination process performed by the image analysis unit 102 in the present embodiment, among the information that can be acquired from the imaging unit 101, the white polarization component (non-spectral) horizontal polarization component P and the vertical polarization component S are compared. Polarization information is used.

  FIGS. 7A and 7B are explanatory diagrams for explaining changes in reflected light when the road surface state is a wet state and when the road surface state is a dry state.

As shown to Fig.7 (a), the road surface in a wet state will be in a state close | similar to a mirror surface, when water accumulates in the uneven part of a road surface. Therefore, the reflected light on the wet road surface exhibits the following polarization characteristics. That is, when the reflectances of the horizontal polarization component P and the vertical polarization component S of the reflected light are Rp and Rs, respectively, the horizontal polarization component Ip and the vertical polarization component Is of the reflected light with respect to the incident light having the light intensity I are expressed by the following formula ( The incident angle dependency can be calculated from 2) and (3) as shown in FIG.
Ip = Rp × I (2)
Is = Rs × I (3)

  As can be seen from FIG. 8, the reflectance Rp of the horizontal polarization component Ip of the reflected light at the mirror surface becomes zero when the incident angle is equal to the Brewster angle (53.1 degrees), and the reflected light intensity of the horizontal polarization component Ip is It becomes zero. Further, since the reflectance Rs of the vertical polarization component Is of the reflected light on the mirror surface shows a characteristic that increases gradually as the incident angle increases, the reflected light intensity of the vertical polarization component Is also increases gradually as the incident angle increases.

  On the other hand, as shown in FIG. 7B, the road surface in the dry state has a rough surface, so that irregular reflection is dominant, the reflected light does not show polarization characteristics, and the reflectance Rp of each polarization component, The difference in Rs becomes smaller.

It is possible to determine whether the road surface state is a wet state or a dry state based on the difference in the polarization characteristics of the reflected light from the road surface. Specifically, the image analysis unit 102 uses the polarization ratio H shown in the following formula (4) in determining the wet and dry state of the road surface. For example, the polarization ratio H is calculated by calculating a ratio (S / P) between the vertical polarization component S of white light (non-spectral) and the horizontal polarization component P of white light (non-spectral) for an image region that reflects a road surface. It can be obtained from the average value. Since the polarization ratio H is a parameter that does not depend on the incident light intensity I as shown in the following formula (4), the polarization ratio H is stably used for determining the wet and dry state of the road surface without being affected by the luminance fluctuation in the imaging region. be able to.
H = Is / Ip = Rs / Rp (4)

Equation (4) can also be expressed as the following equation (5) as a function of the degree of polarization shown in equation (1). Here, Is is regarded as the pixel value I (S) of the polarization degree image, and Ip is regarded as the pixel value I (P) of the polarization degree image.
H = (1−degree of polarization) / (1 + degree of polarization) (5)

  When the polarization ratio H obtained in this way exceeds a predetermined threshold value, the image analysis unit 102 determines that the road surface state is a wet state, and when the road surface state is equal to or lower than the predetermined threshold value, the road surface state is dry. It is determined that it is in a state. When the road surface is dry, the vertical polarization component S and the horizontal polarization component P are substantially equal, so the polarization ratio H is about 1. On the other hand, when the road surface is completely wet, the vertical polarization component S takes a value much larger than the horizontal polarization component P, so the polarization ratio H becomes a large value. When the road surface is slightly wet, The ratio H is an intermediate value between them.

  In the present embodiment, the determination result of the determination process of the wet / dry condition of the road surface state as described above is used for driving support control such as warning to the driver of the vehicle 100 and control of the steering wheel and brake of the vehicle 100. . Specifically, when it is determined that the road surface state is a wet state, the determination result is sent to the vehicle travel control unit 108, and is used for, for example, control of the automatic brake system of the vehicle 100, thereby causing a traffic accident. A reduction effect can be expected. In addition, for example, information that warns that the road surface is slippery may be notified on a CRT screen of the car navigation system of the vehicle 100 to alert the driver.

  Note that the image analysis unit 102 may perform a road surface state determination process only for a pixel group corresponding to an imaging region where the road surface is imaged (particularly below the effective imaging region).

  By the way, when the frozen road surface, in particular, the surface layer of asphalt called black ice is frozen, it is difficult to distinguish the frozen region from the normal luminance image. On the other hand, the frozen region is clearly expressed in the polarization degree image. FIG. 9 shows a luminance image and a polarization degree image obtained by imaging the ice sheet on the concrete surface.

  As shown in FIG. 9 (a), it is difficult to determine where the ice is in the luminance image, whereas in the polarization degree image of FIG. 9 (b), the location (portion enclosed by an ellipse in the figure). You can check the ice plate. Since the light is scattered (diffuse reflection) in the concrete portion without the ice plate, the degree of polarization is almost zero. On the other hand, in the concrete portion where the plate ice is present, the light is scattered inside the plate ice, but the P-polarized component is dominant as the polarization component transmitted through the air interface from the inside of the plate ice. For this reason, in FIG.9 (b), a difference can be seen clearly with a plate ice part and a dry concrete part.

  By using polarization information, it is possible to detect information other than road surface conditions, for example, an obstacle in a shadow that is difficult to detect in a luminance image. FIG. 10 is an explanatory diagram illustrating the imaging result of the obstacle in the shadow. In contrast to the luminance image in FIG. 10A, a vehicle parked beside the road can be recognized in the polarization degree image in FIG. 10B. The polarization degree image is generated by the calculation of the expression (1) already shown, and is independent of the luminance information and can express the angle information and the material difference of the object. Suitable for detecting obstacles in shadows. In other words, in the present embodiment, the image analysis unit 102 can perform obstacle detection with high accuracy by using such a polarization degree image.

  In the present embodiment, the imaging unit 101 may capture not only the vehicle surrounding information in front of the vehicle 100 but also the vehicle surrounding information in the rear, for example. In recent years, it is possible to detect an obstacle or the like hidden in the shadow of the host vehicle by arranging the imaging unit 101 as a rear view camera mounted on various vehicles and capturing a polarization degree image.

  When the imaging unit 101 is used as a rear view camera, it is desirable to use a wide-angle lens having an angle of view of 90 degrees or more as the imaging lens 204. In general, when a wide-angle lens is used, the incident angle of the peripheral image is larger than that of the image center (for example, 20 degrees). When the incident angle increases in this way, the transmittance of incident light decreases in the region where the polarizer of the polarizing filter layer 223 is formed.

Therefore, when a wide-angle lens is used, when the image analysis unit 102 calculates the degree of polarization, a predetermined gain α corresponding to the imaging region may be multiplied by the degree of polarization of Expression (1). That is, instead of equation (1), the following equation (6) may be used.
Polarization degree = α × (I (P) −I (S)) / (I (P) + I (S)) (6)

  For example, the gain α may be α = 1 in the central portion of the effective imaging region and α = 1.1 in the peripheral portion according to the transmittance deterioration. As a result, an image having no difference in polarization degree between the image center and the image periphery can be captured.

  Further, the image analysis unit 102 performs known edge extraction on the luminance image and the polarization degree image, and superimposes the luminance image and the polarization degree image after the edge extraction, thereby comparing the obstacle with the case of only the luminance image. The detection performance can be improved.

  As described above, the image processing system 110 according to the present embodiment can capture a color image in addition to the polarization degree image. FIG. 11 is a schematic diagram illustrating a state in which marker detection is performed by an image processing system 110 (only the imaging unit 101 is illustrated) mounted inside the vehicle 100. The imaging unit 101 captures an image of a road sign (for example, a restriction sign for prohibiting entry of a vehicle, etc.) 300 located in front of the vehicle 100, and alerts the driver by voice or image display on the display via the image analysis unit 102. To do.

  There may be a plurality of types of road signs to be subjected to sign detection. For example, the image processing system 110 stores various road sign information in the memory of the image analysis unit 102 in advance, and stores the various road sign information acquired using the imaging unit 101 and the various road sign information stored in the memory. The structure to compare may be sufficient. Further, the image processing system 110 may recognize the acquired various road sign information, convert the recognized various road sign information into a voice synthesis signal, and notify the driver of the various road sign information by voice. Good.

  Further, by using the polarization degree image, it is possible to sharpen the wrinkled image. In general, fog or haze is a phenomenon in which light from the sun appears to be reflected by floating water components in the air. Since reflected light from the floating water component generally has polarization characteristics, an image with reduced influence of wrinkles can be generated by subtracting a predetermined component based on the polarization degree image from the color image.

  As the predetermined component, for example, the average value or median value of the polarization degree of the pixel group A of interest (see FIG. 6) and the polarization degree of other pixel groups adjacent to the pixel group A can be used. In this case, the image analysis unit 102 calculates the average value or the median value for the pixel group corresponding to at least the blurred imaging region (particularly the upper portion of the effective imaging region), and uses them as the corresponding color image. A process of subtracting from the pixel group is performed.

  In addition, phenomena such as fog and haze tend to become darker at higher positions away from the road surface. Therefore, as shown in FIG. 12, the imaging unit 101 for the vehicle 100 is located at a position closer to the road surface than the height of the driver's line of sight (for example, a position closer to the road surface than the center of the tire 111 of the vehicle 100). When installed, it is easy to capture an image that is not affected by fog or haze.

  For example, as shown in FIG. 12, a configuration in which an opening 114 for the imaging unit 101 is provided below the number plate 113 of the bumper 112 of the vehicle 100 and the front and rear of the vehicle 100 are imaged therefrom is preferable. With such a configuration, even in a situation where fog or haze is occurring, it is possible to take an image with a visual field farther than the visual field of the driver's naked eye, and image them to the driver. It can be provided as information.

  Further, the image pickup unit 101 is installed in two locations, the vicinity of the windshield 105 in the vehicle 100 (see FIG. 1) and the position closer to the road surface than the center of the tire 111 shown in FIG. The unit 102 may determine whether to turn on the fog lamp from the difference in the field of view of the two imaging units 101 (difference in the degree of polarization in this embodiment).

  By the way, the imaging region where the polarizer is arranged (hereinafter referred to as a road surface state detection region) is compared with the other imaging region which is not subjected to polarization restriction (hereinafter referred to as a marker detection region). Is about half. Therefore, the amount of exposure at the time of imaging is changed between the polarization degree image generated based on the light transmitted through the road surface detection area and the color image generated based on the light transmitted through the sign detection area. Is preferred.

  Specifically, by the automatic exposure adjustment of the image analysis unit 102, the imaging unit 101 captures the image frame for the polarization degree image at the first exposure amount (exposure time) in the road surface state detection region and detects the sign. The image area may be configured to capture an image frame for a color image with the second exposure amount (exposure time) in the production area. For example, the image analysis unit 102 may be configured to change the exposure amount (exposure time) by controlling the time for each pixel of the image sensor 206 to convert incident light into an electrical signal.

  That is, with respect to the road surface state detection region, the image analysis unit 102 performs automatic exposure adjustment while detecting the amount of light transmitted through the road surface state detection region, so that the imaging unit 101 has the first exposure amount (exposure). The road surface state is imaged at (time). In the marker detection area, the image analysis unit 102 performs automatic exposure adjustment while detecting the amount of light transmitted through the marker detection area, so that the imaging unit 101 is labeled with the second exposure amount (exposure time). The image of is taken. Thereby, it becomes possible to capture an image with an optimum exposure amount for each image.

  Note that the change in the amount of light is large in the label detection region. Specifically, since the illuminance around the vehicle changes from tens of thousands of lux in the daytime to 1 lux or less at night, it is necessary to adjust the exposure time according to the imaging scene. For this, the image analysis unit 102 may perform known automatic exposure control.

  On the other hand, the road surface state detection region can be imaged with a fixed exposure time that is half the exposure time of the marker detection region (imaged with a fixed exposure amount). Thereby, shortening of exposure control time, simplification of exposure control, and the like can be realized.

  FIG. 13 is a flowchart showing the procedure of exposure control in the present embodiment. First, the image analysis unit 102 performs exposure adjustment on a label detection area where no polarizer is arranged. Then, the imaging unit 101 captures an image frame for a color image with the second exposure amount set by the exposure adjustment by the image analysis unit 102 (step S120).

  Next, the image analysis unit 102 analyzes the image frame for the color image captured in step S120 (step S121), and sends out instruction signals for causing the vehicle travel control unit 108 and the like to perform various controls. (Step S122).

  Next, the image analysis unit 102 performs exposure adjustment on the road surface state detection region where the polarizer is disposed. Then, the imaging unit 101 captures an image frame for the polarization degree image with the first exposure amount set by the exposure adjustment by the image analysis unit 102 (step S123).

  Next, the image analysis unit 102 analyzes the image frame for the polarization degree image captured in step S123 (step S124). In step S124, a process of generating a color image with reduced influence of wrinkles by subtracting a predetermined component based on the polarization degree image obtained in step S123 from the color image obtained in step S120 as necessary. May be executed.

  Then, the image analysis unit 102 sends out an instruction signal for causing the headlamp control unit 103, the vehicle travel control unit 108, etc. to perform various controls based on the analysis result in step S124 (step S125).

  Then, the image analysis unit 102 repeatedly executes the processes of steps S120 to S125 until there is a predetermined end instruction (such as an end instruction by the driver of the vehicle 100) (step S126).

  Hereinafter, the spectral characteristics of the spectral filter layer 221 included in the optical filter 205 will be described. FIG. 14 is a graph showing the spectral characteristics of the spectral filter layer 221. As shown in FIG. 14, the spectral filter layer 221 includes incident light in a so-called visible light region having a wavelength range of 400 nm to 670 nm (here, λ1 = 400 nm, λ2 = 670 nm) and a wavelength range of 940 nm to 1000 nm (here, λ3 = 940 nm, λ4 = 1000 nm) in the infrared light region, and has transmittance characteristics that cut incident light in the wavelength range of 670 nm to 940 nm. The transmittance in the wavelength range of 400 nm to 670 nm and the wavelength range of 940 to 1000 nm is preferably 30% or more, and more preferably 90% or more. The transmittance in the wavelength range of 670 nm to 940 nm is preferably 20% or less, and more preferably 5% or less.

  Here, it is assumed that the light projection wavelength of the headlamp 104 of the vehicle 100 overlaps the transmission wavelength range of 940 nm to 1000 nm on the infrared side of the spectral filter layer 221.

  Incident light in the visible light region is used for detecting vehicle periphery information, and incident light in the infrared light region is used for detecting road surface conditions and obstacles at night. The reason why the incident light in the wavelength range of 670 nm to 940 nm is not transmitted is that when the imaging element 206 takes in the incident light in this wavelength range, the obtained captured image data becomes entirely red, and the red sign indicates the red color This is because it may be difficult to extract the red image portion corresponding to the tail lamp.

  Therefore, if the spectral filter layer 221 having the characteristic of cutting most of the wavelength range (670 nm to 940 nm) of the infrared light region as shown in FIG. 14 is formed over the entire effective imaging region, disturbance light can be removed. For example, it is possible to improve the detection accuracy of red signs such as a stop sign in Japan and the identification accuracy of tail lamps. Note that the wavelength range of 940 to 1000 nm and the wavelength range of 400 nm to 670 nm are representative examples of the wavelength range according to the present invention.

  By the way, when the imaging direction of the imaging unit 101 is tilted downward, the hood of the host vehicle may enter the lower part of the imaging area. In this case, sunlight reflected by the bonnet of the host vehicle or the tail lamp of the preceding vehicle becomes disturbance light, which is included in the captured image data, which may cause misidentification of the head lamp of the oncoming vehicle, the tail lamp of the preceding vehicle, and the white line. Become. Even in such a case, in the present embodiment, the spectral filter layer 221 that blocks the light in the wavelength range of 670 nm to 940 nm is formed over the entire effective imaging region, so that the sunlight reflected by the bonnet, the tail lamp of the preceding vehicle, etc. The disturbance light is removed. Therefore, the head lamp of the oncoming vehicle, the tail lamp of the preceding vehicle, and the white line identification accuracy are improved.

  As already described, the optical filter 205 is disposed close to the surface of the image sensor 206 on the imaging lens 204 side. This is because optical crosstalk is more likely to occur between adjacent pixels as the optical filter 205 and the image sensor 206 are separated from each other. Therefore, it is desirable that the gap between the optical filter 205 and the image pickup element 206 is closely bonded by a method such as adhesion so that the gap is 2 μm or less. As a result, the boundary between the road surface state detection region and the sign detection region of the optical filter 205 and the pixel boundary of the image sensor 206 can be easily matched.

  For example, the optical filter 205 and the imaging element 206 may be bonded with a UV adhesive, or may be bonded to the four sides outside the effective imaging area by UV bonding or thermocompression bonding while being supported by the spacer outside the effective imaging area. Good.

  Hereinafter, each part of the optical filter 205 will be described in detail. The substrate 220 is made of a transparent material, such as glass, sapphire, or quartz, that can transmit light in the used wavelength band (visible light region and infrared light region in this embodiment). In the present embodiment, glass, particularly quartz glass (refractive index 1.46) and Tempax glass (refractive index 1.51) which are inexpensive and durable can be suitably used.

  The polarizing filter layer 223 formed on the substrate 220 is composed of a polarizer formed with a wire grid structure as shown in FIG. 15, and the surface on the image sensor 206 side is an uneven surface. The wire grid structure is a structure in which metal wires (conductor lines) made of metal such as aluminum and extending in a specific direction are arranged at a specific pitch. By making the wire pitch of the wire grid structure a sufficiently small pitch (for example, ½ or less) compared to the wavelength band of incident light (for example, 400 nm to 800 nm), the wire grid structure vibrates in parallel to the longitudinal direction of the metal wire. Therefore, it can be used as a polarizer for producing a single polarized light because it reflects most of the light of the electric field vector component and transmits almost the light of the electric field vector component that vibrates in the direction orthogonal to the longitudinal direction of the metal wire.

  In a wire grid polarizer, the extinction ratio generally increases as the cross-sectional area of the metal wire increases, and the transmittance decreases for metal wires having a predetermined width or more with respect to the period width. Further, when the cross-sectional shape orthogonal to the longitudinal direction of the metal wire is a tapered shape, the wavelength dispersion of the transmittance and the polarization degree is small in a wide band, and a high extinction ratio characteristic is exhibited.

  In the present embodiment, the polarizing filter layer 223 is formed in a wire grid structure, and thus has the following effects. The wire grid structure can be formed using a widely known semiconductor manufacturing process. Specifically, after depositing an aluminum thin film on the substrate 220, patterning is performed, and the sub-wavelength uneven structure of the wire grid may be formed by a technique such as metal etching. By such a manufacturing process, it becomes possible to adjust the longitudinal direction of the metal wire, that is, the polarization direction (polarization axis) corresponding to the imaging pixel size (several μm level) of the imaging element 206. Therefore, as in the present embodiment, it is possible to create the polarizing filter layer 223 in which the longitudinal direction of the metal wire, that is, the polarization direction (polarization axis) is different for each imaging pixel.

  Further, since the wire grid structure is made of a metal material such as aluminum, there is an advantage that it is excellent in heat resistance and can be suitably used even in a high-temperature environment such as a vehicle interior that is likely to become high temperature.

  As described above, the polarizer of the polarizing filter layer 223 has a sub-wavelength sized wire grid structure, has low mechanical strength, and a metal wire is damaged by a slight external force. Since it is desired that the optical filter 205 of the present embodiment is placed in close contact with the image sensor 206, there is a possibility that the optical filter 205 and the image sensor 206 come into contact with each other in the manufacturing stage. The optical filter 205 and the image sensor 206 are preferably arranged in parallel, and a planarization layer is preferably formed therebetween.

  The filler 224 is used to flatten the upper surface of the polarizing filter layer 223 in the stacking direction, and is filled in the recesses between the metal wires of the polarizing filter layer 223. As the filler 224, an inorganic material having a refractive index lower than or equal to that of the substrate 220 can be suitably used. In addition, the filler 224 in this embodiment is formed so as to cover the upper surface in the stacking direction of the metal wire portion of the polarizing filter layer 223.

The filler 224 is preferably made of a low refractive index material whose refractive index is as close as possible to the refractive index of air (refractive index = 1) so as not to deteriorate the polarization characteristics of the polarizing filter layer 223. As a specific material, for example, a porous ceramic material formed by dispersing fine pores in ceramics is preferable, and specifically, porous silica (SiO ₂ ), porous magnesium fluoride (MgF). And porous alumina (Al ₂ O ₃ ). The degree of these low refractive indexes is determined by the number and size of pores in the ceramic (porosity). When the main component of the substrate 220 is made of silica crystal or glass, porous silica (n = 1.2-1.26) can be preferably used.

As a method for forming the filler 224, for example, an inorganic coating film (SOG: Spin On Glass) method can be suitably used. Specifically, a solvent in which silanol (Si (OH) ₄ ) is dissolved in alcohol is spin-coated on the polarizing filter layer 223 formed on the substrate 220, and then the solvent component is volatilized by heat treatment, whereby the silanol itself Thus, the filler 224 is formed in such a manner as to cause a dehydration polymerization reaction.

  As described above, in the present embodiment, the upper surface in the stacking direction of the polarizing filter layer 223, that is, the surface on the image sensor 206 side is covered with the filler 224. The situation that damages is suppressed.

  Further, by filling the filler 224 into the recesses between the metal wires in the wire grid structure of the polarizing filter layer 223 as in the present embodiment, it is possible to prevent foreign matter from entering the recesses.

  The spectral filter layer 221 of the present embodiment is made of a multilayer film structure in which high refractive index thin films and low refractive index thin films are alternately stacked. If such a multilayer film structure is adopted, the degree of freedom in setting the spectral transmittance is increased by using light interference, and a specific wavelength (for example, a wavelength band other than red) is obtained by stacking thin films in multiple layers. On the other hand, it is also possible to realize a reflectance close to 100%.

  As described above, the image processing system according to the present embodiment and the vehicle including the image processing system have the color sensor and the polarization filter layer, and thus are expressed in color in the sign recognition in addition to the detection of the road surface state. The attention level can also be detected, and the accuracy of sign recognition can be improved.

(Second Embodiment)
An image processing system according to a second embodiment of the present invention and a vehicle including the image processing system will be described with reference to the drawings. In addition, description is abbreviate | omitted suitably about the structure and operation | movement similar to embodiment already demonstrated.

  In this embodiment, the example which changed the area | region division method of the polarizing filter layer is shown. In the polarization filter layer in the present embodiment, a polarizer for causing the horizontal polarization component P of incident light to enter one of the two G pixels included in each pixel group, and incident light to the other of the two G pixels. A polarizer for allowing the vertical polarization component S to enter is formed.

  FIG. 16 is a schematic diagram illustrating the correspondence between the polarizing filter layer 223 of the optical filter 205 and the pixels of the image sensor 206 shown in FIG. 5 of the first embodiment, and shows another example of the pattern of the polarizer. Is.

  That is, FIG. 16 shows a polarizer that transmits the horizontal polarization component P of incident light so as to correspond to the G pixel disposed in the region g1 and transmits the vertical polarization component S of incident light. Shows an example of being arranged so as to correspond to the G pixels arranged in the region g2. Polarizers are not disposed in the regions r and b, and light incident on these regions is not subject to polarization restrictions. That is, the area where the polarizer is arranged occupies 1/2 of the effective imaging area.

  As a result, from each pixel group, only the pixel value Ir of the R pixel in the region r where polarization is not restricted, the pixel value Ig1 of the G pixel in the region g1 through which only the horizontal polarization component P is transmitted, and only the vertical polarization component S are transmitted. The pixel value Ig2 of the G pixel in the region g2 and the pixel value Ib of the B pixel in the region b where the polarization is not limited are output.

  The image analysis unit 102 obtains a pixel value Ig1 containing only information on the horizontal polarization component P and a pixel value Ig2 containing information on only the vertical polarization component S from each pixel group, and performs known image interpolation processing. By doing so, two types of images, a horizontal polarization component image and a vertical polarization component image, are generated. For example, when forming a horizontal polarization component image, the pixel values of the regions g2, r, and b may be the average value of the pixel values of the region g1 included in the same pixel group and adjacent pixel groups. On the other hand, when forming a vertical polarization component image, the pixel values of the regions g1, r, and b may be average values of the pixel values of the region g2 included in the same pixel group and adjacent pixel groups.

  Further, the image analysis unit 102 obtains the degree of polarization of each pixel from Equation (1) already shown, where I (P) is the pixel value of the horizontal polarization component image and I (S) is the pixel value of the vertical polarization component image. As a result, a polarization degree image independent of luminance information is generated. In the example shown in FIG. 16, the degree of polarization for each pixel group corresponds to (Ig1-Ig2) / (Ig1 + Ig2).

  Further, the image analysis unit 102 generates a color image by using the image interpolation technique as described above. In the present embodiment, the two G pixels in the pixel group output pixel values including information on the horizontal polarization component P and the vertical polarization component S of the incident light. It may be considered that the pixel value is not affected. Here, the pixel value corresponding to two G pixels is 2 × (Ig1 + Ig2).

  That is, since an R image, a G image, and a B image are obtained from twice the sum of the pixel values Ig1 and Ig2 and the pixel values Ir and Ib that are not subjected to polarization limitation, a color image is generated by superimposing these.

  As described above, the image processing system according to the present embodiment and the vehicle including the image processing system can improve the accuracy of the polarization degree image because the value of the horizontal polarization component P is close to the true value as compared with the first embodiment. Also, a color image can be generated by the above calculation without losing resolution.

(Third embodiment)
An image processing system according to a third embodiment of the present invention and a vehicle including the image processing system will be described with reference to the drawings. In addition, description is abbreviate | omitted suitably about the structure and operation | movement similar to embodiment already demonstrated.

  In the present embodiment, another example in which the region dividing method of the polarizing filter layer is changed is shown. The polarizing filter layer in the present embodiment has a polarizer for causing the vertical polarization component S of incident light to enter one of the two G pixels included in each pixel group, and the horizontal polarization of incident light to the other of the two G pixels. A polarizer for making the component P incident and a polarizer for making the horizontal polarization component P of the incident light incident on the R pixel and the B pixel are formed.

  FIG. 17 is a schematic diagram illustrating the correspondence between the polarizing filter layer 223 of the optical filter 205 and the pixels of the image sensor 206 shown in FIG. 5 of the first embodiment, and shows another example of the pattern of the polarizer. Is. That is, FIG. 17 shows that a polarizer that transmits the vertical polarization component S of incident light is disposed so as to correspond to the G pixel that is disposed in the region g1, and a polarizer that transmits the horizontal polarization component P of incident light. Shows an example of being arranged so as to correspond to the G pixels arranged in the areas g2, r, and b. That is, the area where the polarizer is arranged occupies the entire effective imaging area.

  As a result, from each pixel group, the pixel value Ir of the R pixel in the region r where only the horizontal polarization component P is transmitted, the pixel value Ig1 of the G pixel in the region g1 where only the vertical polarization component S is transmitted, and the horizontal polarization component The pixel value Ig2 of the G pixel in the region g2 through which only P is transmitted and the pixel value Ib of the B pixel in the region b through which only the horizontal polarization component P is transmitted are output.

  The image analysis unit 102 obtains a pixel value Ig1 containing only information of the vertical polarization component S and a pixel value Ig2 containing information of only the horizontal polarization component P from each pixel group, and performs known image interpolation processing. By doing so, two types of images, a horizontal polarization component image and a vertical polarization component image, are generated. For example, when forming a vertical polarization component image, the pixel values of the regions g2, r, and b may be the average value of the pixel values of the region g1 included in the same pixel group and adjacent pixel groups. On the other hand, when a horizontal polarization component image is formed, the pixel value of the region g1 may be an average value of the pixel values of the regions g2, r, and b included in the same pixel group.

  Further, the image analysis unit 102 obtains the degree of polarization of each pixel from Equation (1) already shown, where I (P) is the pixel value of the horizontal polarization component image and I (S) is the pixel value of the vertical polarization component image. As a result, a polarization degree image independent of luminance information is generated. In the example shown in FIG. 17, the degree of polarization for each pixel group corresponds to (Ig2−Ig1) / (Ig2 + Ig1).

  Further, the image analysis unit 102 generates a color image by using the image interpolation technique as described above. In the present embodiment, a color image is generated by superimposing the R image obtained from the pixel value Ir, the G image obtained from the pixel value Ig2, and the B image obtained from the pixel value Ib. Focusing on the pixel value, color information can be obtained by Ir + Ig2 × 2 + Ib.

  In general, when an imaging device is installed in a vehicle and an image outside the vehicle is to be captured, unnecessary light reflected by a dashboard or a hood may be superimposed on the captured image data. These may be noise light and cause a reduction in the recognition rate of original label detection, etc., but most of the polarization component of such unnecessary light is the S polarization component. Therefore, these unnecessary lights can be reduced by suppressing the S polarization component.

  Considering this point, the image processing system according to the present embodiment and the vehicle including the image processing system can generate a color image that is not affected by unnecessary light because the color image is generated only by the P polarization component. .

(Fourth embodiment)
An image processing system according to a fourth embodiment of the present invention and a vehicle including the image processing system will be described with reference to the drawings. In addition, description is abbreviate | omitted suitably about the structure and operation | movement similar to embodiment already demonstrated.

  In the present embodiment, another example in which the region dividing method of the polarizing filter layer is changed is shown. In the polarization filter layer according to the present embodiment, a polarizer is formed for making one of the horizontal polarization component P and the vertical polarization component S of incident light enter only one of the two G pixels included in each pixel group. Has been.

  More specifically, the polarizing filter layer in the present embodiment includes a polarizer for causing the horizontal polarization component P of incident light to enter one of two G pixels included in one pixel group, and the one G pixel. A polarizer for causing the vertical polarization component S of the incident light to enter the G pixel that is 1 pixel vertically and horizontally from the horizontal axis, and a horizontal polarization component of the incident light to the G pixel that is diagonally separated from the one G pixel by 1 pixel Polarizers for entering P are periodically formed in units of four pixel groups that are vertically and horizontally adjacent to each other.

  FIG. 18 is a schematic view illustrating the correspondence between the polarization filter layer 223 of the optical filter 205 and the pixels of the image sensor 206. That is, in FIG. 18, a polarizer that transmits the horizontally polarized component P of incident light is disposed in the region g1 of the pixel groups I and IV, and a vertically polarized component S of incident light is transmitted to the region g1 of the pixel groups II and III. The example by which a polarizer is arrange | positioned is shown. Polarizers are not arranged in the regions g2, r, and b of the pixel groups I to IV, and the incident light to these regions is not subjected to polarization limitation. That is, the area where the polarizer is arranged occupies 1/4 of the effective imaging area.

  As a result, from the pixel groups I and IV, the pixel value Ir of the R pixel in the region r where polarization is not limited, the pixel value Ig1 of the G pixel in the region g1 through which only the horizontal polarization component P is transmitted, and the region g2 where polarization is not limited The pixel value Ig2 of the G pixel and the pixel value Ib of the B pixel in the region b where the polarization is not limited are output.

  On the other hand, from the pixel groups II and III, the pixel value Ir from the region r where the polarization is not restricted, the pixel value Ig1 ′ from the region g1 through which only the vertical polarization component S is transmitted, and the pixel value Ig2 from the region g2 where the polarization is not restricted. Then, the pixel value Ib from the region b where the polarization is not limited is output.

  The image analysis unit 102 obtains a pixel value Ig1 including only the horizontal polarization component P from the pixel groups I and IV and a pixel value Ig1 ′ including only the vertical polarization component S from the pixel groups II and III. By obtaining and performing known image interpolation processing, two types of images, a horizontal polarization component image and a vertical polarization component image, are generated.

  Further, the image analysis unit 102 obtains the degree of polarization of each pixel from Equation (1) already shown, where I (P) is the pixel value of the horizontal polarization component image and I (S) is the pixel value of the vertical polarization component image. As a result, a polarization degree image independent of luminance information is generated. In the example shown in FIG. 18, the degree of polarization for each two pixel groups adjacent vertically or horizontally corresponds to (Ig1−Ig1 ′) / (Ig1 + Ig1 ′).

  Further, the image analysis unit 102 generates a color image by using the image interpolation technique as described above. In the present embodiment, the R image obtained from the pixel value Ir not subject to polarization restriction, the G image obtained from the pixel value Ig2 not subject to polarization restriction, and the B image obtained from the pixel value Ib not subject to polarization restriction are superimposed. As a result, a color image is generated. Focusing on the pixel value, color information can be obtained by Ir + Ig2 × 2 + Ib.

  The image processing system of the present embodiment and the vehicle equipped with the image processing system can take a larger amount of light incident on the image sensor because the area where the polarizer is formed is smaller than in the first to third embodiments. As a result, a bright and high-resolution color image can be generated, so that label detection and the like can be suitably performed.

  On the other hand, in the present embodiment, since the calculation of the degree of polarization is performed with 8 pixels (two pixel groups), the resolution is lowered, but it is only necessary to capture a change in a relatively large area of the captured image data (for example, The change of the road surface state etc.) is also adequately suitable for the polarizer pattern in this embodiment.

  If the resolution can be further reduced, the transmission region of the horizontal polarization component P and the transmission region of the vertical polarization component S are further discretely formed (for example, a polarizer is formed every two or three pairs of vertical and horizontal pixel groups). Or the like.

(Fifth embodiment)
An image processing system according to a fifth embodiment of the present invention and a vehicle including the image processing system will be described with reference to the drawings. In addition, description is abbreviate | omitted suitably about the structure and operation | movement similar to embodiment already demonstrated.

  In the present embodiment, another example in which the region dividing method of the polarizing filter layer is changed is shown. In the polarizing filter layer in the present embodiment, a polarizer for making any one of the horizontal polarization component P and the vertical polarization component S of incident light incident is formed as a stripe pattern. Each pixel group is classified into a first pixel group group corresponding to a striped pattern and a second pixel group group in which no polarizer is arranged.

  FIG. 19 is a schematic diagram illustrating the correspondence between the polarization filter layer 223 of the optical filter 205 and the pixels of the image sensor 206 shown in FIG. 18 of the fourth embodiment, and shows another example of the pattern of the polarizer. Is. That is, FIG. 19 shows an example in which polarizers that transmit the horizontal polarization component P of incident light are arranged in all the pixels of the first pixel group group (pixel groups I and III). All the pixels of the second pixel group group (pixel groups II and IV) are not provided with a polarizer, and light incident on these regions is not subject to polarization restrictions. That is, the area where the polarizer is arranged occupies 1/2 of the effective imaging area.

  As a result, from the pixel groups I and III, the pixel value Ir of the R pixel in the region r through which only the horizontal polarization component P is transmitted, the pixel value Ig1 of the G pixel in the region g1 through which only the horizontal polarization component P is transmitted, and the horizontal The pixel value Ig2 of the G pixel in the region g2 through which only the polarization component P is transmitted and the pixel value Ib of the B pixel in the region b through which only the horizontal polarization component P is transmitted are output.

  On the other hand, from the pixel groups II and IV, the pixel value Ir ′ of the R pixel in the non-polarized region r, the pixel value Ig1 ′ of the G pixel in the non-polarized region g1, and the G pixel of the region g2 in the non-polarized region The value Ig2 ′ and the pixel value Ib ′ of the B pixel in the region b where the polarization is not limited are output.

  That is, the pixel values Ig1 and Ig2 contain only the information on the horizontal polarization component P, and the pixel values Ig1 ′ and Ig2 ′ that are not polarization limited correspond to the sum of the horizontal polarization component P and the vertical polarization component S. Contains information. For example, between the G pixel of the pixel group I belonging to the first pixel group group and the G pixel of the pixel group II belonging to the second pixel group group adjacent to the pixel group I, for example, (Ig1 ′ + Ig2 ′) And the difference between (Ig1 + Ig2) can detect only the vertical polarization component S of the G pixel.

  The image analysis unit 102 acquires the pixel values Ig1 and Ig2 and the pixel values Ig1 ′ and Ig2 ′, and performs a known image interpolation process, so that two types of horizontal polarization component image and vertical polarization component image are obtained from the G pixel. Generate an image. The horizontal polarization component image and the vertical polarization component image may be generated from R pixels or B pixels instead of G pixels, or generated from pixels of all three colors (R pixels, G pixels, and B pixels). May be.

  Further, the image analysis unit 102 obtains the degree of polarization of each pixel from Equation (1) already shown, where I (P) is the pixel value of the horizontal polarization component image and I (S) is the pixel value of the vertical polarization component image. As a result, a polarization degree image independent of luminance information is generated. As described above, the degree of polarization for each two adjacent pixel groups is (Ig1 ′ + Ig2′−2 × (Ig1 + Ig2)) / (Ig1 ′ + Ig2 ′ + 2 × (Ig1 + Ig2)) when only G pixels are used. When all three colors of pixels are used, this corresponds to (Ir ′ + Ig1 ′ + Ig2 ′ + Ib′−2 × (Ir + Ig1 + Ig2 + Ib)) / (Ir ′ + Ig1 ′ + Ig2 ′ + Ib ′ + 2 × (Ir + Ig1 + Ig2 + Ib)) .

  Further, the image analysis unit 102 generates a color image by using the image interpolation technique as described above. In the present embodiment, the color image may be generated by output from the pixel groups I and III belonging to the first pixel group group, or output from the pixel groups II and IV belonging to the second pixel group group. May be generated. The former is a color image from which reflected light is excluded, and the latter is a bright color image.

  For example, in the daytime when there is enough light, or at night when there is enough light such as streetlights, color images generated based on pixel values from pixel groups I and III can be used to reflect images caused by sunlight or streetlights. It is possible to remove the spotlight and perform label recognition. On the other hand, for a dark night where there is no reflected light, the image analysis unit 102 may perform switching such as using a color image generated based on the pixel values of the pixel groups II and IV.

  For example, the image analysis unit 102 may determine a pixel group used for generating a color image in accordance with a driver's action (for example, during high beam irradiation), or a light amount sensor (not shown) may be used as a windshield of the vehicle 100. In the case of being attached to 105, a pixel group used for generating a color image may be determined according to the illuminance output value from the sensor.

  As described above, in this embodiment, since the region where the polarizer is formed has a width of 2 pixels, the bonding accuracy between the polarizing filter layer and the imaging element is reduced. In particular, as shown in FIG. 19, the region where the polarizer is formed is in a stripe shape, so that only the position adjustment in the horizontal direction in the figure needs to be precisely performed, and the adjustment for bonding the polarizing filter layer and the image sensor is more effective. It has an easy configuration.

(Sixth embodiment)
An image processing system according to a sixth embodiment of the present invention and a vehicle including the image processing system will be described with reference to the drawings. In addition, description is abbreviate | omitted suitably about the structure and operation | movement similar to embodiment already demonstrated. In the present embodiment, another configuration example of the spectral filter layer in the optical filter is shown.

  FIG. 20 is a schematic cross-sectional view along the light transmission direction showing another example of the structure of the optical filter. FIG. 21 is a schematic front view of the optical filter 205 ′ in the present embodiment as viewed from the sensor substrate 207 side. In other words, in addition to the configuration of the optical filter 205 shown in FIG. A spectral filter layer 222 (second spectral filter layer) that is formed through the polarizing filter layer 223 and the filler 224 and selectively transmits light having a wavelength component in the wavelength range of λ3 to λ4. The surface of the filter layer 222 on the image sensor 206 side is disposed close to the image sensor 206.

  That is, the optical filter 205 ′ has a structure in which the spectral filter layer 221 and the spectral filter layer 222 are overlapped in the light transmission direction.

  Here, the spectral characteristics of the color filter used in the color sensor generally show characteristics as shown in FIG. In FIG. 22, the spectral characteristic of the R pixel is indicated by a solid line, the spectral characteristic of the G pixel is indicated by a broken line, and the spectral characteristic of the B pixel is indicated by a one-dot chain line. That is, with respect to wavelengths shorter than 800 nm, the spectral characteristics of the R pixel, G pixel, and B pixel are different.

  On the other hand, the spectral characteristic of the spectral filter layer 222 included in the optical filter 205 ′ is preferably as shown in FIG. That is, the spectral filter layer 222 may have spectral characteristics having, for example, a near-infrared light region in the wavelength range of 940 nm to 1000 nm as a transmission band (cutting light on a shorter wavelength side than the wavelength 940 nm).

  The near-infrared light region in the wavelength range of 940 nm to 1000 nm is a wavelength band in which the spectral characteristics of all three colors of pixels (R pixel, G pixel, and B pixel) in the spectral characteristics of the color filter shown in FIG. is there. Therefore, for example, when the polarizing filter layer 223 has polarizers corresponding to pixels of all three colors, the image processing system of the present embodiment and the vehicle including the same have luminance in the region where the spectral filter layer 222 is formed. A stable and high-resolution polarization degree image can be generated using pixels of three colors with almost no difference in information. Thereby, it becomes possible to improve the detection accuracy of the road surface state in the region where the spectral filter layer 222 is formed.

  Next, the structure and manufacturing method of the spectral filter layer 222 will be described. Similar to the spectral filter layer 221, the spectral filter layer 222 can be manufactured as a multilayer film structure in which high refractive index thin films and low refractive index thin films are alternately stacked.

  In the present embodiment, since the use wavelength range of the captured image data is substantially from the visible light wavelength band to the infrared light wavelength band, the image sensor 206 having sensitivity in the use wavelength range is employed. Since the spectral filter layer 222 only needs to transmit a part of infrared light, the transmission wavelength range of the multilayer film portion of the spectral filter layer 222 is set to, for example, 900 nm or more, and other wavelength bands are reflected. A filter (see FIG. 23) may be formed.

Such a cut filter is obtained by forming a multilayer film having a configuration of “substrate / (0.125L 0.25H 0.125L) p / medium A” in order from the lower side in the stacking direction of the optical filter 205. It is done. The “substrate” here means the filler 224 described above. In addition, “0.125L” is obtained by setting nd / λ to 1L in the film thickness marking method of a low refractive index material (for example, SiO ₂ ). Therefore, the film of “0.125L” has 1/8 wavelength. It is a film of a low refractive index material having a film thickness that becomes an optical path length. “N” is a refractive index, “d” is a thickness, and “λ” is a cutoff wavelength.

Similarly, “0.25H” is a film thickness marking method of a high refractive index material (for example, TiO ₂ ) in which nd / λ is 1H. Therefore, a film of “0.25H” has a quarter wavelength. It is a film of a high refractive index material having a film thickness such that the optical path length becomes. “P” indicates the number of times the film combination shown in parentheses is repeated (laminated), and the more “p”, the more the influence of ripple and the like can be suppressed. The medium A is intended for air or a resin or adhesive for tight bonding with the image sensor 206.

Further, the spectral filter layer 222 may be a band-pass filter having a transmission wavelength range of 940 to 970 nm as a multilayer film structure that transmits only the infrared wavelength band. Such a band-pass filter is, for example, “substrate / (0.125L 0.5M 0.125L) p (0.125L 0.5H 0.125L) q (0.125L 0.5M 0.125L) r / It can be obtained by producing a multilayer film having a configuration such as “medium A”. If titanium dioxide (TiO ₂ ) is used as the high refractive index material and silicon dioxide (SiO ₂ ) is used as the low refractive index material, a highly weather resistant spectral filter layer 222 can be realized.

  An example of a method for manufacturing the spectral filter layer 222 will be described. First, the multilayer film described above is formed on the filler 224 formed on the substrate 220 and the polarizing filter layer 223. As a method for forming such a multilayer film, a known method such as vapor deposition may be used. Subsequently, the multilayer film is removed at a location corresponding to the non-spectral region (for example, the upper region of the image sensor 206).

  As this removal method, a general lift-off processing method may be used. In the lift-off processing method, a pattern opposite to the target pattern is formed in advance on a layer of the filler 224 with a metal, a photoresist, or the like, a multilayer film is formed thereon, and then a non-spectral region. The multi-layer film corresponding to the above is removed together with the metal and photoresist.

  In this embodiment, since the multilayer structure is adopted as the spectral filter layer 222, there is an advantage that the degree of freedom in setting the spectral characteristics is high.

  In the present embodiment, the spectral filter layer 222 laminated on the filler 224 is not provided with a protective layer like the filler 224. According to the experiments by the present inventors, even if the spectral filter layer 222 is in contact with the image sensor 206, damage that affects the captured image did not occur. The layer is omitted.

  The height of the metal wire (convex portion) of the polarizing filter layer 223 is generally as low as half or less of the wavelength used, while the height of the spectral filter layer 222 increases the transmittance at the cutoff wavelength as the height (thickness) increases. Since the characteristics can be sharpened, the height is about the same as the used wavelength to several times higher.

  As the thickness of the filler 224 increases, it becomes more difficult to ensure the flatness of the upper surface, which affects the characteristics of the optical filter 205, so that there is a limit to increasing the thickness of the filler 224. Therefore, in the present embodiment, the spectral filter layer 222 is not covered with a filler.

  That is, in this embodiment, since the spectral filter layer 222 is formed after the polarizing filter layer 223 is covered with the filler 224, the layer of the filler 224 can be stably formed. In addition, the spectral filter layer 222 formed on the upper surface of the layer of the filler 224 can be optimally formed.

100 Vehicle 101 Imaging unit (imaging means)
102 Image analysis unit (image analysis means)
103 Headlamp control unit 104 Headlamp 105 Windshield (transparent member)
DESCRIPTION OF SYMBOLS 106 Wiper control unit 107 Wiper 108 Vehicle travel control unit 110 Image processing system 111 Tire 112 Bumper 113 Number plate 114 Opening 204 Imaging lens 205,205 'Optical filter 206 Imaging element 206a Pixel array 206b Color filter 207 Sensor substrate 208 Signal processing part 220 Substrate 221 Spectral filter layer (first spectral filter layer)
222 Spectral filter layer (second spectral filter layer)
223 Polarizing filter layer 224 Filler

Japanese Patent No. 2707426 JP 2007-86720 A

Claims (14)

An imaging device having a Bayer array in which R pixels, G pixels, and B pixels are two-dimensionally arrayed, and an optical filter disposed in a preceding stage of the image sensor;
Image analysis means for analyzing the imaging result of the imaging means,
The optical filter is
A substrate that transmits incident light;
A polarizer for causing one of the horizontal polarization component and the vertical polarization component of the incident light to enter only one of the two G pixels included in each pixel group including four pixels adjacent in the vertical and horizontal directions of the image sensor. A polarizing filter layer formed,
The image analysis unit generates a horizontal polarization component image, a vertical polarization component image, and a polarization degree image using the pixel value of the one G pixel and the pixel value of the other G pixel of the two G pixels. And a color image is generated using the pixel value of the other G pixel, the pixel value of the R pixel, and the pixel value of the B pixel. An imaging device having a Bayer array in which R pixels, G pixels, and B pixels are two-dimensionally arrayed, and an optical filter disposed in a preceding stage of the image sensor;
Image analysis means for analyzing the imaging result of the imaging means,
The optical filter is
A substrate that transmits incident light;
A polarizer for causing a horizontal polarization component of the incident light to enter one of two G pixels included in each pixel group including four pixels adjacent in the vertical and horizontal directions of the image sensor, and the other of the two G pixels. A polarizing filter layer on which a polarizer for making the vertically polarized component of the incident light incident is formed,
The image analysis unit generates a horizontal polarization component image, a vertical polarization component image, and a polarization degree image by using the pixel values of the two G pixels, and the pixel values of the two G pixels and the pixels of the R pixel. A color image is generated using a value and a pixel value of a B pixel. An imaging device having a Bayer array in which R pixels, G pixels, and B pixels are two-dimensionally arrayed, and an optical filter disposed in a preceding stage of the image sensor;
Image analysis means for analyzing the imaging result of the imaging means,
The optical filter is
A substrate that transmits incident light;
A polarizer for causing a vertically polarized component of the incident light to enter one of two G pixels included in each pixel group consisting of four pixels adjacent in the vertical and horizontal directions of the image sensor, and the input to the other of the two G pixels. A polarizer for making the horizontal polarization component of the incident light incident, and a polarization filter layer on which a polarizer for making the horizontal polarization component of the incident light incident on the R pixel and the B pixel is formed,
The image analysis unit generates a horizontal polarization component image, a vertical polarization component image, and a polarization degree image using the pixel values of the two G pixels, and also includes a pixel value of the other G pixel and a pixel of the R pixel. A color image is generated using a value and a pixel value of a B pixel. An imaging device having a Bayer array in which R pixels, G pixels, and B pixels are two-dimensionally arrayed, and an optical filter disposed in a preceding stage of the image sensor;
Image analysis means for analyzing the imaging result of the imaging means,
The optical filter is
A substrate that transmits incident light;
A polarizer for causing one of the horizontal polarization component and the vertical polarization component of the incident light to enter only one of the two G pixels included in each pixel group including four pixels adjacent in the vertical and horizontal directions of the image sensor. A polarizing filter layer formed,
The polarizing filter layer is a polarizer for causing a horizontal polarization component of the incident light to enter one of two G pixels included in one pixel group, and is separated from the one G pixel vertically and horizontally by one pixel. A polarizer for causing the vertical polarization component of the incident light to enter the G pixel, and a polarizer for causing the horizontal polarization component of the incident light to enter the G pixel obliquely separated from the one G pixel by one pixel. , Periodically formed in units of four pixel groups adjacent vertically and horizontally,
The image analysis means uses a pixel value of the one G pixel and a pixel value of the G pixel that is one pixel away from the one G pixel vertically or horizontally, and uses a horizontal polarization component image, a vertical polarization component image, And an image with a degree of polarization, and a color image is generated using the other pixel value of the two G pixels, the pixel value of the R pixel, and the pixel value of the B pixel. An imaging device having a Bayer array in which R pixels, G pixels, and B pixels are two-dimensionally arrayed, and an optical filter disposed in a preceding stage of the image sensor;
Image analysis means for analyzing the imaging result of the imaging means,
The optical filter is
A substrate that transmits incident light;
A polarizer for making any one of a horizontal polarization component and a vertical polarization component of the incident light incident, and a polarization filter layer formed as a stripe pattern,
Each pixel group consisting of four pixels adjacent in the vertical and horizontal directions of the image sensor includes a first pixel group group corresponding to the stripe pattern and a second pixel group group in which the polarizer is not arranged. Classified,
The image analysis means includes a pixel value of a pixel group belonging to the first pixel group group in which the polarizer is disposed, and a pixel group belonging to the second pixel group group adjacent to the pixel group. A horizontal polarization component image, a vertical polarization component image, and a polarization degree image are generated using the pixel value of the corresponding pixel, and the G pixel of the pixel group belonging to one of the first and second pixel group groups A color image is generated using the pixel value of R, the pixel value of the R pixel, and the pixel value of the B pixel. The imaging means images vehicle surrounding information including a road surface,
The said image analysis means discriminates a road surface state based on the degree of polarization of the polarization degree image, and performs sign recognition based on the color image. The image processing system described in 1.   The said image analysis means reduces the wrinkles in the said color image by subtracting the predetermined component based on the said polarization degree image from the said color image, The any one of Claims 1-6 characterized by the above-mentioned. The image processing system described in 1. The optical filter is
It is formed on the entire surface of the effective imaging region on the surface of the substrate opposite to the imaging device, and selectively selects light having a wavelength component in the range of wavelengths λ1 to λ2 and λ3 to λ4 (λ1 <λ2 <λ3 <λ4). The image processing system according to any one of claims 1 to 7, further comprising a first spectral filter layer that transmits the light. The optical filter is
And a second spectral filter layer that is formed in a predetermined region of the effective imaging region on the imaging element side surface of the substrate and selectively transmits light having a wavelength component in the wavelength range of λ3 to λ4. The image processing system according to claim 8.   The exposure amount when the imaging unit captures the frame for the polarization degree image is different from the exposure amount when the imaging unit captures the frame for the color image. Item 10. The image processing system according to any one of Items 9 to 9.   When the pixel value of each pixel of the horizontal polarization component image is I (P) and the pixel value of each pixel of the vertical polarization component image is I (S), the image analysis means is α × (I (P ) −I (S)) / (I (P) + I (S)) (where α is a constant), and α in the periphery of the effective imaging region is calculated as the effective imaging region. The image processing system according to claim 1, wherein the image processing system is larger than a central portion of the image processing system.   The image processing system according to any one of claims 1 to 11, wherein the image analysis unit detects an object existing in a dark part based on the polarization degree image. An image processing system according to any one of claims 1 to 12,
Vehicle driving control means for performing driving support control for supporting driving of a vehicle based on a polarization degree image and a color image generated by the image analysis means provided in the image processing system.   14. The transmission wavelength λ3 to λ4 of the first and second spectral filter layers included in the image processing system includes at least a part of a light projection wavelength band of a vehicle headlamp. Vehicle described in.