JP5867806B2 - Imaging device and object identification device using the same - Google Patents

Description

  In the present invention, light from an imaging region is received by a light receiving element array through an optical filter in which one type or two or more types of polarization filter units and color separation filter units are periodically arranged, and a captured image is obtained from an image signal obtained thereby. The present invention relates to an imaging device that calculates pixel values of image information and an object identification device using the imaging device.

  As this type of imaging apparatus, a predetermined imaging area is imaged using an imaging means such as a digital camera, and image information obtained by extracting only a desired wavelength band or only a desired polarization component is extracted from the image signal. Those that generate image information and the like are known. Such an imaging device, for example, receives light from an imaging region incident through an optical filter using a light receiving element array in which a large number of light receiving elements are two-dimensionally arranged, and according to the amount of light received by each light receiving element. Image information of the imaging region is generated from the image signal output from the light receiving element array. Patent Document 1 describes an imaging device using, as an optical filter, for example, a color filter in which color separation filter portions of R (Red), G (Green), and B (Blue) are arranged according to a predetermined arrangement pattern. ing. Patent Document 2 discloses an image pickup apparatus that receives light from an image pickup region that is incident through an optical filter in which two types of polarization filter units (vertical polarization component and horizontal polarization component) are periodically arranged, by a light receiving element array. Have been described. According to such an imaging apparatus, image information obtained by extracting each of a plurality of different optical components in one imaging operation (for example, image information of R, G, and B, vertical polarization component, and horizontal polarization component, respectively) Image information).

  Such an optical component extraction image generated from a plurality of different optical components is used for object identification processing for identifying an object in an imaging region, and thus is image information (monochrome image information) consisting of only luminance information. This is useful in that object identification can be performed with higher accuracy than when object identification processing is performed. Therefore, in recent years, such an imaging apparatus has been used as, for example, an in-vehicle camera, and performs object detection processing based on a captured image obtained by imaging a forward area (imaging area) in the traveling direction of the host vehicle, and the preceding vehicle or the opposite vehicle. It is used for a driver support system that detects an object such as a vehicle and reduces the driving load on the driver of the own vehicle. In addition, it is expected to be widely used in other fields such as an imaging device for an object identification device used for robot control.

  The inventors of the present invention have developed an imaging apparatus that can output such an optical component extracted image at a frame rate of 30 fps in real time, and faced the following problems in developing the imaging apparatus. In the following description, an imaging device that obtains optical component extraction images of the longitudinal polarization component and the lateral polarization component will be described. However, the same applies to an imaging device that obtains image information having different wavelength components such as R, G, and B. .

  In developing such an imaging apparatus, real-time image processing is performed at a frame rate of 30 fps by extracting an optical component extracted image by extracting a longitudinal polarization component and a lateral polarization component from an image signal output from a light receiving element array. It is required to complete within a short period of time enabling video output. At present, in order to enable image processing that satisfies this requirement, in reality, a predetermined image processing program is executed by a general-purpose CPU, and software processing that realizes the image processing is performed instead of FPGA (Field Programmable Gate Array). It is necessary to employ hardware processing that realizes the image processing using dedicated hardware such as

Here, an example of image processing for generating an optical component extraction image from an image signal output from the light receiving element array will be described.
FIG. 31 is an explanatory diagram showing a correspondence relationship between the polarization filter of the optical filter and the light receiving elements of the light receiving element array.
In this figure, each square arranged vertically and horizontally indicates a light receiving element, a hatching area extending obliquely with respect to the direction in which the light receiving elements are arranged indicates an area of a vertical polarizing filter that cuts a vertical polarization component, and areas other than the hatching area indicate The area | region of the horizontal polarization filter which cuts a horizontal polarization component is shown. The optical filter of the example shown in the figure has a band of longitudinal polarization filters and a band of longitudinal polarization filters arranged alternately and the longitudinal direction of the bands is oblique with respect to the arrangement direction of the light receiving elements. And a lateral polarization filter are arranged.

The vertical polarization component V and the horizontal polarization component H of the target pixel (pixel number 5 in the figure) corresponding to one light receiving element on the light receiving element array are, for example, the image signal value I5 corresponding to the amount of light received by the target pixel 5. In addition to the image signal values I1 to I4 and I6 to I9 corresponding to the received light amounts of the eight peripheral pixels 1 to 4 and 6 to 9 that surround the periphery of the target pixel 5, the following equation 1 is used. More demanded. In the formula shown below, a coefficient matrix (calculation parameters) having elements A1 to A18 for calculating the vertical polarization component V and the horizontal polarization component H of the pixel of interest 5 from the image signal values I1 to I9 is expressed as follows. This is called a pseudo inverse matrix parameter.

This pseudo inverse matrix parameter can be obtained in advance as follows.
First, the entire light receiving element array is irradiated with 100% longitudinally polarized sample light with a substantially uniform light intensity V through the optical filter shown in FIG. 31, and each light receiving element (the pixel number of the light receiving element is set to i). ) To obtain the sample longitudinal polarization intensity avi according to the amount of light received. Similarly, the entire light-receiving element array is irradiated with 100% laterally polarized sample light with an almost uniform light intensity H through an optical filter, and the sample laterally polarized light corresponding to the amount of light received by each light-receiving element at that time. Get strength ahi. It can be considered that the sample longitudinal polarization intensity avi and the sample transverse polarization intensity ahi of each light receiving element thus obtained are proportional to the sensitivities of the longitudinal polarization component and the lateral polarization component received by the light receiving element, respectively. it can. Therefore, if the intensity of light transmitted through the optical filter and received by each light receiving element i is Ii, Ii = avi × V + ahi × H is established. Then, it is assumed that the incident light is the same with respect to a total of nine pixels 1 to 9 of 3 × 3 pixels centered on the target pixel 5, and the determinant represented by the following equation 2 is set for these pixels 1 to 9. Get. In this determinant, an inverse matrix is obtained for sensitivity matrix parameters having av1 to 9 and ah1 to 9 as elements. This inverse matrix becomes the pseudo inverse matrix parameter described above. However, in order to optimize for hardware processing, the pseudo inverse matrix parameter obtained in this way may be slightly modified.

  Here, the optical filter in which the vertical polarization filter and the horizontal polarization filter are alternately arranged has been described as an example. However, the optical filter in which the vertical polarization filter and no filter (area that transmits light as it is) are alternately arranged, Even in the case of an optical filter in which a lateral polarization filter and no filter are alternately arranged, the pseudo inverse matrix parameter can be obtained by the same method as that described above. However, when such an optical filter is used, at the time of actual imaging, the light receiving element array does not receive either one of the horizontally polarized light component and the vertically polarized light component. For this reason, it is necessary to obtain a polarization component that does not receive light by using an image signal value of a polarization component that is received through the polarization filter and a non-polarization image signal value that is received through the absence of the filter. In this case, for example, the pseudo inverse matrix parameter may be generated using the fact that the amount of light received by non-polarized light approximates the sum of the amount of light received by the longitudinally polarized light component and the amount of light received by the horizontally polarized light component.

As described above, the pixel value of the target pixel is calculated using the calculation parameter such as the pseudo inverse matrix parameter and the image signal value I5 of the target pixel 5 and the peripheral pixels 1 to 4 and 6 to 9 located around the target pixel. When calculating from an image signal value group composed of the image signal values I1 to I4 and I6 to I9, if the image signal value group includes an abnormal value, a normal value as the pixel value of the target pixel Cannot be calculated. For example, a pixel whose image signal value is zero (a pixel where the so-called light reception amount is zero and blackening occurs) or a pixel whose image signal value is the upper limit (a pixel where the so-called light reception amount is saturated) is usually In many cases, it cannot exist when receiving light from within the imaging region, and the pixel value is often an abnormal pixel value. When such an abnormal pixel value is included in the image signal value group for calculating the target pixel, an abnormal value deviating from the original pixel value is calculated as the pixel value of the target pixel. When a pixel value indicating an abnormal value is included in the captured image, there is a problem in that the processing result is abnormal when used as it is for subsequent information processing using the pixel value of the captured image. . For example, in the case of object identification processing for identifying an object based on the pixel value of the captured image, the object identification rate is reduced or the object is erroneously identified.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an imaging apparatus capable of avoiding an abnormal processing result from being generated in subsequent information processing based on the pixel value of the captured image. And an object identification device using the same.

To achieve the above object, the present invention is, through the selective optical filter that the transmitted was Ru selection filter regions periodically arranged a predetermined optical component, the light receiving element in which the light-receiving elements are arranged two-dimensionally The array receives light from the imaging region, and outputs an image signal including an image signal value corresponding to the amount of light received for each unit pixel corresponding to one light receiving element or two or more light receiving elements on the light receiving element array. an imaging unit, a pixel value of a unit pixel corresponding to the image signal output from the imaging means, using a predetermined calculation parameters, image signal value of the unit pixel and the unit of peripheral pixels located around the pixel image in the image pickup apparatus having a pixel value calculating means for calculating from the composed image signal value group from the signal value on the basis of the image signal values constituting the image signal value group, the image signal constituting the image signal value group in the, Has an abnormality signal value determining means for determining whether an abnormality determination condition is satisfied image signal values that are out of the constant range of values are included, the pixel value calculating means, constituting the image signal value group when the upper Kikoto normally satisfies the determination condition image signal value in the image signal values are included to the abnormality signal value determination means determines the without the predetermined operation parameters, abnormal value It is characterized in that processing for setting the abnormal instruction value indicating that the pixel value of the unit pixel corresponding to the image signal value group is executed.

In the present invention, the abnormality determination that the image signal value constituting the image signal value group referred to when the pixel value of the unit pixel is calculated using the predetermined calculation parameter is out of the predetermined value range. When the abnormal signal value determining means determines that an image signal value satisfying the condition is included, an abnormal instruction value indicating an abnormal value is assigned to the pixel value. Accordingly, the subsequent information processing using the pixel values of the captured image, can be a pixel value of the unit pixel to grasp an abnormal value based on abnormal determination condition is satisfied abnormal image signal value. As a result, in the subsequent information processing, the pixel value of the unit pixel can be handled as an abnormal value, so that the pixel value of the unit pixel indicating the abnormal value is processed as it is as a normal pixel value. It is possible to avoid abnormal results.

  As described above, according to the present invention, it is possible to obtain an excellent effect that an abnormal processing result can be avoided in subsequent information processing based on the pixel value of the captured image.

It is a functional block diagram of the driver assistance system concerning an embodiment. It is a flowchart which shows the outline | summary of the vehicle detection process in the driver | operator assistance system. It is a flowchart which shows the flow of the process for discriminating a road surface and a solid object. It shows a differential polarization degree image obtained by imaging with a polarization camera mounted on a vehicle traveling on a highway. The image after performing the discrimination | determination process of a road surface and a solid object about the same differential polarization degree image is shown. It is a flowchart for determining a vehicle candidate area | region from the area | region determined to be a solid object area | region. It is explanatory drawing which shows schematic structure of the polarization camera which can be used for the driver | operator assistance system. It is a cross-sectional enlarged view which shows the optical filter and image sensor of the polarization camera. It is explanatory drawing which shows the correspondence of the polarization filter of the same optical filter, and the pixel on an image sensor. It is explanatory drawing for demonstrating the method of classifying the light receiving element on the image sensor for every unit processing area. It is explanatory drawing which shows distribution of four types of common unit pixels in one unit processing area. (A) is explanatory drawing which shows the common unit pixel distribution in 3x3 pixel used when performing an image process by making a 1st common unit pixel into an attention pixel. FIG. 6B is an explanatory diagram illustrating a common unit pixel distribution within 3 × 3 pixels used when image processing is performed using the second common unit pixel as a target pixel. (C) is an explanatory diagram showing a common unit pixel distribution within 3 × 3 pixels used when image processing is performed using the third common unit pixel as a target pixel. (D) is an explanatory diagram showing a common unit pixel distribution in 3 × 3 pixels used when image processing is performed with a fourth common unit pixel as a target pixel. Example of distribution of unit processing area in which pseudo inverse matrix parameter is stored and unit processing area in which pseudo inverse matrix parameter is not stored, using schematic diagram in which light receiving element on same image sensor is divided for each unit processing area It is explanatory drawing which showed. It shows an example of a differential polarization degree image obtained using pseudo inverse matrix parameters (total data amount of pseudo inverse matrix parameters = about 26 Mbytes) prepared individually for all pixels. It shows an example of a differential polarization degree image (total data amount of pseudo inverse matrix parameters = approximately 0.025 Mbytes) obtained by using the same pseudo inverse matrix parameters repeatedly and performing the above-described interpolation processing. It is explanatory drawing which shows the correspondence of the optical filter which concerns on the modification 1, and the pixel on an image sensor. FIG. 10 is an explanatory diagram showing a distribution of six types of common unit pixels existing in one unit processing region in Modification 1. It is explanatory drawing which shows the correspondence of the optical filter which concerns on the modification 2, and the pixel on an image sensor. FIG. 12 is an explanatory diagram showing a distribution of 12 types of common unit pixels existing in one unit processing area in Modification 2. It is a block diagram which shows schematic structure of the polarization camera which concerns on an Example. It is a block diagram which shows the internal structure of the difference polarization degree image generation part provided in the same polarization camera. It is an output timing chart of the BRAM address calculation part provided in the difference polarization degree image generation part. It is an internal block diagram of the coefficient generation part provided in the difference polarization degree image generation part. It is explanatory drawing which shows the division of the unit processing area | region in an Example. It is an address map of a BRAM for storing pseudo inverse matrix parameters in which pseudo inverse matrix parameters are stored. It is an internal block diagram of the polarization separation process part provided in the same differential polarization degree image generation part. It is an internal block diagram of the difference polarization degree calculation part provided in the same difference polarization degree image generation part. It is an internal block diagram of the difference polarization degree image calculation part provided in the difference polarization degree image generation part. It is an address map of the SPI flash memory in which pseudo inverse matrix parameters are stored. It is an address map of the pseudo inverse matrix parameter stored in the sector 24 of the SPI flash memory. It is explanatory drawing which shows the correspondence of the polarizing filter of an optical filter, and the light receiving element of a light receiving element array.

Hereinafter, an embodiment in which an imaging device according to the present invention is applied to a driver support system for reducing the driving load of a driver of the own vehicle will be described.
First, a schematic configuration and an operation outline in the driver assistance system will be described.
FIG. 1 is a functional block diagram of the driver assistance system according to the present embodiment.
FIG. 2 is a flowchart showing an outline of vehicle detection processing in the driver assistance system of the present embodiment.
A landscape around the host vehicle including a road surface (moving surface) on which the vehicle travels is photographed by a polarization camera 10 as an imaging device mounted on a vehicle (not shown) (S1). Thereby, the vertical polarization intensity and the horizontal polarization intensity for each unit pixel (hereinafter simply referred to as “pixel”) corresponding to one light receiving element of the polarization camera 10 are obtained (S2). In the present embodiment, the differential polarization degree is calculated for each unit pixel from the vertical polarization intensity and the horizontal polarization intensity of each pixel obtained in this way (S3). From this differential polarization degree, a differential polarization degree image with the differential polarization degree of each pixel as the pixel value can be generated. Here, “longitudinal polarization intensity” means the light intensity of polarized light having a polarization direction parallel to the column (vertical column) of the light receiving element array in the imaging apparatus installed in the vehicle. The “polarization intensity” means the light intensity of polarized light having a polarization direction parallel to a row (horizontal row) of the light receiving element array in the imaging device installed in the own vehicle.

The differential polarization degree is obtained from the calculation formula shown in the following formula (3). That is, the differential polarization degree is the ratio of the difference value (luminance difference value) between the lateral polarization intensity and the longitudinal polarization intensity to the total value (luminance total value) of the transverse polarization intensity and the longitudinal polarization intensity. In addition, the differential polarization degree can be paraphrased as a difference value between the ratio of the P deflection intensity with respect to the total luminance value (lateral differential polarization degree) and the ratio of the S deflection intensity with respect to the total luminance value (vertical differential polarization degree). . In the present embodiment, a case where the longitudinal polarization intensity is subtracted from the lateral polarization intensity will be described, but the lateral polarization intensity may be subtracted from the longitudinal polarization intensity.
Differential polarization degree = (transverse polarization intensity−longitudinal polarization intensity) / (transverse polarization intensity + longitudinal polarization intensity) (3)

In the present embodiment, differential polarization degree image data is output from the polarization camera 10, and the differential polarization degree image data is input to the road surface / three-dimensional object determination unit 21. The road surface / three-dimensional object discriminating unit 21 discriminates between an image area showing a road surface and an image area showing a three-dimensional object from the differential polarization degree image output from the polarization camera 10 (S4).
FIG. 3 is a flowchart showing a flow of processing for discriminating a road surface and a three-dimensional object.
When the road surface / three-dimensional object discriminating unit 21 receives the differential polarization degree image data from the polarization camera 10, first, a threshold value for binarizing the differential polarization degree image is set (S 41), and the difference is calculated using the threshold value. The polarization degree image is binarized (S42). Specifically, a binary image is created by assigning “1” to pixels having a differential polarization degree equal to or greater than a predetermined threshold and assigning “0” to pixels that do not. Thereafter, in the binarized image, when pixels assigned with “1” are close to each other, a labeling process for recognizing them as one image area is performed (S43). As a result, a set of a plurality of adjacent pixels having a high difference polarization degree is extracted as one high difference polarization degree region. Each high differential polarization degree region extracted in this way is compared with road surface feature data stored in a storage unit (not shown) as road surface feature data storage means, and the high difference polarization degree region that matches the road surface feature data. Is extracted as a road surface area (S44). Then, the shape of the road surface area extracted in this way is estimated by comparing with the road surface shape pattern (S45), and both ends of the road surface area, that is, road edge lines are specified. Thereafter, the remaining image area other than the extracted road surface area is extracted as a three-dimensional object area in which a three-dimensional object is projected (S46).

FIG. 4 shows a differential polarization degree image obtained by imaging with a polarization camera 10 mounted on a vehicle traveling on a highway.
FIG. 5 shows an image after the road surface and the three-dimensional object are subjected to the differential polarization degree image.
The image area of the black part in the image shown in FIG. 5 is a location extracted as a road surface area. Moreover, the line shown with the dotted line in the image shown in FIG. 5 is a road edge line.

The vehicle candidate area determination unit 22 determines a vehicle candidate area from the areas determined to be a three-dimensional object area in S46 using the characteristics of the differential polarization degree of light from the window glass of the vehicle. (S5).
FIG. 6 is a flowchart for determining a vehicle candidate area from among the areas determined to be solid objects areas.
The vehicle candidate area determination unit 22 determines whether or not the differential polarization degree is equal to or greater than a predetermined threshold for each solid object area determined by the road surface / three-dimensional object determination unit 21 (is within a predetermined numerical range). No) is detected, and a three-dimensional object region having a differential polarization degree equal to or greater than a predetermined threshold is detected (S51). Next, it is checked whether or not the area of the three-dimensional object region detected in this way is within the area range corresponding to the window glass of the vehicle (S52). Then, it is checked whether or not the three-dimensional object region that has passed this check has a shape corresponding to the window glass of the vehicle, and the three-dimensional object region that has passed this check is determined as the vehicle rear region (S53).

  The vehicle identification unit 23 identifies whether or not the vehicle candidate region determined by the vehicle candidate region determination unit 22 is a vehicle region by a pattern matching method using the vehicle feature amount pattern stored in the vehicle feature amount pattern storage unit 24. To do. As this pattern matching method, known methods can be widely used.

  In the present embodiment, for example, a monochrome luminance image (front) generated by using the luminance data calculated by the monochrome image processing unit is displayed on the display unit (display) 25 which is an in-vehicle information notification unit composed of a CRT, a liquid crystal, or the like. (View image) is displayed, and information indicating an area in which the other vehicle is displayed is displayed as information useful for the driver in a display form that is easy for the driver to see. According to this, for example, even in a situation where it is difficult for the driver to visually recognize the other vehicle, the driver can view the relative position between the host vehicle and the other vehicle by looking at the front view image on the display unit. The positional relationship can be grasped, and it is easy to travel safely without colliding with another vehicle.

  In addition, the vehicle control unit 26 of the present embodiment performs a process of grasping the relative positional relationship between the own vehicle and the other vehicle from the position information of the vehicle area identified by the vehicle identification unit 23, for example. It is determined whether or not the vehicle is approaching the vehicle, and processing for generating an alarm sound or the like when the vehicle approaches another vehicle is performed. Or when approaching another vehicle, you may perform the process which performs an automatic brake function and reduces the traveling speed of the own vehicle.

Hereinafter, the configuration and operation of the polarization camera 10 which is a characteristic part of the present invention will be described in detail.
FIG. 7 is an explanatory diagram showing a schematic configuration of the polarization camera 10 in the present embodiment.
The polarization camera 10 is mainly output from the image pickup lens 1, the optical filter 2, a sensor substrate 3 on which an image sensor 4 having a light receiving element array in which light receiving elements are two-dimensionally arranged, and a sensor substrate 3. A / D converter 5 that converts an analog electrical signal (the amount of light received by each light receiving element on image sensor 4) into a digital electrical signal, and a digital electrical signal (pixel data) output from A / D converter 5 ), And a signal processing unit 6 that generates and outputs differential polarization degree image data. Light from the imaging region including the subject (test object) passes through the imaging lens 1, passes through the optical filter 2, and is converted into an electrical signal corresponding to the light intensity by the image sensor 4. When an electrical signal (analog signal) output from the image sensor 4 is input to the A / D conversion unit 5, a digital signal indicating the amount of light received by each pixel on the image sensor 4 as pixel data from the electrical signal. Are output to the signal processing unit 6 together with the vertical synchronizing signal and horizontal synchronizing signal of the image.

FIG. 8 is an enlarged cross-sectional view showing the optical filter 2 and the image sensor 4.
The image sensor 4 is an image sensor using a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like, and a photodiode 4a is used as a light receiving element thereof. The photodiodes 4a are two-dimensionally arranged for each pixel, and a microlens 4b is provided on the incident side of each photodiode 4a in order to increase the light collection efficiency of the photodiodes 4a. The image sensor 4 is bonded to a PWB (printed wiring board) by a technique such as wire bonding to form a sensor substrate 3. Since the wavelength range of the image to be captured is substantially visible light, an image sensor 4 having sensitivity in the visible light range may be selected. The image sensor 4 of the present embodiment has a configuration in which light receiving elements (photodiodes 4a) of 480 × 752 arranged at equal intervals in the vertical and horizontal directions orthogonal to each other are effective pixels, and one photodiode 4a. Is a unit pixel.

The optical filter 2 is disposed close to the surface of the image sensor 4 on the microlens 4b side. On the surface of the optical filter 2 on the image sensor 4 side, as shown in FIG. 9, there are a transmission region 2a that is a non-selection filter region that transmits light as it is, and a selection filter region that selectively transmits only the lateral polarization component. The horizontal polarization filter region 2b has stripe-shaped region division patterns alternately arranged so as to incline in both the vertical direction (y direction in the drawing) and the horizontal direction (x direction in the drawing) of the image sensor 4. doing. Specifically, when the pixel interval (distance between the centers of the photodiodes 4a) in the vertical direction of the image sensor 4 is n and the pixel interval in the horizontal direction is m, the vertical direction of the transmission region 2a and the horizontal polarization filter region 2b. The optical filter 2 is configured such that the length N satisfies the following formula (1) and the lateral length M of the transmission region 2a and the lateral polarization filter region 2b satisfies the following formula (2). As shown in FIG. 9, the optical filter 2 of the present embodiment is an example of A = 1 and B = 2 (that is, an example of B / A = 2).
N = A × n (1)
M = B × m (2)

In the present embodiment, as shown in FIG. 9, the ratio of the light receiving area in one pixel of each light transmitted through the two types of filter regions (the transmission region 2a and the lateral polarization filter region 2b) provided in the optical filter 2 is as follows. Four types of common unit pixels that are equal to each other are configured to periodically exist.
More specifically, the first common unit pixel receives the horizontally polarized light component transmitted through the horizontally polarized filter region 2b in the small region at the upper right corner of the pixel, like the pixel of pixel number 1 shown in FIG. These are pixels having a pattern for receiving the non-polarized light transmitted through the transmission region 2a in the remaining large region.
Like the pixel number 2 shown in FIG. 9, the second common unit pixel receives non-polarized light transmitted through the transmission region 2a in a small region at the lower left corner of the pixel, and horizontally in the remaining large region. It is a pixel having a pattern for receiving a laterally polarized light component transmitted through the polarizing filter region 2b.
The third common unit pixel receives the non-polarized light transmitted through the transmission region 2a in the small region at the upper right corner of the pixel, as in the pixel of pixel number 3 shown in FIG. It is a pixel having a pattern for receiving a laterally polarized light component transmitted through the polarizing filter region 2b.
The fourth common unit pixel, like the pixel of pixel number 5 shown in FIG. 9, receives the horizontally polarized light component transmitted through the horizontally polarized filter region 2b in the small region at the lower left corner of the pixel, and in the remaining large region. It is a pixel of the pattern which receives the non-polarized light which permeate | transmitted the transmissive area | region 2a.

In the signal processing unit 6 of the polarization camera 10 according to the present embodiment, the vertical polarization of the pixel of interest (pixel number 5 in FIG. 9) using the pixel data output from the A / D conversion unit 5 and the pseudo inverse matrix parameter. The component V and the lateral polarization component H are calculated from the formula shown in the following equation 3. That is, in the present embodiment, the vertical polarization component V and the horizontal polarization component H of the target pixel 5 are not only the pixel data I5 of the target pixel 5 but also the eight peripheral pixels 1 to 4 surrounding the target pixel 5. Calculations are also made using 6 to 9 pixel data I1 to I4 and I6 to I9. As the pseudo inverse matrix parameter necessary for this calculation, a parameter obtained in advance and stored in an SPI flash memory as a parameter storage means described later is used.

  In the image processing executed by the signal processing unit 6 of the present embodiment, the effective pixels (480 × 752) of the image sensor 4 are divided into 30 equal parts and 47 equal parts as shown in FIG. , A processing process is performed in which the vertical polarization component V and the horizontal polarization component H of the pixel of interest 5 are calculated by the equation shown in the above equation 3 by dividing the pixel processing area into 1410 unit processing regions each composed of 16 × 16 pixels. When divided in this way, each unit processing region has two or more common unit pixels of the same type for each of the four types of common unit pixels described above.

The distribution of the four types of common unit pixels in one unit processing region is shown in FIG.
When each of the four types of common unit pixels is a target pixel, the distribution of the four types of common unit pixels in the 3 × 3 pixels used for the image processing is shown in FIGS. .
For the four types of common unit pixels, the first common unit pixel is indicated by a circled number 1, the second common unit pixel is indicated by a circled number 2, the third common unit pixel is indicated by a circled number 3, and the fourth common When the unit pixel is indicated by a circled numeral 4, in the case of the optical filter 2 configured as shown in FIG. 9, the common unit pixel distribution in each unit processing region is as shown in FIG. In such a distribution, as described above, the distribution of the four types of common unit pixels in the 3 × 3 pixels used for the image processing using the first common unit pixel as the target pixel is which one in the unit processing region. The first common unit pixel is also as shown in FIG. That is, when the first common unit pixel is the target pixel, the 3 × 3 common unit pixel distribution is uniquely determined. Similarly, the second common unit pixel, the third common unit pixel, and the fourth common unit pixel are uniquely determined.

  As described above, regarding the same type of common unit pixels having the same light receiving area ratio, if the pixels are close to each other on the image sensor 4, pseudo inverse matrix parameters (calculation parameters) used in the image processing are mutually different. Approximate. In particular, if not only the light receiving area ratio but also the same light receiving position in the pixel of each light transmitted through the transmission region 2a and the lateral polarization filter region 2b is a common unit pixel of the same type, the pseudo inverse matrix parameter The closeness of is further increased. Therefore, even if the same pseudo inverse matrix parameter is used repeatedly as the pseudo inverse matrix parameter of the same type of common unit pixel located close to each other, the calculation error can be kept within an allowable range.

  Therefore, in the present embodiment, the effective pixels of 480 × 752 on the image sensor 4 are divided into 1410 unit processing regions composed of 16 × 16 pixels adjacent to each other. The number of divisions is appropriately set according to the allowable calculation error. In other words, even with the same type of common unit pixel, the closer the distance between each other, the lower the approximation, so that the range of the same type of common unit pixel that uses the same pseudo inverse matrix parameter is allowed to have a calculation error. The number of divisions is set so as to be limited to a range located at a distance at which the closeness within the range can be obtained.

As the number of divisions increases, the approximation of the original pseudo inverse matrix parameter corresponding to the same type of common unit pixel that uses the same pseudo inverse matrix parameter is higher, so a smaller calculation error is required. The number of pseudo inverse matrix parameters (data amount) increases.
Specifically, in the case of this embodiment, since the number of elements of one pseudo inverse matrix parameter is 18 and the data amount per element is 4 bytes, the data amount per pseudo inverse matrix parameter Is 72 bytes. When dividing into 1410 unit processing areas as in the present embodiment, four pseudo inverse matrix parameters are reused in each unit processing area, so that 4 × 1410 = 5640 pseudo inverse matrix parameters in total. Is required. Therefore, the total data amount of the pseudo inverse matrix parameter is 406080 bytes (about 0.4 Mbytes). On the other hand, if one unit processing area is divided into 8 × 8 pixels, the number of divisions is 5640, and a total of 4 × 5640 = 22560 pseudo inverse matrix parameters are required. In this case, the total data amount of the pseudo inverse matrix parameter is 1624320 bytes (about 1.6 Mbytes).
Conversely, if there is a margin for calculation error, the number of segments is reduced, the number of common unit pixels of the same type that reuse the same pseudo inverse matrix parameter is increased, and the number of pseudo inverse matrix parameters (data amount) is further increased. You may make it reduce.

In the present embodiment, the data amount per element of the pseudo inverse matrix parameter may be reduced to, for example, 1 byte. In this case, the total data amount of the pseudo inverse matrix parameter can be further reduced to 101520 bytes (about 0.1 Mbyte).
In particular, in this embodiment, the four types of pseudo inverse matrix parameters that are reused in each unit processing area are the sum of the pseudo inverse matrix parameters obtained for all of the 16 × 16 pixels that constitute the unit processing area, You may normalize so that the absolute value of the element with the largest absolute value may become 127. With this normalization, the data amount per element of the pseudo inverse matrix parameter is 1 byte. Even if the data amount is reduced in this way, a differential polarization degree with good calculation accuracy can be calculated.

  In the present embodiment, in order to further reduce the total data amount of the pseudo inverse matrix parameter, the SPI flash memory includes a pseudo inverse matrix for a part of the 1410 unit processing areas described above. Does not memorize parameters. For a unit processing area (specific unit processing area) in which no pseudo inverse matrix parameter is stored in the SPI flash memory, the pseudo inverse matrix parameter of the unit processing area in which the pseudo inverse matrix parameter is stored in the SPI flash memory is used. Interpolation processing is performed, and image processing is performed using the calculated pseudo inverse matrix parameter generated by the interpolation processing.

FIG. 13 is a schematic diagram in which the light receiving elements on the image sensor are divided for each unit processing region. The unit processing region in which the pseudo inverse matrix parameter is stored and the unit processing region in which the pseudo inverse matrix parameter is not stored are illustrated. It is explanatory drawing which showed an example of distribution.
In this embodiment, as shown by the bold lines in FIG. 13, 1410 unit processing areas are divided into 4 units of 2 × 2 so as to be 1 unit. Then, pseudo inverse matrix parameters are stored in the SPI flash memory at a rate of one area per unit. As a result, for the unit processing area in which the pseudo inverse matrix parameter is not stored in the SPI flash memory, the pseudo inverse matrix parameter is located between at least two unit processing areas stored in the SPI flash memory.

  Here, as a result of intensive research, the present inventor has found that the unit processing areas adjacent to each other on the image sensor 4 have a fixed relationship (pseudo inverse matrix parameters (calculation parameters) used in each area along the arrangement direction ( It was found that the relationship is close to linear. Therefore, by using this relationship, the pseudo inverse matrix parameter of the unit processing region (specific unit processing region) located between two unit processing regions that are separated from each other can be converted into the pseudo inverse matrix parameter of the two unit processing regions. Can be generated with high accuracy. Then, using the pseudo inverse matrix parameter generated in this way, the calculation error for the specific unit processing region can be within an allowable range.

  Therefore, in the present embodiment, for the unit processing area in which the pseudo inverse matrix parameter is stored in the SPI flash memory, the vertical polarization component V and the lateral polarization component H are calculated using the pseudo inverse matrix parameter in the SPI flash memory. On the other hand, for the unit processing area in which the pseudo inverse matrix parameter is not stored in the SPI flash memory, the longitudinal polarization component V and the lateral polarization component H are calculated using the pseudo inverse matrix parameter generated by the interpolation process.

  Specifically, as shown in FIG. 13, for the unit processing area A, the pseudo inverse matrix parameter corresponding to the area A is read from the SPI flash memory and used. On the other hand, for the unit processing region α located between the unit processing region A and the unit processing region B, the signal processing unit 6 reads out the pseudo inverse matrix parameters of these regions A and B from the SPI flash memory. An interpolation process using two pseudo inverse matrix parameters is performed to generate an interpolation pseudo inverse matrix parameter. Then, the longitudinal polarization component V and the lateral polarization component H of the pixel in the unit processing region α are calculated using the interpolation pseudo inverse matrix parameter. Further, for the unit processing region β located between the unit processing region A and the unit processing region E, the signal processing unit 6 reads the pseudo inverse matrix parameters of these regions A and E from the SPI flash memory. An interpolation process using two pseudo inverse matrix parameters is performed to generate an interpolation pseudo inverse matrix parameter. Then, the longitudinal polarization component V and the lateral polarization component H of the pixel in the unit processing region β are calculated using the interpolation pseudo inverse matrix parameter. For the unit processing region γ located between the unit processing region A and the unit processing region F or between the unit processing region B and the unit processing region E, the signal processing unit 6 uses these regions A, F or The pseudo inverse matrix parameters of the regions B and E are read from the SPI flash memory, and interpolation processing using the two pseudo inverse matrix parameters is performed to generate an interpolation pseudo inverse matrix parameter. Then, the longitudinal polarization component V and the lateral polarization component H of the pixel in the unit processing region γ are calculated using the interpolation pseudo inverse matrix parameter.

  The method of interpolation processing is appropriately selected according to a certain relationship between the pseudo inverse matrix parameters of the two unit processing regions separated from each other and the original pseudo inverse matrix parameters of the specific unit processing region located between them. However, linear interpolation processing is preferable in this embodiment. When performing linear interpolation processing by hardware using FPGA or the like, it is preferable to devise so that only bit shift is required. Therefore, in the present embodiment, as described above, the pseudo inverse matrix parameters are stored in the SPI flash memory at a ratio of 1 area per 2 × 2 4 areas, but 1 area per 16 areas and 1 area per 64 areas. It is also possible to use the area ratio. However, in terms of the accuracy of the interpolation process, it is preferable to store the pseudo inverse matrix parameters in the SPI flash memory at a ratio of 1 area per 2 × 2 4 areas as in the present embodiment.

  When preparing the pseudo inverse matrix parameters individually for all the pixels, the data amount of all the pseudo inverse matrix parameters is about 26 Mbytes, and the memory for storing the pseudo inverse matrix parameters is a large memory of 26 Mbytes or more. When a capacity memory is required, according to the present embodiment, the data amount of all pseudo inverse matrix parameters can be reduced to about 0.4 Mbytes by using the same pseudo inverse matrix parameters, and further, interpolation processing is performed. The amount of data can be reduced to ¼ by using. Therefore, according to the present embodiment, the data amount of all pseudo inverse matrix parameters can be reduced to about 0.1 Mbytes.

  In particular, as described above, when the data amount per element of the pseudo inverse matrix parameter is reduced to 1 byte, for example, the same pseudo inverse matrix parameter is reused and the above-described interpolation process is used to obtain the total pseudo inverse matrix. It is possible to reduce the parameter data amount to about 0.025 Mbytes. That is, the data amount of the pseudo inverse matrix parameter can be compressed to 0.1% as compared with the case where a large data amount of about 26 M times bytes is required as the data amount of the pseudo inverse matrix parameter. .

FIG. 14 shows an example of a differential polarization degree image obtained by using pseudo inverse matrix parameters (total data amount of pseudo inverse matrix parameters = about 26 Mbytes) prepared individually for all pixels. .
FIG. 15 shows a difference polarization degree image (total data amount of pseudo inverse matrix parameters = approximately 0.025 Mbytes) obtained by using the same pseudo inverse matrix parameters as in this embodiment and performing the above-described interpolation processing. An example is shown.
In these figures, the differential polarization degree is expressed by the luminance of each pixel in the image. As can be seen by comparing these figures, there is almost no difference in luminance between the two, and there is almost no difference in the degree of differential polarization. Therefore, even when the total data amount of the pseudo inverse matrix parameter is compressed to 0.1%, a highly accurate differential polarization degree image can be obtained.

  In the present embodiment, the arrangement pattern (area division pattern) of the filter regions 2a and 2b of the optical filter 2 is 2 horizontally with respect to the image sensor 4 composed of 480 pixels vertically and 752 pixels horizontally. Although an example of a stripe pattern having an inclination of one pixel in the vertical direction has been described, the same applies to a stripe pattern having an inclination of one pixel in the horizontal direction and two pixels in the vertical direction. . This inclination is appropriately set. For example, the inclination may be set to 1.

[Modification 1]
FIG. 16 is an explanatory diagram showing a correspondence relationship between the optical filter 2 according to the modification example 1 and the pixels of the image sensor 4 configured such that the slopes of the filter regions 2a and 2b become 1/3.
FIG. 17 is an explanatory diagram showing the distribution of six types of common unit pixels existing in one unit processing area in the first modification.
When the filter regions 2a and 2b are configured to have an inclination of 1/3 as in the optical filter of the first modification, there are six types of common unit pixels in total, and the distribution is The one shown in FIG. 17 is repeated. In the first modification, in consideration of the six types of common unit pixels, one unit processing region is configured by 6 × 6 pixels. In this case, since the number of divisions is approximately 10080, 6 × 10080 = 60480 pseudo inverse matrix parameters are required as a whole. Therefore, if the data amount per pseudo inverse matrix parameter is 18 bytes, the total data amount of the pseudo inverse matrix parameter is 1088640 bytes (about 1.1 Mbytes).

Then, by performing the same interpolation processing as in the above-described embodiment, the total data amount of the pseudo inverse matrix parameter can be further compressed to ¼. Therefore, according to this modification, the total data amount of the pseudo inverse matrix parameter is reduced. It can be reduced to about 0.27 Mbytes.
The same applies when the inclination is 3.

[Modification 2]
FIG. 18 is an explanatory diagram showing a correspondence relationship between the optical filter 2 according to the modified example 2 configured so that the inclination of each of the filter regions 2a and 2b is 2/3 and the pixels of the image sensor 4.
FIG. 19 is an explanatory diagram showing the distribution of 12 types of common unit pixels existing in one unit processing area in the second modification.
When the filter regions 2a and 2b are configured to have an inclination of 2/3 as in the optical filter of the second modification, there are 12 types of common unit pixels as a whole. In the second modification, in consideration of the 12 types of common unit pixels, one unit processing region is configured by 8 × 8 pixels. In this case, since the number of divisions is 5640, 12 × 5640 = 67680 pseudo inverse matrix parameters are required as a whole. Therefore, if the data amount per pseudo inverse matrix parameter is 18 bytes, the total data amount of the pseudo inverse matrix parameter is 1218240 bytes (approximately 1.2 Mbytes).

  Then, by performing the same interpolation processing as in the above-described embodiment, the total data amount of the pseudo inverse matrix parameter can be further compressed to ¼. Therefore, according to this modification, the total data amount of the pseudo inverse matrix parameter is reduced. It can be reduced to about 0.3 Mbytes.

In the present embodiment (including each modification example, the same applies hereinafter), a plurality of types of filter regions provided in the optical filter 2 selectively transmit only the transmission region 2a that transmits light as it is and the lateral polarization component. Although it is a case where it is with the polarizing filter area | region 2b, it is not restricted to this. For example, a transmission region that transmits light as it is and a vertical polarization filter region that selectively transmits only the vertical polarization component, or a vertical polarization filter region and horizontal polarization component that selectively transmits only the vertical polarization component. Even in the case of the lateral polarization filter region that allows selective transmission, a differential polarization degree image can be obtained with the same calculation accuracy.
In the present embodiment, the plurality of types of filter regions provided in the optical filter 2 are configured to be inclined in both the vertical direction and the horizontal direction of the image sensor 4. It is good also as a structure which is parallel to any one of a vertical direction and a horizontal direction. However, in this case, both light beams transmitted through each filter region are received within one pixel.

〔Example〕
Examples of the present invention will be described below.
FIG. 20 is a block diagram illustrating a schematic configuration of the polarization camera 10 according to the present embodiment.
An analog electrical signal corresponding to the amount of light received by each pixel output from the image sensor 4 is converted into pixel data, which is a digital signal, by the A / D converter 5, and the pixel data is a signal along with a vertical synchronization signal and a horizontal synchronization signal. The data is sequentially output to the processing unit 6. The vertical synchronization signal, horizontal synchronization signal, and pixel data sequentially input to the signal processing unit 6 are temporarily held in the line buffer unit 31 in the signal processing unit 6. The line buffer unit 31 holds pixel data for at least the previous two lines. Then, the line buffer unit 31 performs image processing from the pixel data for three lines including the pixel data for the previous two lines and the latest pixel data for one line input to the signal processing unit 6. 3 × 3 pixel data I1 to I9 consisting of the target pixel and its peripheral pixels are selected and output to the differential polarization degree image generation unit 50. The differential polarization degree image generation unit 50 then inputs the pixel data I1 to I9 for 3 × 3 pixels and the pseudo inverse matrix corresponding to the pixel of interest stored in the SPI flash memory 32 as parameter storage means. Using the parameters, the vertical polarization intensity V and the horizontal polarization intensity H of the light received by the target pixel are calculated from the equation shown in Equation 3 above, and the differential polarization degree for the target pixel is calculated from the calculation result. calculate. The calculated differential polarization degree is sequentially output to the subsequent apparatus together with the horizontal / vertical synchronization signal as data (pixel value) of the differential polarization degree image. By performing such image processing for all the effective pixels of the image sensor 4, a differential polarization degree image with the differential polarization degree for each pixel as a pixel value is generated.

Here, the configuration and operation of the differential polarization degree image generation unit 50 in the present embodiment will be described.
FIG. 21 is a block diagram illustrating an internal configuration of the differential polarization degree image generation unit 50.
This differential polarization degree image generation unit 50 is composed of an FPGA, calculates the vertical polarization intensity V and the horizontal polarization intensity H for the pixel from the pixel data I1 to I9 and the pseudo inverse matrix parameter, and the calculation result Thus, image processing for calculating the degree of differential polarization for the pixel is realized by hardware processing. All the pseudo inverse matrix parameters used in this image processing are loaded in advance into a pseudo inverse matrix parameter storage BRAM (Block-RAM) provided in the FPGA.

  The BRAM address calculation unit 51 of the differential polarization degree image generation unit 50 receives the vertical synchronization signal and the horizontal synchronization signal, and calculates a BRAM address value in which a pseudo inverse matrix parameter used for the image processing is stored. The output timing chart of the BRAM address calculation unit 51 in this embodiment is as shown in FIG.

  When the coefficient generation unit 52 of the differential polarization degree image generation unit 50 receives the BRAM address value output from the BRAM address calculation unit 51, the pseudo inverse matrix parameter stored in the BRAM address value is transmitted to the polarization separation processing unit 53. Output. FIG. 23 shows an internal configuration diagram of the coefficient generation unit 52 in the present embodiment. FIG. 24 shows the division of the unit processing area in this embodiment. FIG. 25 shows an address map of the pseudo inverse matrix parameter storage BRAM in which the pseudo inverse matrix parameters are stored. The interpolation process described above is executed in the parameter calculation unit of the coefficient generation unit 52.

  The polarization separation processing unit 53 of the differential polarization degree image generation unit 50 includes nine pieces of pixel data I1 to I9 output from the line buffer unit 31 and 18 elements A1 to A18 output from the coefficient generation unit 52. The pseudo inverse matrix parameter is received, and the lateral polarization component H and the longitudinal polarization component V are calculated from the equation shown in Equation 3 above. FIG. 26 shows an internal configuration of the polarization separation processing unit 53 in this embodiment.

  When the differential polarization degree calculation unit 54 of the differential polarization degree image generation unit 50 receives the horizontal polarization component H and the vertical polarization component V output from the polarization separation processing unit 53, the differential polarization degree P = (HV) / ( H + V) is calculated. In a normal case, the differential polarization degree P takes a value in the range of −1 to 1, and therefore, the differential polarization degree calculation unit 54 outputs a bit representation of the decimal part. FIG. 27 shows an internal configuration diagram of the differential polarization degree calculation unit 54 in the present embodiment.

  When the differential polarization degree image calculation unit 55 of the differential polarization degree image generation unit 50 receives the differential polarization degree P output from the differential polarization degree calculation unit 54, the differential polarization degree P that represents the decimal part in bits is represented by the differential polarization degree P. A process of converting to an image pixel value and outputting it is performed. Specifically, this process scales the differential polarization degree P indicating a value in the range of −1 to 1 to a range of 0 to 1022, and outputs the scaled value as the pixel value of the differential polarization degree image. It is processing. At this time, when the differential polarization degree P is outside the range of −1 to 1, the differential polarization degree P is highly likely to be an abnormal value. Therefore, for the differential polarization degree P indicating a value outside the range of −1 to 1, the abnormality indication value 1023 is output as the pixel value. FIG. 28 shows an internal configuration diagram of the differential polarization degree image calculation unit 55 in the present embodiment.

  Further, the differential polarization degree image calculation unit 55 of the present embodiment acquires pixel data I1 to I9 (RAW data) via the delay processing unit 56, as shown in FIG. If any one of the pixel data I1 to I9 is equal to or less than a predetermined blackout threshold value or equal to or greater than a predetermined saturation threshold value, the differential polarization degree calculated from these pixel data I1 to I9. As the P pixel value, 1023 which is an abnormal instruction value is output.

  As a method for detecting whether any one of the pixel data I1 to I9 is equal to or lower than a predetermined blackout threshold value or equal to or higher than a predetermined saturation threshold value, for example, a register representing these threshold values is prepared. It is preferable to make a determination by comparing the pixel data I1 to I9. The blackout threshold value is set to a value close to zero, but is not necessarily zero. Similarly, the saturation threshold value is set to a value close to the upper limit value of the digital electric signal (pixel data) output from the A / D converter 5, but it is not necessarily required to be the upper limit value. These threshold values are preferably configured to be appropriately changeable.

  By performing such processing, each pixel value of the differential polarization degree image data indicates a range of 0 to 1022 for a normal value and 1023 for an abnormal value. When such differential polarization degree image data is used in a process (discrete information processing) for discriminating a road surface and a three-dimensional object, for example, a pixel having an abnormal value is excluded and processed. Or correct this and use it to perform appropriate outlier response. As a method of correcting and using a pixel having an abnormal value, for example, when a differential polarization degree image is binarized, for a pixel indicating an abnormal value, “0” is calculated from the surrounding normal pixel values. ”Or“ 1 ”. As a result, compared with the case where the normal value and the abnormal value are processed without being distinguished from each other, misdetermination of the road surface and the three-dimensional object can be suppressed, and the determination accuracy of the road surface and the three-dimensional object can be improved.

Next, the configuration and operation of the parameter control unit 40 in the present embodiment will be described.
In this embodiment, all pseudo inverse matrix parameters are calculated in advance and stored in the SPI flash memory 32. The address map of the SPI flash memory 32 in this embodiment is as shown in FIGS. As shown in FIGS. 20 and 21, the parameter control unit 40 includes a general-purpose CPU 41. When the power is turned on, the parameter control unit 40 reads the pseudo inverse matrix parameters from the SPI flash memory 32 through the SPI interface 42, and reads all the pseudo inverses. The matrix parameter is transferred to the pseudo inverse matrix parameter storage BRAM in the FPGA constituting the differential polarization degree image generation unit 50 via the CPU bus 43 and the User interface 44.

Next, a processing flow from when the power is turned on to when the differential polarization degree image is generated will be described.
When the power is turned on, first, the CPU 41 of the parameter control unit 40 loads the FPGA circuit data in the SPI flash memory 32 into the FPGA constituting the differential polarization degree image generation unit 50. Thereafter, the CPU 41 executes a pseudo inverse matrix parameter transfer process. In this pseudo inverse matrix parameter transfer process, first, a read command is issued from the CPU 41 to the SPI interface 42. As a result, the pseudo inverse matrix parameter in the SPI flash memory 32 is read into the CPU 41 via the SPI interface 42. Thereafter, the CPU 41 issues a write command to the user interface 44. As a result, the pseudo inverse matrix parameter is written into the pseudo inverse matrix parameter storage BRAM in the coefficient generation unit 52 via the User interface 44. Until all the pseudo inverse matrix parameters are transferred to the pseudo-inverse matrix parameter storage BRAM in the coefficient generation unit 52, the above-described processing from issuing the Read command to issuing the Write command is repeated.

  When all the pseudo inverse matrix parameters in the SPI flash memory 32 are transferred to the pseudo inverse matrix parameter storage BRAM in the coefficient generation unit 52 and the pseudo inverse matrix parameter transfer process is completed as described above, the differential polarization degree image generation is performed. Move to the process. In this differential polarization degree image generation step, when a vertical synchronization signal, a horizontal synchronization signal, and pixel data are transferred from the image sensor 4, first, coefficients are generated for each pixel with reference to the vertical synchronization signal and the horizontal synchronization signal. The address value of the pseudo-inverse matrix parameter storage BRAM in the unit 52 is calculated, and the pseudo-inverse matrix parameter element (144-bit bus signal) stored in the BRAM is read. Then, the read 144-bit bus signal is divided into 18 elements to obtain 18 elements A1 to A18 of pseudo inverse matrix parameters. Thereafter, the polarization separation processing unit 53 uses the 18-element pseudo inverse matrix parameter to convert the horizontal polarization component H and the vertical polarization component V of the pixel to be processed (target pixel) into the target pixel and its peripheral pixels. The pixel data I1 to I9 are calculated from the equation shown in the above equation (3). The lateral polarization component H and the longitudinal polarization component V calculated in this way are sent to the differential polarization degree calculation unit 54, and the differential polarization degree calculation unit 54 calculates the differential polarization degree P. Then, the differential polarization degree image calculation unit 55 performs a process of converting the differential polarization degree P into a pixel value (image data) of the differential polarization degree image. Thereby, image data for one pixel of the differential polarization degree image is obtained. A differential polarization degree image is generated by performing the above processing for all pixels.

What has been described above is merely an example, and the present invention has a specific effect for each of the following modes.
(Aspect A)
Through selective permeation is allowed Ru selection filter region periodically arranged the optical filter 2 a predetermined optical component, the light-receiving element array of the image sensor 4 such that the light receiving elements are arranged two-dimensionally such as a photodiode 4a An imaging unit that receives light from the imaging region and outputs an image signal including an image signal value corresponding to the amount of received light for each unit pixel corresponding to one light receiving element or two or more light receiving elements on the light receiving element array an imaging unit etc., a pixel value of a unit pixel corresponding to the image signal output from the image pickup means (range 0-1022), using a predetermined calculation parameters, image signal value of the unit pixel and the unit An image forming the image signal value group in an imaging apparatus including a pixel value calculation unit such as a signal processing unit 6 that calculates from an image signal value group composed of image signal values of peripheral pixels located around the pixel Trust Based on the value, the in the image signal value forming the image signal value group, determining whether or not differential polarization degree image includes the abnormality determination satisfies the image signal value that is outside a predetermined value range has an abnormality signal value determination means such calculation unit 55, when the abnormality determination condition is satisfied image signal value in the image signal values constituting the image signal value group contains the abnormality signal value determining means If the determination is made, the process of setting the abnormal instruction value (1023) indicating an abnormal value as the pixel value of the unit pixel corresponding to the image signal value group without using the predetermined calculation parameter is executed. It is characterized by doing.
According to this, it is possible to grasp that the pixel value is an abnormal value in subsequent information processing such as processing for discriminating the road surface and the three-dimensional object using the pixel value of the captured image. As a result, in the subsequent information processing, the pixel value can be handled as an abnormal value, so that an abnormal result such as a misjudgment caused by processing a pixel value indicating an abnormal value as a normal pixel value as it is can be avoided. It becomes possible to do.

(Aspect B)
In the above embodiments A, upper Kikoto normal determination condition corresponding to the case conditions and the received light amount of the received light quantity is the image signal values corresponding to the case is less than a predetermined underexposure threshold is equal to or greater than a predetermined saturation threshold It includes at least one of the conditions that it is an image signal value .
According to this, the image signal value image signal value or saturated with underexposed likely to be outliers, can be handled as a suitably outliers.

(Aspect C)
In the aspect A or B, the pixel value calculation means is a value that is out of a numerical value range (0 to 1022) of pixel values that can be calculated using the predetermined calculation parameter as the abnormality indication value, A value adjacent to the upper limit value of the numerical range or a value adjacent to the lower limit value of the numerical range is used.
This facilitates data processing of normal values and abnormal values.

(Aspect D)
In any one of the aspects A to C, the selection filter region is configured by a polarization filter that selectively transmits a predetermined polarization component.
According to this, a specific polarization component can be extracted from the image signal output from the light receiving element array.

(Aspect E)
In an object identification apparatus having an object identification processing unit that performs identification processing of a detection target existing in an imaging region based on a pixel value of a captured image captured by an imaging unit, any of the above aspects A to D is used as the imaging unit. The pixel value of the unit pixel is output as the pixel value of the captured image using the imaging apparatus according to any of the above aspects, and the object identification processing unit excludes the pixel value indicating the abnormality indication value. The identification process is performed based on the pixel value.
According to this, it is possible to avoid an abnormality in the identification process by a pixel indicating an abnormal value.

In the above description, the selection filter region of the optical filter 2 is configured with a polarization filter that selectively transmits a predetermined polarization component, and the case where a differential polarization degree image is generated from two different polarization components has been described. The present invention is similarly applied to a case where a selective filter region of the optical filter 2 is configured by a spectral filter that selectively transmits a specific wavelength component and, for example, a color separation image is generated from a plurality of different wavelength components. it can.
Further, the driver assistance system according to the present embodiment is mounted on the vehicle as a whole, but the entire system is not necessarily mounted on the vehicle. Therefore, for example, only the polarization camera 10 may be mounted on the own vehicle, and the remaining system components may be remotely arranged at a location different from the own vehicle. In this case, a system in which a person other than the driver objectively grasps the traveling state of the vehicle can be provided.
The imaging apparatus according to the present invention is not limited to the present embodiment, and can be applied to a wide range of fields.

DESCRIPTION OF SYMBOLS 1 Image pick-up lens 2 Optical filter 2a Transmission area 2b Lateral polarization filter area 4 Image sensor 4a Photodiode (light receiving element)
5 A / D conversion unit 6 Signal processing unit 10 Polarization camera 31 Line buffer unit 32 SPI flash memory 40 Parameter control unit 41 CPU
DESCRIPTION OF SYMBOLS 50 Differential polarization degree image generation part 51 Address calculation part 52 Coefficient generation part 53 Polarization separation process part 54 Differential polarization degree calculation part 55 Differential polarization degree image calculation part

JP 2007-086720 A Japanese Patent No. 3777054

Claims (5)

Predetermined selectively transmitted so Ru selection filter region of the optical component through the optical filter that is periodically arranged, and receiving light from the imaging region by the light receiving element array in which the light-receiving elements are arranged two-dimensionally, the An image pickup means for outputting an image signal including an image signal value corresponding to an amount of light received for each unit pixel corresponding to one light receiving element or two or more light receiving elements on the light receiving element array;
The pixel value of the unit pixel corresponding to the image signal output from the imaging means is calculated from the image signal value of the unit pixel and the image signal values of the peripheral pixels located around the unit pixel using a predetermined calculation parameter. In an imaging apparatus comprising a pixel value calculation means for calculating from a configured image signal value group,
Based on the image signal values constituting the image signal value group, whether the image signal values constituting the image signal value group include an image signal value satisfying an abnormality determination condition that the image signal value group is out of a predetermined value range . Having an abnormal signal value determining means for determining whether or not
The pixel value calculation unit, when the upper Kikoto normally satisfies the determination condition image signal value in the image signal values constituting the image signal value group contains the error signal value determination unit determines the An image pickup apparatus that performs a process of using an abnormal instruction value indicating an abnormal value as a pixel value of a unit pixel corresponding to the image signal value group without using the predetermined calculation parameter. . The imaging device according to claim 1.
Upper Kikoto normal determination conditions are image signal values corresponding to when the conditions and the received light amount of the received light quantity is the image signal values corresponding to the case is less than a predetermined underexposure threshold is equal to or greater than a predetermined saturation threshold An image pickup apparatus including at least one of the following conditions. The imaging device according to claim 1 or 2,
The pixel value calculation means is a value that deviates from a numerical range of pixel values that can be calculated using the predetermined calculation parameter as the abnormality indication value, and is a value adjacent to the upper limit value of the numerical range or the numerical value An image pickup apparatus using a value adjacent to a lower limit value of a range. The imaging device according to any one of claims 1 to 3,
The image pickup apparatus, wherein the selection filter region includes a polarization filter that selectively transmits a predetermined polarization component. In an object identification device having object identification processing means for performing identification processing of a detection target existing in an imaging region based on a pixel value of a captured image captured by an imaging means,
The imaging device according to any one of claims 1 to 4 is used as the imaging unit, and the pixel value of the unit pixel is output as the pixel value of the captured image.
The object identification device according to claim 1, wherein the object identification processing means performs the identification processing based on a remaining pixel value excluding a pixel value indicating the abnormality indication value.