JP2011150686A - Object identifying apparatus, moving body control apparatus equipped with the same, and information providing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To increase the types of the property of an object which can be used for the identification of the object by using an image pickup means, and to increase the number of the classifications of the object which can be finally specified. <P>SOLUTION: An object identifying apparatus which identifies an object existing in an image pickup area is provided with: a polarizing camera 10 which receives two polarized lights (P polarized light and S polarized light) having different polarization directions included in the rays of light reflected from the object, and captures two polarization images (P polarization image and S polarization image); a differential polarization degree image processing part 15 which calculates a differential polarization degree as the rate of a luminance differential value between the P polarization image and the S polarization image to a luminance total value for every pixel; and a road surface structure identification part 16 and a three-dimensional object identification part 18 for performing the identification processing of the object by using the differential polarization degree. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an object identification device for identifying an object existing in an imaging region, and movement control of a moving body such as a vehicle, a ship, an aircraft, or an industrial robot using the identification result of the object identification device. The present invention relates to a mobile body control device that performs the above and an information providing device that provides useful information to the driver of the mobile body.

  As this type of object identification device, for example, a device used in a driver support system such as ACC (Adaptive Cruise Control) for reducing the driving load of a driver (driver) of a vehicle is known. In such a vehicle driving support system, an automatic brake function and an alarm function for avoiding the collision of the own vehicle with an obstacle or reducing the impact at the time of the collision, and a distance between the preceding vehicle are maintained. In order to realize various functions such as a function for adjusting the speed of the own vehicle, a function for supporting the prevention of deviation from the traveling lane in which the host vehicle is traveling, obstacles existing around the host vehicle, preceding vehicles, It is necessary to appropriately distinguish and recognize (identify) an object such as a lane. For this reason, various object identification devices have been proposed.

  In patent document 1, in order to detect the relative displacement of the own vehicle with respect to the lane (white line) which divides the driving lane of the vehicle by detecting a line in the image from a road image (captured image) obtained by photographing, An object identification device for identifying a lane (object) is disclosed. In this object identification device, when there is a puddle on the road due to rainy weather, sunlight is specularly reflected by this puddle, and the puddle is photographed with the same brightness as the lane (white line) on the road. This is a solution to the problem of misrecognizing traffic as a lane (white line). Specifically, in order to remove the puddle portion from the road image before performing the white line identification processing, the puddle portion is removed by removing only the specular reflection component from the photographed road image, and the white line is recognized from the remaining scattered light component. . As a method of removing only the specular reflection component, the horizontal polarization component of the specular reflection is almost 0 at the Brewster angle, and the scattered light component includes substantially equal amounts of the vertical polarization component and the horizontal polarization component. Is used as follows. That is, the correction coefficient for calculating the difference between the vertical polarization component and the horizontal polarization component in the captured road image and removing the specular reflection component corresponding to the incident angle included in the horizontal polarization component with respect to the difference value The specular reflection component is calculated by multiplying by. Then, by subtracting the calculated specular reflection component from the horizontal polarization component, an image of the confusion light component obtained by removing only the specular reflection component from the road image is obtained.

The number of types of objects that can finally identify what the object is using a conventional object identification device that identifies an object existing in the imaging region is still not sufficient. This is because the types of object properties used for object identification in the conventional object identification device are limited. In other words, if the types of object properties used to identify an object increase, the classification of the object can be subdivided. As a result, the accuracy of identifying what the object is is increased and the number of object types that can be identified is increased. Can do.
For example, some conventional object identification devices have white lines (objects) on the road due to differences in the intensity of reflected light (amount of received light) from objects existing in the shooting area, that is, differences in brightness in the captured image. ) To identify. In this object identification device, the white line on the road is identified from other objects using the property that the intensity of the reflected light is different, but when identifying an object using only this property, As described in Patent Document 1, an object (such as a puddle) whose reflected light intensity is comparable to that of the white line may not be recognized separately from the white line. On the other hand, as in the object identification device described in the above-mentioned Patent Document 1, if identification is performed by using the property of an object that is specularly reflected, a white line and a puddle that are objects of the same intensity are reflected. Can also be identified.
Further, for example, in the conventional object identification device, the shape of the object grasped from the captured image is used as the shape template of the target object to be finally identified by utilizing the property of the shape of the object in the captured image. Some of them identify the object in light of it. This object identification device also uses other properties such as luminance or specular reflection, and increases the types of object properties used for object identification, so that the accuracy of object identification increases and the number of types of objects that can be identified increases. be able to.

As described above, since the types of object properties used for object identification are limited, the problem that the number of object types that can be finally identified is not sufficient is an object identification device used in a driver assistance system. The problem occurs not only in any object identification apparatus such as an object identification apparatus used for robot control.
In order to solve this problem, it is not preferable to newly prepare a detection device for detecting a new property in order to increase the types of the property of the object used for identifying the object because the cost increases. . Therefore, if a new property can be detected by using an imaging unit that is a detection device generally used for detecting the intensity (luminance) of reflected light from an object in a conventional object identification device, a cost viewpoint can be obtained. Is beneficial from.

  The present invention has been made in view of the above problems, and the object of the present invention is to increase the number of types of object properties that can be used for object identification using an imaging means, and finally the number of types of objects that can be specified. Object identification apparatus capable of increasing the number of objects, and a moving body control apparatus and an information providing apparatus provided with the object identification apparatus.

In order to achieve the above object, according to a first aspect of the present invention, there is provided an object identification device for identifying an object existing in an imaging region, wherein the polarization directions included in the reflected light from the object existing in the imaging region are mutually different. An imaging unit that receives two different polarized lights and captures each polarized image, and an object existing at a location corresponding to each processing region in the imaging region using the two polarized images captured by the imaging unit. And an object identification processing means for performing identification processing.
According to a second aspect of the present invention, in the object identification device for identifying an object existing in the imaging region, two polarized lights having different polarization directions included in the reflected light from the object existing in the imaging region are received. Then, an imaging unit that captures each polarization image, and the two polarization images captured by the imaging unit are each divided into predetermined processing areas, and a luminance difference value between the two polarization images is calculated for each processing area Using the luminance difference value calculating means and the identification index value obtained from the luminance difference value calculated by the luminance difference value calculating means, the identification processing of the object existing at a location corresponding to each processing area in the imaging area is performed. And an object identification processing means for performing.
According to a third aspect of the present invention, in the object identification device of the second aspect, the object identification processing means determines which of a plurality of numerical ranges respectively determined for a plurality of different objects the identification index value belongs to. The identification process is performed by performing a determination process to determine that an object existing at a location corresponding to the processing area is an object corresponding to the numerical range determined to belong to the determination process. It is characterized by carrying out.
According to a fourth aspect of the present invention, in the object identification device according to the third aspect, the imaging means is mounted on a moving body that moves on a moving surface, and includes the moving surface obliquely from above the moving surface. In the above determination process, a numerical range determined for the same object is set for each of at least two or more areas that divide the two polarized images in the vertical direction. It is characterized by determining which of the numerical ranges set for the area the identification index value of the processing area belonging to the area belongs.
According to a fifth aspect of the present invention, in the object identification device according to any one of the second to fourth aspects, the identification processing includes an object existing at a location corresponding to each processing region in the imaging region. A material identification process for identifying the material is included.
According to a sixth aspect of the present invention, in the object identification device according to the fifth aspect, the material of the object identified by the material identification process includes a metal.
The invention of claim 7 is the object identification apparatus according to any one of claims 2 to 6, wherein the imaging means is mounted on a vehicle that is a moving body that moves on a road surface that is a moving surface. The imaging region including the road surface is imaged, and the object identification processing means performs identification processing of an object whose outer surface is exposed on substantially the same plane as the road surface.
The invention according to claim 8 is the object identification device according to any one of claims 2 to 7, wherein the object identification processing means corresponds to a location where a reference object is assumed to exist in advance. A luminance difference value calculated by the luminance difference value calculating unit for a processing region different from the reference processing region, using an identification index value obtained from the luminance difference value calculated by the luminance difference value calculating unit for the reference processing region The identification processing is performed by calculating a relative value of the identification index value obtained from the value with respect to the reference index value and identifying an object existing at a location corresponding to the processing area based on the relative value. It is characterized by doing.
The invention according to claim 9 is the object identification device according to any one of claims 2 to 8, wherein the luminance between two polarized images used when the luminance difference value is calculated as the identification index value. A differential polarization degree indicating a ratio of the luminance difference value to the total value is used.
The invention according to claim 10 is the identification processing result storage means for storing the result of the identification processing performed in the past by the object identification processing means in the object identification device according to any one of claims 2 to 9. And the object identification processing means performs the identification processing using the identification index value and the result of past identification processing stored in the identification processing result storage means. .
Further, the invention of claim 11 is the object identification device according to any one of claims 2 to 10, wherein shape information for storing shape information indicating a shape when a predetermined specific object is imaged by the imaging means is stored. The object identification processing means includes a shape information stored in the shape information storage means in a shape indicated by a plurality of processing regions adjacent to each other identified as the same object by the identification processing. It is determined whether or not the shape is approximated, and when it is determined that the shape is approximated, an object specifying process for specifying an object existing at a location corresponding to the plurality of processing regions as the specific object is performed. It is what.
Further, the invention of claim 12 includes a movement control means for controlling the movement of the moving body, and an object identification means for picking up an image around the moving body as an imaging target and for identifying an object existing in the imaging target. The object identification according to any one of claims 1 to 11, wherein the movement control means, as the object identification means, is a moving body control device that performs the movement control using a result of identification by the object identification means. It is characterized by using an apparatus.
According to a thirteenth aspect of the present invention, an object is identified as an object to be imaged around a moving body that moves in accordance with a driving operation by the driver, and an object that exists in the object is identified by the object identifying means. In the information providing apparatus, comprising: information providing means for generating useful information for the driver using the result; and information notifying means for notifying the driver of information generated by the useful information generating means. As a means, the object identification device according to any one of claims 1 to 11 is used.

  In the present invention, for two polarized images captured by the imaging means, for example, a luminance difference value between these polarized images is calculated for each predetermined processing region, and an identification index value obtained from the luminance difference value is used. Thus, an object existing at a location corresponding to each processing area is identified. The reflected light from the same place where natural light or illumination is illuminated usually includes a plurality of polarized lights having different polarization directions. Even if the intensity | strength and incident angle of the light irradiated to the place are the same, if the polarization characteristics of the object which exists in the place differ, the intensity | strength will change for every polarization direction. The present inventors paid attention to the nature of this polarized light and, as a result of intensive studies, found that the object can be identified from the difference in the polarization characteristics of the object. For example, it is found that an object can be identified by utilizing that the difference in polarization characteristics of an object is strongly reflected in a luminance difference value between two polarized images obtained by receiving two polarized light beams having different polarization directions. It was. Therefore, by using an identification index value (including the luminance difference value itself) obtained from the luminance difference value between the two polarized images, it is possible to identify objects having different polarization characteristics with high accuracy.

  As described above, according to the present invention, it is possible to identify an object using a new property called the polarization property of an object using an imaging unit. An excellent effect is obtained that the number of types of objects that can be specified can be increased.

It is a functional block diagram of the driver assistance system concerning an embodiment. It is explanatory drawing which shows one structural example of the polarization camera which can be utilized for the driver | operator assistance system. It is explanatory drawing which shows the other structural example of the polarization camera which can be utilized for the driver | operator assistance system. It is explanatory drawing which shows the further another structural example of the polarization camera which can be used for the driver | operator assistance system. It is explanatory drawing which shows the further another structural example of the polarization camera which can be used for the driver | operator assistance system. It is explanatory drawing which shows the further another structural example of the polarization camera which can be used for the driver | operator assistance system. It is explanatory drawing which shows the further another structural example of the polarization camera which can be used for the driver | operator assistance system. (A) is a top view of the botsdots that can be identified, and (b) is a side view of the botsdots. (A) is a top view of a cat's eye that can be an identification target, and (b) is a perspective view of the cat's eye. It is a flowchart which shows the flow of the road surface structure specific process in embodiment. It is explanatory drawing which shows an example of the monochrome image (luminance image) produced | generated in the monochrome image process part from the polarization RAW image data acquired with the polarization camera. It is explanatory drawing which shows the difference polarization degree image produced | generated in the difference polarization degree image process part from the same polarization RAW image data. It is the graph which plotted the luminance value obtained along the white broken line arrow in FIG. It is the graph which plotted the differential polarization degree obtained along the white broken-line arrow in FIG. It is a graph which shows an example of a change of a differential polarization | polarized-light degree when a P polarization | polarized-light image and a S-polarized image are image | photographed with the camera which changed and fixedly arranged the light source position with respect to an asphalt surface and a metal surface in a laboratory. It is explanatory drawing for demonstrating the influence of a windshield. It is explanatory drawing which shows an example of the monochrome image (luminance image) which the monochrome image processing part produced | generated from the polarized RAW image data which imaged the road surface in which the botsdots are used as a section line with the polarization camera, and was acquired by this. It is explanatory drawing which shows the difference polarization degree image produced | generated in the difference polarization degree image process part from the same polarization RAW image data. A graph showing the change in the degree of differential polarization when a P-polarized image and an S-polarized image are photographed with a camera that is fixedly arranged by changing the light source position on an asphalt surface and a coated surface obtained by applying paint to steel in a laboratory. It is. It is explanatory drawing which shows an example of the monochrome image (luminance image) which the monochrome image process part 13 produced | generated from the polarized RAW image data which imaged the road surface to which coal tar adhered with the polarization camera, and was acquired by this. It is explanatory drawing which shows the difference polarization degree image produced | generated in the difference polarization degree image process part from the same polarization RAW image data. It is a flowchart which shows the flow of the solid object specific process in embodiment. It is explanatory drawing which shows an example of the monochrome image (luminance image) produced | generated in the monochrome image process part from the polarization RAW image data acquired with the polarization camera. It is explanatory drawing which shows the difference polarization degree image produced | generated in the difference polarization degree image process part from the same polarization RAW image data. 24 is a histogram of luminance value distributions at three white frames in FIG. 23 for 100 frames. FIG. 25 is a histogram obtained by taking differential polarization degree distributions at three positions in a white frame in FIG. 24 for 100 frames.

Hereinafter, an embodiment in which the present invention is applied to a driver support system as a mobile control device and an information providing device will be described.
FIG. 1 is a functional block diagram of the driver assistance system according to the present embodiment.
A landscape around the vehicle including a road surface (moving surface) on which a moving vehicle travels is photographed by a polarization camera 10 as an imaging means mounted on a vehicle (not shown). Polarized RAW image data including “S-polarized light intensity” and horizontal polarized light intensity (hereinafter simply referred to as “P-polarized light intensity”) is acquired. The horizontally polarized image data obtained from the P polarized light intensity data included in the polarized RAW image data is stored in the horizontally polarized image memory 11, and the vertically polarized image data obtained from the S polarized light intensity data included in the polarized RAW image data is displayed in the vertically polarized image memory 12. Respectively. These image data are respectively transmitted to the monochrome image processing unit 13 as a luminance calculating unit and the differential polarization degree image processing unit 15 as a luminance difference value calculating unit.

  The polarization camera 10 captures a surrounding image having pixels of, for example, a megapixel size by an image sensor such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) that is a light receiving element. It is preferable that the polarization camera 10 continuously acquires surrounding images at short time intervals close to real time. The polarization camera 10 may be attached to a room mirror, for example, to capture a landscape in front of the vehicle (a front view including a road surface), or attached to a side mirror, for example, to capture a landscape on the side of the vehicle. For example, it may be attached to a back door and may take an image of a landscape behind the vehicle. In the present embodiment, an example will be described in which a landscape (a front view including a road surface) in front of a vehicle is imaged by being attached to a rearview mirror.

FIG. 2 is an explanatory diagram illustrating a configuration example of the polarization camera 10.
As shown in FIG. 2, the polarization camera 10 </ b> A has a rotating polarizer 102 that is rotationally driven on the front surface of one camera 101 that includes an image pickup device such as a CCD. In this polarization camera 10 </ b> A, the polarization direction of light passing therethrough changes according to the rotation angle of the rotating polarizer 102. Therefore, the camera 101 can capture the P-polarized image and the S-polarized image alternately by imaging while rotating the rotating polarizer 102.

FIG. 3 is an explanatory diagram illustrating another configuration example of the polarization camera 10.
As shown in FIG. 3, the polarization camera 10B uses two cameras 111 and 112 each having an image sensor such as a CCD, and transmits an S-polarized light filter 113 that transmits S-polarized light and a P-polarized light to the respective front surfaces. A P-polarization filter 114 is disposed. In the polarization camera 10A shown in FIG. 2, since one camera 101 alternately captures P-polarized images and S-polarized images, the P-polarized image and S-polarized image could not be captured simultaneously. With the polarization camera 10B shown in FIG. 3, a P-polarized image and an S-polarized image can be taken simultaneously.

FIG. 4 is an explanatory diagram showing still another configuration example of the polarization camera 10.
As shown in FIG. 4, the polarization camera 10C is similar to the polarization camera 10B shown in FIG. 3 in that the imaging elements are individually provided for the P-polarized image and the S-polarized image. The elements are greatly different in that they are arranged closer than in the polarization camera 10B shown in FIG. According to this polarization camera 10C, it can be made smaller than the polarization camera 10B shown in FIG. The polarization camera 10C shown in FIG. 4 is formed by laminating a lens array 122, a light shielding spacer 123, a polarization filter 124, a spacer 125, and a solid-state imaging unit 126. The lens array 122 has two imaging lenses 122a and 122b. The two imaging lenses 122a and 122b are formed of a single lens made of, for example, an aspheric lens having the same shape independent from each other, and the optical axes 121a and 121b are parallel to each other and on the same plane. It is arranged. The light shielding spacer 123 has two openings 123a and 123b, and is provided on the side opposite to the subject side with respect to the lens array 122. The two openings 123a and 123b are penetrated with a predetermined size centering on the optical axes 121a and 121b, respectively, and the inner wall surface is subjected to light reflection prevention processing by blackening, roughening or matting. The polarizing filter 124 is a region-dividing type polarizer filter having two polarizer regions 124 a and 124 b whose polarization planes are different by 90 degrees, and is provided on the side opposite to the lens array 122 with respect to the light shielding spacer 123. The polarizer regions 124a and 124b transmit non-polarized light whose electromagnetic field vibrates in an unspecified direction to linearly polarized light by transmitting only the vibration component (polarized component) in the direction along the polarization plane. Note that a region-divided polarizer filter with a clear boundary can be obtained by using a wire grid method formed with a metal fine concavo-convex shape or an auto-cloning photonic crystal method. The spacer 125 is formed in a rectangular frame shape having an opening 125 a through which regions corresponding to the polarizer regions polarized light a and polarized light b of the polarizing filter 124 pass, and is provided on the side opposite to the light shielding space 123 with respect to the polarizing filter 124. It has been. The solid-state image pickup unit 126 has two solid-state image pickup devices 126 a and 126 b mounted on a substrate 127, and is provided on the side opposite to the polarization filter 124 with respect to the spacer 125. In the present embodiment, in order to perform monochrome sensing, these solid-state imaging devices 126a and 126b do not include a color filter. However, a color filter is arranged when sensing a color image.

FIG. 5 is an explanatory diagram showing still another configuration example of the polarization camera 10.
As illustrated in FIG. 5, the polarization camera 10 </ b> D includes a half mirror 131 having 1: 1 transparency, a reflection mirror 132, an S polarization filter 133, a P polarization filter 134, and an S polarization filter 133. It has an S-polarized CCD 135 that receives S-polarized light and a P-polarized CCD 136 that receives P-polarized light via a P-polarized filter 134. In the polarization cameras 10B and 10C shown in FIGS. 3 and 4, although S-polarized images and P-polarized images can be captured simultaneously, parallax occurs. On the other hand, in the polarization camera 10D shown in FIG. 5, since the S-polarized image and the P-polarized image are simultaneously captured using the same light received through the same imaging optical system (lens) (not shown), the parallax is reduced. Does not occur. Therefore, processing such as parallax deviation correction is not necessary.
Instead of the half mirror 131, a polarizing beam splitter such as a prism that reflects P-polarized light and transmits S-polarized light may be used. By using such a polarization beam splitter, the S-polarization filter 133 and the P-polarization filter 134 can be omitted, the optical system can be simplified, and the light utilization efficiency can be improved.

FIG. 6 is an explanatory diagram showing still another configuration example of the polarization camera 10.
This polarization camera 10E is the same as the polarization camera 10C shown in FIG. 4 in that it is a unit in which camera components are stacked along the optical axis 141 of the imaging lens 142a as shown in FIG. The difference is that an S-polarized image and a P-polarized image are captured by a single imaging lens 142 (a plurality of imaging lenses may be stacked on the optical axis) 142. According to this polarization camera 10E, no parallax occurs between the S-polarized image and the P-polarized image, as in the polarization camera 10D shown in FIG. Moreover, it can be made smaller than the polarizing camera 10D shown in FIG. The polarization filter 144 of the polarization camera 10E shown in FIG. 6 is a region-dividing polarizer filter in which two types of polarizer regions 144a and 144b having different polarization planes of 90 degrees are provided. Accordingly, four solid-state image sensors 146a, 146b, 146c, 146d are provided.

FIG. 7 is an explanatory diagram showing still another configuration example of the polarization camera 10.
As shown in FIG. 7, the polarization camera 10F employs a region division type filter. In FIG. 7, squares arranged vertically and horizontally indicate the light receiving portions 151 of the respective light receiving elements, a region indicated by a vertical line indicates a region of the S polarization filter 152, and a region indicated by a horizontal line indicates the region of the P polarization filter 153. This polarization camera 10F is not made to correspond to the pixels of the light receiving element 1: 1, but the area of each filter 152, 153 has a width corresponding to one light receiving element in the horizontal direction, and the inclination of the boundary line of the area is 2. That is, it takes the shape of an oblique band having an angle that changes by two pixels in the vertical direction while proceeding by one pixel in the horizontal direction. By combining such a special filter arrangement pattern and signal processing, it is possible to reproduce each filter transmission image on the entire screen even if the alignment accuracy when joining the image sensor array and the area division filter is not sufficient. It is possible to realize a low-cost polarization camera that can capture an S-polarized image and a P-polarized image.

The monochrome image processing unit 13 calculates the monochrome luminance (P polarization intensity + S polarization intensity of the pixel) for each pixel from the P polarization intensity data and the S polarization intensity data in the horizontal polarization image memory 11 and the vertical polarization image memory 12. . A monochrome image can be generated using the monochrome luminance data. The monochrome luminance data calculated by the monochrome image processing unit 13 is output to a white line identification unit 14 as a line detection unit.
The differential polarization degree image processing unit 15 calculates the differential polarization degree (identification index value) for each pixel from the P polarization intensity data and the S polarization intensity data in the horizontal polarization image memory 11 and the vertical polarization image memory 12. A differential polarization degree image can be generated using the differential polarization degree. The differential polarization degree is obtained from the calculation formula shown in the following formula (1). That is, the differential polarization degree is the ratio of the difference value (luminance difference value) between the P polarization intensity and the S polarization intensity to the total value (luminance total value) of the P polarization intensity and the S polarization intensity. The differential polarization degree can be paraphrased as a difference value between the ratio of the P deflection intensity to the total luminance value (P polarization ratio) and the ratio of the S deflection intensity to the total luminance value (S polarization ratio). In this embodiment, the case where the S polarization intensity is subtracted from the P polarization intensity will be described. However, the P polarization intensity may be subtracted from the S polarization intensity. The differential polarization degree data calculated by the differential polarization degree image processing unit 15 is output to the road surface structure identification unit 16 and the three-dimensional object identification unit 18 as object identification processing means.
Differential polarization degree = (P polarization intensity−S polarization intensity) / (P polarization intensity + S polarization intensity) (1)

The white line identification unit 14 identifies a white line on the traveling road surface by the following method based on the monochrome luminance data calculated by the monochrome image processing unit 13. The white line referred to here may include any line that divides the road, such as a line of an arbitrary color such as a yellow line, a solid line, a broken line, a dotted line, and a double line.
A normal road lane (division line) is formed in a high-contrast color (white or the like) with respect to a black portion such as asphalt so that the driver can easily see. Therefore, the luminance of such a lane (here, a white line) is sufficiently higher than that of an object such as asphalt existing elsewhere. Therefore, it is possible to determine a portion where the monochrome luminance data is equal to or greater than the predetermined threshold as a white line. Note that the monochrome luminance data used in the present embodiment uses the total value of the P-polarized light intensity and the S-polarized light intensity obtained by the polarization camera 10 described above.

  As a flow of white line identification processing in the present embodiment, first, the monochrome image processing unit 13 uses the total value of the P-polarized light intensity and S-polarized light intensity of each pixel obtained by the polarization camera 10 as the monochrome luminance of each pixel. calculate. The white line identification unit 14 sets a plurality of processing lines for the monochrome image obtained by the monochrome luminance. The processing line of this embodiment is set for each pixel column arranged in one horizontal row in the differential polarization degree image. The direction of the processing line is not necessarily the horizontal direction, and may be the vertical direction or the oblique direction. In addition, the number of pixels in each processing line may be the same or different. Further, the processing line is not necessarily set for all the pixels in the differential polarization degree image, and may be set for a part of appropriately selected pixels in the differential polarization degree image. Further, as will be described later, the processing may be performed in units of processing blocks (blocks each consisting of two or more pixels in the vertical and horizontal directions) instead of the processing line. The white line identification unit 14 calculates a monochrome luminance difference between two adjacent pixels for each processing line, and determines whether the calculation result is equal to or greater than a predetermined white line edge threshold. If it is determined by this determination that the threshold is equal to or greater than the white line edge threshold value, the two adjacent pixels related to the determination are stored as a white line edge. By performing this for all the processing lines, it is possible to extract the white line edge in the monochrome image.

The result of identifying the white line edge by the white line identifying unit 14 can be used for various processes.
For example, a monochrome image (front view image) generated by using the luminance data calculated by the monochrome image processing unit is displayed on a display unit (display) which is an in-vehicle information notification unit composed of a CRT, a liquid crystal or the like, In order to notify the information of the white line portion in the image as information useful for the driver, there is a process of displaying the information in a display form that is easy for the driver to see. According to this, for example, even in a situation where it is difficult for the driver to visually recognize the white line, the driver can see the front view image of the display unit, and thereby the relative positional relationship between the vehicle and the white line. It is easy to keep track of the driving lanes demarcated by the white line.
In addition, for example, from the position information of the white line recognized by the white line identification unit 14, a process of grasping the relative positional relationship between the own vehicle and the white line is performed, and the own vehicle is determined from the appropriate travel position on the travel lane partitioned by the white line. A process for determining whether or not the vehicle is running off and generating an alarm sound or the like when the vehicle is running out of the proper running position can be given. Alternatively, there is a process in which the automatic braking function is executed to reduce the traveling speed of the host vehicle when traveling out of the proper traveling position.

  Based on the differential polarization degree calculated by the differential polarization degree image processing unit 15, the road surface structure identification unit 16 refers to a structure (hereinafter referred to as “road surface structure”) present on the traveling road surface by a method described later. To identify. The road surface structure identifying unit 16 outputs the recognition result to the road surface structure identifying unit 17 and the white line identifying unit 14. Road surface structures include manhole covers, metal objects such as road connections on roads such as highways and overpasses, metal objects such as bots dots and cats eyes that form section lines that divide traveling lanes, reflectors, etc. A composite composed of: In addition, here, it is assumed to include a foreign substance that covers a part of the road surface that is not intentionally made, such as coal tar. The road surface structure identification unit 16 identifies a planar object whose outer surface is located substantially in the same plane as the road surface as a road surface structure. Whether the road surface structure is a manhole cover, a road connection unit, The road surface structure specifying unit 17 specifies whether it is a cat's eye.

  The road surface structure identification unit 16 removes the white line from the differential polarization degree image based on the white line edge identification result by the white line identification unit 14 described above, and the road surface structure object with respect to the differential polarization degree image from which the white line has been removed. An identification process may be performed. In this case, noise including a white line can be appropriately removed, and the road surface structure identification accuracy can be improved.

The manhole cover here refers to a metal plate fitted in the opening of the manhole, and is generally made of cast iron that is strong and heavy.
Botsdots B is made of ceramic, for example, which is mainly used as a section line in North America. As shown in FIG. 8 (b), the botsdots B is obtained by embedding a circular dome-shaped object having a diameter of about 100 mm in the road surface. As shown in FIG. By arranging, it is used as a section line.
The cat's eye C is used as a section line. As shown in FIG. 9B, a reflector D having a characteristic of reflecting incident light in the same direction is attached inside a substantially rectangular body. It is what was done. As shown in FIG. 8A, the cat's eyes C are used as section lines by being arranged in large numbers along the traveling lane on the road.
Both the botsdots and the cat's eyes are arranged in a state of slightly protruding from the road surface.

  The three-dimensional object identification unit 18 identifies a three-dimensional object existing in the imaging region of the polarization camera 10 based on the differential polarization degree calculated by the differential polarization degree image processing unit 15 by a method described later. Such three-dimensional objects include other vehicles traveling on the road surface, guardrails, telephone poles, street lamps, signs, roadside steps, and other obstacles existing near the road edge of the road surface. Any solid object having an outer surface facing in a direction different from the traveling road surface, such as a collision avoidance object such as a person on an upper or shoulder road, an animal, or a bicycle, is included. The three-dimensional object identification unit 18 outputs the recognition result to the three-dimensional object identification unit 19. The three-dimensional object identification unit 18 recognizes such a three-dimensional object by distinguishing it from a road object and a plane object having an outer surface substantially in the same plane as the road surface. The three-dimensional object specifying unit 19 specifies whether the object is an external obstacle or a collision avoidance object.

  Data of various shape templates as shape information used in the road surface structure specifying unit 17 and the three-dimensional object specifying unit 19 are stored in the shape storage unit 20 as the shape information storage unit. The shape template stored in the shape storage unit 20 is a shape (specification in a picked-up image) when an object (specific object) specified by the road surface structure specifying unit 17 and the three-dimensional object specifying unit 19 is imaged by the polarization camera 10. The shape of the object). For example, the shape template of the manhole cover (circular) shows an elliptical shape because the polarizing camera 10 photographs the manhole cover on the road surface from obliquely above. The shape template may include size information. In this embodiment, as a shape template that can be used in the road surface structure specifying unit 17, for example, a shape template for specifying a manhole cover, a shape template for specifying a section line made up of botsdots or cat's eyes And a shape template for identifying a road connecting portion existing on a road such as an expressway or a crossover. In addition, as a shape template that can be used by the three-dimensional object specifying unit 19, for example, a shape template for specifying another vehicle, a shape template for specifying a telephone pole or a streetlight, and a stepped portion on a road edge are specified. The shape template for doing is mentioned. Of course, a template for identifying an object other than those exemplified here may be prepared as long as the object can be identified from its shape.

  The road surface structure specifying unit 17 compares the shape of the image area identified as the road surface structure with the shape template stored in the shape storage unit 20 based on the identification result of the road surface structure identification unit 16, and the road surface Whether the structure is a manhole cover, a road connecting portion, a section line made up of botts or cats eyes, or a road surface structure other than these is specified by a method described later.

  The three-dimensional object specifying unit 19 compares the shape of the image area identified as the three-dimensional object with the shape template stored in the shape storage unit 20 based on the identification result of the three-dimensional object identification unit 18, and the three-dimensional object is Whether it is another vehicle, an off-road obstacle, a collision avoidance object, or a three-dimensional object other than these is specified by a method described later.

Next, the flow of the road surface structure specifying process for specifying the road surface structure in the driver assistance system according to the present embodiment will be described.
FIG. 10 is a flowchart showing the flow of the road surface structure specifying process.
When polarized RAW image data is acquired by the polarization camera 10, the horizontally polarized image data obtained from the P polarization intensity data included in the polarized RAW image data is stored in the horizontally polarized image memory 11 and included in the polarized RAW image data. The vertically polarized image data obtained from the S polarization intensity data is stored in the vertically polarized image memory 12 (S1). Thereafter, the differential polarization degree image processing unit 15 calculates, from the P-polarization intensity data and the S-polarization intensity data in the horizontal polarization image memory 11 and the vertical polarization image memory 12, for each pixel, from the calculation formula shown in the above formula (1). A differential polarization degree (identification index value) is calculated (S2). Data of the differential polarization degree image obtained from the calculation result is stored in an image memory (not shown) in the differential polarization degree image processing unit 15.

Next, the edge determination process will be described.
The differential polarization degree image processing unit 15 that has obtained the differential polarization degree image sets a plurality of processing lines for the differential polarization degree image. The processing line of this embodiment is set for each pixel column arranged in one horizontal row in the differential polarization degree image. The direction of the processing line is not necessarily the horizontal direction, and may be the vertical direction or the oblique direction. In addition, the number of pixels in each processing line may be the same or different. Further, the processing line is not necessarily set for all the pixels in the differential polarization degree image, and may be set for a part of appropriately selected pixels in the differential polarization degree image.

  Further, the processing may be performed in units of processing blocks (blocks each composed of two or more pixels in the vertical and horizontal directions) instead of the processing line. In this case, for example, in an edge determination process described later, a plurality of processing blocks are set for the differential polarization degree image, and a standard deviation indicating a variation degree (scattering degree) of the differential polarization degree is calculated for each processing block. When the calculated standard deviation is equal to or greater than the reference deviation threshold, it can be determined that an edge exists in the processing block. Note that the processing block may be set in a rectangular area, or may be set in an area having another shape. The size of the processing block may be about 10 × 10 pixels, for example. Each processing block may be the same size or a different size. Further, instead of the standard deviation, a statistic such as variance or average deviation may be used.

  The differential polarization degree image processing unit 15 calculates the difference in differential polarization degree between two adjacent pixels for each processing line, and determines whether or not the calculation result is equal to or greater than a predetermined edge threshold (S3). . If it is determined in this determination that the threshold value is equal to or greater than the edge threshold, the interval between two adjacent pixels related to the determination is stored as an edge (S4). By performing this for all the processing lines (S5), it is possible to specify the boundary lines of different objects in the differential polarization degree image.

  Here, in the conventional edge determination processing, for each processing line, a difference in monochrome luminance between adjacent pixels is calculated, and it is often determined whether or not the calculation result is equal to or greater than a predetermined edge threshold value. However, in this conventional edge discrimination process, for example, the edge can be discriminated between dissimilar objects with similar intensities of reflected light (monochrome luminance) received by the camera, such as manhole covers and asphalt. There wasn't. On the other hand, in the edge discrimination processing in the present embodiment, the edge discrimination is performed using the differential polarization degree instead of the monochrome luminance, so that it is possible to accurately discriminate edges between different kinds of objects having the same reflected light intensity. it can. Hereinafter, this point will be described.

FIG. 11 is an explanatory diagram showing an example of a monochrome image (luminance image) generated by the monochrome image processing unit 13 from the polarized RAW image data acquired by the polarization camera 10.
FIG. 12 is an explanatory diagram showing a differential polarization degree image generated by the differential polarization degree image processing unit 15 from the same polarized RAW image data.
FIG. 13 is a graph in which the luminance values obtained along the white dashed arrows in FIG. 11 are plotted.
FIG. 14 is a graph in which the degree of differential polarization obtained along the white dashed arrow in FIG. 12 is plotted.

As can be seen from the graph shown in FIG. 13, as for the luminance value, both the asphalt area and the manhole cover area show almost the same value (range of −0.4 or more and −0.5 or less). No change in value. On the other hand, as can be seen from the graph shown in FIG. 14, the degree of differential polarization is about zero in the asphalt area, and is about −0.3 in the manhole cover area. A large change is seen in the differential polarization degree. Therefore, by setting an appropriate threshold value (for example, -0.2) between 0 and -0.3, the asphalt area and the manhole cover, which are difficult to discriminate with monochrome luminance due to the difference in the degree of differential polarization, are set. The edge with the region can be discriminated with high accuracy.
The reason why the relative differential polarization degree is used instead of the differential polarization degree in the object identification in the present embodiment will be described later.

Next, the road surface structure identification process performed by the road surface structure identification unit 16 will be described.
In describing the road surface structure identification process, first, the reason why the road surface structure can be identified from the differential polarization degree will be described.
Reflected light reflected from an object includes a specular reflection component that is a so-called “light”, a diffuse reflection component that is a matte reflection component that is a fine uneven structure on the surface of the object, and internal scattering that is scattered inside the object. Contains ingredients. The intensity of the reflected light is expressed as the sum of these three components. The specular reflection component can also be considered as a part of the diffuse reflection component. Although the diffuse reflection component and the internal scattering component are observed regardless of the direction of the light source that irradiates the object (that is, the dependency on the incident angle is low), the specular reflection component is the light receiving unit for the reflected light. On the other hand, it is a component having a strong incident angle dependency that is observed only when a light source is present in a substantially regular reflection direction. This is also true for polarization characteristics. As described above, the diffuse reflection component and the internal scattering component are observed regardless of the direction of the light source that irradiates the object, but their polarization characteristics are different from each other. Specifically, the diffuse reflection component can be assumed to divide the object surface into minute regions and satisfy the Fresnel reflection characteristics in each region. Therefore, when non-polarized light is incident, the S-polarized component is P There is a polarization property that is larger than the polarization component. On the other hand, the internal scattering component is a component that is scattered inside the object, so when non-polarized light is incident, it is less affected by the polarization component of the light incident on the object, There is a polarization characteristic that the P-polarized light component becomes stronger when it comes out.

  And, as in this embodiment, most of the objects (such as asphalt and manhole cover) that can exist in the shooting area when shooting the front view from the host vehicle are not a little uneven on the surface. Therefore, it can be considered that the specular reflection component is small. As a result, in the present embodiment, it can be considered that the diffuse reflection component and the internal scattering component are dominant in the reflected light from the object existing in the imaging region of the polarization camera 10. As a result, by comparing the intensity of the S-polarized component and the P-polarized component in the reflected light, it can be understood that if the S-polarized component is strong, the reflected light contains a large amount of diffusely reflected component. If is strong, it can be understood that the reflected light contains many internal scattering components.

FIG. 15 shows a change in the degree of differential polarization when a P-polarized image and an S-polarized image are taken with a camera that is fixedly arranged by changing the light source position with respect to an asphalt surface and a metal surface (smooth surface) in the laboratory. It is a graph which shows an example.
In this graph, the horizontal axis represents the incident angle (light source position), and the vertical axis represents the differential polarization degree. The camera elevation angle is tilted 10 degrees from the horizontal. This differential polarization degree is calculated from luminance information at a substantially central portion of the captured image at each incident angle. The difference polarization degree in this bluff is a ratio of a value obtained by subtracting the S polarization component from the P polarization component with respect to the total value of the P polarization component (Rp) and the S polarization component (Rs). Therefore, when the P polarization component is stronger than the S polarization component, the differential polarization degree takes a positive value, and when the S polarization component is stronger than the P polarization component, the differential polarization degree is negative. Will take the value.

  As can be seen from the graph shown in FIG. 15, regarding the asphalt surface, the differential polarization degree takes a negative value over almost the entire incident angle. That is, the S polarization component is stronger than the P polarization component. This is because the diffuse reflection component is dominant in the reflected light from the asphalt surface. On the other hand, regarding the metal surface, the differential polarization degree has a positive value over the entire region where the incident angle exceeds 30 °. That is, the P-polarized component is stronger than the S-polarized component. This is because the internal scattering component is dominant in the reflected light from the metal surface.

As described above, by taking the difference between the S-polarized component and the P-polarized component included in the reflected light, it is possible to grasp whether the diffuse reflection component or the internal scattering component is strong in the reflection characteristics of the object. it can. Therefore, an object having different reflection characteristics, such as an asphalt having a strong diffuse reflection component and a road surface structure having a strong internal scattering component, can be identified based on the difference between the S polarization component and the P polarization component included in the reflected light. it can.
In general, since the refractive index is different for different materials, this effect also appears in the difference between the S-polarized component and the P-polarized component. Therefore, objects with different materials can be identified based on the difference between the S-polarized component and the P-polarized component included in the reflected light.

  Note that the results of the graph shown in FIG. 15 show the difference in the surface state between the asphalt surface and the metal surface (asphalt is a surface on which many irregularities are distributed, whereas the metal surface is a smooth surface). It is thought that they are also affected. Therefore, the values of the S-polarized component and the P-polarized component that are detected vary depending on the difference in the surface state of the identification target. This is because the differential polarization degree of the manhole cover, which is an asphalt area or a road structure (a metal object on the road) in the graph of FIG. 14 showing the experimental results in an actually used environment, is the laboratory shown in FIG. It can be understood from the difference from the experimental results. However, between objects of different materials, even if the surface states are the same, the reflection characteristics of the objects (diffuse reflection component and internal scattering component strength, refractive index, etc.) differ due to the difference in the material. It is possible to grasp the difference based on the difference between the S-polarized component and the P-polarized component included in the reflected light. Therefore, it is possible to distinguish between objects of different materials by using a threshold value set in consideration of the actual environment.

  In the road surface structure identification process of the present embodiment, by using the differential polarization degree of each area divided by the edge determined by the edge determination process described above as an identification index value, it is determined whether or not the area is a road surface structure. Determine whether. As described above, the road surface structure identification process can determine whether or not the region is a road surface structure even if the difference value between the S-polarized light intensity and the P-polarized light intensity is used as the identification index value. However, when the brightness is insufficient, the difference value between the S-polarized light intensity and the P-polarized light intensity calculated as the identification index value takes a small value, so that it is difficult to determine whether or not the road surface structure is present. On the other hand, as in this embodiment, if the difference polarization degree that is a value obtained by dividing the difference value by the total value of S polarization intensity and P polarization intensity (monochrome luminance) is used, even if the brightness is insufficient, The degree of differential polarization as the identification index value can take a relatively large value, and it can be determined whether or not the road surface structure.

  The flow of the road surface structure identification process of the present embodiment will be described. First, for each area divided by the edge determined by the edge determination process described above, the difference between the difference polarization degree of the area and the reference difference polarization degree is calculated. Calculate (S6). And it is judged whether this difference value is below the threshold value for road surface structures determined beforehand (S7). In this determination, when it is determined that the difference value is equal to or less than the road surface structure threshold, the area is identified and stored as a road surface structure (S8). By performing this for all areas (S9), it is possible to grasp the area where the road surface structure is displayed in the captured image.

  In the present embodiment, the differential polarization degree (absolute amount) of the area is not used as the identification index value used for object identification of the area, but the relative polarization degree of the area is subtracted from the reference differential polarization degree. The differential polarization degree (relative amount) is used. This is because, even if a difference occurs in the calculated value of the differential polarization degree of the identification target area due to the difference in the shooting environment, etc., by using the relative amount with the reference differential polarization degree where the deviation occurs due to the same effect, This is because the influence can be reduced. In the present embodiment, the differential polarization degree in the asphalt region occupying most of the road surface is used as the reference differential polarization degree. Further, in the present embodiment, it has been experimentally found that the differential polarization degree for the road surface structure is at least 0.2 or more larger than the differential polarization degree for asphalt under an actual environment. Yes. Therefore, the relative differential polarization degree with respect to the road surface structure takes a negative value of −0.2 or less. Therefore, in the present embodiment, −0.2 is adopted as the road surface structure threshold value, and a region having a relative differential polarization degree belonging to a range equal to or smaller than the threshold value is identified as a road surface structure.

Here, there is a difference in the intensity of the reflected light from the object between the upper part and the lower part of the captured image. This is because the upper part of the captured image is a part obtained by photographing an object located far away, and the intensity of reflected light is smaller than the lower part obtained by photographing an object located nearby. Therefore, in consideration of this difference, the road surface structure threshold value to be used may be different between the upper part and the lower part of the captured image.
In addition, since the lower part of the captured image obtained by photographing an object located nearby has higher object identification accuracy than the upper part, the processing order of the processing lines is preferably the order from the lower side to the upper side of the image.

Next, the road surface structure type specifying process performed by the road surface structure specifying unit 17 will be described.
First, the road surface structure specifying unit 17 recognizes the shape of the area identified as the road surface structure by the road surface structure identification process described above (S10), and whether there is a shape template that approximates the shape of the area. It is determined whether or not (S11). In this determination, if it is determined that there is an approximate shape template, the road surface structure in the region is specified as a type associated with the shape template and stored (S12). For example, when approximating an oval shape template, it is specified as a manhole cover, and when approximating a bar shape template traversing in the horizontal direction of the image, it is specified as a road connecting portion, and a plurality of road surfaces When an article is posted on a shape template that is linearly arranged along the traveling direction of the vehicle, it is specified as a section line made up of botsdots or cat's eyes. If this is performed for all road surface structures (S13), the process is terminated.

Specifically, using the edge information of the road surface structure region and the shape template information stored in the shape storage unit 20, an approximate curve is obtained, and the least square method, the Hough transform, the model equation, etc. Shape approximation recognition is performed using a technique. Note that when obtaining an approximate curve, it is desirable that the edge information located in the lower portion of the highly reliable captured image is given a higher weight to the voting value for shape approximation. In this way, even if there is edge information that is misrecognized in the upper part of the low-reliability captured image, if there is edge information that is normally recognized in the lower part of the highly reliable captured image, The kind of road surface structure can be specified appropriately.
For example, when a manhole cover is specified, feature points related to an object larger than the manhole cover may be removed in advance by morphological calculation or the like. Thereby, the specific precision of a manhole cover can be improved.

In order to increase the specific accuracy of the type of road surface structure, the following processing may be added.
In the present embodiment, the road surface structure specifying process (S1 to S13) as described above is performed on polarized image data obtained by continuously photographing with a polarization camera 10 at a predetermined time interval. With respect to the area where the type of the road surface structure is specified by the above-described road surface structure type specifying process (S10 to S13), the processing result is stored in a predetermined memory. Using the past processing result (for example, the processing result of the polarization image data captured immediately before) stored in this memory, the type of the road surface structure specified by this processing corresponds to the past. If the processing result is the same as this, it is determined that the current processing result is highly reliable. And this reliability is utilized for specification of the kind of final road surface structure. The past processing result corresponding to the region related to the current processing result is related to the corresponding past processing result from the position of the region related to the current processing result and the traveling direction of the own vehicle, for example, using edge information. The position of the area is searched and the corresponding past processing result is specified.
In addition, although the road surface structure type specifying process (S10 to S13) has been described here, in order to improve the identification accuracy of the road surface structure identifying process (S6 to S9) as well, the past processing result is used. May be used.

  In addition, the edge threshold value used in the determination of the edge determination process (S3) and the road surface structure threshold value used in the determination of the road surface structure identification process (S7) are appropriately switched according to the difference in the photographing environment. Also good. A specific example is switching between time zones such as daytime and nighttime according to the difference in weather such as rainy weather and fine weather. This switching can be realized using time information or information such as a rain sensor or a sunshine sensor.

  In addition, when the polarization camera 10 according to the present embodiment is mounted in a vehicle such as a room mirror, an image captured by the polarization camera 10 is affected by the windshield. Therefore, it is desirable to consider the polarization characteristics of the windshield. As shown in FIG. 16, the windshield is usually disposed with a predetermined angle with respect to the optical axis of the polarizing camera 10. Generally, when a glass flat plate is disposed obliquely with respect to the optical path, the polarization state of the transmitted light changes. The calculation of so-called Fresnel transmission reflection is established, and the amount of S-polarized light component reflected by the glass surface is larger than that of P-polarized light component, and the attenuation factor of the transmitted light is higher for S-polarized light component than for P-polarized light component. large. Specifically, for example, in the example shown in FIG. 16, the P-polarized component is attenuated by about 8% by Fresnel reflection on the glass surface, whereas the S-polarized component is attenuated by about half. In this example, the refractive index at the windshield was 1.5.

  Thus, since the influence of the windshield is included in the polarization information of the light taken into the polarization camera 10, it is desirable to take this into consideration. For example, at the time of calculating the differential polarization degree (S2), an operation for canceling the attenuation component on the windshield is performed for each of the P polarization intensity and the S polarization intensity. In the above example, the S polarization component is approximately doubled and the P polarization component is approximately 1 / 0.9. As another method, for example, an optical element that cancels attenuation components of P-polarized light intensity and S-polarized light intensity is disposed between the windshield and the polarizing camera 10. As the optical element in this case, in the above example, an optical element that transmits the S-polarized component through the dead band and increases the P-polarized component by 0.5 / 0.9 times can be used.

  In the future, it is expected that plastic windshields will be used to reduce weight and cost. It is known that plastic has birefringence due to internal strain compared to glass. In this case, the influence of birefringence needs to be considered. Specifically, for example, with respect to the polarization component of the light captured by the polarization camera 10, the differential polarization degree is calculated in consideration of the P-polarization component and the S-polarization component due to birefringence.

The result of specifying the type of the road surface structure by the road surface structure specifying unit 17 can be used for various processes.
For example, the processing result of the road surface structure specifying unit 17 may be used for white line edge identification processing in the white line identification unit 14. Specifically, since the area where the type of the road surface structure is specified by the processing of the road surface structure specifying unit 17 is not a white line area, this area is excluded from the target of the white line edge identification process by the white line identification unit 14. . Thereby, the possibility of erroneously recognizing a road surface structure such as a manhole cover as a white line can be reduced, and the recognition accuracy of the white line can be improved. Here, the white line edge identifying process by the white line identifying unit 14 has been described. However, when performing the process of identifying an object other than a white line from the captured image, the road surface structure specifying unit 17 generally performs the road surface structure identification process. By excluding the region in which the type of the object is specified from the identification processing target, the identification accuracy of the identification processing can be improved. As a specific example, in a system that identifies obstacles such as a preceding vehicle by sensor fusion based on radar ranging results and camera images, various road surface structures such as manhole covers are misrecognized as obstacles. You can avoid doing that. As a result, it is possible to prevent the occurrence of a state in which the speed is rapidly decreased due to erroneous recognition as various road surface structure obstacles.

  Further, for example, it may be used for a car navigation system. As a specific example, the vehicle position information such as the distance and angle between the vehicle and the manhole cover is generated from the position of the manhole cover specified from the processing result of the road surface structure specifying unit 17, and the vehicle position Using the information, a further detailed position of the own vehicle is specified within the range of the own vehicle position calculated by the car navigation system. Thereby, the specific precision of the own vehicle position in a car navigation system can be improved.

Further, for example, the position and direction of the manhole cover and the road connecting portion relative to the own vehicle can be grasped from the processing result of the road surface structure specifying unit 17, and this is used for a travel support ECU (Electronic Control Unit) or the like. It can also be mentioned.
In particular, for example, the road line structure specifying unit 17 can use the section line specifying result made up of botsdots or cat's eyes for the same processing as the white line edge identifying result by the white line identifying unit 14. Specifically, for example, a monochrome image (front view image) generated using luminance data calculated by the monochrome image processing unit on a display unit (display) that is an in-vehicle information notification unit configured by a CRT, liquid crystal, or the like. ) And the information on the section line in the image is reported as information useful for the driver, and the display is easy to see for the driver. According to this, for example, even in a situation where it is difficult for the driver to visually recognize the section line, the driver can view the relative position between the vehicle and the section line by looking at the front view image on the display unit. It is possible to grasp the positional relationship, and it is easy to maintain the traveling lane divided by the section line and to travel.
In addition, for example, a process for grasping the relative positional relationship between the own vehicle and the section line from the position information of the section line recognized by the road surface structure specifying unit 17 is performed on the travel lane where the own vehicle is partitioned by the section line. There is a process of determining whether or not the vehicle is traveling away from the proper travel position and generating an alarm sound or the like when traveling outside the proper travel position. Alternatively, there is a process in which the automatic braking function is executed to reduce the traveling speed of the host vehicle when traveling out of the proper traveling position.

FIG. 17 is a diagram illustrating an example of a monochrome image (luminance image) generated by the monochrome image processing unit 13 from the polarized RAW image data obtained by capturing an image of a road surface using Botsdots as a section line with the polarization camera 10. FIG.
FIG. 18 is an explanatory diagram showing a differential polarization degree image generated by the differential polarization degree image processing unit 15 from the same polarized RAW image data.
As can be seen by comparing FIG. 17 and FIG. 18, the difference polarization degree image shown in FIG. 18 has higher contrast between asphalt and bottsdots than the monochrome image (luminance image) shown in FIG. Therefore, by using the differential polarization degree image, it is possible to discriminate the edge between the asphalt region and the botsdot region, which are difficult to discriminate with monochrome luminance, with high accuracy.

In the present embodiment, description has been made mainly using metal as an example of an identifiable material. However, even other materials can be identified.
FIG. 19 shows the difference in the degree of polarization when a P-polarized image and an S-polarized image are taken with a camera that is fixedly arranged by changing the light source position on the asphalt surface and the coated surface of steel coated with paint. It is a graph which shows a change. This graph has the same conditions as the graph shown in FIG. 15 in which the asphalt surface and the metal surface are compared. As can be seen from this graph, there is a difference in the degree of differential polarization between the asphalt surface and the painted surface. The differential polarization degree of the painted surface is also different from the differential polarization degree of the metal surface shown in FIG. Therefore, the painted surface and the metal surface can also be distinguished and recognized by the difference in the degree of differential polarization (difference in polarization characteristics).

Similarly, not only such a painted surface but also a road surface structure such as coal tar attached on the road surface during construction or due to deterioration of the road surface, using a differential polarization degree image, compared to a monochrome luminance image. It is possible to detect with high contrast.
FIG. 20 is an explanatory diagram illustrating an example of a monochrome image (luminance image) generated by the monochrome image processing unit 13 from the polarized RAW image data obtained by capturing an image of the road surface to which coal tar is attached with the polarization camera 10.
FIG. 21 is an explanatory diagram showing a differential polarization degree image generated by the differential polarization degree image processing unit 15 from the same polarized RAW image data.
As can be seen by comparing FIG. 20 and FIG. 21, the contrast between asphalt and coal tar is higher in the differential polarization degree image shown in FIG. 21 than in the monochrome image (luminance image) shown in FIG. Therefore, if the differential polarization degree image is used, the edge between the asphalt area and the coal tar area, which is difficult to determine with monochrome luminance, can be determined with high accuracy.

Next, a flow of a three-dimensional object specifying process for specifying a three-dimensional object in the driver assistance system according to the present embodiment will be described.
FIG. 22 is a flowchart showing the flow of the three-dimensional object specifying process.
Note that the processing up to the edge determination processing is the same as the above-described road surface structure specifying processing, and thus description thereof is omitted. However, the edge threshold value used in the edge determination process uses a value different from the road surface structure specifying process described above. A method for setting the edge threshold used in the edge discrimination process will be described below.

FIG. 23 is an explanatory diagram showing an example of a monochrome image (luminance image) generated by the monochrome image processing unit 13 from the polarized RAW image data acquired by the polarization camera 10.
FIG. 24 is an explanatory diagram showing a differential polarization degree image generated by the differential polarization degree image processing unit 15 from the same polarized RAW image data.
FIG. 25 is a histogram of the luminance value distribution of three white frames in FIG. 23 for 100 frames.
FIG. 26 is a histogram of differential polarization degree distributions at three positions in the white frame in FIG. 24 for 100 frames.
As can be seen from the histogram shown in FIG. 25, with respect to the luminance value, the luminance distribution in the asphalt region, the luminance distribution in the vehicle side surface region of the other vehicle, and the luminance distribution in the vehicle rear surface region of the other vehicle overlap each other. On the other hand, as can be seen from the histogram shown in FIG. 26, for the differential polarization degree, the luminance distribution of the asphalt region, the luminance distribution of the vehicle side surface region of the other vehicle, and the luminance distribution of the vehicle rear surface region of the other vehicle, They can be separated without overlapping each other. Therefore, by setting appropriate thresholds that can distinguish these areas, it is difficult to distinguish with monochrome luminance due to differences in the degree of differential polarization, asphalt areas, vehicle side areas of other vehicles, vehicle rear areas of other vehicles Can be discriminated with high accuracy.

Next, the three-dimensional object identification process performed by the three-dimensional object identification unit 18 will be described.
In describing the three-dimensional object identification process, first, the reason why the three-dimensional object can be identified from the differential polarization degree will be described.
The reflected light reflected by the object differs in the incident angle of light taken into the polarization camera 10 between the road surface and the side surface of the three-dimensional object (the outer surface facing the direction different from the road surface). Differences in strength occur. In particular, when the side surface of the three-dimensional object is a surface substantially upright with respect to the road surface, the relative relationship between the P-polarized component and the S-polarized component contained in the reflected light from the side surface of the three-dimensional object is the reflected light from the road surface. Is equivalent to the one in which the relative relationship between the P-polarized component and the S-polarized component included in is replaced. In general, the relative relationship between the P-polarized light component and the S-polarized light component contained in the reflected light is a polarized light component perpendicular to the incident surface rather than the P-polarized light component that is a polarized light component parallel to the incident surface. There is a relationship that the S polarization component is larger. Therefore, when the reflected light from the road surface or a plane parallel to the road surface is received by the polarization camera 10, the S-polarized light component is stronger than the P-polarized light intensity, and the reflected light from the side surface of the three-dimensional object substantially upright with respect to the road surface. Is received by the polarizing camera 10, the P-polarized light component is stronger than the S-polarized light intensity. By comparing the intensity of the S-polarized component and the P-polarized component in the reflected light received by the polarization camera 10 due to the difference in polarization characteristics between the road surface and the three-dimensional object, the road surface is increased if the S-polarized component is strong. It can be understood that the reflected light is from a plane parallel to the road surface. If the P-polarized light component is strong, it can be understood that the reflected light is from a plane perpendicular to the road surface. As a result, by taking the difference value between the S-polarized component and the P-polarized component contained in the reflected light received by the polarization camera 10, depending on whether the difference value is positive or negative, whether the object has a plane parallel to the road surface, Can grasp whether the object has an outer surface facing in a different direction, that is, a three-dimensional object.

  In the three-dimensional object identification process according to the present embodiment, whether or not the area is a three-dimensional object is determined by using the differential polarization degree of each area divided by the edge determined by the edge determination process described above as an identification index value. judge. As described above, the three-dimensional object identification process can determine whether or not the region is a three-dimensional object even if the difference value between the S-polarized light intensity and the P-polarized light intensity is used as the identification index value. However, when the brightness is insufficient, the difference between the S-polarized light intensity and the P-polarized light intensity calculated as the identification index value takes a small value, so that it is difficult to determine whether the object is a three-dimensional object. On the other hand, as in this embodiment, if the difference polarization degree that is a value obtained by dividing the difference value by the total value of S polarization intensity and P polarization intensity (monochrome luminance) is used, even if the brightness is insufficient, The degree of differential polarization as the identification index value can take a relatively large value, and it can be determined whether or not the object is a three-dimensional object.

  The flow of the three-dimensional object identification process according to the present embodiment will be described. For each area divided by the edge determined by the edge determination process described above, the differential polarization degree of the area is equal to or less than a predetermined threshold for a three-dimensional object. It is determined whether or not (S21). In the present embodiment, the differential polarization degree for a three-dimensional object takes a positive value as can be seen from the histogram shown in FIG. Therefore, in the present embodiment, a positive value near zero (for example, +0.05) is adopted as the threshold for a three-dimensional object, and an area having a differential polarization degree belonging to a range equal to or greater than this threshold is identified as a three-dimensional object. In addition, the same process is possible even if it uses relative difference polarization degree like the road surface structure identification process mentioned above. In this determination, when it is determined that the differential polarization degree is equal to or less than the threshold for the three-dimensional object, the region is identified and stored as a three-dimensional object (S22). By performing this for all areas (S23), it is possible to grasp the area where the three-dimensional object is displayed in the captured image.

Also, since there is a difference in the intensity of reflected light from the object between the upper part and the lower part of the captured image, the threshold for solid objects to be used is different between the upper part and the lower part of the captured image in consideration of this difference. You may make it let.
In addition, since the lower part of the captured image obtained by photographing an object located nearby has higher object identification accuracy than the upper part, the processing order of the processing lines is preferably the order from the lower side to the upper side of the image.

Next, the three-dimensional object type specifying process performed by the three-dimensional object specifying unit 19 will be described.
First, the three-dimensional object specifying unit 19 recognizes the shape of the area identified as a three-dimensional object by the above-described three-dimensional object identification process (S24), and determines whether there is a shape template that approximates the shape of the area. (S25). In this determination, when it is determined that there is an approximate shape template, the three-dimensional object in the region is specified as a type associated with the shape template and stored (S26). For example, in the case of approximating a vehicle shape template, it is specified as another vehicle. If this is performed for all three-dimensional objects (S27), the process ends. In addition, since the method of shape approximation recognition in the three-dimensional object specifying unit 19 is the same as the processing method of the road surface structure specifying unit 17 described above, description thereof is omitted.

In order to improve the accuracy of the three-dimensional object type specifying process and the three-dimensional object identifying process, the past processing result may be used as in the case of specifying the type of the road surface structure.
Further, the edge threshold value used in the determination of the edge determination process (S3) and the threshold value for the three-dimensional object used in the determination process of the three-dimensional object (S21) may be switched as appropriate according to the difference in the shooting environment. . A specific example is switching between time zones such as daytime and nighttime according to the difference in weather such as rainy weather and fine weather. This switching can be realized using time information or information such as a rain sensor or a sunshine sensor.
In addition, when the polarizing camera 10 of the present embodiment is mounted in a vehicle such as a rearview mirror, it is desirable to consider the polarization characteristics of the windshield as in the road surface structure specifying process described above.

The result of specifying the type of the three-dimensional object by the three-dimensional object specifying unit 19 can be used for various processes.
For example, from the result of processing by the three-dimensional object specifying unit 19, the driver is warned of the approach of a recognized three-dimensional object that is an obstacle to be avoided, or the collision is avoided or controlled by controlling the vehicle's automatic brake system. To reduce the impact.
Further, for example, the processing result of the three-dimensional object specifying unit 19 may be used for white line edge identification processing in the white line identification unit 14. Specifically, since the region in which the type of the three-dimensional object is specified by the processing of the three-dimensional object specifying unit 19 is not a white line region, this region is excluded from the target of white line edge identification processing by the white line identification unit 14. Thereby, the possibility of erroneously recognizing a solid object such as another vehicle as a white line can be reduced, and the recognition accuracy of the white line can be improved. Here, the white line edge identifying process by the white line identifying unit 14 has been described. However, in the case of performing a process of identifying an object other than a white line from a captured image, in general, the processing of the three-dimensional object specifying unit 19 is performed. By excluding the region whose type is specified from the target of the identification process, the identification accuracy of the identification process can be improved.

  Further, for example, it may be used for a car navigation system. As a specific example, the distance, angle, etc. between the vehicle and the off-road obstacle is determined from the position of the off-road obstacle such as a telephone pole, a streetlight, and a sign specified from the processing result of the three-dimensional object specifying unit 19. Vehicle position information is generated, and the vehicle position information is used to specify a further detailed position of the vehicle within the range of the vehicle position calculated by the car navigation system. Thereby, the specific precision of the own vehicle position in a car navigation system can be improved.

Further, for example, the position and direction of various three-dimensional objects with respect to the own vehicle can be grasped from the processing result of the three-dimensional object specifying unit 19, and this may be used for a travel support ECU (Electronic Control Unit) or the like. .
In particular, a three-dimensional object that should avoid a collision is generated by using, for example, luminance data calculated by a monochrome image processing unit on a display unit (display) that is an in-vehicle information notification unit composed of a CRT, a liquid crystal, or the like. In order to display a monochrome image (front view image) and information of a three-dimensional object in the image as useful information for the driver, there is a process of displaying the information in a display form that is easy for the driver to see. According to this, for example, even in a situation where it is difficult for the driver to visually recognize the three-dimensional object, the driver confirms the presence of the three-dimensional object by looking at the front view image on the display unit. This makes it easy to avoid collisions.

As described above, the object identification device according to the present embodiment receives two polarized lights (P-polarized light and S-polarized light) having different polarization directions included in the reflected light from the object existing in the imaging region, A polarization camera 10 as an imaging means for capturing a polarized image (P-polarized image and S-polarized image), and a P-polarized image and an S-polarized image captured by the polarized camera 10 are each divided into predetermined processing areas (units of one pixel). The differential polarization degree image processing unit 15 as a luminance difference value calculation unit that calculates the luminance difference value between the P-polarized image and the S-polarized image for each pixel, and the luminance calculated by the differential polarization degree image processing unit 15 The road surface structure identification unit 16 as an object identification processing unit that performs identification processing of an object existing at a location corresponding to each pixel in the imaging region using the differential polarization degree that is an identification index value obtained from the difference value. And a fine solid object identifying unit 18. Thereby, it is possible to distinguish and recognize between the road surface and the road surface structure or the road surface and the three-dimensional object that could not be identified when using the difference in monochrome luminance.
Further, in the present embodiment, the road surface structure identifying unit 16 and the three-dimensional object identifying unit 18 perform a determination process for determining which of a plurality of numerical ranges each determined for each of a plurality of different objects belongs to the degree of differential polarization. The identification process is performed by performing a process of identifying an object existing at a location corresponding to the processing area as an object corresponding to the numerical range determined to belong to the determination process. With such a process, the identification process can be realized by a simple process of comparison with a threshold value.
In the present embodiment, the polarization camera 10 is mounted on a vehicle (own vehicle) as a moving body that moves on a road surface that is a moving surface, and images an imaging region including the road surface obliquely from above the road surface. The numerical range defined for the same object is set for each of at least two or more areas that divide the P-polarized image and the S-polarized image in the vertical direction, and the difference between the processing areas belonging to the area is set. An identification process is performed by determining which of the numerical ranges set for the area the polarization degree belongs to. Accordingly, it is possible to appropriately identify the difference in the amount of light received by the polarization camera 10 in the vertical direction of the P-polarized image and the S-polarized image.
In the present embodiment, the road surface structure identification unit 16 performs a material identification process for identifying the material (metal) of an object present at a location corresponding to each processing area in the imaging area. Objects of different materials existing in the same plane, which could not be identified when using the difference in monochrome luminance, can be appropriately identified by the difference in their polarization characteristics.
Further, in the present embodiment, the polarization camera 10 is mounted on a vehicle that is a moving body that moves on a road surface that is a moving surface, and images an imaging region that includes the road surface, and is substantially flush with the road surface. The object (road surface structure) whose outer surface is exposed is identified. Therefore, a road surface structure such as a manhole cover, a road connecting portion, a section line made of botsdots or cat's eyes can be identified.
In the present embodiment, the road surface structure identification unit 16 calculates the difference calculated by the differential polarization degree image processing unit 15 for the reference processing region corresponding to the location of the asphalt as the reference object that is assumed to exist in advance. Using the degree of polarization as a reference index value, the relative polarization degree calculated by the difference polarization degree image processing unit 15 for a processing area different from the reference processing area is calculated with respect to the reference index value (relative difference polarization degree), Based on this relative differential polarization degree, a process of identifying an object present at a location corresponding to the processing region is performed. As a result, even if there is a deviation in the calculated value of the differential polarization degree of the identification target area due to the influence of the difference in the shooting environment, etc., by using the relative amount with the reference differential polarization degree where the deviation is caused by the same influence, The impact can be reduced.
In this embodiment, since the differential polarization degree indicating the ratio of the luminance difference value (P-polarization intensity-S-polarization intensity) to the luminance total value (P-polarization intensity + S-polarization intensity) is used as the identification index value. Even if there is not enough, recognition with high accuracy is possible.
In the present embodiment, the road surface structure identification unit 16 and the three-dimensional object identification unit 18 are stored in a memory serving as an identification process result storage unit that stores the results of identification processes performed in the past. 16 and the three-dimensional object identification unit 18 perform the identification process using the result of the past identification process stored in the memory together with the differential polarization degree. This makes it possible to determine the reliability of the identification result as to whether the same result as the past identification result has been obtained.
Further, in the present embodiment, it has a shape storage unit 20 as a shape information storage unit that stores a shape template that is shape information indicating a shape when a predetermined specific object is imaged by the polarization camera 10, and includes an object identification processing unit. The road surface structure specifying unit 17 and the three-dimensional object specifying unit 19 are adjacent to each other that are identified as the same object (road surface structure or three-dimensional object) by the identification processing of the road surface structure identifying unit 16 and the three-dimensional object identifying unit 18. It is determined whether or not the shape indicated by the plurality of pixels approximates the shape of the shape template stored in the shape storage unit 20, and when it is determined that the shape is approximated, it exists at a location corresponding to the plurality of pixels. An object specifying process for specifying an object as a specific object corresponding to the shape template is performed. Thereby, the object can be specified not only from the polarization characteristic of the object but also from the shape information of the object.
In addition, as described above, the object identification device according to the present embodiment is a travel support ECU as a movement control unit that performs movement control of a vehicle (own vehicle) that is a moving body using the identification result of the object identification device. It can be applied to an automatic brake system as a moving body control device equipped with
In addition, as described above, the object identification device according to the present embodiment generates information useful for a driver who operates a vehicle (own vehicle) that is a moving body using the identification result of the object identification device. The present invention can also be applied to an information providing apparatus that notifies the driver of the generated information.

Note that the driver assistance system according to the present embodiment is mounted on the vehicle as a whole, but the entire system does not necessarily have to be mounted on the vehicle. Therefore, for example, only the polarization camera 10 may be mounted on the own vehicle, and the remaining system components may be remotely arranged at a location different from the own vehicle. In this case, a system in which a person other than the driver objectively grasps the traveling state of the vehicle can be provided.
In the present embodiment, the differential polarization degree is used as the index value, but the differential value itself may be used.

DESCRIPTION OF SYMBOLS 10 Polarization camera 11 Horizontally polarized image memory 12 Vertically polarized image memory 13 Monochrome image processing part 14 White line identification part 15 Differential polarization degree image processing part 16 Road surface structure identification part 17 Road surface structure identification part 18 Three-dimensional object identification part 19 Three-dimensional object identification Unit 20 Shape storage unit 101, 111, 112 Camera 102 Rotating polarizer 113, 114, 124, 133, 134, 144, 152, 153 Polarization filter

Japanese Patent Laid-Open No. 11-175702

Claims (13)

In an object identification device for identifying an object existing in an imaging region,
Imaging means for receiving two polarized light beams having different polarization directions included in reflected light from an object existing in the imaging region and capturing respective polarized images;
An object identification apparatus comprising: object identification processing means for performing identification processing of an object existing at a location corresponding to each processing area in the imaging area using two polarized images captured by the imaging means. In an object identification device for identifying an object existing in an imaging region,
Imaging means for receiving two polarized light beams having different polarization directions included in reflected light from an object existing in the imaging region and capturing respective polarized images;
A luminance difference value calculating unit that divides two polarized images captured by the imaging unit into predetermined processing regions, and calculates a luminance difference value between the two polarized images for each processing region;
Using an identification index value obtained from the luminance difference value calculated by the luminance difference value calculating means, and an object identification processing means for performing identification processing of an object existing at a location corresponding to each processing area in the imaging area. An object identification device. The object identification device according to claim 2,
The object identification processing means performs a determination process to determine which of a plurality of numerical ranges defined for each of a plurality of different objects each belongs to the identification index value, and an object present at a location corresponding to the processing area The object identification device is characterized in that the identification process is performed by performing a process of identifying the object corresponding to the numerical range determined to belong to the determination process. The object identification device according to claim 3.
The imaging means is mounted on a moving body that moves on a moving surface, and picks up an imaging region including the moving surface obliquely from above the moving surface,
In the determination process, a numerical range defined for the same object is set for each of at least two or more sections that respectively divide the two polarized images in the vertical direction, and an identification index of a processing region belonging to the section. An object identification device for determining which of a numerical range set for the area a value belongs to. In the object identification device according to any one of claims 2 to 4,
The object identification device, wherein the identification process includes a material identification process for identifying a material of an object existing at a location corresponding to each processing area in the imaging area. The object identification device according to claim 5, wherein
An object identification device, wherein the material of the object identified by the material identification process includes metal. The object identification device according to any one of claims 2 to 6,
The imaging means is mounted on a vehicle that is a moving body that moves on a road surface that is a moving surface, and images an imaging region that includes the road surface.
The object identification processing means performs identification processing of an object whose outer surface is exposed on substantially the same plane as the road surface. The object identification device according to any one of claims 2 to 7,
The object identification processing means uses an identification index value obtained from the luminance difference value calculated by the luminance difference value calculation means for a reference processing area corresponding to a location where a reference object is assumed to exist in advance as a reference index value. And calculating a relative value of the identification index value obtained from the brightness difference value calculated by the brightness difference value calculation means for the processing area different from the reference processing area with respect to the reference index value, and performing the processing based on the relative value. An object identification apparatus characterized in that the identification process is performed by performing a process of identifying an object existing at a location corresponding to a region. The object identification device according to any one of claims 2 to 8,
An object identification device using a differential polarization degree indicating a ratio of a luminance difference value to a luminance total value between two polarized images used when calculating the luminance difference value as the identification index value. In the object identification device according to any one of claims 2 to 9,
An identification processing result storage means for storing the result of the identification processing performed in the past by the object identification processing means;
The object identification processing unit performs the identification process by using the identification index value and the result of the past identification process stored in the identification process result storage unit. The object identification device according to any one of claims 2 to 10,
Having shape information storage means for storing shape information indicating a shape when a predetermined specific object is imaged by the imaging means;
Whether the object identification processing means approximates the shape of the shape information stored in the shape information storage means with the shape indicated by the plurality of processing regions adjacent to each other identified as the same object by the identification processing An object identification device that performs an object identification process that identifies an object that exists at a location corresponding to the plurality of processing areas as the specific object when it is determined whether the object is approximate. A movement control means for performing movement control of the moving body;
An object identification unit that images the surroundings of the moving object as an imaging target and identifies an object existing in the imaging target;
In the moving body control device that performs the movement control using the identification result by the object identification unit,
A moving body control device using the object identification device according to any one of claims 1 to 11 as the object identification means. An object identification unit that captures an image of a moving object that moves in accordance with a driving operation by a driver as an imaging target, and that identifies an object existing in the imaging target;
Useful information generating means for generating information useful for the driver using the identification result by the object identifying means;
In an information providing apparatus having information notifying means for notifying the driver of information generated by the useful information generating means,
An information providing apparatus using the object identifying apparatus according to any one of claims 1 to 11 as the object identifying means.