JP5686281B2 - Three-dimensional object identification device, and mobile object control device and information providing device provided with the same - Google Patents

Description

  The present invention relates to a three-dimensional object identification device that identifies a three-dimensional object existing in an imaging region, and a moving object such as a vehicle, a ship, an aircraft, or an industrial robot using the identification result of the three-dimensional object identification device. The present invention relates to a mobile control device that performs the above-described movement control and an information providing device that provides useful information to a driver of the mobile body.

  As this type of three-dimensional object identification device, for example, a device used in a driver support system such as ACC (Adaptive Cruise Control) for reducing the driving load of a driver (driver) of a vehicle is known. . In such a vehicle driving support system, an automatic brake function and an alarm function for preventing the own vehicle from colliding with an obstacle such as a side wall or reducing an impact at the time of collision, an inter-vehicle distance from a preceding vehicle In order to realize various functions such as the vehicle speed adjustment function for maintaining the vehicle, obstacles such as side walls around the vehicle and solid objects such as the preceding vehicle are properly distinguished and recognized (identified) ) Is required. Therefore, various three-dimensional object identification devices have been conventionally proposed.

  Patent Document 1 discloses a three-dimensional object identification device that can correctly recognize two objects of the same color even when two objects of the same color overlap. This three-dimensional object identification device takes advantage of the fact that the principal axis directions of polarized light (the direction in which the polarization intensity is maximum) are not the same even for objects of the same color, so that even if two objects of the same color overlap, It distinguishes and recognizes. Specifically, an image is acquired through a plurality of polarizers, and the polarization component at the location is calculated using the received light intensity of a predetermined image location obtained through each of the polarizers having different polarization directions, and A polarization component is calculated for the entire image. Then, a region in which the principal axis directions of polarized light included in the polarization component are considered to be the same is divided, a movement direction of each of the divided regions is calculated, and an image portion of a region having the same movement direction is determined as one object. Identify as.

  A conventional three-dimensional object identification device uses a difference in luminance in a captured image to create a three-dimensional object having a planar object (for example, asphalt on a road surface) existing in a predetermined plane and an outer surface facing in a direction different from the predetermined plane. Extracting edges between objects (for example, roadside obstacles such as side walls, guardrails, telephone poles, street lamps, stepped parts of pedestrian paths, etc. existing on roadsides) It is common to recognize that However, in the conventional method using the difference in luminance as a method for distinguishing and recognizing a three-dimensional object and a planar object, if there is a part with greatly different luminance in the same planar object, the boundary between these parts is represented by an edge. In some cases, these parts are extracted as being, and these parts are mistakenly recognized as different objects even though they are the same planar object. Specifically, for example, there is a large difference in brightness between the sunlit part and the shaded part on the road surface, so the shaded part (part with low brightness) is a different object from the sunlit part (part with high brightness). In some cases, it was mistakenly recognized. Such misrecognition is a cause of erroneous control or mishandling, such as a collision avoidance operation, assuming that the shaded part that was misrecognized is an obstacle such as a side wall existing at the roadside in the case of ACC. It becomes.

Note that this problem is not limited to the three-dimensional object identification device used in the driver assistance system, and similarly occurs in any three-dimensional object identification device such as a three-dimensional object identification device used for robot control or the like.
In addition, it is not preferable to solve this problem by newly preparing a detection device different from the imaging means because the cost increases. Therefore, if the above-mentioned problem can be solved by using an imaging means that is a detection device generally used for detecting the reflected light intensity (luminance) from a three-dimensional object in a conventional three-dimensional object identification device, the cost can be reduced. It is beneficial from a viewpoint.

  The present invention has been made in view of the above problems, and the object of the present invention is to capture an image between a planar object and a three-dimensional object even if there is a portion with greatly different luminance in the same planar object. It is to provide a three-dimensional object identification device that can be identified with high accuracy using a means, and a mobile control device and an information providing device including the same.

To achieve the above object, the present invention 請 Motomeko 1 is present in the imaging area, in the three-dimensional object identifying device for identifying the three-dimensional object having an outer surface facing a direction different from the predetermined plane, the imaging area An imaging unit that receives two polarized light beams having different polarization directions included in reflected light from an existing object and captures the respective polarized images, and two polarized images captured by the imaging unit are respectively predetermined. Luminance calculation means that divides the image into processing regions and calculates a luminance total value between the two polarization images for each processing region; and a luminance difference value between the two polarization images with respect to the luminance total value for each processing region The differential polarization degree calculating means for calculating the differential polarization degree indicating the ratio of the ratio and the luminance of the processing area corresponding to the location of the reference plane object assumed to exist in the same plane as the predetermined plane in advance. The difference polarization degree calculated by the difference polarization degree calculation means for the reference processing area that is a large part is set as the reference difference polarization degree, and the difference polarization degree calculated by the difference polarization degree calculation means for a processing area different from the reference processing area A difference polarization degree difference value calculating means for calculating a difference polarization degree difference value, which is a difference value between a degree and the reference difference polarization degree, a luminance total value calculated by the brightness calculation means, and the difference polarization degree difference value calculating means. Three-dimensional object identification that uses the calculated differential polarization degree difference value as an identification index value and performs a three-dimensional object identification process for determining whether or not an object present at a location corresponding to each processing area in the imaging area is the three-dimensional object. And a processing means.
The invention according to claim 2 is the three-dimensional object identification device according to claim 1 , wherein the three-dimensional object identification processing means is a low-luminance process that is a portion having a low luminance in a processing area corresponding to the location of the reference plane object. A determination process for determining which of the plurality of numerical ranges defined for the low-intensity part of the reference plane object and the three-dimensional object existing at a location corresponding to the area belongs to the processing area; Performing the above three-dimensional object identification process by performing a process of identifying an object existing at a location corresponding to the object as an object corresponding to a numerical range determined to belong to the determination process. It is.
According to a third aspect of the present invention, in the three-dimensional object identification device according to the second aspect, the imaging means is mounted on a moving body that moves on a moving surface, and includes the moving surface obliquely from above the moving surface. In the determination process, a numerical range determined for the same object is set for each of at least two or more areas that divide the two polarized images in the vertical direction. It is characterized in determining which of the numerical ranges set for the area the identification index value of the processing area belonging to the area belongs.
The invention of claim 4 is the three-dimensional object identifying device according to any one of claims 1 to 3, the identification of the three-dimensional object identification processing means stores the results of the three-dimensional object identification processing performed in the past Processing result storage means, and the three-dimensional object identification processing means performs the three-dimensional object identification processing using the result of the past three-dimensional object identification processing stored in the identification processing result storage means together with the identification index value. It is characterized by doing.
Further, the invention of claim 5 is a movement control means for performing movement control of a moving object, a three-dimensional object identification means for picking up an image of the periphery of the moving object as an imaging target, and identifying a three-dimensional object existing in the imaging target; 5. The mobile control device that performs the movement control using the identification result obtained by the three-dimensional object identification unit, and the movement control unit includes any one of claims 1 to 4 as the three-dimensional object identification unit. The three-dimensional object identification device described above is used.
According to a sixth aspect of the present invention, there is provided a three-dimensional object identifying means for capturing an image of a moving body moving in accordance with a driving operation by a driver as an imaging target, and identifying a three-dimensional object existing within the imaging target, In an information providing apparatus comprising: useful information generating means for generating information useful for the driver using the identification result by the means; and information notifying means for notifying the driver of information generated by the useful information generating means. The three-dimensional object identification device according to any one of claims 1 to 4 is used as the three-dimensional object identification means.

As a result of earnest research, the present inventors corresponded to each processing region by using the differential polarization degree, which is the ratio of the luminance difference value to the luminance total value between the two polarized images captured by the imaging means, as the identification index value. We have devised a new three-dimensional object identification method that can distinguish a three-dimensional object existing at a place from a planar object with high accuracy. For example, it is a three-dimensional object identification method using a difference polarization degree (difference polarization degree difference value) obtained by taking a difference between a reference processing area and an identification target processing area. According to this method, it is possible to distinguish and recognize a three-dimensional object existing at a location corresponding to each processing region with higher accuracy than a conventional method using luminance as an identification index value. However, even in the method using the differential polarization degree, if there is a portion having a significantly different luminance in the planar object existing in the reference processing area, the processing in which the planar object having a luminance different from that of the reference processing area exists. In some cases, the region is misrecognized as a three-dimensional object.
As a result of further research, the inventors have used not only the differential polarization degree but also the luminance used as the identification index value of the conventional method, so that the luminance different from the reference processing region can be obtained as described later. It has been found that a processing area having a solid object and a processing area having a three-dimensional object can be distinguished and recognized with high accuracy.
Therefore, according to the present invention using both the differential polarization degree and the luminance, even if there is a portion with a large difference in luminance in the same planar object, the luminance has a luminance different from that of the reference processing region that is easily mistaken for a three-dimensional object. It is possible to discriminate between a planar object and a three-dimensional object in the processing area portion with high accuracy. Moreover, according to the present invention, since the luminance used as the identification index value is the total luminance value between the two polarized images captured by the imaging unit used to calculate the differential polarization degree, the new detection device is It is unnecessary.

  As described above, according to the present invention, even if there is a portion having a significantly different luminance in the same planar object, it is possible to identify the planar object and the three-dimensional object with high accuracy using the imaging unit. An excellent effect is obtained.

It is a functional block diagram of the driver assistance system concerning an embodiment. It is explanatory drawing which shows one structural example of the polarization camera which can be utilized for the driver | operator assistance system. It is explanatory drawing which shows the other structural example of the polarization camera which can be utilized for the driver | operator assistance system. It is explanatory drawing which shows the further another structural example of the polarization camera which can be used for the driver | operator assistance system. It is explanatory drawing which shows the further another structural example of the polarization camera which can be used for the driver | operator assistance system. It is explanatory drawing which shows the further another structural example of the polarization camera which can be used for the driver | operator assistance system. It is explanatory drawing which shows the further another structural example of the polarization camera which can be used for the driver | operator assistance system. (A) is a top view of the botsdots that can be identified, and (b) is a side view of the botsdots. (A) is a top view of a cat's eye that can be an identification target, and (b) is a perspective view of the cat's eye. It is a flowchart which shows the flow of the road surface structure specific process in embodiment. It is explanatory drawing which shows an example of the monochrome image (luminance image) produced | generated in the monochrome image process part from the polarization RAW image data acquired with the polarization camera. It is explanatory drawing which shows the difference polarization degree image produced | generated in the difference polarization degree image process part from the same polarization RAW image data. It is the graph which plotted the luminance value obtained along the white broken line arrow in FIG. It is the graph which plotted the differential polarization degree obtained along the white broken-line arrow in FIG. It is a graph which shows an example of a change of a differential polarization | polarized-light degree when a P polarization | polarized-light image and an S-polarized image are image | photographed with the camera which changed and fixedly arranged the light source position with respect to an asphalt surface and a metal surface. It is explanatory drawing for demonstrating the influence of a windshield. It is explanatory drawing which shows an example of the monochrome image (luminance image) which the monochrome image processing part produced | generated from the polarized RAW image data which imaged the road surface in which the botsdots are used as a section line with the polarization camera, and was acquired by this. It is explanatory drawing which shows the difference polarization degree image produced | generated in the difference polarization degree image process part from the same polarization RAW image data. A graph showing the change in the degree of differential polarization when a P-polarized image and an S-polarized image are photographed with a camera that is fixedly arranged by changing the light source position on an asphalt surface and a coated surface obtained by applying paint to steel in a laboratory. It is. It is explanatory drawing which shows an example of the monochrome image (luminance image) which the monochrome image process part 13 produced | generated from the polarized RAW image data which imaged the road surface to which coal tar adhered with the polarization camera, and was acquired by this. It is explanatory drawing which shows the difference polarization degree image produced | generated in the difference polarization degree image process part from the same polarization RAW image data. It is a flowchart which shows the flow of the solid object specific process in embodiment. It is explanatory drawing which shows an example of the monochrome image (luminance image) produced | generated in the monochrome image process part from the polarization RAW image data acquired with the polarization camera. It is explanatory drawing which shows the difference polarization degree image produced | generated in the difference polarization degree image process part from the same polarization RAW image data. 24 is a histogram of luminance value distributions at three white frames in FIG. 23 for 100 frames. FIG. 25 is a histogram obtained by taking differential polarization degree distributions at three positions in a white frame in FIG. 24 for 100 frames. It is explanatory drawing which shows the other example of the monochrome image (luminance image) produced | generated in the monochrome image process part from the polarization RAW image data acquired with the polarization camera. It is a graph which shows the difference polarization degree in each position on the process line (the straight line extended in the horizontal direction in a figure) drawn in the image shown in FIG. It is a graph which shows the brightness | luminance (monochrome brightness | luminance) in each position on the process line (the straight line extended in the horizontal direction in the figure) drawn in the image shown in FIG. It is a two-dimensional distribution diagram of various identification objects with the brightness of the sun road surface taken along the X axis and the brightness of various object objects other than the sun road surface existing in the imaging area taken as the Y axis. It is a three-dimensional distribution diagram of each identification target object, where the X-axis represents the differential polarization degree of the sun road surface, the Y-axis represents the differential polarization degree of the identification target object, and the Z-axis represents the luminance of the identification target object.

Hereinafter, an embodiment in which the present invention is applied to a driver support system as a mobile control device and an information providing device will be described.
FIG. 1 is a functional block diagram of the driver assistance system according to the present embodiment.
A landscape around the vehicle including a road surface (moving surface) on which a moving vehicle travels is photographed by a polarization camera 10 as an imaging means mounted on a vehicle (not shown), and a vertical polarization intensity (hereinafter simply referred to as “pixel polarization”) for each pixel. Polarized RAW image data including “S-polarized light intensity” and horizontal polarized light intensity (hereinafter simply referred to as “P-polarized light intensity”) is acquired. The horizontally polarized image data obtained from the P polarized light intensity data included in the polarized RAW image data is stored in the horizontally polarized image memory 11, and the vertically polarized image data obtained from the S polarized light intensity data included in the polarized RAW image data is displayed in the vertically polarized image memory 12. Respectively. These image data are respectively transmitted to the monochrome image processing unit 13 as the luminance calculation unit and the differential polarization degree image processing unit 15 as the differential polarization degree calculation unit and the differential polarization degree difference value calculation unit.

  The polarization camera 10 captures a surrounding image having pixels of, for example, a megapixel size by an image sensor such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) that is a light receiving element. It is preferable that the polarization camera 10 continuously acquires surrounding images at short time intervals close to real time. The polarization camera 10 may be attached to a room mirror, for example, to capture a landscape in front of the vehicle (a front view including a road surface), or attached to a side mirror, for example, to capture a landscape on the side of the vehicle. For example, it may be attached to a back door and may take an image of a landscape behind the vehicle. In the present embodiment, an example will be described in which a landscape (a front view including a road surface) in front of a vehicle is imaged by being attached to a rearview mirror.

FIG. 2 is an explanatory diagram illustrating a configuration example of the polarization camera 10.
As shown in FIG. 2, the polarization camera 10 </ b> A has a rotating polarizer 102 that is rotationally driven on the front surface of one camera 101 that includes an image pickup device such as a CCD. In this polarization camera 10 </ b> A, the polarization direction of light passing therethrough changes according to the rotation angle of the rotating polarizer 102. Therefore, the camera 101 can capture the P-polarized image and the S-polarized image alternately by imaging while rotating the rotating polarizer 102.

FIG. 3 is an explanatory diagram illustrating another configuration example of the polarization camera 10.
As shown in FIG. 3, the polarization camera 10B uses two cameras 111 and 112 each having an image sensor such as a CCD, and transmits an S-polarized light filter 113 that transmits S-polarized light and a P-polarized light to the respective front surfaces. A P-polarization filter 114 is disposed. In the polarization camera 10A shown in FIG. 2, since one camera 101 alternately captures P-polarized images and S-polarized images, the P-polarized image and S-polarized image could not be captured simultaneously. With the polarization camera 10B shown in FIG. 3, a P-polarized image and an S-polarized image can be taken simultaneously.

FIG. 4 is an explanatory diagram showing still another configuration example of the polarization camera 10.
As shown in FIG. 4, the polarization camera 10C is similar to the polarization camera 10B shown in FIG. 3 in that the imaging elements are individually provided for the P-polarized image and the S-polarized image. The elements are greatly different in that they are arranged closer than in the polarization camera 10B shown in FIG. According to this polarization camera 10C, it can be made smaller than the polarization camera 10B shown in FIG. The polarization camera 10C shown in FIG. 4 is formed by laminating a lens array 122, a light shielding spacer 123, a polarization filter 124, a spacer 125, and a solid-state imaging unit 126. The lens array 122 has two imaging lenses 122a and 122b. The two imaging lenses 122a and 122b are formed of a single lens made of, for example, an aspheric lens having the same shape independent from each other, and the optical axes 121a and 121b are parallel to each other and on the same plane. It is arranged. The light shielding spacer 123 has two openings 123a and 123b, and is provided on the side opposite to the subject side with respect to the lens array 122. The two openings 123a and 123b are penetrated with a predetermined size centering on the optical axes 121a and 121b, respectively, and the inner wall surface is subjected to light reflection prevention processing by blackening, roughening or matting. The polarizing filter 124 is a region-dividing type polarizer filter having two polarizer regions 124 a and 124 b whose polarization planes are different by 90 degrees, and is provided on the side opposite to the lens array 122 with respect to the light shielding spacer 123. The polarizer regions 124a and 124b transmit non-polarized light whose electromagnetic field vibrates in an unspecified direction to linearly polarized light by transmitting only the vibration component (polarized component) in the direction along the polarization plane. Note that a region-divided polarizer filter with a clear boundary can be obtained by using a wire grid method formed with a metal fine concavo-convex shape or an auto-cloning photonic crystal method. The spacer 125 is formed in a rectangular frame shape having an opening 125 a through which regions corresponding to the polarizer regions polarized light a and polarized light b of the polarizing filter 124 pass, and is provided on the side opposite to the light shielding space 123 with respect to the polarizing filter 124. It has been. The solid-state image pickup unit 126 has two solid-state image pickup devices 126 a and 126 b mounted on a substrate 127, and is provided on the side opposite to the polarization filter 124 with respect to the spacer 125. In the present embodiment, in order to perform monochrome sensing, these solid-state imaging devices 126a and 126b do not include a color filter. However, a color filter is arranged when sensing a color image.

FIG. 5 is an explanatory diagram showing still another configuration example of the polarization camera 10.
As illustrated in FIG. 5, the polarization camera 10 </ b> D includes a half mirror 131 having 1: 1 transparency, a reflection mirror 132, an S polarization filter 133, a P polarization filter 134, and an S polarization filter 133. It has an S-polarized CCD 135 that receives S-polarized light and a P-polarized CCD 136 that receives P-polarized light via a P-polarized filter 134. In the polarization cameras 10B and 10C shown in FIGS. 3 and 4, although S-polarized images and P-polarized images can be captured simultaneously, parallax occurs. On the other hand, in the polarization camera 10D shown in FIG. 5, since the S-polarized image and the P-polarized image are simultaneously captured using the same light received through the same imaging optical system (lens) (not shown), the parallax is reduced. Does not occur. Therefore, processing such as parallax deviation correction is not necessary.
Instead of the half mirror 131, a polarizing beam splitter such as a prism that reflects P-polarized light and transmits S-polarized light may be used. By using such a polarization beam splitter, the S-polarization filter 133 and the P-polarization filter 134 can be omitted, the optical system can be simplified, and the light utilization efficiency can be improved.

FIG. 6 is an explanatory diagram showing still another configuration example of the polarization camera 10.
This polarization camera 10E is the same as the polarization camera 10C shown in FIG. 4 in that it is a unit in which camera components are stacked along the optical axis 141 of the imaging lens 142a as shown in FIG. The difference is that an S-polarized image and a P-polarized image are captured by a single imaging lens 142 (a plurality of imaging lenses may be stacked on the optical axis) 142. According to this polarization camera 10E, no parallax occurs between the S-polarized image and the P-polarized image, as in the polarization camera 10D shown in FIG. Moreover, it can be made smaller than the polarizing camera 10D shown in FIG. The polarization filter 144 of the polarization camera 10E shown in FIG. 6 is a region-dividing polarizer filter in which two types of polarizer regions 144a and 144b having different polarization planes of 90 degrees are provided. Accordingly, four solid-state image sensors 146a, 146b, 146c, 146d are provided.

FIG. 7 is an explanatory diagram showing still another configuration example of the polarization camera 10.
As shown in FIG. 7, the polarization camera 10F employs a region division type filter. In FIG. 7, squares arranged vertically and horizontally indicate the light receiving portions 151 of the respective light receiving elements, a region indicated by a vertical line indicates a region of the S polarization filter 152, and a region indicated by a horizontal line indicates the region of the P polarization filter 153. This polarization camera 10F is not made to correspond to the pixels of the light receiving element 1: 1, but the area of each filter 152, 153 has a width corresponding to one light receiving element in the horizontal direction, and the inclination of the boundary line of the area is 2. That is, it takes the shape of an oblique band having an angle that changes by two pixels in the vertical direction while proceeding by one pixel in the horizontal direction. By combining such a special filter arrangement pattern and signal processing, it is possible to reproduce each filter transmission image on the entire screen even if the alignment accuracy when joining the image sensor array and the area division filter is not sufficient. It is possible to realize a low-cost polarization camera that can capture an S-polarized image and a P-polarized image.

The monochrome image processing unit 13 calculates the monochrome luminance (P polarization intensity + S polarization intensity of the pixel) for each pixel from the P polarization intensity data and the S polarization intensity data in the horizontal polarization image memory 11 and the vertical polarization image memory 12. . A monochrome image can be generated using the monochrome luminance data. The monochrome luminance data calculated by the monochrome image processing unit 13 is output to a white line identification unit 14 as a line detection unit and a three-dimensional object identification unit 18 as a three-dimensional object identification processing unit.
The differential polarization degree image processing unit 15 calculates the differential polarization degree (identification index value) for each pixel from the P polarization intensity data and the S polarization intensity data in the horizontal polarization image memory 11 and the vertical polarization image memory 12. A differential polarization degree image can be generated using the differential polarization degree. The differential polarization degree is obtained from the calculation formula shown in the following formula (1). That is, the differential polarization degree is the ratio of the difference value (luminance difference value) between the P polarization intensity and the S polarization intensity to the total value (luminance total value) of the P polarization intensity and the S polarization intensity. The differential polarization degree can be paraphrased as a difference value between the ratio of the P deflection intensity to the total luminance value (P polarization ratio) and the ratio of the S deflection intensity to the total luminance value (S polarization ratio). In this embodiment, the case where the S polarization intensity is subtracted from the P polarization intensity will be described. However, the P polarization intensity may be subtracted from the S polarization intensity. The differential polarization degree data calculated by the differential polarization degree image processing unit 15 is output to the road surface structure identification unit 16 and the three-dimensional object identification unit 18.
Differential polarization degree = (P polarization intensity−S polarization intensity) / (P polarization intensity + S polarization intensity) (1)

The white line identification unit 14 identifies a white line on the traveling road surface by the following method based on the monochrome luminance data calculated by the monochrome image processing unit 13. The white line referred to here may include any line that divides the road, such as a line of an arbitrary color such as a yellow line, a solid line, a broken line, a dotted line, and a double line.
A normal road lane (division line) is formed in a high-contrast color (white or the like) with respect to a black portion such as asphalt so that the driver can easily see. Therefore, the luminance of such a lane (here, a white line) is sufficiently higher than that of an object such as asphalt existing elsewhere. Therefore, it is possible to determine a portion where the monochrome luminance data is equal to or greater than the predetermined threshold as a white line. Note that the monochrome luminance data used in the present embodiment uses the total value of the P-polarized light intensity and the S-polarized light intensity obtained by the polarization camera 10 described above.

  As a flow of white line identification processing in the present embodiment, first, the monochrome image processing unit 13 uses the total value of the P-polarized light intensity and S-polarized light intensity of each pixel obtained by the polarization camera 10 as the monochrome luminance of each pixel. calculate. The white line identification unit 14 sets a plurality of processing lines for the monochrome image obtained by the monochrome luminance. The processing line of this embodiment is set for each pixel column arranged in one horizontal row in the differential polarization degree image. The direction of the processing line is not necessarily the horizontal direction, and may be the vertical direction or the oblique direction. In addition, the number of pixels in each processing line may be the same or different. Further, the processing line is not necessarily set for all the pixels in the differential polarization degree image, and may be set for a part of appropriately selected pixels in the differential polarization degree image. Further, as will be described later, the processing may be performed in units of processing blocks (blocks each consisting of two or more pixels in the vertical and horizontal directions) instead of the processing line. The white line identification unit 14 calculates a monochrome luminance difference between two adjacent pixels for each processing line, and determines whether the calculation result is equal to or greater than a predetermined white line edge threshold. If it is determined by this determination that the threshold is equal to or greater than the white line edge threshold value, the two adjacent pixels related to the determination are stored as a white line edge. By performing this for all the processing lines, it is possible to extract the white line edge in the monochrome image.

The result of identifying the white line edge by the white line identifying unit 14 can be used for various processes.
For example, a monochrome image (front view image) generated by using the luminance data calculated by the monochrome image processing unit is displayed on a display unit (display) which is an in-vehicle information notification unit composed of a CRT, a liquid crystal or the like, In order to notify the information of the white line portion in the image as information useful for the driver, there is a process of displaying the information in a display form easy for the driver to see. According to this, for example, even in a situation where it is difficult for the driver to visually recognize the white line, the driver can see the front view image of the display unit, and thereby the relative positional relationship between the vehicle and the white line. It is easy to keep track of the driving lanes demarcated by the white line.
In addition, for example, from the position information of the white line recognized by the white line identification unit 14, a process of grasping the relative positional relationship between the own vehicle and the white line is performed, and the own vehicle is determined from the appropriate travel position on the travel lane partitioned by the white line. A process for determining whether or not the vehicle is running off and generating an alarm sound or the like when the vehicle is running out of the proper running position can be given. Alternatively, there is a process in which the automatic braking function is executed to reduce the traveling speed of the host vehicle when traveling out of the proper traveling position.

  Based on the differential polarization degree calculated by the differential polarization degree image processing unit 15, the road surface structure identification unit 16 refers to a structure (hereinafter referred to as “road surface structure”) present on the traveling road surface by a method described later. To identify. The road surface structure identifying unit 16 outputs the recognition result to the road surface structure identifying unit 17 and the white line identifying unit 14. Road surface structures include manhole covers, metal objects such as road connections on roads such as highways and overpasses, metal objects such as bots dots and cats eyes that form section lines that divide traveling lanes, reflectors, etc. A composite composed of: In addition, here, it is assumed to include a foreign substance that covers a part of the road surface that is not intentionally made, such as coal tar. The road surface structure identification unit 16 identifies a planar object whose outer surface is located substantially in the same plane as the road surface as a road surface structure. Whether the road surface structure is a manhole cover, a road connection unit, The road surface structure specifying unit 17 specifies whether it is a cat's eye.

  The road surface structure identification unit 16 removes the white line from the differential polarization degree image based on the white line edge identification result by the white line identification unit 14 described above, and the road surface structure object with respect to the differential polarization degree image from which the white line has been removed. An identification process may be performed. In this case, noise including a white line can be appropriately removed, and the road surface structure identification accuracy can be improved.

The manhole cover here refers to a metal plate fitted in the opening of the manhole, and is generally made of cast iron that is strong and heavy.
Botsdots B is made of ceramic, for example, which is mainly used as a section line in North America. As shown in FIG. 8 (b), the botsdots B is obtained by embedding a circular dome-shaped object having a diameter of about 100 mm in the road surface. As shown in FIG. By arranging, it is used as a section line.
The cat's eye C is used as a section line. As shown in FIG. 9B, a reflector D having a characteristic of reflecting incident light in the same direction is attached inside a substantially rectangular body. It is what was done. As shown in FIG. 8A, the cat's eyes C are used as section lines by being arranged in large numbers along the traveling lane on the road.
Both the botsdots and the cat's eyes are arranged in a state of slightly protruding from the road surface.

  The three-dimensional object identifying unit 18 is calculated by the differential polarization degree difference value, which is a difference value between the differential polarization degree calculated by the differential polarization degree image processing unit 15 and a reference differential polarization degree described later, and the monochrome image processing unit 13. Based on the monochrome luminance, a solid object existing in the imaging region of the polarizing camera 10 is identified by a method described later. As such a three-dimensional object, other vehicles traveling on the road surface, side walls existing near the road edge of the road surface, guardrails, telephone poles, street lights, signs, roadside obstacles such as stepped portions, Any solid object having an outer surface facing in a direction different from the traveling road surface is included, such as a collision avoidance object such as a person, an animal, or a bicycle on the traveling road surface or on the shoulder. The three-dimensional object identification unit 18 outputs the recognition result to the three-dimensional object identification unit 19. The three-dimensional object identification unit 18 recognizes such a three-dimensional object by distinguishing it from a road object and a plane object having an outer surface substantially in the same plane as the road surface. The three-dimensional object specifying unit 19 specifies whether the object is an external obstacle or a collision avoidance object.

  Data of various shape templates as shape information used in the road surface structure specifying unit 17 and the three-dimensional object specifying unit 19 are stored in the shape storage unit 20 as the shape information storage unit. The shape template stored in the shape storage unit 20 is a shape (specification in a picked-up image) when an object (specific object) specified by the road surface structure specifying unit 17 and the three-dimensional object specifying unit 19 is imaged by the polarization camera 10. The shape of the object). For example, the shape template of the manhole cover (circular) shows an elliptical shape because the polarizing camera 10 photographs the manhole cover on the road surface from obliquely above. The shape template may include size information. In this embodiment, as a shape template that can be used in the road surface structure specifying unit 17, for example, a shape template for specifying a manhole cover, a shape template for specifying a section line made up of botsdots or cat's eyes And a shape template for identifying a road connecting portion existing on a road such as an expressway or a crossover. In addition, as a shape template that can be used by the three-dimensional object specifying unit 19, for example, a shape template for specifying another vehicle, a shape template for specifying a telephone pole or a streetlight, and a stepped portion on a road edge are specified. The shape template for doing is mentioned. Of course, a template for identifying an object other than those exemplified here may be prepared as long as the object can be identified from its shape.

  The road surface structure specifying unit 17 compares the shape of the image area identified as the road surface structure with the shape template stored in the shape storage unit 20 based on the identification result of the road surface structure identification unit 16, and the road surface Whether the structure is a manhole cover, a road connecting portion, a section line made up of botts or cats eyes, or a road surface structure other than these is specified by a method described later.

  The three-dimensional object specifying unit 19 compares the shape of the image area identified as the three-dimensional object with the shape template stored in the shape storage unit 20 based on the identification result of the three-dimensional object identification unit 18, and the three-dimensional object is Whether it is another vehicle, an off-road obstacle, a collision avoidance object, or a three-dimensional object other than these is specified by a method described later.

Next, the flow of the road surface structure specifying process for specifying the road surface structure in the driver assistance system according to the present embodiment will be described.
FIG. 10 is a flowchart showing the flow of the road surface structure specifying process.
When polarized RAW image data is acquired by the polarization camera 10, the horizontally polarized image data obtained from the P polarization intensity data included in the polarized RAW image data is stored in the horizontally polarized image memory 11 and included in the polarized RAW image data. The vertically polarized image data obtained from the S polarization intensity data is stored in the vertically polarized image memory 12 (S1). Thereafter, the differential polarization degree image processing unit 15 calculates, from the P-polarization intensity data and the S-polarization intensity data in the horizontal polarization image memory 11 and the vertical polarization image memory 12, for each pixel, from the calculation formula shown in the above formula (1). A differential polarization degree (identification index value) is calculated (S2). Data of the differential polarization degree image obtained from the calculation result is stored in an image memory (not shown) in the differential polarization degree image processing unit 15.

Next, the edge determination process will be described.
The differential polarization degree image processing unit 15 that has obtained the differential polarization degree image sets a plurality of processing lines for the differential polarization degree image. The processing line of this embodiment is set for each pixel column arranged in one horizontal row in the differential polarization degree image. The direction of the processing line is not necessarily the horizontal direction, and may be the vertical direction or the oblique direction. In addition, the number of pixels in each processing line may be the same or different. Further, the processing line is not necessarily set for all the pixels in the differential polarization degree image, and may be set for a part of appropriately selected pixels in the differential polarization degree image.

  Further, the processing may be performed in units of processing blocks (blocks each composed of two or more pixels in the vertical and horizontal directions) instead of the processing line. In this case, for example, in an edge determination process described later, a plurality of processing blocks are set for the differential polarization degree image, and a standard deviation indicating a variation degree (scattering degree) of the differential polarization degree is calculated for each processing block. When the calculated standard deviation is equal to or greater than the reference deviation threshold, it can be determined that an edge exists in the processing block. Note that the processing block may be set in a rectangular area, or may be set in an area having another shape. The size of the processing block may be about 10 × 10 pixels, for example. Each processing block may be the same size or a different size. Further, instead of the standard deviation, a statistic such as variance or average deviation may be used.

  The differential polarization degree image processing unit 15 calculates the difference in differential polarization degree between two adjacent pixels for each processing line, and determines whether or not the calculation result is equal to or greater than a predetermined edge threshold (S3). . If it is determined in this determination that the threshold value is equal to or greater than the edge threshold value, the two adjacent pixels related to the determination are stored as an edge (S4). By performing this for all the processing lines (S5), it is possible to specify the boundary lines of different objects in the differential polarization degree image.

  Here, in the conventional edge determination processing, for each processing line, a difference in monochrome luminance between adjacent pixels is calculated, and it is often determined whether or not the calculation result is equal to or greater than a predetermined edge threshold value. However, in this conventional edge discrimination process, for example, the edge can be discriminated between dissimilar objects with similar intensities of reflected light (monochrome luminance) received by the camera, such as manhole covers and asphalt. There wasn't. On the other hand, in the edge discrimination processing in the present embodiment, the edge discrimination is performed using the differential polarization degree instead of the monochrome luminance, so that it is possible to accurately discriminate edges between different kinds of objects having the same reflected light intensity. it can. Hereinafter, this point will be described.

FIG. 11 is an explanatory diagram showing an example of a monochrome image (luminance image) generated by the monochrome image processing unit 13 from the polarized RAW image data acquired by the polarization camera 10.
FIG. 12 is an explanatory diagram showing a differential polarization degree image generated by the differential polarization degree image processing unit 15 from the same polarized RAW image data.
FIG. 13 is a graph in which the luminance values obtained along the white dashed arrows in FIG. 11 are plotted.
FIG. 14 is a graph in which the degree of differential polarization obtained along the white dashed arrow in FIG. 12 is plotted.

As can be seen from the graph shown in FIG. 13, as for the luminance value, both the asphalt area and the manhole cover area show almost the same value (range of −0.4 or more and −0.5 or less). No change in value. On the other hand, as can be seen from the graph shown in FIG. 14, the degree of differential polarization is about zero in the asphalt area, and is about −0.3 in the manhole cover area. A large change is seen in the differential polarization degree. Therefore, by setting an appropriate threshold value (for example, -0.2) between 0 and -0.3, the asphalt area and the manhole cover, which are difficult to discriminate with monochrome luminance due to the difference in the degree of differential polarization, are set. The edge with the region can be discriminated with high accuracy.
The reason for using the differential polarization degree difference value instead of the differential polarization degree in the object identification in the present embodiment will be described later.

Next, the road surface structure identification process performed by the road surface structure identification unit 16 will be described.
In describing the road surface structure identification process, first, the reason why the road surface structure can be identified from the differential polarization degree will be described.
Reflected light reflected from an object includes a specular reflection component that is a so-called “light”, a diffuse reflection component that is a matte reflection component that is a fine uneven structure on the surface of the object, and internal scattering that is scattered inside the object. Contains ingredients. The intensity of the reflected light is expressed as the sum of these three components. The specular reflection component can also be considered as a part of the diffuse reflection component. Although the diffuse reflection component and the internal scattering component are observed regardless of the direction of the light source that irradiates the object (that is, the dependency on the incident angle is low), the specular reflection component is the light receiving unit for the reflected light. On the other hand, it is a component having a strong incident angle dependency that is observed only when a light source is present in a substantially regular reflection direction. This is also true for polarization characteristics. As described above, the diffuse reflection component and the internal scattering component are observed regardless of the direction of the light source that irradiates the object, but their polarization characteristics are different from each other. Specifically, the diffuse reflection component can be assumed to divide the object surface into minute regions and satisfy the Fresnel reflection characteristics in each region. Therefore, when non-polarized light is incident, the S-polarized component is P There is a polarization property that is larger than the polarization component. On the other hand, the internal scattering component is a component that is scattered inside the object, so when non-polarized light is incident, it is less affected by the polarization component of the light incident on the object, There is a polarization characteristic that the P-polarized light component becomes stronger when it comes out.

  And, as in this embodiment, most of the objects (such as asphalt and manhole cover) that can exist in the shooting area when shooting the front view from the host vehicle are not a little uneven on the surface. Therefore, it can be considered that the specular reflection component is small. As a result, in the present embodiment, it can be considered that the diffuse reflection component and the internal scattering component are dominant in the reflected light from the object existing in the imaging region of the polarization camera 10. As a result, by comparing the intensity of the S-polarized component and the P-polarized component in the reflected light, it can be understood that if the S-polarized component is strong, the reflected light contains a large amount of diffusely reflected component. If is strong, it can be understood that the reflected light contains many internal scattering components.

FIG. 15 shows a change in the degree of differential polarization when a P-polarized image and an S-polarized image are taken with a camera that is fixedly arranged by changing the light source position with respect to an asphalt surface and a metal surface (smooth surface) in the laboratory. It is a graph which shows an example.
In this graph, the horizontal axis represents the incident angle (light source position), and the vertical axis represents the differential polarization degree. The camera elevation angle is tilted 10 degrees from the horizontal. This differential polarization degree is calculated from luminance information at a substantially central portion of the captured image at each incident angle. The difference polarization degree in this bluff is a ratio of a value obtained by subtracting the S polarization component from the P polarization component with respect to the total value of the P polarization component (Rp) and the S polarization component (Rs). Therefore, when the P polarization component is stronger than the S polarization component, the differential polarization degree takes a positive value, and when the S polarization component is stronger than the P polarization component, the differential polarization degree is negative. Will take the value.

  As can be seen from the graph shown in FIG. 15, regarding the asphalt surface, the differential polarization degree takes a negative value over almost the entire incident angle. That is, the S polarization component is stronger than the P polarization component. This is because the diffuse reflection component is dominant in the reflected light from the asphalt surface. On the other hand, regarding the metal surface, the differential polarization degree has a positive value over the entire region where the incident angle exceeds 30 °. That is, the P-polarized component is stronger than the S-polarized component. This is because the internal scattering component is dominant in the reflected light from the metal surface.

As described above, by taking the difference between the S-polarized component and the P-polarized component included in the reflected light, it is possible to grasp whether the diffuse reflection component or the internal scattering component is strong in the reflection characteristics of the object. it can. Therefore, an object having different reflection characteristics, such as an asphalt having a strong diffuse reflection component and a road surface structure having a strong internal scattering component, can be identified based on the difference between the S polarization component and the P polarization component included in the reflected light. it can.
In general, since the refractive index is different for different materials, this effect also appears in the difference between the S-polarized component and the P-polarized component. Therefore, objects with different materials can be identified based on the difference between the S-polarized component and the P-polarized component included in the reflected light.

  Note that the results of the graph shown in FIG. 15 show the difference in the surface state between the asphalt surface and the metal surface (asphalt is a surface on which many irregularities are distributed, whereas the metal surface is a smooth surface). It is thought that they are also affected. Therefore, the values of the S-polarized component and the P-polarized component that are detected vary depending on the difference in the surface state of the identification target. This is because the differential polarization degree of the manhole cover, which is an asphalt region and a road structure in the graph of FIG. 14 showing the experimental results in an actually used environment, is the result of the laboratory experiment shown in FIG. It can be understood from the difference. However, between objects of different materials, even if the surface states are the same, the reflection characteristics of the objects (diffuse reflection component and internal scattering component strength, refractive index, etc.) differ due to the difference in the material. It is possible to grasp the difference based on the difference between the S-polarized component and the P-polarized component included in the reflected light. Therefore, it is possible to distinguish between objects of different materials by using a threshold value set in consideration of the actual environment.

  In the road surface structure identification process of the present embodiment, by using the differential polarization degree of each area divided by the edge determined by the edge determination process described above as an identification index value, it is determined whether or not the area is a road surface structure. Determine whether. As described above, the road surface structure identification process can determine whether or not the region is a road surface structure even if the difference value between the S-polarized light intensity and the P-polarized light intensity is used as the identification index value. However, when the brightness is insufficient, the difference value between the S-polarized light intensity and the P-polarized light intensity calculated as the identification index value takes a small value, so that it is difficult to determine whether or not the road surface structure is present. On the other hand, as in this embodiment, if the difference polarization degree that is a value obtained by dividing the difference value by the total value of S polarization intensity and P polarization intensity (monochrome luminance) is used, even if the brightness is insufficient, The degree of differential polarization as the identification index value can take a relatively large value, and it can be determined whether or not the road surface structure.

  The flow of the road surface structure identification process according to this embodiment will be described. First, for each area divided by the edge determined by the edge determination process described above, the difference between the difference polarization degree of the area and the reference difference polarization degree ( A difference polarization degree difference value) is calculated (S6). And it is judged whether this difference value is below the threshold value for road surface structures determined beforehand (S7). In this determination, when it is determined that the difference value is equal to or less than the road surface structure threshold, the area is identified and stored as a road surface structure (S8). By performing this for all areas (S9), it is possible to grasp the area where the road surface structure is displayed in the captured image.

  In the present embodiment, instead of using the differential polarization degree (absolute amount) of the area as the identification index value used for object identification of the area, the differential polarization obtained by subtracting the differential polarization degree of the area from the reference differential polarization degree. The degree difference value (relative amount) is used. This is because, even if a difference occurs in the calculated value of the differential polarization degree of the identification target area due to the difference in the shooting environment, etc., by using the relative amount with the reference differential polarization degree where the deviation occurs due to the same effect, This is because the influence can be reduced. In the present embodiment, as the reference differential polarization degree, the differential polarization degree in the asphalt region (reference processing region) occupying most of the road surface is used. Further, in the present embodiment, it has been experimentally found that the differential polarization degree for the road surface structure is at least 0.2 or more larger than the differential polarization degree for asphalt under an actual environment. Yes. Therefore, the differential polarization degree difference value for the road surface structure takes a minus value of −0.2 or less at the minimum. Therefore, in this embodiment, -0.2 is adopted as the road structure threshold value, and an area having a differential polarization degree difference value belonging to a range equal to or smaller than the threshold value is identified as a road surface structure.

Here, there is a difference in the intensity of the reflected light from the object between the upper part and the lower part of the captured image. This is because the upper part of the captured image is a part obtained by photographing an object located far away, and the intensity of reflected light is smaller than the lower part obtained by photographing an object located nearby. Therefore, in consideration of this difference, the road surface structure threshold value to be used may be different between the upper part and the lower part of the captured image.
In addition, since the lower part of the captured image obtained by photographing an object located nearby has higher object identification accuracy than the upper part, the processing order of the processing lines is preferably the order from the lower side to the upper side of the image.

Next, the road surface structure type specifying process performed by the road surface structure specifying unit 17 will be described.
First, the road surface structure specifying unit 17 recognizes the shape of the area identified as the road surface structure by the road surface structure identification process described above (S10), and whether there is a shape template that approximates the shape of the area. It is determined whether or not (S11). In this determination, if it is determined that there is an approximate shape template, the road surface structure in the region is specified as a type associated with the shape template and stored (S12). For example, when approximating an oval shape template, it is specified as a manhole cover, and when approximating a bar shape template traversing in the horizontal direction of the image, it is specified as a road connecting portion, and a plurality of road surfaces When an article is posted on a shape template that is linearly arranged along the traveling direction of the vehicle, it is specified as a section line made up of botsdots or cat's eyes. If this is performed for all road surface structures (S13), the process is terminated.

Specifically, using the edge information of the road surface structure region and the shape template information stored in the shape storage unit 20, an approximate curve is obtained, and the least square method, the Hough transform, the model equation, etc. Shape approximation recognition is performed using a technique. Note that when obtaining an approximate curve, it is desirable that the edge information located in the lower portion of the highly reliable captured image is given a higher weight to the voting value for shape approximation. In this way, even if there is edge information that is misrecognized in the upper part of the low-reliability captured image, if there is edge information that is normally recognized in the lower part of the highly reliable captured image, The kind of road surface structure can be specified appropriately.
For example, when a manhole cover is specified, feature points related to an object larger than the manhole cover may be removed in advance by morphological calculation or the like. Thereby, the specific precision of a manhole cover can be improved.

In order to increase the specific accuracy of the type of road surface structure, the following processing may be added.
In the present embodiment, the road surface structure specifying process (S1 to S13) as described above is performed on polarized image data obtained by continuously photographing with a polarization camera 10 at a predetermined time interval. With respect to the area where the type of the road surface structure is specified by the above-described road surface structure type specifying process (S10 to S13), the processing result is stored in a predetermined memory. Using the past processing result (for example, the processing result of the polarization image data captured immediately before) stored in this memory, the type of the road surface structure specified by this processing corresponds to the past. If the processing result is the same as this, it is determined that the current processing result is highly reliable. And this reliability is utilized for specification of the kind of final road surface structure. The past processing result corresponding to the region related to the current processing result is related to the corresponding past processing result from the position of the region related to the current processing result and the traveling direction of the own vehicle, for example, using edge information. The position of the area is searched and the corresponding past processing result is specified.
In addition, although the road surface structure type specifying process (S10 to S13) has been described here, in order to improve the identification accuracy of the road surface structure identifying process (S6 to S9) as well, the past processing result is used. May be used.

  In addition, the edge threshold value used in the determination of the edge determination process (S3) and the road surface structure threshold value used in the determination of the road surface structure identification process (S7) are appropriately switched according to the difference in the photographing environment. Also good. A specific example is switching between time zones such as daytime and nighttime according to the difference in weather such as rainy weather and fine weather. This switching can be realized using time information or information such as a rain sensor or a sunshine sensor.

  In addition, when the polarization camera 10 according to the present embodiment is mounted in a vehicle such as a room mirror, an image captured by the polarization camera 10 is affected by the windshield. Therefore, it is desirable to consider the polarization characteristics of the windshield. As shown in FIG. 16, the windshield is usually disposed with a predetermined angle with respect to the optical axis of the polarizing camera 10. Generally, when a glass flat plate is disposed obliquely with respect to the optical path, the polarization state of the transmitted light changes. The calculation of so-called Fresnel transmission reflection is established, and the amount of S-polarized light component reflected by the glass surface is larger than that of P-polarized light component, and the attenuation factor of the transmitted light is higher for S-polarized light component than for P-polarized light component. large. Specifically, for example, in the example shown in FIG. 16, the P-polarized component is attenuated by about 8% by Fresnel reflection on the glass surface, whereas the S-polarized component is attenuated by about half. In this example, the refractive index at the windshield was 1.5.

  Thus, since the influence of the windshield is included in the polarization information of the light taken into the polarization camera 10, it is desirable to take this into consideration. For example, at the time of calculating the differential polarization degree (S2), an operation for canceling the attenuation component on the windshield is performed for each of the P polarization intensity and the S polarization intensity. In the above example, the S polarization component is approximately doubled and the P polarization component is approximately 1 / 0.9. As another method, for example, an optical element that cancels attenuation components of P-polarized light intensity and S-polarized light intensity is disposed between the windshield and the polarizing camera 10. As the optical element in this case, in the above example, an optical element that transmits the S-polarized component through the dead band and increases the P-polarized component by 0.5 / 0.9 times can be used.

  In the future, it is expected that plastic windshields will be used to reduce weight and cost. It is known that plastic has birefringence due to internal strain compared to glass. In this case, it is necessary to consider the influence of birefringence. Specifically, for example, with respect to the polarization component of the light captured by the polarization camera 10, the differential polarization degree is calculated in consideration of the P-polarization component and the S-polarization component due to birefringence.

The result of specifying the type of the road surface structure by the road surface structure specifying unit 17 can be used for various processes.
For example, the processing result of the road surface structure specifying unit 17 may be used for white line edge identification processing in the white line identification unit 14. Specifically, since the area where the type of the road surface structure is specified by the processing of the road surface structure specifying unit 17 is not a white line area, this area is excluded from the target of the white line edge identification process by the white line identification unit 14. . Thereby, the possibility of erroneously recognizing a road surface structure such as a manhole cover as a white line can be reduced, and the recognition accuracy of the white line can be improved. Here, the white line edge identifying process by the white line identifying unit 14 has been described. However, when performing the process of identifying an object other than a white line from the captured image, the road surface structure specifying unit 17 generally performs the road surface structure identification process. By excluding the region in which the type of the object is specified from the identification processing target, the identification accuracy of the identification processing can be improved. As a specific example, in a system that identifies obstacles such as a preceding vehicle by sensor fusion based on radar ranging results and camera images, various road surface structures such as manhole covers are misrecognized as obstacles. You can avoid doing that. As a result, it is possible to prevent the occurrence of a state in which the speed is rapidly decreased due to erroneous recognition as various road surface structure obstacles.

  Further, for example, it may be used for a car navigation system. As a specific example, the vehicle position information such as the distance and angle between the vehicle and the manhole cover is generated from the position of the manhole cover specified from the processing result of the road surface structure specifying unit 17, and the vehicle position Using the information, a further detailed position of the own vehicle is specified within the range of the own vehicle position calculated by the car navigation system. Thereby, the specific precision of the own vehicle position in a car navigation system can be improved.

Further, for example, the position and direction of the manhole cover and the road connecting portion relative to the own vehicle can be grasped from the processing result of the road surface structure specifying unit 17, and this is used for a travel support ECU (Electronic Control Unit) or the like. It can also be mentioned.
In particular, for example, the road line structure specifying unit 17 can use the section line specifying result made up of botsdots or cat's eyes for the same processing as the white line edge identifying result by the white line identifying unit 14. Specifically, for example, a monochrome image (front view image) generated using luminance data calculated by the monochrome image processing unit on a display unit (display) that is an in-vehicle information notification unit configured by a CRT, liquid crystal, or the like. ) And the information on the section line in the image is reported as information useful for the driver, and the display is easy to see for the driver. According to this, for example, even in a situation where it is difficult for the driver to visually recognize the section line, the driver can view the relative position between the vehicle and the section line by looking at the front view image on the display unit. It is possible to grasp the positional relationship, and it is easy to maintain the traveling lane divided by the section line and to travel.
In addition, for example, a process for grasping the relative positional relationship between the own vehicle and the section line from the position information of the section line recognized by the road surface structure specifying unit 17 is performed on the travel lane where the own vehicle is partitioned by the section line. There is a process of determining whether or not the vehicle is traveling away from the proper travel position and generating an alarm sound or the like when traveling outside the proper travel position. Alternatively, there is a process in which the automatic braking function is executed to reduce the traveling speed of the host vehicle when traveling out of the proper traveling position.

FIG. 17 is a diagram illustrating an example of a monochrome image (luminance image) generated by the monochrome image processing unit 13 from the polarized RAW image data obtained by capturing an image of a road surface using Botsdots as a section line with the polarization camera 10. FIG.
FIG. 18 is an explanatory diagram showing a differential polarization degree image generated by the differential polarization degree image processing unit 15 from the same polarized RAW image data.
As can be seen by comparing FIG. 17 and FIG. 18, the difference polarization degree image shown in FIG. 18 has higher contrast between asphalt and bottsdots than the monochrome image (luminance image) shown in FIG. Therefore, by using the differential polarization degree image, it is possible to discriminate the edge between the asphalt region and the botsdot region, which are difficult to discriminate with monochrome luminance, with high accuracy.

In this embodiment, the metal is described as an example of the identifiable material. However, other materials can be identified.
FIG. 19 shows the difference in the degree of polarization when a P-polarized image and an S-polarized image are taken with a camera that is fixedly arranged by changing the light source position on the asphalt surface and the coated surface of steel coated with paint. It is a graph which shows a change. This graph has the same conditions as the graph shown in FIG. 15 in which the asphalt surface and the metal surface are compared. As can be seen from this graph, there is a difference in the degree of differential polarization between the asphalt surface and the painted surface. The differential polarization degree of the painted surface is also different from the differential polarization degree of the metal surface shown in FIG. Therefore, the painted surface and the metal surface can also be distinguished and recognized by the difference in the degree of differential polarization (difference in polarization characteristics).

Similarly, not only such a painted surface but also a road surface structure such as coal tar attached on the road surface during construction or due to deterioration of the road surface, using a differential polarization degree image, compared to a monochrome luminance image. It is possible to detect with high contrast.
FIG. 20 is an explanatory diagram illustrating an example of a monochrome image (luminance image) generated by the monochrome image processing unit 13 from the polarized RAW image data obtained by capturing an image of the road surface to which coal tar is attached with the polarization camera 10.
FIG. 21 is an explanatory diagram showing a differential polarization degree image generated by the differential polarization degree image processing unit 15 from the same polarized RAW image data.
As can be seen by comparing FIG. 20 and FIG. 21, the contrast between asphalt and coal tar is higher in the differential polarization degree image shown in FIG. 21 than in the monochrome image (luminance image) shown in FIG. Therefore, if the differential polarization degree image is used, the edge between the asphalt area and the coal tar area, which is difficult to determine with monochrome luminance, can be determined with high accuracy.

Next, a flow of a three-dimensional object specifying process for specifying a three-dimensional object in the driver assistance system according to the present embodiment will be described.
FIG. 22 is a flowchart showing the flow of the three-dimensional object specifying process.
Note that the processing up to the edge determination processing is the same as the above-described road surface structure specifying processing, and thus description thereof is omitted. However, the edge threshold value used in the edge determination process uses a value different from the road surface structure specifying process described above. A method for setting the edge threshold used in the edge discrimination process will be described below.

FIG. 23 is an explanatory diagram showing an example of a monochrome image (luminance image) generated by the monochrome image processing unit 13 from the polarized RAW image data acquired by the polarization camera 10.
FIG. 24 is an explanatory diagram showing a differential polarization degree image generated by the differential polarization degree image processing unit 15 from the same polarized RAW image data.
FIG. 25 is a histogram of the luminance value distribution of three white frames in FIG. 23 for 100 frames.
FIG. 26 is a histogram of differential polarization degree distributions at three positions in the white frame in FIG. 24 for 100 frames.
As can be seen from the histogram shown in FIG. 25, with respect to the luminance value, the luminance distribution in the asphalt region, the luminance distribution in the vehicle side surface region of the other vehicle, and the luminance distribution in the vehicle rear surface region of the other vehicle overlap each other. On the other hand, as can be seen from the histogram shown in FIG. 26, for the differential polarization degree, the luminance distribution of the asphalt region, the luminance distribution of the vehicle side surface region of the other vehicle, and the luminance distribution of the vehicle rear surface region of the other vehicle, They can be separated without overlapping each other. Therefore, by setting appropriate thresholds that can distinguish these areas, it is difficult to distinguish with monochrome luminance due to differences in the degree of differential polarization, asphalt areas, vehicle side areas of other vehicles, vehicle rear areas of other vehicles Can be discriminated with high accuracy.

Next, the three-dimensional object identification process performed by the three-dimensional object identification unit 18 will be described.
In describing the three-dimensional object identification process, first, the reason why the three-dimensional object can be identified from the differential polarization degree difference value will be described.
The reflected light reflected by the object differs in the incident angle of light taken into the polarization camera 10 between the road surface and the side surface of the three-dimensional object (the outer surface facing the direction different from the road surface). Differences in strength occur. In particular, when the side surface of the three-dimensional object is a surface substantially upright with respect to the road surface, the relative relationship between the P-polarized component and the S-polarized component contained in the reflected light from the side surface of the three-dimensional object is the reflected light from the road surface. Is equivalent to the one in which the relative relationship between the P-polarized component and the S-polarized component included in is replaced. In general, the relative relationship between the P-polarized light component and the S-polarized light component contained in the reflected light is a polarized light component perpendicular to the incident surface rather than the P-polarized light component that is a polarized light component parallel to the incident surface. There is a relationship that the S polarization component is larger. Therefore, when the reflected light from the road surface or a plane parallel to the road surface is received by the polarization camera 10, the S-polarized light component is stronger than the P-polarized light intensity, and the reflected light from the side surface of the three-dimensional object substantially upright with respect to the road surface. Is received by the polarizing camera 10, the P-polarized light component is stronger than the S-polarized light intensity. By comparing the intensity of the S-polarized component and the P-polarized component in the reflected light received by the polarization camera 10 due to the difference in polarization characteristics between the road surface and the three-dimensional object, the road surface is increased if the S-polarized component is strong. It can be understood that the reflected light is from a plane parallel to the road surface. If the P-polarized light component is strong, it can be understood that the reflected light is from a plane perpendicular to the road surface. As a result, by taking the difference value between the S-polarized component and the P-polarized component contained in the reflected light received by the polarization camera 10, depending on whether the difference value is positive or negative, whether the object has a plane parallel to the road surface, Can grasp whether the object has an outer surface facing in a different direction, that is, a three-dimensional object.

  Therefore, similarly to the road surface structure identification process, whether the area is a three-dimensional object by using the difference polarization degree difference value of each area divided by the edge determined by the edge determination process described above as an identification index value. It is possible to determine whether or not. However, for example, as shown in FIG. 27, when there is a part with significantly different luminance in the asphalt area, as in the case where the asphalt area includes a sunny part (reference processing area) and a shaded part, The boundary between the shaded portion (the portion with low luminance) and the sunny portion (the portion with high luminance) is erroneously recognized as an edge. Therefore, although it is the same road surface, the shaded portion (shaded road surface) is distinguished as an object different from the sunny portion (the sunny road surface). The shaded road surface thus distinguished as a separate object from the Hinata road surface may be misrecognized as a three-dimensional object even if it is distinguished from the solid object using the differential polarization degree difference value as an identification index value. there were.

FIG. 28 is a graph showing the degree of differential polarization at each position on a processing line (straight line extending in the horizontal direction in the figure) drawn in the image shown in FIG.
In this graph, there is a difference between the shaded road surface and the side wall in the differential polarization degree. Therefore, if the differential polarization degree difference value described above is used as an identification index value, the shaded road surface and the side wall (three-dimensional object) are distinguished. It is possible to recognize. However, according to the study by the present inventors, in the actual environment, the shaded road surface and various roadside obstacles (three-dimensional objects) including this side wall are distinguished with high accuracy only from the differential polarization degree difference value. It turned out to be difficult to recognize.

On the other hand, using a conventional method for identifying an object with luminance as an identification index value, even if it is attempted to distinguish between a shaded road surface and a roadside obstacle (three-dimensional object), the accuracy is as follows. The value is much lower than when the value is used as the identification index value.
FIG. 29 is a graph showing the luminance (monochrome luminance) at each position on the processing line (straight line extending in the horizontal direction in the drawing) drawn in the image shown in FIG.
As can be seen from this graph, there is no difference between the shaded road surface and the side wall (three-dimensional object) in the monochrome brightness, so if the monochrome brightness is used as the identification index value, it can be identified using the differential polarization degree. Even between a shady road surface and a side wall is difficult to distinguish.
The luminance here is expressed by a value normalized within the range of −1 to +1 in accordance with the notation of the differential polarization degree. Specifically, for example, when a monochrome luminance value is expressed by gradations of 1 to 256, 1 (black) in 256 gradation display is set to −1 in the luminance here, and 256 (white) in 256 gradation display. ) Is expressed as +1 in the luminance here.

FIG. 30 is a two-dimensional distribution diagram of various identification objects in which the brightness of the sun road surface is taken on the X axis and the brightness of various object objects other than the sun road surface existing in the imaging region is taken on the Y axis.
The objects to be identified are the sunshaded road surface (white frame portion on the right side in FIG. 27) existing on the same road surface as the Hinata road surface (left white frame portion in FIG. 27), and the hinata of the roadside obstacle (three-dimensional object). Part and shade part. Specifically, the roadside obstacles here are rough walls, white walls, guardrails, and wall reflectors. In FIG. 30, black circles in the figure are data on shaded road surfaces, and other marks are data on the sunlit and shaded parts of the various roadside obstacles described above.

  As can be seen from the two-dimensional distribution chart shown in FIG. 30, the shaded road surface distribution area indicated by black circles and the roads indicated by other marks are within the range where the brightness of the sun road surface is 0.6 or less that can be taken in the actual environment. The distribution areas of the end obstacles overlap each other. Therefore, the threshold value for distinguishing the shaded road surface from the roadside obstacle cannot be set only by the luminance data, and these cannot be identified.

Therefore, in the present embodiment, not only the differential polarization degree difference value but also the monochrome luminance used as the identification index value of the conventional method is used as the identification index value, thereby having a luminance different from that of the Hinata road surface. The shaded road surface and roadside obstacles (three-dimensional objects) are distinguished and recognized with high accuracy.
FIG. 31 is a three-dimensional distribution diagram of various identification objects, in which the differential polarization degree of the sun road surface is taken on the X axis, the differential polarization degree of the identification object is taken on the Y axis, and the luminance of the identification object is taken on the Z axis. It is.
As can be seen from such a three-dimensional distribution map, the shaded road surface distribution area indicated by black circles and the roadside obstacle distribution area indicated by other marks are within the range that the brightness of the sun road surface can take in the actual environment. Can be identified without overlapping each other. Therefore, by using a threshold value that defines a boundary surface that divides them, it is possible to identify roadside obstacles in distinction from shaded road surfaces. For such data separation, a known method such as SVM (Support Vector Machine) can be used.

  The flow of the three-dimensional object identification process according to the present embodiment will be described. First, for each area divided by the edge determined by the edge determination process described above, a three-dimensional object in which the difference polarization degree difference value of the area is determined in advance. It is determined whether or not it is equal to or less than the threshold for use (S21). In the present embodiment, the differential polarization degree for a three-dimensional object takes a positive value as can be seen from the histogram shown in FIG. Further, the differential polarization degree for the asphalt region having the reference differential polarization degree is around -0.05. Therefore, in the present embodiment, a positive value near zero (for example, +0.08) is adopted as the threshold for a three-dimensional object, and an area having a differential polarization degree difference value belonging to a range equal to or greater than this threshold is identified as a three-dimensional object. To do.

  However, the shaded road surface may be included in the region identified as a three-dimensional object by this determination as described above. Therefore, in this embodiment, a threshold value (shade road surface exclusion threshold value) that defines a boundary surface that can distinguish the distribution area of the shaded road surface and the distribution area of the roadside obstacle in the three-dimensional distribution diagram shown in FIG. 31 is used. Then, a process of eliminating the area that is a shaded road surface that is erroneously recognized as a three-dimensional object in S21 is performed (S22). Thereafter, the remaining area from which the shaded road area is removed is identified and stored as a three-dimensional object (S23). By performing this for all regions (S24), it is possible to grasp the region where the three-dimensional object is displayed in the captured image.

In addition, since there is a difference in the intensity of reflected light from the object between the upper part and the lower part of the captured image, taking into account this difference, the threshold for three-dimensional objects and the shade used in the upper part and the lower part of the captured image are considered. You may make it vary the threshold value for road surface exclusion.
In addition, since the lower part of the captured image obtained by photographing an object located nearby has higher object identification accuracy than the upper part, the processing order of the processing lines is preferably the order from the lower side to the upper side of the image.

Next, the three-dimensional object type specifying process performed by the three-dimensional object specifying unit 19 will be described.
First, the three-dimensional object specifying unit 19 recognizes the shape of the region identified as a three-dimensional object by the above-described three-dimensional object identification process (S25), and determines whether there is a shape template that approximates the shape of the region. (S26). In this determination, when it is determined that there is an approximate shape template, the three-dimensional object in the region is specified as a type associated with the shape template and stored (S27). For example, in the case of approximating a vehicle shape template, it is specified as another vehicle. If this is performed for all three-dimensional objects (S28), the process is terminated. In addition, since the method of shape approximation recognition in the three-dimensional object specifying unit 19 is the same as the processing method of the road surface structure specifying unit 17 described above, description thereof is omitted.

In order to improve the accuracy of the three-dimensional object type specifying process and the three-dimensional object identifying process, the past processing result may be used as in the case of specifying the type of the road surface structure.
Further, the edge threshold value used in the determination of the edge determination process (S3) described above, the threshold value for solid object used in the determination process of the three-dimensional object (S21, S22), and the threshold value for shading road surface exclusion depend on the difference in the shooting environment. You may make it switch suitably. A specific example is switching between time zones such as daytime and nighttime according to the difference in weather such as rainy weather and fine weather. This switching can be realized using time information or information such as a rain sensor or a sunshine sensor.
In addition, when the polarizing camera 10 of the present embodiment is mounted in a vehicle such as a rearview mirror, it is desirable to consider the polarization characteristics of the windshield as in the road surface structure specifying process described above.

  In the present embodiment, the shade road surface exclusion threshold defines the boundary surface in the three-dimensional distribution diagram shown in FIG. 31, but a parameter for obtaining higher discrimination is added, and the four-dimensional Alternatively, a boundary surface in a high-dimensional distribution map such as 5 or 5 dimensions may be defined. An example of such a parameter is the direction of the sun. In this case, the sun direction information can be obtained by extracting information about the sun direction and the vehicle traveling direction from the navigation system. Since the direction in which the shadow is formed with respect to the roadside obstacle (three-dimensional object) can be grasped by the sun direction information, it becomes possible to further improve the distinguishability. Also, solar altitude information may be added as the parameter. Similarly, solar altitude information can be obtained by obtaining information related to travel date and time via a navigation system or the like. Further, it is possible to add information such as a sunshine sensor used for autolight.

The result of specifying the type of the three-dimensional object by the three-dimensional object specifying unit 19 can be used for various processes.
For example, from the result of processing by the three-dimensional object specifying unit 19, the driver is warned of the approach of a recognized three-dimensional object that is an obstacle to be avoided, or the collision is avoided or controlled by controlling the vehicle's automatic brake system. To reduce the impact.
Further, for example, the processing result of the three-dimensional object specifying unit 19 may be used for white line edge identification processing in the white line identification unit 14. Specifically, since the region in which the type of the three-dimensional object is specified by the processing of the three-dimensional object specifying unit 19 is not a white line region, this region is excluded from the target of white line edge identification processing by the white line identification unit 14. Thereby, the possibility of erroneously recognizing a solid object such as another vehicle as a white line can be reduced, and the recognition accuracy of the white line can be improved. Here, the white line edge identifying process by the white line identifying unit 14 has been described. However, in the case of performing a process of identifying an object other than a white line from a captured image, in general, the processing of the three-dimensional object specifying unit 19 is performed. By excluding the region whose type is specified from the target of the identification process, the identification accuracy of the identification process can be improved.

  Further, for example, it may be used for a car navigation system. As a specific example, the distance, angle, etc. between the vehicle and the off-road obstacle is determined from the position of the off-road obstacle such as a telephone pole, a streetlight, and a sign specified from the processing result of the three-dimensional object specifying unit 19. Vehicle position information is generated, and the vehicle position information is used to specify a further detailed position of the vehicle within the range of the vehicle position calculated by the car navigation system. Thereby, the specific precision of the own vehicle position in a car navigation system can be improved.

Further, for example, the position and direction of various three-dimensional objects with respect to the own vehicle can be grasped from the processing result of the three-dimensional object specifying unit 19, and this may be used for a travel support ECU (Electronic Control Unit) or the like. .
In particular, a three-dimensional object that should avoid a collision is generated by using, for example, luminance data calculated by a monochrome image processing unit on a display unit (display) that is an in-vehicle information notification unit composed of a CRT, a liquid crystal, or the like. In order to display a monochrome image (front view image) and information of a three-dimensional object in the image as useful information for the driver, there is a process of displaying the information in a display form that is easy for the driver to see. According to this, for example, even in a situation where it is difficult for the driver to visually recognize the three-dimensional object, the driver confirms the presence of the three-dimensional object by looking at the front view image on the display unit. This makes it easy to avoid collisions.

As described above, the three-dimensional object identification device according to the present embodiment identifies a three-dimensional object that exists in an imaging region and has an outer surface facing a direction different from a predetermined plane (road surface), and exists in the imaging region. As imaging means for receiving two polarized lights (P-polarized light and S-polarized light) having different polarization directions contained in the reflected light from the object to be picked up and capturing respective polarized images (P-polarized image and S-polarized image) The polarization camera 10 and the P-polarized image and the S-polarized image captured by the polarization camera 10 are each divided into predetermined processing areas (one pixel unit), and each pixel is between the P-polarized image and the S-polarized image. A monochrome image processing unit 13 serving as a luminance calculating means for calculating a monochrome luminance that is a luminance total value, and a difference bias indicating a ratio of a luminance difference value between the P-polarized image and the S-polarized image with respect to the monochrome luminance for each pixel. A differential polarization degree image processing unit 15 serving as a differential polarization degree calculation means for calculating a degree, and a process corresponding to a location where a reference plane object (asphalt) existing in the same plane as the predetermined plane is present. The differential polarization degree calculated by the differential polarization degree image processing unit 15 for the reference processing area (area corresponding to the sun road surface) that is a relatively high luminance portion of the area is defined as a reference differential polarization degree. What is the reference processing area? Differential polarization degree image processing as differential polarization degree difference value calculation means for calculating a differential polarization degree difference value that is a difference value between the differential polarization degree calculated by the differential polarization degree image processing unit 15 and a reference differential polarization degree for different processing regions. And the monochrome luminance calculated by the monochrome image processing unit 13 and the differential polarization degree difference value calculated by the differential polarization degree image processing unit 15 as the identification index values, An object existing at a location corresponding to each pixel definitive has a three-dimensional object recognition unit 18 as a three-dimensional object identification processing means for performing three-dimensional object identification process whether a three-dimensional object. Thereby, it can distinguish and recognize between a shade road surface and a solid object with high precision.
Further, in the present embodiment, the three-dimensional object identification unit 18 is a place corresponding to a low-luminance processing area (area corresponding to a shaded road surface) that is a portion having a low luminance among the processing areas corresponding to the location where the asphalt (road surface) exists. A determination process is performed to determine which of a plurality of numerical ranges defined respectively for a shaded road surface and a three-dimensional object existing in the area belongs to, and an object present at a location corresponding to the processing area is determined The three-dimensional object identification process is performed by performing a process of identifying the object corresponding to the numerical range determined to belong to the process. With such a process, the three-dimensional object identification process can be realized by a simple process of comparison with a threshold value.
In the present embodiment, the polarization camera 10 is mounted on a vehicle (own vehicle) as a moving body that moves on a road surface that is a moving surface, and images an imaging region including the road surface obliquely from above the road surface. The numerical range defined for the same object is set for each of at least two or more areas that divide the P-polarized image and the S-polarized image in the vertical direction, and the difference between the processing areas belonging to the area is set. A solid object identification process is performed by determining which of the numerical ranges set for the area the polarization degree belongs to. Accordingly, it is possible to appropriately identify the difference in the amount of light received by the polarization camera 10 in the vertical direction of the P-polarized image and the S-polarized image.
Moreover, in this embodiment, it preserve | saves in the memory as an identification process result storage means which memorize | stores the result of the identification process which the solid-object identification part 18 performed in the past, and the solid-object identification part 18 is added with the said identification index value. The identification processing is performed using the past identification processing result stored in the memory. This makes it possible to determine the reliability of the identification result as to whether the same result as the past identification result has been obtained.
In addition, as described above, the three-dimensional object identification device in the present embodiment travels as a movement control unit that performs movement control of a vehicle (own vehicle) that is a moving body using the identification result of the three-dimensional object identification device. The present invention can be applied to an automatic brake system as a moving body control device including a support ECU.
In addition, as described above, the three-dimensional object identification device according to the present embodiment generates information that is useful for a driver who drives a vehicle (own vehicle) that is a moving body using the identification result of the three-dimensional object identification device. And it is applicable also to the information provision apparatus which alert | reports the produced | generated information to the said driver | operator.

  Note that the driver assistance system according to the present embodiment is mounted on the vehicle as a whole, but the entire system does not necessarily have to be mounted on the vehicle. Therefore, for example, only the polarization camera 10 may be mounted on the own vehicle, and the remaining system components may be remotely arranged at a location different from the own vehicle. In this case, a system in which a person other than the driver objectively grasps the traveling state of the vehicle can be provided.

DESCRIPTION OF SYMBOLS 10 Polarization camera 11 Horizontally polarized image memory 12 Vertically polarized image memory 13 Monochrome image processing part 14 White line identification part 15 Differential polarization degree image processing part 16 Road surface structure identification part 17 Road surface structure identification part 18 Three-dimensional object identification part 19 Three-dimensional object identification Unit 20 Shape storage unit 101, 111, 112 Camera 102 Rotating polarizer 113, 114, 124, 133, 134, 144, 152, 153 Polarization filter

JP 2009-59260 A

Claims (6)

In a three-dimensional object identification device for identifying a three-dimensional object that exists in an imaging region and has an outer surface facing a direction different from a predetermined plane,
Imaging means for receiving two polarized light beams having different polarization directions included in reflected light from an object existing in the imaging region and capturing respective polarized images;
A luminance calculating unit that divides two polarized images captured by the imaging unit into predetermined processing regions, and calculates a total luminance value between the two polarized images for each processing region;
Differential polarization degree calculating means for calculating a differential polarization degree indicating a ratio of a luminance difference value between the two polarized images with respect to the total luminance value for each processing region;
The differential polarization degree calculation means for the reference processing region that is a portion having a relatively high luminance among the processing regions corresponding to the location where the reference plane object is assumed to exist in advance in the same plane as the predetermined plane. The calculated difference polarization degree is set as a reference difference polarization degree, and a difference polarization degree difference which is a difference value between the difference polarization degree calculated by the difference polarization degree calculation unit and the reference difference polarization degree for a processing region different from the reference processing region. Differential polarization degree difference value calculating means for calculating a value;
The luminance total value calculated by the luminance calculation means and the differential polarization degree difference value calculated by the differential polarization degree difference value calculation means are used as identification index values and are present at locations corresponding to the respective processing areas in the imaging area. A three-dimensional object identification device comprising: a three-dimensional object identification processing unit that performs a three-dimensional object identification process for determining whether or not an object is the above-described three-dimensional object. The three-dimensional object identification device according to claim 1 ,
The three-dimensional object identification processing means includes a low-luminance portion of the reference plane object existing in a location corresponding to a low-luminance processing region that is a portion having a low luminance in a processing region corresponding to the location of the reference plane object, and the three-dimensional object. Judgment processing is performed to determine which of the plurality of numerical ranges defined for each object belongs to the identification index value, and an object existing at a location corresponding to the processing area is determined to belong to the determination processing A three-dimensional object identification device, wherein the three-dimensional object identification process is performed by performing a process of identifying an object corresponding to a numerical range. The three-dimensional object identification device according to claim 2 ,
The imaging means is mounted on a moving body that moves on a moving surface, and picks up an imaging region including the moving surface obliquely from above the moving surface,
In the determination process, a numerical range defined for the same object is set for each of at least two or more sections that respectively divide the two polarized images in the vertical direction, and an identification index of a processing region belonging to the section. A three-dimensional object identification device, wherein a value belongs to which of a numerical range set for the area. The three-dimensional object identification device according to any one of claims 1 to 3 ,
The solid object identification processing means has identification processing result storage means for storing the result of the solid object identification processing performed in the past,
The three-dimensional object identification processing unit performs the three-dimensional object identification process using the identification index value and the result of the past three-dimensional object identification process stored in the identification process result storage unit. Identification device. A movement control means for performing movement control of the moving body;
A solid object identifying means for imaging the periphery of the moving object as an imaging target, and identifying a three-dimensional object existing in the imaging target;
In the moving body control device that performs the movement control using the identification result by the three-dimensional object identification unit,
A mobile object control device using the three-dimensional object identification device according to any one of claims 1 to 4 as the three-dimensional object identification means. Three-dimensional object identification means for imaging a moving object moving according to a driving operation by a driver as an imaging target, and identifying a three-dimensional object existing in the imaging target;
Beneficial information generating means for generating information useful for the driver using the identification result by the three-dimensional object identifying means;
In an information providing apparatus having information notifying means for notifying the driver of information generated by the useful information generating means,
An information providing apparatus using the three-dimensional object identification device according to any one of claims 1 to 4 as the three-dimensional object identification means.