JPWO2015045501A1 - External recognition device - Google Patents

Abstract

Provided is an external environment recognition device capable of recognizing an environment outside a vehicle by comparing a plurality of images having different optical conditions with a single monocular camera or a stereo camera.
Cameras 1L and 1R that are mounted on a vehicle and acquire image information of a region to be photographed in front of the vehicle include an imaging element 13 having first and second imaging regions 13a and 13b, and first and second imaging regions 13a. , 13b, a first optical member 11 that forms an image of the imaging target area, and a second optical element that allows light having different optical conditions to pass through each of the first and second imaging areas 13a, 13b. And a member 12 (for example, a polarizing filter). The image processing unit 2 compares the first image information obtained from the first imaging area 13a with the second image information obtained from the second imaging area 13b, and compares the state of the imaging target area (for example, the road surface). The friction coefficient).

Description

  The present invention relates to an in-vehicle external environment recognition device, for example, an apparatus for recognizing an environment outside a vehicle based on image information of an image sensor.

  2. Description of the Related Art Conventionally, an external recognition device such as a stereo camera device that is mounted on a vehicle and recognizes a situation in front of the vehicle is known. For example, a stereo camera device that accurately measures a distance to a wide-range subject from a short distance to a long distance and secures a wide field of view with a wide-angle lens at a short distance, and a vehicle exterior monitoring device using the stereo camera device are disclosed. (See Patent Document 1 below).

  The stereo camera device described in Patent Document 1 processes first and second imaging units arranged on the left and right sides to obtain two stereo images for short distance and long distance, and a long distance image. A first camera control unit and a second camera control unit that processes an image for short distance are provided.

  The first imaging unit includes a first imaging lens for a long distance, a second imaging lens for a short distance, an imaging element having respective imaging regions for a short distance and a long distance, and each imaging lens. And a prism and a polarization beam splitter disposed between the image pickup device and the image pickup device. Similarly, the second imaging unit includes a third imaging lens for a long distance, a second imaging lens for a short distance, and an imaging element having respective imaging regions for a short distance and a long distance, A prism and a polarizing beam splitter are provided between each imaging lens and the imaging element.

  The prism and the polarization beam splitter of each imaging unit allow only the S-polarized light component of the light transmitted through the long-distance imaging lens to pass through the long-distance imaging region of the imaging device. Further, only the P-polarized light component of the light transmitted through the short-distance imaging lens is passed through the short-distance imaging region of the imaging element. Accordingly, the first camera control unit combines the right image and the left image of the long-distance subject imaged in the long-distance imaging region of the imaging element of each imaging unit as one image. Similarly, the second camera control unit combines the right image and the left image of the short-distance subject imaged in the short-distance imaging region of the imaging element of each imaging unit as one image.

JP 2010-261877 A

  The stereo camera device described in Patent Document 1 can simultaneously obtain a stereo image of a long-distance subject with different polarization components and a stereo image of a short-distance subject. It is difficult to recognize the state of the imaging target region by comparing these images.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an external environment recognition device capable of recognizing an environment outside a vehicle by comparing a plurality of images having different optical conditions.

  In order to achieve the above object, an external environment recognition apparatus according to the present invention includes an external camera that is mounted on a vehicle and acquires image information of a shooting target area in front of the vehicle, and an image processing unit that processes the image information. An apparatus, wherein the camera includes an imaging element having first and second imaging areas, and a first optical member that forms an image of the imaging target area on each of the first and second imaging areas. And a second optical member that allows light of different optical conditions to pass through each of the first and second imaging regions, and the image processing unit is a first obtained from the first imaging region. And the second image information obtained from the second imaging area are compared to recognize the state of the imaging target area.

  According to the external environment recognition apparatus of the present invention, the environment outside the vehicle can be recognized by comparing a plurality of images having different optical conditions. Other problems and effects of the present invention will be clarified by the following embodiments.

The block diagram which shows schematic structure of a vehicle-mounted system provided with the external field recognition apparatus which concerns on Embodiment 1 of this invention. The block diagram which shows schematic structure of the image process part of the external field recognition apparatus shown in FIG. The figure which shows the arrangement | positioning and coordinate system of the camera of the external field recognition apparatus shown in FIG. The figure which shows the imaging | photography object area | region of the camera of the external field recognition apparatus shown in FIG. (A) is a figure which shows the image of the imaging | photography object area | region which let the polarizing filter pass, (b) is a figure which shows the image of the imaging | photography object area | region which does not pass a polarizing filter. Explanatory drawing of the camera with which the external field recognition apparatus shown in FIG. 1 is provided. Explanatory drawing of the image of the imaging | photography object area | region acquired with the camera shown in FIG. (A) And (b) is a figure which shows an example of the image of the imaging | photography object area | region acquired with the camera shown in FIG. (A) to (c) is a diagram showing a modification of the image of the imaging target region shown in FIG. The block diagram which shows schematic structure of the image process part of the external field recognition apparatus which concerns on Embodiment 2 of this invention. Explanatory drawing of the camera with which the external field recognition apparatus which concerns on Embodiment 2 of this invention is provided. The figure which shows the image of the imaging | photography object area | region acquired with the camera with which the external field recognition apparatus which concerns on Embodiment 2 of this invention is provided. Explanatory drawing of the estimation and learning of the road surface friction coefficient based on the image of an imaging | photography object area | region.

  DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of an external environment recognition device according to the present invention will be described below with reference to the drawings.

[Embodiment 1]
FIG. 1 is a block diagram illustrating a schematic configuration of an in-vehicle system including an external environment recognition device 100 according to Embodiment 1 of the present invention. FIG. 2 is a block diagram showing a schematic configuration of the image processing unit 2 of the external environment recognition apparatus 100 shown in FIG.

  The external environment recognition apparatus 100 according to the present embodiment is an apparatus that is mounted on a vehicle and recognizes the environment outside the vehicle based on image information of a shooting target area in front of the vehicle. The external environment recognition apparatus 100 is used for, for example, road surface wetness, snow accumulation and pattern identification, pedestrians, cyclists and animal / background identification, or image processing under bad weather conditions such as snowfall, rain, and fog.

  The external environment recognition apparatus 100 includes two cameras 1L and 1R arranged on the left and right sides for acquiring image information, and an image processing unit 2 for processing the image information. Each of the left camera 1L and the right camera 1R includes a first optical member 11, a second optical member 12, and an image sensor 13. The imaging element 13 has a first imaging area 13a and a second imaging area 13b that convert the images of the imaging target areas of the cameras 1L and 1R into image information.

  The first optical member 11 includes a first lens 11a and a second lens 11b that are arranged adjacent to each other on the optical paths of the cameras 1L and 1R. The first lens 11a is disposed so as to overlap a part of the second lens 11b on the optical path. In the present embodiment, the first lens 11a and the second lens 11b are separate members. However, the first lens 11a and the second lens 11b are integrally formed to form the first lens 11a and the second lens 11b. The optical member 11 may be a single member.

  The second optical member 12 is a polarizing filter, for example, and is an optical filter that selectively transmits light of a predetermined polarization component. The second optical member 12 is disposed on the optical path of the first imaging region 13a and is not disposed on the optical path of the second imaging region 13b, so that light of a predetermined polarization component is present in the first imaging region 13a. L1 is allowed to pass, and unpolarized light L2 is allowed to pass through the second imaging region 13b. That is, the second optical member 12 allows the light L1 and L2 having different optical conditions to pass through the first and second imaging regions 13a and 13b, respectively.

  The first optical member 11 causes light of a predetermined polarization component that has passed through the second optical member 12 to enter the second lens 11 b, and the image of the imaging target region is displayed on the first imaging region 13 a of the imaging device 13. Make an image. In addition, the first optical member 11 causes the non-polarized light that does not pass through the second optical member 12 to enter the first lens 11 a and the second lens 11 b, so that the second imaging region on the imaging element 13. An image of the imaging target area is formed on 13b.

  That is, the cameras 1L and 1R according to the present embodiment include the polarization region A1 in which the first and second optical members 11 and 12 are disposed on the optical path, and the first optical member 11 that is not disposed in the optical path. And a polarization region A2. In addition, the cameras 1L and 1R form an image of the imaging target area on the first imaging area 13a via the polarization area A1, and shoot on the second imaging area 13b via the non-polarization area A2. An image of the target area is formed.

  The image processing unit 2 includes first image information obtained from the first imaging regions 13a and 13a of the imaging devices 13 and 13 of the cameras 1L and 1R, and second imaging of the imaging devices 13 and 13 of the cameras 1L and 1R. The second image information obtained from the areas 13b and 13b is compared to recognize the state of the shooting target areas of the cameras 1L and 1R. Hereinafter, the image processing unit 2 will be described in detail.

  The image processing unit 2 is connected to the imaging devices 13 of the two cameras 1 arranged as the left camera 1L and the right camera 1R of the stereo camera, and as shown in FIG. 2, the image processing unit 21 and the parallax processing unit 22, road surface estimation means 23, and road surface μ data 24. Each means is composed of, for example, a single or a plurality of computer units, and is configured to be able to exchange data with each other.

  The image processing unit 2 converts the signals output from the first imaging regions 13a and 13a of the imaging devices 13 and 13 of the left and right cameras 1L and 1R into first image information by the image processing unit 21. I do. The first image information is image information based on the image of the imaging target area formed on the first imaging area 13a of the imaging element 13 via the polarization areas A1 of the left and right cameras 1L and 1R. Image information through the optical member 12, that is, the polarization filter.

  The image processing unit 2 is connected to the control unit 3, the storage unit 4, and the input / output unit 5 via the bus 6. The control unit 3 includes a single or a plurality of computer units that control the image processing unit 2, the storage unit 4, and the input / output unit 5. The storage unit 4 is configured by, for example, a memory that stores image information and the like obtained by the image processing unit 2. The input / output unit 5 outputs the information output from the external environment recognition device 100 to the control system 8 of the host vehicle via the CAN 7. For example, the control system 8 can issue a warning to the occupant, brake the host vehicle, or perform avoidance control of the object based on information from the external environment recognition device 100.

  The image processing unit 2 uses the image processing unit 21 to convert the signals output from the second imaging regions 13b and 13b of the imaging devices 13 and 13 of the left and right cameras 1L and 1R into second image information. I do. The second image information is image information based on the image of the imaging target area formed on the second imaging area 13b of the imaging element 13 via the non-polarized areas A2 of the left and right cameras 1L and 1R. The second optical member 12, that is, image information not passing through a polarizing filter.

  FIG. 3 is a diagram showing the arrangement and coordinate system of the cameras 1L and 1R of the external environment recognition apparatus 100 shown in FIG. The external recognition apparatus 100 is an xyz orthogonal coordinate system in which an intermediate point between the left and right cameras 1L and 1R is an origin o, a horizontal direction (vehicle width direction) is an x axis, a vertical direction is a y axis, and a traveling direction of the vehicle is a z axis. Is used. The left and right cameras 1L and 1R each have a predetermined angle range α that is symmetrical with respect to the center lines c1 and c2 parallel to the z-axis as an imaging target region.

  The image processing unit 2 uses the parallax processing unit 22 to perform the parallax processing of the first image information based on the video through the second optical member 12 of the left and right cameras 1L and 1R obtained by the image processing unit 21. . In other words, the external environment recognition apparatus 100 uses the parallax processing unit 22 to perform a shooting target area between the first image information of the shooting target area of the left camera 1L and the first image information of the shooting target area of the right camera 1R. The parallax process of the same point is performed, and the distance to each point in the imaging target region can be obtained by the principle of triangulation.

  The image processing unit 2 includes road surface estimation means 23 for estimating a road friction coefficient μ and road surface μ data 24 as a road surface μ estimation dictionary. The road surface estimation means 23 includes first image information based on the image of the imaging target area via the second optical member 12 obtained by the image processing means 21 and the imaging target area not via the second optical member 12. Second image information based on the video is acquired. In the road surface μ data 24, the relationship between the first image information, the second image information, and the friction coefficient μ of the road surface based on the calculation and learning of the road surface estimation means 23 is recorded. The road surface estimating means 23 refers to the road surface μ data 24 based on the first and second image information acquired from the image processing means 21 and estimates the friction coefficient μ of the road surface in the imaging target area. Details of the calculation and learning of the road surface estimation means 23 and the road surface μ data 24 will be described later.

  FIG. 4 is a diagram illustrating the relationship between the image information P1 of the shooting target area of the left camera 1L and the image information P2 of the shooting target area of the right camera 1R. In a stereo camera, normally, parallax processing is performed in a region where the image information P1 of the shooting target region of the left camera 1L and the image information P2 of the shooting target region of the right camera 1R overlap. That is, the stereoscopic imaging target is imaged in the image information P1a on the right side of the image information P1 of the left camera 1L and the image information P2a on the left side of the image information P2 of the right camera 1R. On the other hand, the image information P1b of the left region of the image information P1 of the left camera 1L and the image information P2b of the right region of the image information P2 of the right camera 1R are not subjected to parallax processing and are not used for stereoscopic viewing. Note that the imaging targets of the left and right cameras 1L and 1R are, for example, the road surface RS, road elements RE such as water, ice, snow or oil on the road surface RS, obstacles OB including pedestrians and cyclists, or rain, snow, Includes meteorological phenomena WP such as smoke.

  FIGS. 5A and 5B are explanatory diagrams for estimating the friction coefficient of the road surface. FIG. 5A is a diagram showing image information PA1 of the imaging target region based on light that has passed through the polarizing filter, and FIG. 5B is not through the polarizing filter. It is a figure which shows the image information PA2 of the imaging | photography object area | region based on light. The road surface estimation means 23 includes the road element RE and the road surface RS of the image information PA1 of the imaging target area that has passed through the polarization filter shown in FIG. 5A and the imaging target area that has passed through the polarization filter shown in FIG. The road surface μ data 24 is referred to based on the road element RE and the road surface RS of the image information PA1. Then, the road surface estimation means 23 estimates the road surface friction coefficient μ on the road surface RS and the road element RE based on the reference result of the road surface μ data 24.

  FIG. 6 is an explanatory diagram of a camera provided in the external environment recognition apparatus 100 shown in FIG. As described above, the light L from the imaging targets RE, RS, OB, and WP in the imaging target areas of the left and right cameras 1L and 1R is the polarization filter in the polarization area A1 of the left and right cameras 1L and 1R. Passes through the optical member 12. Then, the light L1 having a predetermined polarization component that has passed through the optical member 12 is collected by the second lens 11b of the first optical member 11, and forms an image on the first imaging region 13a of the imaging element 13. In addition, the light L from the imaging targets RE, RS, OB, and WP does not pass through the second optical member 12 that is a polarization filter in the non-polarization region A2 of the left and right cameras 1L and 1R, and the first optical The light is collected by the first lens 11 a and the second lens 11 b of the member 11, and forms an image on the second imaging region 13 b of the imaging device 13. Thereby, the image information P1, P2 of the imaging target region as shown in FIG. 7 is acquired by the left and right cameras 1L, 1R.

  FIG. 7 is an explanatory diagram of the image information P1 and P2 of the shooting target area acquired by the left and right cameras 1L and 1R shown in FIG. Image information P1a on the right side of the image information P1 on the imaging target area of the left camera 1L is based on an image formed on the first imaging area 13a of the imaging element 13 of the left camera 1L in the polarization area A1. Further, the image information P1b in the left area of the image information P1 is based on a video image formed in the second imaging area 13b of the imaging element 13 in the non-polarized area A2 of the left camera 1L. Similarly, the image information P2a of the left area of the image information P2 of the imaging target area of the right camera 1R is based on the image formed on the first imaging area 13a of the imaging element 13 of the right camera 1R in the polarization area A1. ing. Further, the image information P2b in the right region of the image information P1 is based on a video image formed on the second imaging region 13b of the imaging device 13 in the non-polarized region A2 of the right camera 1L.

  In the example shown in FIGS. 6 and 7, as described above, the polarization area A1 is provided corresponding to the right side of the imaging target area of the left camera 1L and the left side of the imaging target area of 1R of the right camera. The non-polarized area A2 is provided corresponding to the left side of the shooting target area of the left camera 1L and the right side of the shooting target area of the right camera 1R. In addition, the area of the polarization region A1 of the left and right cameras 1L and 1R is larger than the area of the non-polarization region A2.

  FIGS. 8A and 8B are diagrams illustrating examples of image information P1 and P2 of the shooting target area acquired by the cameras 1L and 1R illustrated in FIG. In the example shown in FIG. 8A, among the image information P1 and P2 of the left and right cameras 1L and 1R, the image information P1a and P2a of the region corresponding to the polarization region A1 in which the second optical member 12 is disposed is parallax. Used for processing. Of the image information P1 and P2 of the left and right cameras 1L and 1R, the image information P1b and P2b in the area corresponding to the non-polarized area A2 where the second optical member 12 is not disposed corresponds to the polarized area A1. The image information in the same range as the imaging target area of the area image information P1a and P2a is acquired.

  Note that the optical filters FL and FR may be arranged on the optical path of at least one non-polarization region A2 of the left and right cameras 1L and 1R shown in FIG. In this case, at least one of the image information P1b and P2b in the region corresponding to the non-polarized region A2 of the image information P1 and P2 shown in FIG. 8A is the image information based on the light that has passed through the optical filters FL and FR. Become. As the optical filters FL and FR, for example, a polarizing filter, a near-infrared filter, an ultraviolet filter, or the like that transmits a polarization component different from that of the second optical member 12 can be used. When the optical filters FL and FR are provided in the non-polarization regions A2 of the left and right cameras 1L and 1R, optical filters with the same optical conditions may be used for the optical filters FL and FR, or different optical conditions may be used. An optical filter may be used.

  In the example shown in FIG. 8B, as in the example shown in FIG. 8A, among the image information P1 and P2 of the left and right cameras 1L and 1R, the polarization region A1 in which the second optical member 12 is arranged. The image information P1a and P2a of the area corresponding to is used for the parallax processing. On the other hand, among the image information P1 and P2 of the left and right cameras 1L and 1R, the image information P1b and P2b in the region corresponding to the non-polarized region A2 where the second optical member 12 is not disposed corresponds to the polarized region A1. The image information of the left half and the image information of the right half of the imaging target area of the image information P1a and P2a of the area are acquired.

  As described above, the external environment recognition apparatus 100 having the above configuration captures an imaging target area in front of the vehicle with the left and right cameras 1L and 1R constituting the stereo camera, and obtains it from the first imaging area 13a of the imaging element 13. The first image information P1a and P2a to be obtained and the second image information P1b and P2b obtained from the second imaging region 13b are compared by the image processing unit 2 to recognize the state of the imaging target region. Therefore, according to the external environment recognition apparatus 100 of the present embodiment, the first image information P1a and P2a and the second image information P1b and P2b that are different in the optical conditions acquired by the left and right cameras 1L and 1R constituting the stereo camera, It becomes possible to recognize the environment outside the vehicle.

  Specifically, the cameras 1L and 1R include a polarization region A1 in which the first and second optical members 11 and 12 are disposed on the optical path, and a non-polarization region in which the first optical member 11 is disposed on the optical path. A2 is formed on the first imaging region 13a of the image sensor 13 through the polarization region A1, and the second imaging region 13b is coupled with the image through the non-polarization region A2. Let me image. In this way, by comparing the first image information P1a and P2a through the polarizing filter and the second image information P1b and P2b through the polarizing filter, water, ice, snow on the road surface RS, It is possible to discriminate the road element RE such as oil and the pattern on the road surface RS and estimate the friction coefficient μ of the road surface RS and the road element RE.

  That is, the image processing unit 2 includes road surface μ data 24 that records the relationship between the friction coefficient μ of the road surface RS, the first image information P1a and P2a, and the second image information P1b and P2b, and the first image information. Based on P1a and P2a and the second image information P1b and P2b, the road surface μ data 24 is referred to, and the friction coefficient μ of the road surface RS and the road element RE in the imaging target region is estimated. Thus, when the road surface RS and the road element RE having a low friction coefficient μ are recognized, information is output via the input / output unit 5, a warning is given to the vehicle occupant by the vehicle control system 8, and the vehicle braking control is performed. Or avoidance control for avoiding danger can be performed.

  Two cameras 1R and 1L are arranged as a left camera 1L and a right camera 1R, and the image processing unit 2 includes first image information P1a obtained by the left camera 1L and a first image obtained by the right camera 1R. The parallax processing of the image information P2a is performed. Thereby, the influence of the reflected light from in-vehicle equipment or glass can be reduced, and the distance to an obstacle OB including a road element RE, a pedestrian, or a cyclist, which is a subject to be photographed in the photographing target region, can be accurately measured. it can. Further, by making the area of the polarization region A1 larger than the area of the non-polarization region A2, the image information P1a and P2a to be subjected to the parallax processing can be increased, and a more accurate distance measurement can be performed.

  Further, when the optical filters FL and FR are arranged on the optical path of at least one non-polarized region A2 of the left and right cameras 1L and 1R, not only the determination of the road surface RS and the road element RE and the estimation of the friction coefficient μ are performed. This makes it possible to perform more diverse recognition of the outside world.

  For example, a near-infrared filter can be used for the optical filter FL or FR disposed on the optical path of one non-polarization region A2 of the left and right cameras 1L and 1R. In this case, for example, in the example shown in FIG. 8A, the image information through one of the near-infrared filters of the second image information P1b and P2b and the image not through the other near-infrared filter. By comparing with information, it is possible to detect pedestrians, cyclists, animals, etc., and to improve the accuracy and accuracy of the collision prevention function.

  Moreover, a near infrared filter, an infrared filter, an ultraviolet filter, etc. can be used as the optical filters FL and FR arranged on the optical path of at least one non-polarization region A2 of the left and right cameras 1L and 1R. In this case, for example, in an adverse weather environment such as rain, snow, haze, etc., the scattering between the ideal image and the actually captured image is made using the fact that the scattering coefficient for each wavelength band of light is different. The coefficient can be estimated. Furthermore, a high quality image can be obtained by performing an inverse operation on the estimated scattering coefficient. Further, by making the optical conditions of the optical filters FL and FR of the left and right cameras 1L and 1R different, more various estimations are possible. In this case, not only the polarizing filter but also the fume removal can be performed using a near infrared filter, an infrared filter, and an ultraviolet filter.

  The first optical member 11 includes first and second lenses 11a and 11b, the first lens 11a and the second optical member 12 are disposed on the optical path of the polarization region A1, and the non-polarization region A2. The first lens 11a and the second lens 11b are arranged on the optical path. By adopting such a configuration, the cameras 1L and 1R can be easily manufactured and the manufacturing cost can be reduced.

  As described above, according to the external environment recognition apparatus 100 of the present embodiment, it is possible to recognize the environment outside the vehicle by comparing a plurality of images having different optical conditions depending on the stereo camera.

  In the external environment recognition apparatus 100 of the present embodiment, as shown in FIGS. 6, 7, 8A and 8B, the image information P1b and P2b of the area corresponding to the non-polarized area A2 is stored in the left camera 1L. The image information P1 is arranged on the left side of the image information P1 and on the right side of the image information P2 of the right camera 1R. 9 (a), 9 (b) and 9 (c) show modifications thereof.

  In the example shown in FIG. 6, the polarization areas A1 are provided corresponding to the right and left sides of the imaging target areas of the left and right cameras 1L and 1R. However, the polarization area A1 is provided corresponding to the upper side of the imaging target areas of the left and right cameras 1L and 1R, and the non-polarization area A2 is provided corresponding to the lower side of the imaging target areas of the left and right cameras 1L and 1R. May be.

  As a result, as shown in FIG. 9A, the image information P1a and P2a corresponding to the polarization area A1 is arranged above the image information P1 and P2 of the imaging target areas of the left and right cameras 1L and 1R. On the other hand, the image information P1b and P2b corresponding to the non-polarized area A2 is arranged below the image information P1 and P2 of the shooting target areas of the left and right cameras 1L and 1R. The lower area of the image information P1 and P2 is image information of an area approaching the vehicle and is not normally used for parallax processing, so that it is possible to effectively use this area.

  The polarization area A1 is provided corresponding to the lower side of the imaging target areas of the left and right cameras 1L and 1R, and the non-polarization area A2 is provided corresponding to the upper side of the imaging target areas of the left and right cameras 1L and 1R. May be. As a result, as shown in FIG. 9B, the image information P1a and P2a corresponding to the polarization area A1 is arranged below the image information P1 and P2 of the imaging target areas of the left and right cameras 1L and 1R. On the other hand, the image information P1b and P2b corresponding to the non-polarized area A2 is arranged above the image information P1 and P2 of the shooting target areas of the left and right cameras 1L and 1R. The upper areas of the image information P1 and P2 are image information corresponding to the sky ahead of the vehicle, and are not normally used for parallax processing, so that these areas can be used effectively.

  Further, the arrangement of the image information P1a and P2a and the image information P1b and P2b shown in FIGS. 7, 9A, and 9B can be appropriately combined. For example, as shown in FIG. 9C, the image information P1a and P2a corresponding to the polarization region A1 are arranged on the lower right side and the left side of the image information P1 and P2 of the left and right cameras 1L and 1R, and the non-polarization region The image information P1b and P2b corresponding to A2 can be arranged on the upper side, the left side and the right side of the image information P1 and P2 of the left and right cameras 1L and 1R. As a result, it is possible to effectively use a region that is not normally used for parallax processing.

[Embodiment 2]
Next, Embodiment 2 of the external recognition apparatus of the present invention will be described with reference to FIGS. 10 to 12 with reference to FIG.

  FIG. 10 is a block diagram illustrating a schematic configuration of the image processing unit 2B of the external environment recognition device 100B according to the second embodiment of the present invention. FIG. 11 is an explanatory diagram of the monocular camera 1 provided in the external environment recognition device 100B of the present embodiment. FIG. 12 is a diagram illustrating an image of the imaging target area acquired by the monocular camera 1 provided in the external environment recognition device 100B of the present embodiment.

  The external environment recognition device 100B of the present embodiment does not include a stereo camera composed of left and right cameras 1L and 1R, but includes one monocular camera 1 having the same configuration as the left camera 1L or the right camera 1R shown in FIG. This is different from the external environment recognition apparatus 100 of the first embodiment. Further, the image processing unit 2B included in the external environment recognition device 100B of the present embodiment includes an object recognition processing unit 22B instead of the parallax processing unit 22. Since the other points are the same, the same parts are denoted by the same reference numerals and description thereof is omitted.

  As shown in FIG. 11, the monocular camera 1 provided in the external environment recognition device 100 </ b> B of the present embodiment forms a first image of the imaging target region on each of the first and second imaging regions 13 a and 13 b of the imaging device 13. 1 optical member 11 is provided. The monocular camera 1 also includes a second optical member 12 that allows light of different optical conditions to pass through each of the first and second imaging regions 13a and 13b of the imaging device 13. The first optical member 11 includes a plurality of first lenses 11a. In the present embodiment, for example, two first lenses 11 a and 11 a are arranged on both sides of the second optical member 12 in the y direction. In addition, the imaging element 13 includes second imaging regions 13b on both sides in the y direction of the first imaging region 13a.

  The external environment recognition apparatus 100B having the above configuration captures an imaging target area in front of the vehicle with the monocular camera 1, and recognizes the imaging targets RE, RS, OB, and WP with the object recognition processing unit 22B of the image processing unit 2B. Then, as shown in FIG. 12, the first image information P1a obtained from the first imaging area 13a and the second image information P1b obtained from the second imaging area are compared in the same manner as in the first embodiment. Then, the state of the imaging target area is recognized. Therefore, according to the external environment recognition device 100B of the present embodiment, the first image information P1a and the second image information P1b obtained by the monocular camera 1 with different optical conditions are the same as the external environment recognition device 100 of the first embodiment. It becomes possible to recognize the environment outside the vehicle.

[Estimation and learning of road friction coefficient]
Finally, estimation and learning of the road surface friction coefficient based on the first image information P1a and P2b and the second image information P1b and P2b of the imaging target area common to the first and second embodiments will be described. FIG. 13 is an explanatory diagram of the estimation and learning of the road surface friction coefficient using the road surface μ data 24. FIG. 13 and Table 1 below describe a mechanism for learning a road surface μ estimation dictionary (hereinafter referred to as road surface μ data 24) necessary for estimating a road surface friction coefficient μ (hereinafter also referred to as road surface μ). Yes. Here, a learning method using automatic differentiation will be mainly described.

  In general, the process for learning the road surface μ data 24 for recognizing a specific pattern in an image is as follows. First, as shown in FIG. 13, for example, image / original signals S1, S2,... Such as first image information P1a, P2b and second image information P1b, P2b are stored in the road surface μ data 24. In addition to various parameters, various vector operations, matrix operations, autocorrelation operations, convolution operations, and other operations OP are performed, and the result of determining what the unknown pattern category is is output to verify its correctness . Automatic differentiation AD (Automatic Differentiation) is a basic numerical operation system that supports this calculation process.

  In automatic differential AD, “number” and “calculation” are uniquely defined. Table 1 shows examples of arithmetic functions used in automatic differentiation AD. Here, it is assumed that n variables out of all variables in the program are objects of partial differentiation. At this time, the structure of “number” is represented by the following vector.

  Here, v is a place where a function value is held. Dk (k = 1 to n) is a place for holding a value when the function is partially differentiated by the k-th variable. In automatic differentiation, a number having the above structure is used as an AD number represented by the above formula (1), and various calculations are performed based on the AD number. The detailed process of various operations is shown in Table 1. By using the transition function of the differential function as shown here, the calculation of the value and the calculation of the partial differential function value are performed in the process of all the function calculations. Automatic differentiation can be performed at the same time.

  The reason for introducing such a mechanism is that the parameter adjustment of the recognition dictionary by the learning process can be flexibly configured according to the purpose and situation. For example, regarding the road surface μ estimation, a plurality of images such as an image with a polarization filter and an image without a polarization filter, such as the first image information P1a and P2b and the second image information P1b and P2b, are converted into the original signals S1, Treat as S2, ... For each image used for learning, what kind of situation it is, for example, whether it is a wet road surface, an icy road surface, a snow surface, or dirt on the road surface that is misleading to be wet, and the situation The measured value of the road surface μ value at is attached as correct answer information. Based on this, simple comparison of pixels between images, comparison of coefficients after applying a filter such as Fourier transform, weighting for various image operations based on parameters written in the recognition dictionary, and comparison of numerical values of the operation results In the learning process, the road surface condition and the road surface μ value are finally calculated, and the parameters are finely adjusted so as to be as close as possible to the correct answer information with respect to the learning data.

  In the learning process, when the road surface μ data 24 is updated, the difference between the automatically calculated determination result and the correct answer information is evaluated, and the gap is adjusted to be as small as possible. The process of automatically calculating the discrimination result and the process of calculating the gap between the correct answer information can all be written as a program including conditional branches such as an if statement and arithmetic operations. In that case, the automatic differentiation mechanism that can process the value calculation and the partial differential function value calculation at the same time is useful. When the automatic differentiation mechanism is used, the target numerical value (for example, the estimated value of the road surface μ) is obtained by the calculation process written in the program. At the same time, the partial differential function value for the parameter used in the calculation is obtained simultaneously. . Therefore, if there is a large error between the calculated value (estimated value of the road surface μ) and the correct answer information, for each parameter used in the calculation, negative feedback proportional to the partial differential function value corresponding to the parameter is given. By giving, the fine adjustment amount of the parameter can be calculated so as to reduce the gap with the correct answer information. As a result, it is possible to update and learn the parameters included in the road surface μ data 24.

  The method of finely adjusting the parameters as described above is called a gradient method. In the gradient method, the partial differential function value of the target function is required. In the road surface μ estimation, an image without a polarization filter and an image with a polarization filter and road surface correct information for the image are input, and the road surface for the input image is referred to with reference to the parameters described in the recognition dictionary (road surface μ data 24). The objective function is to calculate the μ value and minimize the error between it and the correct answer. This goal function is often changed by learning situations. For example, when other filters (infrared rays, ultraviolet filters, etc.) are used in addition to the polarizing filter, the number of input images increases to three or more. Even if such a condition change is made, if automatic differentiation is used, the calculation value and parameters of the function can be obtained simply by writing the objective function as a program, so that learning by the gradient method can be easily performed.

  The embodiment of the present invention has been described in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and there are design changes and the like without departing from the gist of the present invention. They are also included in the present invention.

  For example, in the above-described embodiment, the case where a lens is used as the first optical member has been described. However, other optical members including a mirror, a prism, and a beam splitter may be used as the first optical member.

1 Camera 1L Left camera (camera)
1R Right camera (camera)
2, 2B Image processing unit 11 First optical member 11a First lens 11b Second lens 12 Second optical member 13 Imaging element 13a First imaging region 13b Second imaging region 24 Road surface μ data 100, 100B External recognition device A1 Polarization region A2 Unchanged region FL, FR Optical filters P1, P2 Image information P1a, P2a First image information P1b, P2b Second image information L, L1, L2 Optical RS Road surface

Claims (11)

An external environment recognition apparatus including a camera mounted on a vehicle to acquire image information of a shooting target region in front of the vehicle, and an image processing unit that processes the image information,
The camera includes an imaging element having first and second imaging areas, a first optical member that forms an image of the imaging target area in each of the first and second imaging areas, and the first And a second optical member that transmits light of different optical conditions for each of the second imaging regions,
The image processing unit compares the first image information obtained from the first imaging area with the second image information obtained from the second imaging area, and recognizes the state of the imaging target area. An external environment recognition device characterized by: The camera includes a polarization region in which the first and second optical members are disposed on an optical path, and a non-polarization region in which the first optical member is disposed on an optical path,
In the first imaging region, the image is imaged through the polarization region,
The external environment recognition device according to claim 1, wherein the image is formed on the second imaging region through the non-polarized region.   The image processing unit includes road surface μ data in which a relationship between a friction coefficient of a road surface and the first and second image information is recorded, and the road surface μ data is obtained based on the first and second image information. 3. The external environment recognition apparatus according to claim 2, wherein a friction coefficient of a road surface is recognized as a state of the imaging target region with reference to the external environment recognition device. Arrange the two cameras as left camera and right camera,
The said image processing part performs the parallax process of the said 1st image information obtained by the said left camera, and the said 1st image information obtained by the said right camera, The Claim 2 or Claim characterized by the above-mentioned. 3. An external recognition apparatus according to 3.   The external environment recognition device according to claim 4, wherein an optical filter is disposed on an optical path of the non-polarized region of at least one of the left and right cameras.   The external environment recognition apparatus according to claim 5, wherein the optical filters of the left and right cameras have different optical conditions. The polarization region is provided corresponding to the right side of the shooting target region of the left camera and the left side of the shooting target region of the right camera,
5. The external environment recognition device according to claim 4, wherein the non-polarized region is provided corresponding to a left side of the shooting target region of the left camera and a right side of the shooting target region of the right camera. The polarization region is provided corresponding to the lower side of the shooting target region of the left and right cameras,
The external environment recognition device according to claim 4, wherein the non-polarized region is provided corresponding to an upper side of the shooting target region of the left and right cameras. The polarization region is provided corresponding to the upper side of the shooting target region of the left and right cameras,
The external environment recognition device according to claim 4, wherein the non-polarization region is provided corresponding to a lower side of the photographing target region of the left and right cameras.   The external environment recognition device according to claim 4, wherein an area of the polarization region is larger than an area of the non-polarization region. The first optical member includes first and second lenses,
The first lens and the second optical member are disposed on an optical path of the polarization region;
The external environment recognition device according to any one of claims 2 to 10, wherein the first lens and the second lens are arranged on an optical path of the non-polarization region.

Images