JP5254102B2 - Environment recognition device - Google Patents

Description

  The present invention relates to an environment recognition apparatus, and more particularly to an environment recognition apparatus that detects a light source existing on a road.

  2. Description of the Related Art Conventionally, there has been known a device that mounts a CCD camera, a CMOS camera, or the like on a host vehicle and detects a signal lamp or the like in which a traffic light is lit from a captured image captured by the camera.

  For example, in the apparatus described in Patent Document 1, a red light emitting unit is extracted from a captured image captured by a camera, the luminance and size of the extracted red light emitting unit on the captured image, and the high on the captured image. Then, based on the shape, ellipticity, etc., it is determined whether or not the red light emitter of the traffic light is in a light emission state, that is, whether or not it is a red signal.

  In addition, in the apparatus described in Patent Document 1, when a plurality of red signals are extracted, the size of the red light emitting unit on the captured image is larger and the height on the captured image is higher. It has been proposed to recognize each red signal in the order of proximity to the host vehicle.

JP 2007-34693 A

  However, in such a conventional red signal recognition method, since the light emitting unit is recognized based only on the luminance, size, and height on the captured image, for example, as shown in FIG. When the traffic lights A, B, and C are captured in the image T with approximately the same size, there is a high possibility that the traffic signal A captured near the center of the captured image T will be detected.

  However, the traffic light A is a traffic light installed on the side road instead of on the traveling path of the host vehicle. When the traffic signal B on the traveling path of the host vehicle is a green signal, the traffic signal A is often a red signal. In such a case, if the traffic light A on the side road is mistaken as a traffic light on the traveling path of the host vehicle and recognized as a red signal, for example, when brake control of the host vehicle is performed based on the detection result, Although the traffic light B on the traveling path of the host vehicle is a green light, the vehicle is automatically braked in front of the traffic light B, and there is a risk that it will be collided with the following vehicle.

  Thus, especially when detecting an object that is not allowed to be erroneously recognized, such as a signal light with a traffic light on, the method of recognizing based only on the brightness, size, height, etc. on the captured image Reliability is not necessarily high.

  Also, when detecting traffic lights that are far away from the host vehicle, it is difficult to distinguish whether it is a red signal or a red display such as an advertising tower or neon sign by the above method. In some cases, a red display such as the above is erroneously recognized as a red signal.

  There are many light sources on the road that can cause serious accidents if they are misrecognized, such as tail lights, brake lights, turn signals, etc. It is desired to develop an environment recognition device that can reliably detect and accurately recognize the light source.

  The present invention has been made in view of such circumstances, and it is possible to reliably detect and accurately recognize a light source such as a signal lamp that is turned on in a traveling path of the host vehicle from a captured image. An object is to provide a possible environment recognition device.

In order to solve the above problem, the first invention provides:
In the environment recognition device,
Imaging means for imaging an image having image data for each pixel;
Distance detecting means for detecting a distance in real space for each pixel of the image;
Road surface detection means for detecting a road surface from the image;
Integration processing means for integrating adjacent pixels into one group based on image data of adjacent pixels in the image;
Distance calculating means for calculating a distance in the real space of the group based on at least one of the distance in the real space corresponding to the pixels belonging to the group and the pixels existing around the group;
Based on the distance of the group in the real space and the coordinates of the group in the image, the size of the group in the real space, the height in the real space from the road surface, and the host vehicle of the group A light source detection means for calculating a position in real space with respect to the traveling path of the light source, and detecting the group as a light source existing on the road when the size, height, and position respectively satisfy predetermined conditions;
It is characterized by providing.

The second invention is the environment recognition device of the first invention,
Lane detection means for detecting a lane on the side of the host vehicle from the image,
The light source detection unit calculates a traveling path of the host vehicle and a position of the group in the real space relative to the traveling path based on the position of the lane in the real space.

  According to a third aspect of the present invention, in the environment recognition device according to the first or second aspect, the integrated processing unit detects a pixel having the image data that satisfies a preset first condition, and When detecting a pixel having the image data that satisfies the second condition set in a range wider than the range of the image data defined by the condition, the pixel is integrated with the pixel having the image data that satisfies the first condition Thus, one group is used.

A fourth invention is the environment recognition device of the third invention, wherein
The image data is color image data,
The first condition is set so that the pixel having a specific color and a specific brightness is detected from the image,
The second condition is set so that the pixel having the specific color but whose brightness is darker than the specific brightness defined by the first condition is detected. And

  According to a fifth aspect of the present invention, in the environment recognition device according to the fourth aspect, the second condition has the specific color, and the brightness is higher than the specific brightness defined by the first condition. The bright pixels are set so as to be detected.

  According to a sixth aspect of the present invention, in the environment recognition apparatus according to any one of the first to fifth aspects of the invention, the integration processing unit includes a difference in distance between the pixels adjacent to the pixel in the real space, and the real of the pixel. If any of the differences between the distance in space and the distance in the real space of the group exceeds a predetermined threshold, the pixels are not integrated into the group.

  According to a seventh invention, in the environment recognition device according to any one of the first to sixth inventions, when the integration processing means integrates the pixels into the group, the shape of the group is not circular. Is characterized in that the pixels are not integrated into the group.

  According to an eighth aspect of the present invention, in the environment recognition apparatus according to any one of the first to sixth aspects, when the integration processing unit integrates the pixels into the group, the shape of the group is not circular. Is characterized by rejecting the group and excluding it from the object of processing in the light source detection means.

  According to a ninth aspect of the present invention, in the environment recognition device according to any one of the first to eighth aspects, the distance calculating means includes the distance in the real space corresponding to the pixels belonging to the group, and the image of the group. The distance in the real space is detected by searching left and right or up and down from the above range, and the distance in the real space corresponding to the pixel existing at the position closest to the range on the image is detected. A distance in the real space of the group is calculated based on at least one of them.

  A tenth aspect of the invention is the environment recognition device according to the ninth aspect of the invention, in which the distance calculation means is a distance in the real space corresponding to each pixel belonging to the group and / or on the image of the group. Calculating a mean value or a median value of the distances in the real space corresponding to the pixels present closest to the right and left or up and down of the range of the range as the distance in the real space of each group, To do.

  An eleventh invention is the environment recognition device according to the ninth invention, wherein the distance calculation means is a distance in the real space corresponding to each pixel belonging to the group and / or on the image of the group. The distance in the real space corresponding to the pixel existing closest to the left or right or up and down of the range is voted on the histogram for each group, and the peak value is set as the distance in the real space of each group. It is characterized by calculating.

  A twelfth aspect of the invention is the environment recognition device according to the ninth aspect of the invention, wherein the distance calculation means is a distance in the real space corresponding to each pixel belonging to the group and / or on the image of the group. Of the distances in the real space corresponding to the pixels present closest to the left and right or up and down of the range of The average value or median value of the distances in the real space belonging to the group with the largest number is calculated as the distance in the real space of each group.

  A thirteenth aspect of the present invention is the environment recognition apparatus according to any one of the first to twelfth aspects of the present invention, wherein the light source detection means is any one of a signal light with a traffic light lit, a tail lamp of a preceding vehicle, a brake lamp, and a blinker as the light source. It is characterized by detecting.

  According to a fourteenth aspect of the present invention, in the environment recognition apparatus according to any one of the first to thirteenth aspects, the light source detection unit detects a signal light with a traffic light turned on as the light source, and detects the detected signal on the image of the group. When the object does not exist within a predetermined distance below the position, a signal lamp with which the traffic light is lit is detected.

  According to the first invention, based on the distance in the real space of each pixel belonging to each group, the size in the real space of each group, the height in the real space from the road surface, and the road surface of the group Since the position in real space with respect to the traveling path of the host vehicle is determined, whether or not the detected light source is a signal light (for example, a red signal) with a traffic light turned on is determined accurately. Can be recognized. As a result, the light source can be detected accurately and reliably, and the recognition reliability can be greatly improved.

  Even when the light source is at a distance from the host vehicle, it is based on the size in the real space, the height in the real space from the road surface, and the position in the real space with respect to the traveling path of the host vehicle on the road surface. Since the light source is determined, it is possible to accurately prevent, for example, erroneous recognition of a red display such as an advertising tower beside a road or a neon sign as a red signal.

  According to the second invention, it is possible to accurately calculate the traveling path of the host vehicle, the position of the group in real space, and the road surface based on the information on the lane on the side of the host vehicle detected from the image. Thus, the effects of the present invention are more accurately and reliably exhibited.

  According to the third aspect of the invention, on the image, the light source has a feature that the center is bright and the image is darkened as the distance from the center increases. Since all bright portions such as the sky that have been imaged are detected, the light source to be detected cannot be detected efficiently.

  Therefore, in addition to the effects of the respective inventions, the first condition for the image data of the pixel is defined so as to be detected by narrowing down to pixels having the image data of the peripheral portion instead of the central bright portion, and the pixel By detecting the pixels by setting the second condition in the search for the surrounding pixels of the pixel to a range wider than the range of the image data defined by the first condition, it becomes possible to detect the light source efficiently. .

  According to the fourth invention, on the color image, the light source has an image of a white and bright part in the center, an image of a bright part having a color specific to the light source in the periphery, and a color specific to the light source in the periphery. However, if it is configured to detect a white whitish bright part at the center, all whitish bright parts such as the sky captured in the image will be detected. The light source to be detected cannot be detected efficiently.

  Therefore, in addition to the effects of the above-described inventions, the first condition for the image data of the pixel is that the pixel having a specific color and a specific brightness specific to the light source in the peripheral portion, not in the central whitish bright portion The second condition in the subsequent search for pixels around the pixel is wider than the range of image data defined by the first condition, and has a specific color. Focus on colors specific to the light source, including the dark part of the edge that is specific to the light source, by detecting the pixels with the brightness set to a range darker than the specific brightness specified in the first condition Thus, the light source can be detected efficiently.

  According to the fifth invention, the second condition is such that a pixel having a specific color specific to the light source and having a brightness higher than the specific brightness defined by the first condition is also detected. By setting, it becomes possible to efficiently detect the light source by focusing on the color peculiar to the light source including the central whitish bright part peculiar to the light source, and the effects of each of the above inventions can be exhibited more accurately. Is possible.

  According to the sixth aspect of the present invention, when the difference in the real space between the pixel and the adjacent pixel or the difference in the real space between the groups is more than a predetermined threshold value, Since adjacent pixels and groups are considered to be pixels corresponding to different imaging targets, in such a case, pixels corresponding to different imaging targets can be configured by making the pixels non-integrated into groups. Are integrated into one group, and different imaging targets captured on the image can be divided into different groups and detected, and the effects of the above-described inventions can be more accurately exhibited. It becomes possible.

  According to the seventh invention, when detecting a light source such as a signal lamp with a traffic light turned on, the shape of the entire group detected by entering light from another light source into a pixel area where the light source is imaged on the image is a circle. It may lose its shape. In such a case, during the detection, when the entire shape of the group on the image is no longer circular, the pixel is not integrated into the group and the integration process is terminated, so that the light enters from another light source. It is possible to accurately exclude only the light portion and extract only the light emitted from the target light source, and to exhibit the effects of the inventions more accurately.

  According to the eighth invention, in the case of a light source that does not have an environment where light from another light source enters, it is found that the entire shape of the group on the image is not circular during detection. In some cases, it is possible to improve the efficiency of the process by disposing the integration process and rejecting the group. Therefore, in such a case, it becomes possible to efficiently detect the target light source by rejecting the group and excluding it from the processing target in the subsequent distance calculation means and light source detection means. The effect can be exhibited more accurately.

  According to the ninth invention, when calculating the distance in the real space of the group itself detected in an integrated manner, if the distance in the real space is detected by the distance detection means for each pixel belonging to the group, The distance in the real space of the group can be calculated using, but the distance in the real space is not always detected for these pixels.

  Therefore, in such a case, a pixel whose distance in the real space is detected vertically and horizontally beyond the pixel range occupied by the group on the image is searched, and the pixel closest to the group among those pixels is searched. The distance in the real space of the group can be calculated using the distance in the real space. And it becomes possible to calculate the distance in the real space of a group exactly by comprising in that way, and it becomes possible to exhibit the effect of each said invention more exactly.

  According to the tenth invention, when calculating the distance in the real space of the group as described above, the average value of the distance in the real space of each pixel belonging to the group or the pixel closest to the group as described above, By calculating the median and calculating the distance of each group in the real space, it becomes possible to easily and accurately calculate the distance of each group in the real space, and the effects of the above-described inventions can be demonstrated more accurately. It becomes possible to make it.

  According to the eleventh aspect, when calculating the distance in the real space of the group as described above, the data of the distance in the real space of each pixel belonging to the group or the pixel closest to the group as described above is obtained. By voting on each histogram created for each group and calculating the peak value as the distance in the real space of each group, for example, other pixels due to false detection or the like in the distance data of each pixel in the real space Even if there is distance data in the real space that is far away from the distance in the real space, the distance in the real space that is far away is not dragged by the data in the real space. Thus, the effects of the respective inventions can be exhibited more accurately.

  According to the twelfth invention, when calculating the distance in the real space of the group as described above, as the data of the distance in the real space of each pixel belonging to the group or the pixel closest to the group as described above, When distance data is obtained so that close values can be grouped together, the average and median distances in real space belonging to the group with the largest number of data among the group of distances collected By calculating the distance in the real space of the group, the distance in the real space of each group can be accurately calculated, and the effects of the respective inventions can be more accurately exhibited.

  According to the thirteenth invention, even when the light source is a signal light with a traffic light turned on, a tail lamp of a preceding vehicle, a brake lamp, a blinker, etc., the effect of each of the above inventions is accurately exhibited, and at least in the captured image. Therefore, it is possible to reliably detect and accurately recognize a target light source such as a signal light with a traffic light turned on.

  According to the fourteenth invention, in addition to the effects of the above inventions, in the case of detecting a signal light with a traffic light lit as a light source, there is no object within a predetermined distance below the signal light of the traffic light in real space, Since a space should exist, if the group detected by integration meets other conditions, and there is a space part where no object exists within a predetermined distance below it, the group is identified by a signal light with a traffic light. It becomes possible to accurately recognize that there is, and it is possible to further improve the reliability of detection and recognition when detecting a signal light with a traffic light turned on, particularly as a light source.

It is a block diagram which shows the structure of the environment recognition apparatus which concerns on 1st Embodiment. It is a figure which shows the example of the reference | standard image imaged with an imaging means. It is a figure explaining the stereo matching process by the image processor of a distance detection means. It is a figure which shows the example of the produced distance image. It is a figure explaining the example of the lane candidate point detected by searching on the horizontal line of a reference image. It is a figure explaining the example of the lane detected by the own vehicle side. It is a figure explaining the example of the formed lane model, (A) represents the horizontal shape model on a ZX plane, (B) represents the road height model on a ZY plane. It is a figure which shows the example of the light source imaged on the reference | standard image. It is a figure which shows the pixel and group which satisfy | fill the 2nd condition detected by searching in the up-down direction of the detected pixel. FIG. 10 is a diagram illustrating a method for detecting a group in a pixel column on the right side of the pixel column in FIG. 9. It is a figure explaining the change of the length of the vertical direction of a group, when (A) becomes the maximum value, (B) is shorter than a maximum value, (C) represents the case where it is still shorter than (B). . It is a figure explaining the case where the length of the vertical direction of a group has increased from the minimum value. It is a figure showing each pixel which adjoined the upper and lower sides of the range of the group, and the distance in real space is detected. It is a figure explaining each pixel of the right-and-left end of a group, and each pixel of an upper-lower end. It is a figure which shows the example of the reference | standard image by which the some signal apparatus containing the signal apparatus on the advancing path of the own vehicle was imaged. It is a figure explaining the traffic signal detected based on the advancing path of the own vehicle on the reference | standard image of FIG. It is a figure explaining the example which plotted each point on the real space corresponding to each pixel which belongs to a group on the XZ plane. It is a figure explaining the example which plotted on the YZ plane each point on the real space corresponding to each pixel which belongs to a group.

  Hereinafter, an embodiment of an environment recognition apparatus according to the present invention will be described with reference to the drawings.

  As shown in FIG. 1, the environment recognition apparatus 1 according to the present embodiment includes processing including an imaging unit 2, a distance detection unit 6, an integration processing unit 10, a distance calculation unit 11, a light source detection unit 12, a road surface detection unit 14, and the like. The unit 9 is provided.

  The upstream configuration of the processing unit 9 including the distance detecting means 6 and the like is described in detail in Japanese Patent Application Laid-Open No. 2006-72495 previously filed by the applicant of the present application. Leave them to the gazette. A brief description is given below.

  In the present embodiment, the image pickup means 2 includes a built-in image sensor such as a CCD or a CMOS sensor that is synchronized with each other. For example, a pair of image sensors 2 attached in the vicinity of a vehicle rearview mirror with a predetermined interval in the vehicle width direction. The stereo camera is composed of a main camera 2a and a sub camera 2b, and is configured to capture a predetermined sampling period and output a pair of images.

  Of the pair of cameras, the main camera 2a is a camera on the side close to the driver, and takes an image T as shown in FIG. 2, for example. In order to distinguish from the image captured by the sub camera 2b, hereinafter, the image T captured by the main camera 2a is referred to as a reference image T, and the image captured by the sub camera 2b is referred to as a comparative image Tc.

In the present embodiment, the main camera 2a and the sub camera 2b of the image pickup unit 2 acquire color image data D represented by RGB values or the like, but pick up monochrome image data. It is also possible to use an imaging means that performs this, and the present invention is also applied to such a case. Hereinafter, the case where the color image data D is output as RGB values, that is, data values represented by R (red), G (green), and B (blue) color components in the RGB color system. As will be described, the RGB value can be converted into each data value described in, for example, the L ^* a ^* b ^* color system, HLS (also referred to as HSL, HSI), or the like. is there.

  The conversion unit 3 includes a pair of A / D converters 3a and 3b, and each image data of the reference image T and the comparison image Tc captured for each pixel by the main camera 2a and the sub camera 2b of the imaging unit 2. When D is sequentially transmitted, the RGB value of each image data D is converted into a digital value having a luminance value of 0 to 255, for example, and output to the image correction unit 4.

  The image correction unit 4 sequentially performs image corrections such as displacement, noise removal, luminance correction, and the like on the transmitted reference image T and comparison image Tc for each image data D, and the corrected image of the reference image T Each image data D of the comparison image Tc is sequentially stored in the image data memory 5 and is sequentially transmitted to the processing unit 9. Further, the image correction unit 4 sequentially transmits the image data D of the reference image T and the comparison image Tc subjected to the image correction to the distance detection unit 6.

  In the image processor 7 of the distance detection means 6, stereo matching processing and filtering processing are sequentially performed on the image data of the reference image T and the comparison image Tc, and the parallax dp corresponding to the distance in the real space is obtained from the reference image T. Calculation is sequentially performed for each pixel.

In the stereo matching process in the image processor 7, when each image data D of the reference image T and the comparison image Tc is transmitted, for example, 3 × 3 pixels or 4 × 4 pixels on the reference image T as shown in FIG. For each comparison pixel block PBc having the same shape as the reference pixel block PB on the epipolar line EPL in the comparison image Tc corresponding to the reference pixel block PB, a reference pixel block PB having a predetermined number of pixels is set. An SAD value, which is a difference in luminance pattern from the reference pixel block PB, is calculated according to the equation, and a comparison pixel block PBc having the smallest SAD value is specified.
SAD = Σ | D1s, t−D2s, t |
= Σ {| R1s, t-R2s, t | + | G1s, t-G2s, t |
+ | B1s, t-B2s, t |} (1)

  In the above equation (1), D1s, t, etc. represent the image data D, etc. of each pixel in the reference pixel block PB, and D2s, t, etc., represent the image data D, etc., of each pixel in the comparison pixel block PBc. . Further, the above sum is obtained when the reference pixel block PB and the comparison pixel block PBc are set as a 3 × 3 pixel region, for example, a range of 1 ≦ s ≦ 3, 1 ≦ t ≦ 3, and 4 × 4 pixels. When set as a region, calculation is performed for all pixels in the range of 1 ≦ s ≦ 4 and 1 ≦ t ≦ 4.

  For each reference pixel block PB of the reference image T in this way, the image processor 7 calculates the parallax dp from the position on the comparison image Tc of the identified comparison pixel block PBc and the position on the reference image T of the reference pixel block PB. It is calculated sequentially. Hereinafter, an image in which the parallax dp is assigned to each pixel of the reference image T is referred to as a distance image Tz. In addition, information on the parallax dp calculated for each pixel in this way, that is, the distance image Tz is sequentially stored in the distance data memory 8 of the distance detecting unit 6 and is sequentially transmitted to the processing unit 9. ing.

In real space, a point on the road surface directly below the center of the pair of cameras 2a and 2b is used as an origin, and the vehicle width direction (that is, the horizontal direction) of the host vehicle is the X-axis direction and the vehicle height direction (that is, the height). (Direction) is the Y-axis direction and the vehicle length direction (that is, the distance direction) is the Z-axis direction, the point (X, Y, Z) in real space and the coordinates (i, j) of the pixel on the distance image Tz The parallax dp and the parallax dp can be uniquely associated by coordinate conversion based on the principle of triangulation expressed by the following equations (2) to (4).
X = CD / 2 + Z * PW * (i-IV) (2)
Y = CH + Z × PW × (j−JV) (3)
Z = CD / (PW × (dp−DP)) (4)

  In the above equations, CD is the distance between the pair of cameras, PW is the viewing angle per pixel, CH is the mounting height of the pair of cameras, and IV and JV are i on the distance image Tz at the infinity point in front of the host vehicle. The coordinates, j-coordinate, and DP represent vanishing point parallax.

  Further, for the purpose of improving the reliability of the parallax dp, the image processor 7 performs a filtering process on the parallax dp obtained by the stereo matching process as described above, and outputs only the valid parallax dp. It has become.

  For example, even if a stereo matching process is performed on the reference image block PB of 4 × 4 pixels having poor features consisting only of a road surface image, the road surface of the comparison image Tc is captured. Since all the correlations are high, the reliability of the parallax dp is low even if the corresponding comparison pixel block PBc is specified and the parallax dp is calculated. Therefore, the parallax dp is invalidated by performing a filtering process, and 0 is output as the value of the parallax dp.

  Therefore, when the distance image Tz is created by assigning (that is, associating) effectively calculated parallax dp to each pixel of the reference image T, the distance image Tz is, for example, on the reference image T as shown in FIG. Thus, an image in which the effective parallax dp is calculated for the edge portion (edge portion) of the imaging target, which is a portion having a significant feature, is obtained. In creating the distance image Tz, the parallax dp can be converted into the distance Z or the like in advance according to the above equation (4) and the like, and the distance Z or the like can be assigned to each pixel of the reference image T. It is.

  In the present embodiment, the parallax dp in each pixel of the distance image Tz is obtained according to the above equation (4) or the like in each process in the integration processing unit 10, the distance calculation unit 11, the matching processing unit 12 and the like of the processing unit 9, as necessary. It is used after being converted into a distance Z in real space.

  In the present embodiment, as described above, the imaging unit 2 includes the main camera 2a and the sub camera 2b, and the distance detection unit 6 performs stereo matching processing on the reference image T and the comparison image Tc captured by them. The distance Z (that is, the parallax dp) in the real space is calculated for each pixel of the reference image T. However, the present invention is not limited to this, and the imaging unit 2 is a single image such as a monocular camera. Only T may be output.

  Further, the distance detection means 6 only needs to have a function of calculating or measuring the distance Z in the real space and assigning it to each pixel of the image T. For example, the distance detection means 6 irradiates the front of the host vehicle with a laser beam or the like and reflects it. A radar apparatus that measures the distance Z to the imaging target imaged on the image T based on the light information may be used, and the detection method is not limited to a specific method.

  In this embodiment, the processing unit 9 is configured by a computer in which a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), an input / output interface, and the like (not shown) are connected to a bus. The processing unit 9 includes an integrated processing unit 10, a distance calculation unit 11, a light source detection unit 12, a lane detection unit 13, and a road surface detection unit 14.

  Sensors Q such as a vehicle speed sensor, a yaw rate sensor, and a steering angle sensor for measuring the steering angle of the steering wheel are connected to the processing unit 9, and each measurement value is input from them. Note that the processing unit 9 may be configured to perform other processing such as detection of a preceding vehicle.

  Here, before describing the processing in the integrated processing means 10, the distance calculation means 11, the light source detection means 12, and the like of the processing unit 9 of the present embodiment, the processing in the lane detection means 13 and the road surface detection means 14 will be described.

  In the following description, the lane refers to a continuous line or a broken line marked on a road surface such as an overtaking prohibition line or a lane marking that divides the roadside zone and the roadway. In the present embodiment, as described below, the lane detection unit 13 detects the lane marked on the road surface, and the road surface detection unit 14 detects the road surface based on the detection result. However, the road surface detection means 14 is not limited to the form described below as long as it can detect the road surface.

  The lane detection unit 13 detects a lane on the side of the host vehicle from the reference image T captured by the imaging unit 2. Specifically, as shown in FIG. 5, the lane detection means 13 searches the horizontal line j having a width of one pixel using the reference image T, for example, from the center of the reference image T in the left-right direction, and the luminance is Pixels that greatly change from the brightness of adjacent pixels to a set threshold value or more are detected as lane candidate points cl and cr.

  Then, while shifting the horizontal line j on the reference image T upward by one pixel, lane candidate points are detected on each horizontal line j in the same manner. At this time, if it is determined that the lane candidate point is not on the road surface based on the parallax dp of the detected lane candidate point, the lane detection unit 13 excludes the lane candidate point from the lane candidate point. The road surface in this case is estimated based on the road surface detected by the road surface detection means 14 in the previous sampling cycle from the subsequent behavior of the own vehicle.

  The lane detecting means 13 detects the lanes by approximating them in a straight line by Hough transform or the like based on the lane candidate points on the side close to the own vehicle among the remaining lane candidate points. At that time, various straight lines are calculated as candidates in the Hough transform. For example, when a plurality of lanes are detected on one side (for example, the right side) of the own vehicle, the line is detected on the other side (for example, the left side) of the own vehicle. A straight line is selected on each side of the host vehicle, for example, by selecting a lane that is consistent with the selected lane or a lane that is consistent with the lane detected in the previous sampling cycle.

  In this way, when the lane detection means 13 detects the lanes in a straight line on the side closer to the own vehicle, the lane detection means 13 selects the lane candidate points from the positional relationship with the straight lines based on the straight lines on the far side. As shown in FIG. 6, the lanes LL and LR are detected on the side of the host vehicle. The processing configuration of the lane detection means 13 is described in detail in Japanese Patent Application Laid-Open No. 2006-331389 previously filed by the applicant of the present application, and the detailed description should be referred to the same publication.

  The lane detecting means 13 stores information such as the lane positions LL and LR and lane candidate points cl and cr detected in this manner in a memory (not shown).

  The road surface detection means 14 forms a lane model three-dimensionally based on the information on the lane positions LL, LR and lane candidate points detected by the lane detection means 13. In the present embodiment, as shown in FIGS. 7A and 7B, the road surface detection means 14 approximates each lane on the side of the host vehicle with a three-dimensional linear expression for each predetermined section, and forms them in a polygonal line shape. A lane model expressed by connecting to the lane is formed. 7A shows a lane model on the Z-X plane, that is, a horizontal shape model, and FIG. 7B shows a lane model on the Z-Y plane, that is, a road height model.

  Specifically, the road surface detection means 14 divides the real space ahead of the host vehicle into sections such as a distance Z7 from the position of the host vehicle, and detects the positions (X, Y, Z), the lane candidate points in each section are linearly approximated by the least square method, and parameters aL, bL, aR, bR, cL, dL in the following formulas (5) to (8) for each section: A lane model is formed by calculating cR and dR.

[Horizontal shape model]
Left lane X = aL ・ Z + bL (5)
Right lane X = aR ・ Z + bR (6)
[Road height model]
Left lane Y = cL · Z + dL (7)
Right lane Y = cR · Z + dR (8)

  The road surface detection means 14 forms a lane model in this way and detects a road surface in real space. Further, the road surface detection means 14 stores the lane model formed in this way, that is, the calculated parameters aL to dR of each section in the memory.

  Next, processing in the integrated processing unit 10, the distance calculation unit 11, the light source detection unit 12, and the like will be described, and the operation of the environment recognition apparatus 1 according to the present embodiment will also be described.

  In the following, a case will be described in which a signal lamp that is lit red (that is, a so-called red signal) is detected. In the present embodiment, the reference image T captured by the main camera 2a of the imaging unit 2 is used as an object to be processed, but the comparison image captured by the sub camera 2b of the imaging unit 2 is used. It is possible to configure to use Tc, or to use both of them.

  The integration processing unit 10 determines whether or not to integrate the adjacent pixels p into one group based on the image data D of the adjacent pixels p for the adjacent pixels p in the reference image T. When it is determined that they should be integrated, they are integrated into one group g.

  In the present embodiment, the integration processing unit 10 searches each pixel p in the reference image T, and includes a pixel p (hereinafter referred to as a pixel of interest) having image data D that satisfies a preset first condition for the image data D. When p (i, j) is detected), each pixel p adjacent to the periphery of the target pixel p (i, j) is searched, and a range wider than the range of the image data D defined by the first condition. When the pixel p having the image data D satisfying the second condition set in is detected, the pixel p is integrated with the target pixel p (i, j) to form one group g. Further, each pixel p adjacent to the periphery of the group g is searched, and when the pixel p having the image data D satisfying the second condition is detected, it is integrated into the group g.

  Specifically, the integration processing unit 10 receives the image data D of each pixel p of the reference image T from the main camera 2a of the image pickup unit 2 via the image correction unit 4 or the like, and the integration processing unit 10 Each pixel p is searched, and a pixel p having image data D that satisfies the following first preset condition for image data D is searched.

  The first condition is set so that a target pixel p (i, j) having a specific color (red in the present embodiment) and specific brightness is detected from the reference image T. In the embodiment, color components of R (red), G (green), and B (blue) represented by luminance values of 0 to 255 of the image data D of the target pixel p (i, j) are as follows. Conditions are set.

[First Condition] R ≧ 200 and G, B <200 (9)
That is, in the present embodiment, the first condition is set such that a bright pixel p having a reddish color from the reference image T and suitable for the light source is detected as the target pixel p (i, j). .

  When a light source such as a red light of a traffic light is picked up by the image pickup means 2, the center of the light source S is shown in the reference image T or the like as shown in FIG. The portion Sc is picked up in reddish white, and the luminance values of the R, G, and B color components of the image data D of the pixel p of that portion are all detected as large values as exceeding 200. This is not a phenomenon limited to a red signal. For example, even in a light source that emits blue light such as a blue signal of a traffic light, the central portion of the light source is imaged in a bluish white on the reference image T and the comparison image Tc.

In such a case, for the purpose of detecting the central portion Sc of the light source S as the first condition, the first condition is set to, for example, the second condition [Condition 2b] described later,
R, G, B> 200 and R ≧ G, R ≧ B (10)
If all the objects captured in the reference image T, such as the sky captured in white and bright, are detected, the light source S that emits red light such as a red signal from the traffic light cannot be detected efficiently.

  Therefore, in the present embodiment, the pixel p of the pixel area S1 that is imaged in red and brightly around it is detected first, not the pixel p imaged white like the central portion Sc of the light source S. By configuring and narrowing down the conditions of the pixel p (attention pixel p (i, j)) that is the beginning of detection, the light source S that emits red light such as a red signal of a traffic light can be detected efficiently. ing.

  When the integration processing unit 10 detects the target pixel p (i, j) having the image data D satisfying the first condition on the reference image T, subsequently, as shown in FIG. 9, the detected target pixel is detected. A pixel p adjacent to the upper side of the target pixel p (i, j) is shifted upward by one pixel on the pixel row pls having a width of one pixel extending in the vertical direction in the reference image T including p (i, j). It is determined whether or not the image data D of (i, j + 1) satisfies the following second condition set in a range wider than the range of the image data D defined by the first condition.

[Second condition]
[Condition 2a] R ≧ 165 and G, B <150 (11)
[Condition 2b] R, G, B> 200 and R ≧ G, R ≧ B (12)
[Condition 2c] R ≧ 200 and G, B <200 (13)

  Here, [Condition 2a] of the second condition has a color specific to the light source (red in the present embodiment), but the brightness is captured so that the brightness is defined by the first condition. This is a condition for detecting the pixel p in the darker pixel region Sd (see FIG. 8) captured around the pixel region Sl, not the pixel p in the pixel region S1.

  In this way, by detecting even the darker pixel region Sd as the light source S, the light source S is accurately detected by accurately grasping the brightness characteristic of the light source that becomes darker as it goes away from the central bright part. It becomes possible.

  In addition, as described above, [Condition 2b] of the second condition has a light source specific color (red in the present embodiment), and the brightness is even brighter than the brightness defined by the first condition. This is a condition for detecting the pixel region Sc at the center of the light source S imaged in white.

  As described above, by detecting up to the brighter pixel region Sc as the light source S, the brightness characteristic of the light source can be accurately obtained such that the bright portion at the center is imaged whitish with the specific color of the light source S as described above. Thus, the light source S can be accurately detected.

  Furthermore, [condition 2c] of the second condition is the same as the first condition described above, and the pixel adjacent to the target pixel p (i, j) is the same as the target pixel p (i, j). This is a condition for detecting a pixel p of the pixel region S1 that is imaged red and bright around the central portion Sc of S.

  When the integration processing unit 10 determines that the image data D of each pixel p (i, j + 1) adjacent to the upper side of the target pixel p (i, j) satisfies the above second condition, the integration p (i, j, j + 1) and the target pixel p (i, j) are integrated into one group g. Subsequently, the integration processing unit 10 similarly determines the pixel p (i, j + 2) adjacent to the upper side of the pixel p (i, j + 1) based on the second condition, and satisfies the second condition. Is determined, the pixel p (i, j + 2) is integrated into the group g.

  The integration processing means 10 performs this operation until a pixel p that does not satisfy any of the second conditions appears, and integrates up to the pixel p immediately before the pixel p that does not satisfy the second condition into the group g. Similarly, the integration process is performed for each pixel p below the pixel of interest p (i, j).

  Then, as shown in FIG. 10, the integration processing unit 10 does not illustrate the coordinates (i, jmax) of the uppermost pixel and the coordinates (i, jmin) of the lowermost pixel of the group g integrated on the pixel column pls. Then, the pixel row is shifted from the pixel row pls to the right and left directions on the reference image T, and the pixel search is performed in the same manner for each pixel row.

  Specifically, the j-coordinate jmid of the middle point of the j-coordinates jmax and jmin of the upper and lower pixels of the group g in the pixel row pls is calculated. Then, as shown in FIG. 10, the pixel column is shifted to, for example, the pixel column pl adjacent to the right, and the pixel at the coordinate (i + 1, jmid) in the pixel column pl is searched in the vertical direction in the same manner as described above. The upper and lower pixels of the group g in the pixel column pl are detected, and their coordinates (i + 1, jmax) and (i + 1, jmin) are stored in the memory.

  The integration processing means 10 repeats the search while further shifting the pixel column pl to the right, and stores the coordinates of the upper and lower pixels of the group g in each pixel column pl in the memory. Similarly, the left direction of the first pixel column pls is searched in the same manner, and the coordinates of the upper and lower pixels of the group g in each pixel column pl are stored in the memory.

  As described above, each pixel p of the reference image T by the distance detection unit 6 (precisely, each pixel p to which a significant value other than 0 is assigned as the value of the parallax dp in the filtering process described above). The parallax dp is calculated and assigned. Therefore, using the parallax dp or the distance Z in the real space calculated from the parallax dp, even if the pixel p satisfies the second condition in the integration process of the pixel p into the group g, When the difference between the distance Zp of the pixel p in the real space and the distance Z of the adjacent pixel or the group g in the real space exceeds a preset threshold, the pixel p is assigned to the group g. It is possible to configure to be non-integrated, that is, not integrated.

  A pixel p having a distance Zp in the real space that is significantly different from the distance Z in the real space of the adjacent pixel or group g is an imaging target corresponding to the adjacent pixel or group g even if the second condition is satisfied. It is considered that the pixel corresponds to a different imaging target. Therefore, by configuring as described above, it is possible to avoid integrating pixels corresponding to different imaging targets into one group g, and to divide different imaging targets imaged on the reference image T into different groups. Can be detected.

  As a method for obtaining the distance Z in the real space of the group g, it is possible to calculate the average value or the median value of the distances Zp in the real space corresponding to the pixels p integrated in the group g. Yes, every time the pixel p is integrated into the group g, the distance Zp in the real space corresponding to the pixel p is voted on a histogram (not shown), and the peak value is expressed as the distance Z in the real space of the group g. It is also possible to do.

  On the other hand, in this embodiment, the integration processing unit 10 determines whether to integrate a pixel p into the group g in the determination of whether or not to integrate a certain pixel p into the group g. If the overall shape is not a circle, the pixel p is not integrated into the group g and the integration process is terminated.

  Specifically, the integration processing unit 10 monitors the number r of pixels from the uppermost pixel to the lowermost pixel of the group g in each pixel column pls, pl (hereinafter referred to as the vertical length r). As shown in FIG. 11A, the vertical length r of the group g in each pixel column pls, pl once becomes the maximum value rmax, and as shown in FIG. The length r in the direction is less than the maximum value rmax, and the downward trend of the length r in the vertical direction of the group g in the pixel column pl continues as shown in FIG. 11C, and the group g can be found in the pixel column pl. If no pixel is found, that is, if no pixel p that satisfies the second condition is found, the search in the right and left pixel columns is stopped at that time.

  In addition, as shown in FIG. 12, when the vertical length r of the group g in the pixel row pl decreases to once reach the minimum value rmin, and then becomes a value larger than the minimum value rmin again, the reference image Since the entire shape of the group g on T is no longer circular, the search for the right and left pixel columns in the pixel column pl is stopped, and the pixel column having the minimum value rmin immediately before it is grouped. Let g be the detected range. This is because the group g corresponding to a light source such as a signal lamp with a traffic light normally does not increase again after the vertical length r decreases.

  Here, when the entire shape of the group g is a circular shape, it does not mean a complete circle. For example, the vertical length r of the group g while moving the pixel row pl in the left-right direction on the reference image T. Means that the vertical length r once once increased tends to decrease and continues to decrease.

  In addition, as described above, the group g corresponding to the light source such as the signal lamp with the traffic light normally does not increase again after the length r in the vertical direction is decreased. As described above, after the vertical length r of the group g in the pixel row pl decreases to once reach the minimum value rmin, the value again becomes larger than the minimum value rmin, and the entire group g on the reference image T becomes larger. When the shape is no longer circular, the group g can be rejected and excluded from the processing target in the distance calculation means 11 and the light source detection means 12 described later.

  When the integration processing unit 10 ends the integration processing for integrating the pixels p into the group g as described above, the integration processing means 10 repeats the integration processing until there is no pixel p that is not searched on the reference image T, and the reference image Each group g is integrated and formed on T.

  As shown in FIG. 8, in each group g integrated on the reference image T using the characteristic that the light source S is imaged so as to have the pixel region Sc in the center, The group g in which the pixel p satisfying the condition [Condition 2b] does not exist in the group g is determined not to be the image of the light source S, the group g is rejected, and the distance calculation unit 11 and the light source described later are rejected. It is also possible to configure such that the detection unit 12 is excluded from processing targets.

  The distance calculation means 11 calculates the distance Zg in real space for each group g integrated by the integration processing means 10.

  As described above, when the integration processing unit 10 determines whether or not to integrate the pixel p into the group g, the distance Z in the real space of the group g is set to each pixel p integrated into the group g. The average value or median value of each distance Zp in the corresponding real space is calculated, or each time the pixel p is integrated into the group g, the distance Zp in the real space corresponding to the pixel p is voted on the histogram. Since the peak value in the case is calculated, the distance calculation means 11 can be configured to utilize these values as the distance Zg in the real space of the group g.

  However, as described above, the integration processing means 10 basically sets the R, G, and B color values of the image data D of each pixel p based on the first condition and the second condition. Accordingly, the possibility of integration into group g is determined and each pixel p is integrated into group g. For this reason, the group g may be formed by only the pixels p to which 0 is assigned as the value of the parallax dp in the filtering process in the distance detection unit 6 described above. In such a case, the distance Zg in the real space of the group g cannot be calculated by the above method.

  Therefore, in the present embodiment, the distance calculation unit 11 not only includes the distance Zp in the real space of each pixel p belonging to the group g but also the actual pixels corresponding to each pixel p existing around the group g on the reference image T. The distance Zg in the real space of each group g is also calculated using the distance Zp in space.

Specifically, as described above, when each group g is integrated by the reference image T in the integration processing unit 10 and the range on the reference image T of each group g is determined, the distance calculation unit 11 For each pixel column pl having a width of 1 pixel belonging to the range of g on the reference image T and extending in the vertical direction, search is performed upward and downward beyond the range of each group g, as shown in FIG. In addition, among the pixels p in which the distance Zp in the real space is detected, each group is also used by using the distance Zp ^* in the real space corresponding to the pixel p ^* existing closest to the range of the group g. The distance Zg in the real space of g is calculated.

As a method for obtaining the distance Zg in the real space of the group g using the distance Zp ^* in the real space corresponding to the pixel p ^* present at the position closest to the range of the group g, each pixel belonging to the group g can be obtained. It is possible to calculate the average value or the median value of the distance Zp in the real space corresponding to p and each distance Zp ^{* in} the real space corresponding to the pixel p ^* existing at a close position.

Further, the pixel p ^* present in the position closest to the range of the group g in the histogram created by the integrated processing means 10 and voted for the distance Zp in the real space corresponding to each pixel p belonging to the group g ^. It is also possible to vote the distance Zp ^* in the real space corresponding to, and make the peak value the distance Zg in the real space of the group g.

Furthermore, as a result of including the distance Zp ^* in the real space corresponding to each pixel p ^* existing around the group g on the reference image T, each pixel p belonging to the group g and each surrounding pixel p ^In some cases, the distributions of the distances Zp and Zp ^{* in} the real space corresponding to ^* are put into a plurality of groups.

In such a case, the distance Zp in the real space corresponding to each pixel p belonging to the group g, or the pixel p ^* present at the closest position in the vertical and horizontal directions of the range on the reference image T of the group g ^. Among the distances Zp ^{* in} the real space corresponding to, the similar distances Zp, Zp ^* are grouped together, and the distance Zp, Zp ^{* in the} real space belonging to the group having the largest number of data among the group of distances collected ^. It is also possible to calculate the average value and the median value of each as the distance Zg in the real space of each group g.

Note that the pixel p ^* itself present at the position closest to the range of the group g is not integrated into the group g. Further, even if the pixel p ^* is closest to the group g, when the pixel p ^* is too far from the group g on the reference image T, it is in the real space corresponding to the pixel p ^* present at the closest position. Since the distance Zp ^* should not be used for calculating the distance Zg in the real space of the group g, it may be configured to limit the search range beyond the range of each group g.

  Further, it may be configured to always perform a search exceeding the range of each group g, and for example, the corresponding distance Zp in the real space for each pixel p belonging to the group g has a sufficient number of data. In some cases, it may be configured not to be performed.

  Further, in the present embodiment, the comparison corresponding to the reference pixel block PB is performed on the epipolar line EPL extending in the left-right direction of the comparison image Tc by the stereo matching process in the image processor 7 of the distance detection unit 6 described above (see FIG. 3). Since the search for the pixel block PBc is performed, there is a high possibility that the parallax dp is relatively effectively obtained in the left and right edge portions (edge portions) of the group g integrated on the reference image T.

  Therefore, in the present embodiment, as described above, in the distance calculation unit 11, each group g for each pixel column pl having a one-pixel width that extends in the vertical direction belonging to the range on the reference image T of each group g. The search is performed in the upward and downward directions beyond the range, and the search in the horizontal direction is not performed. However, for example, the search is performed in the horizontal direction belonging to the range on the reference image T of each group g. It is also possible to perform a search in the left direction and the right direction beyond the range of each group g for each existing pixel row of one pixel width.

  Next, the light source detection unit 12 includes the coordinates (i, j) on the reference image T of each group g integrated by the integration processing unit 10 and the real space of each group g calculated by the distance calculation unit 11. (A) the size in the real space of the group g, (b) the height in the real space from the road surface, (c) The position in the real space with respect to the traveling path of the own vehicle on the road surface of the group g is calculated, and the group g exists on the road when the size, the height, and the position satisfy predetermined conditions, respectively. It detects as a light source.

[(A) Size of group g in real space]
Specifically, for each group g, the light source detection unit 12 first, as shown in FIG. 14, the i coordinate Imin of the leftmost pixel of the group g, the i coordinate Imax of the rightmost pixel, and the j of the uppermost pixel. The coordinate Jmax and the j coordinate Jmin of the bottom pixel are detected.

Then, by substituting the i-coordinates Imin and Imax of the pixels at the left and right ends of the group g and the distance Zg in the real space of the group g into the above equation (2),
Xmin = CD / 2 + Zg × PW × (Imin−IV) (14)
Xmax = CD / 2 + Zg * PW * (Imax-IV) (15)
Therefore, the size in the real space between the left and right ends of the group g is
Xmax−Xmin = Zg × PW × (Imax−Imin) (16)
And the size in the real space of the group g in the left-right direction.

When the i coordinates Jmax and Jmin of the upper and lower pixels of the group g and the distance Zg in the real space of the group g are substituted into the above equation (3),
Ymin = CH + Zg × PW × (Jmin−JV) (17)
Ymax = CH + Zg × PW × (Jmax−JV) (18)
Therefore, the size in the real space between the upper and lower ends of the group g is
Ymax−Ymin = Zg × PW × (Jmax−Jmin) (19)
And the size in the real space in the vertical direction of the group g.

  In a normal case, the signal lamp of the traffic light is formed in a circular shape having a diameter of 25 cm or 30 cm. Therefore, in this embodiment, including the leakage of light from the signal lamp, the light source detection means 12 includes the right and left of each group g. It is checked whether the sizes Xmax-Xmin and Ymax-Ymin in the real space in the direction and the vertical direction are within a range of, for example, 40 cm.

  As described above, the size of the group g in the real space is calculated for both the size Xmax−Xmin in the horizontal space of the group g and the size Ymax−Ymin in the vertical space of the vertical direction. It may be configured to check, or any one of them may be calculated and checked.

  In addition to those sizes or instead of calculating those sizes, for example, in real space in a predetermined direction such as an oblique direction in each group g (for example, an oblique 45 ° direction on the reference image T). It is also possible to configure to calculate the size. Further, for example, using the coordinates (i, j) of each pixel p of the upper and lower sides of each group g and all the left and right edge portions (edge portions) and the distance Zg in the real space of the group g, According to the equation (3), the position of a point on the real space corresponding to each pixel p is calculated, and the area of the group g on the real space and the perimeter of the group g on the real space are calculated, A predetermined range may be set in advance, and it may be configured to check whether it is within the range.

[(B) Height in real space from the road surface of group g]
For each group g, the light source detection unit 12 also includes an intermediate point Ic between the i-coordinate Imin of the leftmost pixel and the i-coordinate Imax of the rightmost pixel and the j-coordinate Jmax of the uppermost pixel and the lower end of each group g. An intermediate point Jc with the j coordinate Jmin of the pixel, that is,
Ic = (Imin + Imax) / 2 (20)
Jc = (Jmin + Jmax) / 2 (21)
The center point Gc of the group g, where i is the i coordinate and j coordinate is calculated. Then, the height Yc of the center point Gc of the group g in the real space is calculated by substituting Jc and the distance Zg of the group g in the real space into the above equation (3). That is,
Yc = CH + Zg × PW × (Jc−JV) (22)

Then, the light source detection means 12 reads the road height model (see FIG. 7B) of the road surface in the real space detected by the road surface detection means 14 from the memory, and the road at the distance Zg in the real space of the group g. The surface height Y ^* is calculated. Then, the height in the real space from the road surface of the group g is calculated as a difference Yc−Y ^* between the height Yc in the real space of the center point Gc of the group g and the height Y ^* of the road surface. It has become.

In a normal case, the signal light of the traffic light is installed at a height of 5 m or more from the road surface. In this embodiment, the light source detection means 12 is set as described above with a slight width. It is checked whether the height Yc-Y ^* in the real space from the road surface of the group g calculated in the above is within a range of, for example, 4.5 m or more.

  Note that the height in real space from the road surface of group g is not the height from the road surface to the center point Gc of group g, but the height to the lower end of group g (the j coordinate of the pixel is Jmin) It is also possible to set the height to another point determined in the group g by a predetermined determination method.

Further, the height Y ^* of the road surface at the distance Zg in the real space of the group g is obtained by linear interpolation from the road height model given by the above equations (7) and (8) for each section. Can do. Furthermore, the lane model including the road height model is used if the lane model in the current sampling cycle is detected by the road surface detection means 14, and if the lane model in the current sampling cycle is not detected, the previous lane model is used. Based on the lane model detected at the sampling period, the lane model at the current sampling period is estimated and used from the behavior of the host vehicle thereafter.

[(C) Position in real space with respect to traveling path of own vehicle on road surface of group g]
In addition, as in the present embodiment, when detecting a signal lamp that is lit red (that is, a red signal), the signal lamp of the traffic signal that exists on the road of the host vehicle on the road surface should be detected. is there. That is, for example, as shown in FIG. 15, when a plurality of traffic signals A, B, and C are captured in the reference image T, the traffic signal to be detected is also opposite to the traffic signal A captured near the center of the reference image T. It is not the traffic light C on the lane but the traffic light B present on the traveling path of the host vehicle on the road surface.

  Therefore, in the present embodiment, the road in the real space detected by the road surface detection means 14 based on the information of each lane LL, LR, etc. detected to the side of the host vehicle as shown in FIG. Based on the horizontal shape model of the surface (see FIG. 7A), the traveling path of the host vehicle on the road surface is grasped. In this case as well, if the horizontal shape model at the current sampling cycle is not detected, the horizontal shape at the current sampling cycle is determined based on the behavior of the host vehicle thereafter based on the horizontal shape model detected at the previous sampling cycle. A model is estimated and used.

  Then, the light source detection means 12 reads the horizontal shape model of the road surface from the memory, and linearly positions the left and right ends in the X-axis direction (left-right direction) of the traveling path of the host vehicle at the distance Zg in the real space of the group g. It is calculated by a method such as interpolation. That is, the horizontal position of each lane LL, LR on the side of the host vehicle at the distance Zg in the real space of the group g is calculated.

In a normal case, the signal lamp of the traffic light is installed above the traveling path of the host vehicle, and the signal lamp is installed so that the horizontal position of the signal lamp exists between the lanes LL and LR on the side of the host vehicle. Therefore, in the present embodiment, the light source detection unit 12 first calculates the i coordinate Ic (see the above equation (20)) of the center point Gc of the group g and the distance Zg in the real space of the group g from the above equation (2). And the horizontal position Xc in the real space of the center point Gc of the group g is calculated. That is,
Xc = CD / 2 + Zg × PW × (Ic−IV) (23)

  Then, the light source detection means 12 compares the Xc with the horizontal position of each lane LL, LR on the side of the host vehicle at the distance Zg in the real space of the group g calculated as described above. It is checked whether the horizontal position Xc in the real space of the center point Gc of g is within the range of the horizontal position of each lane LL, LR on the side of the host vehicle.

  The range in which the horizontal position Xc in the real space of the center point Gc of the group g can exist in consideration of the detection error of the lanes LL, LR on the side of the host vehicle, the error of the horizontal shape model, etc. The lateral lanes LL and LR can be set to a range of, for example, 1 m outward from the horizontal position.

  In the present embodiment, the light source detection unit 12 turns on the group g that cleared the check items (a) to (c) above as a light source existing on the road, that is, in the case of this embodiment, the traffic light is lit red. Detection is made as a signal lamp (that is, a red signal).

  As described above, according to the environment recognition device 1 according to the present embodiment, at least one image T (reference image T) based on the pair of images (reference image T and comparison image Tc) captured by the imaging unit 2. A distance Z in the real space is calculated for each of the pixels p, and based on this, the distance Zg in the real space of each group g integrated on the one image T based on the image data D is calculated. Then, based on the size in the real space of each group g calculated based on the height, the height in the real space from the road surface, and the position in the real space with respect to the traveling path of the own vehicle on the road surface of the group g. The group g is detected by determining whether or not it is a light source existing on the traveling path of the host vehicle.

  Thus, based on the distance Zp in the real space of each pixel p belonging to each group g, the size in the real space of each group g, the height in the real space from the road surface, and the road surface of the group g Since the position in the real space with respect to the traveling path of the host vehicle is determined, it is accurately determined whether or not the detected light source is a signal light (for example, a red signal) with a traffic light turned on. Thus, the light source can be detected with high accuracy and the reliability of the recognition can be greatly improved.

  Even when the light source is at a distance from the host vehicle, it is based on the size in the real space, the height in the real space from the road surface, and the position in the real space with respect to the traveling path of the host vehicle on the road surface. Since the light source is determined, it is possible to accurately prevent, for example, erroneous recognition of a red display such as an advertising tower beside a road or a neon sign as a red signal.

  In the present embodiment, the traveling path of the host vehicle on the road surface is calculated based on the horizontal shape model of the road surface in the real space detected by the road surface detection means 14 (see FIG. 7A). However, the present invention is not limited to this. For example, based on the turning curvature of the host vehicle calculated from the vehicle speed or the like of the host vehicle, a travel trajectory that the host vehicle is estimated to travel in the future is calculated, and a plane having a predetermined width on the left and right is centered on the travel trajectory. It is also possible to configure so as to calculate the traveling path of the host vehicle.

At that time, the turning curvature Cua of the host vehicle is expressed by the following equation (24) based on information such as the vehicle speed V, yaw rate γ, steering angle δ of the steering wheel, etc. transmitted from the sensors Q (see FIG. 1). Alternatively, it can be calculated according to the following formulas (25) and (26). In the following equations, Re is a turning radius, Asf is a vehicle stability factor, and Lwb is a wheelbase.
Cua = γ / V (24)
Re = (1 + Asf · V ² ) · (Lwb / δ) (25)
Cua = 1 / Re (26)

  Further, in the present embodiment, the signal lamp that is lit red, that is, the case where a red signal is detected has been described, but in addition to this, for example, R (red) in the first condition and the second condition described above Can be configured to detect a blue signal of a traffic light.

  Furthermore, as is well known, among the R (red), G (green), and B (blue) color components of the image data D of each pixel p, when R and G are additively mixed, a yellow color component is obtained. Can be obtained. Therefore, for example, R and G, G and B, and B and R are additively mixed to the image data D of each pixel p of the reference image T obtained by imaging with the imaging unit 2 to yellow, light blue, By calculating each color component of pink, replacing R (red) in the above first condition and second condition with yellow, G (green) and B (blue) with light blue and pink, and applying, It can be configured to detect a yellow signal of a traffic light.

  In this way, predetermined weighted addition is performed on each of the R, G, and B color components of the image data D of each pixel p of the reference image T obtained by imaging with the imaging unit 2, for example, the orange of the preceding vehicle It is possible to detect light sources such as colored tail lamps, turn signals, and orange-red brake lamps.

  At that time, the preset ranges in the check items such as (a) the size of the group g in the real space and (b) the height in the real space from the road surface in the light source detection unit 12 are changed as appropriate. Needless to say, it is used. In addition to the signal light with the traffic light turned on, the light source that may cause a serious accident if it is misrecognized, such as the tail lamp, brake lamp, blinker, etc. of the preceding vehicle. For these, those light sources can be reliably detected and accurately recognized.

  In the present embodiment, as described above, since the distance Zp (parallax dp) in the real space is calculated for each pixel p belonging to the group g, the inclination of each group g in the real space is calculated. Can do.

  Specifically, the equations (2) to (4) are based on the distance Zp in the real space corresponding to each pixel p belonging to the group g and the coordinates (i, j) on the reference image T of each pixel p. Thus, each point (Xp, Yp, Zp) in the real space corresponding to each pixel p can be calculated. If it is plotted on the XZ plane, it can be plotted, for example, as shown in FIG. 17, and the distribution of each point is linearly approximated, for example, the distance to the horizontal direction (X-axis direction) of the distribution of each point. The inclination θ in the direction (Z-axis direction) can be calculated.

  Further, when each point on the real space corresponding to each pixel belonging to the group g is plotted on the YZ plane, it can be plotted as shown in FIG. 18, for example, by linearly approximating the distribution of each point. The slope φ in the distance direction (Z-axis direction) with respect to the height direction (Y-axis direction) of the distribution of each point can be calculated.

  Therefore, for example, threshold values are set in advance for the inclinations θ and φ in the real space calculated in this way, and only light sources that face the host vehicle, that is, light sources having inclinations θ and φ close to 0 ° are used. It can also be configured to detect. If comprised in this way, it will become possible to avoid more correctly detecting erroneously the signal light etc. which the lighting of the traffic light installed in the travel path other than the advancing path of the own vehicle, for example.

  In the environment recognition apparatus 1 according to the present embodiment, when detecting a signal light with a traffic light turned on as a light source, the detected light is further within a predetermined distance from the position on the reference image T of the detected group g. It is possible to confirm that there is no pixel p having a distance Zp in the real space corresponding to the object, and to detect a signal light that is turned on when the pixel p does not exist. .

  If the light source detected as the group g is a signal light with the traffic light turned on, there is usually a space below which no object is present, but the light source detected as the group g is, for example, a tail lamp of a preceding vehicle. For example, since the vehicle body of the preceding vehicle is usually present below the object and there is an object, whether or not the light source is a signal light with a traffic light turned on depending on whether or not an object exists below the detected group g. It is possible to accurately determine whether or not.

  In this case, the predetermined distance below the group g is set to a distance of 1 m as a distance in the real space, for example.

  On the reference image T, the distance Zp in the real space may be detected by the distance detection means 6 for the pixel p existing below the detected group g. And when the same distance Zp is detected about several pixel p below, those pixels p are considered to correspond to the pixel by which the object was imaged. In this case, if the distance in the height direction between the object and the signal light that has been detected as the group g is greater than 1 m, the group g is recognized as a signal light that has been lighted.

  In addition, if the distance in the height direction between the object and the signal light with the traffic light detected as the group g is within 1 m, the group g is not recognized as the signal light with the traffic light lit. Furthermore, when the pixels p that exist below the detected group g and the distance Zp in the real space is detected are scattered, or an object having the same distance Zp in the real space is captured. In such a case, the pixels p are not considered to correspond to the pixels on which the object is imaged, and the group g can be recognized as a signal light with a traffic light turned on.

DESCRIPTION OF SYMBOLS 1 Environment recognition apparatus 2 Imaging means 6 Distance detection means 10 Integrated processing means 11 Distance calculation means 12 Light source detection means 13 Lane detection means 14 Road surface detection means A, B, C Traffic light D Image data g Group Gc Group center point (i, j) Coordinates LL, LR Side lane p of own vehicle Pixel p ^* Pixel present at closest position S Light source T
Xmax-Xmin Group real space size Yc Group real space height Yc-Y ^* Group height from the road surface (difference)
Ymax-Ymin Group vertical size Z, Zp Real space distance Zg Group real space distance Zp ^* Distance in real space corresponding to the closest pixel

Claims (14)

Imaging means for imaging an image having image data for each pixel;
Distance detecting means for detecting a distance in real space for each pixel of the image;
Road surface detection means for detecting a road surface from the image;
Integration processing means for integrating adjacent pixels into one group based on image data of adjacent pixels in the image;
Distance calculating means for calculating a distance in the real space of the group based on at least one of the distance in the real space corresponding to the pixels belonging to the group and the pixels existing around the group;
Based on the distance of the group in the real space and the coordinates of the group in the image, the size of the group in the real space, the height in the real space from the road surface, and the host vehicle of the group A light source detection means for calculating a position in real space with respect to the traveling path of the light source, and detecting the group as a light source existing on the road when the size, height, and position respectively satisfy predetermined conditions;
An environment recognition apparatus comprising: Lane detection means for detecting a lane on the side of the host vehicle from the image,
2. The environment recognition according to claim 1, wherein the light source detection unit calculates a travel path of the host vehicle and a position of the group in real space relative to the traveling path based on a position of the lane in real space. apparatus.   The integration processing means detects a pixel having the image data satisfying a first condition set in advance, and sets a first range set wider than the range of the image data defined by the first condition. 3. When detecting a pixel having the image data satisfying the condition of 2, the pixel is integrated with the pixel having the image data satisfying the first condition to form one group. The environment recognition device described in 1. The image data is color image data,
The first condition is set so that the pixel having a specific color and a specific brightness is detected from the image,
The second condition is set so that the pixel having the specific color but whose brightness is darker than the specific brightness defined by the first condition is detected. The environment recognition device according to claim 3.   The second condition is set such that the pixel having the specific color and the brightness of which is brighter than the specific brightness defined by the first condition is also detected. The environment recognition device according to claim 4.   The integration processing means may be any one of a difference in a distance in the real space between a pixel adjacent to the pixel and a difference in a distance in the real space between the pixel and a distance in the real space of the group. 6. The environment recognition apparatus according to claim 1, wherein when the pixel exceeds a predetermined threshold, the pixels are not integrated into a group.   2. The integration unit according to claim 1, wherein when the pixels are integrated into the group and the shape of the group is not circular, the pixels are not integrated into the group. Item 7. The environment recognition device according to any one of items 6 to 6.   The integration processing means rejects the group and excludes it from the processing target in the light source detection means when the shape of the group is not circular when the pixels are integrated into the group. The environment recognition apparatus according to claim 1, wherein the environment recognition apparatus is characterized.   The distance calculating means searches the distance in the real space corresponding to each pixel belonging to the group and the range on the image of the group to the left or right or up and down to detect the distance in the real space. A distance in the real space of the group is calculated based on at least one of the distances in the real space corresponding to the pixels present at a position closest to the range on the image. The environment recognition device according to any one of claims 1 to 8.   The distance calculation means includes the pixel that is present at a position closest to the distance in the real space corresponding to each pixel belonging to the group and / or the range on the image in the group. The environment recognition device according to claim 9, wherein an average value or a median value of the distances in the real space corresponding to is calculated as a distance in the real space of each group.   The distance calculation means includes the pixel that is present at a position closest to the distance in the real space corresponding to each pixel belonging to the group and / or the range on the image in the group. 10. The environment recognition apparatus according to claim 9, wherein a distance in the real space corresponding to the above is voted on a histogram for each group, and a peak value thereof is calculated as a distance in the real space of each group. .   The distance calculation means includes the pixel that is present at a position closest to the distance in the real space corresponding to each pixel belonging to the group and / or the range on the image in the group. Among the distances in the real space corresponding to the above, the distances close to each other are collected, and the average value of the distances in the real space belonging to the group having the largest number of data among the group of the distances collected or The environment recognition apparatus according to claim 9, wherein a median value is calculated as a distance in the real space of each group.   13. The light source detection unit according to claim 1, wherein the light source detection unit detects any one of a signal light with a traffic light turned on, a tail lamp of a preceding vehicle, a brake lamp, and a winker as the light source. The environment recognition device described.   The light source detection means detects the signal light with the traffic light turned on as the light source, and when the object is not present within a predetermined distance below the detected position of the group on the image, the signal light is turned on. The environment recognition device according to claim 1, wherein a signal light is detected.

Images