JP2012168592A - Environment recognition device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an environment recognition device which can accurately recognize an arrow signal of a traffic light, and recognize the arrow signal on a local vehicle at a position far away from the traffic light.SOLUTION: An environment recognition device 1 includes detection means 12 for red signal which detects a red signal Lr etc., of a traffic light S based upon an image T picked up by imaging means 2; and arrow signal extraction means 13 of extracting an arrow signal A imaged in a search region Rs set based upon a position of the detected red signal Lr etc., in the image T. The arrow signal extraction means 13 switches a method of recognizing a direction that the arrow signal A indicates according to a distance Z from a local vehicle MC to the red signal Lr etc., in the actual space, and recognizes the direction that the arrow signal A indicates based upon mutual position relation between the arrow signal A and red signal Lr etc., in the image T within a first predetermined section Sec1 far from the local vehicle MC.

Description

  The present invention relates to an environment recognition device, and more particularly to an environment recognition device for recognizing a traffic light installed on a road.

  2. Description of the Related Art Conventionally, there has been known a device that mounts a CCD camera, a CMOS camera, or the like on a host vehicle and detects a signal lamp or the like in which a traffic light is lit from a captured image captured by the camera.

  For example, in the apparatus described in Patent Document 1, a red light emitting unit is extracted from a captured image captured by a camera, the luminance and size of the extracted red light emitting unit on the captured image, and the high on the captured image. Then, based on the shape, ellipticity, etc., it is determined whether or not the red light emitter of the traffic light is in a light emission state, that is, whether or not it is a red signal.

  However, in the traffic lights, as shown in FIGS. 28 (A) to (C), in addition to the blue, yellow, and red signal lights Lb, Ly, and Lr, arrows indicating right turn, straight travel, and left turn are possible. There is also a traffic light S with signals Al, As, Ar. If only the red signal Lr is extracted as described above with respect to such a traffic light S, for example, in the case where automatic traveling control of the host vehicle is performed based on the information, an arrow signal indicating that the traffic light S can go straight. Although As is lit, the host vehicle may stop before the stop line.

  In this way, if the vehicle is stopped before the stop line even though the straight-ahead arrow signal As is lit, the braking operation is performed because the straight-ahead arrow signal As is lit. There is a risk that the vehicle will be bumped into a subsequent vehicle that will not be used. Therefore, for example, in Patent Document 2, the red signal Lr and the yellow signal Ly of the traffic light S are detected, a search area is set at a predetermined position with respect to the red signal Lr and the yellow signal Ly, and comparison with a template is performed within this search area ( That is, a traffic signal recognition device that detects arrow signals Al, As, and Ar by template matching) is disclosed.

JP 2007-34693 A JP 2009-43068 A

  By the way, when the traffic light S is imaged by a CCD camera or the like, if the traffic light S is at a position close to the own vehicle, for example, as shown in FIG. 29, an arrow signal (in the case of FIG. 29, an arrow signal in the captured image T). As) is captured relatively clearly, and it is easy to detect an arrow signal by template matching.

  However, when the traffic light S is slightly far from the host vehicle, for example, as shown in FIG. 30, the arrow signal As captured in the image T is blurred, and it is difficult to detect the arrow signal by template matching. When the traffic light S is further away from the host vehicle, for example, as shown in FIG. 31, the arrow signal As captured in the image T becomes further blurred, and the detection of the arrow signal by template matching is no longer possible.

  Note that the size of each pixel in the image T in FIGS. 29 to 31 is the same, and FIGS. 30 and 31 show enlarged photographs. That is, when viewed from the size of the traffic light S shown in FIG. 29, the traffic light S is captured in a smaller size in the image T in FIG. 30, and the traffic light S is captured in a smaller size in the image T in FIG. .

  As described above, when the method of detecting the arrow signal by template matching is employed, the arrow signal cannot be detected while the traffic light S is far from the host vehicle, and the arrow signal can be recognized by template matching. There is a problem that the arrow signal cannot be recognized unless the traffic light S is close to the host vehicle until it is captured in the size of the image T.

  Therefore, from the stage where the traffic light S is still far from the host vehicle, the arrow signal of the traffic signal S is accurately recognized, and the information is used for automatic traveling control of the host vehicle or the driver is appropriately warned. It was difficult to perform control such as.

  The present invention has been made in view of such circumstances, and can accurately recognize an arrow signal of a traffic light, and recognize an arrow signal from a stage where the traffic signal is far from the host vehicle. An object of the present invention is to provide an environment recognition device capable of performing the above.

In order to solve the above problem, the first invention is an environment recognition apparatus,
Imaging means for imaging an image having image data for each pixel;
Distance detecting means for detecting a distance in real space for each pixel of the image;
Integration processing means for integrating adjacent pixels into one group based on image data of adjacent pixels in the image;
Distance calculating means for calculating a distance in the real space of the group based on the distance in the real space corresponding to the pixels belonging to the group and the pixels existing around the group;
Based on the distance of the group in the real space and the size of the group in the real space and the height in the real space from the road surface calculated based on the coordinates in the image of the group, Or a detection means such as a red signal for detecting a yellow signal light;
Based on the detected position of the red or yellow signal lamp in the image, an image area in which an arrow signal may be captured is set as a search area, and the arrow signal captured in the search area Arrow signal extraction means for extracting
With
The arrow signal extraction means includes
It is configured to change the method of recognizing the direction indicated by the arrow signal according to the distance in the real space from the own vehicle to the red or yellow signal light,
When the distance in the real space to at least the red or yellow signal light is within a first predetermined section far from the host vehicle, the arrow signal and the red or yellow signal light in the image Based on the positional relationship, the direction indicated by the arrow signal is recognized.

  According to a second aspect of the present invention, in the environment recognition device according to the first aspect, the arrow signal extraction means has at least a distance in the real space to the red or yellow signal light that is closer to the host vehicle than the first predetermined section. As a method for recognizing the direction indicated by the arrow signal when within the second predetermined interval, the direction indicated by the arrow signal is recognized based on left-right or vertical symmetry of the arrow signal in the image. It is characterized by doing.

  According to a third aspect of the present invention, in the environment recognition apparatus according to the second aspect, the arrow signal extraction means sets an image area circumscribing a pixel area corresponding to the arrow signal in the image, and the center of the image area The image region is divided by a straight line extending in the up-down direction or the left-right direction passing through and the number of pixels belonging to the pixel region corresponding to the arrow signal included in each of the divided image regions is compared. Alternatively, the upper and lower symmetry is determined.

  According to a fourth aspect of the present invention, in the environment recognition device of the second or third aspect, the arrow signal extraction means is configured such that at least the distance in the real space to the red or yellow signal lamp is smaller than the second predetermined section. Each pixel region in the image corresponding to the arrow head portion and the arrow shaft portion that forms the arrow signal as a method of recognizing the direction indicated by the arrow signal when in the third predetermined section close to the vehicle The direction indicated by the arrow signal is recognized based on the mutual positional relationship.

  According to a fifth aspect of the present invention, in the environment recognition device according to the fourth aspect, the arrow signal extraction means is configured to select the pixel regions in the image corresponding to the arrow head portion and the arrow shaft portion that form the arrow signal, It is characterized by identifying by comparing the number of pixels belonging to each pixel region.

  According to a sixth aspect of the present invention, in the environment recognition device according to the fourth or fifth aspect, the arrow signal extraction means determines each center point of each pixel area corresponding to the arrow head portion and the arrow shaft portion, respectively. The direction indicated by the arrow signal is recognized based on the calculated directionality of the vector connecting the center points.

A seventh invention is the environment recognition device according to any one of the first to sixth inventions,
Furthermore, a road surface detection means for detecting a road surface from the image,
The detection means such as a red signal is calculated based on the distance of the group in the real space and the coordinates of the group in the image, and the size of the group in the real space, and the road detected by the road surface detection means. A red or yellow signal light of a traffic light is detected based on the height of the surface in real space.

The eighth invention is an environment recognition apparatus,
Imaging means for imaging an image having image data for each pixel;
Distance detecting means for detecting a distance in real space for each pixel of the image;
Integration processing means for integrating adjacent pixels into one group based on image data of adjacent pixels in the image;
Distance calculating means for calculating a distance in the real space of the group based on the distance in the real space corresponding to the pixels belonging to the group and the pixels existing around the group;
Based on the distance of the group in the real space and the size of the group in the real space and the height in the real space from the road surface calculated based on the coordinates in the image of the group, Or a detection means such as a red signal for detecting a yellow signal light;
Based on the detected position of the red or yellow signal lamp in the image, an image area in which an arrow signal may be captured is set as a search area, and the arrow signal captured in the search area Arrow signal extraction means for extracting
With
The arrow signal extraction means includes a plurality of recognition means for recognizing the direction indicated by the arrow signal by different methods as a method for recognizing the direction indicated by the arrow signal, and the actual signal from the own vehicle to the red or yellow signal lamp is provided. According to a distance in space, the direction recognized by any one of the plurality of recognition means is selected and output.

  According to the first invention, at least when the distance in the real space to the red signal and the yellow signal of the traffic light S is within the first predetermined section far from the host vehicle, the arrow signal and the red signal in the image, etc. Since the direction indicated by the arrow signal is recognized based on the mutual positional relationship with the vehicle, the conventional method of detecting the direction indicated by the arrow signal by template matching is very far from the host vehicle where the direction could not be detected. For the traffic light at the position (see, for example, FIG. 31), the direction indicated by the arrow signal can be accurately recognized and detected.

  Therefore, from the stage where the traffic light is still far from the host vehicle, the arrow signal of the traffic signal is accurately recognized, and the information is used for automatic driving control of the host vehicle, or the driver is appropriately warned. Control can be performed.

  According to the second invention, when the distance in the real space from the host vehicle to the red signal or the like is in the second predetermined section closer to the host vehicle than the first predetermined section (see, for example, FIG. 30), it is also conventional. Although it was difficult to detect the direction indicated by the arrow signal using the template matching method of, the direction of the arrow signal can be accurately recognized based on the left and right symmetry and the vertical symmetry of the arrow signal in the image. Is possible.

  Therefore, in addition to the effects of the present invention, it is possible to accurately recognize and detect the direction indicated by the arrow signal even for a traffic light that is relatively far from the host vehicle. Therefore, from the stage where the traffic light is still relatively far from the host vehicle, the arrow signal of the traffic signal is accurately recognized, and the information is used for automatic driving control of the host vehicle, or the driver is warned appropriately. Etc. can be controlled.

  According to the third invention, the image area circumscribing the pixel area corresponding to the arrow signal set in the image is divided by the straight line extending in the vertical direction and the horizontal direction passing through the center of the image area. By comparing the number of pixels belonging to the pixel area corresponding to the arrow signal included in each image area, and determining the left / right symmetry and the vertical symmetry, the left / right symmetry and the upper / lower symmetry of the arrow signal are determined. It is possible to accurately determine the symmetry of.

  Then, based on the right and left symmetry and the vertical symmetry determined accurately, it becomes possible to accurately recognize the direction indicated by the arrow signal, and the effect of the second invention can be exhibited more accurately. It becomes possible.

  According to the fourth invention, in addition to the effects of the respective inventions described above, when the distance in real space to the red signal of the traffic light is within the third predetermined section closer to the host vehicle than the second predetermined section, In order to recognize the direction indicated by the arrow signal based on the mutual positional relationship between the pixel regions in the image corresponding to the arrow head portion and the arrow shaft portion that form the arrow signal, a traffic signal (for example, FIG. 29)), the direction indicated by the arrow signal can be accurately recognized and detected.

  Therefore, in addition to the effects of the inventions described above, even when the distance in the real space to the red light of the traffic light is within the third predetermined section close to the host vehicle, the direction of the direction indicated by the accurately recognized arrow signal It is possible to perform control such as using the information for automatic traveling control of the own vehicle or appropriately warning the driver.

  At the same time, the conventional method using template matching makes processing heavy, and processing may take time and real-time characteristics may be impaired. Also, a template corresponding to an irregular arrow signal is not prepared. In some cases, there may be a problem that the direction of the arrow signal cannot be recognized. However, by configuring as described above, the processing does not become very heavy, real-time characteristics are ensured, and an irregular arrow signal is obtained. It is possible to accurately recognize the direction of the arrow signal even when is displayed.

  According to the fifth invention, each pixel area in the image corresponding to the arrow head part and the arrow shaft part forming the arrow signal is identified by comparing the number of pixels belonging to each pixel area, The pixel region can be accurately associated with the arrow head portion and the arrow shaft portion, and the effect of the fourth invention can be more accurately exhibited.

  According to the sixth aspect of the invention, by recognizing the direction indicated by the arrow signal based on the directionality of the vector connecting the center points of the respective pixel regions respectively corresponding to the arrow head portion and the arrow shaft portion, The direction indicated by the arrow signal including the arrow signal can be recognized as the vector directionality, and the direction of the arrow signal can be recognized more accurately. Therefore, the effects of the above inventions can be more accurately exhibited.

  According to the seventh aspect of the invention, based on the height in the real space of the road surface itself detected by the road surface detection means and the height in the real space of the group corresponding to the red signal or the like extracted in the image, It becomes possible to calculate the height of the signal from the actual road surface, and based on this, it is possible to detect the red signal of the traffic light.

  Therefore, in addition to the effects of each of the above inventions, a group that may correspond to a red signal or the like extracted in an image is accurately recognized as a group that corresponds to a red signal or the like, and thereafter Processing can be performed, and the detection accuracy of a red signal, the detection accuracy of an arrow signal, the recognition accuracy of the direction indicated by the arrow signal, and the like can be reliably improved.

  According to the eighth aspect of the present invention, the arrow signal extraction means is provided with a plurality of recognition means that adopt different methods as methods for recognizing the direction indicated by the arrow signal, so that the distance in the real space from the own vehicle to the red signal, etc. Accordingly, even if it is configured to select and output the direction indicated by the arrow signal recognized by any of the plurality of recognition means, the same beneficial effects as those of the above inventions can be exhibited. Is possible.

It is a block diagram which shows the structure of the environment recognition apparatus which concerns on this embodiment. It is a figure which shows the example of the reference | standard image imaged with an imaging means. It is a figure explaining the stereo matching process by the image processor of a distance detection means. It is a figure which shows the example of the produced distance image. It is a figure explaining the example of the lane candidate point detected by searching on the horizontal line of a reference image. It is a figure explaining the example of the lane detected on either side of the own vehicle. It is a figure explaining the example of the formed lane model, (A) represents the horizontal shape model on a ZX plane, (B) represents the road height model on a ZY plane. It is a figure which shows the example of the red signal imaged on the reference | standard image. It is a figure which shows the pixel and group which satisfy | fill the 2nd condition detected by searching in the up-down direction of the detected pixel. FIG. 10 is a diagram illustrating a method for detecting a group in a pixel column on the right side of the pixel column in FIG. 9. It is a figure explaining the change of the length of the vertical direction of a group, when (A) becomes the maximum value, (B) is shorter than a maximum value, (C) represents the case where it is still shorter than (B). . It is a figure explaining the case where the length of the vertical direction of a group has increased from the minimum value. It is a figure showing each pixel which adjoined the upper and lower sides of the range of the group, and the distance in real space is detected. It is a figure explaining each pixel of the right-and-left end of a group, and each pixel of an upper-lower end. It is a flowchart which shows the process sequence of the recognition process of the direction which the arrow signal extraction process in the arrow signal extraction means and the arrow signal points. It is a figure showing the vertical signal apparatus and the arrow signal provided in the side. It is a figure showing the search area | region set below the group corresponding to the red signal of a reference | standard image, and the left and right sides. It is a figure explaining the 1st predetermined section Sec1, the 2nd predetermined section Sec2, and the 3rd predetermined section Sec3 which section the distance in the real space from the own vehicle to a red signal. It is a flowchart which shows the process sequence of the recognition process of the direction which the arrow signal points out in the recognition method 1 in the arrow signal extraction means. It is a figure explaining the positional relationship of the pixel area | region corresponding to the extracted arrow signal, and the group corresponding to a red signal. It is a figure explaining the positional relationship of the pixel area | region corresponding to the extracted arrow signal and the group corresponding to a red signal in case a traffic light is a vertical traffic light. It is a flowchart which shows the process sequence of the recognition process of the direction which the arrow signal points in the recognition method 2 in the arrow signal extraction means. It is a figure explaining the image area | region circumscribed to the pixel area | region corresponding to an arrow signal, the straight line passing through the center, etc. It is a figure explaining the image area | region circumscribed to the pixel area corresponding to the arrow signal in case an arrow signal is an arrow signal which shows the right turn possible, the straight line which passes along the center, etc. It is a flowchart which shows the process sequence of the recognition process of the direction which the arrow signal points in the recognition method 3 in the arrow signal extraction means. It is a figure explaining each pixel area | region corresponding to the arrow head part of an arrow signal, an arrow shaft part, each center point, the vector which connects center points, etc. FIG. It is a figure explaining the arrow head part in case an arrow signal is irregular, an arrow shaft part, each center point, the vector which connects center points, etc. (A)-(C) is a figure which shows the example of the general signal apparatus provided with the arrow signal other than the red signal. It is a photograph which shows the example of the image of the red signal and arrow signal imaged when the distance from the own vehicle to a traffic signal is near. It is a photograph which shows the example of the image of the red signal and arrow signal imaged when the distance from the own vehicle to a traffic signal is a little far. It is a photograph which shows the example of the image of the red signal and arrow signal imaged when the distance from the own vehicle to a traffic signal is further far.

  Hereinafter, an embodiment of an environment recognition apparatus according to the present invention will be described with reference to the drawings.

  As shown in FIG. 1, the environment recognition apparatus 1 according to the present embodiment includes a processing unit 9 including an imaging unit 2, a distance detection unit 6, an integrated processing unit 10, and the like.

  The upstream configuration of the processing unit 9 including the distance detecting means 6 and the like is described in detail in Japanese Patent Application Laid-Open No. 2006-72495 previously filed by the applicant of the present application. Leave them to the gazette. A brief description is given below.

  In the present embodiment, the image pickup means 2 includes a built-in image sensor such as a CCD or a CMOS sensor that is synchronized with each other. For example, a pair of image sensors 2 attached in the vicinity of a vehicle rearview mirror with a predetermined interval in the vehicle width direction. The stereo camera is composed of a main camera 2a and a sub camera 2b, and is configured to capture a predetermined sampling period and output a pair of images.

  Of the pair of cameras, the main camera 2a is a camera on the side close to the driver, and takes an image T as shown in FIG. 2, for example. In order to distinguish from the image captured by the sub camera 2b, hereinafter, the image T captured by the main camera 2a is referred to as a reference image T, and the image (not shown) captured by the sub camera 2b is referred to as a comparative image Tc.

  In the present embodiment, the main camera 2a and the sub camera 2b of the image pickup unit 2 acquire color image data D represented by RGB values or the like, but pick up monochrome image data. It is also possible to use an imaging means that performs this, and the present invention is also applied to such a case.

Hereinafter, color image data D is output as RGB values, that is, luminances (R, G, B) of R (red), G (green), and B (blue) color components in the RGB color system. However, the RGB values are converted into data values described in, for example, the L ^* a ^* b ^* color system, HLS (also referred to as HSL, HSI), etc., and the process is performed. Is also possible. Furthermore, when monochrome image data is acquired as described above, the image data is represented by the luminance of each pixel.

  The conversion unit 3 includes a pair of A / D converters 3a and 3b, and each image data of the reference image T and the comparison image Tc captured for each pixel by the main camera 2a and the sub camera 2b of the imaging unit 2. When D is sequentially transmitted, the RGB value of each image data D is converted into a digital value having a luminance of, for example, 0 to 255 and output to the image correction unit 4.

  The image correction unit 4 sequentially performs image corrections such as displacement, noise removal, luminance correction, and the like on the transmitted reference image T and comparison image Tc for each image data D, and the corrected image of the reference image T Each image data D of the comparison image Tc is sequentially stored in the image data memory 5 and is sequentially transmitted to the processing unit 9. Further, the image correction unit 4 sequentially transmits the image data D of the reference image T and the comparison image Tc subjected to the image correction to the distance detection unit 6.

  In the image processor 7 of the distance detection means 6, stereo matching processing and filtering processing are sequentially performed on the image data of the reference image T and the comparison image Tc, and the parallax dp corresponding to the distance in the real space is obtained from the reference image T. Calculation is sequentially performed for each pixel.

In the stereo matching process in the image processor 7, when each image data D of the reference image T and the comparison image Tc is transmitted, for example, 3 × 3 pixels or 4 × 4 pixels on the reference image T as shown in FIG. For each comparison pixel block PBc having the same shape as the reference pixel block PB on the epipolar line EPL in the comparison image Tc corresponding to the reference pixel block PB, a reference pixel block PB having a predetermined number of pixels is set. An SAD value, which is a difference in luminance pattern from the reference pixel block PB, is calculated according to the equation, and a comparison pixel block PBc having the smallest SAD value is specified.
SAD = Σ | D1s, t−D2s, t |
= Σ {| R1s, t-R2s, t | + | G1s, t-G2s, t |
+ | B1s, t-B2s, t |} (1)

  In the above equation (1), D1s, t, etc. represent the image data D, etc. of each pixel in the reference pixel block PB, and D2s, t, etc., represent the image data D, etc., of each pixel in the comparison pixel block PBc. . Further, the above sum is obtained when the reference pixel block PB and the comparison pixel block PBc are set as a 3 × 3 pixel region, for example, a range of 1 ≦ s ≦ 3, 1 ≦ t ≦ 3, and 4 × 4 pixels. When set as a region, calculation is performed for all pixels in the range of 1 ≦ s ≦ 4 and 1 ≦ t ≦ 4.

  For each reference pixel block PB of the reference image T in this way, the image processor 7 calculates the parallax dp from the position on the comparison image Tc of the identified comparison pixel block PBc and the position on the reference image T of the reference pixel block PB. It is calculated sequentially. Hereinafter, an image in which the parallax dp is assigned to each pixel of the reference image T is referred to as a distance image Tz. In addition, information on the parallax dp calculated for each pixel in this way, that is, the distance image Tz is sequentially stored in the distance data memory 8 of the distance detecting unit 6 and is sequentially transmitted to the processing unit 9. ing.

In real space, the point on the road surface directly below the center of the pair of cameras 2a, 2b is the origin, the vehicle width direction (that is, the left-right direction) of the host vehicle is the X-axis direction, and the vehicle height direction (that is, the vertical direction). ) In the Y-axis direction and the vehicle length direction (that is, the distance direction) as the Z-axis direction, the point (X, Y, Z) in the real space, the coordinates (i, j) of the pixel on the distance image Tz, and The parallax dp, that is, (i, j, dp) can be associated one-to-one by coordinate transformation based on the principle of triangulation expressed by the following equations (2) to (4).
X = CD / 2 + Z * PW * (i-IV) (2)
Y = CH + Z × PW × (j−JV) (3)
Z = CD / (PW × (dp−DP)) (4)

  In the above equations, CD is the distance between the pair of cameras, PW is the viewing angle per pixel, CH is the mounting height of the pair of cameras, and IV and JV are i on the distance image Tz at the infinity point in front of the host vehicle. The coordinates, j-coordinate, and DP represent vanishing point parallax.

  Further, for the purpose of improving the reliability of the parallax dp, the image processor 7 performs a filtering process on the parallax dp obtained by the stereo matching process as described above, and outputs only the valid parallax dp. It has become.

  For example, even if a stereo matching process is performed on the reference image block PB of 4 × 4 pixels having poor features consisting only of a road surface image, the road surface of the comparison image Tc is captured. Since all the correlations are high, the reliability of the parallax dp is low even if the corresponding comparison pixel block PBc is specified and the parallax dp is calculated.

  Therefore, the parallax dp is invalidated by performing a filtering process, and 0 is output as the value of the parallax dp. The parallax dp to which 0 is assigned is not converted into the coordinates (Z, Y, Z) in the real space according to the above equation (4). That is, the parallax dp means that at least the distance Z in real space is unknown.

  Then, when the distance image Tz is created by assigning (ie, associating) effectively calculated parallax dp to each pixel of the reference image T, the distance image Tz is displayed on the reference image T as shown in FIG. Thus, an image in which the effective parallax dp is calculated for the edge portion (edge portion) of the imaging target, which is a portion having a significant feature, is obtained. In creating the distance image Tz, the parallax dp can be converted into the distance Z or the like in advance according to the above equation (4) and the like, and the distance Z or the like can be assigned to each pixel of the reference image T. It is.

  In the present embodiment, in each process in the integration processing unit 10, the distance calculation unit 11, the matching processing unit 12 and the like of the processing unit 9, each pixel of the distance image Tz according to the expressions (2) to (4) as necessary. The i-coordinate, j-coordinate, and parallax dp are converted into an X coordinate (that is, a position in the horizontal direction), a Y coordinate (that is, a position in the vertical direction), and a distance Z in the real space.

  In the present embodiment, as described above, the imaging unit 2 includes the main camera 2a and the sub camera 2b, and the distance detection unit 6 performs stereo matching processing on the reference image T and the comparison image Tc captured by them. The distance Z (that is, the parallax dp) in the real space is calculated for each pixel of the reference image T. However, the present invention is not limited to this, and the imaging unit 2 is a single image such as a monocular camera. Only T may be output.

  Further, the distance detection means 6 only needs to have a function of calculating or measuring the distance Z in the real space and assigning it to each pixel of the image T. For example, the distance detection means 6 irradiates the front of the host vehicle with a laser beam or the like and reflects it. A radar apparatus that measures the distance Z to the imaging target imaged on the image T based on the light information may be used, and the detection method is not limited to a specific method.

  In this embodiment, the processing unit 9 is configured by a computer (not shown), a ROM (Read Only Memory), a RAM (Random Access Memory), a RAM (Random Access Memory), an input / output interface or the like connected to the bus, or a dedicated circuit. Has been. The processing unit 9 includes an integrated processing unit 10, a distance calculation unit 11, a red signal detection unit 12, and an arrow signal extraction unit 13. In the present embodiment, the processing unit 9 is further provided with a lane detection unit 14 and a road surface detection unit 15.

  Sensors Q such as a vehicle speed sensor, a yaw rate sensor, and a steering angle sensor for measuring the steering angle of the steering wheel are connected to the processing unit 9, and each measurement value is input from them. Note that the processing unit 9 may be configured to perform other processing such as detection of a preceding vehicle.

  Here, before describing the processing in the integrated processing means 10 and the like of the processing unit 9 of the present embodiment, the processing in the lane detection means 14 and the road surface detection means 15 will be described.

  In the following description, the lane refers to a continuous line or a broken line marked on a road surface such as an overtaking prohibition line or a lane marking that divides the roadside zone and the roadway. In the present embodiment, as described below, the lane detection unit 14 detects a lane marked on the road surface, and the road surface detection unit 15 detects the road surface based on the detection result. However, the road surface detection means 15 is not limited to the form described below as long as it can detect the road surface.

  The lane detection unit 14 detects a lane on the side of the host vehicle from the reference image T captured by the imaging unit 2. Specifically, as shown in FIG. 5, the lane detection unit 14 searches the horizontal line j having a width of one pixel using the reference image T in the horizontal direction from the center of the reference image T, for example. Pixels that greatly change from the brightness of adjacent pixels to a set threshold value or more are detected as lane candidate points cl and cr.

  In this case, as the luminance of the pixel, for example, when the image captured by the imaging unit 2 is a monochrome image as described above, the luminance for each pixel that is image data of each pixel of the captured image is used. In addition, when the image picked up by the image pickup means 2 is a color image as in the present embodiment, for example, the sum of the color components R, G, B of the luminance (R, G, B) for each pixel. Is used as the luminance for each pixel. Alternatively, it may be configured to determine the luminance for each pixel with reference to the value of each color component.

  Then, the lane detection means 14 detects the lane candidate points cl and cr on each horizontal line j in the same manner while shifting the horizontal line j on the reference image T upward by one pixel. At this time, if the lane detection unit 14 determines that the lane candidate points cl and cr are not on the road surface based on the parallax dp of the detected lane candidate points, the lane candidate points cl and cr are determined as lane candidates. Exclude from points.

  In this case, the road surface at the current sampling period is detected based on the lanes LL and LR (see FIG. 6 described later) detected by the lane detection unit 14 as will be described later. At the time of the lane detection process in, the road surface in the current sampling cycle has not been detected yet. Therefore, in the current sampling cycle, for example, based on the information on the position of the road surface detected by the road surface detection means 15 in the previous sampling cycle, the behavior of the host vehicle from the previous sampling cycle to the current sampling cycle, etc. The position of the surface is estimated.

  The lane detecting means 14 performs Hough transform on each lane to the left and right of the host vehicle based on the lane candidate points cl and cr closer to the host vehicle in the distance direction among the remaining lane candidate points cl and cr. Each of them is detected by approximating it with a straight line. Then, on the far side, lane candidate points cl and cr are selected and connected based on the straight line based on the positional relationship with the straight line, and as shown in FIG. LL and LR are detected.

  The processing configuration of the lane detection means 14 described above is described in detail in Japanese Patent Application Laid-Open No. 2006-331389 previously filed by the applicant of the present application. Further, the lane detection means 14 stores information such as the lane positions LL and LR and the lane candidate points cl and cr detected in this manner in a memory (not shown).

  The road surface detection means 15 is configured to form a lane model three-dimensionally based on the information of the lane positions LL and LR and the lane candidate points cl and cr detected by the lane detection means 14.

  In the present embodiment, as shown in FIGS. 7A and 7B, the road surface detection means 15 approximates the left and right lanes of the host vehicle with a three-dimensional linear expression for each predetermined section, and forms them in a polygonal line shape. A lane model expressed by connecting to the lane is formed. 7A shows a lane model on the Z-X plane, that is, a horizontal shape model, and FIG. 7B shows a lane model on the Z-Y plane, that is, a road height model.

  Specifically, the road surface detection means 15 is a section of the real space ahead of the host vehicle from the position of the host vehicle (that is, the position of Z = 0 in FIGS. 7A and 7B) to, for example, a distance Z7. The lane candidate points cl and cr in each section are linearly approximated by the least square method based on the positions (X, Y, Z) of the detected lane candidate points cl and cr in the real space. The lane model is formed by calculating parameters aL, bL, aR, bR, cL, dL, cR, dR in the following equations (5) to (8).

[Horizontal shape model]
Left lane X = aL ・ Z + bL (5)
Right lane X = aR ・ Z + bR (6)
[Road height model]
Left lane Y = cL · Z + dL (7)
Right lane Y = cR · Z + dR (8)

  The road surface detection means 15 forms a lane model in this way and detects a road surface in real space. Further, the road surface detecting means 15 stores the lane model formed in this way, that is, the calculated parameters aL to dR of each section in the memory.

  Next, each processing in the integrated processing means 10, the distance calculation means 11, the red signal detection means 12, and the arrow signal extraction means 13 of the processing unit 9 (see FIG. 1) will be described, and the environment recognition apparatus according to the present embodiment. The operation of 1 will also be described.

  In the following, the case of detecting the red signal Lr of the traffic light S (that is, the red signal lamp Lr; see FIGS. 28A to 28C) will be described, but the conditions relating to the luminance of the color component in each process are changed. Then, it can be configured to detect the yellow signal Ly (that is, the yellow signal lamp Ly). Therefore, it is possible to configure so that the detection processing of the red signal Lr and the yellow signal Ly is performed in parallel.

  In the present embodiment, the reference image T captured by the main camera 2a of the imaging unit 2 is used as an object to be processed, but the comparison image captured by the sub camera 2b of the imaging unit 2 is used. It is possible to configure to use Tc, or to use both of them.

  Note that each processing configuration in the integrated processing means 10, distance calculation means 11, and red signal detection means 12 in this embodiment is described in detail in Japanese Patent Laid-Open No. 2010-224925 previously filed by the applicant of the present application. For details, see the same publication. Hereinafter, the main points of each processing in the integrated processing unit 10, the distance calculation unit 11, and the red signal detection unit 12 will be described.

  The integration processing unit 10 determines whether or not to integrate the adjacent pixels p into one group based on the image data D of the adjacent pixels p for the adjacent pixels p in the reference image T. When it is determined that they should be integrated, they are integrated into one group g.

  In the present embodiment, the integration processing unit 10 searches each pixel p in the reference image T, and includes a pixel p (hereinafter referred to as a pixel of interest) having image data D that satisfies a preset first condition for the image data D. When p (i, j) is detected, each pixel p adjacent to the periphery of the target pixel p (i, j) is searched.

  When a pixel p having image data D satisfying the second condition set in a wider range than the first condition is detected, the pixel p is integrated with the target pixel p (i, j) to form one group g. Further, each pixel p adjacent to the periphery of the group g is searched, and when the pixel p having the image data D satisfying the second condition is detected, it is integrated into the group g.

  Specifically, when the image data D of each pixel p of the reference image T is transmitted from the main camera 2a of the imaging unit 2, the integration processing unit 10 searches for each pixel p in the reference image T, and the image A pixel p having image data D satisfying the following first condition set in advance for data D is searched.

[First condition]
As the first condition, for example, when the luminance R, G, B of each color component of the image data D of the pixel p is expressed by a luminance of 0-255, for example,
R ≧ 200 and G, B <200 (9)
If there is a pixel p that satisfies this condition, the pixel p is set to be detected as the target pixel p (i, j).

  When the red signal Lr of the traffic light S is picked up by the image pickup means 2, the center portion Lc of the red signal Lr is picked up in reddish white on the reference image T as shown in FIG. As the brightness of each of the R, G, and B color components of the image data D exceeds 200, it is detected as a larger value. This is not a phenomenon limited to the red signal Lr. For example, even with a signal lamp that emits green light such as a blue signal Lb of a traffic light, the central portion of the light source is imaged in greenish white on the reference image T.

In such a case, for the purpose of detecting the central portion Lc of the red signal Lr as the first condition, the first condition is set as, for example, [Condition 2b] of the second condition described later.
R, G, B> 200 and R ≧ G, R ≧ B (10)
If all of the objects captured in the reference image T, such as the sky captured in white and bright, are detected, a light source that emits red light such as the red signal Lr of the traffic light S cannot be detected efficiently.

  Therefore, in the present embodiment, the pixel p of the pixel region Ll picked up in the surrounding area that is red and bright is detected first, not the pixel p picked up white like the central portion Lc of the red signal Lr. It is configured.

  When the integration processing unit 10 detects the target pixel p (i, j) having the image data D satisfying the first condition on the reference image T, subsequently, as shown in FIG. 9, the detected target pixel is detected. The image data D of each pixel p is converted into the first data while shifting one pixel in the vertical direction on the pixel column pls extending in the vertical direction in the reference image T including p (i, j). It is determined whether or not the following second condition set in a wider range than the above condition is satisfied.

[Second condition]
[Condition 2a] R ≧ 165 and G, B <150 (11)
[Condition 2b] R, G, B> 200 and R ≧ G, R ≧ B (12)
[Condition 2c] R ≧ 200 and G, B <200 (13)

  Here, [Condition 2a] of the second condition has a red color peculiar to the red signal Lr, but the pixel in the pixel region Ll that is brightly imaged so that the brightness is defined by the first condition described above. This is a condition for detecting the pixel p in the darker pixel region Ld (see FIG. 8) imaged around the pixel region Ll instead of p.

  In addition, as described above, [Condition 2b] of the second condition has a specific red color in the red signal Lr, and the brightness is brighter and whitish than the brightness defined by the first condition. This is a condition for detecting the pixel region Lc at the center of the captured red signal Lr.

  Furthermore, [Condition 2c] of the second condition is the same condition as the first condition described above, and is a condition for detecting the pixel p in the pixel region Ll as with the target pixel p (i, j).

  For example, when the integration processing unit 10 determines that the image data D of each pixel p (i, j + 1) adjacent to the upper side of the target pixel p (i, j) satisfies the second condition, the pixel p (i , J + 1) are integrated with the pixel of interest p (i, j) to form one group g, and the image data D of each pixel p is shifted by one pixel in the vertical direction on the pixel column pls. Whether or not condition 2 is satisfied is determined, and pixels p that satisfy the second condition are integrated into group g until a pixel p that does not satisfy any of the second conditions appears.

  Then, as shown in FIG. 10, the integration processing unit 10 determines the coordinates (i, jmax) of the uppermost pixel p and the coordinates (i, jmin) of the lowermost pixel p of the group g integrated on the pixel column pls. In the same manner, the search for the pixel p and the integration process into the group g are performed while shifting the pixel column pl from the pixel column pls to the right and left directions on the reference image T. .

  As described above, the distance image Tz is created by assigning the parallax dp calculated by the distance detection unit 6 to each pixel p of the reference image T. Therefore, in the integration process of the pixel p into the group g using the parallax dp, for example, even if the pixel p satisfies the second condition, the parallax dp corresponding to the pixel p is adjacent. When the difference from the parallax dp corresponding to the pixel p to be processed is larger than the threshold value, the pixel p can be configured not to be integrated into the group g.

  On the other hand, in this embodiment, the integration processing unit 10 determines whether to integrate a pixel p into the group g in the determination of whether or not to integrate a certain pixel p into the group g. When the overall shape is not the circular shape that is the shape of the red signal Lr, the pixel p is not integrated into the group g, and the integration process is terminated at that time.

  Specifically, the integration processing means 10 is configured to monitor the number r of pixels from the uppermost pixel p to the lowermost pixel p of the group g in each pixel column pls, pl, for example, as shown in FIG. As shown in FIG. 11B, the number r of pixels in the vertical direction of the group g in each pixel column pls, pl once reaches the maximum value rmax, and the number of pixels r in the vertical direction of the group g in the pixel column pl is maximum as shown in FIG. If the value rmax falls below the value rmax, and the downward trend of the number of pixels r in the vertical direction of the group g in the pixel column pl continues as shown in FIG. 11C, the group g cannot be found in the pixel column pl. If no pixel p satisfying the condition is found, the search in the right and left pixel columns is stopped at that time.

  In addition, as shown in FIG. 12, when the number r of pixels in the vertical direction of the group g in the pixel row pl decreases to once reach the minimum value rmin, and then becomes a value larger than the minimum value rmin again, the reference image Since the entire shape of the group g on T is no longer circular, the search in the right and left pixel columns in the pixel column pl is stopped there, and the pixel column having the minimum value rmin immediately before is stopped. The range in which the group g is detected is set.

  When the integration processing unit 10 ends the integration processing for integrating the pixels p into the group g as described above, the integration processing means 10 repeats the integration processing until there is no pixel p that is not searched on the reference image T, and the reference image Each group g is integrated and formed on T.

  The distance calculation means 11 calculates the distance Zg in real space for each group g integrated by the integration processing means 10.

  The distance calculation unit 11 calculates, in the real space, for each group g integrated by the integration processing unit 10 based on each parallax dp assigned to each pixel p belonging to the group g. The average value or median value of each distance Z can be calculated as the distance Zg in the real space of the group g.

  However, there is a possibility that a parallax dp of 0 is assigned to each pixel p belonging to the group g in the filtering process in the distance detection means 6 (see FIG. 1) described above. In the method, the distance Zg in the real space of the group g cannot be calculated.

Therefore, in the present embodiment, the distance calculation unit 11 not only includes each distance Z in the real space corresponding to each pixel p belonging to the group g, but also around the group g on the reference image T as shown in FIG. Also, the distance Z in the real space corresponding to each pixel p ^* existing in is used, and the average value and median value of the distances Z in the real space of the pixels p and p ^* The distance Zg is calculated as a real space distance Zg.

Each distance Z in the real space of these pixels p and p ^* is voted on, for example, a histogram, and the class value of the class to which the mode belongs belongs to the distance Zg in the real space of the group g. It is also possible to configure as described above.

  Next, the red light detection means 12 determines the size of the group g in the real space and the road surface based on the distance Zg in the real space of each group g and the coordinates (i, j) in the image of the group g. The height in real space from the vehicle is calculated, and the red signal Lr (or yellow signal Ly) of the traffic light S is detected based on the calculated size in real space and the height from the road surface. .

  Specifically, as for the size of the group g in the real space, as shown in FIG. 14, the i coordinate Imin of the leftmost pixel p of the group g, the i coordinate Imax of the rightmost pixel p, and the upper end of the group g The j coordinate Jmax of the pixel p and the j coordinate Jmin of the pixel p at the lower end are detected, and the position (X, Y) in the real space corresponding to each pixel p is calculated according to the above equations (2) to (4). Then, the difference between the right end and the left end X coordinate is calculated to obtain the size of the group g in the horizontal direction, and the difference between the upper end and the lower end Y coordinate is calculated to calculate the size of the group g in the vertical direction in the real space. Say it.

  In a normal case, the signal lamp L of the traffic light S is formed in a circular shape having a diameter of about 30 cm or 35 cm. Therefore, in the present embodiment, including the leakage of light from the signal lamp, It is checked whether or not the size of each group g in the left-right direction and the up-down direction in real space is within a range of, for example, 40 cm.

  For the height in the real space from the road surface, the red signal detection means 12 uses the i-coordinate Imin of the leftmost pixel p and the rightmost pixel p of the group g, the i-coordinate Ic of the intermediate point Imax, A center point Gc of the group g is calculated with the intermediate points Jc between the j coordinates Jmax and Jmin of the upper end pixel p and the lower end pixel p as i coordinates and j coordinates, respectively. Then, the height Yc of the center point Gc of the group g in the real space is calculated by substituting Jc and the distance Zg of the group g in the real space into the above equation (3).

Further, the red signal detection means 12 reads the road height model (see FIG. 7B) of the road surface in the real space detected by the road surface detection means 15 from the memory, and in the real space of the group g described above. The height Y ^* of the road surface at the distance Zg is calculated. Then, the height in the real space from the road surface of the group g is calculated as a difference Yc−Y ^* between the height Yc in the real space of the center point Gc of the group g and the height Y ^* of the road surface. It has become.

In a normal case, the signal light of the traffic light is installed at a height of 5 m or more from the road surface. Therefore, in the present embodiment, the detection means 12 for detecting a red signal or the like is given a little width. In this way, it is checked whether or not the height Yc-Y ^* in the real space from the road surface of the group g calculated in this way is within a range of, for example, 4 m or more.

Note that the height Y ^* of the road surface at the distance Zg in the real space of the group g is obtained by linear interpolation from the road height model given by the above equations (7) and (8) for each section. Can do. Further, the lane model including the road height model is used if the lane model in the current sampling cycle is detected by the road surface detection means 15, and if the lane model in the current sampling cycle has not been detected yet, As described above, based on the lane model detected in the previous sampling cycle, the lane model in the current sampling cycle is estimated and used from the behavior of the host vehicle thereafter.

  Further, in the reference image T, a plurality of traffic lights S may be captured, and a plurality of red signals Lr may be captured. Therefore, in the present embodiment, the red signal detection unit 12 detects the position (Xc, Yc, Zg) of the center point Gc of the group g calculated as described above and the road surface detection unit 15. Compared with the horizontal shape model of the road surface in real space (refer to FIG. 7A), the traffic light S that is present on or near the traveling path on which the host vehicle travels and is closest to the host vehicle is selected. It is supposed to be.

  In the present embodiment, the red signal detection means 12 detects the group g that has cleared the above check items as the red signal Lr (or yellow signal Ly) of the traffic light S present on the road. .

  Next, the arrow signal extraction means 13 has an arrow signal Al, an arrow signal As, an arrow signal Ar (FIG. 28 (A) to (A)) in the vicinity of the red signal Lr (or yellow signal Ly) detected as described above. C) see) is searched for and extracted. Hereinafter, the processing in the arrow signal extraction means 13 will be specifically described with reference to the flowchart shown in FIG.

  The arrow signal extraction means 13 first picks up the arrow signals Al, As, Ar based on the position in the reference image T of the red signal Lr (or yellow signal Ly, the same applies hereinafter) detected as described above. An image area that may be present is set as the search area Rs (step S1).

  At that time, the arrow signals Al, As, Ar (hereinafter generally referred to as arrow signal A) are provided below the red signal Lr as shown in FIGS. As shown in FIG. 16, there is a case where an arrow signal A is provided on the side of a vertical traffic light S as shown in FIG.

  Therefore, in the present embodiment, the arrow signal extraction unit 13 sets the search areas Rs below and to the left and right sides of the group g corresponding to the red signal Lr in the reference image T as shown in FIG. It is like that.

  Then, the arrow signal extraction unit 13 performs an arrow signal extraction process for searching the search area Rs and extracting the arrow signal A imaged in the search area Rs (step S2).

  In this embodiment, the arrow signal extraction process (step S2) is performed as follows. In the following description, it is assumed that the luminance R, G, B of each color component in the image data D of each pixel p of the reference image T has a value of 0-255 as described above.

Specifically, since the arrow signal A is usually marked in green, the arrow signal extraction unit 13 searches each pixel p in the search region Rs, and each luminance R, G, B in the image data D is determined. For example,
G> 150 (14)
B> G × 0.6 (15)
R <G × 0.5 (16)
When this condition is satisfied, each pixel p is extracted by extracting the pixel p.

  When pixels p that are adjacent to the extracted pixel p in the vertical and horizontal directions are also extracted, the pixel p is set as one pixel region Ra, so that the search signal R corresponds to the arrow signal A. The pixel region Ra to be extracted is extracted.

  For example, in the traffic light S shown in FIG. 28A, since the arrow signal Al indicating that the vehicle can turn left and the arrow signal As indicating that the vehicle can go straight may be lit simultaneously, the arrow signal A extracted within the search region Rs. The pixel area Ra corresponding to is not limited to one. Further, as shown in FIG. 31, when the traffic light S is at a position far from the host vehicle, for example, only one pixel p may be extracted as the pixel region Ra corresponding to the arrow signal A.

  If the arrow signal extraction means 13 does not extract the arrow signal A in the search region Rs in the arrow signal extraction process (step S2) (step S3; NO), the arrow signal extraction means 13 ends the process in the current sampling cycle and performs the next sampling. The system waits until information such as the group g corresponding to the red signal Lr of the traffic light S is sent from the red signal detection means 12 or the like in a cycle.

  Further, when the arrow signal extraction unit 13 extracts the arrow signal A in the search region Rs by the arrow signal extraction process (step S2) (step S3; YES), the direction indicated by the extracted arrow signal A is subsequently indicated. A process for recognizing (hereinafter simply referred to as the direction of the arrow signal A) is performed.

  In this case, in the present invention, as shown in FIG. 18, the arrow signal extraction means 13 is a distance Z in the real space from the host vehicle MC (more precisely, the imaging means 2 mounted on the host vehicle) to the red signal Lr. Are divided into a first predetermined section Sec1 far from the host vehicle MC, a second predetermined section Sec2 closer to the host vehicle than the first predetermined section, and a third predetermined section Sec3 closer to the host vehicle than the second predetermined section. In each of the sections Sec1 to Sec3, the method for recognizing the direction of the arrow signal A is changed.

  In other words, it is difficult to recognize the direction of the arrow signal A in the recognition methods 2 and 3 described below, but the direction of the arrow signal A can be recognized using the recognition method 1. The section is set to the first predetermined section Sec1, and it is difficult to recognize the direction of the arrow signal A by the recognition method 3, but the section in which the direction of the arrow signal A can be recognized by using the recognition method 2. Is set as the second predetermined section Sec2, and a section in which the direction of the arrow signal A can be recognized using the recognition method 3 is set as the third predetermined section Sec3.

  Then, as shown in the flowchart of FIG. 15, the arrow signal extraction unit 13 sets the distance Zg in the real space to the group g (see FIG. 17) corresponding to the red signal Lr calculated by the distance calculation unit 11 described above. By dividing into sections Sec1 to Sec3, the direction of the extracted arrow signal A is recognized using recognition methods 1 to 3 corresponding to the respective predetermined sections Sec1 to Sec3.

  First, when the distance Zg in the real space from the host vehicle MC to the red signal Lr is not in the third predetermined section Sec3 close to the host vehicle MC (step S4; NO) and not in the second predetermined section Sec2. (Step S5; NO), that is, recognition method 1 (Step S6) applied when the distance Zg in real space from the host vehicle MC to the red signal Lr is in the first predetermined section Sec1 far from the host vehicle MC. explain.

  When the distance Zg in real space from the host vehicle MC to the red signal Lr is in the first predetermined section Sec1 far from the host vehicle MC, the red signal Lr and the arrow signal A are, for example, as shown in FIG. The number of pixels belonging to the group g and the pixel area Ra corresponding to is very small, and all of them are imaged in a state where the outline and the like are unclear. In particular, the arrow signal A is captured in a state where the direction of the arrow signal A cannot be recognized from the captured reference image T.

  In the recognition method 1 (step S6), as shown in the flowchart of FIG. 19, first, the arrow signal extraction unit 13 performs the above processing on the real space of the group g corresponding to the red signal Lr calculated by the distance calculation unit 11 as described above. Based on the distance Zg of the red signal Lr, when a signal lamp having a diameter of, for example, 35 cm exists in the distance Zg, the number of pixels corresponding to the diameter on the reference image T is calculated. The number of pixels r corresponding to the diameter of the signal lamp is calculated (step S61).

  In the process of step S61, instead of configuring to calculate the number r of pixels corresponding to the diameter of the red signal Lr based on the distance Zg in the real space of the group g corresponding to the red signal Lr, the processing is actually performed. The width in the vertical direction (that is, the j-axis direction) or the horizontal direction (that is, the i-axis direction) on the reference image T of the group g corresponding to the detected red signal Lr is defined as the number of pixels r corresponding to the diameter of the red signal Lr. It can also be configured to calculate.

  Subsequently, the arrow signal extraction unit 13 recognizes the direction of the arrow signal A based on the mutual positional relationship between the pixel region Ra corresponding to the arrow signal A in the reference image T and the group g corresponding to the red signal Lr. (Step S62).

At that time, in the actual traffic light S, the positional relationship between the red signal Lr and the arrow signals Al, As, Ar indicating whether the vehicle can turn right, go straight, or turn left is as shown in FIGS. 28 (A) to (C). In this embodiment, as shown in FIG.
(1) When the pixel area Ra corresponding to the extracted arrow signal A is at a position about one signal light of the group g corresponding to the red signal Lr, the arrow signal Ar indicating that the right turn is possible,
(2) If the pixel area Ra corresponding to the extracted arrow signal A is at a position about one signal lamp on the left side from the position about one signal lamp of the group g corresponding to the red signal Lr, it is possible to go straight ahead. Arrow signal As shown,
(3) When the pixel area Ra corresponding to the extracted arrow signal A is located at the left side of about two signal lamps from the position below about one signal lamp of the group g corresponding to the red signal Lr, the left turn is allowed. Arrow signal Al,
As a result, the direction of the extracted arrow signal A is recognized.

Further, in the vertical traffic light S as shown in FIG. 16, when the arrow signal A is on the right side of the red signal Lr, the arrow signal A is the arrow signal Ar indicating that the right turn is possible, and on the left side of the red signal Lr. When there is an arrow signal A (not shown), the arrow signal A is often an arrow signal Al indicating that it is possible to turn left. Therefore, in this embodiment, the arrow signal extraction means 13 is further configured as shown in FIG. In addition,
(4) When the pixel area Ra corresponding to the extracted arrow signal A is on the right side of about one signal lamp of the group g corresponding to the red signal Lr, the arrow signal Ar indicating right turn is possible.
(5) When the pixel area Ra corresponding to the extracted arrow signal A is at a position on the left side of about one signal lamp of the group g corresponding to the red signal Lr, the arrow signal Al indicating that the left turn is possible,
As a result, the direction of the extracted arrow signal A is recognized.

  As described above, when the distance Zg in the real space from the host vehicle MC to the red signal Lr is in the first predetermined section Sec1 far from the host vehicle MC, for example, as shown in FIG. In many cases, it is difficult to recognize the direction of the arrow signal A even when analyzing the shape of the pixel region Ra.

  However, according to the recognition method 1 (step S6) in the present embodiment, the direction of the arrow signal A is recognized based on the shape of the arrow signal A captured on the reference image T as described above. Even if this is not possible, the group g corresponding to the red signal Lr in the reference image T and the pixel region Ra corresponding to the arrow signal A in the reference image T are determined based on the position of the detected red signal Lr on the reference image T. The direction of the arrow signal A is recognized from the positional relationship.

  Therefore, although there is a premise that the traffic light S or red signal Lr captured in the reference image T is at least a general traffic light S as shown in FIGS. 28 (A) to (C) or FIG. The direction of the arrow signal A in the traffic light S can be accurately recognized.

  Next, although the distance Zg in real space from the host vehicle MC to the red signal Lr is not within the second predetermined section Sec3 close to the host vehicle MC (step S4 in FIG. 15; NO), the first predetermined section Sec1 The recognition method 2 (step S7) applied when it is in the second predetermined section Sec2 closer to the host vehicle (step S5; YES) will be described.

  When the distance Zg in real space from the host vehicle MC to the red signal Lr is in the second predetermined section Sec2 (see FIG. 18), the red signal Lr and the arrow signal A are blurred as shown in FIG. 30, for example. Images are taken in a state. The arrow signal A cannot be clearly distinguished from an arrow head portion Ahead and an arrow shaft portion Ashaft (see FIG. 26 and the like to be described later), but barely recognizes the direction of the arrow signal A from the captured reference image T. The image is captured in a state where it can be performed.

  In the recognition method 2 (step S7), processing is performed according to the flowchart shown in FIG. Here, as a method for recognizing the direction of the arrow signal A, the direction of the arrow signal A is recognized based on the left-right symmetry and the vertical symmetry of the arrow signal A in the reference image T.

  As shown in FIG. 23, the arrow signal extraction means 13 first sets, for example, a rectangular image area Ae circumscribing the pixel area Ra corresponding to the arrow signal A in the reference image T (step S71). The image area Ae is divided into left and right by a straight line l1 extending in the vertical direction passing through the center Ce of the image area Ae (step S72).

  The left and right symmetry of the pixel area Aa corresponding to the arrow signal A is determined by comparing the number of pixels belonging to the pixel area Aa corresponding to the arrow signal A included in each of the divided image areas Ae1 and Ae2. It is supposed to be.

  Specifically, the arrow signal extraction unit 13 determines the number of pixels r (Ae1) belonging to the pixel area Aa corresponding to the arrow signal A included in the left image area Ae1 and the arrow signal A included in the right image area Ae2. The absolute value | r (Ae1) −r (Ae2) | of the difference from the number of pixels r (Ae2) belonging to the pixel area Aa corresponding to is calculated.

  If the absolute value | r (Ae1) −r (Ae2) | of the difference is less than the threshold value (step S73; YES), it is recognized that the direction of the arrow signal A is the straight direction (step S74). That is, it is recognized that the arrow signal A that is the object of the current determination is the arrow signal As that indicates that the vehicle can go straight. In this case, the threshold value is set, for example, as 10% of the total number of pixels in the pixel area Aa corresponding to the arrow signal A.

  In the example shown in FIG. 23, the absolute value | r (Ae1) of the difference between the number of pixels r (Ae1) and r (Ae2) belonging to the pixel area Aa corresponding to the arrow signal A included in each of the image areas Ae1 and Ae2. -R (Ae2) | is one pixel. Further, since the total number of pixels in the pixel area Aa corresponding to the arrow signal A is 29 pixels, 10% of which is 2.9 pixels, the determination criterion of Step S73 is satisfied. Therefore, in this case, it is recognized that the direction of the arrow signal A is the straight direction (step S74).

  On the other hand, for example, in the example shown in FIG. 24, the absolute value | r () of the difference between the number of pixels r (Ae1) and r (Ae2) belonging to the pixel area Aa corresponding to the arrow signal A included in the image areas Ae1 and Ae2 Since Ae1) −r (Ae2) | exceeds 10% of the total number of pixels in the pixel area Aa corresponding to the arrow signal A, the determination criterion of step S73 is not satisfied (step S73; NO).

  Therefore, the arrow signal extraction unit 13 subsequently determines that the number of pixels r (Ae1) belonging to the pixel area Aa corresponding to the arrow signal A included in the left image area Ae1 is the arrow signal included in the right image area Ae2. It is determined whether or not the number of pixels r (Ae2) belonging to the pixel area Aa corresponding to A is larger than the above threshold value (step S75). If the determination criterion of step S75 is satisfied (step S75; YES), it is recognized that the direction of the arrow signal A is the left turn direction (step S76). That is, the arrow signal A is recognized as an arrow signal Al indicating that a left turn is possible.

  However, in the example shown in FIG. 24, conversely, the number r (Ae2) of pixels belonging to the pixel area Aa corresponding to the arrow signal A included in the right image area Ae2 is included in the left image area Ae1. Since the number r (Ae1) of pixels belonging to the pixel area Aa corresponding to the arrow signal A is larger than the above threshold (step S75; NO), the arrow signal extraction means 13 determines that the direction of the arrow signal A is the right turn direction. (Step S77). That is, the arrow signal A is recognized as an arrow signal Ar indicating that a right turn is possible.

  In the flowchart shown in FIG. 22 and each figure shown in FIGS. 23 and 24, the image area Ae circumscribing the pixel area Ra corresponding to the arrow signal A set in the reference image T is represented by the image area Ae. A case has been described in which the left-right symmetry of the pixel area Aa corresponding to the arrow signal A is determined by dividing the line area left and right by a straight line 11 extending in the up-down direction passing through the center Ce.

  However, although not shown, the image area Ae circumscribing the pixel area Ra corresponding to the arrow signal A is divided vertically by a straight line 12 extending in the left-right direction passing through the center Ce of the image area Ae. Even when the vertical symmetry of the pixel area Aa corresponding to the signal A is determined, the direction of the arrow signal A can be recognized as described above.

  As described above, when the distance Zg in real space from the host vehicle MC to the red signal Lr is in the second predetermined section Sec2, for example, as shown in FIG. 30, the arrow signal A is imaged in a blurred state. As will be described later in recognition method 3 (step S8), the pixel area Aa corresponding to the arrow signal A is clearly distinguished and analyzed into Ahead and an arrow shaft portion Ashaft (see FIG. 26 described later). The direction of A cannot be recognized.

  However, according to the recognition method 2 (step S7) in the present embodiment, the left and right symmetry and the vertical symmetry of the pixel area Aa corresponding to the arrow signal A imaged on the reference image T as described above. Based on the above, the direction of the arrow signal A can be accurately recognized.

  Next, the recognition method 3 applied when the distance Zg in real space from the host vehicle MC to the red signal Lr is within the third predetermined section Sec3 close to the host vehicle MC (step S4 in FIG. 15; YES). (Step S8) will be described.

  When the distance Zg in real space from the host vehicle MC to the red signal Lr is in the third predetermined section Sec3 (see FIG. 18) close to the host vehicle MC, the red signal Lr and the arrow signal A are shown in FIG. 29, for example. As shown in FIG. 26 to be described later, the arrow signal A is imaged in a state where the arrow head portion Ahead and the arrow shaft portion Ashaft are clearly distinguished.

  Therefore, in the recognition method 3 (step S8), as a method of recognizing the direction of the arrow signal A, each of the pixel regions Rahead in the reference image T corresponding to the arrow head portion Ahead and the arrow shaft portion Ashaft that form the arrow signal A, The direction of the arrow signal A is recognized based on the mutual positional relationship of Rashaf.

  In recognition method 3 (step S8), processing is performed according to the flowchart shown in FIG.

  As shown in FIG. 26, the arrow signal extraction means 13 compares the number of pixels belonging to each pixel area in a set of pixel areas Ra corresponding to the arrow signal A extracted in the reference image T, and is larger. The pixel region Rahead corresponding to the arrow head Ahead of the arrow signal A (that is, the pixel region having a large number of pixels belonging) is defined as the pixel region Rahead, and the smaller pixel region (that is, the pixel region having a small number of pixels belonging to) Is identified as a pixel region Rashaft corresponding to the arrow shaft portion Ashaft (step S81).

  Whether the two pixel regions Ra corresponding to the arrow signal A extracted in the reference image T are a set of pixel regions Ra corresponding to the arrow head portion Ahead and the arrow shaft portion Ashaft belonging to the same arrow signal A. Alternatively, it is necessary to determine whether each pixel area Ra corresponds to each separate arrow signal A.

  In this case, if the two pixel regions Ra belong to the same arrow signal A, the pixel regions Ra are in real space, for example, a circle having a diameter of 35 cm, as in the case of the red signal Lr described above. Should fit within. Therefore, for example, referring to the distance image Tz described above, the parallax dp assigned to the pixels belonging to the two pixel regions Ra is calculated, and the distance Z in the real space between the two pixel regions Ra is calculated.

  Then, based on the calculated distance Z, it is determined whether or not the two pixel regions Ra are within a circle having a diameter of, for example, 35 cm in real space, so that the two pixel regions Ra have the same arrow signal A. It can be determined whether it belongs or not. When it is determined that the two pixel regions Ra belong to different arrow signals A, they are processed separately.

  When the arrow signal extraction unit 13 determines on the reference image T the pixel region Rahead corresponding to the arrow head portion Ahead of the arrow signal A and the pixel region Rashaft corresponding to the arrow shaft portion Ashaft, then, as shown in FIG. As described above, the coordinates (i, j) of the center points Chead and Cshaft of the pixel regions Rahead and Rashaft are calculated (step S82).

  For the center point, for example, the i coordinate is the average value of the left end coordinate and the right end coordinate of the pixel area (that is, the coordinate between the left and right ends), and the j coordinate is the upper end coordinate and the lower end coordinate of the pixel area. It can be calculated as an average value (that is, coordinates between the upper and lower ends).

  The direction of the arrow signal A is recognized based on the directionality of the vector v connecting the center points Chead and Cshaft (step S83).

  That is, for example, a vector from the center point Cshaft to the center point Chead is calculated as the vector v, and an angle formed by the i-axis of the reference image T is calculated. If the calculated angle is about 90 ° (that is, in the case of FIG. 26), the direction of the arrow signal A is recognized as the straight direction (that is, the arrow signal A is recognized as the arrow signal As indicating that the straight travel is possible). To do).

  If the calculated angle is about 0 °, the direction of the arrow signal A is recognized as a right turn direction, and if the calculated angle is about 180 °, the direction of the arrow signal A is recognized as a left turn direction. Configured to do.

  If this detection method 3 is used, as shown in FIG. 27, the center point Chead, even if the arrow signal A is somewhat irregular, for example, an arrow signal indicating that the vehicle is allowed to travel diagonally to the left is displayed. Based on the directionality of the vector v connecting the Cshafts, the direction of the arrow signal A can be accurately recognized. In other words, in this case, since the angle formed by the vector v and the i-axis of the reference image T is about 135 °, the arrow signal extraction means 13 accurately determines that the direction of the arrow signal A is the diagonally left front direction. It becomes possible to recognize.

  As described above, when the distance Zg in the real space from the host vehicle MC to the red signal Lr is in the third predetermined section Sec3 close to the host vehicle MC, for example, as shown in FIG. Imaging is performed by distinguishing between an arrow head portion Ahead and an arrow shaft portion Ashaft (see FIG. 26 and the like).

  Therefore, the recognition method 3 (step S8) is used to recognize the direction of the arrow signal A, whereby each of the arrows in the reference image T corresponding to the arrow head portion Ahead and the arrow shaft portion Ashaft that forms the arrow signal A. Based on the mutual positional relationship between the pixel regions Rahead and Rashaft, the direction of the arrow signal A can be accurately recognized.

  If the distance Zg in real space from the host vehicle MC to the red signal Lr is in the third predetermined section Sec3 (see FIG. 18) close to the host vehicle MC, for example, as shown in FIG. Since the signal A is clearly imaged, as described in Patent Document 2 described above, the direction of the arrow signal A can be recognized by template matching.

  However, the method using template matching often takes a long time to process. For example, when the direction information of the arrow signal A recognized by the arrow signal extraction unit 13 is used to perform automatic traveling control of the own vehicle, for example. May impair real-time performance.

  Further, if a template is prepared for each of the arrow signals Al, As, and Ar (see FIG. 28) indicating whether the vehicle can turn right, go straight, or turn left, the processing becomes heavy. For example, as shown in FIG. If a template corresponding to the irregular arrow signal A is not prepared, there may be a problem that the direction of the arrow signal A cannot be recognized.

  On the other hand, the above recognition method 3 is adopted, and based on the mutual positional relationship between the pixel areas Rahead and Rashaft in the reference image T corresponding to the arrow head part Ahead and the arrow shaft part Ashaft forming the arrow signal A. If the configuration is such that the direction of the arrow signal A is accurately recognized, the processing does not become very heavy, real-time characteristics are secured, and an irregular arrow signal A as shown in FIG. 27 is displayed. Even if it exists, it becomes possible to recognize the direction of the arrow signal A exactly.

  As described above, the arrow signal extraction unit 13 applies the recognition method 1 (step S6 in FIG. 15), the recognition method 2 (step S7), or the recognition method 3 (step S8) to the reference image T. When the direction of the extracted arrow signal A (that is, the direction indicated by the arrow signal A) is recognized, the recognized direction is stored in the memory and output to the outside (step S9).

  Then, the arrow signal extraction unit 13 ends the processing in the current sampling cycle, and information such as the group g corresponding to the red signal Lr of the traffic light S is transmitted from the red signal detection unit 12 or the like in the next sampling cycle. Wait until it comes.

  As described above, according to the environment recognition device 1 according to the present embodiment, the first predetermined section Sec1 (at least the distance Zg in the real space to the red signal Lr and the yellow signal Ly of the traffic light S is far from the host vehicle MC. 18), the above recognition method 1 (step S6) is applied, and the arrows based on the mutual positional relationship between the arrow signal A and the red signal Lr and the yellow signal Ly in the image T are applied. The direction indicated by the signal A is recognized.

  For this reason, the traffic light S (for example, see FIG. 31) located at a very far position from the host vehicle MC that could not detect the direction indicated by the arrow signal A by the conventional template matching cannot be detected. It becomes possible to accurately recognize and detect the direction indicated by the signal A.

  Therefore, from the stage where the traffic light S is still far from the host vehicle MC, the arrow signal of the traffic signal S is accurately recognized, and the information is used for automatic travel control of the host vehicle, or the driver is appropriately warned. It is possible to perform control such as

  When the distance Zg in real space from the host vehicle MC to the red signal Lr and the yellow signal Ly is in the second predetermined section Sec2 (see FIG. 18), the arrow signal A is, for example, as shown in FIG. Since the image is taken in a blurred state, it is difficult to detect the direction by the conventional method of detecting the direction indicated by the arrow signal A by template matching.

  However, according to the environment recognition apparatus 1 according to the present embodiment, when the distance Zg in real space to the red signal Lr and the yellow signal Ly of the traffic light S is within the second predetermined section Sec2, By applying the recognition method 2 (step S7), the direction of the arrow signal A is recognized based on the left-right symmetry and the vertical symmetry of the arrow signal A in the image T.

  Therefore, for the traffic light S (see FIG. 30 for example) located relatively far from the host vehicle MC, which is difficult to detect the direction indicated by the arrow signal A by conventional template matching. It becomes possible to accurately recognize and detect the direction indicated by the arrow signal A.

  Therefore, from the stage where the traffic light S is still relatively far from the host vehicle MC, the arrow signal of the traffic signal S is accurately recognized and the information is used for automatic driving control of the host vehicle, etc. It is possible to perform control such as warning.

  Moreover, according to the environment recognition apparatus 1 which concerns on this embodiment, the distance Zg in real space to the red signal Lr of the traffic light S and the yellow signal Ly is the 3rd predetermined area Sec3 (refer FIG. 18) near the own vehicle MC. If it is within, the above-described recognition method 3 (step S8) is applied to each of the pixel regions Rahead and Rashaf in the reference image T corresponding to the arrow head portion Ahead and the arrow shaft portion Ashaft that form the arrow signal A. The direction of the arrow signal A is recognized based on the mutual positional relationship.

  Therefore, it is possible to accurately recognize and detect the direction indicated by the arrow signal A for the traffic light S (for example, see FIG. 29) located near the host vehicle MC, and the traffic signal S is located near the host vehicle MC. In this case, it is possible to accurately recognize the arrow signal of the traffic light S and use the information for automatic traveling control of the host vehicle or to control the driver appropriately.

  At the same time, the conventional method using template matching makes processing heavy, and processing may take time and real-time performance may be impaired. Also, a template corresponding to an irregular arrow signal A is prepared. If not, there may be a problem that the direction of the arrow signal A cannot be recognized. However, if the above-described recognition method 3 is adopted, the processing does not become very heavy, real-time properties are ensured, and irregularities are obtained. Even when a simple arrow signal A is displayed, the direction of the arrow signal A can be accurately recognized.

  In the above embodiment, as shown in the flowchart of FIG. 15, the arrow signal extraction unit 13 recognizes according to the distance Zg in real space from the host vehicle MC to the red signal Lr and the yellow signal Ly. The case where it is comprised so that any one of methods of ~ 3 may be applied was explained.

  However, although not shown, for example, the arrow signal extraction unit 13 uses the recognition method 1 to recognize the direction indicated by the arrow signal A and the recognition method 2 uses the recognition method 2 to indicate the arrow signal A. A second recognizing unit for recognizing the direction and a third recognizing unit for recognizing the direction indicated by the arrow signal A using the recognizing method 3 are provided. When information on the corresponding group g or the like is sent, processing is performed in parallel by the three recognition means.

  The arrow signal extraction unit 13 is provided with a selection unit that selects and outputs the direction indicated by the arrow signal A recognized by any one of the three recognition units. The selection unit includes a red signal Lr and a yellow signal. The first predetermined section Sec1 in which the distance Zg in real space to the signal Ly is far from the host vehicle MC, the second predetermined section Sec2 closer to the host vehicle MC than the first predetermined section Sec1, or the first predetermined section Sec1. Depending on which predetermined section of the third predetermined section Sec3 that is closer to the host vehicle MC than which of the recognition means recognizes, the direction to be selected and output may be switched. Is possible.

  Even if comprised in this way, it becomes possible to obtain the same beneficial effect as the case of this embodiment mentioned above.

  Further, for example, when there are a plurality of traffic lights S on or in the vicinity of the lane model formed by the road surface detection means 15 described above (see the above formulas (5) to (8)), each of the traffic lights S in the image T is displayed. May be imaged.

  In such a case, it is natural that the direction indicated by the arrow signal A in the traffic light S closest to the host vehicle MC is recognized, but other traffic signals farther from the host vehicle MC are recognized. It is also possible to configure so as to simultaneously recognize the direction indicated by the arrow signal A in S.

  In that case, whether or not to output the direction indicated by the arrow signal A of the traffic light S other than the traffic light S closest to the front side is performed by using the output information to perform automatic traveling control of the host vehicle or issue a warning to the driver. This is appropriately determined based on whether or not the information is required on the external device side.

DESCRIPTION OF SYMBOLS 1 Environment recognition apparatus 2 Imaging means 6 Distance detection means 10 Integration processing means 11 Distance calculation means 12 Red signal etc. detection means 13 Arrow signal extraction means 15 Road surface detection means A, Al, As, Ar Arrow signals Ae1, Ae2 Divided images Area Ahead Arrow head part Ashaft Arrow shaft part Ce Center Chead, Cshaft center point D Image data g Group (i, j) Coordinates l1, l2 Straight line Lr Red signal (red signal light)
Ly yellow signal (yellow signal light)
MC own vehicle p pixel p ^* pixels r (Ae1), r (Ae2) existing around the group Ra pixel area corresponding to the arrow signal Rahead pixel area corresponding to the arrow head part pixel corresponding to the Rashaft arrow shaft part Area Re circumscribed image area Rs search area S traffic light
Sec1 First predetermined section
Sec2 Second predetermined section
Sec3 Third predetermined section T Reference image (image)
v Vector Y connecting each center point Height in real space Y ^* Height in road real space Yc Height in group real space Yc-Y ^* Group height in road from road Z Distance in real space Zg Distance in group real space

Claims (8)

Imaging means for imaging an image having image data for each pixel;
Distance detecting means for detecting a distance in real space for each pixel of the image;
Integration processing means for integrating adjacent pixels into one group based on image data of adjacent pixels in the image;
Distance calculating means for calculating a distance in the real space of the group based on the distance in the real space corresponding to the pixels belonging to the group and the pixels existing around the group;
Based on the distance of the group in the real space and the size of the group in the real space and the height in the real space from the road surface calculated based on the coordinates in the image of the group, Or a detection means such as a red signal for detecting a yellow signal light;
Based on the detected position of the red or yellow signal lamp in the image, an image area in which an arrow signal may be captured is set as a search area, and the arrow signal captured in the search area Arrow signal extraction means for extracting
With
The arrow signal extraction means includes
It is configured to change the method of recognizing the direction indicated by the arrow signal according to the distance in the real space from the own vehicle to the red or yellow signal light,
When the distance in the real space to at least the red or yellow signal light is within a first predetermined section far from the host vehicle, the arrow signal and the red or yellow signal light in the image An environment recognition apparatus that recognizes the direction indicated by the arrow signal based on the positional relationship of   The arrow signal extraction means is configured such that when the distance in the real space to at least the red or yellow signal lamp is within a second predetermined section closer to the host vehicle than the first predetermined section, the arrow signal is The environment recognition apparatus according to claim 1, wherein, as a method for recognizing a pointing direction, the direction indicated by the arrow signal is recognized based on left-right or vertical symmetry of the arrow signal in the image.   The arrow signal extraction means sets an image region circumscribing a pixel region corresponding to the arrow signal in the image, and the image region is a straight line extending in the vertical direction or the horizontal direction passing through the center of the image region. The left-right or up-down symmetry is determined by comparing the number of pixels belonging to a pixel area corresponding to the arrow signal included in each of the divided image areas. The environment recognition device described in 1.   The arrow signal extraction means is configured such that when the distance in the real space to at least the red or yellow signal light is within a third predetermined section closer to the host vehicle than the second predetermined section, the arrow signal is As a method for recognizing the pointing direction, the direction indicated by the arrow signal is recognized based on the mutual positional relationship between the pixel regions in the image corresponding to the arrow head portion and the arrow shaft portion that form the arrow signal. The environment recognition device according to claim 2 or claim 3, wherein   The arrow signal extraction means identifies each pixel area in the image corresponding to the arrow head part and the arrow shaft part forming the arrow signal by comparing the number of pixels belonging to each pixel area. The environment recognition apparatus according to claim 4.   The arrow signal extraction means calculates each center point of each pixel region corresponding to each of the arrow head part and the arrow shaft part, and based on the directionality of the vector connecting the center points, 6. The environment recognition apparatus according to claim 4, wherein the direction indicated by the signal is recognized. Furthermore, a road surface detection means for detecting a road surface from the image,
The detection means such as a red signal is calculated based on the distance of the group in the real space and the coordinates of the group in the image, and the size of the group in the real space, and the road detected by the road surface detection means. The environment recognition device according to any one of claims 1 to 6, wherein a red or yellow signal light of a traffic light is detected based on a height of the surface in real space. Imaging means for imaging an image having image data for each pixel;
Distance detecting means for detecting a distance in real space for each pixel of the image;
Integration processing means for integrating adjacent pixels into one group based on image data of adjacent pixels in the image;
Distance calculating means for calculating a distance in the real space of the group based on the distance in the real space corresponding to the pixels belonging to the group and the pixels existing around the group;
Based on the distance of the group in the real space and the size of the group in the real space and the height in the real space from the road surface calculated based on the coordinates in the image of the group, Or a detection means such as a red signal for detecting a yellow signal light;
Based on the detected position of the red or yellow signal lamp in the image, an image area in which an arrow signal may be captured is set as a search area, and the arrow signal captured in the search area Arrow signal extraction means for extracting
With
The arrow signal extraction means includes a plurality of recognition means for recognizing the direction indicated by the arrow signal by different methods as a method for recognizing the direction indicated by the arrow signal, and the actual signal from the own vehicle to the red or yellow signal lamp is provided. An environment recognition apparatus characterized by selecting and outputting the direction recognized by any of the plurality of recognition means according to a distance in space.