JP5922947B2 - Outside environment recognition device and outside environment recognition method - Google Patents

Description

  The present invention relates to a vehicle environment recognition apparatus and a vehicle environment recognition method for recognizing an environment outside a host vehicle.

  Conventionally, a specific object such as a vehicle or an obstacle positioned in front of the host vehicle is detected, and a collision with the detected specific object is avoided (collision avoidance control), or the distance between the preceding vehicle and the preceding vehicle is kept at a safe distance. Such a control (cruise control) technique is known (for example, Patent Document 1).

  Many of these technologies are configured to track the preceding vehicle, update the position information, and calculate the moving speed of the preceding vehicle, and this result is reflected in, for example, collision avoidance control and cruise control. . In addition, in order to smoothly perform such control, the vehicle environment other than the preceding vehicle, for example, specific objects such as traffic lights and road signs, are recognized to support the traveling of the host vehicle, or the brake lamp of the preceding vehicle is recognized. In preparation for sudden deceleration of the preceding vehicle.

  Such a specific object is specified based on an image obtained by capturing an environment outside the vehicle in front of the host vehicle. For example, a technology is known that captures a color image outside a vehicle, groups adjacent pixels having the same luminance (color), and recognizes a light source such as a red light of a traffic light as a specific object (for example, a patent) Literature 2, 3).

Japanese Patent No. 3349060 JP 2005-301518 A JP 2010-224924 A

  In order to smoothly perform the collision avoidance control and cruise control described above, for example, it is necessary to obtain more accurate traffic information necessary for driving the vehicle through the exterior environment such as red, yellow, and blue lights of traffic lights. There is. And if the red light is lit in the traffic light immediately before the host vehicle, it is conceivable to support the driver's braking operation.

  However, even if the red light of the traffic light is lit, there is no need for braking depending on the traveling direction of the own vehicle as long as the blue light of the arrow traffic light attached (attached) is lit. Therefore, in collision avoidance control and cruise control, the blue light of the arrow traffic light is grasped in addition to the red light of the traffic light, and when both signals are lit, support for the braking operation is suppressed. However, when only the blue light of the arrow traffic light is shielded by the preceding vehicle or the like under the situation where the red light of the traffic light and the blue light of the arrow traffic light are lit, only the red light of the traffic light can be recognized, Support for braking operation may be performed unnecessarily. As described above, when an operation contrary to the driver's intention occurs, the driver feels bothered by the operation, and there is a possibility that attention regarding traffic may be lost. Therefore, even if the traffic light is a red light, setting may be made so as to suppress the support of the braking operation if there is a possibility that the blue light of the arrow traffic light attached thereto is lit.

  The possibility that the blue light of the arrow traffic light attached to the red light of the traffic light is lit is that there are areas where the blue light of the arrow traffic light may exist and certain things that may be located in front of the host vehicle. It is sufficient to compare with the occupied area. However, in the collision avoidance control and the cruise control, in order to improve the processing efficiency, the specific object to be tracked may be limited to a part such as the immediately preceding vehicle and the specific object may be controlled. Therefore, there is a case where it is determined that there is no potential even if the blue light of the arrow traffic signal may be latent only by comparing the traffic light with the immediately preceding vehicle.

  Furthermore, in the case of a vehicle having a loading platform such as a truck, only the rear end portion of the loading platform is tracked, and the blue light of the arrow traffic light is latent with the area occupied by the rear portion of the loading platform even though there is a cabin. There was a case where it was misjudged that there was no possibility of doing.

  In view of such a problem, the present invention determines with high accuracy the possibility that an arrow traffic light is illuminated even when only the specific object immediately before is tracked, and is based on collision avoidance control and cruise control. An object of the present invention is to provide a vehicle exterior environment recognition device and a vehicle exterior environment recognition method capable of avoiding unnecessary intervention in braking operation.

In order to solve the above-described problem, an external environment recognition apparatus according to the present invention includes an image acquisition unit that acquires an image obtained by capturing an external environment, and a position for deriving a relative distance of an object corresponding to the partial image of the image with respect to the host vehicle. An information deriving unit; a first grouping unit that identifies one or more specific objects based on a relative distance; a specific object selection unit that selects one specific object to be tracked from the one or more specific objects; The second grouping unit that identifies stop signal lights that indicate stop or inability to proceed based on the color and size in the image, and the stop based on the positional relationship between the selected specific object that is the selected specific object and the stop signal light Stop if the first possibility determination unit that determines the possibility that the progress signal lamp indicating the progress permission attached to the signal lamp is latent and the first possibility determination unit has not determined the possibility that the progress signal lamp is latent Signal light Relative distance in a short space Ri, determines the possibility of exploring the shield that shields the signal search area defined based on the stop signal lamp, progressive signal lamp based on the occupied area ratio of the shield in the signal search area potentially A second possibility determination unit.

  The second possibility determination unit may determine that there is a possibility that the progress signal lamp is latent if the occupied area ratio of the shielding object in the signal search region is equal to or greater than a predetermined threshold.

  The second possibility determination unit may search for a shielding object that shields the signal search region only in a predetermined relative distance range with the selected specific object as a reference, and determine the possibility that the progress signal lamp is latent.

  The first possibility determination unit may determine the possibility that the progress signal lamp is latent based on the horizontal position and height of the selected specific object and the stop signal lamp.

In order to solve the above-described problem, an external environment recognition method of the present invention acquires an image obtained by capturing an external environment, derives a relative distance of an object corresponding to a partial image of the image with respect to the host vehicle, and is based on the relative distance. Identify one or more specific objects, select one specific object to be tracked from one or more specific objects, and identify a stop signal light that indicates stop or no progress based on the color and size in the image Then, based on the positional relationship between the selected specific object, which is the selected specific object, and the stop signal light, the possibility of the progress signal light indicating the progress permission attached to the stop signal light is determined, and the progress signal light may be latent depending on the determination. when unable to sex determination, in the relative distance is shorter space than the stop signal lamp, searching the shield that shields the predetermined signal search area based on the stop signal lamp, put the signal search area Progressive signal light based on the occupied area ratio of the shield is characterized by determining the possibility of potential.

  According to the present invention, even if only the immediately preceding specific object is to be tracked, it is possible to determine with high accuracy the possibility that the arrow traffic light is on, and to perform unnecessary braking operation by collision avoidance control or cruise control. It is possible to avoid intervention.

It is the block diagram which showed the connection relation of the environment recognition system. It is explanatory drawing for demonstrating a luminance image and a distance image. It is the functional block diagram which showed the schematic function of the external environment recognition apparatus. It is explanatory drawing for demonstrating a color identifier table. It is explanatory drawing for demonstrating the conversion to the three-dimensional positional information by a positional information derivation | leading-out part. It is explanatory drawing for demonstrating a division area and a representative distance. It is explanatory drawing for demonstrating a division area group. It is explanatory drawing for demonstrating the process of a 1st grouping part. It is explanatory drawing for demonstrating a color identifier map. It is explanatory drawing which showed the relationship between a selection specific thing and an arrow traffic light. It is explanatory drawing which showed the relationship between a selection specific thing and an arrow traffic light. It is explanatory drawing which showed the relationship between the selection specific thing and the red light of a signal apparatus. It is explanatory drawing for demonstrating a signal search area | region. It is the flowchart which showed the rough flow of the process of the environment recognition method outside a vehicle. It is the flowchart which showed the flow of the parallax derivation process. It is the flowchart which showed the flow of the positional information derivation processing. It is the flowchart which showed the flow of the representative distance derivation processing. It is the flowchart which showed the flow of the 1st grouping process. It is the flowchart which showed the flow of the color identifier setting process. It is the flowchart which showed the flow of the 2nd grouping process. It is the flowchart which showed the flow of the 1st possibility determination process. It is the flowchart which showed the flow of the 2nd possibility determination process.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The dimensions, materials, and other specific numerical values shown in the embodiment are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. In the present specification and drawings, elements having substantially the same function and configuration are denoted by the same reference numerals, and redundant description is omitted, and elements not directly related to the present invention are not illustrated. To do.

(Environment recognition system 100)
FIG. 1 is a block diagram showing a connection relationship of the environment recognition system 100. The environment recognition system 100 includes a plurality (two in the present embodiment) of imaging devices 110, a vehicle exterior environment recognition device 120, and a vehicle control device 130 provided in the host vehicle 1.

  The imaging device 110 includes an imaging device such as a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS), and is a color image, that is, three hues (R (red), G (pixel-by-pixel)). The brightness of green) and B (blue) can be acquired. In the present embodiment, color and luminance are treated equally, and when both words are included in the same sentence, each other can be read as luminance constituting the color or a color having luminance. Here, a color image picked up by the image pickup apparatus 110 is called a luminance image, and is distinguished from a distance image described later.

  Further, the imaging devices 110 are arranged so that the imaging axes (optical axes) of the two imaging devices 110 are different in position on the same horizontal plane and are substantially parallel on the traveling direction side of the host vehicle 1. Is done. The imaging device 110 continuously generates, for example, every 1/60 seconds (60 fps), image data obtained by imaging an object existing in the detection area in front of the host vehicle 1. Each functional unit in the following embodiment performs each process triggered by such update of image data. Therefore, each functional unit repeats the process in units of frames. Here, the object is an object that actually exists in the environment outside the vehicle corresponding to each part of the image based on the image data. Further, the specific object is an object specified on the collision avoidance control or cruise control among the objects. Specific items include not only specific items that exist independently, such as vehicles, traffic lights, roads, and guardrails, but also those that can be specified as parts of specific items such as each lighting part of a traffic light, brake lamp (tail lamp), blinker, and the like.

  The outside-vehicle environment recognition device 120 acquires image data from each of the two imaging devices 110, and is a block arbitrarily extracted from an image based on one of the image data (here, composed of an array of horizontal 4 pixels × vertical 4 pixels). The parallax is derived using so-called pattern matching, in which a block corresponding to is detected from an image based on the other image data. Here, “horizontal” used in the description of the block indicates the horizontal direction of the captured image and corresponds to the horizontal direction in real space. “Vertical” indicates the vertical direction of the captured image and corresponds to the vertical direction in real space.

  As this pattern matching, it is conceivable to compare luminance values (Y color difference signals) in units of blocks indicating an arbitrary image position between two pieces of image data. For example, the SAD (Sum of Absolute Difference) that takes the difference in luminance value, the SSD (Sum of Squared intensity Difference) that uses the difference squared, and the similarity of the variance value obtained by subtracting the average value from the luminance value of each pixel. There are methods such as NCC (Normalized Cross Correlation). The vehicle exterior environment recognition apparatus 120 performs such block-based parallax derivation processing for all blocks displayed in the detection area (for example, 600 pixels × 200 pixels). Here, the block is 4 pixels × 4 pixels, but the number of pixels in the block can be arbitrarily set.

  However, the vehicle environment recognition apparatus 120 can derive the parallax for each block, which is a unit of detection resolution, through pattern matching, but cannot recognize what kind of specific object the block is. Accordingly, disparity information is derived independently not in units of specific objects but in units of detection resolution (for example, blocks) in the detection region. Here, an image in which the parallax information derived in this way (corresponding to a relative distance described later) is associated with image data is referred to as a distance image.

  FIG. 2 is an explanatory diagram for explaining the luminance image 124 and the distance image 126. For example, it is assumed that a luminance image (image data) 124 as illustrated in FIG. 2A is generated for the detection region 122 through the two imaging devices 110. However, only one of the two luminance images 124 is schematically shown here for easy understanding. The vehicle environment recognition apparatus 120 obtains the parallax for each block from the luminance image 124 and forms a distance image 126 as shown in FIG. Each block in the distance image 126 is associated with the parallax of the block. Here, for convenience of description, blocks from which parallax is derived are represented by black dots.

  Since the parallax is easily specified at the edge portion of the image (the portion where the difference in brightness between adjacent pixels is large), the block to which the black dot is attached in the distance image 126 and from which the parallax is derived is the luminance image 124. Are often edges. Therefore, the luminance image 124 shown in FIG. 2A and the distance image 126 shown in FIG. 2B are similar to the contour of each specific object.

  Then, the vehicle environment recognition apparatus 120 uses the relative distance from the host vehicle 1 based on the distance image 126 (hereinafter simply referred to as “relative distance”) to identify what type of object is displayed in the detection area 122. Identify whether it corresponds to an object. For example, a specific object that maintains a predetermined relative distance (runs at the same speed as the host vehicle) is recognized as a preceding vehicle. Further, the outside environment recognition device 120 further identifies, for example, the brake lamp of the preceding vehicle based on the luminance of the luminance image 124, thereby quickly grasping the deceleration of the preceding vehicle through the lighting of the brake lamp and avoiding the collision. It can be used for control and cruise control.

  The relative distance from the host vehicle 1 based on the distance image 126 is obtained by converting the parallax information for each block in the distance image 126 into three-dimensional position information using a so-called stereo method. Here, the stereo method is a method of deriving the relative distance of the specific object from the imaging device 110 from the parallax of the specific object by using a triangulation method. The processing of the outside environment recognition device 120 will be described in detail later.

  The vehicle control device 130 avoids a collision with a specific object specified by the outside environment recognition device 120 (collision avoidance control), or executes a control (cruise control) that keeps the distance between the vehicle and the preceding vehicle at a safe distance. . Specifically, the vehicle control device 130 acquires the current traveling state of the host vehicle 1 through the steering angle sensor 132 that detects the steering angle, the vehicle speed sensor 134 that detects the speed of the host vehicle 1, and the like, and controls the actuator 136. The distance between the vehicle and the preceding vehicle is kept safe. Here, the actuator 136 is a vehicle control actuator used to control a brake, a throttle valve, a steering angle, and the like. Further, when a collision with a specific object is assumed, the vehicle control device 130 displays a warning (notification) on the display 138 installed in front of the driver and controls the actuator 136 to control the host vehicle 1. Brake automatically. Such a vehicle control device 130 can also be formed integrally with the outside environment recognition device 120.

  In the present embodiment, through the environment recognition system 100, a specific area (part of the detection area in accordance with the specific object) occupied by the preceding vehicle that is the specific object is specified, and the red light or yellow light of the traffic light is specified. (Stop signal light indicating stop or no progress) is identified. And the possibility that the blue light of the arrow traffic light (a progress signal lamp indicating progress permission) is latent is derived from the positional relationship between the two. Hereinafter, the outside environment recognition device 120 will be described in detail.

(Vehicle environment recognition device 120)
FIG. 3 is a functional block diagram showing a schematic function of the outside environment recognition device 120. As shown in FIG. 3, the vehicle exterior environment recognition apparatus 120 includes an I / F unit 150, a data holding unit 152, and a central control unit 154.

  The I / F unit 150 is an interface for performing bidirectional information exchange with the imaging device 110 and the vehicle control device 130. The I / F unit 150 functions as an image acquisition unit that acquires the luminance image 124 from the imaging device 110. The data holding unit 152 includes a RAM, a flash memory, an HDD, and the like. The data holding unit 152 holds a color identifier table and various pieces of information necessary for processing of each functional unit described below, and the luminance image 124 received from the imaging device 110. Hold temporarily. Here, the color identifier table is used as follows.

  FIG. 4 is an explanatory diagram for explaining the color identifier table 200. In the color identifier table 200, a luminance range 202 representing a predetermined number of colors that are determined in advance is associated with the color identifier 204. For example, the color identifier “1” includes a luminance range corresponding to a red specific object, for example, a luminance range (R) “80 or more”, a luminance range (G) “40 or less”, and a luminance range (B) “40 or less. "Is associated. Similarly, for the color identifiers “2”, “3”, “4”, “5”, “6”, “7”, “8”, yellow, blue-green, magenta, orange, vermilion, blue, green A luminance range corresponding to the specific object is associated. However, it goes without saying that the luminance range is not limited to the luminance range shown in FIG. 4, and the number thereof is not limited.

  Here, although not shown, each color identifier in the color identifier table 200 is associated with one or more specific objects. For example, the color identifier “1” is associated with a red specific object “red light of traffic light”. This is for avoiding an increase in calculation load by extracting specific objects having similar brightness with the same reference (brightness range).

  The central control unit 154 is configured by a semiconductor integrated circuit including a central processing unit (CPU), a ROM storing a program and the like, a RAM as a work area, and the like, and through the system bus 156, the I / F unit 150 and the data holding unit. 152 is controlled. In the present embodiment, the central control unit 154 includes the parallax deriving unit 160, the position information deriving unit 162, the representative distance deriving unit 164, the first grouping unit 166, the specific object selecting unit 168, the color identifier setting unit 170, the first It also functions as a two-grouping unit 172, a first possibility determination unit 174, and a second possibility determination unit 176.

  The parallax deriving unit 160 compares the two pieces of image data (luminance image) acquired from the imaging device 110, and correlates with an arbitrary block composed of one or a plurality of pixels in an image (reference image) based on one image data. A high block is extracted from an image (comparison image) based on the other image data. And the parallax of both blocks is calculated | required and it is set as parallax information.

  The position information deriving unit 162 converts the disparity information for each block in the detection region 122 in the distance image 126 into three-dimensional position information including the horizontal distance x, the height y, and the relative distance z using the stereo method described above. Convert. Here, the parallax information indicates the parallax of each block in the distance image 126, while the three-dimensional position information indicates information on the relative distance of each block in the real space. Further, when the disparity information is derived not in pixel units but in block units, that is, in a plurality of pixel units, the disparity information may be regarded as disparity information of all pixels belonging to the block, and calculation in pixel units may be executed. it can.

  FIG. 5 is an explanatory diagram for explaining the conversion into three-dimensional position information by the position information deriving unit 162. First, the position information deriving unit 162 recognizes the distance image 126 as a pixel unit coordinate system as shown in FIG. In FIG. 5, the lower left corner is the origin (0, 0), the horizontal direction is the i coordinate axis, and the vertical direction is the j coordinate axis. Therefore, a pixel having parallax can be expressed as (i, j, dp) by the pixel position i, j and the parallax information dp indicating the parallax.

The three-dimensional coordinate system in real space in the present embodiment is considered as a relative coordinate system with the host vehicle 1 as the center. Here, the right side of the traveling direction of the host vehicle 1 is the positive direction of the X axis, the upper direction of the host vehicle 1 is the positive direction of the Y axis, and the traveling direction (front) of the host vehicle 1 is the positive direction of the Z axis. The intersection of the vertical line passing through the center of 110 and the road surface is defined as the origin (0, 0, 0). At this time, assuming that the road is a plane, the road surface coincides with the XZ plane (y = 0). The position information deriving unit 162 converts the block (i, j, dp) on the distance image 126 to a three-dimensional point (x, y, z) in the real space by the following (Formula 1) to (Formula 3). Convert coordinates.
x = CD / 2 + z · PW · (i-IV) (Equation 1)
y = CH + z · PW · (j−JV) (Formula 2)
z = KS / dp (Formula 3)
Here, CD is the interval (baseline length) between the imaging devices 110, PW is the viewing angle per pixel, CH is the height of the imaging device 110 from the road surface, and IV and JV are The coordinates (pixels) on the image of the point at infinity in front of the vehicle 1 and KS is a distance coefficient (KS = CD / PW).

  Therefore, the position information deriving unit 162 determines the road surface based on the relative distance of the block and the detected distance (for example, the number of pixels) on the distance image 126 between the block and the point on the road surface at the same relative distance as the block. The height from is derived.

  First, the representative distance deriving unit 164 divides the detection area 122 of the distance image 126 into a plurality of divided areas in the horizontal direction. Subsequently, the representative distance deriving unit 164 adds up the relative distances included in each of the predetermined distances divided into a plurality of divided areas based on the position information for blocks located above the road surface, and displays a histogram. Generate. Then, the representative distance deriving unit 164 derives a representative distance corresponding to the peak of the accumulated distance distribution. Here, “corresponding to a peak” means a peak value or a value that satisfies an arbitrary condition in the vicinity of the peak.

  FIG. 6 is an explanatory diagram for explaining the divided region 216 and the representative distance 220. As shown in FIG. 6, when the distance image 126 is divided into a plurality of parts in the horizontal direction, the divided region 216 has a strip shape as shown in FIG. Such a strip-shaped divided region 216 is originally composed of, for example, 150 columns each having a horizontal width of 4 pixels, but here, for convenience of explanation, the detection region 122 is divided into 16 parts.

  Subsequently, the representative distance deriving unit 164 refers to the relative distances of all the blocks in each divided region 216 and creates a histogram (shown as a horizontally long square (bar) in FIG. 6B). A distance distribution 218 as shown in (b) is obtained. Here, the vertical direction indicates the divided predetermined distance, and the horizontal direction indicates the number of blocks in which the relative distance is included in each of the divided predetermined distances. However, FIG. 6B is a virtual screen for performing the calculation, and actually does not involve generation of a visual screen. Then, the representative distance deriving unit 164 refers to the distance distribution 218 derived in this way, and represents a representative distance (indicated by a square filled in black in FIG. 6B) that is a relative distance corresponding to a peak. Is identified.

  The first grouping unit 166 first sequentially compares the representative distances 220 between adjacent divided areas 216, and groups the divided areas 216 that are close to each other (for example, located at 1 m or less) into one or a plurality of divided areas 216. A divided region group is generated. At this time, even when the representative distance 220 is close in three or more divided areas 216, all the continuous divided areas 216 are collected as a divided area group.

  FIG. 7 is an explanatory diagram for explaining the divided region group 222. The first grouping unit 166 compares the divided areas 216 and groups the representative distances 220 as shown in FIG. 7 (virtual group 224 after grouping). By such grouping, the first grouping unit 166 can identify a specific object located above the road surface. Further, the first grouping unit 166 is configured such that the group 224 is changed to a rear portion, a side portion, a guardrail, or the like of the preceding vehicle based on the transition of the relative distance in the horizontal direction and the vertical direction in the grouped divided region group 222. It is possible to recognize which is a structure along the road.

  Subsequently, the first grouping unit 166 identifies a specific object such as a preceding vehicle according to the relative positional relationship with the host vehicle 1, and determines one or more specific areas according to the outer edge of the specific object.

  FIG. 8 is an explanatory diagram for explaining the processing of the first grouping unit 166. The specific area 230 can be expressed by various predetermined shapes that accommodate the outer edge of the specific object. Here, the specific area 230 is expressed by a rectangular frame as shown in FIG. Specifically, the first grouping unit 166 uses the block whose relative distance z corresponds to the representative distance 220 in the divided region group 222 generated by the first grouping unit 166 as a base point, and the horizontal distance x , The difference in height y, and the difference in relative distance z are grouped on the assumption that they correspond to the same specific object in a predetermined range (for example, 0.1 m). The above range is represented by a distance in real space, and can be set to an arbitrary value by a manufacturer or a passenger. The first grouping unit 166 also uses the block as a base point for the newly added block by grouping so that the difference in the horizontal distance x, the difference in the height y, and the difference in the relative distance z are within a predetermined range. A block is further grouped. As a result, all blocks that can be assumed to be the same specific object are grouped.

Here, the horizontal distance x difference, the height y difference, and the relative distance z difference are determined independently, and only when all are included in a predetermined range, the same group is used. You can also. For example, the root mean square of the difference in horizontal distance x, the difference in height y, and the difference in relative distance z ((difference in horizontal distance x) ² + (difference in height y) ² + (difference in relative distance z) ² ) Are included in the predetermined range, the same group may be used. With this calculation, an accurate distance between blocks in real space can be derived, so that the grouping accuracy can be improved.

  Subsequently, the first grouping unit 166 determines the block group as a specific object if the grouped block group satisfies a predetermined condition. For example, when the grouped block group is located on the road, the first grouping unit 166 determines whether the size of the entire block group corresponds to the size of the specific object “vehicle”. If it is determined that the size corresponds to the size of the specific object “vehicle”, the block group is specified as the specific object “vehicle”. The first grouping unit 166 sets the area occupied on the screen by the block group identified as the specific object “vehicle” as the specific area 230 of the vehicle.

  The specific object selection unit 168 selects a specific object (selected specific object) to be tracked from one or more specific objects specified by the first grouping unit 166. Then, the specific object selection unit 168 tracks the selected specific object selected. In the present embodiment, a vehicle (preceding vehicle) positioned in front of the host vehicle 1 where the possibility of a collision is higher than others is selected as the selected specific object. Since the selection means and the tracking means for such a selected specific object can use various existing techniques, detailed description thereof is omitted here.

  The color identifier setting unit 170 sets a color identifier for each pixel based on the luminance of the plurality of pixels in the detection region 122 and the luminance range of the color identifier table 200.

  First, the color identifier setting unit 170 acquires the luminance in units of pixels (the luminance of three hues (R, G, B) in units of pixels) from the luminance image 124 held in the data holding unit 152.

  Then, the color identifier setting unit 170 sequentially selects the color identifiers 204 registered in the color identifier table 200, and whether or not the acquired luminance of one pixel is included in the luminance range 202 of the sequentially selected color identifier 204. To determine. If the luminance of the pixel is included in the luminance range 202 of any one of the color identifiers 204, the color identifier 204 is set for the pixel, and a color identifier map is created.

  The color identifier setting unit 170 sequentially performs a series of comparisons between the luminance of each pixel and the luminance range 202 of the plurality of color identifiers 204 registered in the color identifier table 200 for each of the plurality of pixels. Here, the selection order of the color identifier 204 is ascending order such as “1”, “2”.

  When it is determined that the luminance of the pixel is included in the luminance range 202 of the color identifier 204 with a low ordinal as a result of comparison in accordance with such a selection order, the comparison process for the color identifier 204 with a higher ordinal is performed earlier. Absent. Therefore, two or more color identifiers 204 are not attached to one pixel. This is because a plurality of specific objects do not overlap in space, so whether or not an object for which an arbitrary color identifier 204 is once set by the color identifier setting unit 170 is another color identifier 204 is determined. Based on the reason that there is no need to judge. By exclusively handling pixels in this way, it is possible to avoid duplicate setting processing of pixels for which the color identifier 204 has already been set, and to reduce the processing load.

  FIG. 9 is an explanatory diagram for explaining the color identifier map 240. The color identifier map 240 is obtained by superimposing the color identifier 204 on the luminance image 124. Therefore, the color identifier 204 is collectively set at the position of the object corresponding to the specific object.

  Here, attention is paid to the partial map 240 a in the color identifier map 240. Here, the luminance of the pixel in the partial map 240a is compared with the luminance range 202 of the color identifier “1” in the color identifier table 200, and the color identifier “1” is assigned to the plurality of pixel groups 242 corresponding to the red light of the traffic light. It is associated. In FIG. 9, a diagram in which color identifiers are attached to a plurality of pixels of the luminance image 124 is presented. However, such an expression is conceptual for ease of understanding. An identifier 204 is registered.

  The second grouping unit 172 uses an arbitrary pixel as a base point, the pixel and the color identifier 204 are equal, and the difference in the horizontal distance x, the difference in the height y, and the difference in the relative distance z are predetermined ranges. Pixels within (for example, 0.1 m) are grouped on the assumption that they correspond to the same specific object. The above range is represented by a distance in real space, and can be set to an arbitrary value by a manufacturer or a passenger. Further, the second grouping unit 172 also has the same color identifier with respect to a pixel newly added as a result of grouping, the difference in horizontal distance x, the difference in height y, and the relative distance z. Pixels whose differences are within a predetermined range are further grouped. As a result, all pixels that can be assumed to be the same specific object are grouped.

Also here, the difference in the horizontal distance x, the difference in the height y, and the difference in the relative distance z are independently determined, and the same group is formed only when all are included in the predetermined range. You can also. For example, the root mean square of the difference in horizontal distance x, the difference in height y, and the difference in relative distance z ((difference in horizontal distance x) ² + (difference in height y) ² + (difference in relative distance z) ² ) Are included in the predetermined range, the same group may be used.

  Then, the second grouping unit 172 determines the target object as a specific object if the target object (pixel group) grouped by the second grouping unit 172 satisfies a predetermined condition. For example, in the second grouping unit 172, the size of the target object (both the horizontal distance x width and the height y width of the target object) is included in the width range associated with the specific object. For example, the target object is determined as a target specific object. Moreover, you may set the width range about each of the width | variety of the horizontal distance x of the target object, and the width | variety of height y. Here, it is confirmed that the object has a size that is appropriate to be regarded as a specific object. Therefore, when it is not included in the width range, it can be excluded as information unnecessary for the environment recognition process. For example, the size of the pixel group 242 in the partial map 240a in FIG. 9 is included in the width range of the specific object “red light of the traffic light”, for example, “0.05 to 0.2 m” (not shown). Is identified as a specific item “red light of traffic light”.

  In this way, the outside environment recognition device 120 can extract the pixel group 242 having the same luminance range from the luminance image 124 as a specific object, and can use the information for various controls. For example, by extracting a specific object “red light of a traffic light”, it is understood that the specific object is a fixed object that does not involve movement, and if the specific object is a traffic light related to the own lane, The host vehicle 1 can grasp that it should stop or decelerate, and supports the driver's braking operation.

  However, even if the red light of the traffic light is lit in this way, depending on the state of the surrounding traffic light, support for the braking operation may be unnecessary. For example, if the blue light of the arrow traffic light attached to the red light is lit, it is not necessary to support braking depending on the traveling direction of the host vehicle 1. Therefore, in the outside environment recognition device 120, even if the traffic light is a red light, if there is a possibility that the blue light of the arrow traffic light attached thereto may be lit, the setting is made to suppress the support for the braking operation.

  10 and 11 are explanatory diagrams showing the relationship between the selected specific object 250 and the arrow traffic light 252. FIG. 10A and 10B show examples of the luminance image 124 and the distance image 126, and FIG. 11 shows a side view of the host vehicle 1, the selected specific object 250, and the arrow traffic light 252 in the situation of FIG. . In this embodiment, since the specific object selection unit 168 selects one selected specific object 250 to be tracked, if it is known that the selected specific object 250 does not shield the blue light of the arrow traffic light 252, the blue color It can be determined that there is no potential for the light.

  However, as described above, in the present embodiment, the difference between the horizontal distance x, the difference in the height y, and the difference in the relative distance z is specified as a specific object within a predetermined range (for example, 0.1 m). In addition, the specific object located in front of the host vehicle 1 is selected as the selected specific object 250. Therefore, as shown in FIGS. 10 and 11, if the preceding vehicle is a truck or the like having a loading platform, only the rear end portion thereof is recognized as the selected specific object 250. Then, as shown in FIG. 11, the selected specific object 250 (the rear end portion of the track) itself does not shield the arrow traffic light 252 attached to the traffic light 254, but the cabin of the truck that moves together with the selected specific object 250. (Cab) 256 may be shielded. Therefore, in the present embodiment, the possibility of the blue light of the arrow traffic light 252 is derived for a specific object including the selected specific object 250 and the truck cabin 256, and unnecessary braking operation by collision avoidance control or cruise control is derived. Avoid intervention.

  The potential of such an arrow traffic light 252 is derived from a signal search area in which the blue light of the arrow traffic light 252 may be latent on the basis of the red light and yellow light of the traffic light 254, and the own vehicle from the traffic light 254. It can be grasped by searching for a shielding object that shields the signal search area in a space having a short relative distance to 1. However, if such a process is performed every frame, the processing load may increase. Therefore, in the present embodiment, first, the first possibility determination unit 174 may cause the blue light of the arrow traffic light 252 to be latent based on the positional relationship between the selected specific object 250 and the traffic light 254, which is known information. Judging sex roughly. The second possibility determination unit 176, based only on the positional relationship between the signal search area and the shielding object, determines the potential of the arrow traffic light 252 only when the potential of the determination is not clear (cannot be determined). The possibility of a blue light is specifically determined. Hereinafter, the process of the 1st possibility determination part 174 and the 2nd possibility determination part 176 is explained in full detail.

  The first possibility determination unit 174 selects the specific specified object 250 and the second grouping unit 172 specifies the specific object “red light of traffic light” or “yellow light of traffic light”. Based on the positional relationship (horizontal direction position and height) between the specific object 250 and the red and yellow lights of the traffic light 254, the blue light of the arrow traffic light 252 attached to the red and yellow lights of the traffic light 254 may be latent. Determine gender. Hereinafter, since the red lamp and the yellow lamp are at substantially the same distance, they will be described together as a red lamp. Therefore, what is described as a red light can be understood by replacing it with a yellow light.

  Specifically, the first possibility determination unit 174 first determines whether the relative distance of the selected specific object 250 to the host vehicle 1 is longer than the relative distance of the red light of the traffic light 254 to the host vehicle 1, that is, It is determined whether or not the selected specific object 250 is far from the red light of the traffic light 254. In this determination, when the relative distance of the selected specific object 250 is longer than the relative distance of the red light of the traffic light 254, it can be determined that the blue light of the arrow traffic light 252 is not latent. Here, when the relative distance of the selected specific object 250 is shorter than the relative distance of the red light of the traffic light 254, further determination is performed.

  FIG. 12 is an explanatory diagram showing the relationship between the selected specific object 250 and the red light 254a of the traffic light 254. The first possibility determination unit 174 is determined based on the red lamp 254a of the traffic light 254, and a horizontal region indicated by a double arrow in FIG. 12A (for example, twice the width of the traffic light 254), It is determined whether or not the horizontal position of the selected specific object 250 overlaps the horizontal area. For example, in FIG. 12A, since the area based on the red light 254a of the traffic light 254 is included in the area in the horizontal direction of the selected specific object 250, it is determined that they overlap.

  In this determination, if the area based on the red light 254a of the traffic light 254 and the horizontal area of the selected specific object 250 do not overlap, it can be determined that the blue light of the arrow traffic light 252 is not latent. Here, when the area based on the red light 254a of the traffic light 254 and the area in the horizontal direction of the selected specific object 250 overlap, further determination is performed.

  Subsequently, the first possibility determination unit 174 has a height (shielding height) of the selected specific object 250 in the relative distance of the selected specific object 250 based on the height of the red light 254a of the traffic light 254. If it does, the possibility that the blue light of the arrow traffic light 252 may be derived is derived. For example, in the example of FIG. 12B, the first possibility determination unit 174 derives the height indicated by the vertically upward arrow in the selected specific object 250.

  Next, the first possibility determination unit 174 compares the derived shielding height with the legal maximum vehicle height, and if the shielding height is higher than the legal maximum vehicle height, the blue light of the arrow traffic light 252 may be present. Therefore, it is determined that the blue light of the arrow traffic light 252 is not latent. If the shielding height is lower than the legal vehicle maximum height, the first possibility determination unit 174 continues to compare the shielding height with the height of the selected specific object 250. At this time, if the shielding height is lower than the height of the selected specific object 250, even if there is a blue light of the arrow traffic light 252 it cannot be acquired as an image. It is determined that a blue light may be latent. Here, if the shielding height is higher than the height of the selected specific object 250, the process proceeds to the second possibility determination unit 176.

  When the first possibility determination unit 174 determines that there is a possibility that the blue light of the arrow traffic light 252 is latent, the second possibility determination unit 176 determines the signal search area with the red light 254a of the traffic light 254 as a reference. . And the 2nd possibility determination part 176 searches the shielding object which shields a signal search area | region in the space where relative distance is shorter than the red light 254a of the traffic light 254, and determines the possibility that the blue light of the arrow traffic light 252 is latent. .

  FIG. 13 is an explanatory diagram for explaining the signal search area 258. The blue light of the arrow traffic light 252 may be arranged in the area of two color lights of the traffic light 254 vertically downward from the traffic light 254, and a plurality of blue lights are juxtaposed in the vertical direction on the right side in the horizontal direction of the traffic light 254. There is a possibility. Therefore, the signal search area 258 is a frame based on the red light 254a (or the yellow light 254b) of the traffic light 254 as shown in FIG.

  The second possibility determination unit 176 newly extracts a shielding object existing in front of the relative distance of the red light 254a of the traffic light 254 with respect to the derived signal search region 258, and the occupation area ratio of the shielding object is determined in advance. If it is equal to or greater than the threshold value (for example, 20% of the signal search area), it is determined that the blue light of the arrow traffic light 252 may be latent. Is determined.

  Here, the second possibility determination unit 176 searches for a shielding object that shields the signal search region 258 only in a predetermined relative distance range (for example, ± 5 m) with the selected specific object 250 as a reference, and the arrow traffic light 252. It is also possible to determine the possibility of a blue light. As described above, when the selected specific object 250 is the rear end portion of the truck, the shielding object is likely to be the cabin 256 of the truck. Therefore, the processing load is reduced by extracting the shielding object in the relative distance range ± 5 m from the selected specific object 250. Note that the robustness is enhanced by searching not only the + (plus) direction but also the-(minus) direction as a relative distance.

  As described above, the possibility of the arrow traffic light 252 being lit can be determined with high accuracy even when only the selected specific object 250 is to be tracked by the outside-vehicle environment recognition device 120 described above. It becomes possible to avoid intervention to unnecessary braking operation by control or cruise control.

(External vehicle environment recognition method)
Hereinafter, specific processing of the vehicle environment recognition apparatus 120 will be described based on the flowcharts of FIGS. 14 to 22. FIG. 14 shows an overall flow relating to the interrupt processing when the luminance image 124 is transmitted from the imaging device 110, and FIGS. 15 to 22 show individual subroutines therein. Here, a block or a pixel is cited as a processing target. The lower left corner of the luminance image 124 and the distance image 126 is the origin, and in the block, the range is 1 to 150 blocks in the horizontal direction of the image, and 1 to 50 blocks in the vertical direction. The processing by the outside environment recognition method is performed in the range of 1 to 200 pixels in the direction.

  When an interruption by the vehicle exterior environment recognition method occurs, the parallax deriving unit 160 compares the two image data, extracts a block having a high correlation with an arbitrary block in the reference image, from the search range in the comparison image, and both blocks Is obtained (S300).

  Subsequently, the position information deriving unit 162 converts the disparity information for each block in the detection area 122 into three-dimensional position information (S302). The representative distance deriving unit 164 derives the representative distance 220 for each divided region 216 (S304), and the first grouping unit 166 groups the blocks based on the representative distance 220 (S306). Then, the specific object selecting unit 168 selects a specific object (selected specific object) to be tracked from one or more specific objects specified by the first grouping unit 166, and tracks the selected specific object (S308). ).

  Next, the color identifier setting unit 170 sets the color identifier 204 for each pixel based on the luminance of the plurality of pixels in the detection region 122 and the luminance range of the color identifier table 200 (S310). The second grouping unit 172 groups a plurality of pixels based on the color identifier 204 and the three-dimensional position of each pixel (S312).

  Here, when the selected specific object 250 exists and the specific object “red light of traffic light” is specified by the second grouping unit 172, the first possibility determination unit 174 determines that the selected specific object 250 is Based on the positional relationship of the traffic light 254 with the red light 254a, it is determined whether the blue light of the arrow traffic light 252 attached to the red light 254a of the traffic light 254 is latent (S314). The second possibility determination unit 176 can determine whether the blue light of the arrow traffic light 252 is latent by the first possibility determination unit 174, and determines whether or not the processing is completed (S316).

  Here, if the determination that the blue light of the arrow traffic light 252 is latent is not completed (NO in S316), the second possibility determination unit 176 is in a space where the relative distance is shorter than the red light 254a of the traffic light 254. Then, a shield that shields a predetermined signal search area with reference to the red light 254a of the traffic light 254 is searched, and the possibility that the blue light of the arrow traffic light 252 is latent is determined (S318). If the determination of the possibility that the blue light of the arrow traffic light 252 is latent has been completed (YES in S316), the vehicle exterior environment recognition method is terminated, and a transition to an interrupt waiting state is made. The above processing will be specifically described below.

(Parallax derivation process S300)
Referring to FIG. 15, the disparity deriving unit 160 initializes (assigns “0” to) a vertical variable j for specifying a block (S400). Subsequently, the parallax derivation unit 160 adds “1” to the vertical variable j and initializes the horizontal variable i (substitutes “0”) (S402). Next, the parallax deriving unit 160 adds “1” to the horizontal variable i (S404).

  The parallax deriving unit 160 derives a correlation value between the block (i, j) of the reference image and the block (i, j) of the comparison image shifted by one pixel to the right of the screen (S406). Such derivation of the correlation value is repeated until the search range is completed (NO in S408). When the search range is completed (YES in S408), the disparity deriving unit 160 extracts the block (i, j) of the comparison image having the largest correlation value with the block (i, j) of the reference image, and the distance between both blocks. The parallax information dp indicating (parallax) is derived (S410).

  Subsequently, the parallax deriving unit 160 determines whether or not the horizontal variable i exceeds 150 which is the maximum value of the horizontal block (S412), and if the horizontal variable i does not exceed the maximum value (NO in S412), step The processing from S404 is repeated. If the horizontal variable i exceeds the maximum value (YES in S412), the parallax derivation unit 160 determines whether or not the vertical variable j exceeds 50, which is the maximum value of the vertical block (S414). If the vertical variable j does not exceed the maximum value (NO in S414), the processing from step S402 is repeated. If the vertical variable j exceeds the maximum value (YES in S414), the disparity derivation process S300 ends. Thus, the block (i, j) of the distance image 126 becomes a block (i, j, dp) to which the parallax information dp is added.

(Position information deriving process S302)
Referring to FIG. 16, the position information deriving unit 162 initializes (substitutes “0”) a vertical variable j for specifying a block (S450). Subsequently, the position information deriving unit 162 adds “1” to the vertical variable j and initializes the horizontal variable i (substitutes “0”) (S452). Next, the position information deriving unit 162 adds “1” to the horizontal variable i (S454).

  Next, the position information deriving unit 162 acquires the parallax information dp from the block (i, j, dp) of the distance image 126 (S456). Then, the position information deriving unit 162 converts the block (i, j, dp) including the parallax information dp to a point (x, y, z) in the real space using the above Equations 1 to 3, It is assumed that the block is (i, j, dp, x, y, z) (S458).

  Subsequently, the position information deriving unit 162 determines whether or not the horizontal variable i exceeds 150 which is the maximum value of the horizontal block (S460). If the horizontal variable i does not exceed the maximum value (NO in S460), The processing from step S454 is repeated. If the horizontal variable i exceeds the maximum value (YES in S460), the position information deriving unit 162 determines whether the vertical variable j exceeds 50, which is the maximum value of the vertical block (S462). If vertical variable j does not exceed the maximum value (NO in S462), the processing from step S452 is repeated. If the vertical variable j exceeds the maximum value (YES in S462), the position information deriving process S302 ends. Thus, the parallax information dp of the distance image 126 is converted into three-dimensional position information.

(Representative distance derivation process S304)
Referring to FIG. 17, the representative distance deriving unit 164 reads road shape parameters (S500), and divides the detection area 122 into 150 divided areas 216, for example, in units of 4 pixels in the horizontal direction (S502). Next, the representative distance deriving unit 164 sequentially extracts one divided region 216 from the 150 divided regions 216, for example, from the left side in the horizontal direction, and arbitrarily blocks (i, j, dp, x, y, z) are set (S504).

  The representative distance deriving unit 164 calculates the height yr of the road surface at the coordinate z in the real space of the block (S506), and the block whose coordinate y in the real space of the block is equal to or higher than the height yr of the road surface. For example, the relative distance is added (voted) to the histogram divided at predetermined distance intervals (S508). Here, even if the coordinate y in the real space of the block is equal to or higher than the height yr of the road surface, the block having a height of 0.1 m or less from the road surface is white line, dirt, shadow, etc. on the road. Consider and exclude from processing. Also, blocks positioned above the height of the host vehicle 1 are regarded as pedestrian bridges, signs, etc., and are excluded from processing targets.

  The representative distance deriving unit 164 determines whether or not the integration process to the histogram has been performed for all the extracted blocks in one divided region 216 (S510). Here, if all the blocks are not completed (NO in S510), the processing from step S504 is repeated for the blocks that have not been subjected to the histogram integration processing.

  If all the blocks have been completed (YES in S510), the representative distance deriving unit 164 refers to the histogram generated in this manner, and the frequency of the histogram (the number of relative distances) is set to a threshold (appropriately set). If there is a section as described above, it is determined that a specific object exists in the divided area 216. Then, the representative distance deriving unit 164 sets the relative distance corresponding to the peak as the representative distance 220 (S512).

  Then, the representative distance deriving unit 164 determines whether the representative distance 220 has been derived for all of the plurality of divided regions 216 (S514). Here, if it is not determined that all the divided areas 216 are completed (NO in S514), a new divided area 216 is set (S516), and the processing from step S504 is repeated for the new divided area 216. On the other hand, if all the processes for deriving the representative distance 220 have been completed (YES in S514), the representative distance deriving process S304 ends.

(First grouping process S306)
Referring to FIG. 18, the first grouping unit 166 sequentially identifies an arbitrary divided area 216 from a plurality of divided areas 216, for example, from the left side in the horizontal direction, and divides the arbitrary divided area 216 adjacent to the right side in the horizontal direction. The area 216 is also specified (S550). Then, the first grouping unit 166 determines whether or not the representative distance 220 exists in both divided regions 216 (S552). If the representative distance 220 does not exist in both the divided areas 216 (NO in S552), the process proceeds to the divided area completion determination step S558. On the other hand, if the representative distance 220 exists in both divided areas 216 (YES in S552), the representative distances 220 of both divided areas 216 are compared (S554).

  Here, if the difference between the two representative distances 220 is equal to or smaller than a predetermined threshold value (a value that can be regarded as the same specific object), the two representative distances 220 are regarded as close to each other, and the first grouping unit 166 The divided areas 216 are grouped to form a divided area group 222 (S556). At this time, when one divided region 216 is already set as the divided region group 222, the other divided region 216 is integrated into the divided region group 222.

  Then, the first grouping unit 166 determines whether the setting processing S552, S554, and S556 of the divided region group 222 has been performed for all the divided regions 216 (S558). Here, if not all is completed (NO in S558), a new divided area 216 is set (S560), and the processing from step S550 is repeated for the new divided area 216. On the other hand, if all the setting processes of the divided region group 222 have been completed (YES in S558), the process proceeds to the next step S562.

  Subsequently, the first grouping unit 166 sequentially extracts one divided region group 222 from the plurality of divided region groups 222 grouped, for example, from the left side in the horizontal direction, and arbitrarily exists in the divided region group 222. Block (i, j, dp, x, y, z) is set (S562).

  The first grouping unit 166 compares the set block (i, j, dp, x, y, z) with a block whose relative distance z corresponds to the representative distance 220 in the divided region group 222, It is determined whether the difference in the horizontal distance x, the difference in the height y, and the difference in the relative distance z are within a predetermined range (for example, 0.1 m) (S564). If it is within the predetermined range (YES in S564), the blocks are grouped on the assumption that they correspond to the same specific object (S566). If not within the predetermined range (NO in S564), the process proceeds to block completion determination step S568.

  The first grouping unit 166 determines whether or not the grouping process has been performed for all the blocks in the extracted one divided region group 222 (S568). If the grouping process for all the blocks has not been completed (NO in S568), the process from step S562 is repeated for the blocks that have not been subjected to the grouping process.

  If the grouping process for all the blocks has been completed (YES in S568), the first grouping unit 166 determines whether the size of the entire group of block groups corresponds to the size of the specific object “vehicle”. (S570). If it is determined that the size corresponds to the size of the specific object “vehicle” (YES in S570), the block group is specified as the specific object “vehicle” (S572). If it is not determined that it corresponds to the size of the specific object “vehicle” (NO in S570), the process proceeds to the completion determination step S574 of the divided region group 222.

  Then, the first grouping unit 166 determines whether the specific object specifying processes S570 and S572 have been performed for all of the plurality of divided region groups 222 (S574). If it is not determined that all the divided region groups 222 are complete (NO in S574), a new divided region group 222 is set (S576), and the processing from step S562 is performed on the new divided region group 222. repeat. On the other hand, if the specific object specifying processes S570 and S572 are all completed (YES in S574), the first grouping process S306 is ended.

  In the grouping, the mutual positional relationship is further determined for a plurality of groups. For example, when the position of an end point is close between groups of specific objects of the same type and the transition of the relative distance in the horizontal direction and the vertical direction in the specific object is almost equal (continuous), It is determined that they are the same surface, and these groups are integrated into one group. At this time, the transition of the relative distance in the horizontal direction and the vertical direction in the specific object can be specified by an approximate straight line by the Hough transform or the least square method. In the case of a preceding vehicle, a plurality of groups can be integrated into one group even if the relative movement speed with respect to the z coordinate is equal.

  Further, when the processing so far is performed in units of blocks, it is possible to perform processing in units of pixels by setting the same information for all the pixels in the block.

(Color identifier setting process S310)
Referring to FIG. 19, the color identifier setting unit 170 initializes (substitutes “0”) a vertical variable j for specifying a pixel (S600). Subsequently, the color identifier setting unit 170 adds (increments) “1” to the vertical variable j and initializes the horizontal variable i (substitutes “0”) (S602). Next, the color identifier setting unit 170 adds “1” to the horizontal variable i and initializes the color identifier variable m (substitutes “0”) (S604). Here, the reason why the horizontal variable i and the vertical variable j are provided is to execute the color identifier setting process S310 for all 600 × 200 pixels, and the color identifier variable m is provided. This is because the eight color identifiers 204 are sequentially compared for each pixel.

  The color identifier setting unit 170 acquires the luminance br of the pixel (i, j, dp, x, y, z, br) from the luminance image 124 (S606), and adds “1” to the color identifier variable m (S608). ), The luminance range 202 of the color identifier (m) or the reset luminance range 202 is acquired according to the position of the pixel (i, j, dp, x, y, z, br) (S610), It is determined whether or not the luminance br of the pixel (i, j, dp, x, y, z, br) is included in the luminance range 202 of the color identifier (m) (S612).

  If the luminance br of the pixel is included in the luminance range 202 of the color identifier (m) (YES in S612), the color identifier setting unit 170 associates the pixel with an identification number p indicating the color identifier (m). , Pixel (i, j, dp, x, y, z, br, p) (S614). In this way, a color identifier map 240 in which an identification number is assigned to each pixel in the luminance image 124 is generated. If the luminance br of the pixel (i, j, dp, x, y, z, br) is not included in the luminance range 202 of the color identifier (m) (NO in S612), the color identifier variable m is the maximum number. It is determined whether or not 8 is exceeded (S616). If the color identifier variable m does not exceed the maximum value (NO in S616), the process of incrementing the color identifier variable m in step S608 is repeated. If the color identifier variable m exceeds the maximum value (YES in S616), it is determined that there is no color identifier 204 corresponding to the pixel (i, j, dp, x, y, z, br), and the horizontal pixel Processing proceeds to decision step S618.

  Subsequently, the color identifier setting unit 170 determines whether or not the horizontal variable i exceeds 600, which is the maximum value of the horizontal pixel (S618). If the horizontal variable i does not exceed the maximum value (NO in S618), The process from the horizontal variable i increment process in step S604 is repeated. If the horizontal variable i exceeds the maximum value (YES in S618), the color identifier setting unit 170 determines whether the vertical variable j exceeds 200, which is the maximum value of the vertical pixel (S620). If the vertical variable j does not exceed the maximum value (NO in S620), the process of incrementing the vertical variable j in step S602 is repeated. If the vertical variable j exceeds the maximum value (YES in S620), the color identifier setting process S310 is terminated.

(Second grouping process S312)
Referring to FIG. 20, the second grouping unit 172 refers to a predetermined range (for example, 0.1 m) for grouping pixels (S650), and initializes a vertical variable j for identifying pixels (“ 0 ”is substituted) (S652). Subsequently, the second grouping unit 172 adds “1” to the vertical variable j and initializes the horizontal variable i (substitutes “0”) (S654). Next, the second grouping unit 172 adds “1” to the horizontal variable i (S656).

  The second grouping unit 172 acquires a pixel (i, j, dp, x, y, z, br, p) from the distance image 126, and the pixel (i, j, dp, x, y, z, br). , P), there is a valid (non-zero) color identifier p, and it is determined whether the group number g has not been assigned yet (S658). Here, if there is a valid color identifier p and the group number g has not yet been assigned (YES in S658), the second grouping unit 172 uses the coordinates (x, y) of the pixel in the real space. , Z), it is determined whether there is one or more other pixels for which the color identifier p is set and a group number g is not yet assigned to the pixel (S660). .

  If one or more other pixels (i, j, dp, x, y, z, br, p) for which the color identifier p is set exist and the group number g is not yet attached to the pixel (S660) YES), the smallest value among the numbers not yet used as the group number is newly assigned to all the pixels within the predetermined range including itself (S662). Here, pixels that have already been grouped as other objects are excluded from the grouping process and are not grouped.

  Thus, when there are a plurality of pixels having the same color identifier within a predetermined range, grouping is performed by assigning a group number g of 1. At this time, the reason why the smallest value among the numbers not yet used as the group number is employed is to prevent missing numbers as much as possible in group numbering. By doing so, the maximum value of the group number g is not increased unnecessarily, and the processing load can be reduced.

  The color identifier p is not a valid value (0), or is a valid value but the group number g has already been assigned (NO in S658), or there is another pixel with the same color identifier If not or if the group number g has already been assigned to all the pixels (NO in S660), the process proceeds to the horizontal variable determination step S664.

  Subsequently, the second grouping unit 172 determines whether or not the horizontal variable i exceeds 600, which is the maximum value of the horizontal pixel (S664), and if the horizontal variable i does not exceed the maximum value (NO in S664). The process from step S656 for incrementing the horizontal variable i is repeated. If the horizontal variable i exceeds the maximum value (YES in S664), the second grouping unit 172 determines whether or not the vertical variable j exceeds 200, which is the maximum value of the vertical pixels (S666). If the vertical variable j does not exceed the maximum value (NO in S666), the process of incrementing the vertical variable j in step S654 is repeated. If the vertical variable j exceeds the maximum value (YES in S666), the process proceeds to step S668.

  Subsequently, the second grouping unit 172 initializes a group variable k for specifying a group (substitutes “0”) (S668), and adds “1” to the group variable k (S670).

  The second grouping unit 172 determines whether there is an object whose group number g is the group variable k from the luminance image 124 (S672). If there is an object (YES in S672), the group number g is attached. The size of the target object is calculated (S674). At this time, the size of the object is determined by the horizontal direction component (difference) between the pixel located at the left edge of the object and the pixel located at the right edge of the object, the pixel located at the upper edge of the object, and the lower edge of the screen. It is specified by the vertical direction component which is the height (difference) between the pixels located at. Then, it is determined whether or not the calculated size is included in the width range of the specific object indicated by the color identifier p associated with the object whose group number g is the group variable k (S676). For example, the horizontal component of the size of the object is within the width range of the specific object indicated by the color identifier p, and the vertical component of the size of the object is the specific object indicated by the color identifier p. If it is within the width range, it can be determined that the object is included in the width range of the specific object indicated by the color identifier p.

  If the size is included in the width range of the specific object indicated by the color identifier p (YES in S676), the second grouping unit 172 determines the target object as the specific object (S678). When the size is not included in the width range of the specific object indicated by the color identifier p (NO in S676), or there is no object whose group number g is the group variable k (NO in S672), the group variable Processing proceeds to decision step S680.

  Subsequently, the second grouping unit 172 determines whether or not the group variable k exceeds the maximum value of the group numbers set in the grouping process (S680). If group variable k does not exceed the maximum value (NO in S680), the processing from step S670 is repeated. If the group variable k exceeds the maximum value (YES in S680), the second grouping process S312 ends. In this way, the grouped objects are formally determined as specific objects.

(First possibility determination processing S314)
Referring to FIG. 21, the first possibility determination unit 174 first determines whether or not the relative distance of the selected specific object 250 is equal to or greater than the relative distance of the red light 254a of the traffic light 254 (S700). In this determination, if the relative distance of the selected specific object 250 is equal to or greater than the relative distance of the red light 254a of the traffic light 254 (YES in S700), it is determined that the blue light of the arrow traffic light 252 is not latent, and the determination is completed (S702). ). On the other hand, if the relative distance of the selected specific object 250 is less than the relative distance of the red light 254a of the traffic light 254 (NO in S700), the process proceeds to step S704.

  Next, the first possibility determination unit 174 determines the horizontal area (for example, twice the width of the traffic light 254) determined based on the red light 254a of the traffic light 254 and the horizontal area of the selected specific object 250. It is determined whether or not the horizontal positions overlap with each other (S704). In this determination, if the area based on the red light 254a of the traffic light 254 and the horizontal area of the selected specific object 250 do not overlap (NO in S704), it is determined that the blue light of the arrow traffic light 252 is not latent (S702). If they overlap (YES in S704), the process proceeds to step S706.

  Subsequently, the first possibility determination unit 174 has a height (shielding height) of the selected specific object 250 in the relative distance of the selected specific object 250 based on the height of the red light 254a of the traffic light 254. If so, it is derived whether there is a possibility that the blue light of the arrow traffic light 252 is latent (S706). Then, the first possibility determination unit 174 compares the derived shielding height with the legal maximum vehicle height to determine whether the shielding height is equal to or higher than the legal vehicle maximum height (S708). Here, if the shielding height is not less than the legal maximum vehicle height (YES in S708), it is determined that the blue light of the arrow traffic light 252 is not latent (S702), and the shielding height is the legal maximum vehicle height. If less (NO in S708), the process proceeds to step S710.

  Next, the first possibility determination unit 174 compares the shielding height with the height of the selected specific object 250 and determines whether or not the shield height is less than the height of the selected specific object 250 (S710). If the shielding height is less than the height of the selected specific object 250 (YES in S710), it is determined that the blue light of the arrow traffic light 252 may be latent, and the determination is completed (S712). On the other hand, if the shielding height is equal to or higher than the height of the selected specific object 250 (NO in S710), the first possibility determination process S314 is terminated without completing the determination of the possibility that the blue light is latent.

(Second possibility determination processing S318)
Referring to FIG. 22, the second possibility determination unit 176 first determines a signal search area with reference to the red lamp 254a of the traffic light 254, and starts and starts the horizontal pixel start point Hs and end point He and the vertical pixel start point. Vs and end point Ve are derived (S750). Then, the second possibility determination unit 176 initializes a shielding counter for counting pixels that shield the signal search area (substitute “0”), and initializes a vertical variable j for specifying the pixel. (Vs is substituted) (S752). Subsequently, the second possibility determination unit 176 adds “1” to the vertical variable j and initializes the horizontal variable i (substitutes Hs) (S754). Next, the second possibility determination unit 176 adds “1” to the horizontal variable i (S756).

  The second possibility determination unit 176 acquires the pixel (i, j, dp, x, y, z, br, p) from the distance image 126, and the parallax information dp of the pixel corresponds to the red light 254a of the traffic light 254. It is determined whether it is less than the parallax information dp of the pixel to be processed (S758). Thereby, the shielding object existing before the relative distance of the red light 254a of the traffic light 254 is extracted with respect to the signal search area 258. If the disparity information dp of the pixel is less than the disparity information dp of the pixel corresponding to the red lamp 254a of the traffic light 254 (YES in S758), “1” is added to the shielding counter value of the shielding counter (S760), and the disparity information If dp is equal to or larger than the parallax information dp of the pixel corresponding to the red lamp 254a of the traffic light 254 (NO in S758), the process proceeds to step S762.

  Subsequently, the second possibility determination unit 176 determines whether or not the horizontal variable i exceeds the maximum value He of the horizontal pixels (S762), and if the horizontal variable i does not exceed the maximum value (NO in S762). ), The process of incrementing the horizontal variable i in step S756 is repeated. If the horizontal variable i exceeds the maximum value (YES in S762), the second possibility determination unit 176 determines whether or not the vertical variable j exceeds Ve, which is the maximum value of the vertical pixel (S764). . If the vertical variable j does not exceed the maximum value (NO in S764), the process of incrementing the vertical variable j in step S754 is repeated.

  If vertical variable j exceeds the maximum value (YES in S764), second possibility determination unit 176 derives the occupied area ratio of the shielding object (S766). The occupation area ratio is obtained by a shielding count value / ((He−Hs) × (Ve−Vs)). Then, the second possibility determination unit 176 determines whether or not the derived occupation area ratio is equal to or greater than a predetermined threshold (for example, 20% of the signal search region) (S768). Here, if the exclusive area ratio of the shield is equal to or greater than the threshold (YES in S768), it is determined that the blue light of the arrow traffic light 252 may be latent (S770), and if it is less than the threshold (NO in S768). ), It is determined that the blue light of the arrow traffic light 252 is not latent (S772).

  As described above, according to the outside environment recognition device 120 and the outside environment recognition method, the blue light of the arrow traffic light 252 is lit even when only the selected specific object 250 is tracked. It is possible to judge the possibility with high accuracy and to avoid the intervention to the unnecessary braking operation by the collision avoidance control or the cruise control.

  Also provided are a program that causes a computer to function as the vehicle exterior environment recognition device 120 and a computer-readable storage medium that stores the program, such as a flexible disk, magneto-optical disk, ROM, CD, DVD, and BD. Here, the program refers to data processing means described in an arbitrary language or description method.

  As mentioned above, although preferred embodiment of this invention was described referring an accompanying drawing, it cannot be overemphasized that this invention is not limited to this embodiment. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Is done.

  For example, in the above-described embodiment, the three-dimensional position information of the object is derived based on the parallax between the image data using the plurality of imaging devices 110. However, the present invention is not limited to this. Various known distance measuring devices such as a distance device can be used. Here, the laser radar distance measuring device projects a laser beam onto the detection region 122, receives light reflected by the laser beam upon the object, and measures the distance from the required time to the object.

  In the embodiment described above, the relative distance of the block is obtained using the distance image 126. However, when the specific object can be specified to some extent by the arrangement and size of the object on the screen, the monocular imaging device The present embodiment can be realized by 110. Also, a specific object can be specified by deriving a motion vector by optical flow.

  In the above-described embodiment, it is assumed that the imaging apparatus 110 acquires a color image. However, the present embodiment is not limited to this, and the present embodiment can also be performed by acquiring a monochrome image. In this case, the color identifier table 200 is defined with the luminance of a single color.

  In the above-described embodiment, the parallax deriving unit 160, the position information deriving unit 162, the representative distance deriving unit 164, the first grouping unit 166, the specific object selecting unit 168, the color identifier setting unit 170, and the second grouping unit 172, the 1st possibility determination part 174, and the 2nd possibility determination part 176 are comprised so that it may operate | move by software by the central control part 154. FIG. However, the functional unit described above can be configured by hardware.

  Note that each step of the vehicle environment recognition method of the present specification does not necessarily have to be processed in time series in the order described in the flowchart, and may include processing in parallel or a subroutine.

  INDUSTRIAL APPLICABILITY The present invention can be used for a vehicle environment recognition apparatus and a vehicle environment recognition method that recognize an environment outside the host vehicle.

DESCRIPTION OF SYMBOLS 1 ... Own vehicle 110 ... Imaging device 120 ... Outside environment recognition device 122 ... Detection area 124 ... Luminance image 126 ... Distance image 160 ... Parallax deriving unit 162 ... Position information deriving unit 164 ... Representative distance deriving unit 166 ... First grouping unit 168 ... specific object selection unit 170 ... color identifier setting unit 172 ... second grouping unit 174 ... first possibility determination unit 176 ... second possibility determination unit 250 ... selected specific object 252 ... arrow traffic light 254 ... traffic light 258 ... signal Search area

Claims (5)

An image acquisition unit that acquires an image of the environment outside the vehicle;
A position information deriving unit for deriving a relative distance of the object corresponding to the partial image of the image with respect to the host vehicle;
A first grouping unit that identifies one or more specific objects based on the relative distance;
A specific object selection unit that selects one specific object to be tracked from the one or more specific objects;
A second grouping unit that identifies a stop signal light indicating stop or no progress based on the color and size in the image;
A first possibility determination unit that determines a possibility that a progress signal lamp indicating progress permission attached to the stop signal lamp is latent based on a positional relationship between the selected specific object that is the selected specific object and the stop signal lamp;
A shield that shields a signal search area defined with reference to the stop signal light in a space having a relative distance shorter than that of the stop signal light when the first possibility determination unit cannot determine the possibility that the progress signal light is latent. A second possibility determination unit that determines the possibility that the progress signal lamp is latent based on the occupied area ratio of the shield in the signal search region ;
A vehicle exterior environment recognition device comprising:   The second possibility determination unit determines that there is a possibility that a progress signal lamp is latent if an occupation area ratio of the shielding object in the signal search region is equal to or greater than a predetermined threshold. Item 1. The vehicle environment recognition device according to Item 1.   The second possibility determination unit searches for a shielding object that shields the signal search area only in a predetermined relative distance range with the selected specific object as a reference, and determines a possibility that a progress signal lamp is latent. The external environment recognition device according to claim 1 or 2, characterized in that   The said 1st possibility determination part determines the possibility that a progress signal lamp is latent based on the horizontal position and height of the said selection specific thing and the said stop signal lamp. The outside environment recognition device according to any one of the above. Get an image of the environment outside the car,
Deriving the relative distance of the object corresponding to the partial image of the image with respect to the host vehicle,
Identifying one or more specific objects based on the relative distance;
Selecting one specific object to be tracked from the one or more specific objects;
Identify stop signal lights that indicate stop or no progress based on color and size in the image,
Based on the positional relationship between the selected specific object, which is the selected specific object, and the stop signal light, determine the possibility of a progress signal light indicating the progress permission attached to the stop signal light being latent,
If it is not possible to determine the possibility that the progress signal lamp is latent by the determination, search for a shielding object that shields a signal search area determined in advance with reference to the stop signal lamp in a space having a relative distance shorter than the stop signal lamp. And determining the possibility that the traveling signal lamp is latent based on the occupied area ratio of the shielding object in the signal search area .