JP6654870B2 - Outside environment recognition device - Google Patents

Description

  The present invention relates to an external environment recognizing device that determines whether a lamp of a preceding vehicle is on or off.

  2. Description of the Related Art Conventionally, a specific object such as a vehicle located in front of a host vehicle is detected to avoid a collision with a preceding vehicle (collision avoidance control), and control is performed so as to maintain a safe inter-vehicle distance with a preceding vehicle ( A cruise control technique is known (for example, Patent Document 1). Here, if a process of recognizing whether or not the brake lamp of the preceding vehicle is lit (operation state of the brake) can be incorporated, smoother cruise control can be performed.

  As a technique for detecting the presence or absence of the lighting of the brake lamp of such a preceding vehicle, a technique of detecting the lighting of the brake lamp based on a change in luminance or a change in area of the tail lamp detection area (for example, Patent Document 2), and a technique of detecting the luminance in a red area. There is also disclosed a technique for generating a histogram distribution of (brightness) and determining whether or not a brake lamp is lit based on the standard deviation (for example, Patent Document 3).

Japanese Patent No. 3349060 Japanese Patent No. 3872179 JP-A-9-267686

  In the technique of Patent Literature 2 described above, since the lighting of the tail lamp can be confirmed only at night, there is a problem that a change in luminance or a change in area of the tail lamp detection region may not be acquired in daytime. Further, in the technique of Patent Document 3, the standard deviation of the luminance is greatly offset depending on the shape of the cover of the brake lamp, the type of the light source, and the sunshine condition. There was a risk of recognition.

  The present invention has been made in view of the above problems, and has as its object to provide an outside-vehicle environment recognition device that can accurately determine whether a lamp is lit.

In order to solve the above problems, the environment outside the vehicle recognition system of the present invention is a computer, an image acquiring unit that acquires color image chronologically, in the color image, the vehicle region preceding vehicle is occupied, the preceding vehicle and the vehicle identification unit for identifying the relative distance between, in particular said vehicle area, counting the number of pixels or pixel area satisfying the color condition based on color predetermined threshold, the said number of pixels or pixel area An area conversion unit that converts the detected area into a detected area based on a relative distance to a preceding vehicle; a history storage unit that stores the detected area in a storage unit; and a waveform corresponding to a history of the detected area stored in the storage unit. , with frequency derives the temporal low-frequency waveform of the detection area is less than a predetermined frequency threshold, the frequency is the frequency threshold value larger than the detection area time A waveform separating unit for deriving a high-frequency waveform in a direction, and a section in which the value of the low-frequency waveform is equal to or greater than a value obtained by multiplying the value of the high-frequency waveform by a predetermined separation ratio, and whether or not the lamp is lit in the vehicle area A lighting presence / absence change determining unit that determines that the lighting presence / absence change section has changed , and classifying the detection area in the lighting presence / absence change section into one of two classes based on the size of the detection area. A histogram generation unit that integrates a voting value to an area division including the detected area in the time direction to generate a histogram, and derives an area threshold based on a correspondence relationship between frequencies of the histogram for each classified class. and area threshold value deriving unit that, by comparing the detection area and the area threshold value, the lighting determining unit determines the lighting Without light in the vehicle area Functional.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to determine whether a lamp is lit or not accurately.

FIG. 3 is a block diagram illustrating a connection relationship of the environment recognition system. It is a functional block diagram showing a schematic function of an outside environment recognition device. FIG. 3 is an explanatory diagram for explaining a color image and a distance image. It is an explanatory view for explaining processing of a vehicle specific part. It is an explanatory view for explaining processing of a vehicle specific part. FIG. 4 is an explanatory diagram for explaining a difference between imaging by a first exposure mode and imaging by a second exposure mode. FIG. 9 is an explanatory diagram for explaining a light emission source identification table. FIG. 4 is an explanatory diagram showing a color threshold. FIG. 3 is an explanatory diagram showing a relationship between a relative distance between a host vehicle and a preceding vehicle and the number of pixels. FIG. 4 is an explanatory diagram for explaining integration in a time direction by a histogram generation unit. FIG. 4 is an explanatory diagram for explaining a histogram. FIG. 9 is an explanatory diagram for explaining a tallying process performed by an area threshold deriving unit; It is a flowchart which shows an outside environment recognition process. It is a flowchart which shows an area threshold value derivation process.

  Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings. The dimensions, materials, other specific numerical values, and the like shown in the embodiments are merely examples for facilitating the understanding of the invention, and do not limit the present invention unless otherwise specified. In the specification and the drawings, elements having substantially the same function and configuration will be denoted by the same reference numerals, and redundant description will be omitted. Elements not directly related to the present invention will be omitted. I do.

  In recent years, the on-board camera mounted on the vehicle has taken an image of the road environment ahead of the own vehicle, and specified a three-dimensional object such as a preceding vehicle based on color information and position information in the captured image. 2. Description of the Related Art Vehicles equipped with a collision prevention function for avoiding a collision and an adaptive cruise control (ACC) for keeping a distance between the vehicle and a preceding vehicle at a predetermined distance are becoming widespread.

  In such a collision prevention function or ACC, for example, a relative distance between the three-dimensional object located in front of the own vehicle and the own vehicle is derived, and a collision between the three-dimensional object located in front of the own vehicle is determined based on the relative distance. If the object is avoided or the three-dimensional object is a vehicle (preceding vehicle), control is performed so that the relative distance from the preceding vehicle is kept at a predetermined distance. Further, by incorporating a process for recognizing whether or not the brake lamp of the preceding vehicle is lit (lighted or turned off), it is possible to realize a smoother collision prevention function and ACC. Hereinafter, an environment recognition system for achieving such an object will be described, and an outside environment recognition device as a specific component thereof will be described in detail.

(Environment recognition system 100)
FIG. 1 is a block diagram showing a connection relationship of the environment recognition system 100. The environment recognition system 100 includes an imaging device 110, an outside environment recognition device 120, and a vehicle control device (ECU: Engine Control Unit) 130 provided in the vehicle 1.

  The imaging device 110 includes an imaging device such as a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS), and captures an image of the front of the vehicle 1 to generate a color image represented by a color value. I do. Here, the color value is a numerical value group consisting of one luminance (Y) and two color differences (UV) or a numerical value group consisting of three hues (R (red), G (green), and B (blue)). It is.

  The imaging devices 110 are arranged in a substantially horizontal direction so that the optical axes of the two imaging devices 110 are substantially parallel on the traveling direction side of the vehicle 1. The imaging device 110 continuously generates image data of an image of a detection area in front of the host vehicle 1, for example, every 1/20 second (20 fps). Here, specific objects to be identified from the image data generated by the imaging device 110 include not only three-dimensional objects such as vehicles, pedestrians, traffic lights, roads (traveling paths), guardrails, and buildings, but also brake lamps. , High-mount stop lamps, tail lamps, turn signals, traffic lights, and other parts that can be specified as part of a three-dimensional object are also included. Each functional unit in the following embodiments performs each process for each frame triggered by such update of the image data.

  Furthermore, in the present embodiment, the imaging device 110 images the detection area in a first exposure mode indicating an exposure time and an aperture according to the brightness of the environment outside the vehicle (a measurement result of an illuminometer and the like), and generates a first image. I do. Further, the imaging device 110 generates a second image capable of determining whether a specific light source such as a brake lamp emits light by itself. As the method, an image pickup device having a wide dynamic range may be used to take an image so that a three-dimensional object that does not emit light is not crushed black and the light emission source does not overexpose. The detection area may be imaged in the second exposure mode having different times and apertures to generate a second image. For example, in the daytime, the second image is generated by shortening the exposure time of the second exposure mode or increasing the aperture in comparison with the exposure time of the first exposure mode according to a bright outside environment. In the present embodiment, each of the first image and the second image is used as a color image. Further, the first exposure mode and the second exposure mode are realized as follows.

  For example, the imaging device 110 sequentially generates a first image and a second image by time-sharing a periodic imaging timing and alternately performing imaging in the first exposure mode and imaging in the second exposure mode. . Note that the imaging device 110 is provided with two capacitors for each pixel, and in an imaging device capable of charging electric charges in parallel to the two capacitors, two different exposure modes with different charging times in one exposure. Images may be generated in parallel. In addition, the imaging device 110 may read twice twice at different times during the charging of one capacitor, and generate two images having different exposure modes in parallel. Further, the imaging device 110 is configured by two sets of imaging devices having different exposure modes (here, two imaging devices 110 × 2 sets), and images may be generated from the two sets of imaging devices 110, respectively. . The exposure time that governs the exposure mode is appropriately controlled, for example, in the range of 1 to 60 msec.

  When the three-dimensional object is identified as the preceding vehicle, the outside environment recognition device 120 derives the relative speed of the preceding vehicle, the relative distance to the preceding vehicle, and the like while tracking the preceding vehicle, and the preceding vehicle collides with the host vehicle 1. It is determined whether there is a high possibility of performing. Here, when it is determined that the possibility of collision is high, the outside environment recognizing device 120 gives a warning display (notification) to the driver through the display 122 installed in front of the driver, and the vehicle control device. Information indicating that fact is output to 130.

  The vehicle control device 130 controls the host vehicle 1 by receiving a driver's operation input through the steering wheel 132, the accelerator pedal 134, and the brake pedal 136, and transmitting the input to the steering mechanism 142, the drive mechanism 144, and the brake mechanism 146. Further, the vehicle control device 130 controls the steering mechanism 142, the driving mechanism 144, and the braking mechanism 146 according to the instruction of the outside environment recognition device 120.

  Hereinafter, the configuration of the outside environment recognition device 120 will be described in detail. Here, a configuration characteristic of the present embodiment will be described in detail, and a description of a configuration unrelated to the characteristics of the present embodiment will be omitted.

(External environment recognition device 120)
FIG. 2 is a functional block diagram showing a schematic function of the outside environment recognition device 120. As shown in FIG. 2, the external environment recognition device 120 includes an I / F unit 150, a data holding unit 152, and a central control unit 154.

  The I / F unit 150 is an interface for performing bidirectional information exchange with the imaging device 110 and the vehicle control device 130. The data holding unit 152 includes a RAM, a flash memory, an HDD, and the like, holds various information necessary for processing of each function unit described below, and also stores image data (first image and image data) received from the imaging device 110. (A color image and a distance image based on the second image).

  The central control unit 154 is a computer constituted by a semiconductor integrated circuit including a central processing unit (CPU), a ROM in which programs and the like are stored, a RAM as a work area, and the like. , The data holding unit 152 and the like. In the present embodiment, the central control unit 154 includes an image processing unit 160, a position information deriving unit 162, a vehicle specifying unit 164, a lamp specifying unit 166, an area converting unit 168, a history storing unit 170, a waveform separating unit 172, and lighting. It functions as the presence / absence change determination unit 174, the histogram generation unit 176, the area threshold derivation unit 178, and the lighting presence / absence determination unit 180. Hereinafter, detailed operations of such functional units will be described in the order of image processing, vehicle specifying processing, lamp specifying processing, area threshold value determining processing, lighting presence / absence determining processing, based on the general purpose.

(Image processing)
The image processing unit 160 acquires color image data (first image and second image) from each of the two imaging devices 110, and arbitrarily extracts a block (for example, 4 horizontal pixels × 4 vertical pixels) from one of the first images. Parallax is derived using so-called pattern matching, in which a block corresponding to the (pixel array) is searched from the other first image. The image processing unit 160 also derives parallax for the second image using pattern matching. Here, “horizontal” indicates the horizontal direction of the captured color image on the screen, and “vertical” indicates the vertical direction of the captured color image on the screen.

  As this pattern matching, it is conceivable to compare the luminance (Y color difference signal) between two images in a block unit indicating an arbitrary image position. For example, SAD (Sum of Absolute Difference) for calculating the difference in luminance, SSD (Sum of Squared intensity Difference) using the difference squared, and NCC for calculating the similarity of the variance obtained by subtracting the average value from the luminance of each pixel. (Normalized Cross Correlation). The image processing unit 160 performs such a parallax deriving process in block units for all blocks displayed in the detection area (for example, 600 pixels horizontally × 180 pixels vertically). Here, the block is 4 pixels horizontally × 4 pixels vertically, but the number of pixels in the block can be set arbitrarily.

  However, the image processing unit 160 can derive the parallax for each block that is the unit of detection resolution, but cannot recognize what kind of three-dimensional object the block is. Therefore, the parallax information derived based on the parallax is independently derived not in units of three-dimensional objects but in units of detection resolution (for example, blocks) in the detection area.

  The position information deriving unit 162 uses the so-called stereo method based on the parallax for each block (for each three-dimensional part) in the detection area derived by the image processing unit 160, and includes a horizontal distance, a height, and a relative distance. Deriving dimensional position information. Here, the stereo method is a method of deriving a relative distance of the three-dimensional part with respect to the imaging device 110 from the parallax of the three-dimensional part by using a triangulation method. At this time, the position information deriving unit 162 determines the three-dimensional part road surface based on the relative distance of the three-dimensional part and the distance on the distance image from the point on the road surface at the same relative distance to the three-dimensional part to the three-dimensional part. Deriving the height from Note that an image in which the disparity information (three-dimensional position information) derived in this manner is associated with image data is referred to as a distance image in order to distinguish it from the above-described color image.

  FIG. 3 is an explanatory diagram for explaining the color image 210 and the distance image 212. For example, it is assumed that a color image (image data) 210 as shown in FIG. Here, for ease of understanding, only one of the two color images 210 is schematically shown. In the present embodiment, the image processing unit 160 obtains parallax for each three-dimensional part from such a color image 210, and the position information deriving unit 162 derives three-dimensional position information for each three-dimensional part based on the parallax, A distance image 212 as shown in FIG. 3B is formed. Each three-dimensional part in the distance image 212 is associated with parallax information of the three-dimensional part. Here, for convenience of description, the three-dimensional part from which the parallax information is derived is represented by black dots. In the present embodiment, such a color image 210 and a distance image 212 are generated based on each of the first image and the second image. Therefore, in the present embodiment, a color image 210 based on the first image, a distance image 212 based on the first image, a color image 210 based on the second image, and a distance image 212 based on the second image are used.

(Vehicle identification processing)
FIG. 4 and FIG. 5 are explanatory diagrams for explaining the processing of the vehicle specifying unit 164. First, the vehicle specifying unit 164 divides the detection area 214 of the distance image 212 based on the first image into a plurality of divided areas 216 in the horizontal direction. Then, the divided area 216 has a strip shape as shown in FIG. Such a strip-shaped divided area 216 is originally composed of, for example, 150 rows arranged with a horizontal width of 4 pixels. However, here, for convenience of explanation, the detection area 214 will be described as being divided into 16 equal parts.

  Subsequently, the vehicle identification unit 164 integrates the relative distances included in each of the plurality of divided predetermined distances for all blocks located above the road surface based on the position information for each of the divided areas 216. A histogram (shown by a horizontally long square (bar) in FIG. 4B) is generated. Then, a distance distribution 218 as shown in FIG. 4B is obtained. Here, the vertical direction indicates the divided predetermined distances (distance divisions), and the horizontal direction indicates the number (frequency) of blocks including relative distances in each distance division. However, FIG. 4B is a virtual screen for performing the calculation, and does not actually generate a visual screen. Then, the vehicle specifying unit 164 refers to the distance distribution 218 derived in this manner, and calculates a representative distance (indicated by a black square in FIG. 4B) 220 which is a relative distance corresponding to the peak. Identify. Here, “corresponding to a peak” refers to a peak value or a value near a peak that satisfies an arbitrary condition.

  Next, the vehicle specifying unit 164 compares the adjacent divided areas 216 with each other, and groups the divided areas 216 having the close representative distance 220 (for example, located at 1 m or less) as shown in FIG. A plurality of divided area groups 222 are generated. At this time, even when the representative distance 220 is close in three or more divided areas 216, all the continuous divided areas 216 are combined as a divided area group 222. By this grouping, the vehicle specifying unit 164 can specify a three-dimensional object located above the road surface.

  Subsequently, the vehicle specifying unit 164 sets a block having a relative distance corresponding to the representative distance 220 in the divided area group 222 as a base point, and determines a difference between the block, the horizontal distance difference, the height difference, and the relative distance in advance. Blocks within a predetermined range (for example, 0.1 m) are grouped on the assumption that they correspond to the same specific object. Thus, a three-dimensional object 224, which is a virtual block group, is generated. The above range is represented by a distance in a real space, and can be set to an arbitrary value by a manufacturer or a passenger. Also, the vehicle specifying unit 164 further groups blocks having a difference in horizontal distance, a difference in height, and a difference in relative distance within a predetermined range with respect to the block newly added by the grouping. Become As a result, all blocks that can be assumed to be the same specific object are grouped.

Further, here, the difference in the horizontal distance, the difference in the height, and the difference in the relative distance are each independently determined, and the same group is set only when all are included in the predetermined range. However, another calculation may be performed. . For example, the predetermined range includes the mean square of the difference of the horizontal distance, the difference of the height, and the difference of the relative distance √ ((the difference of the horizontal distance) ² + (the difference of the height) ² + (the difference of the relative distance) ² ). If the same group is used, the same group may be used. By such a calculation, an accurate distance in the real space between the blocks can be derived, so that the grouping accuracy can be improved.

  Next, if the grouped three-dimensional objects 224 satisfy a predetermined condition corresponding to a predetermined vehicle, the vehicle specifying unit 164 determines the three-dimensional object 224 as a specific object “vehicle”. For example, when the grouped three-dimensional object 224 is located on a road, the vehicle specifying unit 164 determines whether or not the entire size of the three-dimensional object 224 corresponds to the size of the specific object “vehicle”. If it is determined that the size corresponds to the size of the specific object “vehicle”, the three-dimensional object 224 is specified as the specific object “vehicle”. Here, the vehicle specifying unit 164 specifies a rectangular area occupied on the screen by the three-dimensional object 224 specified as the specific object “vehicle” as a vehicle area.

  In this way, the outside environment recognition device 120 can extract one or a plurality of three-dimensional objects 224 as a specific object, for example, a vehicle (preceding vehicle) from the distance image 212 based on the first image, and various types of information can be extracted. It can be used for control. For example, if an arbitrary three-dimensional object 224 in the detection area 214 is specified as a vehicle, the specified vehicle (preceding vehicle) is tracked, and a relative distance and a relative acceleration are detected to avoid a collision with the preceding vehicle. Or the distance between the vehicle and the preceding vehicle can be controlled to be a safe distance. In order to more quickly identify such a preceding vehicle and grasp the behavior of the preceding vehicle, a brake lamp existing in the vehicle area is specified, and whether or not the brake lamp is lit is determined.

(Lamp identification processing)
The second image is, for example, an image captured in the second exposure mode in which a specific light source (here, a brake lamp) can be determined. Here, a device that emits light by itself, such as a brake lamp, can acquire a high color value regardless of the brightness of the sun or street lamps. In particular, since the brightness at the time of turning on the brake lamp is generally specified by regulations, by taking an image in an exposure mode (for example, short-time exposure) in which only a predetermined brightness can be exposed, only pixels corresponding to the brake lamp are exposed. Can be easily extracted.

  FIG. 6 is an explanatory diagram for explaining a difference between the imaging by the first exposure mode and the imaging by the second exposure mode. FIG. 6A shows a color image 210 based on the first image according to the first exposure mode. In particular, in the left diagram of FIG. 6A, the tail lamp is turned on, and in the right diagram of FIG. Brake lights are on in addition to tail lights. As can be understood with reference to FIG. 6A, in the first exposure mode according to the brightness of the environment outside the vehicle, the color of the tail lamp position 230 when the brake lamp is off and the tail lamp is on There is almost no difference in color value between the value and the brake lamp position 232 when the brake lamp is on and the tail lamp is on. This is because in the first exposure mode in which the exposure time is long, the color values of all the RGB components of both the tail lamp and the brake lamp are saturated.

  FIG. 6B shows a color image 210 based on the second image according to the second exposure mode. In particular, in the left diagram of FIG. 6B, the tail lamp is turned on, and in the right diagram of FIG. Brake lights are on in addition to tail lights. The second exposure mode is set so that only the color value when the brake lamp is lit can be obtained. Therefore, as can be understood with reference to FIG. 6B, even when the tail lamp is turned on, it is hardly possible to acquire a color value corresponding to the brightness at the tail lamp position 230, and when the brake lamp is turned on. At the brake lamp position 232, a clearly high color value can be obtained.

  In the second exposure mode, it is desirable to set an exposure time such that the R component of the color value of the brake lamp is saturated or not in the image sensor. Since the imaging device 110 usually has a much narrower dynamic range than a human, if the imaging is performed in the first exposure mode with low brightness in the evening, the color value of the brake lamp becomes relatively high with respect to the environment outside the vehicle. Then, not only the R component but also the G component and the B component overlap with the R component and are saturated to the maximum value (for example, the color value is 255), and the pixel becomes white. Therefore, by setting the second exposure mode to an exposure time such as whether or not the R component is saturated when the brake lamp is turned on, the influence of the G component and the B component on the color value is suppressed regardless of the external environment. While extracting only the R component with the maximum value. Thus, for example, it is possible to ensure the maximum difference in color value from the tail lamp.

  More specifically, if a preceding vehicle is present during night driving and the second exposure state is set so that the brake lamp at the time of lighting has a color range (R) of “200 or more”, the tail lamp at the time of lighting Are displayed in the color image 210 based on the second image in, for example, the color range (R) “50”, the color range (G) “50”, and the color range (B) “50”. On the other hand, the brake lamp at the time of lighting is displayed in the second image in, for example, the color range (R) “200 or more”, the color range (G) “50 or less”, and the color range (B) “50 or less”. You. Thus, the lamp specifying section 166 can specify the brake lamp through the color image 210 based on the second image.

  FIG. 7 is an explanatory diagram for explaining the light emission source identification table. In the light emission source specification table, for a plurality of specific objects, a color range indicating a range of color values (here, R, G, and B), a height range indicating a range of height from the road surface, and a specific object. The horizontal distance width range, the vertical distance width range of the specific object, the horizontal distance difference with the same specific object, the vertical distance difference with the same specific object, and the area ratio with the same specific object correspond It is attached. Here, as the specific object, there are various brakes required for specifying a vehicle, such as a “brake lamp (red)”, a “high-mount stop lamp (red)”, a “tail lamp (red)”, and a “blinker (orange)”. Things are assumed. However, it goes without saying that the specific object is not limited to the one described in FIG. Among the specific objects, for example, the specific object “brake lamp (red)” has a color range (R) of “200 or more”, a color range (G) of “50 or less”, a color range (B) of “50 or less”, and high. Height range “0.3 to 2.0 m”, horizontal distance width range “0.05 to 0.2 m”, vertical distance width range “0.05 to 0.2 m”, and horizontal distance difference “1.4”. １1.9 m ”, the vertical distance difference“ 0.3 m or less ”, and the area ratio“ 50-200% ”.

  The lamp identifying unit 166 acquires color values of three hues (R, G, B) in pixel units from an area corresponding to the vehicle area of the color image 210 based on the second image. Then, the lamp specifying unit 166 specifies, for example, a pixel that satisfies the color range of the brake lamp with reference to the light emitting source specifying table. If the detection area 214 is, for example, rainy or cloudy, it may be acquired after adjusting the white balance so that the original color value can be acquired.

  If the horizontal distance difference, the height difference, and the relative distance difference between the specified pixels are within a predetermined range (for example, 0.1 m), the lamp specifying unit 166 groups the plurality of pixels as one brake lamp. As a result, the grouped pixels are temporarily specified as brake lamps. In this way, even if a plurality of pixels constituting the brake lamp straddle, it can be specified as one brake lamp.

  However, with only the second image according to the second exposure mode, the color value of the entire detection area 214 becomes low (dark) at night or the like, and it becomes impossible to grasp anything other than the light source such as the brake lamp. Therefore, the temporarily specified “brake lamp” is associated with the specific object “vehicle” specified by the first image in the first exposure mode described above.

  The lamp identification unit 166 associates the vehicle area grouped as the specific object “vehicle” by the vehicle identification unit 164 with the position of the identified specific object “brake lamp”. Then, the tracking of the specific object “vehicle” by the vehicle specifying unit 164 and the tracking of the specific object “brake lamp” by the lamp specifying unit 166 are supported, and one position information is used to calibrate the other position information. Thus, the positional relationship between the outer edge of the preceding vehicle and the brake lamp of the vehicle can be maintained.

  Further, the lamp specifying unit 166 specifies a combination of a pair of brake lamps among the brake lamps temporarily specified to be present in the same preceding vehicle with reference to the light emission source specifying table, and determines the combination of the vehicle area and the brake lamp. It is determined whether the relative arrangement with the position is an appropriate arrangement. Specifically, for example, the lamp identification unit 166 determines that the brake lamp alone has a height range of “0.3 to 2.0 m”, a horizontal distance width range of “0.05 to 0.2 m”, and a vertical distance. It is determined whether the condition of the width range of “0.05 to 0.2 m” is satisfied. Furthermore, the lamp specifying unit 166 determines that the combination of the pair of brake lamps is such that the horizontal distance difference is “1.4 to 1.9 m”, the vertical distance difference is “0.3 m or less”, and the area ratio is “50 to 200%”. It is determined whether the condition is satisfied. In this way, the brake lamp and the provisionally specified light source correspond to the appropriate position of the vehicle only when it is formally specified as a brake lamp. The illuminated light source can be prevented from being erroneously recognized as a brake lamp.

  However, in order to accurately determine the behavior of the preceding vehicle, it is necessary to determine not only the brake lamp but also whether or not the brake lamp is lit. However, depending on the external environment, such as the shape of the brake lamp cover, the type of light source, and the sunshine conditions, the brightness changes under the influence of sunlight reflection and the like. Simply comparing the threshold value with the appropriate threshold value may erroneously determine whether or not the light is on. For example, even though the brake lamp is not lit, the color value is higher than the fixed threshold, and it is erroneously recognized that the brake lamp is lit, or the color value is fixed when the brake lamp is lit. It may be lower, and may be erroneously recognized as not being lit.

  Therefore, in the present embodiment, the specification of the brake lamp by the lamp specifying unit 166 is used as an auxiliary (redundant) in determining the likelihood of the brake lamp. By paying attention to the difference between the luminance change based on the presence or absence of the brake lamp and the luminance change due to a change in the external environment, an area satisfying a predetermined color condition and a threshold (area threshold) are appropriately compared. The presence or absence of lighting is determined.

(Area threshold decision processing)
The area conversion unit 168 counts the number of pixels that satisfy a color condition based on one or more predetermined color thresholds in the vehicle region of the preceding vehicle specified by the vehicle specification unit 164, and converts the number of pixels into an area. (Normalize). Hereinafter, the color threshold value and the color condition will be described.

  FIG. 8 is an explanatory diagram showing a color threshold. In the present embodiment, the standard shutter speed in the second exposure mode is set to 20 msec, and for example, two color threshold values of “yellow” and “red” as shown in FIG. 8 are provided. In the present embodiment, a color condition that satisfies one of the plurality of color thresholds is employed instead of individually using such a plurality of color thresholds. Here, a plurality of color thresholds are prepared as color conditions for the following reason. That is, when the brightness of the brake lamp increases, the color value of the hue R is saturated, so that the balance of the R, G, and B color values changes, and the red color changes from yellow to whitish. Therefore, by setting the color condition to be the sum of a predetermined color threshold value and another color threshold value higher in brightness than that, it is possible to appropriately obtain an area in which the brightness is higher than the predetermined color threshold value.

  The area conversion unit 168 first counts the number of pixels that satisfy one of the two color conditions shown in FIG. That is, in the vehicle region, a pixel in which the color value of the hue R is higher than 225, the color value of the hue G is lower than 169, and the color value of the hue B is lower than 98, the color value of the hue R is higher than 150, and the hue G , And the pixel whose hue B color value is less than 66 are counted. In the present embodiment, it suffices to derive only one pixel number that satisfies the color condition for each frame, so that the processing load can be reduced.

  By the way, if the relative distance between the host vehicle 1 and the preceding vehicle is long, the number of light emitting sources satisfying the color condition is small, and the number of pixels is also small. On the other hand, when the relative distance to the preceding vehicle is short, the area of the light emitting source that satisfies the color condition increases, and the number of pixels increases. Therefore, even if the brake lamp keeps lighting, the number of pixels satisfying the color condition changes according to the change in the relative distance from the preceding vehicle. For example, if the number of pixels satisfying the color condition differs depending on the positional relationship with the preceding vehicle even though the brake lamp is lit, the brake lamp is originally lit, and there are pixels that satisfy the color condition. Again, the result can be that the relative distance is too long and the number is less than the threshold. Therefore, in the present embodiment, the number of pixels is converted into an actual area based on the relative distance from the preceding vehicle.

  FIG. 9 is an explanatory diagram showing the relationship between the relative distance between the host vehicle 1 and the preceding vehicle and the number of pixels. In FIG. 9, the horizontal axis indicates the relative distance, and the vertical axis indicates the number of pixels occupied by a three-dimensional object of a predetermined size. As can be understood with reference to FIG. 9, even with the same three-dimensional object (same area), the number of pixels decreases as the relative distance increases. Such a transition can be approximated by a function, and is proportional to the relative distance from the relative distance 0 point to the relative distance a point in FIG. 9, and is proportional to the 3/2 power of the relative distance after the point a. Usually, the size of a three-dimensional object in an image is simply proportional to its relative distance, but in the case of a light emitting source, the apparent light emitting range is widened due to the influence of light emission. Therefore, the relationship between the relative distance and the number of pixels becomes non-linear as shown in FIG.

  Therefore, the area conversion unit 168 also specifies the relative distance to the preceding vehicle (light emission source), and, based on the relative distance to the preceding vehicle, divides by the inverse function of FIG. 9 (by dividing by the number of pixels in FIG. 9). The number of pixels that satisfy the color condition based on the color threshold shown in FIG. 8 is converted into an area (hereinafter, referred to as a detection area).

  The history storage unit 170 stores the detected area converted by the area conversion unit 168 in the data holding unit 152 in association with the time T at which the frame was acquired for each frame, thereby obtaining a history of the detected area. Accumulate. Note that the history storage unit 170 accumulates, for example, the history of the detected areas in the history holding period T1 corresponding to the latest 256 (256 frames), and deletes the detected areas earlier than that.

  Subsequently, the waveform separating unit 172 reads the history of the detected area stored in the data holding unit 152, and derives an average value of the read detected area. Then, the waveform separation unit 172 subtracts the average value (DC component) for each of the read detection areas to obtain a curve extending over positive and negative, and performs FFT (Fast Fourier Transform) on the subtracted detection area, The frequency component of the detection area in the history holding period T1 is derived.

  After that, the waveform separation unit 172 extracts only low frequency components equal to or less than a predetermined frequency threshold from the derived frequency components of the detection area. Here, since the brightness change of the brake lamp changes when the driver depresses the brake pedal, it is sufficiently slower than the brightness change due to a change in the external environment such as reflection of sunlight. Therefore, the waveform of the detection area in the time direction has a frequency at which the waveform component due to the change in the brightness of the brake lamp is lower than the waveform component due to the change in the brightness due to a change in the external environment such as sunlight reflection. The frequency threshold is a threshold for separating a frequency component due to a change in brightness of the brake lamp from a frequency component due to a change in brightness due to a change in the external environment, and is set to, for example, 0.3 Hz.

  Then, the waveform separation unit 172 derives a low-frequency waveform in the time direction of the detection area in the history holding period T1 by performing an inverse transform of the FFT on the extracted low-frequency component. That is, the waveform separation unit 172 derives a low-frequency waveform excluding the DC component and the high-frequency waveform larger than the frequency threshold from the waveform in the time direction of the history of the detection area.

  In addition, the waveform separation unit 172 determines, based on the derived low-frequency waveform, a frame interval T2 (20 ms in this case), which is an interval at which frames are acquired, before and after the center of the history holding period T1 in the time direction. Extract every 2 minutes. That is, the waveform separation unit 172 extracts a low-frequency waveform from the time (T-history holding period T1 / 2−frame interval T2 / 2) to the time (T−history holding period T1 / 2 + frame interval T2 / 2). become.

  In addition, the waveform separating unit 172 extracts only high frequency components higher than the frequency threshold from the frequency components of the detection area subjected to the FFT, and performs an inverse transform of the FFT on the extracted high frequency components. , A high-frequency waveform in the time direction of the detection area in the history holding period T1 is derived. Thereafter, the waveform separation unit 172 derives the average value of the derived high-frequency waveform as the high-frequency waveform value at the time (T-history retention period T1 / 2 + frame interval T2 / 2).

  As described above, the waveform separating unit 172 derives a low-frequency waveform resulting from the presence or absence of the brake lamp and a high-frequency waveform value resulting from a change in the external environment from the waveform corresponding to the history of the detected area. Here, in order to determine a change in the presence or absence of the lighting of the brake lamp, only a low-frequency waveform may be derived. On the other hand, when a change in the external environment occurs, that is, when the high-frequency waveform value is high, the low-frequency waveform is also affected. The low frequency waveform may change.

Thus, the lighting presence / absence change determination unit 174 calculates a value obtained by multiplying the derived high-frequency waveform value by an S / N (Signal / Noise) separation ratio (for example, 100) and a value of the low-frequency waveform at each corresponding time T. Compare to Then, the lighting presence / absence change determination unit 174 determines that the value of the low frequency waveform is larger than the value obtained by multiplying the high frequency waveform value by the S / N separation ratio because the value of the low frequency waveform increases when the lighting status of the brake lamp changes. A large section is determined as a lighting presence / absence change section in which the presence / absence of lighting of the brake lamp has changed. In this lighting presence / absence change section, the lighting presence / absence of the brake lamp has changed, that is, it has been determined that only the lighting has changed from off to lighting or the lighting has changed from lighting to lighting. Can not be determined.

  Thus, by comparing the value obtained by multiplying the high-frequency waveform value by the S / N separation ratio with the low-frequency waveform, it is possible to remove the influence of a change in the external environment and to determine that the presence or absence of the brake lamp has changed. The determination can be made with high sensitivity (high accuracy).

  Subsequently, the histogram generation unit 176 generates a histogram based on the detected area of the lighting presence / absence change section. Here, the histogram indicates the frequency for each of the plurality of different area divisions as a class. The histogram generation unit 176 votes a predetermined voting value (for example, 1) for an area division including the detected area converted by the area conversion unit 168 for each frame in the lighting presence / absence change section. However, the histogram generation unit 176 generates a histogram based on the result of integrating the voting values of the voted area divisions in the time direction.

  Here, in the present embodiment, a plurality of different levels of area divisions are defined as classes, and a plurality of histograms indicating the frequency of each area division are prepared. The plurality of histograms include a lighting histogram and a lighting histogram. Specifically, when a predetermined voting condition is satisfied, the area threshold deriving unit 178 integrates a voting value in the time direction and votes for a histogram that satisfies the voting condition.

  More specifically, the area threshold deriving unit 178 sets the detected area of each frame in the lighting presence / absence change section in which the lighting presence / absence of the brake lamp is determined to have changed by the lighting presence / absence change determination unit 174 to the size of the area. Based on, for example, the least squares method, the data is classified into two classes. Then, among the classified classes, the detection area classified into the class having the relatively large detection area is voted on the histogram at the time of lighting, and the detection area classified into the class having the relatively small detection area at the time of turning off the light. Vote on the histogram.

FIG. 10 is an explanatory diagram for explaining integration in the time direction by the histogram generation unit 176. Histogram generation unit 176, the voting value X _n of the area division who voted integrated in the time direction according to Equation 1 below, determine the degree Y _n. Here, n indicates the current frame, and n-1 indicates the previous frame.
Y _n = 0.99 × Y _n−1 + 0.01 × X _n (Formula 1)
Such Equation 1 is first-order lag function, the voting value X _n at the current frame, only affect about 0.01 relative degree Y _n. Further, the time constant of the first-order lag in Equation 1 is sufficiently longer than the time required for switching between turning on and off the brake lamp. Therefore, it is suppressed that the area in the middle of switching between lighting and extinguishing appears in the histogram.

Accordingly, as shown in FIG. 10 (a), if the detection area is a predetermined period of time that has been converted by the area conversion section 168 are included in the same area segment S in _0, while the to area classification S ₀ When the state in which the voting value X _n is 1 is continuous, the frequency Y _n of the area section S ₀ changes to 1 with a first-order delay as shown in FIG. Then, it changes to 0 with a first-order delay. Also, the area varies by more than the area division S _0, power Y _n of the area segment S ₀ becomes possible to vary between 0 and 1.

FIG. 11 is an explanatory diagram for explaining a histogram. When the brake lamp is on, the area that satisfies the color condition is large, and when the brake lamp is off, the area that satisfies the color condition is small. Therefore, when the lighting histogram is generated, as shown in FIG. 11A, the frequency approaches ₁ in the relatively large area section S1 when the brake lamp is lit. On the other hand, when generating the light-off histograms, frequency approaches 1 at a relatively small area divided S ₂ when off the brake lamp.

However, since both the host vehicle 1 and the preceding vehicle are usually moving, the relative positional relationship changes, and the area satisfying the color condition also changes. Therefore, when such a positional relationship generates a histogram in situations varying, as shown in FIG. 11 (b), the lighting time histogram, with a relatively large area division S ₁ predetermined deviation near a predetermined power appeared, and in light-off histograms, predetermined frequency appears with a relatively small area divided S ₂ predetermined deviation in the vicinity.

FIG. 12 is an explanatory diagram for describing the area threshold deriving process performed by the area threshold deriving unit 178. Here, for convenience of explanation, a plurality of frequency groups are indicated by line segments simulating the outer shape. The area threshold deriving unit 178 counts the detected areas classified into the class having the relatively large detected area in the histogram at the time of lighting as shown in FIG. The area is totaled in a histogram at the time of turning off light as shown in FIG. The area threshold value derivation unit 178, the lighting time of the histogram of the aggregated FIG. 12 (a), the the minimum value L _d of the area segment constituting the frequency group, at the off time of the histogram of the aggregated FIG 12 (b) derives a maximum value _{L u} of the area segment constituting the frequency group, as shown in FIG. 12, to determine the average value _{((L d + L u)} / 2) as the area threshold value L.

Here, although the minimum L _d and the maximum value L _u area threshold value using the average value of L, the present invention is not limited in such a case, the average value and the mode value of the frequency group at the histogram on, off time histogram The average value of the frequency group and the average value with the mode may be set as the area threshold L.

(Lighting presence / absence determination processing)
The lighting presence / absence determining unit 180 compares the detected area converted by the area converting unit 168 with the area threshold L, and determines that the brake lamp is turned off if the detected area is smaller than the area threshold L. If the area is equal to or larger than the area threshold L, it is determined that the brake lamp is on.

  In this way, it is possible to derive an appropriate area threshold that can distinguish between lighting and turning off, and the outside environment recognizing device 120 can change the area satisfying the color condition regardless of the change in the positional relationship with the preceding vehicle. , It is possible to accurately determine whether or not the brake lamp is lit in the preceding vehicle.

(Flow of outside environment recognition processing)
Next, the flow of the outside environment recognition processing including the above-described image processing, vehicle specification processing, lamp specification processing, area threshold value determination processing, and lighting presence / absence determination processing executed by the central control unit 154 will be described.

  FIG. 13 is a flowchart illustrating the flow of the outside environment recognition process. FIG. 14 is a flowchart illustrating the flow of the area threshold deriving process. As shown in FIG. 13, first, the image processing unit 160 acquires, from the imaging device 110, a first image captured in the first exposure mode and a second image captured in the second exposure mode (S300). ). Then, the image processing unit 160 derives parallax from the acquired image, and the position information deriving unit 162 derives three-dimensional position information for each three-dimensional part based on the derived parallax (S302). Subsequently, the vehicle specifying unit 164 specifies the vehicle and the vehicle region from the three-dimensional objects grouped based on the three-dimensional position information, and specifies the relative position (relative distance) with the preceding vehicle (S304).

  The lamp specifying unit 166 tentatively specifies, as the brake lamp, a group of pixels in which the color values of the pixels constituting the second image satisfy the color range of the brake lamp (S306). (S308). Further, the lamp specifying unit 166 specifies a combination of a pair of brake lights provisionally specified to be present in the same preceding vehicle, and determines whether or not the relative arrangement between the vehicle area and the position of the brake lights is an appropriate arrangement. (S310).

  After that, the area conversion unit 168 counts the number of pixels satisfying the predetermined color condition, converts the number of pixels into a detection area based on the relative distance from the preceding vehicle (S312), and the history storage unit 170 converts the number of pixels into the detection area. The detected area is stored in the data holding unit in association with the time T (S314). Subsequently, after an area threshold deriving process for deriving an area threshold described below is performed (S400), the lighting presence / absence determining unit 180 compares the area threshold derived in the area threshold deriving process with the detected area. By doing so, it is determined whether or not the brake lamp is lit (S316), and the external environment recognition processing ends.

  Further, the area threshold deriving process (S400) will be specifically described. As shown in FIG. 14, the waveform separation unit 172 reads the history of the detection area stored in the data holding unit 152, and based on the read detection area. Then, a low-frequency waveform at time (T-history holding period T1 / 2 + frame interval T2 / 2) from time ((T-history holding period T1 / 2−frame interval T2 / 2)) is derived (S402).

  Further, the waveform separation unit 172 reads the history of the detected area stored in the data holding unit 152, and based on the read detected area, starts from time ((T−history holding period T1 / 2−frame interval T2 / 2)). The high frequency waveform value at the time (T-history retention period T1 / 2 + frame interval T2 / 2) is derived (S404).

  Subsequently, the lighting presence / absence change determination unit 174 compares the value obtained by multiplying the high frequency waveform value by the S / N separation ratio with the value of the low frequency waveform at each corresponding time T (S406). If the value of the low frequency waveform is greater than the value obtained by multiplying the high frequency waveform value by the S / N separation ratio (YES in S406), the lighting presence / absence change determination unit 174 determines whether the lighting status of the brake lamp has changed. It is determined to be a change section (S408). On the other hand, if the value of the low frequency waveform is smaller than the value obtained by multiplying the high frequency waveform value by the S / N separation ratio (NO in S406), the lighting presence / absence change determination unit 174 determines whether the lighting status of the brake lamp has changed. The process proceeds to the next process without determining that the section is a change section.

  The histogram generation unit 176 classifies the detection area of the lighting presence / absence change section into classes (S410), and determines whether each of the detection areas of the lighting presence / absence change section is classified into a class whose detection area is relatively large (S412). If the detection area is classified into a class having a relatively large detection area (YES in S412), the histogram generation unit 176 votes the detection area in a histogram at the time of lighting (S414), and the class having a relatively large detection area is determined. If it is not classified as (NO in S412), the detected area is voted for in the off-time histogram (S416).

Thereafter, the area threshold derivation unit 178 derives the minimum value L _d of the area segment constituting the power unit in the lighting time of the histogram, and a maximum value L _u of the area segment constituting the power unit at the off time of the histogram, the average determining a value _{((L d + L u)} / 2) as the area threshold value L (S418), and ends the area threshold value derivation process.

  As described above, the preferred embodiments of the present invention have been described with reference to the accompanying drawings, but it goes without saying that the present invention is not limited to such embodiments. It is obvious to those skilled in the art that various changes or modifications can be conceived within the scope of the claims, and it is understood that these also naturally belong to the technical scope of the present invention. Is done.

  Further, in the above embodiment, whether or not the brake lamp is lit is determined. However, the present invention is not limited to this, and it may be determined whether or not another lamp is lit.

  Further, in the above embodiment, the history holding period T1 is a fixed period, but is not limited thereto, and may be a variable period. For example, the history holding period T1 may be lengthened to more accurately detect a change in the presence or absence of the brake lamp, and the history holding period T1 may be shortened to improve responsiveness. .

  Further, in the above embodiment, the S / N separation ratio is a fixed period, but is not limited thereto, and may be a variable value. For example, the S / N separation ratio is increased to more accurately detect the change in the presence or absence of the brake lamp, and the S / N separation ratio is decreased to improve the response. Is also good. As described above, by changing the history retention period T1 and the S / N separation ratio, it is possible to change the detection accuracy of the change in the presence or absence of the brake lamp and the responsiveness.

  Further, the area conversion unit 168 converts the number of pixels satisfying the predetermined color condition in the vehicle area into the detection area, but converts the pixel area satisfying the predetermined color condition in the vehicle area into the detection area. It may be.

  In the above embodiment, the central control unit 154 is configured by a semiconductor integrated circuit including a central processing unit (CPU), a ROM, a RAM, and the like. However, the present invention is not limited to this, and may be configured by an integrated circuit such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Further, it may be configured by one or a plurality of central processing units, an FPGA, and an ASIC.

  Also, a program that causes the computer to function as the outside environment recognition device 120 and a computer-readable storage medium such as a flexible disk, a magneto-optical disk, a DRAM, an SRAM, a ROM, an NVRAM, a CD, a DVD, and a BD in which the program is recorded. Is also provided. Here, the program refers to data processing means described in an arbitrary language or description method.

  In addition, each step of the outside environment recognition process in this specification does not necessarily need to be performed in a time series in the order described in the flowchart, and may include a process in parallel or by a subroutine.

  Also provided are a program for causing the computer to function as the outside environment recognition device 120 and a computer-readable storage medium such as a flexible disk, a magneto-optical disk, a ROM, a CD, a DVD, and a BD in which the program is recorded. Here, the program refers to a data processing unit described in an arbitrary language or description method.

  INDUSTRIAL APPLICATION This invention can be utilized for the exterior environment recognition apparatus which specifies the brake operation state of a preceding vehicle.

120 Outside environment recognition device 152 Data storage unit (storage unit)
160 Image processing unit (image acquisition unit)
164 Vehicle specifying unit 166 Lamp specifying unit 168 Area conversion unit 170 History storage unit 172 Waveform separation unit 174 Lighting presence / absence change determination unit 176 Histogram generation unit 178 Area threshold derivation unit 180 Lighting presence / absence determination unit

Claims (1)

Computer
An image acquisition unit that acquires color images in time series,
In the color image, and the vehicle identification unit the preceding vehicle to identify the vehicle space occupied, the relative distance between the preceding vehicle,
In particular said vehicle area, counting the number of pixels or pixel area satisfying the color condition based on color predetermined threshold, the detection area based on the number of pixels or pixel area to the relative distance between the preceding vehicle An area conversion unit for conversion;
A history storage unit for storing the detection area in a storage unit,
From the waveform corresponding to the history of the detection area stored in the storage unit, as the frequency to derive a temporal low-frequency waveform of the detection area is less than a predetermined frequency threshold, the frequency is the frequency threshold value larger than the A waveform separation unit that derives a high-frequency waveform in the time direction of the detection area,
A lighting presence / absence change in which a section in which the value of the low frequency waveform is equal to or greater than a value obtained by multiplying the value of the high frequency waveform by a predetermined separation ratio is determined as a lighting presence / absence change section in which the lighting presence / absence of the lamp has changed in the vehicle area. A determination unit ;
The detected area in the lighting presence / absence change section is classified into one of two classes based on the size of the detected area, and for each of the classified classes, a voting value for an area section including the detected area is calculated based on time. A histogram generation unit that generates a histogram by integrating in the direction;
An area threshold deriving unit that derives an area threshold based on the correspondence relationship between the frequencies of the histograms for each classified class,
Wherein the detection area by comparing with the area threshold value, the vehicle exterior environment recognition apparatus characterized by functioning as a determining lighting determining unit lighting Without light in the vehicle area.

Images