JP2004173195A - Device and method for monitoring vehicle - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To recognize the winker of another vehicle traveling on an adjacent lane in order to detect the winker states of other vehicles traveling on the adjacent lane. <P>SOLUTION: A lane recognizing part 13a recognizes a lane on the basis of at least distance data. A vehicle recognizing part 13b specifies another vehicle traveling on the adjacent lane from the recognized lane on the basis of the distance data and recognizes a vehicle side face of the specified vehicle. A winker recognizing part 13c sets a winker area in a color image obtained by imaging the scene including the monitoring area or the distance data with an end part position of the vehicle side face as the reference. In addition, the winker recognizing part 13c detects the pixels of a color component of the winkers on the basis of the color image. The detected pixels positionally corresponding to the area on the color image which corresponds to the winker area are recognized as the winkers. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a vehicle monitoring device and a vehicle monitoring method for monitoring the behavior of another vehicle from a blinker state of another vehicle traveling in an adjacent lane adjacent to the lane in which the host vehicle travels.
[0002]
[Prior art]
In recent years, a vehicle monitoring device using various sensors such as a monocular camera, a stereo camera, and a millimeter-wave radar has attracted attention (for example, see Patent Document 1). In a vehicle monitoring device of this kind, a scene in a predetermined monitoring area is imaged or scanned by a sensor mounted on the host vehicle, and a traveling situation is recognized based on information obtained thereby. Typically, the object to be recognized by the vehicle monitoring device includes a lane (road) and a preceding vehicle. Further, in such a vehicle monitoring device, by further combining a control device (for example, an AT control device, an engine control device or a brake control device) and an alarm device, an inter-vehicle distance-sensitive cruise control and an approach warning of another vehicle are provided. It is also considered to include a driving support function such as. In a vehicle monitoring device with a driving support function, basically, a distance to a preceding vehicle and a relative speed are detected, and a control device and an alarm device are controlled based on the detection results.
[0003]
Generally, when a human drives a vehicle, the driver often controls the traveling of the own vehicle while predicting the behavior of another vehicle. For example, when another vehicle traveling in an adjacent lane interrupts the traveling lane of the own vehicle to change lanes, the driver predicts an interruption to the own lane by recognizing the blinker state of the other vehicle. And so on. Then, in consideration of the inter-vehicle distance between the interrupted other vehicle and the own vehicle, an operation such as a brake operation or a downshift is performed as necessary.
[0004]
In addition, for example, Patent Document 2 discloses a method of detecting a blinker of another vehicle. According to Patent Literature 2, a preceding vehicle is detected as a plurality of windows (window groups) based on a stereo image, and a luminance change in windows on both sides of the window group (that is, both ends on the rear side of the vehicle) is detected. Is used to determine the operating state of the blinker.
[0005]
[Patent Document 1]
JP-A-10-283461
[Patent Document 2]
JP-A-11-14346
[0006]
[Problems to be solved by the invention]
By the way, the conventional vehicle monitoring device recognizes the interrupted vehicle as a preceding vehicle to be monitored on the condition that a part of the interrupted vehicle enters the own lane. Therefore, in such a vehicle monitoring device, there is a possibility that control such as deceleration or acceleration suspension may be delayed as compared with a case where a driver recognizes a lane change of another vehicle and performs a driving operation. This delay may make the driver feel uncomfortable with human driving, and in some cases, may increase the risk of contact with other vehicles. Therefore, in order to avoid such a problem, for example, as disclosed in Patent Document 2, a blinker of another vehicle is detected, and the behavior of the other vehicle is grasped based on the detected state of the blinker. It may be reflected in vehicle monitoring.
[0007]
However, when the other vehicle traveling in the adjacent lane is imaged by the camera from the own vehicle side, since this camera images the other vehicle diagonally from the rear, the other vehicle is observed including the vehicle side surface and the vehicle rear surface. Become. Therefore, if the other vehicle is detected as a block of windows and the blinker positions are detected as both ends of the window group, the blinkers cannot be detected because the both ends of the window group do not correspond to the blinker positions of the other vehicle. There is. Further, depending on the positional relationship between the own vehicle and the other vehicle, it is conceivable that the rear surface of the vehicle of the other vehicle does not fall within the angle of view range of the camera. In this case, the blinker detection method disclosed in Patent Literature 2 detects the blinker. It can be difficult to recognize.
[0008]
The present invention has been made in view of such circumstances, and an object thereof is to recognize a blinker of another vehicle traveling in an adjacent lane in order to detect a blinker state of another vehicle traveling in the adjacent lane. is there.
[0009]
Another object of the present invention is to further improve the safety and reliability of a device that controls the traveling of a vehicle according to vehicle monitoring.
[0010]
[Means for Solving the Problems]
In order to solve such a problem, a first invention provides a vehicle monitoring device that monitors the behavior of another vehicle from a blinker state of another vehicle traveling in an adjacent lane adjacent to the lane in which the host vehicle travels. This vehicle monitoring device has a first camera, a sensor, a lane recognition unit, a vehicle recognition unit, and a blinker recognition unit. Here, the first camera outputs a color image by capturing a scene including the monitoring area. The sensor outputs distance data indicating a two-dimensional distribution of the distance in the monitoring area. The lane recognition unit recognizes a lane in the monitoring area based on at least the distance data. The vehicle recognizing unit specifies, based on the distance data, another vehicle traveling in an adjacent lane from the recognized lanes, and recognizes a side surface of the specified other vehicle. The blinker recognizing unit sets a blinker area on the two-dimensional plane defined by the color image or the distance data based on the recognized end position of the side surface of the vehicle, and based on the color image, the pixel of the color component of the blinker. To detect. Then, a detected pixel that corresponds in position to a region on the color image corresponding to the set blinker region is recognized as a blinker.
[0011]
Here, in the first invention, it is preferable that the blinker recognizing unit sets the blinker region in a region corresponding to the side surface of the vehicle based on the end position. At this time, the blinker recognizing unit may set a blinker region also in a region corresponding to the rear surface of the vehicle based on the end position.
[0012]
Further, in the first aspect, it is preferable that the blinker recognizing unit determines whether or not the blinker of another vehicle is blinking based on a time-series area change of the pixel recognized as the blinker.
[0013]
In the first invention, the blinker recognizing unit sets the blinker region based on a boundary position between the vehicle side surface and the vehicle rear surface when the vehicle recognizing unit further recognizes the rear surface of the identified vehicle. Is preferred.
[0014]
Furthermore, the first invention further includes a second camera that functions as a stereo camera by cooperating with the first camera, by outputting a color image by capturing a scene including the monitoring area. Is preferred. In this case, it is desirable that the sensor outputs distance data by stereo matching based on the color image output from the first camera and the color image output from the second camera. On the other hand, the first invention may further include a monochrome stereo camera that outputs a pair of image data by imaging a scene including a monitoring area. In this case, it is preferable that the sensor output as distance data by stereo matching based on a pair of image data output from the monochrome stereo camera.
[0015]
According to a second aspect of the present invention, based on a color image obtained by capturing a scene including a monitoring area with a camera and distance data indicating a two-dimensional distribution of distances in the monitoring area, the host vehicle is configured to execute Provided is a vehicle monitoring method for monitoring a behavior of another vehicle from a blinker state of another vehicle traveling in a lane adjacent to the traveling lane. In this vehicle monitoring method, as a first step, a lane in a monitoring area is recognized based on at least distance data. As a second step, based on the distance data, another vehicle traveling in the adjacent lane is specified from the recognized lanes, and the vehicle side surface of the specified other vehicle is recognized. As a third step, a blinker area is set on a two-dimensional plane defined by the color image or the distance data, based on the recognized end position of the side surface of the vehicle. As a fourth step, pixels of the color component of the blinker are detected based on the color image. As a fifth step, a detected pixel positionally corresponding to a region on the color image corresponding to the set blinker region is recognized as a blinker.
[0016]
Here, in the second invention, it is preferable that in the third step, the blinker region is set in a region corresponding to the side surface of the vehicle with reference to the end position. At this time, in the third step, the blinker region may be set also in a region corresponding to the rear surface of the vehicle based on the end position.
[0017]
BEST MODE FOR CARRYING OUT THE INVENTION
FIG. 1 is a schematic block diagram of a vehicle monitoring device with a driving support function according to the present embodiment. This vehicle monitoring device monitors the behavior of another vehicle based on the color image and the distance data from the blinker state of another vehicle traveling in a lane adjacent to the lane in which the own vehicle travels.
[0018]
A stereo camera (a color stereo camera in the present embodiment) that captures an image of the front of the host vehicle is mounted near a rearview mirror and includes a pair of cameras 1 and 2 having a built-in image sensor such as a CCD or a CMOS sensor. ing. The cameras 1 and 2 are mounted at a predetermined interval (camera base line length) in the vehicle width direction. The main camera 1 that outputs a reference image is mounted on the right side in the traveling direction of the host vehicle. On the other hand, the sub camera 2 that outputs the comparison image is mounted on the left side in the traveling direction.
[0019]
Each of the cameras 1 and 2 incorporates a separate image sensor for each of red, green and blue (for example, a three-plate color CCD). The color image (hereinafter, referred to as “reference image”) output from the main camera 1 is actually a red image (hereinafter, referred to as “R image”) and a green image (hereinafter, referred to as “G image”). And a blue image (hereinafter, referred to as a “B image”). Similarly, a color image (hereinafter, referred to as a “comparison image”) output from the sub camera 2 is also configured by three primary color images of an R image, a G image, and a B image. Therefore, a total of six images are output by the pair of cameras 1 and 2 at one shooting timing. Hereinafter, the R image constituting the reference image may be referred to as a reference image (R). Similarly, the G image forming the reference image may be referred to as a reference image (G), and the B image may be referred to as a reference image (B). As for the comparative image, the R, G, and B images may be referred to as a comparative image (R), a comparative image (G), and a comparative image (B), respectively.
[0020]
In a state where the pair of cameras 1 and 2 are synchronized, the analog image output from each of the cameras 1 and 2 is adjusted by the analog interface 3 so as to match the input range of the component at the subsequent stage. A gain control amplifier (GCA) 3a in the analog interface 3 adjusts the brightness balance of a pair of analog image signals. This adjustment is performed between images of the same color (R, G, B). The analog image pair adjusted in the analog interface 3 is converted by the A / D converter 4 into a digital image having a predetermined luminance gradation (for example, 256 gray scales).
[0021]
The image correction unit 5 performs luminance correction, image geometric conversion, and the like for each of the six primary color images that make up the digitized image pair. Usually, since the mounting positions of the pair of cameras 1 and 2 have an error to some extent, a shift caused by the error exists in the left and right images. Therefore, geometric transformation such as rotation or translation of the image is performed using affine transformation or the like. As a result, matching of horizontal lines (epipolar lines) in an image pair, which is a prerequisite for performing stereo matching, is guaranteed. Through such image processing, reference image data having 512 pixels (pixels) in the horizontal direction (i-coordinate direction) and 200 pixels (pixels) in the vertical direction (j-coordinate direction) is generated from the output signal of the main camera 1. You. Also, from the output signal of the sub-camera 2, comparative image data having the same vertical length as the reference image data and a larger horizontal length than the reference image data is generated (for example, 640 pixels (pixels) in the horizontal direction). , The vertical direction is 200 pixels). Here, an image plane defined by each of the images is represented by an ij coordinate system, with the lower left corner of the image as the origin, the horizontal direction as the i coordinate axis, and the vertical direction as the j coordinate axis. The reference image data corresponding to one frame and the comparison image data corrected by the image correction unit 5 are output to the subsequent stereo image processing unit 6 and stored in the image data memory 7.
[0022]
The stereo image processing unit 6 performs stereo matching based on the reference image data and the comparison image data, and calculates distance data for a captured image corresponding to one frame. Here, “distance data” is a set of parallaxes calculated for each small area in an image plane defined by image data, and each parallax is associated with a position (i, j) on the image plane. ing. In other words, the distance data is a two-dimensional distribution of the distance in the monitoring area. One parallax is calculated for each pixel block having a predetermined area which forms a part of the reference image. In the present embodiment in which three primary color images are output from each of the pair of cameras 1 and 2, this stereo matching is performed separately for the primary color image pairs of the same color. That is, R image (reference image (R) and comparative image (R)), G image ((reference image (G) and comparative image (G)), B image (reference image (B) and comparative image (B)) Is done independently for each.
[0023]
In the present embodiment, a pixel block of 4 × 4 pixels is used as a parallax calculation unit. Therefore, in the entire image corresponding to one frame, parallaxes of the number of pixel blocks included in the reference image, that is, 128 × 50 parallaxes can be calculated. As is well known, the parallax is the amount of displacement in the horizontal direction with respect to a pixel block, which is the unit of calculation, and has a large correlation with the distance to the object projected on the pixel block. That is, the parallax of this pixel block becomes larger as the object projected in the pixel block is closer to the cameras 1 and 2, and the parallax becomes smaller as the object is farther (if the object is infinitely far, the parallax becomes 0). ).
[0024]
When calculating parallax for a certain pixel block (correlation source), an area (correlation destination) having a correlation with the luminance characteristic of this pixel block is specified in the comparison image. As described above, the distance from the cameras 1 and 2 to the object appears as a horizontal shift amount between the reference image and the comparison image. Therefore, when searching for a correlation destination in the comparison image, it is only necessary to search on the same horizontal line (epipolar line) as the j-coordinate of the pixel block serving as the correlation source. The stereo image processing unit 6 determines the correlation between the correlation source and the correlation destination candidate while shifting the pixel on the epipolar line by one pixel within a predetermined search range set based on the i coordinate of the correlation source. Evaluate sequentially (stereo matching). Then, in principle, the horizontal shift amount of the correlation destination (one of the candidates of the correlation destination) determined to have the highest correlation is set as the parallax of the pixel block.
[0025]
The correlation of the luminance characteristics between the pixel blocks can be evaluated, for example, by calculating a city block distance. Basically, the pixel block having the minimum value is the pixel block having the correlation. Then, the horizontal shift amount between the pixel blocks having the correlation is output as the parallax. The hardware configuration for calculating the city block distance is disclosed in Japanese Patent Application Laid-Open No. H5-114099, and should be referred to if necessary.
[0026]
The stereo image processing unit 6 performs stereo matching between the primary color images of the same color (for example, the reference image (R) and the comparison image (R)), and outputs the result to the subsequent fusion processing unit 8. Thereby, for one pixel block in the reference image, three parallaxes, that is, the parallax Nr calculated from the R image, the parallax Ng calculated from the G image, and the parallax Nb calculated from the B image are calculated (hereinafter, referred to as “the parallax Nb”). Each is called "primary color parallax."
[0027]
The fusion processing unit 8 fuses the three primary color parallaxes Nr, Ng, and Nb calculated for a certain pixel block, and calculates a unified parallax Ni (hereinafter, referred to as “unified parallax Ni”) for the pixel block. This fusion is performed based on a product-sum operation (weighted average in the present embodiment) as shown in Expression 1. Specifically, the unified parallaxes Ni for the pixel block are calculated by adding weights to the primary color parallaxes Nr, Ng, Nb for a certain pixel block. Here, Or is a weight coefficient for red (R), Og is a green (G), and Ob is a weight coefficient for blue (B).
(Equation 1)
Ni = Or.Nr + Og.Ng + Ob.Nb
[0028]
The set of weight coefficients (Or, Og, Ob) is specified by the detection target selection unit 9. The detection target selection unit 9 selects an object to be detected at the present time (for example, an overtaking prohibited center line or the like). The detection target can appear when the vehicle is running, and a plurality of targets (for example, an overtaking prohibited center line, a stop lamp, a blinker, and the like) are set in advance. The detection target selection unit 9 selects any one of the targets as a detection target in accordance with a traveling situation specified by a speed sensor, a steering angle sensor, or the like of the own vehicle (not shown), and determines a weight coefficient of the detection target. The set (Or, Og, Ob) is output to the fusion processing unit 8.
[0029]
When the set of weight coefficients (Or, Og, Ob) is regarded as a weighted average weight coefficient, the specific coefficient values Or, Og, Ob need to satisfy the four conditions (expressions) shown in Expression 2. is there.
(Equation 2)
Or + Og + Ob = 1
0 ≦ Or ≦ 1
0 ≦ Og ≦ 1
0 ≦ Ob ≦ 1
[0030]
In order to improve the recognition accuracy of the detection target, it is preferable that the color of the target is reflected in the set of weight coefficients (Or, Og, Ob). Specifically, the respective weighting factors Or, Og, Ob are defined as in Equation 3. In the equation, Ir is a red (R) color component value of the object, Ig is a green (G) color component value, and Ib is a blue (B) color component value. Specific values of the weighting factors Or, Og, and Ob are determined by executing the calculation of Expression 3 based on the contents of a color information table 11a described later.
[Equation 3]
Or = Ir / (Ir + Ig + Ib)
Og = Ig / (Ir + Ig + Ib)
Ob = Ib / (Ir + Ig + Ib)
[0031]
The detection target selection unit 9 includes a microcomputer 10 that selects a detection target by executing a preset program, and a memory 11 in which programs and various data are stored. The memory 11 includes a ROM in which programs and data are fixedly stored, and a RAM used as a work area of the microcomputer 10 and the like. The information stored in the memory 11 includes, for example, specific values of the weighting factors Or, Og, Ob determined by the above-described arithmetic processing, and the color information table 11a.
[0032]
The set of weight coefficients (Or, Og, Ob) determined by the calculation shown in Expression 3 is stored and held in a predetermined address area (hereinafter, referred to as “weight setting area”) in the memory 11 (RAM). When the weight coefficient set (Or, Og, Ob) is updated, the value written in the weight setting area is also immediately updated. The fusion processing unit 8 obtains a set of weight coefficients (Or, Og, Ob) to be applied at the present time by accessing the weight setting area.
[0033]
FIG. 2 is a diagram illustrating an example of the color information table 11a. In the color information table 11a, a detection target 111 and color information 112 representing the color are described in association with each other. The color information 112 is a color component (Ir, Ig, Ib) expressed by 256 gradation luminance values. For example, an overtaking prohibited center line (yellow or orange in Japan) is set to Ir = 240, Ig = 190, and Ib = 0. The rear lamp and the stop lamp of the vehicle are set to Ir = 240, Ig = 0, and Ib = 0.
[0034]
The set of one frame of the unified parallax Ni obtained in this manner is associated with the position (i, j) on the image plane, and stored in the distance data memory 12 as distance data (i, j, Ni). Is stored. The details of the fusion processing unit 8 for calculating the unified parallax Ni are disclosed in Japanese Patent Application No. 2001-343801 already filed by the present applicant.
[0035]
The microcomputer 13 includes a CPU, a ROM, a RAM, an input / output interface, and the like. When the microcomputer 13 is functionally grasped, the microcomputer 13 includes a lane recognition unit 13a, a vehicle recognition unit 13b, and a blinker recognition unit 13c. The microcomputer 13 reads image data corresponding to one frame from the image data memory 7 and reads distance data corresponding to the read image data from the distance data memory 12 on the premise of executing these functions.
[0036]
The lane recognition unit 13a recognizes a lane in the monitoring area based on at least the read distance data. Then, the recognized lanes (more precisely, parameters defining these) are output to the vehicle recognition unit 13b. The vehicle recognizing unit 13b specifies another vehicle traveling in an adjacent lane from the recognized lanes based on the read distance data, and recognizes a vehicle side surface of the specified other vehicle. At this time, when the rear surface of the specified other vehicle can be recognized, the vehicle recognition unit 13b recognizes the other vehicle as a combination of the vehicle side surface and the vehicle rear surface. The vehicle recognizing unit 13b may specify another vehicle traveling in the adjacent lane using the read image data. The turn signal recognizing unit 13c uses the recognized end position of the side surface of the vehicle on an image (in the present embodiment, a two-dimensional image plane defined by image data) or a two-dimensional plane defined by distance data. , And detects the pixels of the color components of the blinker based on the read image data. Then, a pixel positionally corresponding to a region on the image plane corresponding to the set blinker region is recognized as a blinker.
[0037]
If the microcomputer 13 determines that an alarm is required based on these recognition results, the microcomputer 13 activates an alarm device 14 such as a monitor or a speaker to call the driver's attention. Further, by controlling the control device 15 as needed, vehicle control such as downshifting of an automatic transmission (AT), suppression of engine output, or operation of a brake is appropriately performed. As one of the features of the present embodiment, the microcomputer 13 performs a process shown in a flowchart described later to change the behavior of the other vehicle from the blinker state of the other vehicle traveling in the lane adjacent to the lane in which the own vehicle travels. Monitoring.
[0038]
FIG. 3 is a flowchart illustrating a vehicle monitoring procedure according to the present embodiment. This routine is called at a predetermined interval and executed by the microcomputer 13. First, in step 1, the microcomputer 13 reads one frame of image data from the image data memory 7 and reads distance data corresponding to the image data from the distance data memory 12.
[0039]
In step 2, the lane recognition unit 13a recognizes a lane in the monitoring area based on the read image data and distance data. Specifically, the lane recognition unit 13a recognizes a white line defining a lane drawn on the road by calculating a road model. This road model is expressed by a straight line formula in a horizontal direction and a vertical direction in a coordinate system of a real space. That is, "recognition of the lane" is to set the parameter of the straight line type to a value that matches the actual white line shape.
[0040]
Based on the knowledge that the white line has higher luminance than the road surface, the lane recognition unit 13a evaluates the luminance change in the width direction of the road and specifies the position of the white line in the monitoring area on the image plane. . The position (x, y, z) of this white line in the real space is known based on the position (i, j) on the image plane and the parallax Ni calculated for this position, that is, based on the distance data. Is calculated from the coordinate conversion formula. The coordinate system of the real space set based on the position of the own vehicle has an origin on the road surface just below the center of the stereo camera, the x-axis in the vehicle width direction, the y-axis in the vehicle height direction, and the vehicle length direction (distance direction). Is the z axis. At this time, the xz plane (y = 0) coincides with the road surface when the road is flat. The road model is represented by dividing the own lane on the road into a plurality of sections in the distance direction, approximating the left and right white lines in each section with a three-dimensional straight line, and connecting them in a polygonal line.
[0041]
In step 3, the vehicle recognizing unit 13b detects a three-dimensional object existing in the monitoring area based on the distance data, and specifies another vehicle in the monitoring area based on the detection result. Specifically, the vehicle recognizing unit 13b divides the read distance data at predetermined intervals (for example, four pixel intervals in the horizontal direction) on the ij plane as shown in FIG. , Into a plurality of grid-shaped (vertical strip) sections. Then, the following processing is performed on each of the divided sections as a processing target.
[0042]
First, arbitrary data (for example, data whose (i, j) position is closest to the origin) is selected from all data existing in the section to be processed. Here, “data” refers to the parallax Ni associated with the position (i, j). Then, for the selected data, a three-dimensional position (x, y, z) is calculated using a well-known coordinate conversion formula. Next, the height of the road surface at the calculated distance z is calculated using the above-described straight line formula of the road model. By comparing the height of the road surface with the height y of the coordinate-converted data, if the data is located above the road surface, the data is extracted as three-dimensional object data. Then, the same processing is repeatedly executed for all data in this section, and three-dimensional object data existing in the section is sequentially extracted.
[0043]
When the extraction processing of the three-dimensional object data is completed, a histogram is calculated next with the extracted three-dimensional object data as a processing target. This histogram is represented as, for example, the appearance frequency of the three-dimensional object data with the distance z on the horizontal axis by counting the number of three-dimensional object data included in a section of a preset distance (for example, 15 m). When this histogram is created, a section in which the appearance frequency is equal to or higher than a predetermined threshold value and is the mode distance is detected. At this time, if there is a section that satisfies this condition, the vehicle recognition unit 13b determines that a three-dimensional object exists in that section. Then, an average value or a representative value of the distance z corresponding to the number of data existing in the section or an intermediate value of the section is calculated as the distance Z to the three-dimensional object.
[0044]
Then, the above-described processing is repeatedly executed for all the sections relating to the distance data, and the distance Z corresponding to the divided sections is calculated.
[0045]
Next, based on the calculated distances Z corresponding to the number of sections, the sections whose distances Z are approximate are grouped into groups. In this process, the distance Z between the three-dimensional objects in each section is checked, and when the difference in the distance Z to the three-dimensional object in the adjacent section is equal to or smaller than the determination value, the three-dimensional object is regarded as the same three-dimensional object, and the adjacent sections belong to the same group. Summarized. On the other hand, if the difference between the distances Z exceeds the determination value, the distances Z regarding these sections are regarded as indicating the distances of different three-dimensional objects, and the adjacent sections are classified as different groups. Then, each of the grouped groups is classified into “side wall” or “object”, and parameters of each group are calculated.
[0046]
In this processing, the slope of the entire group is calculated by obtaining an approximate straight line by the Hough transform or the least square method from the position (x, Z) of each section in the group. Then, the inclination of this straight line is compared with a set value (for example, 45 °). When the calculated inclination of the straight line is equal to or less than the set value and the arrangement of the distances Z in the group is substantially in the z-axis direction, the group is labeled as “side wall”. On the other hand, when the calculated inclination of the straight line exceeds the set value and the arrangement of the distances Z in the group is substantially in the x-axis direction, the group is labeled as “object”. Then, in the group labeled “object”, the average distance calculated from the distance Z in the group, the x coordinate of the left end and the right end, and the like are calculated as parameters of the group. In the group labeled “side wall”, the direction of the distance Z in the group (inclination with the z-axis), the position of the front and rear ends (z, x coordinates), and the like are calculated as parameters of the group.
[0047]
Then, among the labeled groups, the groups that are individually labeled as “object” and “sidewall” despite being the same three-dimensional object are a combination of “object” and “sidewall”. Will be recognized as This is because other vehicles traveling in the adjacent lane may be grouped as "side wall" on the side of the vehicle and "object" on the rear side of the vehicle, and each of these groups is recognized as an independent three-dimensional object. Because there is fear. Therefore, in this processing, the difference between the position of the end point of the group labeled "object" and the position of the end point of the group labeled "side wall" is calculated. When the “object” is on the right side of the front of the host vehicle (corresponding to the z-axis), the positions of the end points are the positions of the left end of the “object” and the positions of the end points on the near side of the “side wall”. On the other hand, when the “object” is on the left side of the front of the host vehicle, the positions of the end points are the right end position of the “object” and the end point on the near side of the “side wall”. Then, it is checked whether or not the positions of the end points of each group are close to each other within a determination value (for example, about 1 m). At this time, if the difference between the positions of the end points of each group is within the determination value, it is determined that these groups are the same three-dimensional object. Because, when the rear part and the side of one solid object are visible at the same time, the corner formed by the two surfaces is convex toward the host vehicle, so the position of the left end of the "object" and the position just before the "side wall" This is because the position of the end point on the side (or the position of the right end of the “object” and the position of the end point on the near side of the “side wall”) are originally considered to be the same. Therefore, when the difference between the positions of the two groups is within the determination value, the two groups are determined to be one three-dimensional object.
[0048]
Through the above processing, each of the three-dimensional objects in the monitoring area is recognized as a group, and the group of three-dimensional objects existing on the adjacent lane and the own lane is specified as another vehicle. Specifically, the vehicle recognition unit 13b sets the group labeled “combination of“ object ”and“ sidewall ”or the group labeled“ sidewall ”as another vehicle existing in the adjacent lane. Identify. On the other hand, the vehicle recognition unit 13b basically specifies the group labeled “object” as another vehicle existing in the same lane as the own vehicle.
[0049]
FIG. 5 is an explanatory diagram illustrating an example of a result of specifying another vehicle. As shown in the figure, since the other vehicles in the monitoring area are specified as the groups labeled "object" and "side wall", the vehicle recognizing unit 13b determines the group labeled "side wall". By recognizing, the side of the vehicle can be recognized. Further, the vehicle recognizing unit 13b can recognize the rear surface of the vehicle by recognizing the group labeled "object". A detailed method for detecting a three-dimensional object as an “object” or “side wall” is disclosed in Japanese Patent Application Laid-Open No. 10-283461, and should be referred to if necessary.
[0050]
Then, in step 4, it is determined whether or not another vehicle exists in the adjacent lane based on the above-described identification result of the other vehicle. If an affirmative determination is made in step 4, the process proceeds to step 5. On the other hand, if a negative determination is made in step 4, the process proceeds to the subsequent step 8.
[0051]
In step 5 following step 4, the blinker detecting unit 13c recognizes the blinker of another vehicle existing in the specified adjacent lane and detects the blinking state of the blinker. FIG. 6 is a detailed flowchart showing the blinker blink detection routine. First, in step 51, a blinker region is set in a region where a blinker of another vehicle will exist on the image plane defined by the distance data.
[0052]
FIG. 7 is an explanatory diagram showing a blinker area set by the blinker recognition unit 13c. Specifically, assuming that a blinker area is set for another vehicle existing in the adjacent lane, a reference is made on the distance data based on a boundary position between the vehicle side surface and the vehicle rear surface of the other vehicle detected by the vehicle detection unit 13b. The position Ps (is, js) is determined. In the present embodiment, the reference position Ps (is, js) is determined at a point on the boundary position closest to the road. Then, based on the reference position Ps (is, js), at a position at a certain height above the road surface (that is, a position shifted in the j direction from the reference position Ps (is, js)), A first blinker region Awin1 having a predetermined area is set. This height is determined based on a position corresponding to a blinker provided on the vehicle, and as a result, the first blinker area Awin1 corresponds to a blinker position on the rear surface of the vehicle of another vehicle. Further, a second blinker area Awin2 (the second blinker area Awin2 has a predetermined width) is set on the side surface of the vehicle and extends in the vehicle length direction along the side surface of the vehicle. That is, the second blinker area Awin2 corresponds to the position of the blinker provided on the side of the vehicle of the other vehicle.
[0053]
When an image of an adjacent other vehicle is taken from the own vehicle, the other vehicle existing slightly ahead of the position of the own vehicle may not be able to detect only the side of the vehicle. Therefore, the reference position for setting the blinker area may be based on the end of the vehicle side surface (more precisely, the end on the host vehicle side). That is, it can be said that the above-described embodiment is a case where the end position of the vehicle side surface coincides with the position of the vehicle rear surface. For example, as shown in FIG. 7, in the other vehicle (the right vehicle in the figure) recognized only from the side of the vehicle, only the second blinker area Awin2 is set based on the end position of the side of the vehicle. It becomes. Further, in the process of step 51 described above, the blinker area is set on the distance data, but an area corresponding to this blinker area may be set as a blinker area at a corresponding position on the image data.
[0054]
Then, in step 52, the color component pixels of the blinker are extracted based on the image data. Here, the blinker color component is generally orange to yellow, and in this step 52, orange to yellow pixels are extracted. Specifically, in each of the three primary color images, a pixel G corresponding to the position is processed, and a ratio G / R of the luminance value between the G image and the R image and a luminance value ratio between the G image and the B image are processed. The ratio G / B is required. Pixels in which the obtained G / R and G / B are within a preset setting range (that is, a G / R range and a G / B range of orange or yellow) are determined by the blinker color. It is extracted as a component pixel.
[0055]
In step 53, among the extracted blinker color component pixels, pixels that are continuous in coordinate position are grouped as a group Gn (grouping). Then, the group area Wgn and the barycentric position Pgn (ign, jgn) on the image plane are calculated for each group Gn to be processed (step 54).
[0056]
Next, in step 55, the position of the center of gravity Pgn (ign, jgn) of the group Gn is set as a position on the image plane corresponding to the first and second blinker regions Awin1 and Awin2 set in step 51. It is determined whether or not they correspond. If a negative determination is made in step 55, it is determined that the group Gn is not a blinker, and the process proceeds to step 8 shown in FIG. On the other hand, when an affirmative determination is made in step 55, it is determined that the possibility that the group Gn is a blinker is high, and the process proceeds to step 56.
[0057]
In a step 56 following the step 55, the area Wgn of the group Gn, whose center of gravity position Pgn (ign, jgn) of the group Gn is determined to correspond to the first and second blinker areas Awin1, Awin2, is predetermined. It is determined whether or not the value is equal to or more than the value Wth. Here, the predetermined set value Wth is determined as a value at which the area Wgn of the group Gn is recognized to be equivalent to the size of the blinker, and the distance data of the first and second blinker areas Awin1 and Awin2 is determined. The value differs depending on the upper position. That is, when the position in the real space corresponding to the set positions of the first and second blinker areas Awin1 and Awin2 is far from the host vehicle, the predetermined determination value Wth is set to a certain small value in consideration of the distance. It becomes. On the other hand, when the position in the real space corresponding to the set positions of the first and second blinker areas Awin1 and Awin2 is close to the host vehicle, the predetermined determination value Wth is a certain large value in consideration of the distance. It has become.
[0058]
If a negative determination is made in step 56, it is determined that the group Gn does not have an area large enough to be recognized as a blinker, and the process proceeds to step 8 shown in FIG. On the other hand, if an affirmative determination is made in step 56, the group Gn is determined as a blinker, the area Wgn of the group Gn is stored in, for example, a RAM area, and the process proceeds to step 57.
[0059]
In step 57 following step 56, the average value Wa of the group areas Wgn detected in the past M times among the stored areas Wgn of the group Gn is calculated. The data of the past M times is data corresponding to the number of times that can determine whether or not the blinker is blinking. For example, if the time for determining the blinking state of the blinker is 2 seconds, and if the execution interval of this routine is 0.01 second, the number M of data to be processed is 200 times. If the current number of detections is M or less, this average value Wa is calculated as the average value of the group area Wgn detected so far.
[0060]
In step 58, a blinking judgment value B is calculated. The blink determination value B is the sum of the difference (absolute value) between the detected group area Wgn and the average value Wa of the group area Wgn in the past M times in the past M times (Σ | Wgn−Wa |). The blink determination value B tends to increase as the number of blinkers of the blinker increases, and to decrease as the number of blinks of the blinker decreases. Then, with the calculation of the blinking determination value B, the process exits this routine.
[0061]
Referring again to FIG. 3, in step 6, the blinking determination value B calculated in the blinker blink detection routine is compared with the determination value Bth. Here, the determination value Bth to be compared with the blinking determination value B is determined as the minimum value of the blinking determination value B at which the blinker is recognized as blinking. Therefore, based on the comparison result, when the blinking determination value B is equal to or greater than the determination value Bth, it is determined that the blinker of another vehicle existing in the adjacent lane is blinking, and the process proceeds to step 7. On the other hand, if the blinking determination value B is smaller than the determination value Bth based on the comparison result, it is determined that the blinker of another vehicle existing in the adjacent lane is not blinking, and the process proceeds to step S8.
[0062]
In step 7 following step 6, the blinker of the other vehicle traveling in the adjacent lane is determined to blink, and the blinker recognizing unit 13c determines that there is a vehicle joining the own lane. At this time, the microcomputer 13 may operate the alarm device 14 to warn the driver of the presence of the merging vehicle. Further, when there is a merging vehicle in the own lane, the target other vehicle serving as a control reference of the control device 15 is switched to this merging vehicle, and the AT (automatic transmission) downshift and engine output are performed as necessary. Vehicle control such as suppression of brakes or operation of a brake.
[0063]
On the other hand, in step 8, the blinker recognizing unit 13c receives the determination that the blinker of the other vehicle traveling in the adjacent lane does not blink, and determines that there is no vehicle joining the own lane. In this case, the microcomputer 13 performs vehicle control such as collision prevention and lane departure prevention based on the inter-vehicle distance and the relative speed with the preceding vehicle (or obstacle) in order to perform normal vehicle monitoring.
[0064]
As described above, in the vehicle monitoring method according to the present embodiment, first, another vehicle traveling on an adjacent lane in the monitoring area is recognized, and the other vehicle is recognized as a vehicle side surface and a vehicle rear surface. Then, a blinker area is set based on the recognized end position of the side surface of the vehicle. Thus, even when the blinker position does not exist at both ends (for example, another vehicle traveling in the adjacent lane viewed from the own vehicle), the blinker position is determined based on the end of the side surface of the vehicle of the other vehicle. Since the area is set on the blinker position of another vehicle, the blinker can be recognized effectively. Also, even in a case where the rear surface of the vehicle of another vehicle does not fit within the angle of view of the camera, the blinker area is located above the blinker position on the side of the vehicle by using the end of the vehicle side surface of the other vehicle as a reference. Since the setting is made, the blinker can be recognized effectively. By recognizing the blinkers in this manner, the behavior of another vehicle can be monitored from the recognized blinkers. For example, by judging the blinking state, it can be grasped whether or not another vehicle existing in the adjacent lane is a vehicle joining the own lane. Thus, the driver can be alerted to the presence of the merged vehicle, or the vehicle can be controlled by focusing on the merged vehicle, so that the safety and reliability of vehicle monitoring can be improved.
[0065]
In the present embodiment, the distance data is calculated using stereo matching. However, the distance data may be output by using various sensors such as a laser radar or a millimeter wave radar. That is, in the vehicle monitoring device shown in FIG. 1, the stereo image processing unit 6 that calculates distance data based on the images captured by the cameras 1 and 2 has the same function as these sensors.
[0066]
In the above-described embodiment, the distance data is calculated and the blinker of the blinker is detected by using the color images output from the cameras 1 and 2. However, the present invention is limited to such a form. Not done. FIG. 8 is a schematic block diagram of another vehicle monitoring device with a driving support function according to the present embodiment. The vehicle monitoring device shown in the figure differs from the above-described embodiment in that the color stereo cameras (cameras 1 and 2) shown in FIG. Camera 18). Further, the vehicle monitoring device shown in FIG. 8 does not have the fusion processing unit 8 shown in FIG. 1 and the detection target selection unit 9 as constituent elements due to the use of the monochrome stereo camera. That is, in the vehicle monitoring device shown in FIG. 8, distance data is calculated based on the output image from the monochrome stereo camera, and lanes and vehicles in the monitoring area are recognized based on the distance data. In addition, blinking of the blinker is detected based on an output image from the color camera (camera 18) together with the recognition results. Even with such a configuration, the same operation and effect as the above-described embodiment can be achieved. Also, such a vehicle monitoring device has an advantage that the calculation process can be simplified as compared with the vehicle monitoring device shown in FIG. 1 because a color image is not used when performing stereo matching.
[0067]
【The invention's effect】
As described above, according to the method of the present invention, the blinker region is set on the blinker position of the other vehicle based on the end of the side surface of the other vehicle, so that the blinker can be effectively recognized. can do. Thus, since the blinker of another vehicle traveling in the adjacent lane can be recognized, the behavior of the other vehicle can be monitored effectively.
[Brief description of the drawings]
FIG. 1 is a schematic block diagram of a vehicle monitoring device with a driving support function according to an embodiment;
FIG. 2 shows an example of a color information table.
FIG. 3 is a flowchart showing a vehicle monitoring procedure according to the embodiment;
FIG. 4 is an explanatory diagram showing distance data divided into a plurality of sections;
FIG. 5 is an explanatory diagram showing an example of a vehicle identification result.
FIG. 6 is a detailed flowchart showing a blinker blink detection routine;
FIG. 7 is an explanatory diagram showing a blinker area set by a blinker recognition unit;
FIG. 8 is a schematic block diagram of another vehicle monitoring device with a driving support function according to the embodiment;
[Explanation of symbols]
1 Main camera
2 Sub camera
3 Analog interface
3a Gain control amplifier
4 A / D converter
5 Image correction unit
6. Stereo image processing unit
7 Image data memory
8 Fusion processing unit
9 Detection target selection section
10. Microcomputer
11 Memory
11a Color information table
12 Distance data memory
13 Microcomputer
13a Lane recognition unit
13b Vehicle recognition unit
13c blinker recognition unit
14 Alarm device
15 Control device

Claims (10)

In a vehicle monitoring device that monitors the behavior of another vehicle from the blinker state of another vehicle traveling in an adjacent lane adjacent to the lane in which the own vehicle travels,
A first camera that outputs a color image by imaging a scene including a monitoring area;
A sensor that outputs distance data indicating a two-dimensional distribution of distances in the monitoring area,
A lane recognition unit that recognizes a lane in the monitoring area based on at least the distance data;
Based on the distance data, while identifying another vehicle traveling in the adjacent lane from the recognized lane, a vehicle recognition unit that recognizes the side surface of the identified other vehicle,
On a two-dimensional plane defined by the color image or the distance data, a blinker area is set based on the recognized end position of the side surface of the vehicle, and a pixel of a color component of the blinker is detected based on the color image. And a blinker recognizing unit for recognizing, as a blinker, the detected pixel positionally corresponding to a region on the color image corresponding to the set blinker region. The vehicle monitoring device according to claim 1, wherein the blinker recognition unit sets a blinker region in a region corresponding to the side surface of the vehicle based on the end position. The vehicle monitoring device according to claim 2, wherein the blinker recognition unit sets a blinker region also in a region corresponding to a rear surface of the vehicle based on the end position. The said blinker recognition part determines whether the blinker of said other vehicle is blinking based on the time-sequential area change of the pixel recognized as said blinker. A vehicle monitoring device according to any one of claims 1 to 3. The blinker recognizing unit sets the blinker area based on a boundary position between the vehicle side surface and the vehicle rear surface when the vehicle recognizing unit further recognizes the rear surface of the identified other vehicle. The vehicle monitoring device according to any one of claims 1 to 4, wherein: A second camera that functions as a stereo camera by cooperating with the first camera, by outputting a color image by imaging a scene including the monitoring area,
2. The sensor according to claim 1, wherein the sensor outputs the distance data by stereo matching based on a color image output from the first camera and a color image output from the second camera. 6. The vehicle monitoring device according to any one of claims 1 to 5, By capturing a scene including the monitoring area, further comprising a monochrome stereo camera that outputs a pair of image data,
The vehicle monitoring device according to claim 1, wherein the sensor outputs the distance data by stereo matching based on a pair of image data output from the monochrome stereo camera. Based on a color image obtained by capturing a scene including a monitoring area with a camera and distance data indicating a two-dimensional distribution of distances in the monitoring area, a lane adjacent to the lane in which the host vehicle travels. A vehicle monitoring method for monitoring the behavior of the other vehicle from the blinker state of the other vehicle traveling on the vehicle,
A first step of recognizing a lane in the monitoring area;
A second step of identifying another vehicle traveling in an adjacent lane from the recognized lane, and recognizing a vehicle side surface of the identified other vehicle;
On a two-dimensional plane defined by the color image or the distance data, a third step of setting a blinker area based on the recognized end position of the side surface of the vehicle,
A fourth step of detecting a blinker color component pixel based on the color image;
A fifth step of recognizing, as a blinker, the detected pixel that physically corresponds to a region on the color image corresponding to the set blinker region. 9. The vehicle monitoring method according to claim 8, wherein the third step sets a blinker area in an area corresponding to the side surface of the vehicle with reference to the end position. 10. The vehicle monitoring method according to claim 9, wherein the third step sets a blinker area also in an area corresponding to a rear surface of the vehicle based on the end position.

Images