JP2010224930A - Road recognition device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a road recognition device detecting a road surface region of a road surface from within a taken image with precision and in real time. <P>SOLUTION: The road recognition device 1 includes: an integration means 6 for integrating one element pi, j in an image T and an adjacent pixel p into one group g based on a difference ΔD of the brightness Di, j of the one pixel pi, j to the brightness D of the pixel p adjacent to the one pixel pi, j and a difference δD of the brightness Di, j to the average value Dave of the brightness D of the group g to which the adjacent pixel p belongs; and a road surface region detection means 9 which detects the group g as a group G belonging to the road surface region of the road surface if a pixel row j exists where breadth Wj per pixel row j of the group g is equal to or greater than a breadth threshold Wth, the average value Nave of the breadth Nj is equal to or greater than an average value threshold Nave_th, the average value Dave of the brightness of the group g is equal to or less than a brightness average threshold Dave_th, and an area A within the region R of the group g is equal to or greater than an area threshold Ath. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a road recognition device, and more particularly to a road recognition device that detects a road surface area of a road surface from a captured image.

  Conventionally, various techniques for detecting a lane or the like marked on a road surface from an image taken in front of a vehicle using, for example, a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera have been developed ( For example, see Patent Documents 1 and 2). In the present specification, a continuous line or a broken line marked on a road surface such as an overtaking prohibition line or a lane marking that divides a roadside zone and a roadway is referred to as a lane.

  Also, for example, in the vehicle environment recognition device described in Patent Document 3, an area surrounded by a lane that exists on the side of the host vehicle detected in the image at the previous sampling period on the image captured at the current sampling period. Set a window with a range larger than the predetermined amount, recognize the reference color near the center of the lower end of the window where the probability that the road surface just before the vehicle is imaged is high, and extract the same color as the reference color for the entire window Then, a technique for recognizing the region as a road surface region of a road surface and recognizing a lane has been proposed.

Japanese Patent No. 3352655 JP 2006-331389 A Japanese Patent No. 3120648

  However, there are roads where the lane is not marked on either the left or right side of the host vehicle. In such a case, the technique disclosed in Patent Document 3 uses the left and right sides of the host vehicle detected in the image as described above. Since the window is set based on the lane, the window cannot be set, and it may be difficult to detect the road surface area of the road surface.

  In addition, if the pixel color near the center of the bottom edge of the window is used as the reference color for extracting the road surface area of the road surface, a pedestrian crossing or a stop marked on the road surface is located near the center of the bottom edge of the window. When a white display such as a line or a number, a character, or an arrow on the road surface is captured, the road surface region is extracted using the white color as a reference color, and it is difficult to correctly extract the road surface region. There is a risk of becoming.

  Furthermore, even if a window is set for an image, the luminance of each pixel in the window once stored in the memory is read, and whether or not the color is the same color as the reference color for all the pixels of the window In other words, the process for determining is a very heavy process. Therefore, for example, when the luminance of the image for several frames or several tens of frames is captured per second and processing must be performed one after another, the processing may not be in time, and the real-time property of road surface area detection may be impaired. there were.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a road recognition device capable of accurately detecting a road surface area of a road surface from a captured image in real time.

In order to solve the above problem, the first invention provides:
A road recognition device that processes an image captured by an imaging unit and detects a road surface area of a road surface from the image,
The one pixel and the adjacent pixel based on the difference between the luminance of the one pixel relative to the luminance of the pixel adjacent to the one pixel in the image and the average value of the luminance of the group to which the adjacent pixel belongs An integration means to integrate
For the group integrated by the integration unit, a calculation unit that calculates an average value of the horizontal width of each pixel row in the horizontal direction of the image and the average value of the luminance of the pixels belonging to the group;
An area calculating means for calculating an area occupied by the group in a region set in the traveling direction of the host vehicle in the image;
There is a pixel row that is equal to or greater than a predetermined width threshold in the pixel row of the group, an average value of the width for each pixel row of the group is equal to or greater than a predetermined average value threshold, and each pixel belonging to the group If the average luminance value is equal to or smaller than a predetermined luminance average threshold value and the area of the group calculated by the area calculating means is equal to or larger than a predetermined area threshold value, the group belongs to the road surface area of the road surface. Road surface area detecting means for detecting as a group;
It is characterized by providing.

  According to a second aspect of the present invention, in the road recognition device according to the first aspect, the luminance average threshold value is variable according to a shutter level of the imaging means.

3rd invention is the road recognition apparatus of 1st or 2nd invention,
Lane detection means for detecting a lane present on the side of the host vehicle from the image,
The lane detecting means calculates an average value of luminance of pixels in which the detected lane is imaged,
The luminance average threshold value may be varied according to an average luminance value of pixels in which the lane is imaged.

A fourth invention is the road recognition device according to any one of the first to third inventions,
Lane detection means for detecting a lane present on the side of the host vehicle from the image,
The region set in the traveling direction of the host vehicle in the image is a lane estimated in the current sampling cycle based on the lane information detected by the lane detection unit in the past sampling cycle and the behavior of the host vehicle. It is characterized in that it is set based on the position of.

  According to a fifth aspect of the present invention, in the road recognition device according to any one of the first to third aspects, the region set in the traveling direction of the host vehicle in the image is lateral to the estimated trajectory estimated from the behavior of the host vehicle. A range up to a position separated by a predetermined distance in the direction is set.

A sixth invention is the road recognition device according to any one of the first to fifth inventions,
When the integration unit integrates the one pixel and the adjacent pixel into one group, the horizontal width in the real space in the pixel row of the group is equal to or greater than the horizontal width threshold. In addition, the information that the condition is satisfied is associated with the group,
The road surface area detecting means determines whether or not there is a pixel row in which the horizontal width of each pixel row of the group is equal to or larger than a predetermined horizontal width threshold, and whether or not the information is associated with the group. It is characterized by being judged.

A seventh invention is the road recognition device according to any one of the first to sixth inventions,
A distance detecting means for detecting a distance in real space of each pixel in the image;
The integrating means includes a height from the road surface of a point in the real space corresponding to the one pixel calculated based on a distance in the real space of the one pixel and an actual pixel corresponding to the adjacent pixel. Among the heights of the points on the space from the road surface, when one is not less than a predetermined first height threshold and the other is less than the first height threshold, the one pixel and the adjacent It is characterized in that non-integrated pixels are to be integrated.

An eighth invention is the road recognition device according to any one of the first to seventh inventions,
A distance detecting means for detecting a distance in real space of the pixel in the image;
When the height of the group in the real space from the road surface of the group is greater than or equal to a predetermined second height threshold based on the distance in the real space of the pixels belonging to the group The group is excluded from the detection target of the road surface area of the road surface.

  According to the first invention, after integrating each pixel of the image into each group based on the luminance by the integrating means, the horizontal width for each pixel row of the group and its average value, the average value of the luminance of each pixel belonging to the group, Based on the area that the group occupies in the region of the image, it is determined whether or not the group belongs to the road surface region of the road surface. In other words, since the determination is based on the size and shape of the group in the image, the average luminance, and the position occupied in the image, it is determined accurately whether or not the group belongs to the road surface area of the road surface. Thus, it is possible to accurately detect the road surface area of the road surface from the image.

  In particular, as in the vehicle environment recognition device described in Patent Document 3 described above, each pixel belonging to the group is not subjected to processing such as setting the color of the pixel near the center of the lower end of the window set in the image as a reference color. Since it is determined whether or not the group belongs to the road surface area of the road surface based on the average value of the luminance of the pixel, it is possible to accurately prevent the problem of erroneous extraction and erroneous determination, and to prevent the road surface of the road surface. It is possible to accurately detect the region.

  In addition, since each process by the calculation means and the area calculation means is performed in parallel with the integration process of each group by the integration means, and after the integration process is completed, a determination process by the road surface area detection means is performed. The road surface area of the road surface can be detected very quickly. Therefore, for example, even when the luminance of an image for several frames or several tens of frames is captured per second and processing must be performed one after another, the road surface area of the road surface is accurately determined from the image for each frame. It becomes possible to detect and process in real time.

  Furthermore, according to the first invention, since the road surface area of the road surface is detected from the image as each group obtained by integrating the pixels of the image based on the luminance by the integrating means, for example, the road on which the host vehicle is traveling Even on a road where no lane is marked on either the left or right side of the vehicle, the road surface area of the road surface can be accurately detected from the image. Further, since the groups detected as the groups belonging to the road surface area are connected and the road surface area of the road surface is extracted and detected in the image, as shown in FIG. It is possible to accurately detect the road B that merges with the existing road and the distant area C of the road on which the vehicle is traveling, and it is possible to accurately detect the road surface area of the road surface captured in the image. It becomes.

  According to the second invention, each marking that is marked white on the road surface such as a lane, a pedestrian crossing, a stop line, a number, a letter, an arrow, and the like and an imaging target that is darkly imaged like a road surface area on the road surface Thus, it is possible to accurately select based on the luminance average threshold value that is varied according to the shutter level of the imaging means. Therefore, it is possible to accurately detect a group with low luminance as a group belonging to the road surface area of the road surface, and it is possible to exhibit the effect of the invention more accurately.

  According to the third aspect of the invention, the brightness average threshold is varied according to the average value of the brightness of the pixels in which the lane is imaged, and is marked in white on the road surface such as a lane, a pedestrian crossing, a stop line, a number, a character, or an arrow. It is possible to appropriately exclude the group corresponding to each sign, and to detect a group with low brightness as a group belonging to the road surface area of the road surface, and to exhibit the effect of each invention more accurately. It becomes possible.

  According to the fourth invention, based on the lane information detected by the lane detection means at the side of the host vehicle in the past sampling cycle such as the previous sampling cycle and the behavior of the host vehicle, By estimating the position of the lane and setting the area based on the estimated position, the area can be set with high accuracy, and the effects of the respective inventions can be exhibited accurately.

  According to the fifth aspect, the area can be set based on, for example, an estimated trajectory estimated from the behavior of the host vehicle. Therefore, even when the lane is not marked on the road on which the vehicle is traveling, the road area of the road surface is accurately detected from the image by effectively setting the area without depending on the lane. Thus, the effects of the above-described inventions can be exhibited accurately.

  According to the sixth aspect of the invention, by setting the width threshold value as a width in the real space, for example, 60 cm or the like, the group corresponding to the marking such as each white line block of the pedestrian crossing marked on the road surface is displayed on the road surface. It can be configured not to be detected as a group belonging to the road surface area, and the calculation unit determines whether the width in the real space for each image row of the group is equal to or greater than the width threshold by the integration unit By configuring so as to be performed at the same time, it is possible to speed up the processing, and it is possible to accurately exhibit the effects of the above-described inventions.

  According to the seventh aspect, it is possible to integrate the imaging target existing on the road surface including the road surface area and the imaging target existing above the road surface such as a three-dimensional object into different groups. Therefore, it is possible to accurately prevent a three-dimensional object or the like from being erroneously detected as a road surface area of the road surface, and it is possible to exhibit the effects of the above-described inventions more accurately.

  According to the eighth invention, the imaging target existing above the road surface such as a three-dimensional object is excluded, and only the imaging target existing on the road surface including the road surface region is set as the detection target of the road surface region of the road surface. Is possible. Therefore, it is possible to accurately prevent a three-dimensional object or the like from being erroneously detected as a road surface area of the road surface, and it is possible to exhibit the effects of the above-described inventions more accurately.

It is a block diagram which shows the structure of the road recognition apparatus which concerns on 1st Embodiment. It is a figure which shows the example of the image imaged with an imaging means. It is a figure which shows the example of the stop line marked continuously to the lane imaged in the image. It is a figure explaining the area | region set based on an estimated locus | trajectory. It is a photograph which shows the example of the image imaged with an imaging means, and the example of the area | region set in the image. It is a flowchart which shows the process sequence in an integration means etc. It is a flowchart which shows the process sequence in an integration means etc. It is a flowchart which shows the process sequence in a calculation means etc. It is a flowchart which shows the process sequence in a road surface area detection means etc. It is a figure explaining the input 1 pixel and the pixel adjacent to the left already input. It is a figure explaining the example of the group to which one pixel and the pixel adjacent on the left belong. It is a figure explaining the inputted 1 pixel and the pixel adjacent below already inputted. It is a figure explaining the example of the group to which one pixel and the adjacent pixel below belong. (A) It is a figure explaining the example of the group which two pixels adjoin to the left and the bottom of one pixel, (B) Since one pixel was grouped with the pixel adjacent to the left, It is a figure explaining the example grouped. (A) It is a figure explaining the example of the group to which the adjacent pixel below belongs, and the group to which the pixel adjacent to the left belongs, (B) The example in which two groups are grouped with one pixel and become one group FIG. 6 is a photograph showing a plurality of groups integrated based on the image of FIG. 5. It is a photograph showing the road surface area of the road surface extracted from the image of FIG. It is a figure showing the distant area | region of the road and the traveling path of the own vehicle which merge with the traveling path of the own vehicle. It is a block diagram which shows the structure of the road recognition apparatus which concerns on 2nd Embodiment. It is a figure explaining the stereo matching process by the image processor of a distance detection means. It is a figure which shows the example of a distance image. It is a figure explaining the example of the lane candidate point detected by searching on the horizontal line of a reference image. It is a figure explaining the brightness | luminance of the pixel corresponding to a lane. It is a figure explaining the example of the lane detected to the side of the own vehicle. It is a figure explaining the example of the formed lane model, (A) represents the horizontal shape model on a ZX plane, (B) represents the road height model on a ZY plane.

  Hereinafter, an embodiment of a road recognition device according to the present invention will be described with reference to the drawings.

[First Embodiment]
As shown in FIG. 1, the road recognition device 1 according to the first embodiment mainly includes an imaging unit 2 and a processing unit 5. In the present embodiment, the imaging unit 2 is a monocular imaging unit such as a camera with a built-in image sensor such as a CCD or CMOS sensor. However, a single or a plurality of imaging units 2 are provided. It is also possible to configure so that the processing according to the present invention is performed on one or a plurality of images picked up by the image pickup means.

  Further, in the present embodiment, for example, when an image T as shown in FIG. 2 is captured, the imaging unit 2 scans in the right direction sequentially from the leftmost imaging element of each horizontal line j of the image T. The horizontal line j to be picked up is picked up while switching upward from the lowermost line in order, and each data of the luminance D is sequentially transmitted to the conversion means 3 in the order picked up by each image pickup device.

  The conversion means 3 is composed of an A / D converter, and when the data of the luminance D imaged for each imaging element (each pixel p) by the imaging means 2 is sequentially transmitted, the luminance of each pixel p. The D data is converted into luminance D data of a gray scale digital value such as 256 gradations and output to the image correction unit 4. The image correction unit 4 sequentially performs image correction such as displacement and noise removal, luminance correction, and the like on the transmitted data of the luminance D, and sequentially transmits the image corrected luminance D data to the processing unit 5. It is supposed to be.

  In this embodiment, the processing unit 5 is configured by a computer in which a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), an input / output interface, and the like (not shown) are connected to a bus. The processing unit 5 includes an integration unit 6, a calculation unit 7, an area calculation unit 8, and a road surface area detection unit 9.

  Sensors Q such as a vehicle speed sensor, a yaw rate sensor, and a steering angle sensor for measuring the steering angle of the steering wheel are connected to the processing unit 5, and each measurement value is input from them. Note that the processing unit 5 may be configured to perform other processing.

  In the following description, for example, with respect to the pixels in the image T shown in FIG. 2, the coordinates of the pixels (i, i, i) and (i, j) is used to represent a pixel pi, j. Further, the luminance D of the pixel pi, j is expressed as Di, j.

  In the present embodiment, when the processing unit 5 sequentially inputs the data of the luminance D of each pixel p of the image T captured by the imaging unit 2 as described above through the processing of the conversion unit 3 or the like, the integration is performed. The means 6 compares the luminance Di, j of the input pixel pi, j (hereinafter referred to as the input pixel pi, j) with the luminance D of the pixel p adjacent to the input pixel pi, j, The luminance Di, j of the pixel pi, j is compared with the average value of the luminances D of all the pixels p belonging to the group g to which the adjacent pixel p belongs, and the input pixel pi, j and the adjacent pixel p are integrated into one group g.

  This integration processing is performed every time the luminance D data of each pixel p of the image T is sequentially input as described above, and is finally performed for all the pixels p of one image transmitted. Each pixel p of the image T is divided into a plurality of groups g.

  When the integration unit 6 integrates the input pixels pi, j into the group g, the calculation unit 7 calculates and updates the average value Dave of the luminance D of each pixel p belonging to each group g in parallel. It is like that. The average value Dave of the luminance D of each pixel p belonging to the group g is calculated by the integration means 6 described above for the luminance Di, j of the input pixel pi, j and all the pixels p belonging to the group g to which the adjacent pixel p belongs. It is used when the average value Dave of each luminance D is compared, and also used to determine whether or not the group g belonging to the road surface area of the road surface is detected by the road surface area detection means 9 described later. .

  Further, the calculating means 7 calculates the horizontal width Wj for each pixel row j extending in the horizontal direction on the image T for each group g. In the present embodiment, two types of determinations are performed using the horizontal width Wj for each pixel row j of each group g.

  First, in this embodiment, in order not to detect the group g corresponding to each white line block of the pedestrian crossing marked on the road surface as the group G belonging to the road surface area of the road surface, A horizontal width threshold Wth is provided for the horizontal width in the space. The lateral width threshold Wth is set to a lateral width of 60 cm, for example, as the lateral width in real space.

  As described above, the data of the luminance D of each pixel p is input from the imaging unit 2 for each horizontal line j (pixel row j). At this time, the calculation unit 7 outputs the data for each horizontal line j. The pixel number threshold value Nth corresponding to the horizontal width threshold value Wth set as the horizontal width in the real space is calculated and set, and the integration unit 6 sets the input pixel pi, j and the adjacent pixel p to 1 as described above. In the integration process for integrating into one group g, the horizontal width Wj of the group g in the image row j is counted as the pixel number Nj, and the counted pixel number Nj is equal to or greater than the pixel number threshold Nth corresponding to the horizontal width threshold Wth. In this case, information indicating that the condition is satisfied is associated with the group g.

  As described above, in the present embodiment, the information indicating that the condition is satisfied is associated, so that the group g has a horizontal width equal to or larger than the horizontal width threshold value Wth in at least a part thereof in the real space. It can be seen that this corresponds to the object to be imaged.

  In the present embodiment, the horizontal width in the real space per pixel for each horizontal line j (pixel row j) of the image T is calculated and assigned in advance and stored in a storage means (not shown). The pixel number threshold value Nth for each horizontal line j corresponding to the horizontal width threshold value Wth that is the horizontal width in the real space is obtained by dividing the horizontal width threshold value Wth by the horizontal width in the real space per pixel of the horizontal line j. Can be calculated.

  Further, as another determination, the group g detected as a group extending in the vertical direction or the diagonal direction in the image T, such as a lane marked on the side of the host vehicle, is a road surface on the road surface. The horizontal width Wj for each pixel row j of each group g is used so that it is not determined that the group G belongs to the region.

  Normally, the lane is marked with a width of 15 cm on the road surface, and when the group g on the image T corresponds to only the lane, it is excluded by the determination process based on the width threshold Wth described above. Is not detected as a group G belonging to the road surface area of the road surface.

  However, as shown in FIG. 3, when a stop line SL or the like that is continuously displayed in the lanes LL and LR is captured in the image T, for example, the lateral width Wj of the group g corresponding to the lane LR is It is detected so that it suddenly increases at the stop line SL and becomes equal to or greater than the width threshold Wth. Therefore, the group g corresponding to the lane LR and the stop line SL may be detected as the group G belonging to the road surface area only by the determination process based on the lateral width threshold Wth.

  Therefore, in order to avoid this, in the present embodiment, the calculation means 7 calculates the average value Wave of the width Wj for each pixel row j of each group g.

  At that time, the lateral widths of the lanes LL and LR captured in the image T are very small compared to the lateral width of the road surface and the number of pixels is small. Therefore, in the present embodiment, in the calculation process for calculating the average value Wave, for the purpose of simplifying the process, the calculation unit 7 assigns the input pixel pi, j to the group g by the integration process in the integration unit 6. The number N of pixels belonging to the group g is simply added every time integration is performed, and when the integration processing in the integration unit 6 is completed, the number N of pixels belonging to the group g is converted to the vertical direction in the image T of the group g. Divide by the number of rows. The average value Wave of the horizontal width Wj for each pixel row j of the group g is calculated as the average value Nave of the number of pixels in the horizontal direction for each pixel row j thus calculated.

  If comprised in this way, even if the stop line SL labeled continuously with the lane LR as mentioned above is imaged, there are few pixel rows j corresponding to the stop line SL, and the pixel row corresponding to the lane LR Since j is overwhelmingly larger, the average value Nave of the number of pixels as the average value Wave of the horizontal width Wj for each pixel row j of the group g is a value close to the number of pixels corresponding to the lane LR.

  Therefore, the stop line SL is continuously indicated on the lane by setting the number of pixels corresponding to the lateral width of the lane or slightly larger than the average threshold value Nave_th for the average value Nave of the number of pixels. Even in such a case, the group g corresponding to the lane can be configured not to be detected as the group G belonging to the road surface area of the road surface.

  On the other hand, in this embodiment, as will be described later, an area R (hereinafter referred to as a check area R) is set in the traveling direction of the host vehicle in the image T. When the integration unit 6 integrates one pixel pi, j into the group g, the area calculation unit 8 simultaneously selects one pixel pi, j if the one pixel pi, j exists in the check region R. Each time j is integrated into the group g, addition is performed, and the area A that each group g occupies in the check region R of the image T is calculated as the number of pixels An.

  As described later, the road surface area detection unit 9 belongs to the horizontal width Wj (Nj) for each pixel row j of the group g calculated by the calculation unit 7 and the average value Wave (Nave) and the group g as described above. Based on the average value Dave of the luminance D of each pixel p, it is configured to determine whether or not the group g is a group G belonging to the road surface area of the road surface. Group g to be detected may be detected as a group G belonging to the road surface area of the road surface.

  Therefore, the area calculating unit 8 is configured to calculate the area A occupied by each group g in the check region R of the image T as the number of pixels An as described above, and the determination condition in the road surface region detecting unit 9 is as follows. By adding a condition regarding the area A that the group g occupies in the check region R, the group g corresponding to the sky or the like that is captured in the image T but does not exist in the check region R is changed to a road surface region on the road surface. It can be configured not to detect as a group G belonging to.

  In the present embodiment, the check region R can be set based on an estimated trajectory Lest estimated from the behavior of the host vehicle MC, for example, as shown in FIG. In the present embodiment, the area calculation means 8 turns the host vehicle MC based on information such as the vehicle speed V, yaw rate γ, steering angle δ of the steering wheel, etc. detected by the sensors Q (see FIG. 1). The curvature Cua is calculated according to the following formula (1) or the following formulas (2) and (3).

Cua = γ / V (1)
Re = (1 + Asf · V ² ) · (Lwb / δ) (2)
Cua = 1 / Re (3)
Here, in each of the above equations, Re is a turning radius, Asf is a vehicle stability factor, and Lwb is a wheelbase.

  Then, based on the calculated turning curvature Cua of the host vehicle MC, an estimated trajectory Lest that the host vehicle is estimated to travel in the future is calculated, and a range up to a position separated by a predetermined distance in the left-right direction around the estimated trajectory Lest is calculated. Calculated as the traveling path of the vehicle MC. Then, for example, as shown in FIG. 5, the check area R is set in the image T by assigning the above range, that is, the traveling path of the host vehicle MC to each corresponding pixel p on the image T.

  For example, the check region R can be set as a region having the left and right ends of the position of the lane detected on the side of the host vehicle MC. This is described in the second embodiment to be described later. I will explain it.

  As described above, the road surface area detection unit 9 calculates the horizontal width Wj (Nj) for each pixel row j of the group g calculated by the calculation unit 7, the average value Wave (Nave), and the luminance of each pixel p belonging to the group g. It is determined whether or not the average value Dave of D and the area A of the group g calculated by the area calculating means 8 are within a predetermined numerical range. Are detected as a group G belonging to the road surface area.

  In the present embodiment, as described above, when the number of pixels Nj as the width Wj in the pixel row j of the group g becomes equal to or larger than the pixel number threshold Nth corresponding to the width threshold Wth, as described above, In order to associate information indicating that the condition is satisfied with the group g, the road surface area detection unit 9 determines whether or not the above information is associated with the group g, and determines the width of each pixel row j of the group g. It is determined whether or not there is a pixel row j in which Wj is greater than or equal to a predetermined width threshold Wth.

  Hereinafter, each process in the processing unit 5 including the integration unit 6, the calculation unit 7, the area calculation unit 8, and the road surface area detection unit 9 will be described according to the flowcharts shown in FIGS. The operation of the road recognition device 1 will be described.

  In the processing in the processing unit 5, when the imaging unit 2 starts imaging at the current sampling cycle (step S1 in FIG. 6), the area calculating unit 8 inputs from the sensors Q at that time as described above. The turning curvature Cua of the host vehicle MC is calculated based on the information such as the vehicle speed V and yaw rate γ of the host vehicle MC and the steering angle δ of the steering wheel, and a check region R is set (step S2).

  Then, the calculating means 7 first sends the data of the brightness Di, j of each pixel pi, j from the imaging means 2 to the horizontal line j with j = 0, that is, the horizontal threshold value Wth described above in the horizontal line 0. A pixel number threshold Nth (0) corresponding to is calculated and set (step S3). Further, the integration unit 6 sets 0 as the values of i and j (step S4).

  Then, as described above, the input of the data of the luminance D0,0 of the leftmost pixel p0,0 (that is, the origin pixel) on the horizontal line 0 imaged by the imaging means 2 to the processing unit 5 is started ( Step S5). Subsequently, data of luminance D1,0, D2,0, D3,0,... Of the pixels p1,0, p2,0, p3,0,. If the integration process has not been completed up to the rightmost pixel of the horizontal line j (step S6; NO), the integration unit 6 increments the i coordinate by 1 (step S7), and the luminance Di of the pixel pi, j , j continues the integration process every time j data is input.

  Further, when the integration process is completed up to the rightmost pixel of the horizontal line j (step S6; YES), the integration unit 6 determines that the integration process has not been completed up to the uppermost horizontal line of the image T (step S8; NO). , J is incremented, i is set to 0 (step S9), the horizontal line j to be integrated is shifted to the horizontal line j one row above, and pixels sequentially input from the shifted horizontal line j The integration process is continued in order from the data of luminance D0, j of p0, j.

  Further, the calculating means 7 calculates and sets a pixel number threshold Nth (j) corresponding to the above-described width threshold Wth in the horizontal line j shifted from the imaging means 2 (step S10).

  Next, each process (after step S11 of FIG. 7) in the processing unit 5 when data of luminance Di, j of the pixel pi, j is input will be described.

  First, the integration means 6 inputs the input pixel pi, j (input pixel pi, j) and the input pixel pi, j before the input as shown in FIG. It is determined whether or not the following conditions 1 and 2 are satisfied for the pixels pi-1, j adjacent to (step S11).

[Condition 1] The difference ΔDleft (i, j) between the luminance Di, j of the input pixel pi, j and the luminance Di-1, j of the pixel pi-1, j adjacent to the left, that is,
ΔDleft (i, j) = | Di, j−Di−1, j | (4)
Is less than a preset first luminance threshold value ΔDth. Hereinafter, the difference ΔD of the luminance D between adjacent pixels as described above is referred to as edge strength.

[Condition 2] As shown in FIG. 11, the average value Dave-left of the luminance D i, j of the input pixel p i, j and the luminance D of all the pixels belonging to the group g to which the adjacent pixel p i-1, j on the left belongs. Difference δDleft (i, j), that is,
δDleft (i, j) = | Di, j−Dave-left | (5)
Is less than the preset second luminance threshold value δDth.

  Hereinafter, as described above, the difference δD between the luminance Di, j of the input pixel pi, j and the average value Dave of the luminance D of the group g to which the adjacent pixel belongs is referred to as an average value difference. Further, the average value Dave of the luminance D of the group g to which the adjacent pixel belongs is calculated by the calculation means 7 in the calculation process in step S25 of FIG. Furthermore, the group g to which the pixel pi-1, j adjacent to the left belongs may be composed of only the pixel pi-1, j adjacent to the left, and in this case, the luminance D of all the pixels belonging to the group g. The average value Dave-left is equal to the luminance Di-1, j of the pixel pi-1, j adjacent to the left.

  If the integration unit 6 determines that both the condition 1 and the condition 2 are satisfied (step S11; YES), the integration unit 6 proceeds to the determination process in step S12, and determines that at least one of the condition 1 and the condition 2 is not satisfied. (Step S11; NO), the process proceeds to the determination process of step S15.

  In the present embodiment, the first luminance threshold value ΔDth and the second luminance threshold value δDth are set to the same value, but may be set to different values and set appropriately. It is also possible to vary the values of the first luminance threshold value ΔDth and the second luminance threshold value δDth according to the situation.

  If the integration unit 6 determines that both the conditions 1 and 2 are satisfied in the determination process of step S11 (step S11; YES), then the integration unit 6 continues to input pixel pi, j and input pixel pi, j as shown in FIG. In the same manner as described above, it is determined whether or not the following condition 3 or condition 4 is satisfied for the pixel pi, j-1 that is input before j is input and is adjacent to the input pixel pi, j. It performs (step S12).

[Condition 3] Edge intensity ΔDlower (i, j) between the luminance Di, j of the input pixel pi, j and the luminance Di, j-1 of the pixel pi, j-1 adjacent below, that is,
ΔDlower (i, j) = | Di, j−Di, j−1 | (6)
Is less than the preset first luminance threshold value ΔDth.

[Condition 4] As shown in FIG. 13, the average value Dave-lower of the luminance D i, j of the input pixel p i, j and the luminance D of all the pixels belonging to the group g to which the adjacent pixel p i, j−1 belongs. Mean value difference δDlower (i, j), that is,
δDlower (i, j) = | Di, j−Dave-lower | (7)
Is less than the preset second luminance threshold value δDth.

  Also in this case, the group g to which the pixel pi, j-1 adjacent to the lower side may be composed of only the pixel pi, j-1 adjacent to the lower side, in which case all of the groups g belonging to the group g are included. The average value Dave-lower of the luminance D of the pixel is equal to the luminance Di, j-1 of the adjacent pixel pi, j-1.

  When the integration unit 6 determines that at least one of the condition 3 and the condition 4 is not satisfied (step S12; NO), the integration unit 6 defines the input pixel pi, j as the pixel pi, j-1 adjacent thereto below. Although not integrated, since it is determined in the determination process of step S11 that the above conditions 1 and 2 are satisfied, the input pixel pi, j and the pixel pi-1, j adjacent to the left are grouped into one group. Integration is performed (step S13).

  In this case, for example, as shown in FIG. 10, if the pixel pi-1, j adjacent to the left is not grouped with other pixels, the pixel pi-1, j adjacent to the left of the input pixel pi, j is used. Are integrated into one group, and a group of two adjacent pixels on the left and right is newly formed. For example, as shown in FIG. 11, if the pixel pi-1, j adjacent to the left is integrated with other pixels and belongs to the group g, the input pixel pi, j is added to the group g. The group g is enlarged by one pixel by the input pixel pi, j.

  When the group g to which the pixel pi-1, j adjacent to the left belongs has a shape as shown in FIG. 14A, for example, the determination process in step S12 satisfies at least one of the condition 3 and the condition 4. Even if the input pixel pi, j is not integrated with the lower adjacent pixel pi, j-1 as shown in FIG. 14B, it is determined that the input pixel pi, j is not integrated (step S12; NO). By integrating with the pixels pi-1, j adjacent to the left (step S13), the input pixel pi, j is integrated with the adjacent pixels pi, j-1 and one group g as a result. In some cases.

  Next, in the determination process in step S12, the integration unit 6 determines that both the condition 3 and the condition 4 are satisfied (step S12; YES), and the input pixel pi, j and the adjacent pixel pi, j-1 and the pixel pi-1, j adjacent to the left are integrated into one group (step S14).

  At this time, for example, as shown in FIG. 12, if the pixel pi, j-1 adjacent below and the pixel pi-1, j adjacent to the left are not grouped with other pixels, the input pixel pi, j and the pixel pi, j-1 adjacent to the lower side and the pixel pi-1, j adjacent to the left are integrated into one group to form a new group of three pixels. Further, for example, as shown in FIGS. 11 and 13, one of the pixel pi, j-1 adjacent to the lower side and the pixel pi-1, j adjacent to the left is integrated with the other pixels, and the group g , The input pixel pi, j and the pixel that does not belong to the group g are integrated so as to be added to the group g, and the group g is expanded by two pixels.

  Further, for example, as shown in FIG. 15A, when the pixel pi-1, j adjacent to the left belongs to the group g1, and the pixel pi, j-1 adjacent to the lower belongs to the other group g2, When the input pixel pi, j is integrated with the pixel pi, j-1 adjacent to the lower side and the pixel pi-1, j adjacent to the left side (step S14), as shown in FIG. The group g1 and the group g2 are integrated through j to form one group g.

  On the other hand, when the integration unit 6 determines that at least one of the condition 1 and the condition 2 is not satisfied in the determination process of step S11 (step S11; NO), the integration unit 6 proceeds to the determination process of step S15, and similarly to the above. Then, it is determined whether or not the conditions 3 and 4 are satisfied (step S15).

  If the integration unit 6 determines that both the condition 3 and the condition 4 are satisfied (step S13; YES), the integration unit 6 determines that at least one of the condition 1 and the condition 2 is not satisfied in the determination process of step S9. Therefore (step S9; NO), the input pixel pi, j and the pixel pi-1, j adjacent to the left are not integrated, and only the pixel pi, j-1 adjacent below is integrated into one group ( Step S14).

  At this time, when the input pixel pi, j and the lower adjacent pixel pi, j-1 are integrated into one group (step S16), the result is integrated with the left adjacent pixel pi-1, j. It can be easily inferred from the cases shown in FIGS. 14A and 14B that there are cases.

  If the integration unit 6 determines in step S15 that at least one of condition 3 and condition 4 is not satisfied (step S15; NO), the integration unit 6 sets the input pixel pi, j to the pixel adjacent to the left. Both pi-1, j and the adjacent pixels pi, j-1 are registered as a new group g to which only the input pixel pi, j belongs without being integrated (step S17).

  In addition, when the integration unit 6 performs integration processing, if the number of pixels of the integrated group g is very small and can be ignored such as noise, the group g is deleted from the group registration. Thus, it may be configured not to be a target for determining whether or not the group G belongs to the road surface area of the road surface.

  When the integration unit 6 integrates the input pixels pi, j into the group g (steps S13, S14, and S16), for example, as shown in FIG. 15B, a plurality of groups are integrated into one group. For example, the group number of the group g integrated into one is updated as necessary by selecting, for example, the smallest number among the group numbers of the plurality of groups to be integrated. When the input pixel pi, j is registered as a new group g (step S17), a group number is assigned to the new group g (step S18).

  Further, when the input pixel pi, j is registered as a new group g (step S17), the calculation means 7 sets a later-described condition satisfaction flag F associated with the group g to 0 (step in FIG. 8). S19). Further, when the input pixels pi, j are integrated into the group g (step S14), and a plurality of groups are integrated into one group (see, for example, FIG. 15B), the calculating means 7 If there is a group whose condition satisfaction flag F is 1 among a plurality of groups targeted for integration, the condition satisfaction flag F of the group g integrated into one is set to 1, and If all the condition satisfaction flags F of the plurality of groups are 0, the condition satisfaction flag F of the group g integrated into one is set to 0 (step S19).

  If the condition satisfaction flag F of the group g currently undergoing the integration process is 0 (step S20; YES), the number of pixels Nj in the pixel row j of the group g is counted (step S21). Then, referring to the pixel number threshold Nth (j) corresponding to the horizontal width threshold Wth in the horizontal line j set in the setting process in step S3 or step S10 in FIG. 6, the counted number of pixels Nj in the pixel row j is If it is greater than or equal to the pixel number threshold Nth (j) (step S22; YES), the condition establishment flag F of the group g is switched to 1 (step S23), and information indicating that the condition is established is set in the group g. Associate.

  That is, in this embodiment, setting the condition satisfaction flag F of the group g to 1 corresponds to associating information indicating that the condition is satisfied with the group g. If the counted pixel number Nj in the pixel row j is not greater than or equal to the pixel number threshold Nth (j) (step S22; NO), the condition satisfaction flag F of the group g is kept at 0.

  Further, if the condition satisfaction flag F of the group g currently undergoing the integration process is 1 (step S20; NO), in the subsequent integration process, the pixel number Nj is once set to the pixel number threshold Nth ( j) Since the number of pixels belonging to the above pixel row j does not decrease and the condition satisfaction flag F of the group g is not set to 0, it is not necessary to perform the processes of steps S21 to S23. .

  Therefore, in this embodiment, for the purpose of speeding up the processing, if the condition establishment flag F of the group g for which the current integration processing has been performed is 1 (step S20; NO), each processing of steps S21 to S23 is performed. It is supposed to skip without doing.

  Subsequently, the calculating unit 7 calculates the number N of pixels of the group g that has been subjected to the integration process (step S24). That is, when the input pixel pi, j is registered as a new group g, the number of pixels N of the group g is set to 1, and when only the input pixel pi, j is integrated into the group g, When the number of pixels N is increased by 1 and the input pixel pi, j is integrated into the group g, and the other pixel p is also integrated into the group g, the number of pixels N of the group g is increased by 2. Let

  In addition, when two groups g1 and g2 are integrated into one group g via the input pixel pi, j, the number N of pixels of the group g integrated into one group g1 and g2 The number of pixels is calculated by adding one pixel of the input pixel pi, j to the sum of the number of pixels.

  Subsequently, the calculation means 7 calculates the sum of the luminance D of each pixel p belonging to the group g, divides it by the number N of pixels of the group g, and calculates the luminance D of each pixel p belonging to the group g. The average value Dave is calculated and updated (step S25). In this case, when the input pixel pi, j is registered as a new group g, the number of pixels N of the group g is 1, so that the luminance Di, j of the input pixel pi, j belongs to the group g. The average value Dave of the luminance D of the pixel p is obtained.

  If necessary, the coordinates of the left and right pixels of the group g, the coordinates of the upper and lower pixels, the center point of the group g, and the like can be calculated and updated. It is.

  The area calculation means 8 determines whether or not the input pixel pi, j is in the check region R based on the check region R set in the setting process in step S2 of FIG. 6 (step S26), and the input pixel pi. , j is in the check region R (step S26; YES), the area A occupied by the group g in the check region R is calculated as the number of pixels An (step S27).

  That is, when the input pixel pi, j is in the check region R (step S26; YES), when the input pixel pi, j is registered as a new group g, the number of pixels An of the area A of the group g is set. When only one input pixel p i, j is integrated into group g, the number of pixels An of area A of group g is increased by one.

  Further, when the other pixel p is also integrated into the group g by integrating the input pixel pi, j into the group g, if the other pixel p is within the check region R, the If the number An of the area A of the group g is increased by 2 and the other one pixel p is not in the check region R and only the input pixel pi, j is in the check region R, the group g The number of pixels An of area A is increased by one.

  Further, when two groups g1 and g2 are integrated into one group g via the input pixel pi, j, the number of pixels An of the area A of the group g integrated into the one group The pixel number is calculated by adding one pixel of the input pixel pi, j to the sum of the pixel number An of each area A of g1 and g2.

  Until the integration process of the uppermost horizontal line of the image T is completed (step S8 in FIG. 6), the above-described processes by the integration unit 6, the calculation unit 7, and the area calculation unit 8 are performed on the input pixels pi, j. Repeated for each input.

  When the integration process is completed up to the uppermost horizontal line of the image T (step S8; YES), each pixel p is integrated into each group g by the integration process by the integration unit 6, and finally each pixel of the image T. p is divided into a plurality of groups g. For example, each pixel p of the image T shown in FIG. 5 is integrated into a plurality of groups g as shown in FIG.

  Subsequently, the calculation means 7 divides the number of pixels N belonging to the group g calculated for each group g by the number of rows of the pixel rows j in the vertical direction of each group g in the calculation processing of step S24 in FIG. An average value Nave of the number of pixels is calculated, and an average value Wave of the horizontal width Wj for each pixel row j of each group g is calculated as the average value Nave of the number of pixels (step S28 in FIG. 9).

  Next, the road surface area detection means 9 determines, for each group g, whether or not the group g is a group G belonging to the road surface area of the road surface. It is detected and registered as a group G belonging to the road surface area.

  Specifically, if there is a group g that has not yet been determined (step S29; YES), the road surface area detection means 9 first determines whether or not the condition satisfaction flag F of the group g is 1 ( In step S30), the condition establishment flag F is 0 (step S30; NO), and if the information indicating that the condition is established is not associated with the group g, the determination process for the group g is ended.

  If the condition satisfaction flag F of the group g is 1, and information indicating that the condition is satisfied is associated with the group g (step S30; YES), the group g has a width Nj for each pixel row j. Satisfies the condition that there is a pixel row j having a pixel number threshold Nth or more corresponding to a predetermined width threshold Wth.

  Then, it can be seen that the group g corresponds to an imaging target in which at least part of the lateral width Wj in the real space is equal to or greater than the lateral width threshold Wth (for example, 60 cm). Therefore, the group g corresponding to each white line block of the pedestrian crossing which is about 50 cm even if the width Wj in the real space is large, and the markings on the road surface such as lanes, numbers, letters, arrows without a stop line, In the determination processing in step S30, the group g detected as the group G belonging to the road surface area of the road surface is excluded.

  Subsequently, the road surface area detection unit 9 is preset with an average value Nave of the number of pixels calculated by the calculation unit 7 as the average value Wave of the width Wj for each pixel row j of the group g in the calculation process of step S28. It is determined whether or not the average value threshold value Nave_th is equal to or greater than the average value threshold value Nave_th (step S31). If the average value Nave of the number of pixels is not equal to or greater than the average value threshold value Nave_th (step S31; NO), the determination process for the group g ends. .

  As shown in FIG. 3, even if a stop line SL marked on the road surface continuously to the lane LR is captured in the image T, the group g corresponding to the lane LR with the stop line SL Since the average value Nave of the number of pixels does not exceed the average value threshold value Nave_th (step S31; NO), the group g corresponding to the lane LR with the stop line SL is determined in the determination process of step S31. Are excluded from the determination targets of the group g detected as the group G belonging to.

  If the average value Nave of the number of pixels of the group g is equal to or greater than the average value threshold value Nave_th (step S31; YES), the road surface detection unit 9 calculates the calculation unit 7 in the calculation process of step S25 in FIG. It is determined whether or not the average value Dave of the luminance D of each pixel p of the group g is equal to or less than a predetermined luminance average threshold value Dave_th (step S32), and the average value Dave of the luminance D is the luminance average threshold value Dave_th. If it is below (step S32; NO), the determination process for the group g ends.

  This luminance average threshold value Dave_th is normally set to a luminance that is low enough to be detected as the luminance D of the pixel p in which the road surface area of the road surface is imaged. For this reason, the group g corresponding to the marking on the road surface such as a pedestrian crossing, a stop line, a lane, a number, a letter, an arrow, etc. in the determination process of step S30 or step S31 is a group G belonging to the road surface area of the road surface. Even if it is not excluded from the determination target of the group g to be detected, it is excluded from the determination target in the determination process of step S32.

  In addition, in each group g in the image T shown in FIG. 16, the group gsky corresponding to the sky is captured with high brightness, so that the group gsky corresponding to the sky is also determined in the determination process of step S32 in the road surface. Are excluded from the determination targets of the group g detected as the group G belonging to the road surface area.

  However, in the case of cloudy weather or rainy weather, the average value Dave of the luminance D of each pixel p of the group gsky corresponding to the sky may be equal to or less than the luminance average threshold value Dave_th (step S32; YES). In addition, as shown in FIG. 16, the group gtree corresponding to the roadside roadside tree is also a determination target of the group g detected as the group G belonging to the road surface area of the road surface in the determination processing of step S30 and step S31 described above. In addition, since the luminance D of each pixel p belonging to the group gtree is low, the average value Dave of the luminance D of each pixel p of the group gtree corresponding to the roadside tree may be equal to or less than the luminance average threshold Dave_th. (Step S32; YES).

  Therefore, the road surface area detection means 9 subsequently sets in advance the number of pixels An as the area A occupied in the check area R of the group g calculated by the area calculation means 8 in the calculation process of step S27 in FIG. Whether or not it is equal to or greater than the area threshold Ath (step S33). If the number of pixels An is not equal to or greater than the area threshold Ath (step S33; NO), the determination process for the group g ends.

  As shown in FIG. 5, the check region R is set in the traveling direction of the host vehicle in the image T based on the estimated trajectory Lest of the host vehicle, and is a pixel region in which street trees on the sky and the roadside are imaged. Does not exist in this check region R, the number of pixels An as the area A occupied in the check region R of the groups gsky and gtree corresponding to the sky and street trees shown in FIG. For this reason, the groups gsky and gtree corresponding to the sky and the roadside tree are excluded from the determination targets of the group g detected as the group G belonging to the road surface area of the road surface in the determination process of step S33.

  In addition, as shown in FIG. 16, the low-luminance group gko detected in the lower right in the image T in the entry prohibition sign may clear the determination processes in the above steps S30 to S32. Further, as shown in FIG. 5, since a part of the check region R covers a portion corresponding to the group gko, the number of pixels An as the area A occupied in the check region R of the group gko does not become zero. There is a case.

  However, even in such a case, by setting the area threshold value Ath to a certain large value in advance, a group such as the group gko can be assigned to the group G belonging to the road surface area of the road surface in the determination process of step S33. It is possible to accurately exclude from the determination target of the group g detected as.

  When the group g clears all the determination processes in steps S30 to S33, the road surface area detection unit 9 detects the group g as a group G belonging to the road surface area of the road surface (step S34). For example, as a result of the processing on the image T shown in FIG. 5, as shown in FIG. 17, the groups G1 to G4 are detected as the group G belonging to the road surface area of the road surface. Then, by connecting these groups G1 to G4 on the image T, it is possible to extract and detect the road surface area of the road surface from the image T.

  The road surface area detection means 9 stores each information of the groups G1 to G4 and information on the road surface area of the road surface detected by connecting them together in the storage means, and outputs them to an external device as necessary. In addition, when each information of each detected group G1-G4 does not have consistency with the information of the road surface area of the road surface extracted by the last sampling period, the process of excluding the information of the group G which does not have consistency etc. It is also possible to configure so that

  When the processing for all the groups g integrated on the image T by the integration unit 6 is completed (step S29; NO), the processing for the image T for one image input from the imaging unit 2 in the current sampling cycle is performed. When the imaging unit 2 starts imaging at the next sampling cycle (step S1 in FIG. 6), each processing from step S2 is executed again by the processing unit 5.

  As described above, according to the road recognition device 1 according to the present embodiment, the integration unit 6 integrates the pixels p of the image T into the groups g based on the luminance D, and then performs the process for each pixel row j of the group g. Based on the width Wj (Nj), the average value Wave (Nave), the average value Dave of the luminance D of each pixel p belonging to the group g, and the area A (An) that the group g occupies in the check region R of the image T, It is determined whether or not the group g is a group G belonging to the road surface area of the road surface.

  That is, the determination is made based on the size and shape of the group g in the image T (width and average value thereof), average luminance (average value of luminance), and position occupied in the image T (area occupied in the check region). Therefore, it is possible to accurately determine whether or not the group g is the group G belonging to the road surface area of the road surface, and to accurately detect the road surface area of the road surface from the image T.

  In particular, in the vehicle environment recognition device described in Patent Document 3 above, a window is set in the image T, and the pixel color near the center of the lower end of the window is used as a reference color for extracting the road surface area of the road surface. Since the reference color is used, when a white display such as a stop line is imaged near the center of the lower end of the window, the white color becomes the reference color, and it may be difficult to correctly extract the road surface area. was there.

  However, in the road recognition device 1 according to the present embodiment, it is determined whether or not the group g is the group G belonging to the road surface area of the road surface based on the average value Dave of the luminance D of each pixel p belonging to the group g. Therefore, it is possible to accurately prevent the occurrence of such erroneous extraction and misjudgment problems and accurately detect the road surface area of the road surface.

  Further, in parallel with the integration process of each group g by the integration unit 6, each process by the calculation unit 7 and the area calculation unit 8 is performed, and after the integration process is completed, the calculation unit 7 calculates step S28 in FIG. Since the process and the determination process of steps S30 to S33 by the road surface area detection means 9 are performed, the road surface area of the road surface from the image T can be detected very quickly.

  Therefore, for example, even when the luminance of an image for several frames or several tens of frames is captured per second and processing must be performed one after another, the road surface area of the road surface is accurately determined from the image T for each frame. Can be detected in real time, and processing can be performed in real time.

  Furthermore, according to the road recognition apparatus 1 according to the present embodiment, the road surface area of the road surface is detected from the image T as each group g obtained by integrating the pixels p of the image T based on the luminance D by the integration unit 6. For example, even if the road on which the host vehicle is traveling is a road in which no lane is marked on either the left or right side of the host vehicle, the road surface area of the road surface can be accurately detected from the image T. .

  At this time, the check region R (see FIGS. 4 and 5) can be set based on, for example, the estimated trajectory Lest estimated from the behavior of the host vehicle MC, so that the road on which the host vehicle is traveling is a lane. Even if the road is not marked, it is possible to set the check area R effectively and accurately detect the road surface area from the image T without depending on the lane.

  Further, as described above, the road recognition device 1 according to the present embodiment extracts the road surface area of the road surface from the image T by connecting the groups G detected as the group G belonging to the road surface area of the road surface. In order to detect, as shown in FIG. 18, the road B having the same luminance D as the road on which the host vehicle is traveling and the road B joining the road on which the host vehicle is traveling is also detected effectively. It becomes possible.

  Further, in the far region C of the road on which the host vehicle is traveling, the lanes on the side of the host vehicle are captured very thinly on the image T, making it difficult to detect. By connecting the groups G detected as the groups G belonging to the road surface area of the road surface, it is possible to detect from the image T effectively.

  As described above, the road recognition device 1 according to the present embodiment can accurately detect the road B that joins the road on which the host vehicle is traveling, and the distant region C of the road on which the host vehicle is traveling. It becomes possible, and it becomes possible to detect the road surface area of the road surface imaged by the image T exactly.

  In addition, since the brightness of the road surface area of the road surface in the image T also changes according to the shutter level of the imaging unit 2, the luminance average threshold value Dave_th is configured to vary according to the shutter level of the imaging unit 2. It is also possible to do.

  Specifically, when the image pickup means 2 is configured with a CCD camera or the like as in the present embodiment, as a function normally provided in the CCD camera or the like, appropriate brightness is obtained by determining the brightness around the own vehicle. Therefore, the exposure adjustment is automatically performed. That is, for example, exposure adjustment such as shutter time adjustment, amplifier gain switching, LUT (Look Up Table) selection, and luminance value conversion based on it is automatically performed.

  Therefore, for example, when the image T is transmitted from the imaging unit 2, the exposure adjustment level is also transmitted to the processing unit 5 as the shutter level, and the road surface area detection unit 9 performs the imaging unit 2. The luminance average threshold value Dave_th can be varied according to the shutter level.

  If comprised in this way, as above-mentioned, it will image darkly like each road marking, such as a lane, a pedestrian crossing, a stop line, a number, a character, an arrow, etc., and the road surface area of a road surface. The imaging target can be accurately selected based on the luminance average threshold value Dave_th that is varied according to the shutter level of the imaging unit 2.

  Therefore, for example, even when the group g corresponding to the marking on the road surface such as a lane is not excluded in the determination processing in step S30 or step S31 in FIG. 9, it is accurately excluded from the determination target in the determination processing in step S32. Thus, the group g having a low luminance D to be detected as the group G belonging to the road surface area of the road surface can be accurately detected as the group G belonging to the road surface area of the road surface.

[Second Embodiment]
In the first embodiment, the road recognition device that processes the luminance D of each pixel p of one image T captured by the imaging unit 2 and detects road surface areas G1 to G4 of the road surface in the image T. 1 was described.

  However, when a pair of images is captured using a pair of cameras (that is, so-called stereo cameras) as the imaging unit 2 and processing is performed based on the pair of images, a distance Z in the real space to the imaging target captured in the image T is obtained. Therefore, the road surface area of the road surface can be detected more accurately based on this. In the second embodiment, a road recognition device configured as described above will be described.

  As shown in FIG. 19, the road recognition device 10 according to the present embodiment is similar to the road recognition device 1 (see FIG. 1) in the first embodiment, the imaging unit 2, the conversion unit 3, the image correction unit 4, The processing unit 5 includes an integration unit 6, a calculation unit 7, an area calculation unit 8, and a road surface area detection unit 9, and further includes a distance detection unit 11 including an image processor 12.

  Note that the configuration on the upstream side of the processing unit 5 including the image processor 12 and the like is described in detail in Japanese Patent Application Laid-Open No. 2006-72495 and the detailed description of the configuration is left to those publications. A brief description is given below.

  In the present embodiment, the image pickup means 2 includes a built-in image sensor such as a CCD or a CMOS sensor that is synchronized with each other. For example, a pair of image sensors 2 attached in the vicinity of a vehicle rearview mirror with a predetermined interval in the vehicle width direction. The stereo camera is composed of a main camera 2a and a sub camera 2b, and is configured to capture a predetermined sampling period and output a pair of images.

  Of the pair of cameras, the main camera 2a is a camera closer to the driver, and captures an image T as shown in FIG. 28, for example. Hereinafter, an image T captured by the main camera 2a is referred to as a reference image T, and an image captured by the sub camera 2b is referred to as a comparative image.

  Also in the present embodiment, when a reference image T as shown in FIG. 2 or a comparison image not shown is taken by the main camera 2a and the sub camera 2b of the imaging means 2, each horizontal line such as the reference image T is displayed. The reference image is scanned in the right direction in order from the leftmost image sensor of j, and the horizontal line j to be scanned is imaged while being switched upward in order from the lowermost line, so that the reference images are picked up by the respective image sensors. Data of T and each brightness D of the comparison image are sequentially transmitted to the conversion means 3, respectively.

  The conversion means 3 is composed of a pair of A / D converters 3a and 3b, and each luminance D of the reference image T and the comparison image taken for each pixel p by the main camera 2a and the sub camera 2b of the image pickup means 2. Are sequentially transmitted, each luminance D is converted into a luminance D of a gray scale digital value of, for example, 256 gradations and output to the image correction unit 4.

  The image correction unit 4 sequentially performs image correction such as displacement, noise removal, luminance correction, and the like on the transmitted reference image T and data of each luminance D of the comparison image, and the corrected image of the reference image T and Data of each brightness D of the comparison image is sequentially stored in the image data memory 14 and is sequentially transmitted to the processing unit 5. Further, the image correction unit 4 sequentially transmits the data of each luminance D of the reference image T and the comparison image subjected to the image correction to the distance detection unit 11.

  In the image processor 12 of the distance detection unit 11, stereo matching processing and filtering processing are sequentially performed on the data of each luminance D of the reference image T and the comparison image, and the parallax dp corresponding to the distance in the real space is obtained as the reference image T. Are sequentially calculated for each pixel p.

In the stereo matching process in the image processor 12, when data of each luminance D of the reference image T and the comparison image Tc are transmitted, as shown in FIG. 20, for example, 3 × 3 pixels or 4 × 4 on the reference image T. A reference pixel block PB having a predetermined number of pixels such as pixels is set, and for each comparison pixel block PBc having the same shape as the reference pixel block PB on the epipolar line EPL in the comparison image Tc corresponding to the reference pixel block PB, the following (8 ) Is used to calculate a SAD value that is a difference in luminance pattern from the reference pixel block PB, and a comparison pixel block PBc having the smallest SAD value is specified.
SAD = Σ | D1s, t−D2s, t | (8)

  In the above equation (8), D1s, t represents the luminance D of each pixel p in the reference pixel block PB, and D2s, t represents the luminance D of each pixel p in the comparison pixel block PBc. Further, the above sum is obtained when the reference pixel block PB and the comparison pixel block PBc are set as a 3 × 3 pixel region, for example, a range of 1 ≦ s ≦ 3, 1 ≦ t ≦ 3, and 4 × 4 pixels. When set as a region, calculation is performed for all pixels p in the range of 1 ≦ s ≦ 4 and 1 ≦ t ≦ 4.

  For each reference pixel block PB of the reference image T in this way, the image processor 12 calculates the parallax dp from the position on the comparison image Tc of the identified comparison pixel block PBc and the position on the reference image T of the reference pixel block PB. It is calculated sequentially. Hereinafter, an image in which the parallax dp is assigned to each pixel p of the reference image T is referred to as a distance image. In addition, information on the parallax dp calculated for each pixel p in this way, that is, a distance image is sequentially stored in the distance data memory 13 of the distance detecting unit 11 and is sequentially transmitted to the processing unit 5. ing.

In real space, a point on the road surface directly below the center of the pair of cameras 2a and 2b is used as an origin, and the vehicle width direction (that is, the horizontal direction) of the host vehicle is the X-axis direction and the vehicle height direction (that is, the height). (Direction) is the Y-axis direction and the vehicle length direction (that is, the distance direction) is the Z-axis direction, the point (X, Y, Z) in the real space and the coordinates (i, j) of the pixel p on the distance image And the parallax dp can be uniquely associated by coordinate transformation based on the principle of triangulation expressed by the following equations (9) to (11).
X = CD / 2 + Z * PW * (i-IV) (9)
Y = CH + Z × PW × (j−JV) (10)
Z = CD / (PW × (dp−DP)) (11)

  In each of the above formulas, CD is the distance between the pair of cameras, PW is the viewing angle per pixel, CH is the mounting height of the pair of cameras, and IV and JV are i coordinates on the distance image at the infinity point in front of the host vehicle. And j coordinate and DP represent vanishing point parallax.

  Further, for the purpose of improving the reliability of the parallax dp, the image processor 12 performs a filtering process on the parallax dp obtained by the stereo matching process as described above, and outputs only the valid parallax dp. It has become.

  For example, even if a stereo matching process is performed on the reference image block PB of 4 × 4 pixels having poor features consisting only of a road surface image, the road surface of the comparison image Tc is captured. Since all the correlations are high, the reliability of the parallax dp is low even if the corresponding comparison pixel block PBc is specified and the parallax dp is calculated. Therefore, the parallax dp is invalidated by performing a filtering process, and 0 is output as the value of the parallax dp.

  Therefore, when the distance image Tz is created by assigning (ie, associating) effectively calculated parallax dp to each pixel p of the reference image T, the distance image Tz is, for example, as shown in FIG. This is an image in which the effective parallax dp is calculated for the edge portion (edge portion) of the imaging target, which is a portion having a significant feature above. In creating the distance image Tz, the parallax dp is converted into the distance Z or the like in advance according to the above equation (11), and the distance Z or the like is assigned to each pixel p of the reference image T for creation. Is possible.

  In the present embodiment, in each process in the processing unit 5, the parallax dp in each pixel p of the distance image Tz is converted into a distance Z in the real space and used according to the above equation (11) as necessary.

  Also in this embodiment, the processing unit 5 is configured by a computer in which a CPU, a ROM, a RAM, an input / output interface and the like (not shown) are connected to a bus, and the integration unit 6 and the calculation unit similar to those in the first embodiment. 7, an area calculating means 8 and a road surface area detecting means 9 are provided. In the present embodiment, the processing in the integration unit 6 and the like is performed on the reference image T. However, the processing may be performed on the comparison image Tc, and the reference image Tc may be configured. It is also possible to configure so that both the image T and the comparison image Tc are processed.

  In the present embodiment, the processing unit 5 is further provided with lane detection means 15 for detecting a lane marked on the side of the host vehicle from the reference image T. Further, similarly to the first embodiment, the processing unit 5 can be configured to be provided with a preceding vehicle detection means for detecting a preceding vehicle, and a vehicle speed sensor, a yaw rate sensor, a steering wheel, and the like can be provided as necessary. It is also possible to configure so that measurement values from sensors such as a steering angle sensor for measuring the steering angle of the wheel are input.

  Here, the lane detection unit 15 will be described before describing the processing in the integration unit 6 and the like of the processing unit 5 of the present embodiment.

  The lane detection unit 15 detects a lane that exists on the side of the host vehicle from the reference image T captured by the imaging unit 2. Specifically, as shown in FIG. 22, the lane detection unit 15 searches the horizontal line j having a width of one pixel, for example, from the center of the reference image T in the left-right direction using the reference image T, and the luminance D Are detected as lane candidate points cl and cr that greatly change from the luminance D of the adjacent pixel p to a threshold value or more.

  Further, for example, at the lane candidate point cr detected by searching from the center of the reference image T to the right, the luminance D increases greatly as shown in FIG. 23, but the search continues further to the right and the lane ends. The lane end point candidate point cre is reduced to a low brightness corresponding to the road surface. Although not shown, the same applies to the lane candidate point cl detected by searching in the left direction.

  Therefore, in this embodiment, when the lane detection unit 15 detects the lane candidate points cl and cr, the lane detection unit 15 continues to search leftward or rightward until the lane end point candidate points cle and cre are detected. The brightness D of the pixel p corresponding to the lane is voted on a histogram (not shown) created in advance, and the histogram is updated. Note that the brightness of the road surface may be calculated by, for example, voting the brightness D of the low brightness pixel p corresponding to the road surface before reaching the lane candidate point cr or the like to the histogram. .

  Then, the lane detection means 15 detects lane candidate points on each horizontal line j in the same manner while shifting the horizontal line j on the reference image T upward by one pixel at a time. At that time, voting on the histogram of the luminance D of the pixel p corresponding to the detected lane is continued.

  If the lane detection unit 15 determines that the lane candidate point is not on the road surface based on the parallax dp or the like of the detected lane candidate point, the lane detection unit 15 excludes the lane candidate point from the lane candidate point. In this case, the position of the road surface in the current sampling cycle is estimated from the behavior of the host vehicle and the like based on the road surface detected in the previous sampling cycle.

  Then, of the remaining lane candidate points, the lane is approximated by a straight line by Hough transform or the like based on the lane candidate points on the side close to the own vehicle, and detected to the side of the own vehicle. At that time, various straight lines are calculated as candidates in the Hough transform. For example, when a plurality of lanes are detected on one side (for example, the right side) of the own vehicle, the line is detected on the other side (for example, the left side) of the own vehicle. A straight line is selected on each side of the host vehicle, for example, by selecting a lane that is consistent with the selected lane or a lane that is consistent with the lane detected in the previous sampling cycle.

  In this way, when the lane is detected in a straight line on the side closer to the host vehicle, the lane candidate points are selected and connected based on the positional relationship with the straight line based on the straight line on the side farther than that. As shown in the figure, lanes LR and LL are detected on the sides of the host vehicle. The processing configuration of the above lane detection means 15 is described in detail in Japanese Patent Application Laid-Open No. 2006-331389 previously filed by the applicant of the present application, and the detailed description should be referred to the same publication.

  The lane detection means 15 stores information such as the detected lane positions LL and LR and lane candidate points cl and cr in a storage means (not shown).

  In the present embodiment, the lane detection means 15 is configured to form a lane model three-dimensionally based on information on the detected lane positions LL and LR and lane candidate points. That is, as shown in FIGS. 25A and 25B, the lane detection means 15 approximates the lane detected on the side of the host vehicle with a three-dimensional linear expression for each predetermined section, and forms them in a broken line shape. It is designed to form a connected lane model. 25A shows a lane model on the ZX plane, that is, a horizontal shape model, and FIG. 25B shows a lane model on the ZY plane, that is, a road height model.

  Specifically, the lane detection means 15 divides the real space ahead of the host vehicle into sections such as a distance Z7 from the position of the host vehicle, and the positions (X, Y, Z), the lane candidate points in each section are linearly approximated by the least square method, and parameters aL, bL, aR, bR, cL, dL in the following formulas (12) to (15) for each section: A lane model is formed by calculating cR and dR.

[Horizontal shape model]
Left lane X = aL · Z + bL (12)
Right lane X = aR ・ Z + bR (13)
[Road height model]
Left lane Y = cL · Z + dL (14)
Right lane Y = cR · Z + dR (15)

  The lane detection means 15 forms a lane model in this way and detects a road surface in real space. The lane detection means 15 stores the lane model thus formed, that is, the calculated parameters aL to dR of each section, in the storage means.

  Also in the road recognition device 10 according to the present embodiment, the integration means 6, the calculation means 7, the area calculation means 8, and the road surface area detection means 9 are the same in the present embodiment as in the case of the first embodiment. Each process is performed in accordance with the flowcharts shown in FIGS.

  The road recognition device 10 according to the present embodiment further has the following functions as a result of providing the lane detection means 15 as described above.

  First, the area calculation means 8 sets the check area R in the reference image T as an area having the left and right ends of the lanes LL and LR detected by the lane detection means 15 to the side of the host vehicle. Yes.

  Specifically, the area calculation means 8 determines the lanes LL and LR in the current sampling cycle from the behavior of the own vehicle based on the information on the lanes LL and LR detected in the past sampling cycle such as the previous sampling cycle. The position is estimated, and the check region R is set based on the estimated position. In the present embodiment as well, as in the first embodiment, the check region R is set based on, for example, the estimated trajectory Lest (see FIG. 4) estimated from the behavior of the host vehicle MC in the current sampling cycle. It is also possible to configure so as to.

  In the present embodiment, as described above, when the lane detection unit 15 detects the lanes LL and LR, the luminance D of the pixel p corresponding to the detected lane is voted on the histogram. Therefore, the brightness used for determining whether or not each group g integrated on the image T can be detected as a group belonging to the road surface area of the road surface by the road surface area detection means 9 using the result of voting on the histogram. The average threshold value Dave_th can be variably applied in accordance with the average value of the luminance D of the pixels p corresponding to the lane calculated by the lane detection unit 15.

  That is, the average value of the luminance D of the pixels p corresponding to the lane in the current sampling cycle can be calculated from the voting result on the histogram by the lane detection means 15. Based on the average value, for example, a value smaller than the predetermined value is set as the luminance average threshold value Dave_th, and the luminance average threshold value Dave_th is set according to the average value of the luminance D of the pixel p corresponding to the lane. Configure to be variable.

  And if comprised in this way, the group g corresponding to each label | marker marked white on the road surfaces, such as a lane, a pedestrian crossing, a stop line, a number, a character, and an arrow, will perform the determination process of FIG.9 S30 or step S31. Even if it is not excluded in step S32, it is possible to accurately exclude from the determination target in the determination processing in step S32, and the group g having a small luminance D to be detected as the group G belonging to the road surface area of the road surface is selected as the road surface of the road surface. It becomes possible to accurately detect the group G belonging to the region.

  On the other hand, as a result of providing the distance detection means 11 as described above, the road recognition device 10 according to the present embodiment has the following functions.

  In the first embodiment described above, when the calculating means 7 calculates the horizontal width Wj for each pixel row j extending in the horizontal direction on the image T as the horizontal width in real space for each group g, The calculation was performed based on the horizontal width in the real space per pixel for each T horizontal line j (pixel row j). However, the actual road surface is not necessarily flat as shown in the road height model of FIG. 25B, and the actual space per pixel for each horizontal line j calculated on the assumption that the road surface is flat. The top width may not always be properly applied.

  However, the parallax dp for each pixel p calculated based on the principle of triangulation by the stereo matching process using the captured reference image T and the comparison image Tc using the distance detection unit 11 as in the present embodiment. If this information is used, the lateral width Wj in the real space for each pixel row j of the group g can be accurately calculated as follows.

  That is, the parallax dpleft at the leftmost pixel pleft of the pixel row j of group g is substituted into the above equation (11) to calculate the distance Zleft in the real space of the leftmost pixel pleft, and the reference image of the leftmost pixel p By substituting the coordinate ileft on T into the above equation (9), the X coordinate Xleft of the point Pleft on the real space corresponding to the leftmost pixel pleft can be calculated.

  Similarly, from the parallax dpright at the rightmost pixel pright of the pixel row j of the group g and the coordinate iright on the reference image T, the point Pright in the real space corresponding to the rightmost pixel pright according to the above equations (9) and (11). The X coordinate Xright of can be calculated. Then, by calculating | Xleft−Xright |, the horizontal width Wj in the real space for each pixel row j of the group g can be calculated with high accuracy.

  Therefore, based on the horizontal width Wj in the real space for each pixel row j of the group g calculated accurately as described above, the calculating means 7 is equal to or larger than the horizontal width threshold Wth set to 60 cm, for example. It is possible to accurately determine whether or not the group g should be detected as the group G belonging to the road surface area of the road surface.

  In the road recognition device 1, the road surface area of the road surface is detected based on the group g integrated on the image T by the integration unit 6, but the captured image T is not a target for detection of a three-dimensional object or the like. The imaging target is also imaged.

  Therefore, at the stage where the integration unit 6 performs integration processing based on the luminance D of each pixel p of the image T, the imaging target existing on the road surface including the road surface area and the imaging target existing above the road surface such as a three-dimensional object. If the integration process is performed so that the three-dimensional object is separated in advance, it is possible to accurately prevent a three-dimensional object or the like from being erroneously detected as a road surface area of the road surface. And if it uses the detection result by the distance detection means 11 in this embodiment, it will be realizable.

  Specifically, when the integrating unit 6 determines that the input pixel pi, j and the pixel p adjacent to the input pixel pi, j satisfy the above condition 1 and condition 2, or the condition 3 and condition 4, the input pixel pi, j And adjacent pixels p corresponding to the pixels p according to the above equations (10) and (11) based on the parallax dp of the pixels p and the coordinates (i, j) on the reference image T. The height Y of the point P in the real space from the road surface is calculated.

The road surface height Y ^* at the distance Z in the real space of the input pixel pi, j is the lane model (road height expressed by the above equations (14) and (15) detected by the lane detection means 15. The height YY ^* of each point from the road surface is calculated by linear interpolation from the model. The integration means 6 then calculates the height YY ^* of the point P in the real space corresponding to the input pixel pi, j from the road surface and the road surface of the point P in the real space corresponding to the adjacent pixel p. of height Y-Y ^* from, one is the first high threshold Yth1 more predetermined, when the other is less than the first height threshold Yth1 the input pixel pi, pixels adjacent to the j p Are not integrated into one group, but are combined into another group.

  By configuring in this way, it becomes possible to integrate the imaging target existing on the road surface including the road surface area and the imaging target existing above the road surface such as a three-dimensional object into different groups. Can be accurately prevented as a road surface area of the road surface.

  Note that the first height threshold Yth1 is set to a value such as 10 cm, for example, that can accurately separate the road surface and the imaging target existing above the road surface. Further, if the lane model at the current sampling cycle is detected by the lane detection means 15, it can be used. If the lane model at the current sampling cycle is not detected, it is detected at the previous sampling cycle. Based on the lane model, the lane model in the current sampling period is estimated and used from the behavior of the host vehicle thereafter.

The height YY ^* from the road surface in the real space may be used to determine whether or not the road surface area detection means 9 detects as a group belonging to the road surface area of the road surface. In this case, for example, the road surface area detection unit 9 determines the parallax dp and the coordinates on the reference image T for each pixel p in which the parallax dp is effectively detected among the pixels p belonging to the group g. Based on (i, j), the height Y from the road surface of each point P in the real space corresponding to each pixel p is calculated according to the above equations (10) and (11).

Then, an average value Yave of the height Y from the road surface of each point P in the real space corresponding to each pixel p is calculated, and the average value Yave of the height Y, the above formulas (14) and (15) From the road surface of the group g by comparing the height Y ^* of the road surface at the position in the real space of the group g obtained by linear interpolation or the like from the lane model (road height model) represented by the equation When the height Yave-Y ^* in the real space is equal to or greater than a predetermined second height threshold Yth2, the road surface is determined as the group g being an imaging target such as a three-dimensional object existing above the road surface. Are excluded from detection of the road surface area.

  By configuring in this way, the imaging target existing above the road surface such as a three-dimensional object is excluded, and only the imaging target existing on the road surface including the road surface region is set as the detection target of the road surface region of the road surface. Thus, it is possible to accurately prevent a three-dimensional object or the like from being erroneously detected as a road surface area of the road surface.

1, 10 Road recognition device 2 Imaging means 2a, 2b Main camera, sub camera (a pair of cameras)
6 Integration means 7 Calculation means 8 Area calculation means 9 Road surface area detection means 11 Distance detection means 15 Lane detection means A Area An Number of pixels occupied by the group in the area Ath Area threshold D, Di, j Brightness Dave Average value Dave_th Luminance average threshold F Condition establishment flag (information indicating that the condition has been established)
G Group G belonging to road surface area G Group G1 to G4 Road surface area j of road surface Pixel row Lest Estimated trajectory LL, LR Lane MC Own vehicle Nave Average number of pixels occupied by width of each pixel row Nave_th Average value threshold Nj Number of pixels occupied by the width of each pixel row P Point in real space Pixel pi, j Input pixel (one pixel)
R check area (area)
T Image Wave Width average value Wj Width Wth Width threshold YY ^* Height from the road surface Yth1 First height threshold Yth2 Second height threshold Z Distance in real space ΔD Difference δD Difference

Claims (8)

A road recognition device that processes an image captured by an imaging unit and detects a road surface area of a road surface from the image,
The one pixel and the adjacent pixel based on the difference between the luminance of the one pixel with respect to the luminance of the pixel adjacent to the one pixel in the image and the average value of the luminance of the group to which the adjacent pixel belongs An integration means to integrate
For the group integrated by the integration unit, a calculation unit that calculates an average value of the horizontal width of each pixel row in the horizontal direction of the image and the average value of the luminance of the pixels belonging to the group;
An area calculating means for calculating an area occupied by the group in a region set in the traveling direction of the host vehicle in the image;
There is a pixel row that is equal to or greater than a predetermined width threshold in the pixel row of the group, an average value of the width for each pixel row of the group is equal to or greater than a predetermined average value threshold, and each pixel belonging to the group If the average luminance value is equal to or less than a predetermined luminance average threshold value and the area of the group calculated by the area calculating means is equal to or greater than a predetermined area threshold value, the group belongs to the road surface area of the road surface. Road surface area detecting means for detecting as a group;
A road recognition device comprising:   The road recognition apparatus according to claim 1, wherein the luminance average threshold is variable according to a shutter level of the imaging unit. Lane detection means for detecting a lane present on the side of the host vehicle from the image,
The lane detecting means calculates an average value of luminance of pixels in which the detected lane is imaged,
The road recognition apparatus according to claim 1 or 2, wherein the luminance average threshold is varied according to an average luminance value of pixels in which the lane is imaged. Lane detection means for detecting a lane present on the side of the host vehicle from the image,
The region set in the traveling direction of the host vehicle in the image is a lane estimated in the current sampling cycle based on the lane information detected in the past sampling cycle by the lane detecting unit and the behavior of the host vehicle. The road recognition device according to any one of claims 1 to 3, wherein the road recognition device is set on the basis of the position.   The region set in the traveling direction of the host vehicle in the image is set to a range from an estimated trajectory estimated from the behavior of the host vehicle to a position spaced a predetermined distance in the lateral direction. The road recognition device according to any one of claims 3 to 4. When the integration unit integrates the one pixel and the adjacent pixel into one group, the horizontal width in the real space in the pixel row of the group is equal to or greater than the horizontal width threshold. In addition, the information that the condition is satisfied is associated with the group,
The road surface area detecting means determines whether or not there is a pixel row in which the horizontal width of each pixel row of the group is equal to or larger than a predetermined horizontal width threshold, and whether or not the information is associated with the group. The road recognition apparatus according to claim 1, wherein the road recognition apparatus performs the determination. A distance detecting means for detecting a distance in real space of each pixel in the image;
The integrating means includes a height from the road surface of a point in the real space corresponding to the one pixel calculated based on a distance in the real space of the one pixel and an actual pixel corresponding to the adjacent pixel. Among the heights of the points on the space from the road surface, when one is not less than a predetermined first height threshold and the other is less than the first height threshold, the one pixel and the adjacent The road recognition device according to any one of claims 1 to 6, wherein a pixel to be integrated is not integrated. A distance detecting means for detecting a distance in real space of the pixel in the image;
When the height of the group in the real space from the road surface of the group is greater than or equal to a predetermined second height threshold based on the distance in the real space of the pixels belonging to the group The road recognition apparatus according to any one of claims 1 to 7, wherein the group is excluded from a target of detection of a road surface area of the road surface.