JP6126849B2 - Lane identification device and lane identification method - Google Patents

Description

  The present invention relates to a technique for identifying a road lane based on an image captured by an in-vehicle camera.

  Conventionally, a technique for identifying a road lane based on an image captured by an in-vehicle camera has been developed. For example, in Patent Document 1, an image obtained by capturing an image of a road surface by an in-vehicle camera is converted into an overhead image. Then, there is disclosed a technique for detecting the position of a white line by integrating the luminance of the overhead view image in parallel with the vehicle traveling direction to obtain a luminance profile with respect to the direction perpendicular to the traveling direction and filtering the luminance profile. Yes.

Japanese Patent No. 4927908

  However, a large hardware configuration is required to convert an image into a bird's-eye view image, and when a white line is detected by a luminance profile, if a vehicle or a road marking (such as a zebra zone) is on the road, they are disturbed. As a result, the detection accuracy of the white line is lowered, which causes false detection. In particular, since the zebra zone is white, erroneous detection becomes very frequent.

  The present invention has been made to solve the above-described problems, and provides a lane identification technique that requires high lane detection accuracy and a small circuit scale even when there is a disturbance factor on the road. The purpose is to do.

A first aspect of a lane identification device according to the present invention is a lane identification device that identifies a lane on a road based on an image captured by an in-vehicle camera, and that includes image data for one frame of the image as a plurality of unit blocks. A complexity detection unit that detects whether the complexity of the image that is a disturbance factor when identifying the lane from the image for each unit block is greater than or equal to a predetermined value, and among the plurality of unit blocks A mask processing unit that performs mask processing on a unit block having the image complexity equal to or higher than the predetermined value, and a straight line that defines the lane for the unit block that is not the target of the mask processing among the plurality of unit blocks. a line detection unit which detects an approximately, the detected straight lines for each of the unit blocks, by integrating the entire frame and a integration processing unit to the lane, before Complexity detecting unit, the complexity of the image, you determination by comparing the variance value and the threshold value of the brightness in the unit block.

In a second aspect of the lane identification device according to the present invention, the complexity detection unit determines the complexity of the image by comparing a variance value of color differences in the unit block with a threshold value.

A third aspect of the lane identification device according to the present invention is characterized in that the straight line detection unit performs an edge detection process on the unit block that is not a target of the mask process and constitutes a detected edge. A straight line defining the lane is approximately detected by performing a Hough transform on.

A lane identification method according to the present invention is a lane identification method for identifying a lane of a road based on an image captured by an in-vehicle camera, wherein image data for one frame of the image is divided into a plurality of unit blocks. (A) performing complexity detection for detecting whether or not the complexity of the image, which becomes a disturbance factor when identifying the lane from the image for each block, is greater than or equal to a predetermined value, and among the plurality of unit blocks A step (b) of performing mask processing on a unit block having a complexity of the image equal to or greater than the predetermined value, and defining the lane for the unit block that is not the target of the mask processing among the plurality of unit blocks. Step (c) for performing a straight line detection process for detecting a straight line approximately, and the straight line detected for each unit block are integrated in the entire frame to form the vehicle. And a step (d) performing the integration process to the step (a), the complexity of the image, comprising the step of determining by comparing the variance value and the threshold value of the brightness in the unit block Yes.

  According to the present invention, a mask process is performed on a unit block including a disturbance factor, and a disturbance line is removed by approximately detecting a straight line that defines a lane for a unit block that is not subject to a mask process. The accuracy of lane detection can be increased.

It is a block diagram which shows the structure of the lane identification device of embodiment which concerns on this invention. It is a figure which illustrates the read-out order of image data typically. It is a figure which shows an example of the image acquired with the camera. It is a figure which shows an example at the time of dividing a picked-up image into a some block. It is a flowchart explaining a lane identification operation | movement. It is a figure explaining the concept of Hough conversion. It is a flowchart explaining the modification of lane identification operation | movement.

<Embodiment>
<Device configuration>
FIG. 1 is a block diagram showing a configuration of a lane identification device 10 according to an embodiment of the present invention. As shown in FIG. 1, the lane identification device 10 includes a camera 11 that digitally processes an image captured by an imaging device such as a CCD (charge coupled device), an image processing unit 12, a frame memory 13, and an information processing unit. 14 and an intermediate memory 15.

  The camera 11 is provided in a front portion of the vehicle (for example, in the vicinity of a rearview mirror (a rearview mirror in the vehicle interior)), and can photograph a road around the vehicle centering on the front in the traveling direction.

  The image processing unit 12 is a part that performs various types of image processing on the image data of the image acquired by the camera 11, and as image processing executed by the image processing unit 12, for example, an insufficient color component is interpolated. There are pixel interpolation processing (Bayer interpolation) to be obtained, color space conversion processing for converting the color space of image data, and the like.

  The frame memory 13 is a memory for temporarily storing image data after image processing output from the image processing unit 12 in units of frames.

  The information processing unit 14 is configured by a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like, and reads out a program stored in the ROM and executes the program by the CPU. The variance calculation unit 141, the mask processing unit 142, the straight line detection unit 143, and the integration processing unit 144 are functionally realized. Each function realized in the information processing unit 14 may be realized by a hardware circuit.

  When reading the image data for one frame from the frame memory 13, the variance calculation unit 141 divides the image data into a plurality of blocks and reads them in units of blocks, and calculates the variance of at least one of luminance and color difference in the unit blocks. Processing to calculate is performed.

  FIG. 2 is a diagram for schematically explaining the reading order of image data. As shown in FIG. 2, image data for one frame is divided into a plurality of blocks BK in the horizontal direction (X direction) and the vertical direction (Y direction). Read out. For example, image data is read out in block units from the block BK at the uppermost left end as shown in the figure. When the processing of the block is completed, the block on the right is read out. When reading is sequentially performed and the right end is reached, the block at the left end of the next stage is read out.

  The reading order of image data in each block is such that reading is performed from the data at the left end of the uppermost column of the block, and when the right end is reached, reading is performed from the data at the left end of the next column.

  The mask processing unit 142 masks the image data of the highly dispersed block so as not to be used in the subsequent processing based on at least one of the luminance and color difference variances in the unit block calculated by the variance calculating unit 141. For example, processing such as changing all image data of the block to predetermined unintentional data is performed.

  The straight line detection unit 143 performs edge detection processing on the blocks that are not masked by the mask processing unit 142, and performs straight line detection processing on the detected edges. A plurality of lines (for example, white lines) that define road lanes are detected by the edge detection process, and a plurality of straight lines along each of the plurality of lines are approximately detected by the straight line detection process. Each process in the straight line detection unit 143 is performed while storing the data being processed in the intermediate memory 15.

  As the straight line detection processing, for example, Hough transformation can be adopted, but other methods may be adopted. Note that as a straight line detection process other than the Hough transform, a method using the least square method and the like can be mentioned.

  The integration processing unit 144 is a part that allows straight lines detected in units of blocks to be integrated over the entire frame and displayed as a lane.

<Operation>
Next, a lane identification operation in the lane identification device 10 will be described with reference to FIGS. FIG. 3 shows an example of an image acquired by the camera 11 (FIG. 1).

  In FIG. 3, vehicles OB1 and OB2 run along two lanes ahead of the host vehicle, and the white lines WL1 and WL2 drawn on the road shoulder are hidden by the vehicles OB1 and OB2 and are difficult to see. A soundproof wall OB3 is provided along the road on the vehicle OB1 side, and a guard rail OB4 is provided along the road on the vehicle OB2 side.

  In this way, the lane to be detected may be hidden behind the preceding vehicle and may not be visible, and if there is a soundproof wall OB3 or guardrail OB4 near the lane, they become disturbance factors when detecting the lane. , They can occur anywhere.

  FIG. 4 shows an example when the captured image is divided into a plurality of blocks. FIG. 4 shows an example in which the photographed image is divided into 5 × 5 blocks. In the figure, the upper left block BK is numbered as block A1, and the rightward direction is A2, A3, A4, A5. In the next stage, A6, A7 ... are assigned from the left end.

  In FIG. 4, only the road surface is displayed in the blocks A21 to A25. However, the vehicle, the soundproof wall, and the guardrail are displayed above the image, that is, as the distance from the host vehicle increases, and the complexity of the image is increased. It turns out to be high. In other words, it can be said that the area where the lane is displayed is relatively low in complexity.

  And if the complexity of the image becomes high, it becomes a disturbance factor when detecting the lane, and the accuracy of the lane detection decreases, so if there is a feature quantity that can quantitatively calculate the complexity of the image, By excluding a region with a high degree of complexity of the image from the lane detection target based on the complexity, the disturbance factor can be removed and the accuracy of the lane detection can be improved.

  For example, road surfaces are displayed in blocks A16 to A25 and white lines WL1 to WL3 are displayed. Of these, vehicles OB1 and OB2 are also displayed in blocks A17 to A19, which is considered to be a high complexity area. .

  Here, a spatial frequency is cited as a feature quantity that can quantitatively calculate the complexity of an image, and an area with a higher spatial frequency can be defined as an area with a higher image complexity. Therefore, it is possible to suppress the disturbance by defining the complexity of the image using the spatial frequency and excluding the high complexity area from the lane detection target, but the disturbance factor to be removed is, for example, the preceding vehicle Since it occupies a relatively large area, such as a soundproof wall, a relatively large block size of 32 × 32 or 64 × 64 is required to remove disturbance in units of blocks. On the other hand, calculating the spatial frequency requires frequency conversion to convert the image data into the frequency domain. However, when performing frequency conversion in units of blocks, the amount of computation is extremely large with a large block size as described above. turn into.

  Here, as an example of frequency conversion, a general DCT (Discrete Cosine Transform) is shown in the following formula (1).

  In order to perform DCT according to the above equation (1), multiplication including a decimal component is required, and the amount of calculation becomes relatively large.

  Therefore, in the present invention, luminance value dispersion is used as a feature value for quantitatively calculating the complexity of an image. In addition, color difference dispersion is used as a feature value for quantitatively calculating the color diversity.

  Here, the formula for calculating the luminance and the color difference is shown in the following formula (2).

  In the above equation (2), R represents red, G represents green, B represents blue, Y represents luminance, Cb represents a blue color difference component, and Cr represents a red color difference component. From the image data, R, G for each pixel is represented. By acquiring the B and B color data, it is possible to calculate the luminance and the color difference for each pixel.

  The dispersion of the luminance thus obtained within the unit block can be calculated by the following mathematical formula (3).

The above mathematical formula (3) is an expression representing a general sample variance, p _i represents a luminance value, and P _mean represents an average value of the luminance values.

  It can be said that the luminance value variation with the high luminance value distribution calculated from Equation (3) is large and the image complexity is high, and the luminance value variation is small with the low luminance value block. It can be said that the complexity of the image is low. As a result, the complexity of the image can be easily and quantitatively calculated. In addition, since the complexity is quantitatively calculated by the distribution of luminance values, there is an effect that the amount of calculation can be reduced as compared with the frequency conversion when calculating the complexity of the image using the spatial frequency.

Further, the variance of the color difference within the unit block can also be expressed by using a general sample variance as in the above formula (3). In this case, p _i is the color difference value, and P _mean is the average of the color difference values. Represents a value.

  A block having a high variance of color difference values calculated from Equation (3) has a variety of colors and can be said to be a complex image. Therefore, it can be said that the complexity of the image is high, and a block having a low variance of color difference values. It can be said that there are few color types and the complexity of the image is low. As a result, the complexity of the image due to color diversity can be easily and quantitatively calculated.

  There are two types of color difference components, Cb and Cr, but Cb and Cr may be used, or (Cb-Cr) or (Cb + Cr) may be used.

  Next, a lane identification operation in the lane identification device 10 will be described using the flowchart shown in FIG. 5 with reference to FIG. When the lane identification operation starts, as shown in FIG. 5, in step S <b> 1, first, image data with a road in the traveling direction of the vehicle as a subject is acquired by the camera 11. The acquired image data is subjected to predetermined image processing by the image processing unit 12.

  The image data that has been subjected to image processing is temporarily stored in the frame memory 12, and then read out for each unit block as described with reference to FIG. 4, and is provided to the variance calculation unit 141 of the information processing unit 14 (step). S2).

  In the variance calculation unit 141, after obtaining R, G, and B color data for each pixel from the image data using Equation (2), the luminance is calculated for each pixel using Equation (3). The variance of luminance within the unit block is calculated (step S3).

  Then, it is determined whether or not the calculated luminance dispersion value is equal to or greater than a predetermined threshold value (for example, a pixel average value such as a unique and significant value for each frame) (step S4). Determines that the unit block has high image complexity and includes a disturbance factor, and proceeds to step S5.

  In step S <b> 5, the mask processing unit 142 performs mask processing so that a block having a high image complexity and including a disturbance factor is not used in subsequent processing.

  On the other hand, if it is determined in step S4 that the calculated luminance variance value is less than the threshold value, the process proceeds to step S11, where the color difference is calculated for each pixel using Equation (3), Calculate the variance of the color difference.

  Then, it is determined whether or not the calculated color difference variance value is equal to or greater than a predetermined threshold value (a constant value regardless of the frame) (step S12). It is determined that the block has a complex color, has a high degree of image complexity, and includes a disturbance factor, and proceeds to step S5. That is, while the lane and the road surface are poor in color, it is determined that blocks having various colors include disturbance factors rather than the lane and the road surface.

  For example, in FIG. 4, since the block A20 only includes the soundproof wall OB3 in addition to the white line WL1, it can be said that the complexity of the image is low, but the soundproof wall OB3 is colored or an advertising signboard is provided. In other words, the block has various colors, the image has a high degree of complexity, and includes disturbance factors.

  As described above, the variance calculation unit 141 detects whether the image of the unit block is complex by calculating the variance of luminance and the variance of chrominance, so that the part that functions as the complexity detection unit It can be said that.

  In FIG. 4, the blocks A1 to A5 and A6 to A10 are areas for displaying the sky and the background, but it is clear that these areas do not include lanes. By setting a block as a default area in which variance calculation processing and straight line detection processing are not performed, unnecessary processing can be omitted and lane identification operation can be speeded up.

  In step S <b> 5, the mask processing unit 142 performs mask processing so that the block with high image complexity is not used in the subsequent processing.

  On the other hand, if it is determined in step S12 that the calculated color difference variance is less than the threshold value, the process proceeds to step S13 to perform straight line detection.

  In step S13, the line detection unit 143 performs edge detection processing on the image in the unit block, and performs line detection processing on the detected edge. Hough transform can be used as the straight line detection process.

  FIG. 6 is a diagram for explaining the concept of the Hough transform. As shown in FIG. 6, when there are feature points P0, P1, P2, P3, and P4 constituting the detected edge, in the Hough transform, a predetermined value from 0 ° to 180 ° is set around each of the feature points P0 to P4. The straight line is determined so as to change with the angle. FIG. 6 shows a plurality of straight lines passing through the feature point P0. In this way, a plurality of straight lines are generated for each of a plurality of feature points on a plane, and a common straight line is determined, thereby defining an edge, that is, an edge of a lane.

  In the case of a method of determining a straight line by voting such as Hough transform, since the acquisition order of feature points does not affect the detection performance, there is no problem even if processing proceeds in units of blocks, and the image is divided into multiple blocks. It can be said that this is a method suitable for the present invention in which the detection accuracy is improved by not making the block including the disturbance factor the target of the straight line detection process.

  The processes in steps S2 to S5 and S11 to S13 described above are repeated until they are executed for all unit blocks in one frame (step S6). After the processes for all unit blocks are completed, step S7 is performed. Proceed to

  In step S7, the integration processing unit 144 integrates straight lines that are highly likely to exist among the straight lines detected in the blocks that are not subject to mask processing, and finally performs processing so as to be displayed as lanes.

  In this processing, for example, when straight lines corresponding to the white lines WL1, WL2, and WL3 are detected from the blocks A16, A20, A21, A22, and A25 in FIG. 4, the white line WL3 can be displayed as a lane, but the white line WL1 And WL2 cannot be displayed because blocks A12 and A19 are masked. Therefore, processing for generating white line data by a method such as extrapolating the white lines WL1 and WL2 in the areas corresponding to the blocks A12 and A19 with respect to the white lines WL1 and WL2 obtained in the blocks A16, A20, and A25 is included. It is out.

  After the straight line integration process is completed, the input of the next frame data from the camera 11 is received (step S8), and the processes after step S1 are repeated.

  In the above description, the straight line detection unit 143 performs straight line detection processing. However, a lane in a road curve can also be detected by approximating it with a straight line.

  According to the lane identification device 10 of the embodiment described above, the complexity of the image, which is a disturbance factor when detecting the lane, is calculated by the distribution of luminance and color difference, and the calculated complexity of the image is calculated. Based on the above, it is possible to remove the disturbance factor and improve the accuracy of the lane detection by excluding the area with high image complexity from the lane detection target. Also, by calculating the complexity of the image not only for the luminance but also for the color difference, it is possible to determine the complexity of the image from the viewpoint of color diversity, and to improve detection accuracy without increasing the amount of computation. Can be raised. Further, when detecting the lane, it is not necessary to convert the image into a bird's-eye view image, so that the hardware configuration such as the circuit scale does not increase.

<Modification>
In the embodiment described above, the configuration in which the luminance and chrominance variance in a unit block is calculated to identify a block having a high image complexity and including a disturbance factor has been described. It is good also as a structure which specifies the block in which a disturbance factor is included only by dispersion | distribution of a brightness | luminance, without calculating. As a result, the amount of calculation can be reduced as compared to the case of calculating the variance of the color difference.

  Hereinafter, a lane identification operation in a case where a block having a high image complexity and including a disturbance factor is specified only by luminance distribution will be described with reference to a flowchart shown in FIG. In FIG. 7, the operations in steps S1 to S4 are the same as the operations in steps S1 to S4 shown in FIG.

  In step S4, it is determined whether or not the calculated luminance dispersion value is equal to or greater than a predetermined threshold value (for example, a pixel average value such as a unique and significant value for each frame). The unit block is determined to be a block having a high image complexity and a disturbance factor, and the process proceeds to step S5.

  In step S <b> 5, the mask processing unit 142 performs mask processing so that a block having a high image complexity and including a disturbance factor is not used in subsequent processing.

  On the other hand, if it is determined in step S4 that the calculated luminance variance is less than the threshold value, the process proceeds to step S9 to perform straight line detection.

  In step S9, the line detection unit 143 performs edge detection processing on the image in the unit block, and performs line detection processing on the detected edge. Hough transform can be used as the straight line detection process.

  The processes in steps S2 to S5 and S9 described above are repeated until all the unit blocks in one frame are executed (step S6). After the processes for all the unit blocks are completed, the process proceeds to step S7. .

  In step S7, the integration processing unit 144 integrates straight lines that are highly likely to exist among the straight lines detected in the blocks that are not subject to mask processing, and finally performs processing so as to be displayed as lanes.

  After the straight line integration process is completed, the input of the next frame data from the camera 11 is received (step S8), and the processes after step S1 are repeated.

  Further, in the above-described modification, the configuration in which the luminance variance in the unit block is calculated to identify a block having a high image complexity and including a disturbance factor is shown. It is good also as a structure which specifies the block where the complexity of is high and a disturbance factor is contained. In that case, in the flowchart shown in FIG. 7, the variance of the color difference is calculated in step S3, and it is determined whether or not the variance value of the color difference is greater than or equal to the threshold value in step S4.

  In the present invention, the embodiments can be appropriately modified and omitted within the scope of the invention.

DESCRIPTION OF SYMBOLS 10 Lane identification device 141 Variance calculation part 142 Mask process part 143 Straight line detection part 144 Integrated process part

Claims (4)

A lane identification device for identifying a road lane based on an image captured by an in-vehicle camera,
The image data for one frame of the image is divided into a plurality of unit blocks, and it is detected whether the complexity of the image, which becomes a disturbance factor when identifying the lane from the image for each unit block, is greater than or equal to a predetermined value. A complexity detection unit,
A mask processing unit that performs mask processing on a unit block of which the complexity of the image is not less than the predetermined value among the plurality of unit blocks;
A straight line detection unit that approximately detects a straight line that defines the lane for the unit block that is not the target of the mask processing among the plurality of unit blocks;
The detected straight line for each of said unit blocks, by integrating the entire frame and a integration processing unit to the lane,
The complexity detector
The complexity of the image, the lane identifying unit you determined by comparing the variance value and the threshold value of the brightness in the unit block. The complexity detector
The lane identification device according to claim 1, wherein the degree of complexity of the image is determined by comparing a dispersion value of a color difference in the unit block with a threshold value. The straight line detector
An edge detection process is performed on the unit block that is not subject to the mask process, and a Hough transform is performed on a plurality of feature points that constitute the detected edge, thereby approximating a straight line that defines the lane. detected, the lane identification apparatus according to claim 1. A lane identification method for identifying a road lane based on an image captured by an in-vehicle camera,
(A) Whether the image data for one frame of the image is divided into a plurality of unit blocks, and the complexity of the image, which becomes a disturbance factor when identifying the lane from the image for each unit block, is greater than or equal to a predetermined value Performing complexity detection to detect if;
(B) performing a mask process on a unit block of which the complexity of the image is not less than the predetermined value among the plurality of unit blocks;
(C) performing a straight line detection process of approximately detecting a straight line defining the lane for the unit block that is not the target of the mask process among the plurality of unit blocks;
(D) the detected straight line for each of said unit blocks, by integrating the entire frame and a step of performing integration processing to the lane,
The step (a)
A lane identification method including a step of determining the complexity of the image by comparing a luminance dispersion value in the unit block with a threshold value .

Images