JP7390899B2 - lane mark recognition device - Google Patents

Description

TECHNICAL FIELD The present invention relates to a lane mark recognition device, for example lane mark recognition for a sensing system such as a stereo camera system.

In recent years, with the spread of in-vehicle camera devices, there has been an increasing demand for various recognition functions for safe driving and autonomous driving. Among these, stereo camera devices simultaneously measure visual information from images and distance information to objects in the image, so they can detect various objects around the vehicle (people, cars, three-dimensional objects, road surfaces, etc.). The system is said to be able to grasp detailed information on road signs, signboards, etc., and contribute to improving safety during driving support. For example, driving support functions include LKA (Lane Keeping Assist), which recognizes lane marks on the road and supports driving in the center of the lane, and lane mark recognition technology is needed to improve the performance of such functions. demand is increasing.

2. Description of the Related Art Various techniques and devices have been proposed in the past regarding in-vehicle camera devices that are mounted on a vehicle and that recognize the situation in front of the vehicle. For example, with regard to improving the performance of lane mark recognition, Patent Document 1 is cited as a technique for accurately recognizing lane marks even when the state of light irradiation on the road surface is partially different.

Japanese Patent Application Publication No. 2007-26106

In the conventional technology described in Patent Document 1, the color component of a first pixel at the leading end in the moving direction of an area including multiple pixels is corrected based on the luminance value of a second pixel included in the area together with the first pixel. Then, by calculating the feature amount of the first pixel according to the color of interest based on the corrected color component of the first pixel, the lane mark of the color of interest is recognized in the road surface image.

However, in the above-mentioned conventional technology, the color of the road surface is not sufficiently reflected in the color component of the first pixel due to the influence of shadows and light on the road surface, and the feature amount corresponding to the color of interest in the first pixel is suppressed. , there is a risk that the recognition accuracy of the lane mark of the color of interest may be reduced.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a lane mark recognition device that can improve the recognition accuracy of lane marks of a specific color.

In order to solve the above-mentioned problems, a lane mark recognition device according to the present invention is a lane mark recognition device that recognizes lane marks on a road surface using a parallax image obtained from an image acquired by a camera device. Information on the lane mark is acquired from the image, a proportion of the road surface area estimated based on the information on the lane mark is occupied by the first color component, and based on the proportion of the first color component, the The present invention is characterized by changing the threshold value for recognizing the second color component of the lane mark.

According to the present invention, lane marks of a specific color can be recognized accurately or more reliably even at night, for example. For example, it is possible to reduce the possibility that a white line, which is a white lane mark, will be recognized as a yellow line, which is a yellow lane mark, and it is possible to accurately distinguish between a white line and a yellow line even at night. This makes it possible to control the vehicle based on a lane mark of a specific color from a road surface with lane marks of a plurality of colors, even at night, for example. For example, in a construction zone in Europe, it becomes possible to perform lane maintenance control that accurately follows lanes that have been temporarily divided into lanes.

Problems, configurations, and effects other than those described above will be made clear by the description of the embodiments below.

1 is a diagram illustrating an installation outline of an in-vehicle stereo camera system according to an embodiment of the present invention. FIG. 1 is a block diagram showing an internal configuration (functional block configuration) of an in-vehicle stereo camera system according to an embodiment of the present invention. FIG. 2 is a block diagram showing the internal configuration (functional block configuration) of the image signal processing unit of the present embodiment. FIG. 3 is a diagram showing an example of a layer arrangement of RCCC and RGGB. FIG. 3 is a diagram showing color distribution by R and C in RCCC. FIG. 3 is a diagram showing an overview of parallax processing. It is a figure explaining a road surface image. FIG. 3 is a diagram illustrating changes in the yellow component of a specific portion due to the influence of light. FIG. 3 is a diagram illustrating fluctuations in a threshold value for yellow line determination.

Embodiments of the present invention will be described below with reference to the drawings. Note that in each figure, parts having the same functions may be designated by the same reference numerals and repeated explanations may be omitted.

FIG. 1 is a diagram showing an installation outline of a vehicle-mounted stereo camera system 100 including a vehicle-mounted stereo camera device 101 according to an embodiment of the present invention.

In the vehicle-mounted stereo camera system 100 of the illustrated embodiment, the vehicle-mounted stereo camera device 101 is installed in front of the windshield at the upper front of the vehicle. In the in-vehicle stereo camera system 100, the in-vehicle stereo camera device 101 is connected to an in-vehicle network through an image signal processing unit 102 and a control unit 103, and controls an accelerator 105, a brake 106, a steering wheel 104, and the like. The in-vehicle stereo camera system 100 has a configuration in which these (101 to 106) are mounted on a vehicle 107. In addition, the image signal processing unit 102 and the control unit 103 may be omitted as the in-vehicle stereo camera device 101 itself may also perform the processing.

FIG. 2 is a block diagram showing the internal configuration (functional block configuration) of the in-vehicle stereo camera system 100 according to the embodiment of the present invention, particularly the internal configuration (functional block configuration) of the in-vehicle stereo camera device 101 and the image signal processing unit 102. It is a diagram.

The in-vehicle stereo camera device 101 has two cameras 201 and 202 (left camera 201 and right camera 202) arranged on the left and right (side by side) to acquire image information. The two cameras 201 and 202 are installed in front of the front upper windshield of the vehicle 107 so that their fields of view (imaging areas) partially overlap.

In this embodiment, the two cameras 201 and 202 constituting the in-vehicle stereo camera device 101 are installed at the front upper part of the vehicle 107 to take an image of the front of the vehicle 107 (in other words, the two cameras 201 and 202 constituting the in-vehicle stereo camera device 101 are (information)), but it goes without saying that images of the rear and sides of the vehicle 107 may also be captured.

The image signal processing unit 102 has an image input interface 203 for controlling imaging by the cameras 201 and 202 and importing the captured images. The image captured through the image input interface 203 is sent through the bus 209 and processed by the image processing unit 204 and the arithmetic processing unit 205, and the intermediate processing results and final result image data are stored in the storage unit 206. be remembered.

The image processing unit 204 compares the first image (left image) obtained from the image sensor of the camera 201 and the second image (right image) obtained from the image sensor of the camera 202, and adds an image to each image. On the other hand, image correction such as correction of device-specific deviation caused by the image sensor and noise interpolation is performed, and the correction is stored in the storage unit 206. In this example, the RCCC layer is used, which can minimize the decrease in spatial resolution due to image processing correction, and is superior to the conventional RGGB layer in situations where light sources are scarce, such as at night, due to its high sensitivity ( (See Figure 4). Furthermore, the image processing unit 204 calculates mutually corresponding locations between the first and second images, calculates parallax information (parallax image), and stores this in the storage unit 206 as before. Remember.

The arithmetic processing unit 205 uses the image and parallax information (distance information for each point on the image) stored in the storage unit 206, and uses the recognition necessary to perceive the environment around the vehicle 107 (for example, lane marks, walking distance information). persons, vehicles, other three-dimensional objects, signals, signs, etc.). A portion of these recognition results and intermediate calculation results are recorded in the storage unit 206 as before.

After performing the above-described recognition on the captured image, the control processing unit 208 calculates a control policy for the vehicle 107 using these recognition results.

Part of the control policy for the vehicle 107 obtained as a result of calculation and the recognition results such as lane marks are transmitted to the external in-vehicle network CAN 210 through the CAN interface 207, thereby controlling the braking of the vehicle 107 via the control unit 103. (Control of accelerator 105, brake 106, steering 104, etc.) is performed.

As described above, the image signal processing unit 102 recognizes various objects around the vehicle 107 using parallax information (parallax images) obtained from images captured from the cameras 201 and 202, and based on the recognition results. The control policy for the vehicle 107 is calculated, and the process of recognizing lane marks provided on the road surface will be described in detail below.

FIG. 3 is a block diagram showing the internal configuration (functional block configuration) of the image signal processing unit (lane mark recognition device) 102 of this embodiment.

The image signal processing unit 102, which is the lane mark recognition device of this embodiment, includes a left image acquisition unit 301, a right image acquisition unit 302, a parallax image generation unit 303, a lane mark It has a detection section 304, a road surface estimation section 305, a road surface color determination section 306, a lane mark color determination section 307, and a vehicle control section 308. For example, the left image acquisition unit 301 and the right image acquisition unit 302 are the image input interface 203 in FIG. The color determination unit 306 and the lane mark color determination unit 307 are incorporated in the arithmetic processing unit 205 in FIG. 2, and the vehicle control unit 308 is incorporated in the control processing unit 208 in FIG.

As described above, in this embodiment, the cameras 201 and 202 constituting the in-vehicle stereo camera device 101 have an image sensor having an RCCC layer that is superior to a conventional RGGB layer in a situation where a light source is scarce.

FIG. 4 shows an example of the arrangement of the RCCC layer and the RGGB layer. Further, FIG. 5 shows the color distribution using an R (red) element and a C (clear) element in RCCC. In FIG. 5, the vertical axis represents the ratio of R (red) and C (clear) in RCCC, and the horizontal axis represents the brightness value of C (clear) in RCCC. As shown in FIG. 5, it is possible to extract and determine specific color components in images acquired by cameras 201 and 202 from the ratio of R (red) and C (clear) and the brightness value of C (clear). It is.

However, in this case, as shown in FIG. 5, for example, a white line illuminated by light such as the headlights of the vehicle 107 may be extracted as a yellow component and determined to be a yellow line.

Therefore, in the image signal processing unit (lane mark recognition device) 102 of this embodiment, each of the processes described below is executed in each of the above-mentioned parts. Note that this processing assumes that the white line, which is a lane mark mainly composed of white components, and the road surface are similarly achromatic.

First, the left image acquisition unit 301 and the right image acquisition unit 302 acquire road surface images (images showing the road surface) in front of the vehicle from the left and right cameras 201 and 202, respectively (see FIG. 7).

Next, the parallax image generation unit 303 calculates the parallax by comparing the two left and right road surface images acquired by the left image acquisition unit 301 and the right image acquisition unit 302 with each other.

The process of parallax calculation will be described using FIG.

First, areas 401 and 404 are the target areas for parallax calculation in the left and right images. Here, the process of dividing the image 401 into certain small regions and searching the image 404 for a portion corresponding to the small region of interest is the process of parallax calculation. Specifically, an image 402 of the same size as the small area of interest on the image 401 is cut out from the image 404, and this cutout range is shifted horizontally as indicated by the arrow 403 to find the best matching location. This is a parallax matching process.

Next, the lane mark detection unit 304 performs lane mark recognition using the parallax information (parallax images including parallax) obtained as a result of the calculation by the parallax image generation unit 303 and the left and right road surface images or one of the road surface images. conduct. For example, the lane mark detection unit 304 extracts the brightness change portion from the left and right images or the C image of one image, and detects the edge of the lane mark. The lane mark detection unit 304 then converts the distance information of the detected lane mark portion from two-dimensional coordinates to world coordinates based on the parallax (parallax image) calculated by the parallax image generation unit 303 and registers the distance information. In addition, the lane mark detection unit 304 refers to the R image and C image of the detected lane mark portion, and based on the ratio of R and C of the pixel and the brightness value of C (see FIG. 5), the lane mark detection unit 304 uses a specific color component (for example, Register the proportion of yellow component) that occupies the lane mark part.

The road surface estimating section 305 calculates the shape of the lane mark on the road surface and the road surface slope from the lane mark information (distance information) detected (obtained) by the lane mark detecting section 304, and estimates the road surface area in the road surface image.

The road surface color determination section 306 determines the color of the pixel from the ratio of R and C of the pixel in the road surface area estimated by the road surface estimation section 305 and the brightness value of C, and determines the color of the pixel occupied by a specific color component (for example, yellow component) of the road surface area. Calculate and register the ratio.

Here, the road surface area refers to an area that is a certain distance away from the detected lane mark toward the center of the lane.

FIG. 7 is a diagram illustrating a road surface image according to this embodiment.

FIG. 7 shows a road surface image 506 obtained from one of the left and right cameras 201 and 202, with white lane marks (that is, white lines) 504 and lane marks of a specific color other than white, such as yellow lane marks (that is, yellow lines). 505 exists. In FIG. 7, a light irradiation range 503 is an area that is irradiated with light, for example, light from the headlights of the vehicle 107. The lane mark color component extraction range 501 represents the range of lane marks from which the lane mark detection unit 304 extracts the pixel color, and the road color component extraction range 502 represents the range from which the pixel color is extracted by the road color determination unit 306. It represents the range of the road surface area (estimated by the road surface estimation unit 305). As shown in FIG. 7, the road surface color component extraction range 502 is an area that is a predetermined distance away from the lane mark color component extraction range 501 toward the center of the lane (for example, a few centimeters from the inside of the lane mark color component extraction range 501). (area ranging from 10cm to 1m inside).

The lane mark color determination unit 307 sets a threshold value for determining whether the lane mark portion is a specific color based on the proportion of the color component (for example, yellow component) of the specific color registered by the road surface color determination unit 306. Here, in order to improve the determination accuracy, the average value of a plurality of frames is used as the proportion of the specific color in the road surface area. The lane mark color determination unit 307 determines whether the lane mark is of a specific color by comparing the set threshold value with the ratio occupied by the color component of the specific color registered by the lane mark detection unit 304. For example, when the proportion of yellow components in the detected road surface area is 30%, add 50% (30/2=15%) and add 45% (30+15=45%) to the yellow line judgment of the lane mark area. By using this as a threshold, the color of the lane mark is determined.

FIG. 8 shows changes (increases) in yellow components on the road surface, white lines, and yellow lines due to the influence of light such as headlights. As explained in FIG. 5, due to the influence of light such as headlights, the influence of light (influence of increase in yellow component) is added to the white line part, so the fixed threshold value 601 set in advance for yellow line determination, There is a possibility that a white line may be mistakenly recognized as a yellow line.

FIG. 9 shows fluctuations (variable settings) of the threshold value for yellow line determination by the lane mark color determination unit 307. Through the processing of the lane mark color determination unit 307 described above, the conventional threshold value 601 for yellow line determination becomes (increases) the threshold value 602 for yellow line determination based on the proportion of the road surface yellow component. Therefore, it is possible to reduce the possibility that a white line will be mistakenly recognized as a yellow line due to the influence of light such as headlights.

The magnitude of the influence of light such as headlights, streetlights, sunset light, and other environmental light (influence of increased yellow component), which is considered in threshold setting, depends on the brightness of the surrounding environment such as day and night, inside and outside of tunnels, and ambient lighting. Therefore, through the above processing of the lane mark color determination unit 307, an appropriate threshold value (for example, a threshold value for yellow line determination) is variably set according to the brightness of the surrounding environment such as day and night, inside and outside of a tunnel, and surrounding lighting. That will happen.

The vehicle control unit 308 calculates the vehicle position in the lane based on the lane mark information (distance information, color component information, etc.) detected by the lane mark detection unit 304 and the lane mark color determination unit 307, and determines the center of the lane. A control signal is generated to control the steering wheel 104 (FIG. 1) and the like so as to maintain the steering wheel 104 (FIG. 1). Further, when a yellow line and a white line exist at the same time, the vehicle control unit 308 gives priority to the yellow line and uses it as a lane mark for the traveling route. This makes it possible to control the vehicle based on a lane mark of a specific color from a road surface with lane marks of a plurality of colors.

In addition, in the above-mentioned embodiment, a case where a white line and a yellow line, which are achromatic colors, are identified is exemplified, but this embodiment can also be used when identifying a white line and an appropriate lane mark other than a white line (such as a blue line). Of course, it is applicable. Furthermore, in the above-described embodiment, the proportion of the yellow component of the road surface area estimated based on the distance information of the lane mark is acquired, and the yellow component of the lane mark is recognized based on the obtained proportion of the yellow component. In other words, when changing the threshold value (threshold value for yellow line determination), in other words, the case where the color component to be extracted and recognized in the road surface area and the color component to be extracted and recognized in the lane mark is the same. , it is also applicable when they are different.

As explained above, the image signal processing unit (lane mark recognition device) 102 of the present embodiment uses parallax information obtained from images acquired by the in-vehicle stereo camera device (camera device having an image sensor having an RCCC layer) 101. Information on lane marks on the road surface is acquired from the image, a proportion of the road surface area estimated based on the information on the lane marks occupied by a first color component (for example, a yellow component) is acquired, and the proportion of the first color component (for example, a yellow component) is A threshold value for recognizing the second color component (for example, yellow component) of the lane mark is changed based on the ratio of the second color component (for example, yellow component).

That is, the image signal processing unit (lane mark recognition device) 102 of this embodiment extracts a specific color component of the road surface portion from the recognized road surface image, and sets a threshold value that reflects the specific color component of the road surface portion. This will identify the color of the lane mark. For example, in this embodiment, a yellow lane mark is recognized by extracting the proportion of the yellow component from the recognized road surface area and setting a threshold corresponding to the proportion of the yellow component.

According to this embodiment, it is possible to accurately or more reliably recognize a lane mark of a specific color even at night when the vehicle is affected by headlights of a vehicle or street lights. For example, it is possible to reduce the possibility that a white line, which is a white lane mark, will be recognized as a yellow line, which is a yellow lane mark, and it is possible to accurately distinguish between a white line and a yellow line even at night. This makes it possible to control the vehicle based on a lane mark of a specific color from a road surface with lane marks of a plurality of colors, even at night when the vehicle is affected by headlights or street lights. For example, in a construction zone in Europe, it becomes possible to perform lane maintenance control that accurately follows lanes that have been temporarily divided into lanes.

Note that the present invention is not limited to the embodiments described above, and includes various modifications. For example, the above-described embodiments have been described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all the configurations described.

Further, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be partially or entirely realized in hardware by designing, for example, an integrated circuit. Furthermore, each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, files, etc. that implement each function can be stored in a memory, a storage device such as a hard disk, an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

Further, the control lines and information lines are shown to be necessary for explanation purposes, and not all control lines and information lines are necessarily shown in the product. In reality, almost all components may be considered to be interconnected.

100 In-vehicle stereo camera system 101 In-vehicle stereo camera device (camera device)
102 Image signal processing unit (lane mark recognition device)
103 Control unit 104 Steering 105 Accelerator 106 Brake 107 Vehicle 201 Left camera 202 Right camera 203 Image input interface 204 Image processing section 205 Arithmetic processing section 206 Storage section 207 CAN interface 208 Control processing section 209 Bus 210 In-vehicle network CAN
301 Left image acquisition section 302 Right image acquisition section 303 Parallax image generation section 304 Lane mark detection section 305 Road surface estimation section 306 Road surface color determination section 307 Lane mark color determination section 308 Vehicle control section

Claims (4)

A lane mark recognition device that recognizes lane marks on a road surface using a parallax image obtained from an image acquired by a camera device,
obtaining information on the lane mark from the parallax image;
obtaining a proportion of the first color component of the road surface area estimated based on the lane mark information;
A lane mark recognition device characterized in that a threshold value for recognizing a second color component of the lane mark is changed based on a ratio of the first color component. The lane mark recognition device according to claim 1,
obtaining the proportion of the yellow component of the road surface area estimated based on the lane mark information;
A lane mark recognition device characterized in that a threshold value for recognizing a yellow component of the lane mark is changed based on a ratio of the yellow component. The lane mark recognition device according to claim 1,
A lane mark recognition device, wherein the road surface area is an area a predetermined distance away from the lane mark. The lane mark recognition device according to claim 1,
A lane mark recognition device characterized in that the camera device has an image sensor having an RCCC layer.

Images