JP2611326B2 - Road recognition device for vehicles - Google Patents

Description

Description: Object of the Invention [Industrial application field] The present invention relates to a vehicle road recognition device that recognizes a road from an image outside the vehicle around the vehicle.

2. Description of the Related Art There have been proposed some devices that capture an image of a field of view in the front-rear direction of a vehicle using an image capturing unit such as a TV camera mounted on a vehicle, and extract and use various information included in the image. For example, JP-A-59-127200 discloses an "in-vehicle traffic sign notification device" that recognizes a road sign from an image.
Further, as a military vehicle, an unmanned tank that detects a flat area and automatically travels has been developed.

[Problems to be Solved by the Invention] However, since a road sign or the like does not always have a fixed positional relationship with a running vehicle, it is extremely difficult to directly recognize a road sign. There was a problem. Therefore, in order to realize such a technology, a device that specifies an object having a substantially fixed positional relationship with a running vehicle and that reliably recognizes the object is indispensable. As such an object, a road is considered to be suitable. However, a device for accurately recognizing a road shape in a three-dimensional real space has not been proposed.

Further, in the automatic driving of a general vehicle, for example, in the automatic driving of following a preceding vehicle, it is not sufficient to simply recognize a flat area from a captured two-dimensional image or simply recognize a road image in a two-dimensional plane. Yes, it is necessary to correctly recognize the three-dimensional shape of the road for traveling. From such a viewpoint, a device that accurately recognizes the shape of the road in the real space is required.

The present invention satisfies the above-mentioned demands, and has an object to extract a road region from an image outside a vehicle and to appropriately recognize a shape of the road in a real space.

Configuration of the Invention The configuration of the present invention that achieves the above object will be described below.

[Means for Solving the Problems] As shown in FIG. 1, the vehicle road recognizing device of the present invention has an image pickup means M2 for picking up an image outside the vehicle M1 and an image picked up by the image pickup means M2. From the two-dimensional image, road area recognition means M3 for recognizing the road area, and two-dimensional coordinates representing the boundary of the road area recognized by the road area recognition means M3 are stored in the actual image of the road imaged by the imaging means M2. Coordinate conversion means for converting to three-dimensional coordinates representing a position
M3 and the three-dimensional coordinates generated by the coordinate conversion means M4 are corrected to the three-dimensional coordinates so that the relational expression derived from the morphological characteristics of the actual road is satisfied. And a coordinate correcting means M5 for correcting the shape of the region represented by

Here, the means M3 for recognizing the road area extracts various areas existing in the image by, for example, performing various image processing on the captured image, and from these areas to the road area. It can be configured to realize various methods of image recognition, such as a configuration for selecting a corresponding area or a configuration for storing the color of an area designated as a road and tracking the color area.

The means for converting coordinates M4 calculates the two-dimensional coordinates representing the boundary of the road area by using a predetermined arithmetic expression determined based on the arrangement position of the imaging means M2, etc., and calculates the actual position of the road imaged by the imaging means M2. Is converted into three-dimensional coordinates.

The coordinate correcting means M5 corrects the three-dimensional coordinates generated by the coordinate converting means M4 so as to satisfy a relational expression derived from the morphological characteristics of the actual road. Note that the morphological characteristics of an actual road include that the road width is constant in a predetermined section, that both edges of the road are formed of smooth straight lines or curves except for intersections and branches, and that intersections and intersections have different shapes. The road shape at the branch point has such a characteristic that it has a known shape in advance.

[Operation] The vehicle road recognition device of the present invention having the above configuration recognizes a road area by the road area recognition means M3 from a two-dimensional image outside the vehicle around the vehicle M1 captured by the imaging means M2.

Then, the coordinate conversion means M4 converts the two-dimensional coordinates representing the boundary of the road area recognized by the road area recognition means M3 into three-dimensional coordinates representing the actual position of the road imaged by the imaging means M2, The correction means M5 corrects the three-dimensional coordinates so as to satisfy a relational expression derived from the morphological characteristics of the actual road, thereby obtaining an area represented by the three-dimensional coordinates, that is, a road to be recognized. Modify the shape of.

As a result, the vehicular road recognition device of the present invention corrects the form of the road area due to a change in the gradient of the road or the presence of an intersection or the like, and recognizes the shape of the road in the real space.

Embodiment In order to further clarify the configuration and operation of the present invention described above, a preferred embodiment of the vehicle road recognition device of the present invention will be described below. FIG. 2 is a schematic configuration diagram of a vehicle road recognition device as one embodiment of the present invention, and FIG. 3 is a block diagram showing a schematic configuration mainly of the image processing device.

As shown in FIG.
A full-color television camera 3 using a CCD for imaging the front of the vehicle, an image processing device 5 for processing an image captured by the television camera 3, and a color monitor 7 for displaying the processed image so that an occupant can view it. It is composed of The image processing device 5 is configured as an arithmetic and logic operation circuit including a well-known CPU 11, a ROM 12, a RAM 13, an image memory (hereinafter, referred to as a VRAM) 15, and the like. The image processing apparatus 5 includes, in addition to the CPU 11 and the like, an image signal input port 18, an input circuit 19, and a CRTC 20 connected to each other via a bus 17, and further, an RGB signal from the television camera 3 for each color. And an A / D converter 22 for converting into a digital signal.

A confirmation switch 25 is connected to the input circuit 19. The switch 25 is used when a final determination by the occupant is requested in a road recognition process described later.
The CPU 11 can read through the input circuit 19 at any time whether or not the switch 25 has been operated. The image signal input port 18 is connected to the A / D converter 22 and sequentially reads the image signal from the television camera 3 and develops the image signal in a predetermined area of the VRAM 15. Therefore, the CPU 11 can freely process the image outside the vehicle by accessing the predetermined address of the VRAM 15. The CRTC 20 is connected to the color monitor 7, and receives an instruction from the CPU 11, reads out image data after image processing developed in a predetermined area in the VRAM 15, and displays the image data on the color monitor 7.

Road recognition processing in the vehicle road recognition device of the present embodiment having the above configuration will be described. The apparatus of the present embodiment executes a road recognition processing routine shown in FIGS. 4 (A) and 4 (B). After turning on the power, first enter the image signal input port 18
Performs a process of inputting an image outside the vehicle captured by the television camera 3 (step 100). The input image data is developed in a predetermined area of the VRAM 15 as described above.
An example of the captured image is shown in FIG. continue,
The captured image data is binarized by applying a well-known spatial filter such as a Laplacian operator, and a process of extracting edges from the captured image is performed (step 110). By edge extraction, the image is binarized into edges and other parts. After edge extraction, edge enlargement / reduction processing is performed to perform processing for connecting portions where edges that should have occurred are missing in the process of capturing images (step 120). FIG. 5B shows an example of the result of such edge extraction and processing.

After the edge is repaired, a process of extracting a portion surrounded by the edge as one region is performed (step 130 in FIG. 4A). The extraction of the area surrounded by the edge is performed in order from the area existing at the bottom of the screen in consideration of the probability of the existence of the road, and by tracing the edge clockwise, the area of the area is extracted. The calculation is also performed. Subsequently, it is determined whether or not the extracted region corresponds to a road. In this embodiment, regarding the extracted area, the area calculated in step 130 is equal to or more than 1/10 of the entire screen (step 140), and the position where the area exists is a position corresponding to the road. When it is determined (step 150), the area is determined to be an area corresponding to a road. The area that should be occupied by the road with respect to the entire screen depends on which part of the vehicle 1 is provided with the television camera 3 and its elevation angle, magnification, or change in the gradient of the road. Determined experimentally. In this embodiment, the determination of the position of the region is made based on whether or not the region exists from the bottom to the center of the screen. TV camera 3
Is provided at the front of the vehicle 1, the road is usually imaged at this position.

If the extracted area does not correspond to a road, it is determined whether all the areas existing on the screen have been extracted and determined (step
160) If the determination of all the areas has not been completed, the process returns to step 130 to continue the determination of the next area. On the other hand, when the determination for all the regions is completed (step 160), the process returns to step 100 to execute again from input of a new image.

When it is determined that the extracted area is a road by the processing of steps 130 to 150, processing for displaying this on the color monitor 7 is performed (step 170). In this case, a high-visibility process (for example, yellow coloring or blinking of the area) is applied to the area extracted as a road, and displayed superimposed on the captured original image, so that it can be recognized at a glance. Is preferred. This example is shown in FIG. 5 (C). In the figure, the hatched area is colored yellow on the color monitor 7.

In this manner, it is determined whether the area shown on the color monitor 7 can be finally determined as the road area by operating the confirmation switch 25 (step 180). If the confirmation switch 25 is operated within a certain period, the area displayed on the color monitor 7 may be determined to be a road. The attributes of this area, in this embodiment, the road width L and its color, are stored in the RAM 13. (Step 190).

With the above processing, it is determined that the initial recognition of the road has been completed, and the recognition of the road using the attributes of the road area is continued from step 200 onward. First, the steps in the initial recognition process
As in the case of 100, 110, and 120, input of an image captured by the television camera 3 (step 200), extraction of edges (step 210),
The edge is repaired (step 220), and subsequently, a detection point is set using the road width L which is one of the attributes stored in the RAM 13 (step 230). The detection point SP is set as a half point of the road width L as shown in FIGS. 6 (A) and 6 (B). The road changes its shape when approaching the carp, but focuses on the fact that there is no significant change in the width occupying the bottom of the screen. The extraction of the area is started from the area to be processed. If the extracted area cannot be recognized as a road, the detected point SP is successively moved to an upper area and a new area is extracted. If the detected point SP reaches the top of the screen, ( In step 240), it is determined that the road area cannot be recognized in the image, and the above-described processing is repeated again from the input of the image (step 200).

If the detection point SP does not reach the upper part of the screen, an area where the detection point SP exists is extracted (step 250), and it is determined whether or not the area of the area is 1/10 or more of the entire screen (step 250). Step 260). If the area is equal to or smaller than the size corresponding to the road, the above-described processing is repeated from the setting of the detection point SP (step 230).

If the area of the extracted area occupies 1/10 or more of the screen, the color which is one of the attributes of the area is determined (step 270). If the color of the area can be regarded as the same as the color of the road stored in the RAM 13 in step 190, it is determined that the area is a road, and it is determined whether or not this area is continuous to the center of the screen (continuous). Sex determination) (step 280). If the continuity is satisfied, the area can be recognized as a road area, and the process proceeds to step 400 and later described later. An example of the road area thus recognized is shown in FIG.

When imaging on a real road, the road area is divided by the shadow of an overpass, building, telephone pole, etc. reflected on the road, or the edge is generated by a white line such as a pedestrian crossing or center line. Sometimes the area is divided. Such an example is shown in FIG.

When the road area is divided, even the area determined to have the area and the color corresponding to the road may not be continuous to the center of the screen. Therefore, when it is determined that there is no continuity (step 280), the detection point SP is moved to the upper or lower part of the divided area to extract the area (step 300), and the area and the road ahead are extracted. To compare the color with the area determined to be
310). If the colors of the two areas can be regarded as the same, the two areas are combined (step 320), and the continuity is determined again (step 280). If the area is not continuous to the center of the screen even after combining, it is further determined whether the area above and below the area cannot be regarded as the same color. Repeat until the sex determination is satisfied (steps 280, 300, 310, 320). If the region of the same color does not exist without the continuity being satisfied, the above-described processing (steps 200 to 320) is repeated from the image input processing (step 200).

If the continuity is satisfied (step 280), it is determined whether the recognized road is a straight road based on the obtained shape of the road area (step 400). The determination as to whether or not the road is a straight road can be easily made based on whether or not the left and right ends of the recognized road area can be regarded as straight lines. If the road is not a straight road, the following equations (1), (2), and (3) are used. On the other hand, if the road is a straight road, the coordinates of the recognized road area are calculated by the following equations (1), (4), and (5). Coordinate conversion is performed (steps 410 and 420).

The image in which the road area is recognized is obtained by developing a three-dimensional space existing in the visual field of the television camera 3 into a two-dimensional plane. This plane is hereinafter referred to as an image plane, and its coordinates are represented by (x, y), and the coordinates of the real space are represented by (X, Y, Z)
It is assumed that At this time, as shown in FIG. 7, assuming that the height of the TV camera 3 from the road surface is H and the distance from the TV camera 3 to the image plane is f between the two coordinates, as shown in FIG. A relationship exists.

Therefore, the coordinates of the road, that is, the coordinates (X, Y, Z) of the left end and the right end of the road can be obtained by performing coordinate conversion from the image plane to the real space according to the equation (1). However, such a conversion is correctly established when it is assumed that the road captured by the television camera 3 is a two-dimensional plane that does not include a gradient change or the like.

Based on the road coordinates (X, Y, Z) thus obtained, the relative position of the vehicle with respect to the road and the equations at the left and right ends of the road are calculated. When the road has curvatures R1 and R2 (see FIG. 8), or when the road is a straight road making an angle θ with respect to the vehicle 1 (see FIG. 9), the left end and the right end are represented in real space. The equations are given by the following equations (2), (3) and (4),
(5). The expressions W1 and W2 indicate the distance of the vehicle 1 to the left and right ends of the road, respectively.

Leftmost case-road curve road ^{Z 2 + (X + R1 +} W1) 2 = R1 2 H 2 x 2 -2xyH (R1 + W1) - (2R1W1-W1 2) y 2 = -f 2 H 2 ... (2) · right edge of the road Z ² + (X + R 2 + W 2) ² = R ^{2 2} H ² x ² −2xyH (R 2 + W 2) − (2R 2 W ² −W 2 ² ) y ² = −f ² H ² … (3) For a straight road ・ The left end of the road ・ Right edge of road The equation of the left and right ends of the road thus obtained is compared with the coordinates of the left and right ends of the road area extracted from the captured image (step 430). If they do not match, the position is corrected (step 440). The two do not match: (A) when the road includes an uphill or downhill with a gradient change and plane approximation cannot be performed; (B) when an error occurs due to noise included in the image signal; C) When an intersection or a branch point is included, etc. In such a case, it is conceivable that an accurate road position cannot be recognized. Therefore, in this case, the road position is corrected using the condition that the road width is constant and the road position information on the screen processed up to that time. For example, in the case of the above (A), as shown in FIG. 10, with respect to the right and left ends Rl and Rr of the road where there is no gradient change,
The left and right ends Fl and Fr of the road area on the captured image are shifted. Therefore, the value of y on the image plane changes,
A shift also occurs in the coordinates in the real space. In such a case, using the condition that the road width W is constant, the coordinates y are corrected so that the following equation (6) is satisfied.

W = X2−X1 = x1 · H / y1−x2 · H / y2 (6) Since the road width direction is parallel to the normal screen,
In this case, y1 = y2. In the case (B), the left and right ends of the road are smoothed using an approximation of being a smooth straight line or curve. Further, in the case of (C) above, it is determined whether or not an intersection or a branch point is included from the shape of the road area obtained by the recognition of the road area. The correction can be performed by superimposing the stored information on the shape of the intersection and the like.

After the position is corrected in this manner, or in step 43
If the expression and the image data match in the judgment of 0, the overhead view of the road and the position of the vehicle are obtained based on the coordinates of the road in the real space obtained by the above processing (step 45).
0), which is displayed on the color monitor 7 (step 46).
0). An example of a bird's-eye view is shown in FIG.

According to the vehicle road recognition device of the present embodiment configured as described above, after extracting an area corresponding to a road based on an image captured by the television camera 3 provided in the front of the vehicle 1, Attributes such as the width and color of the area are stored, and the extraction of the road area is corrected based on the attribute, and the areas are combined here. Therefore, the road recognition device of the present embodiment can recognize roads extremely accurately and continuously, and further corrects the shape of the road in the real space based on the road thus recognized by correcting a gradient change or the like. Can be sought. In this embodiment, the shape of the road thus determined can be displayed on the monitor 7 as an overhead view together with the position of the vehicle 1. Therefore, the vehicle occupant can use this information, and can further improve the safety in driving the vehicle.

The three-dimensional shape of the road and the positional relationship of the vehicle with respect to the road thus obtained can be used by various devices such as a key plane traveling device (not shown). When used in such an apparatus, it is also possible to realize automatic or semi-automated driving, accident prevention such as rear-end collision, and the like. Further, since it is not necessary to lay a device for identification on the road side, it becomes easy to actually mount the road recognition device on the vehicle.

Although the embodiments of the present invention have been described above, the present invention is not limited to such embodiments at all. For example, a configuration for realizing tracking of a preceding vehicle and key plane traveling using a recognized three-dimensional shape, a vehicle It does not deviate from the gist of the present invention, such as a device that also processes the tail image and recognizes the road and displays a bird's-eye view of the road before and after the vehicle based on the road, or a device that uses the roughness of the area (texture) as the attribute of the road. Of course, it can be implemented in various ways within the scope.

Effects of the Invention As described in detail above, according to the vehicle road recognition device of the present invention, a road region existing in a two-dimensional image is recognized, and two-dimensional coordinates representing the boundary of the recognized road region are used. The three-dimensional coordinates representing the actual position of the road are obtained, and the three-dimensional coordinates are corrected so as to satisfy a relational expression derived from the morphological characteristics of the actual road. The shape of the region to be recognized, that is, the three-dimensional shape of the road to be recognized, is accurately obtained. As a result, the recognized road can be output, for example, displayed as a bird's-eye view, and the road information can be provided to the occupant to contribute to safe driving. Further, since it is not necessary to lay a device for identification on the road side, it becomes easy to actually mount the road recognition device on a vehicle.

Further, by combining with the vehicle road recognition device of the present invention, for example, a configuration for recognizing a gradient change from the three-dimensional shape of the road and using it for constant speed traveling control, a configuration for performing automatic tracking traveling or key plane traveling, and the like. This makes it possible to increase the accuracy of the processing of these devices. The realization of such a device can contribute to the improvement of driving safety of a vehicle and is extremely useful.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a basic configuration of the present invention, FIG. 2 is a schematic configuration diagram of a vehicle road recognizing device as one embodiment of the present invention, FIG. 3 is a block diagram of the same, and FIG. 5A and 5B are flowcharts showing a road recognition processing routine executed by the image processing apparatus according to the embodiment, and FIGS. 5A, 5B, 5C, 5D, and 5E each show road recognition. FIGS. 6A and 6B are explanatory diagrams illustrating an example of an image processed by the processing, FIGS. 6A and 6B are explanatory diagrams illustrating a state of detection point SP setting, and FIG. 7 is a conceptual diagram illustrating a relationship between an image plane and a real space , FIG. 8 is an explanatory view showing a coordinate conversion method on a curved road, and FIG.
FIG. 10 is an explanatory diagram showing a method of coordinate transformation on a straight road. FIG. 10 is an explanatory diagram showing a state of recognizing a road having a gradient change.
The figure is an explanatory view showing an example of an overhead view obtained in the present embodiment. 1 vehicle 3 television camera 5 image processing device 7 color monitor 15 VRAM SP detection point GS search area

Claims (1)

(57) [Claims] 1. An image pickup means for picking up an image of the outside of a vehicle around a vehicle, a road area recognition means for recognizing a road area from a two-dimensional image picked up by the image pickup means, and a road recognized by the road area recognition means. Coordinate conversion means for converting the two-dimensional coordinates representing the boundary of the area into three-dimensional coordinates representing the actual position of the road imaged by the imaging means, and the three-dimensional coordinates generated by the coordinate conversion means A coordinate correcting unit that corrects the three-dimensional coordinates so as to satisfy a relational expression derived from the morphological characteristics of the road and corrects the shape of the area represented by the three-dimensional coordinates. Road recognition device.