JPH0676065A - Method and device for recognizing road circumstances - Google Patents

Abstract

PURPOSE:To reduce the capacity of a memory to be used, to increase the processing speed, and to meet the personal needs with an inexpensive device with respect to the road circumstance recognition method using picture recognition as the driver aid. CONSTITUTION:This method consists of a picture input device 1, a picture data storage memory 2 where input picture data is stored, a feature point extraction processing 3 which extracts feature points of a picture, an area extraction processing 4 which extracts a mark area, a feature point interpolation processing 5 which connects feature points, a sample area designating part 6 which designates a sample area, a sample data storage memory 7, a picture data comparison processing 8 which compares the input picture with sample data, and a display device 9 on which the recognition result is displayed. Since the preprocessing of picture processing is omitted and partial data is processed, the memory capacity to be used is reduced, and the processing speed is increased. Since sample data is inputted, recognition meeting the personal purpose is realized.

Description

Detailed Description of the Invention

[0001]

[Field of Industrial Application] As a method for supporting a driver when driving a vehicle, there is a method of recognizing a lane, an obstacle, a sign, etc. on a road to inform the driver or automatically stopping.
The present invention relates to a road environment recognition method and apparatus for processing a road environment image to assist a driver and obtain basic information for automatic driving.

[0002]

2. Description of the Related Art In image recognition technology for road environments, various recognition methods for signs, signals, road displays, etc. have been proposed. For example, regarding the recognition of a sign, JP-A-1-265 is available.
In No. 400, the road area is obtained, the area where the sign or the like exists is estimated from the result, and the feature points are extracted from the area,
A method of recognizing a sign from color, shape, size, etc. has been proposed. Further, regarding the recognition of roads, JP-A 1-2
Japanese Patent No. 65399 proposes a method of recognizing a road region from feature points, colors, areas, and the like.

[0003]

In the above prior art, feature points are extracted and noise processing is performed in all areas of the input image, and the areas are divided into several areas, and the road is recognized based on their colors and shapes. There is. Further, the position of a sign or the like is estimated from the recognized road to recognize the sign. Therefore, the processing time per screen of the input image is long, and the memory area required for the processing is large. Further, regarding the recognition of a wide variety of signs and the like, the difference in their feature amounts is not taken into consideration. Further, there is no particular description about the method of notifying the user of the recognition result.

A first object of the present invention is to solve the above problems and to provide a road environment recognition method capable of processing in a short processing time and a small memory area. The second purpose is
The user specifies various kinds of signs and the like to individually recognize the specified kinds of signs. A third object is to enable recognition by a low-resolution video camera or an arithmetic device having a slow processing speed, and to enable road environment recognition by an inexpensive device. The fourth purpose is to efficiently notify the user of the recognition result of lanes, signs, and the like.

[0005]

FIG. 1 is a structural diagram of the present invention for achieving the above object.

The image data input by the image data input device 1 is stored in the image data storage memory 2. The sample area designating unit 6 is a configuration for designating a marker or the like that the user wants to recognize. The feature point extraction process 3 is a process of extracting a part of the features of the image data. Based on this feature, the region extraction process 4 extracts a candidate for a marker, and the feature point interpolation process 5 extracts a lane. The image data comparison processing 8 is
The characteristics of the sample data and the input image data are compared to determine whether the marker is the designated marker. The display device 9 displays the input image and the recognition result.

[0007]

[Function] Since the sample data is input and the recognition is performed by comparing the sample data with the input data, it becomes possible to individually recognize the target signs and the like, and the recognition corresponding to the intention of the user. Is now possible. Since the shape, size, and color of signs, etc. can be recognized even if strict calculation and noise processing are omitted, it is possible to recognize images with blurry or noisy images input with a low-resolution camera. It became so. Furthermore, since one or more parts of the input image data are extracted and processed, and preprocessing for image recognition such as noise processing is omitted, the processing time required for recognition can be shortened and the memory used Could be reduced. By this,
Even if an image input device (for example, a camera) to be used, an arithmetic processing device including a memory, or (for example, a computer) that is inexpensive to some extent, road environment recognition can be performed.
Since the safety area on the lane is displayed over the image data, the user can check the safety area at a glance. Since the recognition result display unit is provided in the display device, even if the user misses the sign or the like, the recognition result can be confirmed on the display device.

[0008]

Embodiments of the present invention will now be described in detail with reference to the drawings.

FIG. 1 is a system configuration diagram of the present invention. Image data input by an image data input device 1 and converted into digital data is stored in an image data storage memory 2. The sample area designation unit 6 takes out the designated area from the image data storage memory 2 and stores it in the sample data storage memory 7 together with the feature data extracted by the feature point extraction processing. The feature point extraction process 3 extracts the features of the image data in the image data storage memory 2 and the sample region designating unit 6, and the region extraction process 4 uses the extracted feature points to identify the region corresponding to the recognition target. Extract. When the area of the sample data is extracted, the characteristics of the area are recorded in the sample data storage memory 7, and in the case of the data of the image data storage memory 2, the area data is passed to the image data comparison processing 8. The image data comparison processing 8 is
The data of the sample data storage memory 7 and the data of the area extraction processing 4 are compared to determine whether they are the same as the sample data,
When it is determined that they are the same, it is displayed on the display device 9. Figure 2
Is an example of image data. By recording the shaded portion 10 and its characteristics as sample data, it is possible to recognize markers having the same characteristic amount. The display device 9 is divided into an input image data display section and a recognition result display section, and the recognition results such as signs are displayed on the recognition result display section.

By inputting the lines 11 to 16 in FIG. 2 as partial data and connecting the characteristic points on each line,
It is possible to cut out a safety area on the lane in which the vehicle is currently located, that is, an area in the lane where there is no preceding vehicle or obstacle. The feature point extraction processing 3 in FIG. 1 extracts partial image data, extracts the feature points, and passes the feature point data to the feature point interpolation processing 5. The feature point interpolation process 5 rearranges the feature points in the lateral coordinate order and creates a polygon having these points on the sides. By displaying the polygon on the display device 9 by superimposing it on the image data, the safety area on the lane is indicated. Alternatively, the center line of a vehicle ahead or an obstacle is recognized in consideration of the shape and size of the polygon.

The detailed operation of each process will be described below.

FIG. 3 is an operation flowchart of the feature point extraction processing 3 in FIG. The feature point extraction processing 3 cuts out a necessary portion from the image data in the image data storage memory 2 (step 101). The part to be cut out is, for example,
If it is a road sign, it is the upper left part, if it is a signal, it is the upper part, and if it is a road recognition, it is a line segment such as lines 11 to 16 shown in FIG. The larger the size of the portion cut out for the recognition of the road sign, the lower the probability of missing the sign, but the larger the area to be processed, the larger the memory used and the processing time. The size of the area is determined by the processing speed of the arithmetic device and the capacity of the memory. Next, the edges of the cut out image data are extracted (102). At this time, in the case of sign recognition, all edges of the cut out image data are extracted. In the case of road recognition, the vertical line (11 to 13 in FIG. 2) searches for an edge from the lower part of the image, that is, a part near the vehicle, and ends the edge extraction when the edge is first found. The horizontal lines (14 to 16 in FIG. 2) search for an edge in the center of the image, that is, from the front of the vehicle to the left and right, and end the edge extraction when the edge is first found.
Similar to the case of the vertical and horizontal lines, it is also possible to use an oblique line that goes diagonally forward and to the left and right from the vehicle and search for the edge in the direction away from the vehicle. An area having the edge found on each line as a point on the side is the required safety area. In the case of lane recognition, such an edge search is performed in order to recognize the front safety area.Therefore, the edges outside the safety area are not required. Because it was done. By this search method, unnecessary edge extraction is not performed, so the processing time can be further shortened. Since noise processing is not performed, false edges may be picked up, but noise can be dealt with by increasing the threshold value for edge processing. Next, if the cut out data is lane data (103), the edge data is passed to the feature point interpolation processing 5 (105), and if not, the image and the edge data are passed to the area extraction processing 4 (104). .

FIG. 4 is an operation flowchart of the area extraction processing 4. When the image data and the edge data are input from the feature point extraction processing 3 (121), a continuous area in the vertical and horizontal directions having no edge is extracted. That is, when pixels adjacent to each other in the vertical and horizontal directions have no edges, it is considered that these pixels are continuous. At this time, even if an edge exists in the area, if the area is continuous in another portion, this is regarded as one area. For example, in a sign as shown in FIG. 11, a part corresponding to a character is extracted as an edge, but there is no edge around the character, and a part other than the character in the sign is a part without continuous edges. 1 other than characters
Considered as one area. Instead of looking at the continuity of the edges,
The continuity of the part without edges is used for the recognition of the sign because the parts other than the characters and arrows of the sign are uniform, there is not much color change, there are few edges, and the part without edges is continuous. This is because there are many cases. Even if there is noise, noise often appears as a group of small edges, and if there is a continuous area without edges around it, it is recognized as one region and the noise is ignored, so in practice This means that noise processing is being performed. If the sign is recognized based on the continuity of the edges, noise may be detected as an edge or, on the contrary, the edge may be erased by the noise. Is difficult to recognize unless noise elimination is performed in the preprocessing.
In addition, there is an advantage that it is easy to distinguish between a place where edges are complicated such as a tree or a grass or the like, and a place where edges hardly appear such as a sign.

Next, a color histogram of the extracted continuous area is created (123). In FIG. 5, it is assumed that each pixel of the digital image is indicated by the luminance of the ternary color of red R, green G, and blue B. Create a histogram for each of the three primary colors. In the histogram, the horizontal axis represents luminance and the vertical axis represents the number of pixels corresponding to each luminance. At this time, assuming that each color is represented by 256 levels of brightness, the histogram does not take the number of pixels in all of 256 levels, but divides the brightness into 16 levels to create a histogram. . Now, the red brightness of a pixel is r in 256 levels.
At this time, in the histogram, this pixel is added to the r / 16th position. This method of creating a histogram saves memory and further shortens the time for subsequent processing.
Next, the shape and area of the region are calculated (124). However, the calculation of the shape can be omitted depending on the surrounding environment. If there are buildings or signs with various colors and shapes around the sign, such as in urban areas, it is necessary to use the shape as a judgment material to prevent misrecognition, but the background is such as a forest or a field. For objects that have it, it is possible to recognize the sign only by the color information and the area. Since the calculation of the shape requires correction for the distortion of the shape depending on the viewing direction,
Since it takes a long processing time, if it is possible to omit it, avoid it. Next, when the input data is sample data (125), the color histogram and information on the shape and area of the area are stored in the sample data storage memory 7 (126). If not, the same data is sent to the image data comparison process (127).

FIG. 6A is a diagram showing the positions of the characteristic points of the lane extracted by the characteristic point extraction processing 3. Screen 2
00, the x-axis is from left to right in the horizontal direction, and the y-axis is from top to bottom in the vertical direction. The points 201 to 208 are the lines 11 in FIG.
16 are the feature points extracted above. FIG. 6B is a polygon created by connecting these feature points with line segments in ascending order of the x coordinate. The inside of this polygon is the safety area on the lane.

FIG. 7 is an operation flowchart of the feature point interpolation processing 5. The position of the edge is input from the feature point extraction processing 3 (141), the edges are rearranged in ascending order of the x coordinate (142), and a polygon is created by connecting line segments in the rearranged order (143) and displayed. Yes (144). The shape of a lane is not limited to a straight line, but various shapes such as a curve can be obtained. As described above, the polygon connecting the characteristic points can be approximated to these shapes. It is possible to obtain a more approximate polygon by increasing the number of lines taken out from the image data storage memory, but since the number of lines is proportional to the memory capacity and processing time required for processing, Determined by ability.

FIG. 8 is an operation flowchart of the image data comparison processing 8. Image data and edge data are input from the area extraction processing 4 (161), and sample data are input from the sample data storage memory 7 (162). When the two input data are compared and judged to be equivalent data (163), the result is displayed on the display device 9 (164). At this time, the image data and the sample data may be slightly different in color, size, and shape depending on the ambient brightness, the angle of the light beam, the distance from the vehicle, the angle of the camera, and the like. Therefore, it is necessary to take the above-mentioned difference in surrounding environment into consideration when determining whether the data are equivalent.

As described above, the road environment recognition is realized by the operation of each constituent element of FIG.

Next, a method of expressing the recognition result will be described. FIG. 9 shows an example of the display screen of the display device 9. Figure 9
(A) and (b) are the examples which displayed the recognition result of a lane. FIG. 9A is a diagram showing a recognition result polygon overlaid on an input image. This allows the user to see the safe area in front at a glance. On the other hand, figure (b)
Is a line connecting characteristic points instead of a polygon. This simplifies the display of (a) and does not completely cover the road in the safety area, so that the user can confirm the recognition result. In addition to this, a method of displaying the position of the front characteristic point as a character or notifying the user by voice may be considered. This is because the vertical position of the screen, that is, the distance of the y-coordinate from the own vehicle is measured in advance from the installation position and angle of the camera, and obstacles such as vehicles ahead of the vehicle are determined from the y-coordinate of the characteristic point. This is possible by calculating the distance to.

FIG. 9C is a display example of the sign recognition result. The display screen is divided into an input image display unit 301 and a sign recognition result display unit 302. When it is recognized that there is a marker in the partial data cut out from the image data storage memory 2, the partial data is displayed on the marker result display unit 302. There is also a method of displaying the area of the sign from the partial data and displaying it, but it seems that it takes time to extract the sign. Further, as shown in FIG. 9C, some recognition results are displayed on the sign recognition result display section, and the latest recognition result is marked with a thick frame or the like. This allows the user to confirm the overlooked sign and the like.

FIG. 10 shows a method of designating a sample area in the sample area designating section 6. FIG. 10A is a designation method in the case where the marker designated as a sample is a rectangle, and the positions of the start point and the end point are input, or the start point position and the vertical and horizontal lengths are input. The input of the position can be simplified by using a device such as a mouse, a trackball, a light pen, or a touch panel, which can directly input a point on the screen, in addition to inputting a numeral from a keyboard. FIGS. 10B and 10C show an input method in the case of a shape other than the sign rectangle. (B) is a method of tracing the contour of the sign with a light pen or the like, and (c) is a method of designating a rectangular area including the mark as in (a). In the method of (c), the probability of erroneous recognition can be reduced by designating that the portion other than the sign is not included as much as possible. Besides these methods,
Prepare standard areas of some sizes (rectangles, circles, triangles, etc.), and select the appropriate area from among them, and touch the area such as the center of the area on the touch panel. There is also a method of inputting from etc.

[0022]

According to the present method, since the sample data can be input, it is possible to recognize the mark which is intended by the user. Further, since image recognition can be performed even if pre-processing such as noise processing is omitted, processing time for both sign recognition and lane recognition is shortened. Since I took out only the necessary part from the input image and processed it,
The memory required for processing has decreased and the processing time has shortened. Since it became possible to recognize the road environment even with a home video camera or a personal computer, an inexpensive road environment recognition device became possible.

[Brief description of drawings]

FIG. 1 is a system configuration diagram of the present invention.

FIG. 2 is an explanatory diagram of partial data.

FIG. 3 is an operation flowchart of feature point extraction processing.

FIG. 4 is an operation flowchart of region extraction processing.

FIG. 5 is an explanatory diagram of a color histogram.

FIG. 6 is an explanatory diagram of characteristic points on a lane.

FIG. 7 is an operation flowchart of feature point interpolation processing.

FIG. 8 is an operation flowchart of image data comparison processing.

FIG. 9 is an explanatory diagram of a screen display example.

FIG. 10 is an explanatory diagram of a sample area specifying method.

FIG. 11 is an explanatory diagram of a continuous area.

[Explanation of symbols]

1 ... Image data input device, 2 ... Image data storage memory,
3 ... Feature point extraction process, 4 ... Region extraction process, 5 ... Feature point interpolation process, 6 ... Sample region designation unit, 7 ... Sample data storage memory, 8 ... Image data comparison process, 9 ... Display device.

Claims (7)

[Claims] 1. An image data input device inputs image data, the input image data is stored in an image data storage memory, partial feature points of the image data are extracted, and a region is extracted from the image data and the feature point data. Extract, interpolate partial feature point data, specify the area of the sample, record the characteristics of the sample area in the sample data storage memory, compare the characteristics of the sample data and the input image data, and display the comparison result. Road environment recognition method displayed on the device. 2. The sample area designating section designates an area including road signs and the like in the image data, inputs characteristic quantities such as color information of the signs as sample data, and inputs them to a sample data storage memory, 2. The road environment recognition method according to claim 1, wherein a road sign or the like is recognized by comparing the feature amount information of the data in the input image with the information in the sample data storage memory by the image data comparison process. 3. A part or a plurality of parts of the image data input by the image data input device is extracted, the characteristic points are extracted from the partial image data extracted by the characteristic point extraction processing, and the partial image data is partially extracted by the area extraction processing. The road environment recognition method according to claim 1, wherein a lane or the like is recognized by extracting a region from the image data and the feature point. 4. A road environment recognition process from digital data, comprising a video camera for inputting a road environment image, a video data conversion device for converting a video signal into digital data, and a memory for storing the converted digital data and sample data. A road environment recognition device composed of an arithmetic processing unit for performing. 5. A road environment recognition device comprising a home video camera for inputting a road environment image, a video data conversion device for converting a video signal into digital data, and a personal computer for performing road environment recognition processing from the converted digital data. 6. The road environment recognition method according to claim 1, wherein a polygon or a polygonal line representing a safety area on the lane obtained from the recognition result of the lane or the like is displayed on the display device while being superimposed on the input image data. 7. The road environment recognition method according to claim 1, wherein a recognition result display portion is provided on a display screen of the display device, and images of one or more recognized road signs are displayed on the recognition result display portion.