KR101778724B1 - Device and method for reconition of road markings - Google Patents

Abstract

The present invention relates to a road sign recognition technique, and more particularly, to an apparatus for recognizing a damaged road sign on a road surface by using a convolutional neural network technique, and a method thereof. According to an embodiment of the present invention, the road sign can be accurately recognized by using a model that has learned various road signs based on deep learning. The road sign recognition apparatus comprises: a road sign image input part; a road sign recognition part; and a road sign guide part.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a road sign recognition apparatus,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a road mark recognition technology, and more particularly, to an apparatus and a method for recognizing a road mark on a road surface using a CNN (Convolutional Neural Network).

Recently, there has been a system for warning the occurrence of a vehicle accident by using images and sensors, etc., caused by the development of an advanced driver assistance system (advanced driver assistance system) of a vehicle. However, the road signs and lanes on the road are considerably old, and the budget of the government that paints this damaged lane or road mark is in short supply. Here, the meaning of the damaged road sign means that it is corroded, worn, discolored. These damages are caused by natural factors such as snow, rain, and external factors due to road surface contact of the vehicle.

The background art of the present invention is disclosed in Korean Patent Laid-Open Publication No. 10-2016-0122452 (October 26, 2014).

The present invention provides a road mark recognition apparatus and method for recognizing a photographed road image using a line neural network technology model in which various road signs are learned.

According to an aspect of the present invention, a road sign recognition apparatus is provided.

The road mark recognition apparatus according to an embodiment of the present invention includes a road sign image input unit to which a photographed road image is input, a road sign image extraction unit that extracts a road sign image from the input road image, A road sign recognition unit for recognizing the road sign by applying the Neural Network model, and a road sign information unit for outputting the recognized road sign.

According to another aspect of the present invention, a road mark recognition method and a computer program for executing the same are provided.

A road mark recognition method and a computer program for executing the same according to an embodiment of the present invention include steps of inputting a road image, designating an area of interest in a road image, performing inverse perspective transformation on a designated area of interest, Extracting a road mark image from the inverse perspective transformed projected image, and performing line neural network based learning on the road mark image and recognizing and outputting the detected road mark image.

According to the embodiment of the present invention, it is possible to accurately recognize the photographed road mark using a line neural network model in which various road markings are learned based on Deep Learning.

Further, according to the embodiment of the present invention, it is possible to correctly recognize the road sign and guide the driver to the correct road, to support accurate driving and safe driving, and to prevent a traffic accident.

1 is a view for explaining a road sign recognition system according to an embodiment of the present invention;
2 to 5 are views for explaining a road sign recognition apparatus according to an embodiment of the present invention.
6 to 18 are views for explaining a road mark recognition method according to an embodiment of the present invention.

While the present invention has been described in connection with certain exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and similarities. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. In addition, the singular phrases used in the present specification and claims should be interpreted generally to mean "one or more " unless otherwise stated.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals refer to like or corresponding components throughout. .

1 is a view for explaining a road sign recognition system according to an embodiment of the present invention.

Referring to FIG. 1, the road marking recognition system includes a photographing device 10, a road sign recognition device 20, and a road sign 30.

The image photographing apparatus 10 photographs the road. The image capturing apparatus 10 may be, for example, a CCD or a CMOS digital camera, and may be a black box camera.

The road sign recognition device 20 recognizes the road sign in the photographed road image. The road mark recognition apparatus 20 detects a road mark image on the photographed road image and recognizes the road mark image by applying a deep learning based circuit neural network model. The road sign recognition apparatus 20 will be described in more detail below with reference to FIG. 2 to FIG.

The road sign 30 includes a forward arrow (FA), a forward-left arrow (FLA), a forward-left-right arrow (FLRA), a forward- Right arrow, right arrow (LA), and right arrow (RA).

FIGS. 2 to 5 are views for explaining a road mark recognition apparatus according to an embodiment of the present invention.

Referring to FIG. 2, the road sign recognition apparatus 20 includes a road sign image input unit 100, a road sign recognition unit 200, and a road sign guide unit 300.

The road sign image input unit 100 inputs the photographed road image.

The road mark recognition unit 200 extracts a road mark image from the input road image, and recognizes the road mark by applying a deep-learning-based line neural network model to the extracted road mark image.

3, the road mark recognizing unit 200 includes a region of interest specifying unit 220, an inverse perspective transforming unit 240, a road mark detecting unit 260, and a circuit neural network based recognizing unit 280.

The interest area designation unit 220 designates the road surface information as an area of interest in the input road image and removes other unspecified unnecessary areas.

The inverse perspective transform unit 240 performs inverse perspective transformation of the designated road surface information to generate transformed road surface information. The inverse perspective transform unit 220 projects the input image as a bird's eye view. The inverse perspective transforming unit 220 may determine that the bird's eye view is good when the lanes become straight lines using camera parameters such as the tilt angle and the focal length of the camera.

The road sign detecting unit 260 detects the road sign image from the converted road surface information. The road sign detecting unit 230 may detect a road sign by applying at least one of algorithms such as Labeling, Morphology, Canny Edge Detection, and Sobel Edge.

The circuit neural network based recognition unit 280 recognizes the detected road sign image by applying a recognition method based on a CNN (Convolutional Neural Network).

4 and 5, the circuit-based neural network based recognition unit 280 includes an image input layer 282, a feature extraction layer 284, and a classification layer 286.

The image input layer 282 inputs a road cover image of a predetermined scale. The image input layer 282 may utilize a gray-scale image and may be of a size of 265 x 137 pixels.

The feature extraction layer 284 extracts features by applying a line neural network model to the inputted road mark image. Feature extraction layer 284 may include (1) three convolution layers; (2) each convolution layer may comprise a rectified linear unit; (3) a channel-wise local response normalization layer, and (4) a maximum pooling layer.

Here, the first convolution layer receives an image of 265 x 137 x 1 pixels, and can be convoluted at intervals of two pixels using 180 filters of size 5x5x1. In this case, the weight per filter size is 5 x 5 x 1 = 25, and the total number of parameters in the first convolution layer is (25 + 1) x 180 = 4680, where 1 represents a bias. The size of the feature map in the first convolution layer is 131 x 67 x 180, and 131 and 67 are the output height and width. The output height (or width) can be computed using the input height (or width) - filter height (or width) + 2 × padding) / stride +1. The outputs pass through the ReLU layer and the cross-channel normalization layer and are processed and downsampled by a first Max-Pulling layer that includes 3x3 filters applied at 2-pixel intervals. Thus, the output from the first max-pulling layer is 65 (= ((131 -)) by the output height (or width) = input height (or width) - filter height (or width) + 2 x padding) / stride + 3 + 2 × 0) / 2 + 1)) × 33 (= ((67-3 + 2 × 0) / 2 + 1)) × 180. The depth dimension remains unchanged at 180 after the application of Max Pooling. The size of the feature map following the first max-pulling layer is 65 x 33 x 180. Thereafter, in the second convolution layer, convolution is performed using 250 filters of 5 × 5 × 1, a second max-pulling layer having a filter of 3 × 3 is applied, and a process of reducing the size of the feature map Is applied at intervals of two pixels. After the third convolution layer has 250 3 × 3 filters at 2 pixel intervals and a third max-pooling layer with 3 × 3 filters at 2 consecutive pixels, the output is 750 (= 3 X250), which can be used as the input of the first fully connected layer.

In addition, the feature extraction layer 284 may further comprise four fully connected layer sets next to the convolution layers. The first three full connection layer sets may include a full connection layer and a ReLU layer, a dropout layer may be inserted before the fourth full connection layer, and a softmax layer . ≪ / RTI >

Here, the first complete connection layer may have 750 and 1920 nodes as inputs and outputs, respectively. The second fully connected layer may have 1920 and 1024 nodes as inputs and outputs, respectively. The third fully connected layer may have 1024 and 512 nodes respectively as inputs and outputs. Prior to the fourth complete connection layer, a dropout technique is applied, which can set the output of each hidden node to zero based on a randomly predetermined probability. Here, the preset probability may be 0.65. The next fourth full connection layer may have 512 and 6 nodes as inputs and outputs, which may ultimately be obtained through a soft max function.

The classification layer 286 classifies the road markers into a predetermined number of road markers based on features extracted by applying a line neural network model. Here, the preset number may be six, and may be a forward direction arrow (FA), a forward-left arrow (FLA), a forward-left-right arrow (FLRA) And may include at least one of a forward-right arrow (FRA), a left arrow (LA), and a right arrow (RA).

Table 1 below sets parameters of a circuit neural network model according to an embodiment of the present invention.

Table 1. Circuit neural network structure Layer type Number
of filters Size of
Feature Map Size of
kernel Number
of stride Image input layer 265 (height) x 137 (width) x 1 (channel) One ^st convolutional  layer 180 131 x 67 x 180 [5 5] [2 2] ReLU  layer Cross channel normalization layer Max pooling layer 180 65 x 33 x 180 [3 3] [2 2] 2 ^nd convolutional  layer 250 31 x 15 x 250 [5 5] [2 2] ReLU  layer Cross channel normalization layer Max pooling layer 250 15 x 7 x 250 [3 3] [2 2] 3 ^rd convolutional  layer 250 7 x 3 x 250 [3 3] [2 2] ReLU  layer Cross channel normalization layer Max pooling layer 250 3 x 1 x 250 [3 3] [2 2] One ^st  fully connected layer 1920 ReLu  layer 2 ^nd  fully connected layer 1024 ReLu  layer 3 ^rd  fully connected layer 512 ReLu  layer Dropout layer 4 ^th  fully connected layer 6 Softmax  layer Classification layer (output layer)

Referring again to FIG. 2, the road sign guide 300 outputs the recognized road sign to guide the driver.

6 to 18 are views for explaining a road mark recognition method according to an embodiment of the present invention.

Referring to FIG. 6, in step S610, the road marker recognition apparatus 20 inputs a road image. For example, the road mark recognition apparatus 20 can input a road image as shown in Fig. 7 (a).

In step S620, the road mark recognition apparatus 20 designates a region of interest in the road image.

In step S630, the road mark recognition apparatus 20 performs inverse perspective transformation on the designated ROI image to generate a projected image. The road sign recognition apparatus 20 can convert the input road image into various images and directions by using inverse perspective mapping, as shown in FIG. 7 (b). Here, the inverse perspective mapping can be used by calculating the perspective transformation matrix.

In step S640, the road mark recognition apparatus 20 extracts the road mark image from the inverse perspective transformed projected image. The road mark recognizing apparatus 20 can extract the road mark image located in the dotted line of FIG. 7 (c). The road sign recognition apparatus 20 can extract a road sign by applying at least one of algorithms such as Labeling, Morphology, Canny Edge Detection and Sobel Edge.

In step S650, the road mark recognition apparatus 20 performs the line-by-line neural network-based learning on the road mark image and recognizes and outputs the detected road mark image.

The road sign recognition apparatus 20 is configured to receive, from six data sets including various illumination changes, shadows and severe damage, ten road signs including a general road sign, a damaged road sign and a road sign including a shadow, Were used for learning and verification. Here, the six data sets include the load marking data set shown in FIG. 9, the KITTI data set shown in FIG. 10, the Malla data set shown in FIG. 11, the Malla city data set shown in FIG. 12, The Naver distance view data set shown and the data load data set shown in Fig. The data of the six data sets includes a forward arrow (FA), a forward-left arrow (FLA), a forward-left-right arrow (FLRA), a forward right turn indication (FRA) Right arrow, LA arrow, right arrow LA and right arrow (RA). In order to learn the circuit neural network, the road sign recognition apparatus 20 transforms all road mark images into 194 × 94 pixel size by size normalization and bi-linear interpolation for data of different sizes. The road sign recognition apparatus 20 performed learning and verification using a total of 163,809 data as shown in Table 2 below in six data sets.

Table 2. Number of Data FA FLA FLRA FRA LA RA Total Number of data 32,686 17,885 22,344 17,885 36,766 36,243 163,809

The learning and validation was done using a NVIDIA GeForce GTX TITAN X (3,072 CUDA cores) computing device with an Intel® Core ™ i7-6700 CPU @ 3.40 GHz (4 CPUs), 64 GB memory and 12 GB of memory.

The road sign recognition apparatus 20 learned a line neural network model, and the image of 194 × 94 pixels was finally changed to a size of 265 × 137 pixels of 8 bits gray by bilinear interpolation method and used for learning and verification. Here, learning of the circuit neural network model is the same as the method described with reference to Figs. 4 and 5 above.

15 to 18 are diagrams showing road sign classification results of a road mark recognition method through learning based on a circuit neural network according to an embodiment of the present invention.

15 to 18, the y axis of the graph represents the road mark classification learning accuracy, and the x axis represents a training epoch. Here, a training epoch means that 10 epochs are learning 100,000, assuming that the data to be learned is 10,000.

Verification performed 13 cross validations by dividing the data into 13 groups and 92.7% (151,809 images) and 7.3% (12,000 images) were randomly selected among the data sets of the entire 163,809 images for use in learning and verification . Based on these results, the average accuracy of the road sign recognition was calculated using the following equations (1) - (4).

(One)

(2)

(3)

(4)

Here, #TN, #TP, #FN, and #FP represent the number of true negatives (TN's), true positives (TP's), false negatives (FN's), and false positives (FP's).

Referring to Table 3 below, the accuracy of the road marking recognition according to the present invention was confirmed to be higher than 99.8% by learning using six data sets.

Table 3. Accuracy of road mark recognition (unit:%) #of Testing FA FLA FLRA FRA LA RA Testing 1 ACC 100 100 100 100 99.45 100 F_score 100 100 100 100 99.72 100 Testing 2 ACC 100 100 100 100 99.44 100 F_score 100 100 100 100 99.72 100 Testing 3 ACC 99.91 100 100 100 99.05 100 F_score 99.96 100 100 100 99.52 100 Testing 4 ACC 100 100 100 100 99.02 100 F_score 100 100 100 100 99.51 100 Testing 5 ACC 100 100 100 100 99.12 100 F_score 100 100 100 100 99.56 100 Testing 6 ACC 100 100 100 100 99.41 100 F_score 100 100 100 100 99.70 100 Testing 7 ACC 99.96 100 100 100 99.34 100 F_score 99.98 100 100 100 99.67 100 Testing 8 ACC 100 100 100 99.11 100 100 F_score 100 100 100 99.55 100 100 Testing 9 ACC 100 100 100 100 99.44 100 F_score 100 100 100 100 99.72 100 Testing 10 ACC 99.96 100 100 100 99.48 100 F_score 99.98 100 100 100 99.74 100 Testing 11 ACC 100 100 99.22 100 100 100 F_score 100 100 99.61 100 100 100 Testing 12 ACC 100 100 100 100 99.44 100 F_score 100 100 100 100 99.72 100 Testing 13 ACC 100 99.93 100 100 98.99 100 F_score 100 99.96 100 100 99.49 100 Average ACC 99.99 99.99 99.94 99.93 99.40 100 99.88 Average F_score 99.99 99.997 99.97 99.97 99.70 100 99.94

The road mark recognition method according to an exemplary embodiment of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, and the like, alone or in combination. Program instructions to be recorded on a computer-readable medium may be those specially designed and constructed for the present invention or may be available to those skilled in the computer software arts. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Includes hardware devices specifically configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory, and the like. The above-mentioned medium may also be a transmission medium such as a light or metal wire, wave guide, etc., including a carrier wave for transmitting a signal designating a program command, a data structure and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The embodiments of the present invention have been described above. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

10: Video shooting device
20: Road sign recognition device
30: Road sign

Claims (10)

delete A road mark recognition apparatus comprising:
A road sign image input unit to which the photographed road image is input;
A road sign recognition unit for extracting a road sign image from the input road image and recognizing a road sign by applying a deep learning based convolutional neural network model to the extracted road sign image; And
And a road sign information unit for outputting the recognized road sign,
The road sign recognition unit
An interest area designation unit for designating road surface information as an area of interest in the input road image and removing unnecessary areas that are not designated;
An inverse perspective transform unit for performing inverse perspective transformation on the designated road surface information to generate transformed road surface information;
A road sign detector for detecting a road sign image from the converted road surface information; And
And a circuit-based neural network-based recognition unit for recognizing the road mark image detected by applying the line-by-line neural network model-based recognition method.
3. The method of claim 2,
The circuit-based neural network-based recognition unit
An image input layer for inputting a road cover image of a preset scale;
A feature extraction layer for extracting features by applying a line neural network model to the inputted road mark image; And
And a classification layer that classifies the road sign into a predetermined number of road signs according to the extracted feature.
The method of claim 3,
The feature extraction layer
A road sign recognition apparatus comprising three convolution layer sets and four fully connected layer sets.
5. The method of claim 4,
The three convolution layer sets
A first convolution layer for inputting an image of 265 x 137 x 1 pixels and convoluting using 180 filters of size 5 x 5 x 1 size;
A first max-pulling layer for applying a 3x3 filter to output 65x33x180 pixels;
A second convolution layer for convoluting the 65 × 33 × 180 pixels using 250 filters of size 5 × 5 × 1;
A second max-pulling layer for applying a 3x3 filter to output 15x7x250 pixels;
A third convolution layer for convoluting the 15 × 7 × 250 pixels using 250 filters of size 3 × 3 × 1; And
And a third maximum pulling layer for applying 3 × 3 size filters to output 3 × 1 × 250 pixels.
5. The method of claim 4,
The four fully connected layer sets
A first full connection layer having 750 and 1920 nodes as inputs and outputs, respectively;
A second complete connection layer having 1920 and 1024 nodes as inputs and outputs, respectively;
A third full connection layer having 1024 and 512 nodes as inputs and outputs, respectively; And
512, and 6 nodes as inputs and outputs, respectively.
In a road mark recognition method,
Inputting a road image;
Designating a region of interest in the road image;
Performing inverse perspective transformation on the designated ROI image to generate a projected image;
Extracting a road mark image from the inverted perspective transformed projected image; And
Performing line-by-line neural network-based learning on the road mark image, and recognizing and outputting the detected road mark image.
8. The method of claim 7,
The step of extracting the road mark image from the inverse perspective transformed projected image
A road sign recognition method for extracting a road sign by applying at least one algorithm among Labeling, Morphology, Canny Edge Detection and Sobel Edge in the inverse perspective transformed projected image .
8. The method of claim 7,
Performing line neural network based learning on the road sign image, and recognizing and outputting the detected road sign image,
Classifying the road images included in the data set into a predetermined number of road marking categories;
Normalizing the road image included in the data set and transforming the road image into a predetermined size by an interpolation method;
Learning a line neural network model using a modified road image; And
And recognizing the road mark by applying the learned line neural network model to the detected road mark image.
A computer program for executing the road mark recognition method according to any one of claims 7 to 9 and recorded on a computer-readable recording medium.

Images