JP2020098389A - Road sign recognition device and program thereof - Google Patents

Abstract

To provide a road sign recognition device or the like which can robustly recognize a road sign even at night or even when it is faded due to degradation with age without the need for a special component.SOLUTION: A road sign recognition device 1 comprises: an image input unit 11 in which a taken image is input from an image-taking device 2; an extracted image generation unit 12 which detects a road sign included in the taken image, extracts a part of the taken image, and generates an extracted image; a binarized image generation unit 13 which binarizes the extracted image and generates a binarized image; a content recognition unit 14 which recognizes a content of the road sign from the binarized image; and a road sign tracking unit 15 which tracks a position of the road sign in a following taken image of a frame which follows the taken image from which the content of the road sign is recognized by the content recognition unit 14. The binarized image generation unit 13 determines a threshold value of binarization on the basis of a histogram of a brightness value of a pixel included in a center portion of the extracted image, enabling recognition of a maximum speed sign faded due to degradation with age.SELECTED DRAWING: Figure 1

Description

The present invention relates to a road sign recognition device and the like for recognizing road signs. In particular, the present invention is suitable for recognizing road signs at night and road signs that have faded due to deterioration over time.

The road sign is a display board which is installed on the side or above the road and provides information necessary for road traffic to vehicle drivers and pedestrians. An example of a road sign is a circular maximum speed sign. Since the maximum speed of the vehicle (also referred to as a speed limit or a restricted speed) varies depending on the road on which the vehicle is traveling, the vehicle driver must drive the vehicle while checking the maximum speed from time to time. If the vehicle driver misses the maximum speed sign, an accident may occur due to overspeed. Therefore, in recent years, a driving support system that automatically recognizes a speed sign and presents the recognition result to a vehicle driver has been developed and put on the market.

However, a commercially available driving support system may not operate normally in recognizing road signs at night and road signs that have faded due to deterioration over time. In particular, at night, there is a problem that the fatal accident rate due to a maximum speed violation is extremely high, and a driving support system capable of robustly recognizing road signs at night is desired.

Patent Document 1 discloses a vehicular road sign recognition device for the purpose of surely recognizing the type and content of a road sign that has deteriorated over time in a dark environment such as at night or in a tunnel. The vehicle road sign recognition device described in Patent Document 1 includes an imaging unit having sensitivity in a visible light region and a near-infrared region, and a near-infrared light irradiation unit that irradiates near-infrared light. Based on the brightness distribution of the captured image, it is determined whether or not the road sign is reflected, and when it is determined that the road sign is not reflected, the near-infrared light irradiation unit irradiates the image capturing range of the image capturing unit. When the image of the road sign is determined to be reflected by repeatedly changing the light amount of the near-infrared light and capturing the image again, the content of the road sign is recognized. As a result, an image having a recognizable contrast can be captured even when the brightness of the road sign is reduced due to deterioration over time.

JP, 2016-196233, A

Mikio Takagi, Haruhisa Shimoda (supervised), "New Edition Image Analysis Handbook", The University of Tokyo Press, 2004. Shinichi Murakami, "Image Processing Engineering", Tokyo Denki University Press, 1996.

However, the vehicle road sign recognition device described in Patent Document 1 requires special components such as an imaging unit having sensitivity in the near infrared region and a near infrared light irradiation unit that irradiates near infrared light. .. Moreover, since it is necessary to repeatedly capture images in a dark environment, it is presumed that there may be a delay in recognition of road signs.

The present invention has been made in view of the above-described problems, and an object thereof is to provide a road sign at night or a road sign that has faded due to deterioration over time without requiring a special component. , Robustly recognizable road sign recognition device and the like.

A first invention for achieving the above-mentioned object is a road sign recognition device for recognizing a road sign, comprising: an image input unit for inputting a photographed image; and a portion including the road sign from the photographed image. An extracted image generation unit that extracts a part of the captured image and generates an extracted image; a binarized image generation unit that binarizes the extracted image to generate a binarized image; and the binarized image From the content recognition unit for recognizing the content of the road sign, the binarized image generation unit, the binarization threshold value based on the histogram of the brightness value of the pixels included in the central portion of the extracted image. It is a road sign recognition device characterized by making a decision. According to the first aspect of the present invention, even a road sign that has faded due to aging deterioration can be robustly recognized without requiring special components.

In the first aspect of the invention, the extracted image generation unit performs predetermined preprocessing on the captured image, applies a plurality of edge detection filters to the captured image after preprocessing, and calculates a logical sum to extract edges. An image may be generated, the road sign may be detected from the edge extraction image, and the extraction image may be generated. This improves the accuracy of road sign recognition in a dark environment.

Further, the road sign in the first invention is a maximum speed sign indicating a maximum speed, and the numeral of the maximum speed sign is constituted by a characteristic numeral characterizing the maximum speed sign and a common numeral common to the maximum speed signs. The content recognition unit determines a division position based on a vertical distribution of the binarized image, and divides the binarized image at the division position, thereby including the feature number. A common number image including the image and the common number may be generated. As a result, the characteristic number and the common number can be accurately divided, and the highest speed sign can be accurately recognized.

Further, the content recognition unit in the first invention executes template matching processing between a plurality of common number template images having different image sizes and the common number image, and the position and size of the common number in the common number image. May be specified. As a result, the position and size of the common numeral can be accurately specified, and the highest speed sign can be recognized with high accuracy.

Further, the content recognition unit in the first invention estimates a characteristic numeral existing region in which the characteristic numeral exists in the characteristic numeral image based on a position and a size of the common numeral, and a plurality of characteristics having different numerals. It is also possible to execute template matching processing of the numeral template image and the characteristic numeral existing area. As a result, the characteristic number can be accurately specified, and the maximum speed sign can be recognized accurately.

Further, the image input unit in the first invention inputs the captured images of a plurality of frames in chronological order, and stores the captured images of frames after the captured image in which the content of the road sign is recognized by the content recognition unit. A road sign tracking unit that tracks the position of the road sign in the subsequent captured image, wherein the road sign tracking unit tracks the position of the road sign by applying a particle filter in the subsequent captured image. Yes, for each particle, a first area corresponding to the symbol of the road sign, a second area corresponding to the background of the road sign, and a third area corresponding to a portion excluding the first area and the second area. It is also possible to calculate three likelihoods and estimate the center coordinates of the road sign based on the weighting of the likelihoods. This allows the road sign to be tracked accurately.

The road sign in the first aspect of the invention is a vehicle entry prohibition sign, and the content recognition unit extracts a characteristic portion of the vehicle entry prohibition sign by performing labeling processing on the binarized image. You may do it. This allows the vehicle entry prohibition sign to be accurately recognized.

A second invention is a program for causing a computer to function as a road sign recognition device for recognizing a road sign, the computer including an image input unit for inputting a picked-up image and the road sign from the picked-up image. An extraction image generation unit that extracts a portion including the extracted image, a binarized image generation unit that binarizes the extracted image and generates a binarized image, and a road sign of the road sign from the binarized image. A content recognition unit for recognizing content, wherein the binarized image generation unit determines a binarization threshold value based on a histogram of brightness values of pixels included in the central portion of the extracted image. It is a program for functioning as a device. The road sign recognition device of the first invention can be obtained by installing the second invention in a general-purpose computer.

According to the present invention, it is possible to provide a road sign recognition device that can robustly recognize even a road sign at night or a road sign that has faded due to deterioration over time without requiring special components.

Figure showing the outline of the road sign recognition device Flowchart showing the flow of road sign recognition processing regarding the maximum speed sign A drawing-substitute photograph showing an example of a maximum speed sign, an example of a captured image, and an example of execution of preprocessing Drawing substitute photograph showing execution example of edge extraction processing Drawing substitute photograph showing execution example of road sign detection processing Drawing substitute photograph showing an example of extracted image and binarized image Drawing substitute photograph explaining the binarized image generator Drawing substitute photograph explaining binarized image division processing Drawing substitute photograph explaining template matching process of common numbers Drawing substitute photograph explaining template matching process of characteristic number Flowchart showing the flow of road sign tracking processing Drawing substitute photograph showing an example of particle scattering Drawing substitute photograph explaining the extraction processing of the first area and the second area Diagram illustrating the likelihood calculation process Flowchart showing the flow of road sign recognition processing related to vehicle entry prohibition signs Drawing substitute photograph showing an example of execution of road sign recognition processing related to vehicle entry prohibition sign

FIG. 1 is a diagram showing an outline of a road sign recognition device. The road sign recognition device 1 is a device mounted on the vehicle 100, and is configured, for example, as a part of a driving support system of the vehicle 100 or a car navigation system. The road sign recognition device 1 is connected to an image capturing device 2 whose image capturing direction is in front of the traveling direction of the vehicle 100, and an output device 3 such as a display or a speaker, and is mounted on the vehicle 100 as necessary. ) 4 is also connected.

The road sign recognition device 1 includes a CPU (abbreviation of “Central Processing Unit”) as a control unit, a memory as a main storage unit, an HDD (abbreviation of “Hard Disk Drive”) as an auxiliary storage unit, and a hard disk such as a flash memory. Have wear. The auxiliary storage unit stores an OS (abbreviation of "Operating System"), application programs, data necessary for processing, and the like. The control unit of the road sign recognition device 1 reads the OS and the application program from the auxiliary storage unit, stores them in the main storage unit, controls the other devices while accessing the main storage unit, and executes the processing described later.

The road sign recognition device 1 detects an image input unit 11 that inputs a photographed image from the photographing device 2 and a road sign included in the photographed image, extracts a part of the photographed image, and generates an extracted image. By the unit 12, the binarized image generation unit 13 that binarizes the extracted image and generates the binarized image, the content recognition unit 14 that recognizes the content of the road sign from the binarized image, and the content recognition unit 14. A road sign tracking unit 15 that tracks the position of the road sign in a subsequent captured image in a frame after the captured image in which the content of the road sign is recognized. The road sign recognition device 1 outputs the recognition result by the content recognition unit 14 and the tracking result by the road sign tracking unit 15 to the output device 3, the ECU 4, and the like.

The image capturing device 2 captures moving image data whose image capturing direction is in front of the traveling direction of the vehicle 100, and outputs the image data to the road sign recognition device 1. The captured image is a still image of a frame included in the moving image data. The image input unit 11 of the road sign recognition device 1 inputs captured images of a plurality of frames in chronological order and executes road sign recognition processing and road sign tracking processing described later.

FIG. 2 is a flowchart showing the flow of road sign recognition processing regarding the maximum speed sign. In the processing shown in FIG. 2, the road sign recognition device 1 targets a single captured image as a processing target and recognizes the content of the maximum speed sign included in the captured image.

As shown in FIG. 2, when the image input unit 11 of the road sign recognition device 1 inputs a photographed image from the photographing device 2 (step S1), the extracted image generation unit 12 sets a target range of the process described below (step S1). S2), a predetermined pre-process is executed (step S3).

FIG. 3 is a drawing-substituting photograph showing an example of a maximum speed sign, an example of a captured image, and an example of execution of preprocessing. 3(a) is the maximum speed sign 20, FIG. 3(b) is the photographed image 21, and FIG. 3(c) is the pre-processed image 22.

As shown in FIG. 3A, the maximum speed sign 20 is located inside the background 20a colored in white, the donut-shaped peripheral portion 20b colored in red, and the peripheral portion 20b, and colored in blue. And the numbers 20c and 20d indicating the maximum speed of the road at the installation location. In the process shown in FIG. 2, the maximum speed sign 20 indicates a two-digit number and a maximum speed that is a multiple of 10. In this case, the first digit number 20d is "0" in common in all the maximum speed signs 20, and is defined as a common number. The second digit 20c is a number that characterizes the maximum speed sign 20 and is defined as a characteristic number. The characteristic numbers are, for example, “3”, “4”, “5”, “8” and the like. In other words, in the present embodiment, the number of the maximum speed sign 20 is constituted by the characteristic number that characterizes the maximum speed sign 20 and the common number common to the maximum speed signs 20.

As shown in FIG. 3B, the captured image 21 has a horizontal direction that is substantially parallel to the horizontal direction and a vertical direction that is substantially parallel to the vertical direction. The number of pixels of the captured image 21 is 640 pixels in the horizontal direction and 480 pixels in the vertical direction.

In order to shorten the processing time, the extracted image generation unit 12 sets a straight line extending in the left-right direction as a dividing line, 75% of the upper part is a target range of the later-described process, and 25% of the lower part is not a target of the later-described process. .. Therefore, the number of pixels in the target range is 640 pixels in the horizontal direction and 360 pixels in the vertical direction.

Next, the extracted image generating unit 12 sets a 2×2 size filter in order to smooth the image, and executes a coarse-graining process in which the average value of this filter is set to one new pixel. As a result, the number of pixels in the target range is 320 pixels in the horizontal direction and 180 pixels in the vertical direction.

Next, the extracted image generation unit 12 executes gamma correction processing (see Non-Patent Document 1) in order to increase the brightness of the entire image. This improves the recognition accuracy of road signs in a dark environment.

Next, the extracted image generation unit 12 performs smoothing processing (see Non-Patent Document 1) to remove fine noise, and uniformizes sky color information. The result is the pre-processed image 22 shown in FIG. Compared with the photographed image 21, in the post-processed image 22, the road sign is visible at the lower left, and there are few fine noises.

Returning to the explanation of FIG. Next, the extracted image generation unit 12 executes edge extraction processing to generate an edge extracted image (step S4). In the present embodiment, the extracted image generation unit 12 generates an edge extracted image by applying a plurality of edge detection filters to the preprocessed image 22 and calculating a logical sum.

FIG. 4 is a drawing-substituting photograph showing an execution example of the edge extraction processing. FIG. 4A is an image 23 after applying the Laplacian filter to the post-processing image 22, FIG. 4B is an image 24 after applying the Canny edge filter to the post-processing image 22, and FIG. 4C. 4D is an image 25 obtained by synthesizing the image 23 and the image 24 by OR, and FIG. 4D is an image 26 after performing a predetermined noise removal process on the image 25.

As described above, the peripheral portion 20b of the maximum speed sign 20 is colored red, but color information is largely lost in a dark environment. Therefore, the extraction image generation unit 12 applies the result of applying the Laplacian filter (see Non-Patent Document 1) in which red strongly reacts in a bright environment, and the Canny edge filter (Non-Patent Document 1) in which a contour is not detected in a dark environment. The result of applying (see) is synthesized by logical sum. This improves the accuracy of road sign recognition in a dark environment. However, in the image 24 shown in FIG. 4B, the threshold of the Canny edge filter is set low in order to improve the recognition accuracy of the road sign in a dark environment, so noise other than the contour of the road sign is generated. Are also detected as many edges. Similarly, a lot of noise remains in the image 25 shown in FIG.

The extracted image generation unit 12 performs noise removal processing based on the features of objects and landscapes other than the road signs in order to remove noise on edges other than the contours of the road signs. Specifically, the extracted image generation unit 12 includes (first edge noise determination condition) pixels having a brightness value of less than 10 in a region such as sky, (second edge noise determination condition) telephone pole, building, A pixel that satisfies any of the three conditions, that is, a linear pixel having a connection number of 25 pixels or more as seen in a signboard or the like, (third edge noise determination condition) 1 pixel or an isolated point pixel of 2 pixel connection is removed as noise. .. The result is the image 26 shown in FIG. In the image 26, the road sign to be extracted appears in a circle in the lower left part. Hereinafter, the image 26 will be referred to as an edge extraction image.

Returning to the explanation of FIG. Next, the extracted image generation unit 12 detects a road sign from the edge extracted image and generates an extracted image (step S5). In the present embodiment, the extraction image generation unit 12 removes noise from the edge extraction image based on the contour feature of the road sign (=circular shape for the highest speed sign) and detects the road sign. The road sign detection process according to the present embodiment is not limited to the maximum speed sign and can be applied to any circular road sign.

FIG. 5 is a drawing-substituting photograph showing an execution example of the road sign detection process. FIG. 5A is an image 27 showing the circle detection result by the generalized Hough transform, and FIG. 5B is an image 28 after noise removal from the circle detection result shown in the image 27.

As shown in FIG. 3A, since the contour of the highest speed sign is circular, the extracted image generation unit 12 detects a candidate for the highest speed sign using the generalized Hough transform useful for detecting a circle. The generalized Hough transform is a method that transforms the coordinates in the image into another parameter space and detects the figure in that space, and can be estimated even when the pixels that make up the circle are missing. , Is useful for detecting the contour of the highest speed sign (for details, see Non-Patent Document 1). The extracted image generation unit 12 detects a circle having a voting number of 6 to 22 pixels by the generalized Hough transform as a candidate for the maximum speed marker. The result is the image 27 shown in FIG.

In FIG. 5A, the portions marked with A to K are detected as circles. A and B are circular road signs. However, like C to K, the edges extracted by the influence of a signboard, a streetlight, or the headlight of a vehicle other than a circular road sign become noise, and are detected as a circle. In the present embodiment, the extracted image generation unit 12 focuses on the characteristics of a circular road sign, regards a circle that meets any of the following determination conditions as noise, and removes it from candidates for the maximum speed sign.

(First Circle Noise Judgment Condition) The extracted image generation unit 12 judges that a part of the circle is noise when it passes outside the image and removes it. It should be noted that the circle forming the road sign may pass outside the image in some cases, but it is excluded from the target in that frame, and detection is attempted in the subsequent frames.

(Second circle noise determination condition) A circular road sign appears as a small circle at the bottom of the image. Therefore, the extracted image generation unit 12 determines, as a noise, a circle having a radius of 10 pixels or more among the circles passing through the lower 1/3 of the image and removes it.

(Third Circle Noise Judgment Condition) A circular road sign appears as a large circle in the upper part of the image. Therefore, the extracted image generation unit 12 determines, as a noise, a circle having a radius of 10 pixels or less among the circles passing through the upper 1/3 of the image and removes it.

(Fourth Circle Noise Judgment Condition) A circular road sign has a feature that the number of votes forming a circle is high. However, since the estimated circles are sorted by the number of votes, even if the density of the votes forming the circles is low, noise with a large radius is detected as a higher circle than a circular road sign. Therefore, the extracted image generation unit 12 determines that a circle whose number of votes is 60% or less of the circumference is noise and removes it.

(Fifth Circle Noise Judgment Condition) Since the generalized Hough transform reacts with white pixels, an infinite number of circles are detected when a white area is input. On the other hand, in the contour detection image, the inside of the circular road sign is characterized by a small number of white pixels. Therefore, the extracted image generation unit 12 determines that a circle in which 70% or more of the pixels existing in the circle are white pixels is noise and removes it.

(Sixth Circle Noise Judgment Condition) Circular road signs are mostly included in the top 15 places when sorted by the number of votes. Therefore, the extracted image generation unit 12 determines that the circles at the 16th place and below are noise and removes them.

The result after removing the noise is the image 28 shown in FIG. Compared to image 27, in image 28 only circle A and circle B remain as candidates for the highest speed sign. The circle A is a maximum speed sign, while the circle B is a circular road sign different from the maximum speed sign. At this stage, it is sufficient to detect a circular road sign, not limited to the maximum speed sign. The extracted image generating unit 12 sets the circle A and the circle B in the image 28 as candidates for the maximum speed marker, projects a rectangular area including each circle on the captured image 21, extracts the area, and generates an extracted image.

Returning to the explanation of FIG. Next, the binarized image generation unit 13 binarizes the extracted image to generate a binarized image (step S6). The process of step S6 targets all extracted images. In the present embodiment, the binarized image generation unit 13 determines the binarization threshold value based on the histogram of the brightness values of the pixels included in the central portion of the extracted image.

FIG. 6 is a drawing-substituting photograph showing an example of the extracted image and the binarized image. FIG. 6A shows an extracted image 29 after preprocessing, and FIG. 6B shows a binarized image 30 obtained by binarizing the extracted image 29. The binarized image generation unit 13 executes gamma correction processing (see Non-Patent Document 1) with a gamma value of 2.0 in order to brighten the brightness of the entire image. Next, the binarized image generation unit 13 uses the bilinear interpolation method (see Non-Patent Document 1) to expand the image size so that the height becomes 100 pixels while maintaining the aspect ratio. FIG. 6A shows the extracted image 29 after performing these preprocessing.

Next, the binarized image generation unit 13 divides the extracted image 29 into 5×5, and divides the central portion 29a (see FIG. 6A), which is a central grid in both the left-right direction and the vertical direction, into the central portion 29a. The threshold value for binarization is determined using the Otsu's discriminant analysis method (see Non-Patent Document 1) for only pixels, and the entire extracted image 29 is binarized. Specifically, the binarized image generation unit 13 binarizes the extracted image 29 using a threshold value that maximizes the separability of the histogram of the brightness value of the central portion 29a. The binarized image 30 in FIG. 6B is the result of binarizing the extracted image 29 by executing this process. In the binarized image 30, the two numbers “4” and “0” indicating the maximum speed can be clearly confirmed as a set of white pixels.

Here, the central portion of the extracted image is a region having a certain area including at least the center point of the extracted image, including a part of the numeral indicating the maximum speed, and having a donut shape colored in red. A region that does not include the peripheral portion is desirable. For example, the central portion of the extracted image is not limited to the central grid when divided into 5×5 described above, but is the central grid when divided into 3×3, 3×5, 5×7, and the like. May be. Further, for example, the shape of the central portion of the extracted image is not limited to a quadrangle, and may be a polygon other than a quadrangle, or a circle or an ellipse.

FIG. 7 is a drawing-substituting photograph for explaining the binarized image generation unit. The reason why the binarized image generation unit 13 in the present embodiment limits the pixels used for determining the binarization threshold value to the central portion of the image will be described with reference to FIG. 7. 7(a) is an extracted image 31 of the highest speed sign that has not faded due to aged deterioration, FIG. 7(b) is an extracted image 32 of the highest speed sign that has been faded due to aged deterioration, and FIG. 7(c) is a binarized image. The binarized image 33 in the case where the pixels used for determining the threshold value are the entire extracted image 32, and FIG. 7D shows the case where the pixel used for determining the threshold value for binarization is limited to the central portion of the extracted image 32. It is the binarized image 34.

In the extracted image 31, the brightness value of the pixel 31a in the donut-shaped peripheral portion colored in red is “47”, and the brightness value of the background pixel 31b inside the peripheral portion is “84”. On the other hand, in the extracted image 32, the brightness value of the pixel 32a in the peripheral portion is "99", and the brightness value of the pixel 32b in the background inside the peripheral portion is "82". In this way, the magnitude relationship of the brightness value between the peripheral portion and the background is reversed depending on the degree of deterioration over time. If the pixels used for determining the binarization threshold value for the extracted image 32 are the entire image, as shown in FIG. 7C, the two-digit number indicating the maximum speed cannot be recognized. The converted image 33 is generated. On the other hand, if the pixels used for determining the threshold value for binarization are limited to the central portion of the image in the extracted image 32 as in the present embodiment, as shown in FIG. The binarized image 34 in which the digit number can be recognized as a set of white pixels is generated. In this way, by limiting the pixels used for determining the binarization threshold value to the central portion of the image, it becomes possible to recognize the maximum speed sign that has faded due to deterioration over time.

Returning to the explanation of FIG. Next, the content recognizing unit 14 divides the binarized image into left and right parts, and generates a characteristic number image including a characteristic number and a common number image including a common number (step S7). The process of step S7 targets all binarized images. In the present embodiment, the content recognition unit 14 determines the division position based on the vertical distribution of the binarized image, and divides the binarized image into the left and right at the division positions, thereby Generate a common number image.

FIG. 8 is a drawing-substituting photograph for explaining the binarized image division processing. 8A shows a binarized image 30 to be processed, FIG. 8B shows a mask image 35, and FIG. 8C shows a binarized image 30 and a binarized image 36 in which the mask image 35 is combined. The marginal distribution graph 37 and FIG. 8D are a characteristic numeral image 42 and a common numeral image 43 generated by dividing the binarized image 36.

In step S5, since a rectangular area including a circle is generated as an extracted image, the area outside the circle is included. Then, the area outside the circle becomes noise in the processing described later. Therefore, in order to eliminate the influence of noise, the content recognition unit 14 performs mask processing (see Non-Patent Document 1) on the binarized image 30 using the mask image 35 to generate a binarized image 36. To do. The mask image 35 has white pixels inside the contour of the highest speed sign and black pixels outside.

Next, the content recognition unit 14 determines the division position based on the vertical distribution of the binarized image 36, and divides the binarized image 36 into the left and right at the division position to obtain the characteristic numeral image. And generate a common digit image. Here, the vertical distribution of the binarized image 36 is the number of pixel values (=black pixels or white pixels) of one of the binarized images 36 counted in the vertical direction at each coordinate in the horizontal direction. (See Non-Patent Document 2). In the marginal distribution graph 37 of FIG. 8C, the black portion corresponds to the number of black pixels, and the white portion corresponds to the number of white pixels. In the present embodiment, the content recognition unit 14 corresponds to the background of the black pixel (=maximum speed sign) within the range of the central portion when the binarized image 36 is divided into three equal parts by lines extending in the vertical direction. The position where the number of pixels is the maximum is the division position.

In FIG. 8C, a dotted line 38 indicates a line that divides the binarized image 36 into two equal parts, and a dotted line 39 indicates a line that passes through the division positions. If the binarized image 36 is simply divided into two, it may happen that the binarized image 36 is divided at a position passing through the characteristic numeral or the common numeral as shown by the dotted line 38. On the other hand, in the present embodiment, it is possible to divide at a position passing only the background located between the characteristic number and the common number. Therefore, as shown in FIG. 8D, in the present embodiment, the white pixels corresponding to the characteristic numeral 40 are present only in the characteristic numeral image 42, and the white pixels corresponding to the common numeral 41 are only the common numeral image 43. , And the characteristic numeral 40 and the common numeral 41 can be accurately divided even if the image cannot be taken from the front of the sign.

Returning to the explanation of FIG. Next, the content recognizing unit 14 executes template matching processing of the common numeral image to identify the position and size of the common numeral (step S8). The process of step S8 targets all common number images. In the present embodiment, the content recognizing unit 14 executes template matching processing between a plurality of common numeral template images having different image sizes and the common numeral image to specify the position and size of the common numeral. As a result, the position and size of the common numeral can be accurately specified regardless of the size of the road sign included in the captured image.

FIG. 9 is a drawing-substituting photograph for explaining the template matching process for common numerals. FIG. 9A relates to the template matching process, and FIG. 9B relates to the specification of the position and size of the common numeral. The template matching process (see Non-Patent Document 1) is a process in which the template image is moved within the search range to find the best matching place, that is, the place where the degree of similarity is maximum. In the present embodiment, the content recognition unit 14 uses the template matching process of the image correlation method (see Non-Patent Document 1) in which the correlation coefficient is the similarity. However, the present invention is not limited to the template matching process of the image correlation method, and can be appropriately selected from known template matching processes.

As shown in FIG. 9A, the content recognizing unit 14 stores, for example, a common numeral template image 44 having an image size of 100 in height in advance, and uses the bilinear interpolation method (see Non-Patent Document 1). The height is reduced from 100 to 24 in 2 steps, and template matching processing is performed for each image size. As a result, the template matching process is executed 39 times, and the same number of places on the common numeral image 43 where the degree of similarity is maximized is obtained. In FIG. 9A, there is only one common number image 43, but if there are a plurality of candidates for the maximum speed sign as in the example of FIG. 5B described above, there are also a plurality of common number images 43. Will be done. The content recognition unit 14 performs the template matching process on all the common number images 43. For example, when there are two common numeral images 43, the template matching process is performed 2×39=78 times, and the same number of places on the common numeral image 43 having the highest degree of similarity are obtained. If the maximum similarity among these is greater than or equal to the threshold value, the content recognition unit 14 determines that the original captured image includes the highest speed sign, and executes the process described below.

In FIG. 9A, the 27×46 common numeral template image 44a has the maximum similarity of 0.92. As shown in FIG. 9B, the content recognition unit 14 determines the image size of the common numeral template image 44a, that is, the width w (=horizontal length) and the height h (=vertical length). get. Further, the content recognizing unit 14 acquires the coordinates regarding the place on the common number image 43 having the highest similarity. Since the width w and the height h of the common numeral are known, the content recognition unit 14 may acquire the coordinates of one vertex of the rectangle surrounding the common numeral, for example, the coordinates 45 of the upper left vertex. The content recognition unit 14 identifies the coordinates 45, the width w, and the height h as the position and size of the common numeral included in the common numeral image 43.

Returning to the explanation of FIG. Next, the content recognizing unit 14 executes template matching processing of the characteristic number image to identify the characteristic number (step S9). Then, the content recognition unit 14 recognizes the content of the road sign (step S10). In the present embodiment, the content recognizing unit 14 estimates the characteristic numeral existing area in which the characteristic numeral exists in the characteristic numeral image based on the position and the size of the common numeral, and the plurality of characteristic numeral template images having different numerals. And template matching processing of the characteristic numeral existing area is executed. Thereby, the characteristic number can be accurately specified.

FIG. 10 is a drawing-substituting photograph for explaining template matching processing of characteristic numbers. FIG. 10A relates to the specification of the position and the size of the characteristic number, and FIG. 10B relates to the template matching process of the characteristic number. As shown in FIG. 10A, the content recognition unit 14 calculates the distance d from the coordinate 45 to the left end of the common number image 43. Next, the content recognizing unit 14 identifies the coordinate 45 at the right end of the characteristic numeral image 42, which has the same vertical position as the coordinate 45. Next, the content recognizing unit 14 extracts a rectangular characteristic number existing area 47 having a coordinate of 46 at the upper right corner and a width of d+w and a height of h. Next, the content recognizing unit 14 converts the image sizes of a plurality of template images, which are stored in advance and have different numbers, so as to have the height h by using the bilinear interpolation method, and sets the template images for the characteristic numbers. Then, the content recognizing unit 14 executes the template matching process of the plurality of characteristic number template images having different numbers and the characteristic number existing area 47.

In the example shown in FIG. 10B, a template image 48a for characteristic numbers having a number "3", a template image 48b for characteristic numbers having a number "4", and a template image 48c for characteristic numbers having a number "5" are shown. Has been done. For example, when information about whether the road on which the vehicle 100 is traveling is a general road or an expressway is obtained from the car navigation system or the like, the content recognition unit 14 uses only a part of the characteristic numeral template image 48. , Template matching processing of characteristic numbers may be executed. For example, the content recognizing unit 14 is a template image 48 for characteristic numbers such as numbers “3”, “4”, and “5” if it is an ordinary road, and “7” and “8” if it is an expressway. It is also possible to use only the template image 48 for characteristic numbers such as "." The content recognizing unit 14 recognizes the content of the highest speed sign based on the number of the characteristic number template image 48 having the highest degree of similarity. In the example shown in FIG. 10B, the maximum similarity is 0.76 in the characteristic numeral template image 48b with the numeral "4", so the content recognition unit 14 determines that the maximum speed indicated by the maximum speed sign is "40 km." /H”.

As described above, in the present embodiment, the content recognition unit 14 accurately divides the common numeral image and the characteristic numeral image, and based on the position and size of the common numeral included in the common numeral image, the characteristic numeral image Since the template matching process is executed, the maximum speed sign can be recognized with high accuracy. In particular, in the process of the extraction image generation unit 12, even if the elements of the edges that form the extracted circle change depending on the sign or the noise cannot be completely removed by the noise removal process, the maximum speed sign is robust. Can be recognized.

The common numeral template image 44 and the characteristic numeral template image 48 described above are preferably extracted from moving image data obtained by photographing the highest speed sign actually installed on the road. This improves the accuracy of the template matching process.

Further, in the above description, the maximum speed sign 20 is assumed to indicate a two-digit number and a maximum speed that is a multiple of 10, but in the present invention, a three-digit number such as "100" or "110" is used. Moreover, even if the maximum speed is a multiple of 10, it is applicable. In this case, the number in the first digit is “0” commonly in all the maximum speed signs 20, and is defined as a common number. The second and third digits are numbers that characterize the maximum speed and are defined as characteristic numbers. The content recognition unit 14 determines whether it is a two-digit number or a three-digit number based on the vertical distribution of the binary image, and based on the vertical distribution of the binary image. 1 or 2 division positions are determined, and the binary image is divided into left or right two or three divisions at the one or two division positions to generate one or two characteristic numeral images and one common numeral image. To do.

FIG. 11 is a flowchart showing the flow of road sign tracking processing. The road sign tracking unit 15 tracks the position of the road sign by applying the particle filter in the subsequent captured image of the frame after the captured image in which the content of the road sign is recognized by the content recognition unit 14. A particle filter is a state estimation algorithm for a nonlinear, non-Gaussian state space model based on Bayes' theorem. The road sign tracking unit 15 includes, for each particle, a first region corresponding to the symbol of the road sign, a second region corresponding to the background of the road sign, and a third region corresponding to a portion excluding the first region and the second region. The three likelihoods of the area are calculated, and the center coordinates of the road sign are estimated by weighting the likelihoods. Here, the symbol is a figure, a picture, or the like to which characters or meanings are attached, and indicates the content of the road sign. Below, similar to the above, the maximum speed sign will be described as an example of the road sign.

As shown in FIG. 11, the road sign tracking unit 15 inputs the recognition result of the road sign recognition process (step S21). The recognition result is specified in various images generated in the road sign recognition process, that is, a captured image, an extracted image, a binarized image, a common numeral image, a characteristic numeral image, and the like, in addition to the content of the road sign. The coordinates of various positions are included.

Next, the road sign tracking unit 15 scatters particles around the road sign recognized by the road sign recognition process in the captured image of the current frame (step S22). For example, the scatter range is a square area centered on the center point of a recognized circular sign and having a side length equal to 2.5 times the radius of the circular sign, and the scatter number is 3 of the area in the scatter range. Let's divide. Let R be the radius of a recognized circular road sign, L be the length of one side on which particles are scattered, and N be the number of dispersed particles, then L=2.5R, N=0.3×L squared. ..

FIG. 12 is a drawing-substituting photograph showing an example of particle scattering. FIG. 12A shows an example of the spraying range, and FIG. 12B shows an example of the spraying result. The example shown in FIG. 12 is a case where a circular road sign with a radius of 50 pixels is recognized by the road sign recognition process. As shown in FIG. 12A, the length L of one side on which the particles are scattered is L=2.5×50 pixels=125 pixels. Further, the number N of scattered particles is N=0.3×125 pixels×125 pixels≈4,688 pixels. The road sign tracking unit 15 uniformly disperses particles so that there is no bias by using a random number whose seed value is the target frame and coordinates.

Returning to the description of FIG. Next, the road sign tracking unit 15 corresponds to the first area (=area corresponding to the sign of the road sign) and the second area (=background of the road sign) in the binarized image generated by the road sign recognition processing. Area) to be extracted (step S23), and the center of gravity of the color information (=RGB value) of the first area and the second area is calculated (step S24).

FIG. 13 is a drawing-substituting photograph for explaining the extraction processing of the first area and the second area. 13A shows an example of an extracted image, FIG. 13B shows an example of a binarized image, and FIG. 13C shows an example of a reverse image. The extracted image 51 shown in FIG. 13A is an image including the highest speed sign. The binarized image 52 shown in FIG. 13B is an image generated from the extracted image 51. In the following processing, the rectangular area image 53 surrounding the common numeral is targeted. The rectangular area surrounding the common numeral can be specified based on the result of the template matching processing of the common numeral image. The inverted image 54 shown in FIG. 13C is an image in which the pixel values of the rectangular area image 53 are inverted. The road sign tracking unit 15 extracts the black pixels of the reverse image 54 as the first area and the white pixels of the reverse image 54 as the second area.

Next, the road sign tracking unit 15 acquires the color information of the pixels forming the first area and the second area from the image having the color information (=RGB value) such as the extracted image 51, and the expression (1) and The center of gravity is calculated for each of the RGB values according to equation (2), and color information C ₁ representative of the first area and color information C ₂ representative of the second area are set.

Returning to the description of FIG. Next, the road sign tracking unit 15 calculates the likelihood of each particle, and each particle is a region corresponding to the first region, the second region, and the third region (=the region excluding the first region and the second region). Which region of () belongs to (step S25). In the present embodiment, the road sign tracking unit 15 sets the color information C ₁ representing the first area, the color information C ₂ representing the second area, and the color information C _{p of the} particle of interest as three-dimensional coordinates. The distance is calculated, and the belonging degrees δ _{1 to} δ ₄ regarding each region are calculated according to the equations (3) to (6). Here, δ ₁ is a degree of belonging close to the first area, δ ₂ is a degree of belonging far from the first area, δ ₃ is a degree of belonging close to the second area, and δ ₄ is a degree of belonging far from the second area. Further, the road sign tracking unit 15 calculates the likelihood μ ₁ to μ ₄ for each area according to the equations (7) to (10). Here, μ ₂ is the likelihood of the first region, μ ₃ is the likelihood of the second region, and μ ₁ and μ ₄ are the likelihoods of the third region. Then, the road sign tracking unit 15 sets the maximum _one of the likelihoods μ _{1 to} μ ₄ as the estimation result of the particle of interest.

FIG. 14 is a diagram illustrating the likelihood calculation process. 14A is a likelihood estimation table, FIG. 14B is a graph of the degree of belonging, FIG. 14C is a likelihood estimation table of the first calculation example, and FIG. 14D is a likelihood of the second calculation example. It is a degree estimation table. The likelihood estimation table shown in FIG. 14A shows the correspondence between the membership degrees δ _{1 to} δ ₄ and the likelihoods μ _{1 to} μ ₄ . FIG. 14B is a graph of the membership degrees δ ₁ and δ ₃ .

Here, a first calculation example of the likelihood will be described. In the first calculation example, the color information C ₁ representing the first area is (R, G, B)=(78, 51, 47), and the color information C ₂ representing the second area is (R, G, B). )=(104, 87, 84), and the color information C _{p of the} particle of interest is (R, G, B)=(79, 54, 39). At this time, Euclidean distance d ₁₂ between C ₁ and C ₂ =57.8, Euclidean distance d _1p between C ₁ and C _p =8.6, and Euclidean distance d _2p between C ₂ and C _p =61.1. When the membership degrees δ _{1 to} δ ₄ and the likelihoods μ _{1 to} μ ₄ are calculated according to the equations (3) to (10), the likelihood estimation table of the first calculation example shown in FIG. 14C is obtained. Since the maximum value of the likelihoods μ _{1 to} μ ₄ is “0.94” of the likelihood μ ₂ , the target particle in the road sign tracking unit 15 corresponds to the first region, that is, the sign of the road sign. Presumed to belong to the area.

Next, a second example of calculating the likelihood will be described. In the second calculation example, the color information C ₁ representing the first area is (R, G, B)=(78, 51, 47), and the color information C ₂ representing the second area is (R, G, B). )=(104, 87, 84), and the color information C _{p of the} particle of interest is (R, G, B)=(110, 93, 93). At this time, the Euclidean distance d ₁₂ between C ₁ and C ₂ =57.8, the Euclidean distance d _1p between C ₁ and C _p =70.0, and the Euclidean distance d _2p between C ₂ and C _p is 12.4. When the membership degrees δ _{1 to} δ ₄ and the likelihoods μ _{1 to} μ ₄ are calculated according to the equations (3) to (10), the likelihood estimation table of the second calculation example shown in FIG. 14D is obtained. Since the maximum value of the likelihoods μ _{1 to} μ ₄ is “0.91” of the likelihood μ ₃ , the road sign tracking unit 15 indicates that the particle of interest corresponds to the second region, that is, the background of the road sign. Presumed to belong to the area.

Returning to the description of FIG. Next, the road sign tracking unit 15 estimates the center coordinates of the road sign in the captured image of the current frame (step S26). In the present embodiment, the road sign tracking unit 15 estimates the center coordinates of the road sign based on the weighting of the likelihoods of the first area, the second area, and the third area calculated in step S25. Specifically, the road sign tracking unit 15 calculates the central coordinates O _{(x, y)} of the road sign according to the equation (11). This makes it possible to accurately estimate the center coordinates of the road sign and accurately track the road sign.

Next, the road sign tracking unit 15 confirms whether or not particles are present in the captured image of the current frame (step S27). If the determination in step S27 is Yes, the road sign tracking unit 15 maintains the recognition result of the road sign recognition process (step S28), scatters particles on the captured image of the next frame (step S29), and the process of step S25. Repeat from. The distribution range of particles in step S29 is determined in the same manner as in step S22, with the center coordinates of the road sign estimated in step S26 as the center. On the other hand, when the determination in step S27 is No, the road sign tracking unit 15 ends the road sign tracking process.

FIG. 15 is a flowchart showing a flow of road sign recognition processing regarding a vehicle entry prohibition sign. In the process shown in FIG. 15, the road sign recognition device 1 targets a single captured image as a processing target and recognizes the content of the vehicle entry prohibition sign included in the captured image. The processing of steps S31 to S36 is the same as the processing of steps S1 to S6 shown in FIG.

FIG. 16 is a drawing-substituting photograph showing an execution example of road sign recognition processing regarding a vehicle entry prohibition sign. 16A shows a vehicle entry prohibition sign 60, FIG. 16B shows an extracted image 61, and FIG. 16C shows a binarized image 62. As shown in FIG. 16( a ), the vehicle entry prohibition sign 60 is composed of a background 60 a colored red and a rectangular symbol 60 b colored white and having a longitudinal direction. In the example shown in FIG. 16B, the binarized image generation unit 13 divides the extracted image 61 into five parts in the left-right direction and three parts in the up-and-down direction, and is a central grid in both the left-and-right direction and the up-and-down direction. The threshold value for binarization is determined by using the Otsu's discriminant analysis method only for the pixels of the central portion 61a, and the entire extracted image 61 is binarized. As a result, the binarized image 62 shown in FIG. 16C is generated.

Next, the content recognizing unit 14 executes labeling processing (see Non-Patent Document 1) on the binarized image, extracts the characteristic portion of the road sign (step S37), and recognizes the content of the road sign (step S38). ). In the present embodiment, the content recognition unit 14 extracts a region having the maximum area as a characteristic portion of the road sign as a result of the labeling process. As a result, the symbol 60b of the vehicle entry prohibition sign 60 can be accurately extracted, and the vehicle entry prohibition sign 60 can be accurately recognized. Then, the content recognition unit 14 divides the width (=horizontal length) of the characteristic portion of the road sign by the height (=vertical length) within a range of 2.5 to 4.0. If there is, it is recognized as a vehicle entry prohibition sign.

As described above, the road sign recognition device 1 according to the present embodiment can robustly recognize a road sign at night or a road sign that has been faded due to deterioration over time without requiring special components. .. In particular, the extracted image generation unit 12 can apply the same processing to various circular road signs. Further, the binarized image generation unit 13 and the road sign tracking unit 15 can apply the same processing to road signs of various shapes, not limited to a circle. On the other hand, the content recognition unit 14 can recognize the maximum speed sign and the entry prohibition sign that are particularly important at night in the present embodiment, although the contents of the processing differ depending on the type of road sign.

Hereinafter, an example of the road sign recognition processing regarding the maximum speed sign by the road sign recognition device 1 of the present invention will be described. The moving image data is taken by a vehicle traveling on an ordinary road and a highway in Akita Prefecture at night. As the photographing device, α7S manufactured by Sony Corporation was used. The shooting conditions are 640×480 pixels, 256 gradations of RGB, 30 fps, ISO sensitivity of 20,000 to 51,200, shutter speed of 1/2000 to 1/3200 seconds, aperture value of F2.8, and auto focus. did. The conditions for the target maximum speed sign are (1) the number of the maximum speed sign can be visually confirmed, (2) part of the tree or building does not overlap the maximum speed sign, and can be visually confirmed ( 3) There are three lanes, one being the target lane. For the purpose of solving the problem in the existing technology, the light of the vehicle 100 was always set to a low beam for photographing.

Data set A of moving image data includes 312 maximum speed signs (30 km/h limit: 120, 40 km/h limit: 103, 50 km/h limit: 89) installed on general roads. It was Further, the data set B contained 62 maximum speed signs (80 km/h limit) installed on the expressway. The extraction was successful if the highest speed sign was extracted well in one frame or more, and the recognition was successful if the number in the highest speed sign and the output result in the recognition process match in one frame or more.

For data set A, of the 312 highest velocity markers, 307 (98.4%) were successfully extracted and 303 (97.1%) were successfully recognized. The road sign recognition device 1 of the present invention is either a maximum speed sign that is discolored due to aging, a maximum speed sign that is installed in a dark environment, or a maximum speed sign that is installed in a bright environment such as an urban area. I was able to recognize it well.

For dataset B, of the 62 highest velocity markers, 60 (96.8%) were successfully extracted and 59 (95.2%) were successfully recognized. Since it was raining when the data set B was photographed, the wiper of the vehicle 100 was operated to obtain it with no noticeable water droplets adhering to the windshield. However, the road sign recognition device 1 of the present invention has the maximum speed. The label was well recognized.

As described above, the road sign recognizing device 1 of the present invention is capable of recognizing a road sign at night which is difficult to recognize by the conventional technique or a road sign which is faded due to deterioration over time, and further, in an environment where the vehicle 100 is traveling at high speed. I was able to recognize the road signs and road signs when it was raining. That is, the road sign recognition device 1 of the present invention was able to robustly recognize the road sign under various environments.

The preferred embodiments of the road sign recognition device and the like according to the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious to those skilled in the art that various changes or modifications can be conceived within the scope of the technical idea disclosed in the present application, and it is obvious that they also belong to the technical scope of the present invention. Understood.

1…………Road sign recognition device 2…………Shooting device 3…………Output device 4…………ECU
11: Image input unit 12: Extracted image generation unit 13: Binary image generation unit 14: Content recognition unit 15: Road sign tracking unit 100: Vehicle

Claims (8)

A road sign recognition device for recognizing road signs,
An image input section for inputting captured images,
An extracted image generation unit that detects the road sign included in the photographed image, extracts a part of the photographed image, and generates an extracted image;
A binarized image generation unit that binarizes the extracted image and generates a binarized image,
A content recognition unit that recognizes the content of the road sign from the binarized image,
Equipped with
The road sign recognition device, wherein the binarized image generation unit determines a binarization threshold value based on a histogram of brightness values of pixels included in a central portion of the extracted image. The extracted image generation unit performs a predetermined preprocessing on the captured image, and generates an edge extracted image by calculating a logical sum by applying a plurality of edge detection filters to the captured image after preprocessing, The road sign recognition device according to claim 1, wherein the road sign is detected from the edge extraction image and the extracted image is generated. The road sign is a maximum speed sign indicating the maximum speed,
The number of the maximum speed sign is constituted by a characteristic number characterizing the maximum speed sign and a common number common to the maximum speed signs,
The content recognition unit determines a division position based on a vertical distribution of the binarized image, and divides the binarized image at the division position to obtain a characteristic numeral image including the characteristic numeral, The road sign recognition device according to claim 1 or 2, wherein a common numeral image including the common numeral is generated. The content recognition unit performs template matching processing between a plurality of common numeral template images having different image sizes and the common numeral image, and specifies the position and size of the common numeral in the common numeral image. The road sign recognition device according to claim 3. The content recognition unit estimates a characteristic number existing area in which the characteristic number exists in the characteristic number image based on the position and size of the common number, and a plurality of characteristic number template images with different numbers and the characteristic number. The road sign recognition device according to claim 4, wherein template matching processing of a numeral existing area is executed. The image input unit inputs the captured images of a plurality of frames in chronological order,
A road sign tracking unit that tracks the position of the road sign in a subsequent captured image in a frame after the captured image in which the content of the road sign is recognized by the content recognition unit;
The road sign tracking unit tracks the position of the road sign by applying a particle filter in the subsequent captured image, and for each particle, a first area corresponding to the symbol of the road sign, the road sign The three coordinates of the second area corresponding to the background and the third area corresponding to the portion excluding the first area and the second area are calculated, and the center coordinates of the road sign are calculated based on the weighting of the likelihood. The road sign recognition device according to claim 1, wherein the road sign recognition device estimates. The road sign is a vehicle entry prohibition sign,
The road sign recognition device according to claim 1 or 2, wherein the content recognition unit extracts a characteristic portion of the vehicle entry prohibition sign by executing a labeling process on the binarized image. .. A program for causing a computer to function as a road sign recognition device that recognizes a road sign,
The computer,
An image input section for inputting captured images,
An extraction image generation unit that extracts a portion including the road sign from the captured image and generates an extraction image,
A binarized image generation unit that binarizes the extracted image and generates a binarized image,
A content recognition unit that recognizes the content of the road sign from the binarized image,
Equipped with
A program for causing the binarized image generation unit to function as a road sign recognition device that determines a binarization threshold value based on a histogram of brightness values of pixels included in a central portion of the extracted image.