JP2007140828A - Sign recognition method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a traffic sign recognition method recognizing a traffic sign at as early as possible, and accurately recognizing even a traffic sign appearing and disappearing by a roadside tree or the like. <P>SOLUTION: In this traffic sign recognition method recognizing the traffic sign included in sequentially arranged respective static images S obtained by photographing a front part in a travel direction of a vehicle by a camera 11, the first tracking image T including only the traffic sign is extracted from the first static image S, the tracking image T is temporarily stored as a distinction image D, the second and succeeding tracking image T is extracted from the second and succeeding static image S, a new distinction image D obtained by composing the tracking image T and the previously temporarily stored distinction image D, is temporarily stored, and the traffic sign present ahead of the travel direction of the vehicle is recognized on the basis of characteristic information CI extracted from the distinction image D. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a sign recognition method for accurately recognizing a traffic sign ahead of a vehicle in the traveling direction from a moving image V obtained by photographing the front of the vehicle in the traveling direction with a camera.

  The vehicles targeted by the present invention are mainly automobiles (including two-wheeled vehicles and four-wheeled vehicles; the same applies hereinafter) or trains, but vehicles on land or water that need to recognize traffic signs in the direction of travel are roughly The present invention can be applied.

  Patent Document 1 discloses a road marking recognition device that extracts and recognizes traffic signs such as road markings and traffic signs ahead of the direction of travel of a vehicle from a moving image V obtained by capturing the forward direction of the vehicle with a camera. is doing. This Patent Document 1 has a problem of reducing the processing burden when recognizing a traffic sign and recognizing the traffic sign even when only a part of the traffic sign can be photographed as the automobile advances. A specific recognition device for road marking, etc., combines an image processing basic unit that extracts a still image S (one frame image) from a moving image V, an image processing unit that extracts a predetermined image from the still image S, and the extracted predetermined image. An image data identification unit that obtains composite data, and an image recognition unit that compares the composite data with specific image data in a database and outputs whether the two data match or not (Patent Document 1, claim 1 and others). reference).

  The reduction of the processing load is solved by processing a predetermined image obtained by dividing the still image S by the image processing unit. If a predetermined image necessary for traffic sign recognition is processed, the processing load is naturally reduced as compared to processing the entire still image S (see paragraph [0091] in Patent Document 1). Further, even when only a part of the traffic sign can be captured, the point of recognizing the traffic sign is solved by matching the composite data obtained by synthesizing the predetermined image extracted by the image data identification unit with the database. That is, by using the predetermined image acquired earlier as a component and adding the component to a missing part of the newly acquired predetermined image, complete composite data used for recognition is obtained (Patent Document 1, paragraph [0093] or [ 0094]). From this, it can be said that the feature of Patent Document 1 is that the still image S extracted from the moving image V is divided and processed.

JP 2003-123197 A

  As in Patent Document 1, the purpose of recognizing traffic signs is to inform the driver of restriction information ahead in the direction of travel. From this, it is desirable to recognize traffic signs at the earliest possible time, that is, to recognize traffic signs at a point farther from the car, and to be able to accurately recognize traffic signs that are visible and hidden by street trees and the like.

  First, when it sees about the point which recognizes a traffic sign as early as possible, it is not clear only by the content which patent document 1 discloses. Patent Document 1 discloses that composite data is obtained by synthesizing predetermined images extracted from still images S. However, the composite data is only an image that reproduces the entire traffic sign using predetermined data as a component. The resolution of the composite data (the number of pixels included in the image, hereinafter the same) appears to depend on the resolution of the still image S. From this, if Patent Document 1 uses a camera with a high resolution, it seems that the resolution of the composite data is also increased and the traffic sign can be recognized at a relatively early point. However, this has a problem of increasing the cost of the camera. . Further, even if a high-resolution camera is used, the resolution of the predetermined image and thus the composite data is not necessarily increased. According to Patent Document 1, it is considered difficult to recognize traffic signs at an early point.

  Further, although Patent Document 1 does not touch on the point of accurately recognizing traffic signs that are visible and hidden by roadside trees and the like, composite data is formed by combining predetermined images extracted by dividing still images S. By excluding predetermined images including roadside trees, it is considered that composite data that is relatively easy to recognize can be formed and traffic signs can be recognized more accurately. However, when a traffic sign is always visible and hidden by a roadside tree or the like, only a predetermined image including the roadside tree or the like can be obtained, and a predetermined image that can be recognized as a component of composite data cannot be prepared. As a result, it is considered that the recognition of traffic signs by composite data must be reduced.

  Thus, in order to recognize a traffic sign at the earliest possible time or to accurately recognize a traffic sign that is visible and hidden by a roadside tree, the recognition method of Patent Document 1 is still used even if a high-resolution camera is used. Is not enough. Therefore, even if the resolution of the camera is low, we studied to develop a recognition method for recognizing traffic signs at the earliest possible time and for accurately recognizing traffic signs that are visible and hidden by street trees.

  What has been developed as a result of the study is a sign recognition method for recognizing traffic signs included in still images S arranged in time series in which the front of the vehicle in the traveling direction is captured by a camera, and includes only traffic signs from the first still image S The first tracking image T is extracted, the first tracking image T is temporarily stored as the discrimination image D, the second and subsequent tracking images T are extracted from the second and subsequent still images S, and the second and subsequent tracking images are extracted. A new discriminating image D obtained by combining T and the discriminating image D temporarily stored earlier is temporarily stored, and a traffic sign ahead of the traveling direction of the vehicle based on the feature information CI extracted from the new discriminating image D Is a traffic sign recognition method. The still image S used in the present invention may be a still image S extracted in units of frames from a moving image V taken by a camera in front of the vehicle in the traveling direction. As described above, the present invention may be either a still image camera or a moving image camera as long as the still image S in the forward direction of the vehicle can be acquired.

  In the present invention, a traffic sign is not recognized for each tracking image T extracted from the still image S, but is generated by combining the tracking image T repeatedly extracted and the discrimination image D temporarily stored. A traffic sign is recognized using the new discrimination image D. The resolution of the new discrimination image D is increased in accordance with the tracking image T to be synthesized. The feature information CI included in the contour and color of the traffic sign of the new discrimination image D is obtained by accumulating the feature information CI of each tracking image T. For this reason, the amount of feature information CI obtained from the discrimination image D is larger than the amount of feature information CI obtained from the tracking image T of the same resolution. As a result, the use of a new discrimination image D increases the recognition of traffic signs. be able to. This means that the traffic sign can be recognized using the discrimination image D at an early stage where the traffic sign cannot be directly recognized from the tracking image T (for example, a stage where the car and the traffic sign are far apart). This brings about the effect of realizing early recognition.

  The amount of feature information CI of an image is proportional to the resolution of the image. However, an enlarged discrimination image D ′ described later increases the number of apparent pixels, but does not increase the resolution, and thus does not increase the amount of feature information CI. This is because the increased pixels are only complemented by the original pixels, and the increased pixels do not produce new feature information CI. From this, the synthesis of the tracking image T and the discrimination image D not only adds the feature information CI of the tracking image T to the feature information CI that the discrimination image D originally has but also the features of the tracking image T with respect to the increased pixels. Information CI is newly added to increase the amount of feature information CI of the combined discrimination image D.

  The still image S is a panoramic image in front of the direction of travel of the car at a specific point in time, and includes scenery in addition to traffic signs to be recognized. The shooting interval of the still image S and the extraction interval from the moving image V may be arbitrary and may be equal intervals or unequal intervals. This photographing interval or extraction interval is preferably longer than the total processing time required for extracting the tracking image T, generating the discrimination image D, and recognizing the traffic sign from the discrimination image D. Conversely, if the processing time is sufficiently short, for example, the time interval inversely proportional to the traveling speed of the automobile, that is, if the automobile is moving at high speed, the shooting interval or extraction interval is shortened, and conversely if the automobile is moving at low speed. It is conceivable to increase the shooting interval or the extraction interval.

  The tracking image T is an image having a size or shape including the traffic sign selected in the still image S. For the purpose of reducing the processing burden, the tracking image T has a shape that follows the outer shape of the traffic sign or the traffic sign has the maximum outer shape. A rectangular or square image is preferred. The tracking image T may be searched for and extracted from the entire still image S for each still image S. In this case, not only is the burden of extracting the tracking image T heavy, but each still image S is also extracted. It is difficult to guarantee the identity of the tracking image T extracted from the. From this, it is preferable to limit the search range of the second and subsequent still images S based on the tracking image T extracted from the first still image S, and extract the second and subsequent tracking images T from the limited search range. . Henceforth, the present invention is called “tracking” image T in the sense of extracting the same traffic sign from each still image S. As a method for extracting the tracking image T, various conventionally known methods can be used.

  The feature information CI is information representing a geometric feature sufficient to identify a traffic sign, for example. For example, Japan's regulatory sign is basically a red figure on a white background, and a blue figure is added if necessary, so if the discrimination image D is binarized with white, red and blue, respectively, the shape of the white background, Three types of specific color extraction images consisting only of the shape of a red graphic and the shape of a blue graphic can be generated. Then, if the contour vector of each specific color extraction image is extracted as feature information CI and compared with the feature information CI consisting of the contour vector for each traffic sign stored in the database, the traffic sign included in the discrimination image D is discriminated. it can. As described above, the feature information CI only needs to be able to determine a traffic sign by matching with a database, and various known methods can be used as a method of generating the feature information CI.

  The present invention is characterized in that a discrimination image D in which the feature information CI of the past tracking image T is accumulated is generated. However, the second and subsequent tracking images T and the discriminating image D temporarily stored earlier naturally have different sizes and resolutions. Therefore, even if they are simply superimposed, the characteristic information CI is weakened instead. There is a risk of it. Therefore, the new discrimination image D calculates the size ratio R between the second and subsequent tracking images T and the discrimination image D temporarily stored previously, and the temporarily stored discrimination image D is calculated as the size ratio R. The discriminating image D ′ enlarged by the above is generated, and the tracking image T is combined with the weight W, and the enlarged discriminating image D ′ is combined with the weight (1-W).

  The second and subsequent tracking images T and the discrimination image D temporarily stored in advance have different sizes and resolutions because the vehicle travels forward in the traveling direction while including the same traffic sign. From this, in order to synthesize the tracking image T and the discrimination image D, it is necessary to match one to the other size. In this case, the tracking image T may be reduced by the size ratio R and combined with the discrimination image D. However, the reduction of the tracking image T unnecessarily impairs the feature information CI included in the tracking image T. Further, as described above, the synthesis of the tracking image T and the discrimination image D has a meaning of newly adding the feature information CI of the tracking image T to the increased pixels of the enlarged discrimination image D ′. From this, it is understood that it is reasonable to combine the discrimination image D ′ enlarged by the size ratio R with the tracking image T.

  In a basic case where the vehicle is traveling horizontally and straight in the direction of travel, the second and subsequent tracking images T and the temporarily stored discrimination image D are images including the same traffic sign, and the discrimination image Since D is generated from the past tracking image T, it can be considered that the height and width are in a similar relationship with the same ratio. In this case, the size ratio R of both can be calculated by simply comparing the height and width of each image or the specific dimension r (see later) of each image. Here, the “specific dimension r” is a specific dimension r of a traffic sign included in the tracking image T or the discrimination image D, for example, an estimated radius of a circular sign or an estimated length of one side of a square sign or a rhombus sign. means. However, when the tracking image T or the discrimination image D is extracted following the outline of the traffic sign, the radius of the tracking image T is set to the specific dimension r, and the tracking image T is extracted with the outline of the traffic sign being maximized. The length of one side of the tracking image T may be the specific dimension r.

  Also, when the car is turning in the direction of travel or moving up and down, such as when the road is curved or the car is turning at an intersection, the ratio of height and width changes depending on the angle of the car with respect to the traffic sign For example, a corrected tracking image T is obtained when the tracking image T is viewed from the front, and the height and width of the corrected tracking image T and the temporarily stored discrimination image D or the specific dimension r of each image are obtained. The size ratio R may be calculated by comparison. In any case, each image may be binarized in order to facilitate comparison of the height, width, or specific dimension r of the image. Whether the comparison control is height, width, or specific dimension r, and whether or not to use intermediate processing such as binarization may be determined by the capability of hardware or software to which the present invention is applied.

  Next, since the tracking image T and the discrimination image D have different resolutions, the present invention combines the tracking image T with a weight W and the enlarged discrimination image D ′ with a weight (1-W) ratio, that is, a weighted average. did. Here, in order to make better use of the feature information CI of the high-resolution tracking image T, it is conceivable that the weight W is set to 0.5 or more, and the influence of the tracking image T on the new discriminating image D obtained by synthesis is increased. . However, if the weight W is increased too much, the synthesized discriminant image D is not different from the tracking image T itself, and the feature information CI accumulated in the discriminating image D on which the tracking image T is superimposed is evaluated relatively low. End up. Therefore, the weight W is a value that monotonously increases with respect to the specific dimension r in the tracking image T in a range smaller than 0.5, and the tracking image T is evaluated according to the size of the specific dimension r while the tracking image T having the weight W is evaluated. In comparison, the enlarged discrimination image D ′ with the weight (1-W) is evaluated relatively high, and the feature information CI of the enlarged discrimination image D ′ can be effectively used for traffic sign recognition. Specifically, the relative value of the tracking image T with respect to the enlarged discrimination image D ′ is lower than 0.5, and the weight W is set to 0.01 to 0.40, preferably 0.05 to 0.35.

The weight W is a function W (r) having a specific dimension r in the tracking image T as a variable,
dW (r) / dr> 0, d ² W (r) / dr ² <0
It is good to be. Although the function W (r) increases as the specific dimension r increases, it is sufficient that the increase amount decreases. Specifically, the logarithmic function log (r) and the power function r ^t (0 <t <1 ). As described above, the specific dimension r is, for example, an estimated radius of a circular sign or an estimated length of one side of a square sign or a rhombus sign.

A preferable function W (r) is expressed by the following formula 1.
A ₁ : Amplification factor A ₂ : Threshold value for specific dimension r A ₃ : Threshold value for weight W Where A ₁ is 0.05 to 0.40 and A ₂ is 4.0 when the weight W is within the range of 0.05 to 0.35 described above. to 8.0, a ₃ is preferably set in a range of from 0.01 to 0.30.

The characteristic information CI included in the tracking image T when the traffic sign is quite far away is small, rather there is a lot of unnecessary information (noise), and it is not preferable to combine it with the enlarged discrimination image D ′. The threshold A ₂ for the specific dimension is a threshold for excluding the small tracking image T that is considered to have a lot of unnecessary information from the combination with the enlarged discrimination image D ′ based on the specific dimension r. As a result, log (r−A ₂ ) may become too small. Therefore, by increasing the value of log (r−A ₂ ) by the amplification factor A ₁ and further subtracting the threshold value A ₃ for the weight W. The excess increment by the amplification factor A ₁ is adjusted. Here, since the weight W must be a positive value less than 1, if the specific dimension r is smaller than the threshold A ₂ or if W (r) becomes a negative value by subtraction of the threshold A _{3 from} the weight W, The tracking image T may be temporarily stored as a new discrimination image D without combining the tracking image T with the enlarged discrimination image D ′.

  The present invention has an effect that a traffic sign can be recognized at an early point in time. These are the effects obtained by recognizing the traffic sign based on the discrimination image D in which the feature information CI of the tracking image T extracted in the past is accumulated. Since the discrimination image D stores the characteristic information CI of the tracking image T extracted in the past, the recognition accuracy of the traffic sign using the discrimination image D is higher than that of the tracking image T of the same resolution. Even if the traffic sign cannot be recognized from the tracking image T, the new determination image D obtained by synthesizing the tracking image T can recognize the traffic sign. This leads to the effect of highly accurate recognition of traffic signs at an early point.

  Further, by recognizing the traffic sign based on the discrimination image D in which the characteristic information CI of the tracking image T extracted in the past is accumulated, an effect of making it possible to recognize even a traffic sign that appears and hides in a roadside tree or the like can be obtained. Conventionally, the same kind of traffic sign recognition method recognizes a traffic sign using only the acquired still image S or the tracking image T extracted from the still image S. Often, traffic signs could not be recognized when obstacles such as trees overlapped. However, since the discrimination image D used in the present invention for the traffic sign recognizes the traffic sign using the discrimination image D obtained by synthesizing the tracking images T extracted in the past, the obstacle included in each tracking image T In addition to being able to reduce the effects of traffic, the feature information CI is increased in areas where there are no obstacles, enabling traffic signs to be recognized.

  In addition, the present invention proposes a mode of using the extracted tracking image T, and can be used in combination with various conventionally known traffic sign recognition methods. That is, conventionally known various methods can be used as the method for acquiring the still image S, the method for extracting the tracking image T, and the recognition method using the discrimination image D in the present invention. This means the versatility of the present invention. Thus, the present invention has an effect of establishing a practical traffic sign recognition method by being applied to various conventionally known traffic sign recognition methods.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of a traffic sign recognition system to which the present invention is applied, FIG. 2 is a usage state reference diagram showing an automobile 5 equipped with the traffic sign recognition system of this example, and FIG. 3 is a traffic sign recognition system. 4 is a reference diagram showing an example of the still image S, FIG. 5 is a reference diagram showing the tracking image T extracted from the still image S, and FIG. 6 is a discrimination image D with respect to the tracking image T. FIG. 7 is a reference diagram schematically showing the process of synthesizing the tracking image T and the enlarged discrimination image D ′, and FIG. 8 is a reference image (a). It is a reference figure showing the contour trace image (d) which traced the contour vector image (b) (c) extracted from a specific color extraction image, and a contour vector continuously. In this example, a traffic sign recognition system for recognizing a traffic sign 6 ahead of the automobile 5 in the traveling direction from a moving image V obtained by photographing the front of the automobile 5 in the traveling direction with a camera.

  For example, as shown in FIG. 1, the traffic sign recognition system includes a camera 11, an in-vehicle computer 2, a hard disk 3, and a display 4. The camera 11 is a moving image camera that captures the moving image V. For example, as shown in FIG. 2, the camera 11 is mounted on the roof of the automobile 5 and constantly photographs a landscape in front of the traveling direction of the automobile 5. The hard disk 3 is an external storage device that permanently stores a large amount of data, and stores a database composed of feature information CI of the traffic sign 6 and map data to be displayed on the display 4. Instead of the hard disk 3 that is a magnetic external storage device, an external storage device such as a DVD-ROM that is an optical external storage device may be used. For example, data is transmitted from a server outside the automobile via a wireless connection Internet. May be obtained sequentially.

  The in-vehicle computer 2 includes a processing unit 26 including a CPU and a memory 25. The processing unit 26 can be divided into functional still image extraction means 21, tracking image extraction means 22, image synthesis means 23, and sign recognition means 24 by executing a processing program that implements the processing procedure based on the present invention. it can. The processing program is stored in the hard disk 3, for example, and is read out and executed by the processing unit 26 of the in-vehicle computer 2 by the activation of the traffic sign recognition system. Each means configured as software by executing a processing program has versatility that is easy to cope with specification changes. However, when shortening the processing time and downsizing the apparatus configuration, each means is configured as hardware. May be.

  The still image extracting means 21 takes in the moving image V from the camera 11 and extracts a still image S in frame units from the moving image V. The tracking image extraction unit 22 takes in the still image S from the still image extraction unit 21 and extracts the tracking image T including the traffic sign 6 from the still image S. Instead of the camera 11, the still image S may be directly taken into the tracking image extraction unit 22 using a camera 12 (see FIG. 1) such as a still image camera. If this tracking image T is the first tracking image T extracted from the first still image S, it is temporarily stored in the memory 25. The image synthesizing unit 23 extracts the second and subsequent tracking images T extracted from the second and subsequent still images S from the tracking image extracting unit 22, and the tracking image T and the discrimination image D temporarily stored in advance. Are generated to generate a new discrimination image D, and this discrimination image D is temporarily stored in the memory 25. The sign recognition unit 24 takes in the new discrimination image D generated from the image synthesis unit 23, extracts the feature information CI from the discrimination image D by various conventionally known methods, and stores the feature information CI in the hard disk 3. The traffic sign 6 is recognized against the feature information CI of the database. Then, the display 4 takes out the image data of the traffic sign 6 recognized by the sign recognition means 24 from the database of the hard disk 3 and synthesizes it with, for example, map data and displays it.

The traffic sign recognition system of this example recognizes the traffic sign 6 in the moving image V according to the processing procedure seen in FIG. For convenience, a case where one traffic sign 6 appears in the moving image V will be described. When a plurality of traffic signs 6 appear (see FIGS. 4 and 5), the following processing procedures are executed in parallel. First, at the beginning of the processing procedure, the counter n is reset, and the still image extracting means 21 extracts the first still image S ₁ from the moving image V captured by the camera 11 (see FIG. 1, the same applies hereinafter). tracking image extracting means 22 from S ₁ is to extract the first tracking image T _1. Here, if the first tracking image T ₁ is not extracted, this processing procedure is terminated, the counter n is reset again, and the next processing procedure is started.

  For the extraction of the tracking image T, various conventionally known methods can be used. However, a method of extracting an image including only a traffic sign from each still image S and the image including the same traffic sign over all the still images S are extracted. And a method of extraction. First, as a method for extracting an image including only a specific traffic sign from each still image S, an extraction method using area limitation (“Extraction of circular traffic signs using specific color discrimination and area limitation (Daisuke Matsuura, Hitoshi Yamauchi , Hiromitsu Takahashi) ”, Journal of the Institute of Electronics, Information and Communication Engineers, D-II, Vol. J85-D-II, No. 6, pp. 1075-1083, June 2002). The extraction method using this area limitation is 1) a specific color extraction procedure using the RGB subtraction method, and 2) a search area that specifies an image area where a traffic sign can exist and reduces the target area of the subsequent shape determination procedure. It consists of a limited procedure and 3) a shape discriminating procedure used for traffic signs, such as a circle and a diamond.

  The RGB subtraction method used for the specific color extraction procedure subtracts the green component having a high contribution rate to the luminance component from the red component to subtract the red component or the yellow component, and similarly subtracts the green component from the blue component to obtain the blue component. It is a technique to extract. Here, the red component and the yellow component are distinguished by the magnitude of the value obtained by subtracting the green component from the original red component, and the red component is distinguished when the value is large, and the yellow component is distinguished when the value is small. . This RGB subtraction method has a small amount of computation compared to generally used HSL conversion, etc., and can extract specific colors at high speed, and when the sky component is photographed in white by subtracting the green component, for example However, there is an advantage that the white color can be removed.

  Since the specific color extracted images of the red component, the green component, the blue component, and the yellow component extracted by the above RGB subtraction method are grayscale images having gradations, each of them is subjected to threshold processing to form a binarized image. As can be seen in FIG. 4, it is used in subsequent mitigation and shape discrimination procedures. Here, the specific color extraction image may contain minute noise composed of isolated pixels of about 1 to 2 pixels. Such noise is controlled by a simple filter process that removes or adds the target pixel by threshold processing of the number of effective pixels included in the 3 × 3 pixel centered on the target pixel, while suppressing a significant increase in processing time. It is good to remove.

  Mitigation uses the fact that traffic signs are represented by combinations of specific shapes. In a region including a traffic sign, a specific shape representing the traffic sign has an effective pixel number equal to or greater than a certain value on a horizontal line or a vertical line. From this, a horizontal line or a vertical line whose number of effective pixels is less than the predetermined value can be excluded from the area for searching for traffic signs = search area. Further, if the width or height of the search area is smaller than the size of the traffic sign to be extracted, no traffic sign can exist in the search area, and therefore the search area can be excluded. If such exclusion from the search area is performed on all of the specific color extraction images, the range for searching for traffic signs is limited. Here, with one exclusion from the search area, it is not always possible to leave only the traffic sign to be extracted due to the influence of other traffic signs and similar color objects existing in the specific color extraction image. For this reason, the exclusion from the search area may be performed twice or more, and the excess area may be removed.

  The shape discrimination procedure discriminates a circular figure or a polygon figure which is the outer shape of the traffic sign for the search area limited by the reduction. For example, in the case of a traffic sign of a circular figure, the presence of a circular figure is obtained by provisional determination of center coordinates by edge search of the search area and threshold processing of the number of pixels on the circumference of the radius of the circular figure centered on the center coordinates. Determine. In the case of traffic signs with polygonal figures such as rhombus figures, central coordinates are provisionally determined in the same way as with circular figures, and the center position is determined by scanning from the center coordinates in the vertical or horizontal direction to search for edges. After that, the outer periphery and inner periphery of the outer portion based on the distance between the obtained edges are scanned to count the number of effective pixels, and the presence of a polygonal figure is determined by threshold processing. Here, in order to select a signboard having the same shape as the traffic sign, the area ratio of each specific color in the limited search area is calculated, and when the ratio that cannot be a traffic sign is calculated, the search area Is excluded.

Thus, the specific color extraction procedure, the search area limited procedures and by detection using area limiting consisting shape determination procedure, as seen in FIG. 5, for example, an image comprising only traffic signs from the still image S _n, Can be extracted as the tracking image T _n . However, to extract the tracking image T _n via each same procedure for all the still image S _n, to the processing time is too long, tracking image T _n includes the same traffic signs extracted from the still image S _n Whether it is an image or not must be determined separately. Therefore, as a method for extracting the image including the same traffic sign over the entire still image S _n, CONDENSATION ( "CONDENSATION - Conditional Density Propagation for Visual Tracking (Michael Isard, and Andrew Blake) ""International Journal of Computer Vision, Vol 29, No. 1, pp. 5-28, 1998.), using the first extracted tracking image T ₁ , the second and subsequent tracking images T _n (n is an integer of 2 or more, and so on) ) Should be extracted efficiently.

  CONDENSATION assumes a multidimensional normal distribution centered on the parameter value in the image extracted immediately before when a specific area extracted from an image can be expressed by multiple parameters. This is a method of determining the parameter value having the highest evaluation value from the parameter candidate group as the parameter value at that time. Here, in order to use CONDENSATION, it is necessary to appropriately evaluate the parameter candidate group. From this, in the case of the present invention in which the tracking image T is extracted from the still image S, the shape of the marker such as a circle and a rhombus and a specific color determined at the initial extraction stage are set as parameter values.

For example, in the case of a circular sign mainly made of red, the parameters (X, y) and radius (specific dimensions) r necessary for expressing a circle that is the shape of the sign can be used as parameters estimated by CONDENSATION. . In this case, each parameter value can be expressed as (xk, yk, rk), where k is the number of the parameter value generated by CONDENSATION. The evaluation value Ek of the generated parameter value is such that P _R (x, y) is the red pixel value at the center coordinates (x, y) and n is the circumference since the outer periphery of the target sign is a red circle. The number of divisions, c, can be calculated by the following formula 2 as a weight for correcting the edge strength.

When the evaluation value Ek calculated from Equation 2 is a large value, there is a possibility that the search area represented by the parameter value that is the basis for calculating the evaluation value Ek is the outermost peripheral portion of the red circle. Estimated to be high. From this, if the search area having the highest evaluation value Ek is extracted for each still image S, the second and subsequent tracking images T _n including the same traffic signs as the first extracted tracking image T ₁ can be obtained.

The first tracking image T ₁ extracted from the first still image S ₁ in this way is temporarily stored in the memory 25 as a new discrimination image D. In this example, the sign recognition means 24 generates feature information CI from the discrimination image D from the viewpoint of standardization of the processing procedure, and the feature data in the database of the hard disk 3 is matched with the feature information CI to recognize traffic signs. Try. In the present invention, since the recognition accuracy is improved using the discrimination image D obtained by superimposing the past tracking images T, the next processing procedure may be performed without extracting the feature information CI from the discrimination image D. A method for extracting the feature information CI from the discrimination image D will be described later.

When the discrimination image D is obtained from the first tracking image T _{1 in this} way, the counter n is incremented by 1 and the processing procedure returns to the beginning, and the still image extracting means 21 from the moving image V captured by the camera 11 is the second still image. The image S ₂ is extracted, and the tracking image extraction means 22 extracts the second tracking image T ₂ from the still image S ₂ . Then, the image synthesizing unit 23 synthesizes the previously stored discrimination image D and the second tracking image T ₂ to obtain a new discrimination image D, and the sign recognition unit 24 further obtains the new discrimination image D. based on the characteristic information CI extracted from Do discrimination image D, and determine the traffic sign including the tracking image T _2.

In order to synthesize the temporarily stored discrimination image D and the second tracking image T ₂ , the image synthesis means 23 first enlarges the discrimination image D according to the tracking image T ₂ . In this example, as shown in FIG. 6, for example, when the tracking image T is extracted, the specific ratio (radius in the circular label) r used for CONDENSATION is compared to calculate the size ratio R. According to the ratio, the height and width of the temporarily stored discrimination image D ′ are enlarged to obtain an enlarged discrimination image D ′. Thereby, the resolution of the enlarged discrimination image D ′ is apparently the same as that of the tracking image T. However, as described above, the enlarged discrimination image D ′ includes pixels that do not have the feature information CI. Yes. The process of assigning the feature information CI of the tracking image T to this pixel is the composition of the present invention, and this composition is performed by enlarging the discrimination image with the weight W as shown in FIG. It is a weighted average obtained by combining D ′ with a weight (1−W). Since the range and calculation of the weight W have already been described, the synthesis of the tracking image T and the enlarged discrimination image D ′ is performed in units of pixels of each image, preferably pixels of a characteristic color image obtained by converting each image. Synthesize in units.

  A synthesis of the tracking image T and the enlarged discrimination image D ′ will be schematically described. For the convenience of explanation, the tracking image T represents the traffic sign feature information CI with a precision of 70% (a solid line portion of the tracking image T in FIG. 7) and the traffic sign feature information CI with a precision of 30%. It is assumed that it is composed of a pixel b (a broken line portion of the tracking image T in FIG. 7). Further, the enlarged discrimination image D ′ of the discrimination image D in which the first tracking image T is temporarily stored as it is is considered to have less feature information CI than the tracking image T with a considerable probability. The pixel c (in FIG. 7, the solid line part of the tracking image T) and the pixel d (in FIG. 7, the broken line part of the tracking image T) expressing the traffic sign feature information CI with the accuracy of 20% It is assumed that The weight W calculated from the specific dimension r is assumed to be 0.10.

  In this case, the portion where the pixel a of the tracking image T and the pixel c of the enlarged discrimination image D ′ are combined is 70% × 0.10 + 50% × 0.90 = 52% (the bold line portion of the discrimination image D in FIG. 7). The portion where the pixel b and the pixel c are combined is 30% × 0.10 + 50% × 0.90 = 48% (the solid line portion of the discrimination image D in FIG. 7), and the portion where the pixel a and the pixel d are combined is 70. % × 0.10 + 20% × 0.90 = 25% (the solid line portion of the discrimination image D in FIG. 7), and the portion where the pixel b and the pixel d are combined is 30% × 0.10 + 20% × 0.90 = 21% (FIG. 7 Among these, a new discrimination image D representing the feature information CI with the accuracy of the solid line portion of the discrimination image D is obtained. Due to the influence of the tracking image T, for example, the feature information CI seems to be slightly weakened partially, such as a portion where the pixel b and the pixel c are combined, but the obtained new discrimination image D has more features as a whole. In addition to the information CI, it can be seen that the characteristic information CI is averaged as a whole.

  Now consider the case of approaching a traffic sign. In this case, in the same manner as in the above example, the tracking image T represents the pixel a (the solid line portion of the tracking image T in FIG. 7) representing the traffic sign feature information CI with 80% accuracy, and the traffic sign feature information CI is 40. It is assumed that the pixel b is represented by a pixel b represented by% accuracy (a broken line portion of the tracking image T in FIG. 7). In addition, since the feature information CI that the past tracking image T has is accumulated in the discrimination image D, the enlarged discrimination image D ′ is a pixel c that represents the feature information CI of the traffic sign with an accuracy of 80%. (A solid line portion of the tracking image T in FIG. 7) and a pixel d (the broken line portion of the tracking image T in FIG. 7) that represents the traffic sign feature information CI with 60% accuracy. The weight W calculated from the specific dimension r is assumed to be 0.30.

  In this case, the portion where the pixel a of the tracking image T and the pixel c of the enlarged discrimination image D ′ are combined is 80% × 0.30 + 80% × 0.70 = 80% (the bold line portion of the discrimination image D in FIG. 7). The portion where the pixel b and the pixel c are combined is 40% × 0.30 + 80% × 0.70 = 68% (the solid line portion of the discrimination image D in FIG. 7), and the portion where the pixel a and the pixel d are combined is 80 % × 0.30 + 60% × 0.70 = 66% (the solid line portion of the discrimination image D in FIG. 7), and the portion where the pixel b and the pixel d are combined is 40% × 0.30 + 60% × 0.70 = 54% (FIG. 7 Among these, a new discrimination image D representing the feature information CI with the accuracy of the solid line portion of the discrimination image D is obtained. This new discrimination image D also has more feature information CI as a whole, and it can be seen that the feature information CI is averaged as a whole.

  Various known methods can be used as a method for discriminating traffic signs based on the new discrimination image D synthesized in this way. For example, template matching found in many image recognitions may be used. The template matching needs to enlarge or reduce a new discrimination image D according to the template image. Unlike the present invention, the template matching corresponds to the tracking image T to be combined with the enlarged or reduced new discrimination image D. Since there is no image, the new discriminating image D that has been enlarged or reduced merely dilutes the feature information CI, and as a result, the recognition accuracy of the traffic sign may be lowered. If advanced interpolation processing is used in order to improve the image quality of the new discriminated image D that has been enlarged or reduced, another problem of increasing the processing time arises.

  Thus, in this example, the outline of a traffic sign in the discrimination image D is extracted without enlarging or reducing the synthesized new discrimination image D, and the contour vector obtained by tracking the extracted contour is characterized. A method for discriminating a traffic sign included in the tracking image T by matching feature information CI including a contour vector of a traffic sign stored in the database in advance as information CI (“Recognition of a road sign by tracking a contour vector (Jin Yamauchi , Hiromitsu Takahashi) ”, Journal of the Institute of Image Information and Television Engineers, Vol. 57, No. 7, pp. 847-853, July 2003.).

  In general, a Sobel filter, a Laplacian filter, or the like is used to extract a contour of a specific shape in an image. However, the contour extraction using these filters only obtains contour points of a specific shape, and it is impossible to determine whether there is a region with a large pixel value, that is, a specific color, inside or outside the contour. Not suitable for. In view of this, the Sobel filter was improved and a method of extracting a vector indicating a certain positional relationship with respect to a region having a large pixel value (hereinafter referred to as a vector extraction method for convenience of explanation) was used. Since the contour of a specific shape can be extracted by connecting the vectors, the vector can be referred to as a “contour vector”.

The contour vector (x _{i, j} , y _{i, j} ) in this vector extraction method is derived by the following equations 3 and 4 at the pixel at the coordinates (i, j).
Here, (ui, j, vi, j) is a pixel value at coordinates (i, j).

  The contour vector has a region with a large pixel value on the right side orthogonal to the vector direction. From this, if it is possible to extract a contour that reaches the initial position by connecting a plurality of calculated contour vectors clockwise, it can be determined that there is a region with a large pixel value inside the contour, and conversely, a plurality of calculated contours If the contour reaching the initial position can be extracted by connecting the vectors to the left, it can be determined that there is a region with a large pixel value outside the contour. If this is applied to traffic signs, it will be understood whether there is a specific color such as red inside the contour connected with the contour vector, or conversely, there is a specific color outside the contour. Will be understood to facilitate.

  Actually, since it is impossible to calculate a contour vector by capturing only pixels along a contour of a specific shape, a contour vector is calculated for all pixels of the discrimination image D, and the direction and length of the obtained total contour vector. Are compared with each other to select a contour vector to be connected, and only a contour having a specific shape is extracted. In this case, the tracking of the contour vector forming the contour is preferably started from a pixel having a large contour vector. Here, since the tracking of the contour vector is performed in units of pixels, the direction of the contour vector in each pixel is normalized to the direction of adjacent pixels, that is, eight directions. In this way, the operation of moving from the start point to the pixel in the normalized direction is repeated until the start point is reached again, the outside of the discrimination image D is reached, or the length of the contour vector becomes 0, and the contour vectors are connected. Get a vector sequence.

  In the present invention, since a traffic sign composed only of a specific color is a recognition target, the discrimination image D is converted into a specific color image composed only of red, blue, etc., and the contour vector is tracked for each specific color image. It is good to carry out. For example, as shown in FIG. 8, taking the reference image (FIG. 8 ()) of the traffic sign prohibiting overtaking as an example, the contour vector image of the red extracted image is FIG. The vector image is represented as FIG. 8C, and a contour trace image obtained by connecting and tracing the respective contour vectors can be obtained as shown in FIG. 8D. Here, each contour vector of the red extracted image and the blue extracted image expresses the direction of the vector with shading.

  Traffic signs are commonly used for different types of simple specific shapes, and there is a high possibility that a more specific type of traffic sign cannot be determined only by contour vectors. From this, when recognizing a traffic sign, it is only necessary to know in which position in the discrimination image D the vector sequence obtained by tracking the contour vector exists. In this case, when the red specific shapes having different sizes are concentrically like a circular sign, it is necessary to distinguish each specific shape. Therefore, it is not the average position as the vector sequence, but the direction of each contour vector. It is good to get the average position.

  When a vector sequence representing a contour is considered to be a concatenation of codes that proceed continuously in the same direction, for example, a contour vector that faces another direction once or twice in a vector sequence is a noise or a blur caused by normalization. Can see. In order to remove this noise or blur, the vector sequence representing the contour is regarded as a “run-length code” that is a set of the normalized contour vector direction and the continuous length of the contour vector, and only the run length is considered. Decrease it a few times. The vector sequence that has been subjected to noise removal and the like thus becomes the feature information CI of the discrimination image D, and the vector sequence is used for discrimination of traffic signs.

  The feature information CI of the traffic sign that compares the feature information CI extracted from the discrimination image D is generated in advance from the created reference image of the traffic sign, and stored in an external storage, and a database is constructed. deep. The recognition of the traffic sign is performed by collating the feature information CI extracted from the discrimination image D with the contour vector sequence stored in the database. Specific collation is determination of coincidence between the average position and direction of the vector sequence. The coincidence determination of the vector string is performed by the same method as the search for the partial character string. Here, when the discrimination image D is still small, it is expected that the details of the traffic sign are crushed and the contour is short-circuited, or an unnecessary contour generated due to partial concealment by an obstacle or the like is included, The extracted vector sequence may be divided into small units, and matching may be determined for each divided unit. Here, since the start point of the extracted contour vector cannot be expected to match the start point of the contour vector of the reference image, the contour vector on the reference image side is searched in an annular shape. In this way, when the vector direction sequence length extracted from the discrimination image D as the feature information CI and a certain ratio or more of the vector direction sequence length of the reference image coincide with each other, it is determined that the traffic sign is the reference image. Recognition for image D is completed.

  Since the recognition of the traffic sign based on the new discriminating image D is repeatedly performed, for example, since the recognition results are the same and continue for a certain number of times, it can be determined that the traffic sign can be correctly recognized. In this case, it is no longer necessary to recognize the discrimination image D, and processing such as extraction of the tracking image T, synthesis of the tracking image T and the discrimination image D, and recognition based on the discrimination image D may be omitted. Further, if the tracking image T is not extracted from the middle, it is determined that the traffic sign that has been the object of recognition has passed, the determination image D that has been temporarily stored is erased, and the same processing procedure is terminated. The next processing procedure may be started again.

  In order to confirm the effectiveness of the present invention, a verification test was conducted. The camera was equipped with an industrial image sensor (SONY DFW-VL500) in a car, and the moving image V of 15fps was captured in the daytime while the car was moving straight at 40km / h. The still image extraction means, the tracking image extraction means, the image composition means, and the image recognition means are configured by executing a processing program on a commercially available notebook computer (IBM ThinkPad (registered trademark) T42p), and the memory of the notebook computer is stored in the memory. The internal hard disk was used as the hard disk, and the notebook computer display was used as the display. In the verification test, the traffic sign to be recognized was a restriction sign (red circular sign) indicating “parking prohibited”. In addition, the above-described CONDENSATION is used for extracting the tracking image T, and “specific color determination for road sign recognition using contour vectors (Jin Yamauchi, Hiromitsu Takahashi) ”(Technical Report of IEICE, IE200488, 11-16, Nov. 2004.).

FIG. 9 is a reference diagram showing the tracking image T, the discrimination image D, and the recognition result of each image in the verification test of the restriction sign indicating “parking prohibited”. The weights W (see Equation 1) used for superimposing the tracking image T and the enlarged discrimination image D ′ were set to A ₁ = 0.15, A ₂ = 6.0, and A ₃ = 0.025 by preliminary experiments. The still image S is extracted from the continuous moving image V at an appropriate interval. That is, the tracking image T is extracted from the moving image V at an appropriate interval. FIG. 9 shows only the tracking image T and the discrimination image D at intervals of 10 frames. In FIG. 9, the size (number of pixels) of the specific dimension r obtained by CONDENSATION is described at the left end where the tracking images T and the discrimination images D are arranged. Furthermore, in the lower part of each tracking image T and discrimination image D, the name of the recognized traffic sign and the numerical value of the rate at which the traffic sign was correctly recognized in each of the 10 frames before and after each tracking image T and discrimination image D are shown. These are shown as recognition results.

  When the tracking image T and the discrimination image D at the same point in time (the specific dimension r is the same) are visually compared, the tracking image T has only the feature information CI of the still image S with a low resolution. It can be seen that the discrimination image D is a smooth image because the individual pixels interfere with each other. This is considered to visually indicate that the characteristic information CI is averaged and accumulated in the entire discrimination image D.

  Further, comparing the recognition results of the tracking image T and the discrimination image D, the recognition based on the tracking image T correctly recognizes the traffic sign when the specific dimension r is 9.94, 13.72, and 16.08, but the specific dimension r is In case of 11.30 and 11.91, it is wrong and it is hard to say that the traffic sign is recognized stably. On the other hand, in the recognition based on the discrimination image D, all of the specific dimensions r are recognized correctly at any stage after 9.94, that is, from the beginning in this verification test, and can be recognized stably. From this, it can be confirmed that the traffic sign recognition method of the present invention is a highly robust recognition method.

  Furthermore, when the recognition rates of the tracking image T and the discrimination image D are compared, in the recognition of the tracking image T, the specific rate r once shows a high value at the stage of 9.94, but the recognition rate is once decreased, and the correct and stable. The recognition result is obtained after the specific dimension r becomes 13.72. On the other hand, the recognition of the discrimination image D stably shows a recognition rate of 80% or more at any stage. From this, it can be confirmed that the traffic sign recognition method of the present invention is a highly reliable recognition method.

  Next, a similar verification test was carried out using a regulatory sign indicating “40 km / h limit” including a numerical notation as a recognition target instead of the regulatory sign indicating “parking prohibited” as a recognition target in the verification test. FIG. 10 is a reference diagram showing a tracking image T and a discrimination image D and a recognition result of each image in a verification test of a restriction sign indicating “40 km / h restriction”. In FIG. 10, the tracking image T and the discrimination image D are intervals of 20 frames of the moving image V. In this verification test, the light conditions were relatively poor as compared to the verification test (Example 1). For this reason, the recognition of the tracking image T has a large change in the recognition rate, and is not stable. On the other hand, the recognition of the discriminating image D has a relatively high recognition rate after the specific dimension r of 11.78 or later although it does not reach the verification test due to the influence of the light beam condition. From this, it can be confirmed that the traffic sign recognition method of the present invention is a robust and highly reliable recognition method.

Finally, a verification test was conducted when the traffic signs were obscured by roadside trees or other traffic signs. FIG. 11 is a reference diagram showing a recognition result of the tracking image T and the discrimination image D and each image in the verification test of the regulation sign indicating “prohibition of overtaking” hidden and hidden by street trees and other traffic signs. Is a correlation diagram of a specific dimension r and a recognition rate in this verification test. This verification test is performed when the direction of travel of the car turns to the left and the traffic sign forbidden on the sidewalk on the left side of the road is obscured by the roadside tree trunk or other traffic signs. The tracking image T and the discrimination image D are compared. In this verification test, A ₁ = 0.20, A ₂ = 6.00, and A ₃ = 0.10 in the weight W (see Equation ₁ ) in the synthesis of the tracking image T and the discrimination image D are set. As is clear from FIG. 11, the tracking image T alone cannot be recognized or is erroneously recognized, but it can be confirmed that the discrimination image D correctly recognizes “prohibition of overtaking”. In addition, the recognition image D has a high recognition rate, and it can be seen that the present invention is very effective for the recognition of traffic signs that are visible and hidden by obstacles.

  When the result of this verification test is shown by the correlation between the specific dimension (radius) r and the recognition rate, the tracking image T and the discrimination image D are recognized as the specific dimension (radius) r increases as shown in FIG. Although the rate of improvement in the rate is the same, the recognition rate of the discrimination image D is generally high, and particularly in the range of a specific dimension (radius) r = 14 to 19 pixels where a traffic sign is obscured by an obstacle, the discrimination image D Is significantly different from the recognition rate of the tracking image T. As described above, it is confirmed from the correlation diagram that the present invention is effective in recognizing a traffic sign that appears and disappears. From the above results, it can be said that the present invention provides a traffic sign recognition method that recognizes traffic signs correctly and stably at an early stage, has high reliability, and has few misrecognitions even with visible and hidden traffic signs. .

It is a block diagram showing the schematic structure of the traffic sign recognition system to which this invention is applied. It is a use condition reference figure showing the car carrying the traffic sign recognition system of this example. It is a flowchart showing the process sequence in a traffic sign recognition system. 3 is a reference diagram illustrating an example of a still image S. FIG. 4 is a reference diagram illustrating a tracking image T extracted from a still image S. FIG. FIG. 6 is a reference diagram schematically illustrating processing for enlarging a discrimination image D with respect to a tracking image T. FIG. 6 is a reference diagram schematically illustrating a process of combining a tracking image T and an enlarged discrimination image D ′. It is a reference figure showing the outline trace image (d) which traced the outline vector image (b) (c) extracted from the specific color extraction image of the reference image (a), and the outline vector continuously. It is a reference figure showing the recognition result of the tracking image T and the discrimination | determination image D, and each image in the verification test of the control sign which shows "parking prohibition". It is a reference figure showing the recognition result of the tracking image T and the discrimination | determination image D, and each image in the verification test of the control sign which shows "40km / h restriction | limiting". It is a reference figure showing the recognition result of the tracking image T and the discrimination | determination image D, and each image in the verification test of the control sign which shows the "prohibition overtaking" which is visible and hidden by a roadside tree, other traffic signs, etc. It is a correlation diagram of the specific dimension r and recognition rate in a verification test.

Explanation of symbols

11 Camera (video camera)
12 Camera (still image camera)
2 Onboard computer
21 Still image extraction means
22 Tracking image extraction means
23 Image composition means
24 Sign recognition means
25 memory
26 Processing Unit 3 Hard Disk 4 Display 5 Car 6 Traffic Sign

Claims (6)

In a sign recognition method for recognizing a traffic sign contained in a time-series still image S obtained by photographing a forward direction of a vehicle with a camera,
The first tracking image T including only the traffic sign is extracted from the first still image S, the first tracking image T is temporarily stored as the discrimination image D, and the second and subsequent tracking images T from the second and subsequent still images S are temporarily stored. And a new discriminant image D obtained by synthesizing the second and subsequent tracking images T and the discriminant image D previously stored temporarily is temporarily stored in the feature information CI extracted from the new discriminant image D. A traffic sign recognition method characterized by recognizing a traffic sign ahead of a vehicle in the traveling direction. The traffic sign recognition method according to claim 1, wherein the still image S is extracted in units of frames from a moving image V obtained by photographing a forward direction of the vehicle with a camera. The new discriminant image D calculates the size ratio R between the second and subsequent tracking images T and the discriminant image D temporarily stored, and enlarges the temporarily stored discriminant image D by the size ratio R. 3. A traffic sign recognition according to claim 1 or 2, wherein said discriminant image D 'is generated, and tracking image T is synthesized with a weight W and said enlarged discriminant image D' is synthesized at a ratio of weight (1-W). Method. 4. The traffic sign recognition method according to claim 3, wherein the weight W is a value that monotonously increases with respect to the specific dimension r in the tracking image T within a range smaller than 0.5. The weight W is a function W (r) having a specific dimension r in the tracking image T as a variable,
dW (r) / dr> 0, d ² W (r) / dr ² <0
The traffic sign recognition method according to any one of claims 3 and 4. 6. The traffic sign recognition method according to claim 5, wherein the function W (r) is expressed by the following equation (1).
A ₁ : Amplification factor A ₂ : Threshold for specific dimension r A ₃ : Threshold for weight W