JP6527617B2 - Outside environment recognition device - Google Patents

Description

  The present invention relates to an outside environment recognizing device for recognizing contents of a road sign confirmed outside the vehicle.

  Conventionally, a specific object such as a vehicle located in front of the own vehicle is detected to avoid collision with the preceding vehicle (collision avoidance control) or to control the distance between the preceding vehicle to be a safe distance ( Cruise control) technology is known (for example, patent document 1). In addition, in order to reduce accidents caused by overspeeding, there is a growing demand for a technology for controlling the speed of the vehicle by recognizing the speed limit provided for each road.

  In order to make the vehicle travel safely with the speed of the host vehicle within the speed limit, it is necessary to recognize the contents of the road sign arranged on the road shoulder and the gate, and to correctly grasp the speed limit of the road currently being traveled. For example, Patent Document 2 describes a technique for performing Hough transform on a portion corresponding to an edge on a screen to recognize an image of a circular road sign (hereinafter, the image of a road sign is also simply referred to as a road sign). . According to such a technology, it is possible to improve the identification efficiency of the road sign by reducing the processing load spent on the Hough transform.

Patent No. 3349060 JP 2012-243051 A

  In order to perform the Hough transformation and recognize a circular road sign, first, feature points corresponding to partial portions of the circumference must be identified. In Patent Document 1, a block having an edge as a feature point is identified. It is also conceivable to specify a feature point on the basis of an arbitrary color component instead of the edge. However, only by extracting edges and color components under extraction conditions using a fixed threshold, feature points are identified innumerably and processing load increases. Then, there is a possibility that the processing time assigned to the feature point identification may be exceeded.

  SUMMARY OF THE INVENTION In view of such problems, the present invention provides an outside environment recognition system capable of suppressing the number of feature points within a predetermined range by changing the extraction conditions of feature points by edge or color component as needed. The purpose is to

  In order to solve the above problems, an environment recognition apparatus for the outside of a vehicle according to the present invention includes: an image acquisition unit that acquires an image; and a feature point identification unit that identifies a pixel whose edge intensity is a predetermined value or more , A sign identification unit that identifies a circular road sign based on feature points, and a sign content recognition unit that recognizes the contents of the identified road sign, and the feature point identification unit is characterized in order from the top of the image The points are identified, and when the identified feature points become equal to or more than a predetermined number, identification of the feature points is stopped, and if the identified feature points exceed a predetermined feature point upper limit, the predetermined point is determined. If the value is increased and the specified number of feature points is less than the predetermined feature point lower limit value, the predetermined value is decreased and used for the next comparison with the edge strength.

  According to the present invention, it is possible to suppress the number of feature points within a predetermined range by changing the extraction condition of feature points by edge or color component as needed.

It is a block diagram showing the connection relation of the environment recognition system outside a car. It is an explanatory view for explaining a color image and a distance image. It is a functional block diagram showing a schematic function of an outside environment recognition device. It is an explanatory view explaining a road sign. It is a flowchart which shows the flow of an environment recognition process outside a vehicle. It is an explanatory view for explaining a color image which an image acquisition part acquires. It is an explanatory view for explaining Hough transformation. It is an explanatory view for explaining Hough transformation. It is an explanatory view for explaining the 4th extraction condition. It is the flowchart which showed an example of the feature point identification process. It is an explanatory view showing an example of feature point specific processing. It is an explanatory view for explaining voting processing. It is an explanatory view for explaining a voting table. It is an explanatory view for explaining a central point candidate list. It is explanatory drawing for demonstrating a flag table. It is the flowchart which showed an example of the label | marker identification process. It is explanatory drawing for demonstrating the process of a label | marker correction | amendment part. It is the flowchart which showed the flow of the concrete processing of mark contents recognition processing. It is an explanatory view for explaining a recognition object field. It is an explanatory view for explaining a road sign which presented cancellation of speed limit. It is an explanatory view for explaining vertical direction alignment processing. It is an explanatory view for explaining a template. It is explanatory drawing for demonstrating the matching process of a horizontal direction. It is an explanatory view for explaining DP matching. It is explanatory drawing for demonstrating the matching process of an attention site. It is an explanatory view for explaining an evaluation judging result. It is the time chart which showed the flow of the result information of the road sign. It is explanatory drawing which showed the display mode of the road sign according to country. It is an explanatory view for explaining a template of a road sign.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The dimensions, materials, and other specific numerical values and the like shown in the embodiment are merely examples for facilitating the understanding of the invention, and do not limit the present invention unless otherwise specified. In the specification and the drawings, elements having substantially the same functions and configurations will be denoted by the same reference numerals to omit repeated description, and elements not directly related to the present invention will not be illustrated. Do.

(Outside environment recognition system 100)
FIG. 1 is a block diagram showing the connection relationship of the outside environment recognition system 100. As shown in FIG. The external environment recognition system 100 includes an imaging device 110, an external environment recognition device 120, and a vehicle control unit (ECU: Engine Control Unit) 130 provided in the host vehicle 1.

  The imaging device 110 is configured to include an imaging element such as a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS), captures an environment corresponding to the front of the host vehicle 1, and is represented by color values. Color images can be generated. Here, the color value is a YUV color space consisting of one luminance (Y) and two color differences (U, V), and three hues (R (red), G (green), B (blue)). It is a numerical value group represented by either the RGB color space or the HSB color space consisting of hue (H), saturation (S) and lightness (B). In the present embodiment, a color image by color values in YUV format is described as an example of an image, but for an application that can execute processing without depending on the color image in any partial processing, only color images may be used. Instead, a luminance image (monochrome image) represented by luminance (Y) can be used.

  Moreover, the imaging devices 110 are spaced apart in the substantially horizontal direction so that the optical axes of the two imaging devices 110 are substantially parallel on the traveling direction side of the host vehicle 1. The imaging device 110 continuously generates, for example, a frame image (60 fps) obtained by imaging an object present in a detection area in front of the host vehicle 1 every 1/60 second. Here, the objects to be recognized are not only three-dimensional objects that exist independently such as vehicles, pedestrians, traffic lights, roads (traveling roads), road signs, gates, guard rails, buildings, etc., but also the contents of road signs, brake lamps, It includes items that can be identified as part of a three-dimensional object, such as high-mounted stop lamps, tail lamps, blinkers, and lighting parts of traffic lights. Each functional unit in the following embodiments performs each process for each frame triggered by such updating of a color image.

  Furthermore, in the present embodiment, the imaging device 110 captures a detection area in a first exposure mode that indicates an exposure time and an aperture according to the brightness of the environment outside the vehicle (measurement results of an illuminance meter and the like), and generates a first image. Do. Further, the imaging device 110 generates an image that can determine whether or not a specific light emission source, such as a road sign of an electric light display type, emits light by itself. As the method, an image pickup element having a wide dynamic range may be used to pick up an image so that the non-emitting object is not blacked out and the light source is not overexposed, and the first exposure mode is an exposure mode (exposure The detection area may be imaged in a second exposure mode different in time and aperture) to generate a second image. For example, in the daytime, the exposure time of the second exposure mode is shorter than the exposure time of the first exposure mode according to the bright external environment, or the aperture is made stronger to generate the second image. In the present embodiment, the first image and the second image are used as a color image and a distance image, respectively. Further, the first exposure mode and the second exposure mode are realized as follows.

  For example, by periodically dividing the periodic imaging timing of the imaging device 110 and alternately performing imaging in the first exposure mode and imaging in the second exposure mode, the first image and the second image are sequentially generated. Can. In addition, in an image pickup element in which two capacitors are provided for each pixel and charge can be charged in parallel to the two capacitors, two images having different exposure modes are made in parallel by changing the charging time in one exposure. It can also be generated. Furthermore, the above object can also be achieved by reading out twice at different times during charge charging of one capacitor and generating two images in different exposure modes in parallel. Alternatively, two sets of imaging devices 110 may be prepared in advance with different exposure modes (here, two imaging devices 110 × 2 sets), and images may be generated from the two sets of imaging devices 110, respectively. It is possible.

  The environment outside environment recognition device 120 acquires a color image from each of two imaging devices 110, and the block corresponding to a block arbitrarily extracted from one color image (for example, an array of 4 horizontal pixels × 4 vertical pixels) The parallax information including the parallax and the screen position indicating the position of the arbitrary block in the screen is derived using so-called pattern matching which is searched from the image. Here, horizontal indicates the screen horizontal direction of the captured image, and vertical indicates the screen vertical direction of the captured image. As this pattern matching, it is conceivable to compare the luminance (Y) in arbitrary block units between a pair of images. For example, SAD (Sum of Absolute Difference) which takes a difference of luminance, SSD (Sum of Squared intensity Difference) which uses the difference as a square, NCC which takes similarity of the dispersion value which subtracted the average value from the luminance of each pixel There are methods such as (Normalized Cross Correlation). The environment outside environment recognition device 120 performs such block-based parallax deriving processing for all blocks displayed in the detection area (for example, 600 pixels × 200 pixels). Here, the block is 4 pixels × 4 pixels, but the number of pixels in the block can be set arbitrarily.

  However, in the external environment recognition device 120, although the parallax can be derived for each block which is a detection resolution unit, it is not possible to recognize what part of the object the block is. Therefore, the disparity information is not independently derived from the object unit, but is independently derived on a detection resolution unit (for example, block unit) in the detection area. Here, the image associated with the parallax information derived in this manner is referred to as a distance image to distinguish it from the color image described above.

  FIG. 2 is an explanatory diagram for explaining the color image 126 and the distance image 128. As shown in FIG. For example, it is assumed that a color image 126 as shown in FIG. 2A is generated for the detection area 124 through the two imaging devices 110. However, here, only one of the two color images 126 is schematically shown to facilitate understanding. The external environment recognition device 120 obtains the parallax for each block from such a color image 126, and forms a distance image 128 as shown in FIG. 2 (b). Each block in the distance image 128 is associated with the parallax of that block. Here, for convenience of explanation, the block from which the parallax is derived is represented by black dots. In the present embodiment, the color image 126 and the distance image 128 are generated based on the first image and the second image.

  Further, the external environment recognition device 120 uses color values based on the color image 126 and three-dimensional positional information in real space including the relative distance to the vehicle 1 calculated based on the distance image 128, Blocks having equal values and close three-dimensional position information are grouped as objects, and it is specified to which specific object (for example, a preceding vehicle) an object in a detection area in front of the host vehicle 1 corresponds. For example, it is possible to identify the preceding vehicle traveling ahead by the relative distance or the like, and further, by accurately recognizing whether or not the brake lamp of the preceding vehicle is lit by the color value, it is possible to grasp acceleration / deceleration of the preceding vehicle. In addition, the environment recognition device outside the vehicle 120 identifies a road sign arranged on the road shoulder or gate, further recognizes the content of the road sign, for example, the speed limit, and limits the speed of the vehicle 1 through the vehicle control device 130. Control to a safe speed within the speed.

  The relative distance can be obtained by converting disparity information of each block in the distance image 128 into three-dimensional position information using a so-called stereo method. Here, the stereo method is a method of deriving the relative distance of the object to the imaging device 110 from the parallax of the object by using the triangulation method.

  Referring back to FIG. 1, the vehicle control device 130 receives the operation input of the driver through the steering wheel 132, the accelerator pedal 134, and the brake pedal 136, and transmits it to the steering mechanism 142, the drive mechanism 144, and the braking mechanism 146. The host vehicle 1 is controlled. Further, the vehicle control device 130 controls the drive mechanism 144 and the braking mechanism 146 according to the instruction of the external environment recognition device 120.

  Hereinafter, the configuration of the external environment recognition device 120 will be described in detail. Here, the process of identifying a road sign, which is characteristic of the present embodiment, will be described in detail, and the description of the configuration irrelevant to the feature of the present embodiment will be omitted.

(Exterior environment recognition device 120)
FIG. 3 is a functional block diagram showing a schematic function of the external environment recognition device 120. As shown in FIG. As shown in FIG. 3, the external environment recognition apparatus 120 includes an I / F unit 150, a data holding unit 152, and a central control unit 154.

  The I / F unit 150 is an interface for performing two-way information exchange with the imaging device 110 and the vehicle control device 130. The data holding unit 152 is configured by a RAM, a flash memory, an HDD, and the like, holds various information necessary for processing of each functional unit described below, and receives an image received from the imaging device 110 (first image and first image The color image 126 based on the two images, the distance image 128) is temporarily held.

  The central control unit 154 is constituted by a semiconductor integrated circuit including a central processing unit (CPU), a ROM in which programs and the like are stored, and a RAM as a work area, and the I / F unit 150 and the data holding unit Control 152 and the like. Further, in the present embodiment, the central control unit 154 includes an image acquisition unit 160, a position information derivation unit 162, a feature point identification unit 164, a voting unit 166, a sign identification unit 168, a sign correction unit 170, a sign content recognition unit 172, It also functions as the sign content determination unit 174. Hereinafter, the road sign which is the recognition object in the present embodiment will be described, and then the outside environment recognition processing characteristic of the present embodiment will be described in detail in consideration of the operation of each functional unit of the central control unit 154.

  FIG. 4 is an explanatory view for explaining a road sign. As the type of road sign, there are a road sign presenting a speed limit and a road sign presenting a release of the speed limit. As for the road sign which presented speed limit, the numerical value which shows speed limit is described in the circle-like frame like Fig.4 (a). In addition, on the road sign that presented the release of the speed limit, as shown in Fig. 4 (b), a solid background with diagonal lines extending from the upper right to the lower left is described, or as shown in Fig. 4 (c). In some cases, the same diagonal lines are written against a numerical value indicating speed.

  Moreover, as a display mode of a road sign, there are an electro-optical display type having a light emission source such as an LED or the like and a non-electro-optical display type only having a color classification without having a light emission source. In addition, as the arrangement of the road sign, there are those which are arranged on the road shoulder of the road, and those which are arranged on the upper side of the road of the gate erected in arch shape straddling both road shoulders. .

  The external environment recognition device 120 according to the present embodiment recognizes the contents of each of the road signs of different types, such as the arrangement, the display mode, and the like, by the respective functional units of the central control unit 154 described above. When recognizing the contents of the road sign, the outside environment recognition device 120 notifies the driver of the contents, for example, the speed limit of the road on which the vehicle is currently traveling, or when the current speed exceeds the speed limit. This can be notified to the driver, or the vehicle control device 130 can be controlled so as not to exceed the speed limit. For this reason, it is not necessary to recognize the road sign at the moment when the vehicle 1 reaches a position at which the road sign can be confirmed, and it is sufficient if it can be recognized when passing the road sign or later. Therefore, it is sufficient to recognize the road sign over a plurality of frames and to determine the content of the road sign from the information of the plurality of frames.

(Outside environment recognition processing)
FIG. 5 is a flow chart showing the flow of the outside environment recognition process. In the environment recognition process outside the vehicle, the image acquisition process (S200) that acquires an image is roughly divided, the road sign, especially the sign detection process that detects a circular frame (S202), the contents of the road sign (numerical value and pattern pattern) The marker content recognition process (S204) for recognizing the mark content, and the marker content confirmation process (S206) for accumulating and confirming the content of the recognized road marker in the time direction are executed in that order.

(Image acquisition processing S200)
The image acquisition unit 160 acquires the color image 126 from the imaging device 110. As described above, some road signs targeted in the present embodiment have different display modes such as an electric display type and a non-electric display type, and some have a different arrangement such as a road shoulder and a gate. . Therefore, the imaging device 110 images two detection areas capable of detecting the road shoulder and the gate in the two exposure modes of the first exposure mode and the second exposure mode, and the image acquisition unit 160 captures an image in this manner. The acquired four color images 126 are acquired.

  FIG. 6 is an explanatory diagram for explaining the color image 126 acquired by the image acquisition unit 160. As shown in FIG. For example, in the imaging device 110, the color image 126 shown in FIGS. 6A and 6B is captured in the first exposure mode in which the exposure time is relatively long. However, FIG. 6 (a) shows a color image 126 captured at an angle of view that can detect the road shoulder, and FIG. 6 (b) is a direction in which the angle of view is wider than in FIG. , And the color image 126 captured and displayed. Then, the imaging device 110 switches the exposure mode to the second exposure mode, for example, sets the exposure time relatively short, and captures the color image 126 shown in FIGS. 6C and 6D. However, FIG. 6C and FIG. 6D, like FIGS. 6A and 6B, are captured by switching the angle of view according to the object to be detected. Here, four color images 126 are listed, but the number and type can be set arbitrarily as long as the road shoulder and the gate can be detected.

  Thus, four color images 126 with different exposure modes and detection areas are obtained. By imaging in a plurality of exposure modes and detection areas as described above, it is possible to solve problems such as saturation of the luminance of the light source or excessive resolution and a decrease in resolution, thereby sufficiently enhancing detection accuracy. Can. The four color images 126 are captured in time division, and the order can be set arbitrarily. However, in the present embodiment, four color images 126 do not necessarily have to be imaged at the same timing, as long as the road sign can be recognized when or after passing the road sign.

  The position information deriving unit 162 acquires color images (FIG. 6A and FIG. 6B) based on the first image captured at the first exposure from each of the two imaging devices 110, and generates a parallax using pattern matching. And disparity information including a screen position indicating a position in the screen of an arbitrary block is derived to generate a distance image 128. Then, using the stereo method, the position information deriving unit 162 uses the stereo method to generate horizontal information that is a horizontal relative distance around the center of the host vehicle 1 in the horizontal direction, using the parallax method for each block in the detection area 124 in the distance image 128 It is converted into three-dimensional position information including the distance x, the height y from the road surface, and the relative distance z with the vehicle 1 in the depth direction. However, the position information deriving unit 162 obtains the vertical position of the road surface in advance before converting into three-dimensional position information, and the height from the road surface from the relative distance between the vertical position of each block and the vertical position of the road surface. derive y. Here, the parallax information indicates the parallax of each block in the distance image 128, while the three-dimensional position information indicates the information of the relative distance z of each block in the real space. In addition, when disparity information is derived not in units of pixels but in units of blocks, that is, in units of a plurality of pixels, the disparity information may be regarded as disparity information of all pixels belonging to a block to perform calculation in units of pixels. it can. About conversion to such three-dimensional position information, since existing technology, such as JP 2013-109391, A, can be referred to, the detailed description thereof is omitted here.

(Label detection process S202)
In the present embodiment, it is an object of the present invention to recognize, among road signs, particularly circular road signs. Such circular road signs are detected using Hough transformation. Here, the Hough transform is a technique of voting from a feature point on a color image to a point where an object may exist and detecting an object having a large number of votes (equal to or more than a predetermined value). As described above, although the Hough transform is specifically described in this embodiment, not only the Hough transform but also an application that can specify a road sign without relying on the Hough transform in any partial process of the environment recognition process outside the vehicle Various existing shape recognition methods such as template matching other than the Hough transform and the least squares method can be used.

  7 and 8 are explanatory diagrams for describing the Hough transform. Here, it is assumed that three pixels 220c, 220d, and 220e having edges as shown in FIG. 7A are extracted from the color image 126. The three pixels 220c, 220d, and 220e are originally part of the circular road sign 222, but normally it can not be clearly understood from the color image 126 that they are circular.

  The Hough transform is a method of detecting a geometric shape such as a circle or a straight line from a plurality of points, and the center of a circle passing through any pixel 220 and having a radius n is centered on that any pixel 220 It is based on the theory that it exists on the circumference of radius n. For example, the center of the circle passing through the three pixels 220c, 220d, and 220e in FIG. 7A is on the circumference centered on each of the three pixels 220c, 220d, and 220e. However, since the radius n can not be specified by the information of only the edge, the radius n of different stages is prepared, and pixels on the circle of the radius n of multiple stages centered on the three pixels 220c, 220d and 220e If the number of votes is equal to or greater than a predetermined value, the radius n and the center are determined as the road sign 222.

  For example, as shown in FIGS. 7 (b), 7 (c) and 7 (d), circles having different radii n = 4, 5, 6 are formed around three pixels 220c, 220d, 220e. And vote for the pixels included in the locus of the circle (associate the unit index). Then, in FIG. 7B, the number of votes obtained is two for two pixels 224 (two unit indices are associated). Further, in FIG. 7C, the number of votes obtained is three in three pixels 224, and the number of votes obtained is three in one pixel 226. Similarly, in FIG. 7D, the number of votes obtained is six for six pixels 224.

  At this time, only the pixel 226 has 3 votes (more than a predetermined value), and the pixel 226 is the center of a circle passing through the three pixels 220c, 220d and 220e, and the radius n when the pixel 226 is derived We can specify = 5 as the radius of the circle. Thus, as shown in FIG. 7E, a circle 228 passing through the three pixels 220c, 220d and 220e is identified. Here, for convenience of explanation, although three pixels 220c, 220d, and 220e were mentioned and explained, the pixel which is not included in circle 228 becomes a feature point, or a position different from the original position by pixelization (discretization) Since the pixel which appeared may become a feature point, in order to avoid the influence of such noise, in fact, a large number of points are used for voting, and the stable detection is performed by the principle of number. In the present embodiment, such Hough transform is applied to, for example, the color image 126 shown in FIG. 6B, and the feature point identifying unit 164, the voting unit 166, and the sign identifying unit 168 use circular road signs. Identify. The basic processing of each functional unit will be described below with reference to FIG.

  First, the feature point identifying unit 164 identifies a feature point corresponding to a partial portion of the circumference from the color image 126 (feature point identifying process). For example, it is assumed that the feature point identification unit 164 identifies the pixels 220f, 220g, 220h, 220i, 220j, and 220k having an edge as a feature point in the color image 126 of FIG. 8A. The dotted line in FIG. 8A corresponds to the traveling lane.

  Next, the voting unit 166 votes at a predetermined distance corresponding to the radius n from the feature point (voting process). Here, with respect to the six pixels 220, for convenience of description, the radius n is temporarily set to 30 pixels for the pixels 220f, 220g, and 220h, and the radius n is temporarily set to 23 pixels for the pixels 220i, 220j, and 220k. The voting unit 166 is located on a circle having a radius n (30 pixels, 23 pixels) from each of the pixels 220 around the pixels 220f, 220g, 220h, 220i, 220j, and 220k in the color image 126 of FIG. Vote for all the pixels. Then, 1 is voted (added) to the voting destination (pixel, radius n) in the voting table 230 shown in FIG. 8C. Here, the voting table 230 is a Hough transform voting space, and is represented in three dimensions of the screen position (x, y) of the pixel to be voted on and the radius n.

  Subsequently, the sign specifying unit 168 detects the number of votes in the voting table 230, and derives the center of the circle and the radius n based on the pixel of the voting destination having a large number of votes and the radius n. Then, as shown in FIG. 8D, a circle 236 with a radius n read about the pixel 234 having a large number of votes is formed, and it is identified as a road sign (sign identification process). Hereinafter, specific operations of the feature point identifying process, the voting process, and the sign identifying process by the feature point identifying unit 164, the voting unit 166, and the sign identifying unit 168 will be described.

(Feature point identification process)
The feature point identifying unit 164 uses the color image 126 and sets a pixel 220 having a predetermined edge intensity among the pixels 220 as a feature point candidate as a first extraction condition. The edge strength may, for example, be represented by a Sobel filter. Assuming that the coordinates of each pixel 220 are (i, j) and the luminance is A (i, j), the feature point identifying unit 164 uses the following equation 1 to set the Sobel filter in the vertical direction and the horizontal direction: The sum of absolute values of the Sobel filter is derived, and if the value (edge strength) is equal to or greater than a predetermined value, the pixel (i, j) is set as a candidate of the feature point.
Edge strength = | A (i + 1, j + 1) +2 A (i + 1, j) + A (i + 1, j-1) -A (i-1, j + 1) -2 A (i- 1, j)-A (i-1, j-1) |
+ | A (i + 1, j + 1) + 2A (i, j + 1) + A (i-1, j + 1) -A (i + 1, j-1) -2A (i, j- 1) -A (i-1, j-1) | ... (Equation 1)

  Here, although the example which derives edge intensity with Sobel filter was mentioned, it is not restricted to such a case, and various existing techniques, such as a Prewitt (Prewitt) filter, can be applied.

  In addition, the feature point specifying unit 164 uses the color image 126, and as the second extraction condition, a predetermined color component of the predetermined color value of the pixel 220, for example, a V component in the YUV color space If the value is equal to or greater than a predetermined value, the pixel 220 is set as a feature point candidate. A road sign presenting a speed limit is composed of a circle with a red circumference, and a road sign presenting a release of the speed limitation is composed of a circle with a white or black circumference. Therefore, only the color belonging to the region where the V component is equal to or more than the predetermined value is extracted as a candidate of the feature point. In this way, it is possible to exclude green-based pixels such as trees that are identified on many occasions while traveling, and to enable appropriate narrowing of feature points.

  If the color image is configured in the RGB color space, it is converted to the YUV color space by an arbitrary conversion formula. Since this conversion formula is an existing technology, the detailed description thereof is omitted here.

  Further, the feature point identifying unit 164 uses the distance image 128, and as the third extraction condition, the relative distance z is within a predetermined range, the height y from the road surface is within a predetermined range, and the horizontal distance among the pixels 220. A pixel 220 in which x satisfies any one or more conditions within a predetermined range is set as a feature point candidate.

  Specifically, the feature point identifying unit 164 extracts an arbitrary pixel 220 from the distance image 128, refers to three-dimensional position information of the pixel 220, and the relative distance z of the object corresponding to the pixel 220 is, for example, When the pixel 220 is positioned at 10 m or more and less than 50 m, the pixel 220 is set as a feature point candidate. This is because if the distance is less than 10 m, the amount of movement of the object on the image in the exposure time becomes long, and the influence of the image blur increases accordingly, and if it is 50 m or more, the road sign In many cases, the content of the can not be recognized correctly. By limiting the relative distance z in this manner, it is possible to reduce the processing load and to reduce misrecognition.

  The feature point identifying unit 164 sets the pixel 220 as a feature point candidate when the height y of the object corresponding to the pixel 220 from the road surface is, for example, 0.5 m or more and less than 6.0 m. Do. By setting the range to 0.5 m or more, road markings and lanes can be excluded as a recognition target, and by setting the range to less than 6.0 m, trees etc. located above can be excluded. . Such a condition also makes it possible to reduce processing load and reduce false recognition.

  In addition, when the horizontal distance x of the object corresponding to the pixel 220 is, for example, within a range of 12 m (−12 m or more and less than 12 m), the feature point identifying unit 164 sets the pixel 220 as a feature point candidate. . By setting this range to 12 m, it is possible to exclude road signs other than the road signs related to the lane in which the vehicle 1 is traveling. Such a condition also makes it possible to reduce processing load and reduce false recognition.

  In addition, the feature point identifying unit 164 uses the color image 126 and the distance image 128, and the difference between at least one color component (for example, U component) of adjacent pixels 220 is within a predetermined range as the fourth extraction condition. In addition, pixels in which such pixels 220 are not continuous with each other in a predetermined direction by a predetermined distance (length) or more are set as feature point candidates.

  FIG. 9 is an explanatory diagram for explaining the fourth extraction condition. For example, there may be a case where a tree 240 or the like with colored leaves shown on the road shoulder in FIG. 9A satisfies all of the first to third extraction conditions. In addition, since there are many places that can be recognized as textures, the trees 240 that have fallen leaves are often extracted as feature points in a wide range. Then, feature points are identified in a wide range in a tree 240 or the like with many textures, and the processing load increases.

  Therefore, the feature point identifying unit 164 determines that the distance between the pixels 220, that is, the distance in the depth direction, the distance in the vertical direction, and the combined distance in the horizontal direction is, for example, a predetermined distance (for example, a predetermined distance) It is determined whether the difference is less than 0.5 m and the difference between one color component (e.g. U component) is within a predetermined value (e.g. 10). Then, in the feature point identifying unit 164, pixels 220 whose combined distance is less than 0.5 m and whose U component difference is within 10 are 30 consecutive pixels in one direction (for example, the horizontal direction) of the screen. In the case, all the pixels 220 are excluded from the feature points. In the road sign, as shown in FIG. 9 (b), the periphery is constituted by circles of the same color, but there is no 30 pixels continuous in a predetermined one direction, for example, the horizontal direction or the vertical direction. By specifying the feature points based on the features of the display format of such a road sign, it is possible to exclude the pixels 220 that satisfy the first to third extraction conditions that should not be extracted as feature points originally. It is possible to improve the efficiency of specifying feature points.

  However, the size of the road sign on the color image 126 changes with the relative distance z. Therefore, the feature point identifying unit 164 may change a predetermined distance, which is a threshold value as to whether the pixels 220 are continuous in a predetermined direction, according to the relative distance z to the host vehicle 1. Specifically, the predetermined distance is made longer as the relative distance z is shorter, and the predetermined distance is made shorter as the relative distance z is longer. By doing this, it is possible to provide an appropriate threshold value according to the size of the road sign on the color image 126, and the pixels 220 that satisfy the first to third extraction conditions that should not be extracted as feature points originally. Can be properly excluded.

  Then, the feature point identifying unit 164 identifies, as a feature point, a pixel 220 satisfying the first or second extraction condition among the pixels 220 satisfying both the third and fourth extraction conditions. Thus, a pixel 220 appropriate as a feature point is identified.

  In addition, the feature point identifying unit 164 may cancel the feature point identifying process in one frame when the number of feature points becomes equal to or greater than a predetermined value. The color image 126 changes variously according to the environment outside the vehicle, and there may be a large number of feature points depending on the captured environment. As the number of feature points increases in this manner, the processing load increases accordingly, and the processing time allocated in one frame may be exceeded. Therefore, when the number of feature points becomes equal to or more than a predetermined value, the feature point identifying unit 164 cancels the feature point identifying process in one frame, and performs voting process only for the feature points identified up to that point. Perform the following processing.

  However, since the road sign is often positioned relatively above the color image 126, the feature point identification unit 164 identifies feature points in order from the top of the color image 126. In this way, it is possible to appropriately extract feature points corresponding to partial portions of the circumference of the road sign.

  In addition, the feature point identifying unit 164 frames the predetermined value of the edge strength in the first extraction condition and the predetermined value of the V component in the second extraction condition in order to keep the number of feature points within the predetermined value as described above. It may be changed every time. Since the environment outside the vehicle has little change on a frame basis, the number of feature points does not change much. Therefore, when a large number of feature points are extracted in one frame, a large number of feature points are continuously extracted in subsequent frames. Therefore, the predetermined value of the edge strength in the first extraction condition and the predetermined value of the V component in the second extraction condition are within the predetermined range so that the processing time assigned to identify the feature point in one frame is not exceeded. While adjusting at (40 to 150), the number of feature points is suppressed within a predetermined range (200 to 2000).

  FIG. 10 is a flowchart showing an example of feature point identification processing. As shown in FIG. 10A, the feature point identifying unit 164 determines whether the number of feature points extracted under the first extraction condition in one frame exceeds the feature point upper limit (here, 2000). It determines (S240). As a result, if the number of feature points exceeds the feature point upper limit (YES in S240), the feature point identifying unit 164 determines whether the predetermined value of the edge strength is less than the edge strength upper limit (here, 150) It determines (S242). As a result, if the predetermined value of the edge strength is less than the edge strength upper limit (YES in S242), the feature point identifying unit 164 increments the predetermined value of the edge strength (S244). The predetermined value of the edge strength is reflected from the specific object identification process of the next frame. If the predetermined value of the edge strength is equal to or higher than the upper limit of edge strength (NO in S242), the feature point identifying unit 164 does not increment the predetermined value of the edge strength, and the predetermined value reaching the upper limit of edge strength is used. Maintain (maintain within the given range).

  In step S240, if the number of feature points is equal to or less than the feature point upper limit (NO in S240), the feature point identifying unit 164 determines that the number of feature points extracted in one frame is the feature point lower limit value (here 200). It is determined whether it is less than (S246). As a result, if the number of feature points is less than the feature point lower limit (YES in S246), the feature point identifying unit 164 determines whether the predetermined value of the edge strength exceeds the edge strength lower limit (here, 40). It determines (S248). As a result, if the predetermined value of the edge strength exceeds the edge strength lower limit (YES in S248), the feature point identifying unit 164 decrements the predetermined value of the edge strength (S250). If the predetermined value of the edge strength is equal to or less than the lower limit value of the edge strength (NO in S248), the feature point identifying unit 164 does not decrement the predetermined value of the edge strength, and the predetermined value reaching the feature point lower limit value Maintain (maintain within the given range). If the number of feature points is equal to or more than the feature point lower limit (NO in S246), no processing is performed.

  Also, as shown in FIG. 10B, the feature point identifying unit 164 determines whether the number of feature points extracted under the second extraction condition in one frame exceeds the feature point upper limit value (here, 2000) It is determined whether or not (S260). As a result, if the number of feature points exceeds the feature point upper limit (YES in S260), the feature point identifying unit 164 determines whether the predetermined value of the V component is less than the V component upper limit (here, 150) It determines (S262). As a result, if the predetermined value of the V component is less than the V component upper limit value (YES in S262), the feature point identifying unit 164 increments the predetermined value of the V component (S264). The predetermined value of the V component is reflected from the specific object identification process of the next frame. Further, if the predetermined value of the V component is equal to or more than the V component upper limit (NO in S262), the feature point identifying unit 164 does not increment the predetermined value of the V component, and the predetermined value reaching the feature point upper limit Maintain (maintain within the given range).

  In step S260, if the number of feature points is equal to or less than the feature point upper limit (NO in S260), the feature point identifying unit 164 determines that the number of feature points extracted in one frame is the feature point lower limit value (here 200). It is determined whether it is less than (S266). As a result, if the number of feature points is less than the feature point lower limit (YES in S266), the feature point identifying unit 164 determines whether the predetermined value of the V component exceeds the V component lower limit (here, 40). It determines (S268). As a result, if the predetermined value of the V component exceeds the V component lower limit value (YES in S268), the feature point identifying unit 164 decrements the predetermined value of the V component (S270). If the predetermined value of the V component is less than or equal to the V component lower limit (NO in S268), the feature point identifying unit 164 does not decrement the predetermined value of the V component, and the predetermined value reaching the feature point lower limit Maintain (maintain within the given range). If the number of feature points is equal to or more than the feature point lower limit value (NO in S266), no processing is performed.

  As described above, in the present embodiment, the pixel 220 satisfying either the first extraction condition or the second extraction condition is specified as the feature point. Therefore, as for the first extraction condition and the second extraction condition, the number of feature points is adjusted independently as shown in FIGS. 10 (a) and 10 (b). Thus, the number of feature points is within the predetermined range (200 to 200) while the predetermined value of the edge strength under the first extraction condition and the predetermined value of the V component under the second extraction condition are suppressed within the predetermined range (40 to 150). It becomes possible to adjust by -2000) and to maintain the processing time allocated to the specification of the said feature point in 1 flame | frame.

  Further, the feature point identifying unit 164 may change the predetermined value of the V component under the second extraction condition for each frame according to the color component of the road surface. The color components of the entire color image 126 change depending on the solar radiation conditions and the illumination environment. For example, in a tunnel in which orange illumination is installed, the V component of the entire color image 126 is increased. Therefore, the feature point identifying unit 164 suppresses the influence of changes in solar radiation and illumination on the feature point identification by changing the predetermined value of the V component in the second extraction condition according to the color component of the road surface. .

  FIG. 11 is an explanatory view showing an example of feature point identification processing. As shown in FIG. 11A, the feature point identifying unit 164 is a color of RGB format at four points 280 at a predetermined relative distance z at which the road surface ahead of the vehicle in the color image 126 is likely to appear. Get the value. Then, the feature point identifying unit 164 obtains the G component / R component of each of the four points 280, and derives the average value AV of the values. Subsequently, the feature point identifying unit 164 multiplies the R component of all the pixels of the color image 126 by the average value AV, converts the result to YUV format, and compares the V component after the conversion with a predetermined value. Identify candidates for

  However, as shown in FIG. 11 (b), when the relative distance z of the preceding vehicle recognized by the external environment recognition device 120 is located within a predetermined range (for example, within 20 m), the average value AV is obtained from the color image 126. And use the average value AV (used) derived in the previous frame. This is for acquiring the color component of the preceding vehicle as the color component of the road surface and affecting the V component to avoid the extraction of an erroneous feature point candidate.

  Also, although the average value AV calculated in the current frame and the average value AV derived (used) in the previous frame do not change the exposure mode (there is no significant change in the ambient brightness). Also in the case where the predetermined value (for example, ± 50%) or more differs in advance, the average value AV is not obtained from the color image 126 but the average value AV (used) derived in the previous frame is used. This is for the purpose of affecting the V component when the road surface is colored with a red-based paint, and for avoiding extraction of an erroneous feature point candidate. However, when there is a shadow on the road surface, gray affects the color components (RGB) of the RGB system equally, and does not affect the values of the G component / R component, which is not a problem.

(Vote process)
The voting unit 166 votes on the circumference separated by radius n from the feature point identified by the feature point identifying unit 164. This is based on the premise that, assuming that a feature point corresponds to a partial portion of an arbitrary circumference, the center of a circle having the feature point as a partial portion of the circumference on the circumference of radius n from the feature point Based on. Therefore, the voting unit 166 votes at a point of a further radius n of the corresponding point which can be the center of the circle having the feature point as a partial portion of the circumference.

  FIG. 12 is an explanatory diagram for explaining the voting process. Here, assuming that a radius n of 1 is mentioned and described, as shown in FIG. 12A, the voting section 166 usually includes all the pixels 220 on the circumference 302 of the radius n centered on the feature point 300. Is a corresponding point 304, and a vote is made on a point of radius n of the screen position. However, since the corresponding points 304 with respect to such a radius n become all the pixels 220 on the circumference 302, they become a large number, and further, if the radius n is changed, the number becomes infinitely large. Therefore, the number of corresponding points 304 and the radius n with respect to the radius n of 1 is limited in order to improve the voting efficiency.

  Here, it is known that the tangent of the circle is perpendicular to the line connecting the contact point and the center of the circle. Also, the tangent of the circle corresponds to the edge extension direction in the pixel 220. Therefore, the corresponding points 304 appear only on line segments perpendicular to the edge extending direction of the feature points 300. Therefore, the voting unit 166 can grasp the edge extending direction of the feature point 300 and determine the corresponding point 304 in the direction perpendicular to the edge extending direction.

Here, assuming that the luminance when the coordinates of each pixel 220 is (i, j) is A (i, j), the voting unit 166 outputs a Sobel filter in the vertical direction as shown in Formula 2 below. A line segment perpendicular to the edge extending direction is derived by the ratio of absolute values of the Sobel filter in the horizontal direction.
Line segment perpendicular to edge extension direction = atan (
| A (i + 1, j + 1) + 2A (i, j + 1) + A (i-1, j + 1) -A (i + 1, j-1) -2A (i, j-1) ) -A (i-1, j-1) |
/ | A (i + 1, j + 1) + 2A (i + 1, j) + A (i + 1, j-1) -A (i-1, j + 1) -2A (i-1, j) j) -A (i-1, j-1) |
) ... (Equation 2)

  Here, although the example which derives the line segment perpendicular | vertical to the edge extension direction with the Sobel filter is mentioned, it is not restricted to such a case, Various existing techniques can be applied. In addition, although division and arctangent (atan) are used in Equation 2, when the processing load increases due to these, the edge stretch direction with the absolute value of the Sobel filter in the vertical direction and the Sobel filter in the horizontal direction as an input A lookup table may be used to uniquely derive a line segment perpendicular to.

  For example, as shown in FIG. 12B, when the edge extension direction of the feature point 300 is indicated by a broken line segment 306, the corresponding point is on the line segment 308 of a dashed dotted line corresponding to the direction perpendicular to the line segment 306. 304 can be defined. Here, if a radius n of 1 is mentioned, the corresponding point 304 can be narrowed to two points separated from the feature point 300 by a radius n on the line segment 308 in a direction perpendicular to the edge extension direction of the feature point 300 it can.

  In addition, the road sign may be one or more in size according to the laws and regulations of each country. Then, the relative distance z determines the size of the road sign on the color image 126. Therefore, the voting unit 166 estimates the size (radius n) of the road sign on the color image 126 using the inverse function of the function from which the three-dimensional position information is derived according to the relative distance z, and uses it for voting. Narrow the number of radius n. For example, when the size of the road sign presenting the speed limit or the road sign presenting the release of the speed limit is limited to three, as shown in FIG. 12C, the number (3) × 2 corresponds to the points 304 is squeezed.

  Thus, the corresponding point 304 for the radius n of 1 is restricted on the line segment 308 in the direction perpendicular to the edge extension direction of the feature point 300, and the number of the radius n is determined according to the predetermined size and relative distance z. By limiting the number to one or more, it is possible to avoid an unwilling vote that the corresponding point 304 should not exist. Therefore, it is possible to prevent the erroneous detection of the road sign due to the erroneous setting of the corresponding point 304, and to reduce the processing load by avoiding the useless Hough transform processing.

  The voting unit 166 votes the voting table 230 described above after limiting the corresponding points 304 in this manner. Here, although a three-dimensional voting space is mentioned and explained, M dimension (M is positive) which expanded the dimension (for example, rotation) about sideways and up-and-down direction as a response to the case where a road sign inclines sideways or up-down direction. Can also form a voting space of

  FIG. 13 is an explanatory diagram for explaining the voting table 230. As shown in FIG. The voting table 230 usually comprises a three-dimensional voting space such as the number of horizontal pixels H of the color image 126 × the number of vertical pixels V × the possible value N of the radius n, as shown in FIG. The number of votes is held at the three-dimensional position (point) of the screen position (x, y) of the pixel and the radius n. For example, assuming that the maximum value of the number of votes is 255 (1 BYTE), the size of the voting table 230 is H × V × N (BYTE). Then, if the color image 126 of high resolution is used, in addition to the problem that the memory storage area spent in the voting table 230 becomes large, and if the number of votes is limited (small), the number of votes obtained due to the influence of noise or the like. The problem that a peak becomes difficult to come out may also arise.

  Regarding the latter problem, taking measures against the voting process with a margin added in consideration of noise, for example, voting not only on the radius n of the corresponding point 304 but also on the radius n near the corresponding point 304 However, the processing load increases. Further, it is conceivable to lower the resolution and vote, for example, in a block unit of horizontal 2 pixels × vertical 2 pixels. However, because the resolution is lowered, deterioration in the specific accuracy of the road sign of the corresponding point 304 can not be avoided.

  Therefore, in the present embodiment, two voting tables 230a (first voting table) and 230b (second voting table) having different dimensions and resolutions are provided, and the voting unit 166 simultaneously performs each of the voting tables 230a and 230b. I will vote.

  In the voting table 230a, as shown in FIG. 13B, the resolution of the color image 126 remains the number of horizontal pixels H × the number of vertical pixels V, and information of the radius n is omitted to reduce one dimension. It is shown in the voting space of (horizontal pixel position, vertical pixel position). Therefore, the size of the voting table 230a is H × V (BYTE), and the number of votes for all the radius n is held at the two-dimensional position of the screen position (x, y) of the pixel to be voted. On the other hand, in the voting table 230b, as shown in FIG. 13C, the resolution with respect to the radius n is the value N as it is, and the dimensions are not omitted. Instead, the resolution of the color image 126 is 1 / H each in horizontal and vertical. Three-dimensional voting space compressed to 4 (with reduced resolution) horizontal pixel number H / 4 (value obtained by compressing horizontal pixel number) × vertical pixel number V / 4 (value obtained by compressing vertical pixel number) It is indicated by. Therefore, the size of the voting table 230b is H / 4 × V / 4 × N (BYTE), and the block (four horizontal pixels × four vertical pixels) to which the screen position (x, y) of the pixel to be voted belongs The number of votes is held at the three-dimensional position with the radius n. As described above, the voting tables 230a and 230b are the one with the resolution of the radius n lowered and the one with the resolution of the screen position lowered.

  When the voting unit 166 derives the corresponding point 304 based on the feature point 300, the voting unit 166 simultaneously votes in the respective voting tables 230a and 230b. However, for the voting table 230a, regardless of the radius n, it votes for the point of 1 corresponding to the corresponding point 304, and for the voting table 230b, votes for the point of the radius n of the block to which the corresponding point 304 belongs. Thus, when the voting is completed, the voting unit 166 sets a point (corresponding point 304) having a large total number of votes of the radius n in the voting table 230a as a candidate for the center position of the road sign. It is possible to set the radius n having a large number of votes in the corresponding block as a candidate for the radius n of the road sign.

  In this manner, the overall storage capacity of the voting table 230 can be reduced to H × V + H / 4 × V / 4 × N (BYTE) while maintaining the accuracy of identifying the center of the road sign with high accuracy. It becomes possible. Here, assuming that H = 600 pixels, V = 200 pixels, and N = 20 pixels, the originally required 600 × 200 × 20 = 2,400,000 BYTE is equivalent to approximately 1/10, 600 × 200 + 600/4 × 200. It can be reduced to / 4 × 20 = 270,000 BYTE.

  By the way, the voting section 166 extracts the number of votes obtained from each point of the voting table 230 after finishing the voting process for all the feature points, and the corresponding point 304 where the total number of votes obtained with the radius n is equal to or more than a predetermined value. As a candidate for the center point of the road sign. However, although the storage capacity of the voting table 230 is reduced, when the voting space has a certain size, determining the size of the number of votes obtained in the entire voting table 230 increases the processing load. Therefore, the voting unit 166 improves the extraction efficiency of the center point candidate by selecting the center point candidate in parallel with the voting.

  FIG. 14 is an explanatory diagram for explaining the center point candidate list 310. As shown in FIG. Here, in addition to the voting table 230, a center point candidate list 310 is provided. In the center point candidate list 310, at least screen position (x, y) related to the voting table 230a is registered. When the number of votes on the voting table 230a of the point currently voting is equal to or more than a predetermined number, the voting unit 166 additionally registers the corresponding point 304 corresponding to the point in the center point candidate list 310 as needed. Then, the voting unit 166 sets only those registered in the center point candidate list 310 as road sign candidates after finishing the voting process at all feature points.

  With this configuration, it is possible to appropriately extract center point candidates while avoiding determination of the number of votes of the entire voting table 230, that is, reducing the processing load. In addition, the voting unit 166 limits the candidates of the center point registered in the center point candidate list 310 to a predetermined value (for example, 50). This is due to the following reasons. That is, when the corresponding points 304 are dispersed into a plurality of center point candidates due to the influence of noise or the like, although it is originally a road sign of 1, there are cases where a plurality of center points become candidates. In this case, candidates for center points should not be extracted without limit, and usually, the possibility that 50 or more labels will be present in the color image 126 is small. As described above, when the number of center point candidates in the center point candidate list 310 becomes 50 or more, the voting unit 166 cancels the voting process in one frame, and for the center point candidates identified up to that point Perform the processing of

  Thus, the center point candidate list 310 is generated by the voting unit 166, and the center point candidate list 310 includes the radius, the number of votes obtained in the voting tables 230a and 230b, and the third order of the pixel in addition to the screen position (center position). Information such as the original position is also associated.

  When the voting unit 166 thus votes on the voting tables 230a and 230b, each point of the voting tables 230a and 230b is initialized so that the number of votes becomes 0 for voting in the next frame. However, depending on the resolution of the color image 126, the processing time spent for initializing the voting tables 230a and 230b can not be ignored, and may occupy, for example, 40% of the entire marker detection process S202. In the voting tables 230a and 230b, as the number of dimensions increases, the storage capacity of the memory increases, and in particular, the influence of the load of the initialization processing on the three-dimensional voting table 230b increases.

  FIG. 15 is an explanatory diagram for explaining the flag table 320. As shown in FIG. Here, a flag table 320 is provided in addition to the voting tables 230a and 230b. The flag table 320 is a table in which the number of dimensions of the voting table 230b is reduced by one radius (one dimension reduced) as shown in FIG. 15A, and the screen position (x, y) of the pixel to be the voting destination A flag is set at the two-dimensional position of the block (four horizontal pixels × four vertical pixels) to which X belongs. Therefore, the size of the flag table 320 is H / 4 × V / 4 (BYTE).

  When the voting unit 166 votes at an arbitrary radius n of an arbitrary block of the voting table 230b indicated by cross hatching in FIG. 15 (b), in the flag table 320 indicated by cross hatching in FIG. 15 (a), Turn on the flag of the block equal to that block. Therefore, when one of the radius n in each block of the voting table 230b is voted, the corresponding block of the flag table 320 is turned ON. Then, when the voting on the voting tables 230a and 230b is completed, the voting unit 166 corresponds to the block indicated by hatching in FIG. 15A and for which the flag is turned on in the flag table 320, FIG. In each of the N blocks of the voting table 230b indicated by hatching, each point is initialized so that the number of votes is 0. In other words, the voting unit 166 does not perform initialization processing for the block of the voting table 230 b corresponding to the block whose flag is turned off in the flag table 320.

  Here, although the voting table 230b is listed as the target of the flag table 320, the concept of the flag table 320 is not limited to this case, and the concept of the flag table 320 is also applied to the voting table 230 of size H × V × N (BYTE). it can. In that case, the size of the flag table 320 is H × V (BYTE).

  Further, instead of providing the flag table 320, it is judged whether or not the voting is done in the area corresponding to each block of the voting table 230a, and if the voting is done, only the blocks of the voting table 230b corresponding to that block Each point may be initialized so that the number of votes is zero. In this way, it is possible to significantly reduce the load of the initialization process because it is not necessary to initialize all the points of the voting table 230b.

  Further, when the voting on the voting tables 230a and 230b is completed, the voting unit 166, as indicated by hatching in FIG. 15 (c), of the voting table 230a corresponding to the block for which the flag is ON in the flag table 320. Only the plurality of pixels (four horizontal pixels × four vertical pixels) may be initialized so that each point is zero. Thus, as with the voting table 230b, there is no need to initialize all the points of the voting table 230a, so the load of the initialization process can be significantly reduced.

  Then, the voting unit 166 initializes all the flags of the flag table 320 to OFF after the initialization process is completed. In this way, it is possible to properly initialize the voting table without increasing the load of the initialization process.

(Sign identification process)
The sign specifying unit 168 narrows down the road sign candidate derived by the voting unit 166 based on the first to third narrowing conditions, and specifies the road sign.

  As a first narrowing condition, the sign specifying unit 168 sets a radius n where the number of votes in the voting table 230a is a predetermined value or more and the number of votes in a block in the voting table 230b is a predetermined value or more as a center point and a radius, respectively. Narrow down. However, as described above, the voting unit 166 also registers the corresponding points 304 in the center point candidate list 310 as needed according to the number of votes obtained in the voting tables 230a and 230b. However, the registration to the center point candidate list 310 is performed during a vote where the final number of votes is still unknown, and it is not a determination of the number of votes obtained when the voting is completed. Therefore, in the present embodiment, the corresponding points 304 corresponding to noise are excluded by again comparing the corresponding points 304 registered in the center point candidate list 310 with a predetermined value larger than that at the time of registration. It is possible to leave only the appropriate corresponding point 304.

  Subsequently, based on the center position and the radius n, the sign specifying unit 168 derives a rectangle whose center is twice the length of the radius n and whose center is the center position as the occupied area. However, with regard to any two corresponding points 304, when such occupied areas overlap (overlap), one may cause the other to be unrecognizable. In this case, when the road sign at the other corresponding point 304 which has become unrecognizable is an important road sign such as a road sign presenting a speed limit, such an important road sign may not be recognized. It will Therefore, when the occupied area of any corresponding point 304 overlaps the occupied area of another corresponding point 304 on the screen as the second narrowing condition, the sign specifying unit 168 has low reliability as a road sign Leave a reliable road sign by excluding people. The reliability of such a road sign is obtained based on the magnitude relationship between the number of votes obtained on the two-dimensional voting table 230a and the number of votes obtained on the three-dimensional voting table 230b.

FIG. 16 is a flowchart showing an example of the marker identification process. As shown in FIG. 16, the sign specifying unit 168 sequentially selects two candidates out of the road sign candidates (S330). Then, the label specifying unit 168 determines whether or not the occupied areas of the two selected candidates overlap (S332). As a result, if the occupied areas of the two candidates overlap (YES in S332), the marker identifying unit 168 determines that the votes C ₁ and D ₁ of the voting tables 230a and 230b of one candidate are the other candidates. It is determined whether or not the number of votes C ₂ and D _{2 of the} voting tables 230 a and 230 _b is greater than either of the _two (S 334). As a result, if all are large, that is, if C ₁ > C ₂ and D ₁ > D ₂ (YES in S334), the label identifying unit 168 excludes the other candidate (S336). If the two candidate occupied areas do not overlap (NO in S332), the process moves to step S344.

Further, if C ₁ > C ₂ and D ₁ > D ₂ (NO in S334), the marker identifying unit 168 determines that the number of votes C ₁ and D ₁ of the voting tables 230a and 230b of one candidate is the other candidate It is determined whether or not the number of votes C ₂ and D ₂ of the voting tables 230 a and 230 _b is smaller than any of the votes C ₂ and D ₂ (S 338). As a result, if all are small, that is, if C ₁ <C ₂ and D ₁ <D ₂ (YES in S338), the label identifying unit 168 excludes one candidate (S340). As described above, when the number of votes obtained in the candidate voting tables 230a and 230b is large, it is assumed that the reliability of the road sign is high, and the candidates with small numbers of votes are excluded and only the candidates with large numbers of votes are excluded. leave.

Further, if C ₁ <C ₂ and D ₁ <D ₂ (NO in S 338), the marker identifying unit 168 lowers any one of the candidate and the other candidate based on the position on the color image 126. The candidate located at is excluded (S342). As described above, when one of the votes obtained from the candidate voting tables 230a and 230b is large and any one is small, it can not be determined only by the number of votes obtained, so only the candidate positioned above is adopted and positioned below. Exclude candidates. This is because if two road signs are arranged vertically in the vertical direction, a road sign presenting a relatively important speed limit is arranged at the upper side.

  If two candidates selected in this manner overlap, and it is determined which to exclude, the label identifying unit 168 determines whether all combinations of the two candidates to be selected have ended. (S344). As a result, if it has ended (YES in S344), the label identification processing is ended, and if it has not ended (NO in S344), the processing from step S330 is repeated. In this way, even when the occupied areas of the two road sign candidates overlap, it is possible to appropriately narrow down to highly reliable candidates.

  Next, as a third narrowing-down condition, the sign specifying unit 168 determines whether the road sign candidate narrowed down by the first and second narrowing-down conditions exceeds the candidate upper limit value (here, 3). Here, if the candidate upper limit value is exceeded, the marker identifying unit 168 narrows the candidates to the candidate upper limit value or less, and does not execute the subsequent processes for other candidates. Specifically, when the road sign candidate exceeds the candidate upper limit, the horizontal distances x at the three-dimensional positions of all the candidates are compared, and the candidate upper limit is ordered in order of decreasing horizontal distance x from the lane of the vehicle 1 Focus on candidates for In this manner, it is possible to appropriately extract a candidate close to the lane of the host vehicle 1 which is likely to be a road sign for the host vehicle 1.

  Subsequently, the sign correction unit 170 corrects the position and the size of the road sign narrowed down to the candidate upper limit value or less. This is because, in the present embodiment, template matching is used to recognize the content of a road sign, and in template matching, positional deviation of an image greatly affects recognition accuracy. Therefore, here, the central position and radius n derived by the voting unit 166 are corrected, and the occupied area of the road sign is set again. For this reason, the sign correction unit 170 detects a red frame existing in four directions in the horizontal direction and the vertical direction from the center position of each candidate of the road sign, and the red frame (circumferential portion of the road sign) circumscribes a rectangle Correct the occupied area so that Specifically, the occupied area is corrected by the following procedures (1) to (7).

  FIG. 17 is an explanatory diagram for explaining the process of the sign correction unit 170. As shown in FIG. (1) First, the sign correction unit 170 sets V as the entire occupied area 346, with a rectangle having a length twice the radius n as one side and the center position as the center shown in FIG. The histogram of the components is determined (the horizontal axis is voted as the V component). Then, when the difference between the maximum value and the minimum value of the histogram of the V component is equal to or more than a predetermined value, the marker correction unit 170 determines that the candidate is a red frame. If the difference between the maximum value and the minimum value of the V component histogram is less than a predetermined value, it is determined that the candidate is a black frame, and the V component histogram is replaced with the Y component histogram. In the following, correction processing of the occupied area 346 will be described by citing red frame candidates, but it goes without saying that the present invention can naturally be applied to a black frame.

  (2) The marker correction unit 170 obtains a predetermined% (for example, 30%) from the top of the histogram of the V component (upper: lower area ratio of 3: 7 when the histogram is converted to an area) V The component value (threshold Vthr) is derived. The threshold value Vthr is different from the predetermined value of the V component used to identify the feature point. This is to set an optimal threshold for each candidate. When the threshold value Vthr becomes equal to or less than a predetermined value (for example, -5), the subsequent processes are not performed as an inappropriate threshold value.

  (3) The sign correction unit 170 determines whether the V component of each pixel is equal to or more than the threshold value Vthr from the center position of each candidate of the road sign, as indicated by the arrow in FIG. The detection pixel is moved in four directions, the direction and the vertical direction. Then, when a predetermined number (for example, three pixels) of pixels whose V component exceeds the threshold Vthr continue, the pixel whose V component above the threshold Vthr starts to be detected is set as the inner edge 348 of the red frame. Here, four horizontal and vertical directions are described as detection directions. However, the present invention is not limited to this case, and four orthogonal radial directions may be sufficient, and an oblique direction or the like may be added. The detection accuracy can also be enhanced.

(4) Subsequently, the marker correction unit 170 determines whether or not the position of the inner edge 348 of the red frame is within the predetermined range to be originally positioned. Specifically, for example, when detecting in the horizontal right direction, assuming that the horizontal coordinate of the center is J, the x coordinate of the horizontal right end of the occupied area 346 is R, and the coordinates of the inner edge 348 of the obtained red frame are RE If Equation 3 is satisfied, the coordinate RE of the inner edge 348 of the red frame is regarded as an inappropriate value, and the subsequent processing is not performed.
RE <(R−J) × K + J (Equation 3)

  Here, K is a coefficient taking any value of 0 to 1. For example, 0.6 in horizontal detection and 0.5 in vertical detection. Further, (R−J) × K is a lower limit value (inner edge lower limit value) in the radial direction that the inner edge 348 can take. Such processing is a measure for preventing the erroneous inner edge 348 from being affected by affecting the V component when, for example, the numerical value becomes orange in a road sign of the electronic display type.

  (5) Next, the sign correction unit 170 re-selects the center position and the radius n of the image based on the position of each inner edge 348 when all four directions in the horizontal direction and the vertical direction are matched in any road sign. To derive. Specifically, of the horizontal inner edge 348, the center position when the inner edge 348 located on the left is LE and the inner edge 348 located on the right is RE is defined by (LE + RE) / 2, and the radius n Is determined by (RE−LE) / 2. Further, the center position and the radius n are determined for the vertical inner edge 348 in the same manner. Thus, the occupied area 346 specified by the center position of the road sign is newly defined.

  (6) Subsequently, the sign correction unit 170 compares, for the road sign, the radius n before correction and the radius n after correction, and the ratio is within a predetermined range (for example, 0.75 times or more and less than 1.5 times If it deviates from), the subsequent processing will not be performed as an inappropriate value.

  (7) Finally, the sign correction unit 170 resizes the corrected occupied area 346 into a rectangle of horizontal predetermined pixels × vertical predetermined pixels, and ends the correction processing. As described above, by re-adjusting the center position and the radius of the road sign, it is possible to achieve high accuracy in recognition by pattern matching. In addition, as resizing, a general method such as arrest neighbor can be used.

(Mark content recognition processing S204)
FIG. 18 is a flowchart showing a specific processing flow of the tag content recognition processing S204. The sign content recognition unit 172 discretizes the brightness of the image corresponding to the occupied area 346 with the image as it is for the road sign corrected by the sign correction unit 170 (S350), and the road sign presents the release of the speed limit. It is determined whether or not the road sign is a road sign (S352). Then, the sign content recognition unit 172 determines that the road sign is not the road sign presenting the release of the speed limit, and performs the vertical alignment (S354), unless the road sign presents the release of the speed limit. After alignment, matching in the horizontal direction is performed (S356). Subsequently, the sign content recognition unit 172 focuses on a predetermined portion of the contents of the road sign, performs template matching of the portion of interest (S 358), derives the comprehensive evaluation value, and determines which speed limit it is (S360).

  By the way, as above-mentioned, there exist a lightning display type and a non-light display type in the road sign which this embodiment aims at. The electronic display type is characterized in that the content of the road sign, for example, the brightness of the numerical value is higher than the brightness of its surroundings, and the non-electric display type is such that the content of the road sign, for example, the brightness of the numerical value is lower Have.

  In the label content recognition process S204, since it is not grasped which display type it is, the recognition process is executed on the assumption that both display types are used. For example, the sign content recognition unit 172 executes a series of processes from step S350 to S360 using the image as it is on the road sign corrected by the sign correction unit 170, and determines whether the contents of the road sign can be effectively recognized. (S362). As a result, if it is determined that the contents of the road sign can not be effectively recognized in any of the processes in steps S350 to S360 (NO in S362), the luminance of the road sign corrected by the sign correction unit 170 is reversed. (S364) A series of processes from step S350 to step S360 are executed again on the road sign whose inverted luminance (inverted road sign).

  In addition, if it is determined that the contents of the road sign can be effectively recognized in any of the processes in steps S350 to S360 (YES in S362), the road sign is not reversed or the contents of the reversed road sign The process moves to step S366 without recognizing. Thus, it is possible to recognize either the road sign or the inverted road sign corrected by the sign correction unit 170, and it is possible to appropriately recognize the contents of the road sign regardless of the difference in display type such as lightning non-lighting It becomes.

  However, if it is determined that the contents of the road sign can not be effectively recognized in any of the steps S350 to S360 before the reversal, the process is interrupted even during the recognition process, and the subsequent processes are omitted. The processing can also be transferred to step S362. Thus, unnecessary recognition processing can be avoided and processing load can be reduced. The image after correction and the image obtained by inverting the luminance thereof are subjected to the same processing. Therefore, for convenience of explanation, in the following, processing of the image after correction will be described, and a detailed description of the image whose luminance is inverted I omit it.

  Here, after the processing of the image after correction, the processing of the image whose luminance is reversed is performed, but the order may be reversed. For example, if the road sign presenting the speed limit is located on the road shoulder according to the environment outside the vehicle, the processing of the image after correction is processed first because the road sign is likely to be of the non-electric display type. If the road sign is located, the road sign is likely to be of the electric display type, so that the process is first performed on the image whose brightness is reversed. In this manner, by performing processing in descending order of possibility of evaluation being completed in one cycle (steps S350 to S360), it is possible to achieve efficiency of the label content recognition processing S204.

  Further, if the contents of the road sign can be effectively recognized in any of the image after correction or the inverted image obtained by inverting the luminance thereof (YES in S362), the sign content recognition unit 172 recognizes all the road signs corrected by the sign correction unit 170. It is determined whether or not the process of steps S350 to S364 has been executed (S366). As a result, if all the road signs are not completed (NO in S366), the sign content recognition unit 172 repeats the processing from step S350 until the completion (YES in S366). The contents of the processes of steps S350 to S360 will be described in detail below.

(Luminance discretization processing S350)
The sign content recognition unit 172 discretizes the luminance of each pixel in the occupied area 346 of each road sign corrected by the sign correction unit 170. Thus, the image pattern is converted into a recognizable image pattern regardless of the imaging state.

  FIG. 19 is an explanatory diagram for describing the recognition target area 370. As shown in FIG. First, the tag content recognition unit 172 sets an area (hereinafter referred to as a recognition target area) 370 to be subjected to recognition processing in a rectangle of horizontal predetermined pixels × vertical predetermined pixels. The recognition target area 370 is a rectangular area in contact with the inner edge 348 of the red frame in the occupied area 346 shown in FIG. 19, and in the subsequent processing, the recognition target area 370 is processed.

  Next, the tag content recognition unit 172 discretizes each pixel of the recognition target area 370 to achieve N-value conversion. For example, in the case of N = 2, the luminance of each pixel will have a value of either 0 or 255. Here, N is a value of 2 or more, but in the present embodiment, for example, N = 5, in order to suppress the influence on pattern matching in the case where binarization does not go well due to the influence of setting of a threshold or the like. Do. In the case of pentanization, there are four discretization thresholds, and four predetermined percentages (for example, 20, 25, 35, 40%) from the top of the luminance histogram are set. The predetermined percentage can be set independently and arbitrarily.

  According to this configuration, the contents of the road sign can be appropriately recognized regardless of the difference in the distribution of the luminance values. Further, since the upper value of the luminance histogram is used as a reference here, it is possible to appropriately perform five-value conversion regardless of the luminance distribution mode of each of the recognition target areas 370, and normalization can be achieved along with discretization. It will be.

(Speed limit release determination process S352)
The sign content recognition unit 172 performs recognition processing corresponding to the road sign presenting the release of the speed limit or the road sign presenting the speed limit with respect to the 5-valued recognition target area 370, but the former process is more effective. Since the load can be reduced, first, processing is performed on the premise that the former is used, and processing for the latter is performed if not. In this way, it is possible to avoid performing recognition processing of the road sign that presented the speed limit unnecessarily.

FIG. 20 is an explanatory view for explaining a road sign presenting cancellation of the speed limit. The tag content recognition unit 172 is a recognition target area 370 shown in FIG. 20, which intersects on the recognition target area 370 and has a plurality of inclined line segments L1, L2, L3 and L4 corresponding to four inclined line segments (tilted). The luminances of the pixels are accumulated, and the accumulated luminance values at the line segments L1, L2, L3, and L4 are S1, S2, S3, and S4, respectively. The inclination angle of such a line segment is an angle in accordance with the display mode of the road sign for which release of the speed limit has been presented. Here, threshold values for determining the respective accumulated luminance values S1, S2, S3, and S4 are taken as TS1, TS2, TS3, and TS4. However, TS4 is comprised from two threshold values which have a relationship of TS4a <TS4b. Also, TS2 = TS3 may be set. The sign content recognition unit 172 presents the release of the speed limit of the road sign when the formula 4 is all satisfied, based on the following formula 4 for quantifying the difference in density in the four line segments L1, L2, L3, L4. Recognize it as a road sign.
S1 <TS1
S2> TS2
S3> TS3
TS4a <S4 <TS4b (Equation 4)
Here, since the deviation of the luminance due to the positional deviation and the brightness is corrected, it is possible to recognize the content of the road sign by a very simple process as in the above-mentioned equation 4.

(Vertical alignment processing S354)
If the road sign is not determined to be the road sign presenting the release of the speed limit by the above processing, the road sign is determined as the road sign presenting the speed limit. The label content recognition unit 172 first performs alignment in the vertical direction with respect to the numerical value area occupied by the numerical value in the recognition target area 370. This is because the recognition target area 370 may include minute positional deviations, or the size, shape, distance between the numerical values, and the like may differ depending on the installation mode of each country or country.

  FIG. 21 is an explanatory view for explaining the vertical alignment processing. The sign content recognition unit 172 accumulates the luminance of each pixel of the recognition target area 370 in the horizontal direction, arranges the accumulated luminance values in the vertical direction, and generates a luminance distribution 372 in the vertical direction, as shown in FIG. However, in the case of the road sign of FIG. 21, since the numerical value portion has low luminance relative to the surroundings, in the luminance distribution 372 of FIG. 21, the cumulative luminance value is inverted after the luminance is extracted to extract the numerical value portion. I'm asking. Then, the tag content recognition unit 172 obtains the maximum value 374 of the accumulated luminance value for all vertical upper and lower areas from the central part of the recognition target area 370, and using a predetermined% (for example 25%) of the maximum value as a threshold The detection pixels are moved vertically and vertically, and when the predetermined number (for example, twice) of pixels whose accumulated luminance value falls below the threshold value continue, that location is set as the upper end and lower end of the numerical value area 376.

  Next, the tag content recognition unit 172 uses the upper end and the lower end of the numerical value area 376 derived in this manner to enlarge or reduce the size in the vertical direction and normalize it. For example, assuming that the distance between the upper end and the lower end of the numerical value area 376 is HI, and the distance in the vertical direction of the template is HT, the tag content recognition unit 172 vertically multiplies the numerical value area 376 by HT / HI. Thus, the size of the numerical value area 376 can be adjusted to the vertical size of the template to be matched later. Also, the correction is performed by the nearest neighbor.

  By such processing, for example, although noise is generated in the accumulated luminance value in the lower part of FIG. 21, the influence thereof can be excluded, and the normalized numerical region 376 can be appropriately extracted.

(Horizontal matching process S356)
FIG. 22 is an explanatory diagram for describing a template. The marker content recognition unit 172 recognizes the content of the recognition target area 370 in which the vertical alignment is performed, that is, the numerical value. Such recognition is performed by matching with a previously prepared template. As shown in FIG. 22, for example, 13 types of templates, 10 to 90 (2 digits), 100 to 130 (3 digits) and 10 steps, are prepared.

  FIG. 23 is an explanatory diagram for explaining the matching processing in the horizontal direction. The sign content recognition unit 172 accumulates each pixel of the recognition target area 370 normalized in the vertical direction in the vertical direction and arranges the accumulated luminance values in the horizontal direction as shown in FIG. Generate Thus, one-dimensionalization of the two-dimensional image is performed. However, as in the alignment in the vertical direction, since the numerical value portion has low luminance, in the luminance distribution 380 of FIG. 23, in order to extract the numerical value portion, the accumulated luminance value is obtained after inverting the luminance. Also, the cumulative range in the vertical direction is limited to only the range from the upper end to the lower end of the numerical value area 376 derived by the vertical alignment. Therefore, it is possible to avoid the accumulation of luminance for unnecessary areas other than the vertical numerical area. The sign content recognition unit 172 DP-matches the brightness distribution 380 of FIG. 23 derived in this manner and the brightness distribution based on the template so that the correlation evaluation value with each template (the lower the correlation evaluation value, the higher the correlation is Calculate).

  Here, as in the case of the alignment in the vertical direction, since there are differences in numerical value size and spacing between the numerical values, sufficient performance can not be obtained when using a fixed size template. Therefore, DP matching is used in which horizontal expansion and contraction are allowed. Although it is possible in principle to perform such DP matching in two dimensions, one-dimensional DP matching is used in the present embodiment because the processing amount becomes enormous.

  FIG. 24 is an explanatory diagram for explaining the DP matching. The label content recognition unit 172 performs, for example, DP matching with the template of “130” to obtain a result as shown in FIG. Here, the broken line indicates the luminance distribution 380 of the recognition target area 370 before DP matching, the solid line indicates the luminance distribution 382 of the recognition target area 370 after DP matching, and the alternate long and short dash line indicates the luminance distribution 384 of the template. In DP matching, matching is performed by expanding and contracting the luminance distribution 380 of the recognition target area 370 so as to match the luminance distribution 384 of the template. Therefore, as can be understood with reference to FIG. 24, although the correlation between the luminance distribution 380 of the recognition target area 370 before expansion and contraction and the luminance distribution 384 of the template is low, the luminance distribution 382 of the recognition target area 370 after expansion and contraction and the template The correlation with the luminance distribution 384 is high.

  However, here, all correlation evaluation values between the luminance distribution 382 of the recognition target area 370 and the luminance distribution 384 of the plurality of templates after expansion and contraction are obtained regardless of the level of the correlation. Specifically, when the luminance distribution 382 of the recognition target area 370 after expansion and contraction is im, and the numerical value (limit speed) of the template is T, the marker content recognition unit 172 determines the correlation evaluation value after expansion and contraction (square sum of differences ) Is derived sequentially from DP (im, 10) to DP (im, 130).

  However, for the candidate that is obviously different from the template, the subsequent processing is not performed. For example, while the luminance distribution 382 of the recognition target area 370 is “130”, the two-digit “10” to “90” originally have different numbers of digits, so DPs corresponding to “10” to “90” im, 10) to DP (im, 90) have a low correlation. Therefore, the subsequent processing is omitted for a template in which the value of DP (im, T) exceeds the threshold (the correlation is low).

(Target part matching process S358)
Here, the correlation evaluation value DP (im, T) is obtained regardless of the number of digits of the numerical value, but when the tendency of change of the numerical value is determined in advance as in the present embodiment, all two-digit and three-digit numerical values It is not a good idea to perform matching in This is because, for example, in the numerical values of "10" to "90", the portion of "0" in the first digit is common to all the numerical values, and in the case of "100" to "130," the "0" in the first digit This is because the part and the part of "1" in the third digit are common. Therefore, with regard to the common part, since any numerical value is the same, when matching is performed for all the digits, a difference in the correlation evaluation value hardly occurs.

  Therefore, the marker content recognition unit 172 finds the correlation evaluation value DP (im, T) as described above, and matches only the second digit that causes a difference in the shape of the numerical value. However, since the luminance distribution 382 of the recognition target area 370 is expanded and contracted in the horizontal direction, it is necessary to derive which part of the luminance distribution 382 of the recognition target area 370 matches which part of the luminance distribution 384 of the template. You must. Therefore, the marker content recognition unit 172 calculates the horizontal coordinate DPR (corresponding to the start position of the second digit of the luminance distribution 382 of the recognition target area 370) corresponding to the horizontal coordinate TS (T) of the start position of the second digit of the template. Derivate im, T). Such horizontal coordinates can be obtained from the history of processing for combining feature points in DP matching. Specifically, information (for example, a route) related to the combination of feature points is stored in advance, and the horizontal coordinate is derived by backward calculation. According to this configuration, horizontal coordinates can be efficiently obtained by using an intermediate result of DP matching. The specific procedure of such DP matching has already been disclosed through various technical documents, and thus the detailed description thereof is omitted here.

  FIG. 25 is an explanatory view for explaining the matching processing of the target portion. As described above, when the horizontal coordinate corresponding to the start position of the second digit numerical value area is obtained, the tag content recognition unit 172 performs simple template matching. Although any index may be used for template matching, for example, the sum of absolute differences (SAD) is used. The target range of the matching is, as shown in the example of “130” in FIG. 25, the horizontal coordinate DPR (im, T) corresponding to the start position of the luminance distribution 382 of the recognition target area 370, and the second digit of the template. From the horizontal coordinate TS (T) corresponding to the start position of, the horizontal length of the numerical value of the second digit of the template. Here, since alignment in the horizontal direction has already been performed, there is no need to perform processing such as optimum value search by matching with shifted positions, and the processing load is significantly reduced.

  However, for example, when the number is two digits and three digits, since the horizontal length (width of the number) is different, the difference in the width of the number may affect the matching result. Therefore, the tag content recognition unit 172 uses the second evaluation value TM (im, T) of the DP matching in the second digit, the horizontal width of the recognition target area 370 having a large number of digits, and the recognition target area 370 having a small number of digits. In accordance with the ratio to the width of, the template is multiplied by a predetermined normalization factor. For example, if the width ratio of the 2-digit numerical value to the 3-digit numerical value is 3: 2, the marker content recognition unit 172 evaluates the second digit correlation evaluation value TM (im, T) of "100" to "130". After deriving), the value obtained by multiplying the value by 3/2 is taken as TM (im, T). Thus, appropriate evaluation can be performed regardless of the number of digits.

(Evaluation value determination processing S360)
Subsequently, the marker content recognition unit 172 uses the correlation evaluation value DP (im, T) obtained for each template and the correlation evaluation value TM (im, T) in the second digit, and uses the templates of Equations 5 and 6 below. The total evaluation value E (im, T) for each is derived.
Comprehensive evaluation value E (im, T) = DP (im, T) x TM (im, T) / F (im) ... (Equation 5)
F (im) = max (min (TM (im, T)), th) (6)
Here, since the correlation of the whole numerical value is expressed by DP (im, T), the value as it is is used, and the partial correlation of the second digit is compared with other templates TM (im, T) / F ( Expressed by im). Here, F (im) is the minimum value min (TM (im, T)) of the correlation evaluation value, but if the value of min (TM (im, T)) becomes too small, the comprehensive evaluation value E (im) , T) may diverge, so if min (TM (im, T)) becomes smaller than a predetermined value th, the value th is adopted as F (im).

  FIG. 26 is an explanatory diagram for describing an evaluation determination result. Here, the broken line indicates the correlation evaluation value DP (im, T), the dashed-dotted line indicates the second digit correlation evaluation value TM (im, T), and the solid line indicates the comprehensive evaluation value E (im, T). There is. Referring to FIG. 26, for "100" to "130", the difference between the original numerical value and the other numerical values is small in matching in all digits of the numerical value, but adding the matching only in the second digit actually It can be understood that the integrated evaluation value E (im, T) of the matching with the template of “130” which is the numerical value of is the smallest (the correlation is the largest). Note that FIG. 26 also displays a template for which the calculation is originally omitted, and also displays templates up to “150” for the convenience of description.

  As described above, the accuracy can be improved and the processing time can be reduced by performing the matching in all the digits of the numerical value and the matching of only the second digit.

(Label contents determination processing S206)
Above, the contents of the road sign were recognized. However, as described above, it is not necessary to recognize the road sign at the moment when the road sign reaches the position where it can be confirmed forward, and it is sufficient if it can be recognized when passing the road sign or later. Therefore, the road sign may be recognized over a plurality of frames, and the contents of the road sign may be determined from the information of the plurality of frames. Therefore, the sign content determination unit 174 accumulates and determines the contents of the road sign recognized in one frame in this manner in the time direction.

  Here, in order to determine the content of the road sign, four variables of a sign accumulation point, a speed limit candidate, a sign no detection time, and a speed limit output are used. Here, the sign accumulation point is provided for each of one or more candidates of the road sign, and various evaluation values (E (im, T), DP (im, T), TM (im, T) in the sign content recognition processing S 204 are provided. Indicate points according to T)). The speed limit candidate indicates one candidate for the speed limit. The marker non-detection time indicates a continuous time when no road sign is detected. The speed limit output is used to latch speed limit candidates. When the speed limit output is updated by the speed limit candidate, while the value is held in the speed limit output, the value is notified to the driver or used as a control input of the vehicle control device 130.

The marker content determination unit 174 determines various evaluation values (E (im, T), DP (im, T), TM (im, T)) for evaluating the certainty of the speed limit derived by the marker content recognition unit 172. The marker accumulation point is accumulated according to the following conditions (1) to (4) using.
(1) If E (im, T) <ETHR1 & DP (im, T) <DTHR1 & TM (im, T) <TTHR1 + 4 points (2) (1) is not satisfied, and E (im, T) <ETHR2 & DP (im, T) <DTHR2 & TM (im, T) + 2 points if TTHR2 but ETHR1 <ETHR2, DTHR1 <DTHR2, TTHR1 <TTHR2
(3) Of the templates satisfying the condition of (1), the minimum value EM of E (im, T) and the difference between E (im, T) of all the other templates are at least a predetermined value. Among the templates satisfying the condition of +2 points (4) (1) in the template of value EM, the difference between the minimum value EM of E (im, T) and E (im, T) of one or more other templates Is within the predetermined value (ETHR3), +1 point is added to all the templates within the predetermined value. Thus, basic points are added according to the conditions of (1) and (2), (3) and (4) Add points based on comparison with other templates according to the condition of.

  For example, in the example of FIG. 26, assuming that the conditions (1) to (4) are ETHR1 = 80, ETHR2 = 100, ETHR3 = 5, DTHR1 = 80, DTHR2 = 100, TTHR1 = 100, and TTHR2 = 150 "120", "130" +4 points based on the condition 1), "100" and "150" +2 points based on the condition (2), "130" +2 points based on the condition (3) It becomes a point. In total, "130" is 6 points, "120" is 4 points, "100" and "150" are 2 points, and the other numerical values are 0 points. In addition, when it is recognized as a road sign which presented speed limit in the speed limit cancellation determination processing, it is uniformly 6 points.

  The sign content determination unit 174 performs final output by accumulating the contents of the road sign in the time direction as follows, based on the sign accumulated points thus obtained.

  FIG. 27 is a time chart showing the flow of notification of the result of a road sign. The sign content determination unit 174 always accumulates sign accumulation points for all the detected road sign candidates for each frame. Then, the sign content determination unit 174 compares the sign accumulation point of the current frame with the sign accumulation point of the previous frame, and if the point does not change, as shown in FIG. The label no detection time is incremented as no label is detected. Also, if the point has changed, as shown in FIG. 27 (2), it is reset that the marker non-detection time is 0, as the road marker is detected in this frame.

  In addition, if a road sign is detected in the current frame, the sign content determination unit 174 is, as shown in FIG. 27 (3), signs of two road sign candidates from the top from a plurality of road sign candidates. Extract the accumulated points and the recognized speed limit. At this point, since it is determined that the speed limit output is newly updated, the currently maintained speed limit output is reset. Therefore, notification of the speed limit is not performed. In this way, it is possible to avoid continuing unnecessary notification of the previous speed limit.

  When the maximum value of the sign accumulation points exceeds a predetermined value (for example, 8 points), the sign content determination unit 174 determines that the difference with the second largest sign accumulation point is a predetermined value (for example, 4 points) or more. As shown in FIG. 27 (4), the speed limit candidate is updated (fixed as an output candidate) at the speed limit (for example, 40) of the road sign at which the sign cumulative point is the maximum value. Thus, candidates for the speed limit are extracted. If it is less than the predetermined value, the speed limit candidate is updated to "undefined". Also, if the maximum value of the sign accumulation point is equal to or less than the predetermined value, the speed limit candidate is not updated. Therefore, the speed limit candidate maintains "no candidate".

  Then, after the change in the sign accumulation point disappears (the road sign is passed) and the output setting time (for example, 3 seconds) elapses, the sign content determination unit 174 is performed, as shown in FIG. 27 (5). Determines whether there is a speed limit candidate. Then, if there is a speed limit candidate, the speed limit output is updated with the speed limit candidate (for example, 40), and notification of the speed limit is resumed. In this way, noisy input of the content of the road sign can be eliminated.

  Subsequently, after the change in the marker accumulation point disappears, when the marker non-detection time elapses a reset time (for example, 5 seconds) longer than the output setting time, as shown in FIG. In preparation for the next road sign, reset the sign accumulation point and speed limit candidate. Thus, preparation is made for recognition of a new road sign.

  Subsequently, after the change in the marker accumulated point disappears, when the marker non-detection time has passed the notification upper limit time (for example, 10 minutes), as shown in FIG. 27 (7), the marker content determination unit 174 outputs the limited speed. Reset. In this way, it is possible to avoid continuing unnecessary notification of the previous speed limit.

  In addition, when it is determined that the vehicle 1 is turning left or right (for example, the absolute value of the steering angle exceeds 360 degrees), the sign content determination unit 174 determines the sign accumulation point, the speed limit candidate, and the limit. Reset all speed outputs. This is because when the host vehicle 1 turns left or right, the road on which the vehicle travels changes, and the speed limit of the road that has been traveled by that time is not applied.

  By configuring in this manner, the sign content determination unit 174 notifies the speed limit 3 seconds after passing the road sign, and until 10 minutes pass, a left turn, or another road sign is detected. Can be maintained. Moreover, it becomes possible to eliminate the noise-like input of the content of a road sign, and to improve the specification precision of the content of a road sign.

  Also, the following processing can be additionally performed to further enhance the practicability. For example, in the case of a plurality of lanes of the vehicle, the road sign arranged at the gate may have different speed limits for each lane. In the present embodiment, since the number of road signs to be recognized is limited to three or less, when the speed limit differs for each lane as described above, a sign accumulation point for each of the different road signs is present. Assuming that only the correct speed limit is accumulated for each road sign, the sign accumulation point is, for example, 6 points in any speed limit. Then, in the above determination, the speed limit candidate is updated to "indefinite" even though the points are appropriately accumulated.

  Therefore, in the present embodiment, the sign content determination unit 174 derives each horizontal distance x when there are a plurality of road sign candidates having significant sign accumulated points at one time in one frame, and any one of them is 1 If the horizontal distance x of the road sign is less than a threshold (for example, 3 m) and the other road sign is equal to or more than the threshold, 1 point is added to the road sign and 1 point is subtracted from the other road sign. In this way, the speed limit of the road sign closest to the host vehicle 1 can be prioritized as the speed limit candidate.

(Difference of road sign every country)
FIG. 28 is an explanatory view showing a display mode of a road sign according to country. The road sign presenting the speed limit is such that it can be understood by comparing the road sign presenting the German speed limit of FIG. 28 (a) with the road sign presenting the Swiss speed limit of FIG. 28 (b) The size, shape, and distance between numbers may differ from country to country. In addition, the road sign indicating the release of the German speed limit in FIG. 28 (c) can be understood by comparing it with the road sign presenting the release of the Italian speed limit in FIG. 28 (d). The angles may be different.

  Therefore, the sign content determination unit 174 determines which country's road sign the speed limit candidate is in parallel with determining and accumulating the contents of the road sign in the time direction as described above. Then, correctly grasp the country where you are currently traveling, and use the template of that country to properly recognize the speed limit, etc.

  FIG. 29 is an explanatory view for explaining a template of a road sign. The country determination process is basically performed using a template for each country. Therefore, as shown in FIG. 29, templates arranged in two dimensions for each country (country A to country Z) and each speed limit ("10" to "150") are prepared. In addition, as for the road sign for which release of the speed limit has been presented, angle information of oblique lines is held instead of the template. In addition, information such as the radius n in the marker detection process S202 and the threshold (predetermined%) of N-value conversion in the marker content recognition process S204 is also held by country.

  Here, when the external environment recognition system 100 is linked with the navigation system, the template may be switched according to the current country information obtained from the navigation system, but when it is not linked with the navigation system, the following procedure Make a country decision.

  In the country determination process, since the request for real-time performance is low, once the image of the recognition target area is acquired, it is temporarily stored in the temporary image memory, and the space after the outside environment recognition process for each frame ends. Perform processing across frames in time. Also, here, in order to determine a country, a country-by-country cumulative point provided for each of one or more country candidates is used as a variable. The labeled content determination unit 174 initializes a temporary image memory area and a country-by-country cumulative point at a predetermined timing.

  In the country determination process, the label content determination unit 174 determines whether the country determination process has already been performed in the current frame. As a result, if the country determination process has already been executed, the process is continued, and if the previous country determination process is completed, the country determination process is newly started. Since the country determination process is performed in the idle time as described above, when the specified process time of a frame is reached in the middle of the process, the process is paused and the next frame is continued.

  Then, in the above-described sign content determination process S206 of the present frame, the sign content determination unit 174 determines that only one speed limit (including the same case among a plurality of road signs) is accumulated in one or more road sign candidates. Determine if you are earning points. As a result, if no road sign is detected, or if only one road sign's speed limit has not obtained a sign cumulative point, such as when points are obtained for a plurality of speed limits with a plurality of road signs, The process is ended, and the determination of the sign accumulated point is repeated in the next frame.

  Subsequently, when only the speed limit of 1 obtains marker accumulated points, the sign content determination unit 174 sets the recognition result of the speed limit as the speed limit V and stores the image of the recognition target area 370 in the temporary image memory. Do. Here, in the case where the speed limit V has obtained the sign cumulative point for all of the plurality of road signs that have become candidates, recognition of a candidate with the lowest overall evaluation value E (im, V) (the correlation is maximized) The image of the target area 370 is stored. The image stored here is an occupied area 346 normalized to a rectangle of horizontal predetermined pixels × vertical predetermined pixels after completion of the marker detection processing S202.

  Next, based on the image of the occupied area stored in the image memory, the marker content determination unit 174 executes the marker content recognition process S204 on the template of the speed limit V of each country. That is, in the sign content recognition processing S204 described above, as shown by the broken line in the template of FIG. 29, the template of each speed limit T for one country is used, but here, as shown by the solid line in the template of FIG. Then, the template of each country regarding the speed limit V of 1 will be used. Therefore, when the national identifier is CN and the speed limit is V, the comprehensive evaluation value E (CN, V) is derived instead of the comprehensive evaluation value E (im, T). In this process, it is possible to obtain an evaluation value E (CN, V) of the national weight prepared as a template.

  However, in the present embodiment, the weighting is made different for the currently recognized country and the other evaluation values E (CN, V). For example, the tag content determination unit 174 multiplies only the evaluation value E (CN, V) of the currently recognized country by a weighting factor (for example, 0.8 of 1 or less). This is to make the evaluation value E (CN, V) of the currently recognized country relatively low (increase the correlation) so that hunting does not occur in the result of the country determination. In addition, when a country adjacent to the currently recognized country is known, the adjacent country may also be multiplied by a weighting factor (for example, 0.95 greater than or equal to 0.8 and less than or equal to 1).

  The tag content determination unit 174 compares the evaluation values E (CN, V) of all the countries thus derived to derive the minimum value ECM. Then, when the difference between the minimum value ECM and the evaluation values E (CN, V) of all the other templates becomes equal to or more than a predetermined value, +1 is added to the country cumulative point of the minimum value ECM template.

  Subsequently, the sign content determination unit 174 compares the maximum value of the country-by-country cumulative points with all other country-by-country cumulative points, and when the difference is equal to or more than a predetermined value (for example, 30), the maximum value is obtained. Determine whether the country is equal to the currently recognized country. As a result, if the country with the maximum value and the country currently recognized are equal, all the country cumulative points are multiplied by 1⁄2 to reduce the overall marker cumulative points for fair judgment. Also, if the country with the highest value is different from the country currently recognized, it is determined that the country you are driving has changed, and the country currently recognized is updated with the country with the highest value. , Temporary image memory area, and country cumulative points are initialized. In addition, if the difference between the maximum value of accumulated points by country and the accumulated points by all other countries is less than a predetermined value, the temporary image memory area is initialized, and the judgment of the relevant accumulated points in the next frame is performed. repeat.

  As described above, by appropriately determining the country in which the vehicle is currently traveling, it is possible to enhance the accuracy in identifying the contents of the road sign. Moreover, it becomes possible to suppress increase of processing load by performing the above-mentioned country judging processing under the recognition processing of a road sign of one country.

  As described above, in the outside environment recognition apparatus 120 according to the present embodiment, it is possible to improve the recognition accuracy of the content of the road sign while suppressing an increase in the processing load.

  In addition, a program that causes a computer to function as the environment recognition device outside the vehicle 120 and a computer readable flexible disk, a magneto-optical disk, a ROM, a CD, a DVD, a BD, and other storage media storing the program are also provided. Here, the program refers to data processing means described in any language or description method.

  Although the preferred embodiments of the present invention have been described above with reference to the accompanying drawings, it goes without saying that the present invention is not limited to such embodiments. It is obvious that those skilled in the art can conceive of various changes or modifications within the scope of the claims, and it is naturally understood that they are also within the technical scope of the present invention. Be done.

  Note that each process of the outside environment recognition process of the present specification does not necessarily have to be processed in chronological order according to the order described as the flowchart, and may include processes in parallel or by a subroutine.

  The present invention can be used for an external environment recognition device that recognizes the contents of a road sign installed on a road.

120 Vehicle outside environment recognition device 160 Image acquisition unit 162 Position information derivation unit 164 Feature point identification unit 166 Voting unit 168 Sign identification unit 170 Sign correction unit 172 Sign content recognition unit 174 Sign content determination unit

Claims (1)

An image acquisition unit for acquiring an image;
A feature point identification unit that identifies pixels having edge strengths equal to or greater than a predetermined value as feature points using the image;
A sign identification unit that identifies a circular road sign based on the feature points;
A sign content recognition unit that recognizes the contents of the identified road sign;
Equipped with
The feature point identification unit
The feature points are specified sequentially from the upper side of the image, and when the specified number of feature points becomes equal to or more than a predetermined number determined in advance, the specification of the feature points is stopped.
If the identified feature score exceeds the predetermined feature point upper limit, the predetermined value is increased, and if the identified feature score is less than the predetermined feature point lower limit, the predetermined value is decreased. An outside environment recognition device characterized in that it is used for next comparison with edge strength.

Images