JP6339840B2 - Outside environment recognition device - Google Patents

Description

  The present invention relates to a vehicle exterior environment recognition device that recognizes the contents of a road sign confirmed outside a vehicle.

  Conventionally, a specific object such as a vehicle positioned in front of the host vehicle is detected, and a collision with a preceding vehicle is avoided (collision avoidance control), or the distance between the preceding vehicle and the preceding vehicle is controlled to be a safe distance ( (Cruise control) technology is known (for example, Patent Document 1). In addition, there is an increasing demand for a technology for recognizing a speed limit provided for each road and controlling the speed of the host vehicle in order to reduce accidents caused by excessive speed.

  In order to drive the vehicle safely with the speed of the host vehicle within the speed limit, it is necessary to recognize the contents of the road signs placed on the shoulders and gates and to correctly grasp the speed limit of the road currently being driven. For example, Patent Document 2 describes a technique for recognizing a circular road sign image (hereinafter, a road sign image is also simply referred to as a road sign) by performing a Hough transform on a portion corresponding to an edge on the screen. . With this technique, it is possible to improve the efficiency of identifying road signs by reducing the processing load spent on Hough conversion.

Japanese Patent No. 3349060 JP 2012-243051 A

  In order to recognize a circular road sign by performing Hough transform, first, a feature point corresponding to a partial part of the circumference is specified, and then a vote is given on the circumference separated from the feature point by a predetermined distance. A candidate road sign having a center position and a radius of a circle is identified according to the number. The content of the road sign is recognized through pattern matching or the like for the candidate road sign. However, the distribution of road sign luminance values varies depending on the external environment such as solar radiation and lighting, and the imaging state. Not only can the contents be recognized correctly with simple processing, but also the area where the contents are displayed is specified correctly. Sometimes you can't.

  In addition, there are two types of road signs: the light-emitting display type and the non-light-emitting display type, and the distribution of brightness differs between the content portion and its surroundings. Depending on the uniform processing, the content is displayed for either one of the display types. There was a problem that it could not be recognized.

  In view of such a problem, the present invention provides a vehicle environment recognition device capable of appropriately recognizing the contents of a road sign regardless of differences in luminance value distribution and display types such as lightning and non-lightning. The purpose is to provide.

In order to solve the above-described problem, an external environment recognition apparatus according to the present invention includes an image acquisition unit that acquires an image, and a sign specification that specifies a circle with a predetermined radius centered on an arbitrary pixel as a road sign using the image. Content recognition that recognizes the content of the road sign from either the N-valued road sign or the inverted road sign obtained by inverting the road sign. The road sign includes a road sign that indicates the release of the speed limit and a road sign that indicates the speed limit, and the sign content recognition unit presents the road sign that indicates the release of the speed limit and the speed limit If the road sign is a road sign that indicates the release of the speed limit, the road sign that indicates the speed limit is not recognized .

  The sign content recognition unit may not reverse the road sign or may not recognize the content of the inverted road sign when the content of the N-valued road sign can be recognized effectively.

  If the sign content recognition unit determines that effective recognition cannot be performed during the recognition of the N-valued road sign content, the sign content recognition unit may interrupt the recognition of the road sign and recognize the inverted road sign.

  The sign content recognizing unit may recognize a road sign presenting the release of the speed limit based on the accumulated luminance value in a plurality of line segments having an inclination angle that intersect on the road sign.

  According to the present invention, it is possible to appropriately recognize the contents of a road sign regardless of a difference in luminance value distribution or a display type such as lightning / non-lightning.

It is the block diagram which showed the connection relation of the external environment recognition system. It is explanatory drawing for demonstrating a color image and a distance image. It is the functional block diagram which showed the schematic function of the external environment recognition apparatus. It is explanatory drawing explaining a road sign. It is a flowchart which shows the flow of a vehicle exterior environment recognition process. It is explanatory drawing for demonstrating the color image which an image acquisition part acquires. It is explanatory drawing for demonstrating Hough conversion. It is explanatory drawing for demonstrating Hough conversion. It is explanatory drawing for demonstrating 4th extraction conditions. It is the flowchart which showed an example of the feature point specific process. It is explanatory drawing which showed an example of the feature point specific process. It is explanatory drawing for demonstrating a voting process. It is explanatory drawing for demonstrating a voting table. It is explanatory drawing for demonstrating a center point candidate list. It is explanatory drawing for demonstrating a flag table. It is the flowchart which showed an example of the label | marker specific process. It is explanatory drawing for demonstrating the process of a label | marker correction | amendment part. It is the flowchart which showed the flow of the specific process of a label | marker content recognition process. It is explanatory drawing for demonstrating a recognition object area | region. It is explanatory drawing for demonstrating the road sign which showed cancellation | release of the speed limit. It is explanatory drawing for demonstrating a vertical direction alignment process. It is explanatory drawing for demonstrating a template. It is explanatory drawing for demonstrating the matching process of a horizontal direction. It is explanatory drawing for demonstrating DP matching. It is explanatory drawing for demonstrating the matching process of an attention site | part. It is explanatory drawing for demonstrating an evaluation determination result. It is the time chart which showed the flow of the result notification of a road sign. It is explanatory drawing which showed the display mode of the road sign according to country. It is explanatory drawing for demonstrating the template of a road sign.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The dimensions, materials, and other specific numerical values shown in the embodiment are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. In the present specification and drawings, elements having substantially the same function and configuration are denoted by the same reference numerals, and redundant description is omitted, and elements not directly related to the present invention are not illustrated. To do.

(External vehicle environment recognition system 100)
FIG. 1 is a block diagram showing a connection relationship of the external environment recognition system 100. The vehicle exterior environment recognition system 100 includes an imaging device 110, a vehicle exterior environment recognition device 120, and a vehicle control device (ECU: Engine Control Unit) 130 provided in the host vehicle 1.

  The imaging device 110 includes an imaging element such as a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS), images an environment corresponding to the front of the host vehicle 1, and is represented by a color value. A color image can be generated. Here, the color value is from a YUV format color space consisting of one luminance (Y) and two color differences (U, V), and three hues (R (red), G (green), B (blue)). A numerical value group represented by any one of the RGB color space or the HSB color space consisting of hue (H), saturation (S), and brightness (B). In the present embodiment, a color image using color values in the YUV format will be described as an example as an image. However, an application that can perform processing without depending on a color image in an arbitrary partial process is only a color image. Instead, a luminance image (monochrome image) represented by luminance (Y) can be used.

  In addition, the imaging devices 110 are arranged in a substantially horizontal direction so that the optical axes of the two imaging devices 110 are substantially parallel on the traveling direction side of the host vehicle 1. The imaging device 110 continuously generates a color image obtained by capturing an object existing in the detection area in front of the host vehicle 1, for example, every 1/60 second frame (60 fps). Here, the objects to be recognized are not only solid objects such as vehicles, pedestrians, traffic lights, roads (traveling paths), road signs, gates, guardrails, buildings, but also the contents of road signs, brake lamps, Also included are those that can be specified as part of a three-dimensional object, such as high-mount stop lamps, tail lamps, blinkers, and lighting parts of traffic lights. Each functional unit in the following embodiment performs each process for each frame in response to the update of the color image.

  Further, in the present embodiment, the imaging device 110 captures the detection region in the first exposure mode indicating the exposure time and the aperture according to the brightness of the environment outside the vehicle (measurement result of the illuminometer, etc.), and generates the first image. To do. Further, the imaging device 110 generates an image that can determine whether or not a specific light source emits light such as an electric display type road sign. As the method, an imaging device having a wide dynamic range may be used to capture an image so that an object that does not emit light is not crushed black and the light emission source is not overexposed. The first exposure mode is an exposure mode (exposure). The detection area may be imaged in a second exposure mode with different time and aperture to generate a second image. For example, during the daytime, the second image is generated by shortening the exposure time in the second exposure mode or increasing the aperture in comparison with the exposure time in the first exposure mode according to the bright outside environment. In the present embodiment, the first image and the second image are used as a color image and a distance image, respectively. Moreover, the said 1st exposure aspect and a 2nd exposure aspect are implement | achieved as follows.

  For example, the first image and the second image are sequentially generated by time-sharing the periodic image capturing timing of the image capturing apparatus 110 and alternately performing image capturing in the first exposure mode and image capturing in the second exposure mode. Can do. In addition, in an image pickup device in which two capacitors are provided for each pixel and charges can be charged in parallel to the two capacitors, two images having different exposure modes are set in parallel by changing the charging time in one exposure. It can also be generated. Further, the above-described object can be achieved by reading twice at different times and generating two images having different exposure modes in parallel while charging one capacitor. In addition, two sets of imaging devices 110 may be prepared in advance with different exposure modes (here, two imaging devices 110 × 2 sets), and images may be generated from the two sets of imaging devices 110, respectively. Is possible.

  The outside environment recognition device 120 acquires a color image from each of the two imaging devices 110, and selects a block corresponding to a block arbitrarily extracted from one color image (for example, an array of horizontal 4 pixels × vertical 4 pixels) as the other color. The parallax information including the parallax and the screen position indicating the position of the arbitrary block in the screen is derived using so-called pattern matching that is searched from the image. Here, the horizontal indicates the horizontal direction of the captured image, and the vertical indicates the vertical direction of the captured image. As this pattern matching, it is conceivable to compare the luminance (Y) in an arbitrary block unit between a pair of images. For example, SAD (Sum of Absolute Difference) that takes a luminance difference, SSD (Sum of Squared Intensity Difference) that uses the difference squared, or NCC that takes the similarity of a variance value obtained by subtracting an average value from the luminance of each pixel There are methods such as (Normalized Cross Correlation). The vehicle exterior environment recognition apparatus 120 performs such block-based parallax derivation processing for all blocks displayed in the detection area (for example, 600 pixels × 200 pixels). Here, the block is 4 pixels × 4 pixels, but the number of pixels in the block can be arbitrarily set.

  However, the vehicle environment recognition apparatus 120 can derive the parallax for each block, which is a detection resolution unit, but cannot recognize what kind of target object the block is. Accordingly, the disparity information is derived independently not in units of objects but in units of detection resolution (for example, blocks) in the detection region. Here, an image in which the parallax information derived in this way is associated is referred to as a distance image in distinction from the color image described above.

  FIG. 2 is an explanatory diagram for explaining the color image 126 and the distance image 128. For example, assume that a color image 126 as shown in FIG. 2A is generated for the detection region 124 through the two imaging devices 110. However, only one of the two color images 126 is schematically shown here for easy understanding. The vehicle environment recognition apparatus 120 obtains the parallax for each block from the color image 126 and forms a distance image 128 as shown in FIG. Each block in the distance image 128 is associated with the parallax of the block. Here, for convenience of description, blocks from which parallax is derived are represented by black dots. In the present embodiment, such a color image 126 and distance image 128 are generated based on the first image and the second image, respectively.

  Further, the outside environment recognition device 120 uses the color value based on the color image 126 and the three-dimensional position information in the real space including the relative distance from the host vehicle 1 calculated based on the distance image 128, and uses the color information. Blocks having the same value and having similar three-dimensional position information are grouped as objects, and the specific object (for example, the preceding vehicle) corresponding to the object in the detection area in front of the host vehicle 1 is specified. For example, it is possible to grasp the acceleration / deceleration of the preceding vehicle by specifying the preceding vehicle that travels ahead based on the relative distance or the like, and by accurately recognizing whether or not the brake lamp of the preceding vehicle is turned on based on the color value. Further, the outside environment recognition device 120 identifies a road sign placed on the shoulder or gate, recognizes the content of the road sign, for example, a speed limit, and limits the speed of the host vehicle 1 through the vehicle control device 130. Control to a safe speed within the speed.

  The relative distance is obtained by converting the disparity information for each block in the distance image 128 into three-dimensional position information using a so-called stereo method. Here, the stereo method is a method of deriving a relative distance of the target object from the imaging device 110 from the parallax of the target object by using a triangulation method.

  Returning to FIG. 1, the vehicle control device 130 receives a driver's operation input through the steering wheel 132, the accelerator pedal 134, and the brake pedal 136, and transmits it to the steering mechanism 142, the drive mechanism 144, and the brake mechanism 146. The own vehicle 1 is controlled. In addition, the vehicle control device 130 controls the drive mechanism 144 and the braking mechanism 146 in accordance with instructions from the outside environment recognition device 120.

  Hereinafter, the configuration of the outside environment recognition device 120 will be described in detail. Here, the road sign specifying process characteristic of the present embodiment will be described in detail, and the description of the configuration unrelated to the characteristics of the present embodiment will be omitted.

(Vehicle environment recognition device 120)
FIG. 3 is a functional block diagram showing a schematic function of the outside environment recognition device 120. As shown in FIG. 3, the vehicle exterior environment recognition apparatus 120 includes an I / F unit 150, a data holding unit 152, and a central control unit 154.

  The I / F unit 150 is an interface for performing bidirectional information exchange with the imaging device 110 and the vehicle control device 130. The data holding unit 152 includes a RAM, a flash memory, an HDD, and the like. The data holding unit 152 holds various pieces of information necessary for processing of each function unit described below, and also receives images (first image and first image) received from the imaging device 110. A color image 126 based on two images and a distance image 128) are temporarily stored.

  The central control unit 154 is configured by a semiconductor integrated circuit including a central processing unit (CPU), a ROM storing a program, a RAM as a work area, and the like, and through the system bus 156, an I / F unit 150, a data holding unit 152 and the like are controlled. In this embodiment, the central control unit 154 includes an image acquisition unit 160, a position information deriving unit 162, a feature point specifying unit 164, a voting unit 166, a sign specifying unit 168, a sign correcting unit 170, a sign content recognizing unit 172, It also functions as the sign content determination unit 174. Hereinafter, the road signs that are the object of recognition in the present embodiment will be described, and then the outside environment recognition processing characteristic of the present embodiment will be described in detail based on the operation of each functional unit of the central control unit 154.

  FIG. 4 is an explanatory diagram for explaining a road sign. As the types of road signs, there are road signs that present the speed limit and road signs that indicate the release of the speed limit. As shown in FIG. 4A, the road sign indicating the speed limit has a numerical value indicating the speed limit in a circular frame. In addition, the road sign indicating the release of the speed limit is shown in FIG. 4 (b), with a plain background with diagonal lines from the upper right to the lower left, and as shown in FIG. 4 (c). Some of them have the diagonal lines on the background of numerical values indicating speed.

  Moreover, as a display mode of a road sign, there are an electric light display type having a light emission source such as an LED and a non-electric light display type only having a light source and being color-coded. In addition, there are road signs arranged on the shoulders of roads and those arranged at positions corresponding to above the roads of gates erected in a arch shape across both road shoulders. .

  The vehicle environment recognition apparatus 120 according to the present embodiment recognizes the contents of each of the road signs having different arrangements, display modes, and types by the function units of the central control unit 154 described above. When the environment recognition device 120 outside the vehicle recognizes the content of the road sign, it notifies the driver of the content, for example, the speed limit of the road that is currently running, or when the current speed exceeds the speed limit. This can be notified to the driver, or the vehicle control device 130 can be controlled so as not to exceed the speed limit. For this reason, it is not always necessary to recognize the road sign at the moment when the host vehicle 1 reaches a position where the road sign can be confirmed, and it is sufficient if the vehicle 1 can recognize it at the time of passing the road sign or thereafter. Therefore, it is only necessary to recognize the road sign over a plurality of frames and determine the content of the road sign from the information of the plurality of frames.

(External vehicle environment recognition processing)
FIG. 5 is a flowchart showing the flow of the external environment recognition process. The environment recognition process outside the vehicle is roughly divided into an image acquisition process (S200) for acquiring an image, a sign detection process (S202) for detecting a road sign, particularly a circular frame, and the contents of the road sign (numeric value and pattern pattern). A sign content recognition process (S204) for recognizing and a sign content confirmation process (S206) for accumulating and confirming the recognized road sign contents in the time direction are executed in that order.

(Image acquisition process S200)
The image acquisition unit 160 acquires the color image 126 from the imaging device 110. As described above, the road signs targeted in the present embodiment have different display modes such as an electric display type and a non-electric display type, and also have different arrangements such as a road shoulder and a gate. . Therefore, the imaging apparatus 110 captures two detection areas in which the road shoulder and the gate can be detected in the two exposure modes of the first exposure mode and the second exposure mode, respectively, and the image acquisition unit 160 captures images in this way. The obtained four color images 126 are acquired.

  FIG. 6 is an explanatory diagram for explaining the color image 126 acquired by the image acquisition unit 160. For example, the imaging device 110 captures the color image 126 shown in FIGS. 6A and 6B in the first exposure mode with a relatively long exposure time. However, FIG. 6A shows a color image 126 picked up with an angle of view capable of detecting the shoulder, and FIG. 6B shows a direction in which the angle of view is wider than that of FIG. 6A in order to detect the gate. A color image 126 picked up by switching to is shown. Then, the imaging device 110 switches the exposure mode to the second exposure mode, for example, sets the exposure time to be relatively short, and captures the color image 126 shown in FIGS. 6C and 6D. However, FIG. 6C and FIG. 6D are imaged by switching the angle of view according to the detection target, as in FIG. 6A and FIG. 6B. Here, four color images 126 are listed, but the number and type can be arbitrarily set as long as the shoulder and gate can be detected.

  In this way, four color images 126 with different exposure modes and detection areas are acquired. By capturing images in multiple exposure modes and detection areas in this way, problems such as saturation of the luminance of the light source and reduction of resolution due to a wide angle of view can be solved, and detection accuracy can be sufficiently increased. Can do. The four color images 126 are picked up in time division, and the order can be arbitrarily set. However, in the present embodiment, it is only necessary to recognize the road sign at the time of passing the road sign or after that, so it is not always necessary to capture the four color images 126 at the same timing.

  The position information deriving unit 162 acquires color images (FIGS. 6A and 6B) based on the first image captured by the first exposure from each of the two imaging devices 110, and uses the pattern matching to perform parallax The parallax information including the screen position indicating the position of the arbitrary block in the screen is derived, and the distance image 128 is generated. Then, the position information deriving unit 162 uses the stereo method to calculate the disparity information for each block in the detection area 124 in the distance image 128 as a horizontal relative distance centered on the horizontal center of the host vehicle 1. The information is converted into three-dimensional position information including the distance x, the height y from the road surface, and the relative distance z in the depth direction with the host vehicle 1. However, the position information deriving unit 162 obtains the vertical position of the road surface in advance before converting into the three-dimensional position information, and calculates the height from the road surface from the relative distance between the vertical position of each block and the vertical position of the road surface. y is derived. Here, the parallax information indicates the parallax of each block in the distance image 128, while the three-dimensional position information indicates information on the relative distance z of each block in the real space. Further, when the disparity information is derived not in pixel units but in block units, that is, in a plurality of pixel units, the disparity information may be regarded as disparity information of all pixels belonging to the block, and calculation in pixel units may be executed. it can. Regarding the conversion to the three-dimensional position information, since existing techniques such as JP2013-109391A can be referred to, detailed description thereof is omitted here.

(Label detection process S202)
In the present embodiment, the object is to recognize circular road signs, among road signs. Such circular road signs are detected using Hough transform. Here, the Hough transform is a technique for voting from a feature point on a color image to a point where an object may exist and detecting an object having a large number of votes (a predetermined value or more). As described above, in the present embodiment, the description will be given specifically for the Hough conversion, but the application that can identify the road sign without depending on the Hough conversion in any partial processing of the outside environment recognition process is not only the Hough conversion Various existing shape recognition methods such as template matching and least square method other than the Hough transform can be used.

  7 and 8 are explanatory diagrams for explaining the Hough transform. Here, it is assumed that three pixels 220c, 220d, and 220e having edges are extracted from the color image 126 as shown in FIG. These three pixels 220 c, 220 d, and 220 e are originally a part of the circular road sign 222, but it is usually impossible to grasp from the color image 126 that the shape is clearly circular.

  The Hough transform is a technique for detecting a geometric shape such as a circle or a straight line from a plurality of points. The center of a circle that passes through an arbitrary pixel 220 and has a radius n is centered on the arbitrary pixel 220. It is based on the theory of existing on the circumference of the radius n. For example, the center of the circle passing through the three pixels 220c, 220d, and 220e in FIG. 7A is on the circumference centered at each of the three pixels 220c, 220d, and 220e. However, since the radius n cannot be specified by only the edge information, a plurality of different radii n are prepared, and pixels on a circle having a plurality of radii n around the three pixels 220c, 220d, and 220e are prepared. And the radius n and the center are determined as the road sign 222 when the number of votes is equal to or greater than a predetermined value.

  For example, as shown in FIGS. 7B, 7C, and 7D, circles having different radii n = 4, 5, and 6 are formed around the three pixels 220c, 220d, and 220e. Then, vote for the pixels included in the locus of the circle (associate the unit index). Then, in FIG. 7B, the number of votes obtained is 2 for two pixels 224 (two unit indices are associated). In FIG. 7C, the number of votes obtained is 3 for three pixels 224, and the number of votes obtained is 3 for one pixel 226. Similarly, in FIG. 7D, the number of votes is 2 for the six pixels 224.

  At this time, the number of votes obtained is 3 (predetermined value or more) only in the pixel 226. The pixel 226 is the center of a circle passing through the three pixels 220c, 220d, and 220e, and the radius n when the pixel 226 is derived is n. = 5 can be specified as the radius of the circle. Thus, as shown in FIG. 7E, a circle 228 passing through the three pixels 220c, 220d, and 220e is specified. Here, for convenience of explanation, the three pixels 220c, 220d, and 220e have been described. However, a pixel that is not included in the circle 228 becomes a feature point, or at a position different from the original position by pixelization (discretization). Since the appearing pixel may become a feature point, in order to avoid the influence of such noise, in practice, a large number of points are used for voting and stable detection is performed based on the principle of number. In the present embodiment, such a Hough transform is applied to, for example, the color image 126 shown in FIG. 6B, and a circular road sign is formed by the feature point specifying unit 164, the voting unit 166, and the sign specifying unit 168. Identify. Hereinafter, basic processing of each functional unit will be described with reference to FIG.

  First, the feature point specifying unit 164 specifies feature points corresponding to a partial portion of the circumference from the color image 126 (feature point specifying process). For example, it is assumed that the feature point specifying unit 164 specifies the pixels 220f, 220g, 220h, 220i, 220j, and 220k having edges as feature points in the color image 126 in FIG. In addition, the dotted line of Fig.8 (a) respond | corresponds to a travel lane.

  Next, the voting unit 166 votes for a predetermined distance corresponding to the radius n from the feature point (voting process). Here, for convenience of explanation, the radius n is assumed to be 30 pixels for the pixels 220f, 220g, and 220h, and the radius n is assumed to be 23 pixels for the pixels 220i, 220j, and 220k. In the color image 126 of FIG. 8B, the voting unit 166 is on a circle having a radius n (30 pixels, 23 pixels) from each of the pixels 220 with the pixels 220f, 220g, 220h, 220i, 220j, and 220k as the center. Vote for all pixels. Then, 1 is voted (added) to the vote destination (pixel, radius n) in the vote table 230 shown in FIG. Here, the voting table 230 is a voting space based on the Hough transform, and is represented in three dimensions, that is, the screen position (x, y) of the pixel that is the voting destination and the radius n.

  Subsequently, the label specifying unit 168 detects the number of votes obtained from the voting table 230, and derives the center and radius n of the circle based on the voted pixels and the radius n having a large number of votes. Then, as shown in FIG. 8D, a circle 236 having a radius n read out around the pixel 234 having a large number of votes is formed and specified as a road sign (sign specifying process). Hereinafter, specific operations of the feature point specifying process, the voting process, and the sign specifying process by the feature point specifying unit 164, the voting unit 166, and the sign specifying unit 168 will be described.

(Feature point identification processing)
The feature point specifying unit 164 uses the color image 126 and, as a first extraction condition, the pixel 220 having a predetermined edge strength among the pixels 220 as a feature point candidate. The edge strength may be represented by, for example, a Sobel filter. When the coordinates of each pixel 220 are (i, j) and the luminance is A (i, j), the feature point specifying unit 164 uses the following Equation 1 to calculate the vertical Sobel filter and the horizontal direction. The sum of absolute values of the Sobel filter is derived, and if the value (edge strength) is equal to or greater than a predetermined value, the pixel (i, j) is determined as a candidate feature point.
Edge strength = | A (i + 1, j + 1) + 2A (i + 1, j) + A (i + 1, j-1)-A (i-1, j + 1) -2A (i- 1, j) -A (i-1, j-1) |
+ | A (i + 1, j + 1) + 2A (i, j + 1) + A (i-1, j + 1)-A (i + 1, j-1) -2A (i, j- 1) -A (i-1, j-1) | ... (Formula 1)

  Here, an example in which the edge strength is derived using a Sobel filter has been described. However, the present invention is not limited to this, and various existing techniques such as a Prewitt filter can be applied.

  In addition, the feature point specifying unit 164 uses the color image 126 and, as a second extraction condition, a predetermined color component of a predetermined color value of the pixel 220, for example, a V component in a YUV color space. If it is equal to or greater than a predetermined value, the pixel 220 is set as a feature point candidate. The road sign indicating the speed limit is configured with a red circle around the road sign, and the road sign indicating the release of the speed limit is configured with a white or black circle around the road speed. Therefore, only the colors belonging to the region where the V component is equal to or greater than the predetermined value are extracted as feature point candidates. In this way, green pixels such as trees that are confirmed at many occasions during traveling can be excluded, and feature points can be appropriately narrowed down.

  If the color image is composed of an RGB color space, the color image is converted to a YUV color space by an arbitrary conversion formula. Since such a conversion formula is an existing technique, a detailed description thereof is omitted here.

  In addition, the feature point specifying unit 164 uses the distance image 128, and, as a third extraction condition, the relative distance z is within a predetermined range, the height y from the road surface is within a predetermined range, and the horizontal distance among the pixels 220. A pixel 220 that satisfies any one or a plurality of conditions in which x is within a predetermined range is set as a feature point candidate.

  Specifically, the feature point specifying unit 164 extracts an arbitrary pixel 220 from the distance image 128, refers to the three-dimensional position information of the pixel 220, and the relative distance z of the object corresponding to the pixel 220 is, for example, If the pixel 220 is located at 10 m or more and less than 50 m, the pixel 220 is set as a feature point candidate. This is because if the distance is less than 10 m, the amount of movement of the object on the image within the exposure time becomes longer, and the influence of image blur increases accordingly. This is because the contents of are often not recognized correctly. By limiting the relative distance z in this way, it is possible to reduce the processing load and reduce erroneous recognition.

  Further, the feature point specifying unit 164 sets the pixel 220 as a feature point candidate when the height y of the object corresponding to the pixel 220 from the road surface is, for example, 0.5 m or more and less than 6.0 m. To do. By setting this range to 0.5 m or more, road markings and lanes can be excluded as recognition targets, and by setting the range to less than 6.0 m, trees and the like located above that can be excluded. . Even under such conditions, it is possible to reduce the processing load and reduce misrecognition.

  Further, the feature point specifying unit 164 sets the pixel 220 as a candidate for the feature point when the horizontal distance x of the object corresponding to the pixel 220 is located within a range of, for example, 12 m (−12 m or more and less than 12 m). . By setting this range to 12 m, it is possible to exclude road signs other than road signs related to the lane in which the host vehicle 1 is traveling. Even under such conditions, it is possible to reduce the processing load and reduce misrecognition.

  The feature point specifying unit 164 uses the color image 126 and the distance image 128, and the difference between at least one color component (for example, U component) between adjacent pixels 220 is within a predetermined range as a fourth extraction condition. In addition, a pixel in which such pixels 220 do not continue for a predetermined distance (length) in a predetermined direction is set as a candidate feature point.

  FIG. 9 is an explanatory diagram for explaining the fourth extraction condition. For example, the autumnal tree 240 shown on the shoulder of FIG. 9A may satisfy all the first to third extraction conditions. Further, since the autumnal tree 240 has many places that can be recognized as a texture, it is often extracted as a feature point in a wide range. If it does so, a feature point will be specified in a wide range in trees 240 etc. with many textures, and processing load will increase.

  Therefore, the feature point specifying unit 164 has a distance between the pixels 220, that is, one or a plurality of combined distances of a distance in the depth direction, a distance in the vertical direction, and a distance in the horizontal direction, for example, a predetermined distance (for example, It is determined whether or not the difference of one color component (for example, U component) is within a predetermined value (for example, 10). Then, the feature point specifying unit 164 has 30 pixels that are continuous in one direction (for example, the horizontal direction) of the pixels 220 having a combined distance of less than 0.5 m and a U component difference of 10 or less. In this case, all the pixels 220 are excluded from the feature points. As shown in FIG. 9B, the road sign is composed of circles of the same color as shown in FIG. 9B, but does not continue 30 pixels in a predetermined direction, for example, the horizontal direction or the vertical direction. By specifying the feature points based on the characteristics of the display format of the road sign, the pixels 220 that satisfy the first to third extraction conditions that should not be extracted as the feature points can be excluded. Thus, it is possible to improve the identification efficiency of the feature points.

  However, the size of the road sign on the color image 126 varies depending on the relative distance z. Therefore, the feature point specifying unit 164 may change a predetermined distance, which is a threshold value as to whether or not the pixels 220 are continuous in a predetermined direction, according to the relative distance z to the host vehicle 1. Specifically, the shorter the relative distance z, the longer the predetermined distance, and the longer the relative distance z, the shorter the predetermined distance. In this way, an appropriate threshold value can be provided according to the size of the road sign on the color image 126, and the pixel 220 that satisfies the first to third extraction conditions that should not be extracted as a feature point originally. Can be appropriately excluded.

  Then, the feature point specifying unit 164 specifies, as a feature point, the pixel 220 that satisfies the first or second extraction condition among the pixels 220 that satisfy both the third and fourth extraction conditions. In this way, a pixel 220 suitable as a feature point is specified.

  The feature point specifying unit 164 may stop the feature point specifying process in one frame when the number of feature points is equal to or greater than a predetermined value. The color image 126 changes variously depending on the environment outside the vehicle, and the feature points may become very large depending on the captured environment. If the number of feature points increases in this way, the processing load increases accordingly, and the processing time allocated in one frame may be exceeded. Therefore, when the number of feature points exceeds a predetermined value, the feature point specifying unit 164 stops the feature point specifying process in one frame and performs voting processing only for the feature points specified up to that point. Perform the following processing.

  However, since the road sign is often positioned relatively above the color image 126, the feature point specifying unit 164 specifies the feature points in order from the top of the color image 126. In this way, it is possible to appropriately extract the feature points corresponding to the circumferential part of the road sign.

  In addition, the feature point specifying unit 164 frames the predetermined value of the edge strength in the first extraction condition and the predetermined value of the V component in the second extraction condition in order to keep the number of feature points within the predetermined value as described above. It may be changed every time. Since the environment outside the vehicle is poor in frame units, the number of feature points does not change much. Therefore, when many feature points are extracted in one frame, many feature points are continuously extracted in subsequent frames. Therefore, the predetermined value of the edge intensity in the first extraction condition and the predetermined value of the V component in the second extraction condition are within a predetermined range so as not to exceed the processing time allocated for specifying the feature point in one frame. While adjusting at (40 to 150), the number of feature points is kept within a predetermined range (200 to 2000).

  FIG. 10 is a flowchart showing an example of the feature point specifying process. As shown in FIG. 10A, the feature point specifying unit 164 determines whether or not the number of feature points extracted under the first extraction condition in one frame exceeds the feature point upper limit value (2000 here). Determine (S240). As a result, if the number of feature points exceeds the feature point upper limit value (YES in S240), the feature point specifying unit 164 determines whether the predetermined value of the edge strength is less than the edge strength upper limit value (150 in this case). Determination is made (S242). As a result, if the predetermined value of edge strength is less than the upper limit value of edge strength (YES in S242), the feature point specifying unit 164 increments the predetermined value of edge strength (S244). The predetermined value of the edge strength is reflected from the specific object specifying process of the next frame. If the predetermined value of the edge strength is equal to or higher than the upper limit value of the edge strength (NO in S242), the feature point specifying unit 164 does not increment the predetermined value of the edge strength and sets the predetermined value that has reached the upper limit value of the edge strength. Maintain (maintain within a predetermined range).

  If the number of feature points is equal to or less than the feature point upper limit value in step S240 (NO in S240), the feature point specifying unit 164 determines that the number of feature points extracted in one frame is the feature point lower limit value (here, 200). It is determined whether it is less than (S246). As a result, if the number of feature points is less than the feature point lower limit value (YES in S246), the feature point specifying unit 164 determines whether the predetermined value of the edge strength exceeds the edge strength lower limit value (here, 40). Determination is made (S248). As a result, if the predetermined value of edge strength exceeds the edge strength lower limit value (YES in S248), the feature point specifying unit 164 decrements the predetermined value of edge strength (S250). If the predetermined value of the edge strength is equal to or lower than the edge strength lower limit value (NO in S248), the feature point specifying unit 164 does not decrement the predetermined value of the edge strength and sets the predetermined value that has reached the lower limit value of the feature point. Maintain (maintain within a predetermined range). If the number of feature points is equal to or greater than the feature point lower limit value (NO in S246), no processing is performed.

  Further, as shown in FIG. 10B, the feature point specifying unit 164 determines whether the number of feature points extracted under the second extraction condition in one frame exceeds the feature point upper limit value (2000 in this case). It is determined whether or not (S260). As a result, if the number of feature points exceeds the feature point upper limit value (YES in S260), the feature point specifying unit 164 determines whether the predetermined value of the V component is less than the V component upper limit value (150 here). Determination is made (S262). As a result, if the predetermined value of the V component is less than the V component upper limit value (YES in S262), the feature point specifying unit 164 increments the predetermined value of the V component (S264). The predetermined value of the V component is reflected from the specific object specifying process in the next frame. If the predetermined value of the V component is equal to or greater than the V component upper limit value (NO in S262), the feature point specifying unit 164 does not increment the predetermined value of the V component, and determines the predetermined value that has reached the upper limit value of the feature point. Maintain (maintain within a predetermined range).

  If the number of feature points is equal to or smaller than the feature point upper limit value in step S260 (NO in S260), the feature point specifying unit 164 determines that the number of feature points extracted in one frame is the feature point lower limit value (here, 200). It is determined whether it is less than (S266). As a result, if the number of feature points is less than the feature point lower limit value (YES in S266), the feature point specifying unit 164 determines whether or not the predetermined value of the V component exceeds the V component lower limit value (here, 40). Determination is made (S268). As a result, if the predetermined value of the V component exceeds the V component lower limit value (YES in S268), the feature point specifying unit 164 decrements the predetermined value of the V component (S270). If the predetermined value of the V component is equal to or less than the V component lower limit value (NO in S268), the feature point specifying unit 164 does not decrement the predetermined value of the V component, and determines the predetermined value that has reached the feature point lower limit value. Maintain (maintain within a predetermined range). If the number of feature points is equal to or greater than the feature point lower limit value (NO in S266), no processing is performed.

  As described above, in the present embodiment, the pixel 220 that satisfies either the first extraction condition or the second extraction condition is specified as a feature point. Therefore, the number of feature points is adjusted independently between the first extraction condition and the second extraction condition as shown in FIGS. 10 (a) and 10 (b). In this way, the number of feature points is within the predetermined range (200) while the predetermined value of the edge strength under the first extraction condition and the predetermined value of the V component under the second extraction condition are kept within the predetermined range (40 to 150). ˜2000), it is possible to maintain the processing time assigned to identify the feature point in one frame.

  The feature point specifying unit 164 may change the predetermined value of the V component in the second extraction condition for each frame according to the color component of the road surface. The color components of the entire color image 126 change depending on the solar radiation conditions and the lighting environment. For example, in a tunnel in which orange illumination is installed, the V component of the entire color image 126 increases. Therefore, the feature point identification unit 164 suppresses the influence of changes in solar radiation and lighting on the identification of feature points by changing the predetermined value of the V component in the second extraction condition according to the color component of the road surface. .

  FIG. 11 is an explanatory diagram showing an example of the feature point specifying process. As shown in FIG. 11A, the feature point specifying unit 164 performs RGB format color at four points 280 at a predetermined relative distance z where the road surface ahead of the vehicle in the color image 126 is highly likely to be reflected. Get the value. Then, the feature point specifying unit 164 obtains the G component / R component of each of the four points 280 and derives an average value AV of the values. Subsequently, the feature point specifying unit 164 multiplies the R component of all the pixels of the color image 126 by the average value AV, converts it to the YUV format, compares the converted V component with a predetermined value, and compares the feature points. Identify candidates.

  However, as shown in FIG. 11B, when the relative distance z of the preceding vehicle recognized by the outside environment recognition device 120 is located within a predetermined range (for example, within 20 m), the average value AV is calculated from the color image 126. The average value AV derived (used) in the previous frame is used. This is for avoiding that the color component of the preceding vehicle is acquired as the color component of the road surface, affects the V component, and erroneous feature point candidates are extracted.

  In addition, the average value AV calculated in the current frame and the average value AV derived (used) in the previous frame are not changed in spite of the fact that the exposure mode has not changed (the ambient brightness is not significantly changed). Even when they differ by a predetermined value (for example, ± 50%) or more, the average value AV derived from the previous frame is used instead of obtaining the average value AV from the color image 126. This is to avoid the extraction of erroneous feature point candidates because the V component is affected when the road surface is colored with a red paint. However, when there is a shadow on the road surface, gray does not affect the G / R component values evenly because it affects the RGB color components (RGB) equally.

(Voting process)
The voting unit 166 votes on a circle that is separated from the feature point specified by the feature point specifying unit 164 by a radius n. This is based on the assumption that the center of a circle with the feature point as a partial part of the circumference should be on the circumference of the radius n from the feature point, assuming that the feature point corresponds to a partial part of the circumference. Based. Therefore, the voting unit 166 votes for a point having a radius n, which is a corresponding point that can be the center of a circle having a feature point as a partial portion of the circumference.

  FIG. 12 is an explanatory diagram for explaining the voting process. Here, supposing that the radius n of 1 is given and explained, as shown in FIG. 12A, the voting unit 166 normally has all the pixels 220 on the circumference 302 of the radius n centering on the feature point 300. Is a corresponding point 304, and a vote is performed for a point having a radius n of the screen position. However, since the corresponding points 304 related to the radius n are all the pixels 220 on the circumference 302, the number of the corresponding points 304 becomes a great number. Further, when the radius n is changed, the number becomes infinite. Therefore, in order to increase the efficiency of voting, the number of corresponding points 304 and radius n for one radius n is limited.

  Here, it is known that the tangent line of a circle is perpendicular to the line segment connecting the contact point and the center of the circle. The tangent line of the circle corresponds to the edge extension direction in the pixel 220. Therefore, the corresponding point 304 appears only on the line segment perpendicular to the edge extension direction of the feature point 300. Therefore, the voting unit 166 can grasp the edge extension direction of the feature point 300 and determine the corresponding point 304 in a direction perpendicular to the edge extension direction.

Here, the voting unit 166 has a vertical Sobel filter as shown in Equation 2 below, where A (i, j) is the luminance when the coordinates of each pixel 220 are (i, j). A line segment perpendicular to the edge extension direction is derived from the ratio of the absolute values of the horizontal Sobel filters.
Line segment perpendicular to edge stretching direction = atan (
| A (i + 1, j + 1) + 2A (i, j + 1) + A (i-1, j + 1)-A (i + 1, j-1) -2A (i, j-1 ) -A (i-1, j-1) |
/ | A (i + 1, j + 1) + 2A (i + 1, j) + A (i + 1, j-1)-A (i-1, j + 1) -2A (i-1, j) -A (i-1, j-1) |
) ... (Formula 2)

  Here, an example of deriving a line segment perpendicular to the edge extending direction by a Sobel filter is given, but the present invention is not limited to this, and various existing techniques can be applied. Further, in Formula 2, division and arc tangent (atan) are used. However, when the processing load increases due to these, the edge extension direction is input using the absolute values of the vertical and horizontal sobel filters as inputs. A look-up table for uniquely deriving a line segment perpendicular to may be used.

  For example, as shown in FIG. 12B, when the edge extension direction of the feature point 300 is indicated by a broken line segment 306, the corresponding point is located on the alternate long and short dash line segment 308 corresponding to the direction perpendicular to the segment 306. 304 can be defined. Here, if the radius n of 1 is given, the corresponding point 304 may be narrowed down to two points separated from the feature point 300 by a radius n on the line segment 308 in the direction perpendicular to the edge extension direction of the feature point 300. it can.

  Further, the size of the road sign may be set to one or more according to the laws and regulations of each country. Then, the size of the road sign on the color image 126 is determined by the relative distance z. Therefore, the voting unit 166 estimates the size (radius n) of the road sign on the color image 126 using the inverse function of the function derived from the three-dimensional position information according to the relative distance z, and uses it for voting. Reduce the number of radii n. For example, when the size of the road sign indicating the speed limit or the road sign indicating the release of the speed limit is limited to three, as shown in FIG. 12 (c), the corresponding number corresponds to the number (3) × 2. 304 is narrowed down.

  In this way, the corresponding point 304 for one radius n is limited to a line segment 308 in the direction perpendicular to the edge extension direction of the feature point 300, and the number of radii n depends on the predetermined size and the relative distance z. By limiting the number to one or more, it is possible to avoid unwilling voting that the corresponding point 304 should not originally exist. Therefore, it is possible to prevent erroneous detection of a road sign due to an erroneous setting of the corresponding point 304, and to avoid useless Hough conversion processing and reduce processing load.

  The voting unit 166 votes the voting table 230 described above after limiting the corresponding points 304 in this way. Here, a three-dimensional voting space will be described as an example. However, as a response to a case where the road sign is inclined in the horizontal direction or the vertical direction, an M dimension (M is a positive value) that expands a dimension (for example, rotation) in the horizontal direction or the vertical direction Voting space) can be formed.

  FIG. 13 is an explanatory diagram for explaining the voting table 230. The voting table 230 is usually composed of a three-dimensional voting space such as the number of horizontal pixels H × the number of vertical pixels V × the radius n of the color image 126 as shown in FIG. The number of votes is held at the three-dimensional position (point) of the pixel screen position (x, y) and radius n. For example, if the maximum value of the number of votes is 255 (1 BYTE), the size of the voting table 230 is H × V × N (BYTE). In this case, if the high-resolution color image 126 is used, in addition to the problem that the storage area of the memory spent on the voting table 230 becomes large, if the number of votes is limited (small), the number of votes obtained is affected by noise or the like. There may also be a problem that the peak is difficult to appear.

  The latter problem is taken into account by taking into account the noise and taking measures by adding a margin, for example, voting not only on the radius n of the corresponding point 304 but also on the radius n near the corresponding point 304. However, there is a problem that the processing load increases accordingly. Further, it is conceivable that the resolution is lowered and, for example, voting is performed in units of blocks of 2 horizontal pixels × 2 vertical pixels. However, the accuracy of identifying the road sign at the corresponding point 304 is inevitably deteriorated as the resolution is lowered.

  Therefore, in this embodiment, two voting tables 230a (first voting table) and 230b (second voting table) having different dimensions and resolutions are provided, and the voting unit 166 simultaneously applies to the voting tables 230a and 230b. I will vote.

  In the voting table 230a, as shown in FIG. 13B, the resolution of the color image 126 remains the same as the number of horizontal pixels H × the number of vertical pixels V, and the information on the radius n is omitted and one dimension is dropped. It is shown in the voting space (horizontal pixel position, vertical pixel position). Therefore, the size of the voting table 230a is H × V (BYTE), and the number of votes for all radii n is held at the two-dimensional position of the screen position (x, y) of the pixel that is the voting destination. On the other hand, in the voting table 230b, as shown in FIG. 13C, the resolution with respect to the radius n is the same value N, and the dimension is not omitted. Instead, the resolution of the color image 126 is Three-dimensional voting space compressed to 4 (with reduced resolution) and the number of horizontal pixels H / 4 (value obtained by compressing the number of horizontal pixels) × number of vertical pixels V / 4 (value obtained by compressing the number of vertical pixels) Indicated by Therefore, the size of the voting table 230b is H / 4 × V / 4 × N (BYTE), and the block (horizontal 4 pixels × vertical 4 pixels) to which the screen position (x, y) of the pixel to be voted belongs belongs to. The number of votes is held at a three-dimensional position with the radius n. In this way, the voting tables 230a and 230b are those in which the resolution of the radius n is intentionally lowered and the resolution of the screen position is intentionally lowered.

  When the voting unit 166 derives the corresponding point 304 based on the feature point 300, the voting unit 166 votes simultaneously to the respective voting tables 230a and 230b. However, the voting table 230a is voted for one point corresponding to the corresponding point 304 regardless of the radius n, and the voting table 230b is voted for the point of the radius n of the block to which the corresponding point 304 belongs. Thus, when the voting is completed, the voting unit 166 sets a point (corresponding point 304) having a large total number of votes with the radius n in the voting table 230a as a candidate for the central position of the road sign, and sets the central position in the voting table 230b. A radius n having a large number of votes in the corresponding block can be set as a candidate for the radius n of the road sign.

  In this way, it is possible to reduce the total storage capacity of the voting table 230 to H × V + H / 4 × V / 4 × N (BYTE) while maintaining the accuracy of identifying the center of the road sign with high accuracy. It becomes possible. Here, if H = 600 pixels, V = 200 pixels, and N = 20 pixels, 600 × 200 × 20 = 2,400,000 BYTE is necessary, but 600 × 200 + 600/4 × 200 corresponding to about 1/10. / 4 × 20 = 270,000 BYTE.

  By the way, the voting unit 166, after finishing the voting process for all the feature points, extracts the number of votes from each point of the voting table 230, and the corresponding points 304 at which the total number of votes of the radius n becomes a predetermined value or more. Is a candidate for the center point of the road sign. However, even if the storage capacity of the voting table 230 is reduced, when the voting space has a certain size, it is a processing load to determine the number of votes obtained for the entire voting table 230. Therefore, the voting unit 166 improves the extraction efficiency of the center point candidates by selecting the center point candidates in parallel with the voting.

  FIG. 14 is an explanatory diagram for explaining the center point candidate list 310. Here, in addition to the voting table 230, a center point candidate list 310 is provided. In the center point candidate list 310, at least the screen position (x, y) related to the voting table 230a is registered. The voting unit 166 additionally registers the corresponding point 304 corresponding to the point in the center point candidate list 310 as needed when the number of votes on the voting table 230a of the currently voting point becomes a predetermined number or more. The voting unit 166, after finishing the voting process for all the feature points, sets only those registered in the center point candidate list 310 as road sign candidates.

  With this configuration, it is possible to appropriately extract the center point candidate while avoiding the determination of the number of votes in the entire voting table 230, that is, while reducing the processing load. The voting unit 166 limits the center point candidates registered in the center point candidate list 310 to a predetermined value (for example, 50). This is due to the following reason. In other words, if the corresponding points 304 are dispersed to a plurality of center point candidates due to the influence of noise or the like, there are cases where a plurality of center points are candidates although they are originally one road sign. In this case, the center point candidates should not be extracted indefinitely, and the possibility that there are usually 50 or more labels in the color image 126 is small. When the number of center point candidates in the center point candidate list 310 is 50 or more in this way, the voting unit 166 stops the voting process in one frame, and only the center point candidate specified up to that point is the sign specifying process and after. Perform the process.

  Thus, the voting unit 166 generates the center point candidate list 310. The center point candidate list 310 includes the radius, the number of votes in the voting tables 230a and 230b, and the tertiary of the pixel, in addition to the screen position (center position). Information such as the original position is also associated.

  When the voting unit 166 performs voting on the voting tables 230a and 230b as described above, the voting unit 166 initializes the voting tables 230a and 230b so that the number of votes becomes 0 for voting in the next frame. However, depending on the resolution of the color image 126, the processing time spent initializing the voting tables 230a and 230b is not negligible, and may occupy 40% of the entire sign detection processing S202, for example. In the voting tables 230a and 230b, as the number of dimensions increases, the storage capacity of the memory increases. In particular, the influence of the load of initialization processing on the three-dimensional voting table 230b increases.

  FIG. 15 is an explanatory diagram for explaining the flag table 320. Here, a flag table 320 is provided in addition to the voting tables 230a and 230b. As shown in FIG. 15A, the flag table 320 is a table in which the number of dimensions of the voting table 230b is reduced by a radius n (one dimensional reduction), and the screen position (x, y) of the pixel that is the voting destination. A flag is set at a two-dimensional position of a block (4 horizontal pixels × 4 vertical pixels) to which the group belongs. Therefore, the size of the flag table 320 is H / 4 × V / 4 (BYTE).

  When the voting unit 166 votes for an arbitrary radius n of an arbitrary block of the voting table 230b shown by cross hatching in FIG. 15B, the voting unit 166 in the flag table 320 shown by cross hatching in FIG. The flag of the block equal to that block is turned ON. Therefore, when a vote is given to any of the radii n in each block of the voting table 230b, the corresponding block of the flag table 320 is turned ON. When the voting on the voting tables 230a and 230b is completed, the voting unit 166 corresponds to the block whose flag is turned on in the flag table 320 shown by hatching in FIG. 15A. Only points corresponding to N blocks in the voting table 230b indicated by hatching are initialized so that each point is obtained so that the number of votes becomes zero. In other words, the voting unit 166 does not perform the initialization process on the block of the voting table 230b corresponding to the block whose flag is turned off in the flag table 320.

  Here, the voting table 230b is cited as the target of the flag table 320, but the concept of the flag table 320 is not limited to this case, and is applied to the voting table 230 having a size of H × V × N (BYTE). it can. In that case, the size of the flag table 320 is H × V (BYTE).

  Further, instead of providing the flag table 320, it is determined whether or not voting is made in an area corresponding to each block of the voting table 230a. If voting is made, only the block of the voting table 230b corresponding to that block is determined. Each point may be initialized so that the number of votes becomes zero. In this way, it is not necessary to initialize all the points in the voting table 230b, so that the load of the initialization process can be significantly reduced.

  When the voting on the voting tables 230a and 230b is completed, the voting unit 166 displays the voting table 230a corresponding to the block whose flag is turned on in the flag table 320, as indicated by hatching in FIG. Only for a plurality of pixels (horizontal 4 pixels × vertical 4 pixels), each point may be initialized so that the number of votes is zero. Thus, as with the voting table 230b, it is not necessary to initialize all the points in the voting table 230a, so that the load of the initialization process can be significantly reduced.

  Then, the voting unit 166 initializes all the flags in the flag table 320 to OFF after the initialization process is completed. Thus, it is possible to appropriately initialize the voting table without causing an increase in the load of the initialization process.

(Sign identification processing)
The sign identifying unit 168 narrows down the road sign candidates derived by the voting unit 166 based on the first to third narrowing conditions, and identifies the road sign.

  As a first narrowing-down condition, the sign specifying unit 168 sets a radius n where the number of votes in the voting table 230a is equal to or larger than a predetermined value and the number of votes of a block in the voting table 230b is equal to or larger than a predetermined value, Narrow down. However, as described above, the voting unit 166 also registers the corresponding points 304 in the central point candidate list 310 as needed according to the number of votes in the voting tables 230a and 230b. However, the registration to the center point candidate list 310 is performed during the voting in which the final number of votes is still unknown, and the number of votes obtained when the vote is completed is not determined. Therefore, in the present embodiment, the corresponding point 304 registered in the center point candidate list 310 is again compared with a predetermined value larger than that at the time of registration to exclude the corresponding point 304 corresponding to noise, Only appropriate corresponding points 304 can be left.

  Subsequently, the label specifying unit 168 derives, as an occupied area, a rectangle centered on the center position with a length twice as long as the radius n based on the center position and the radius n. However, regarding any two corresponding points 304, if such occupied areas overlap (superimpose), there is a risk that one of the two corresponding points will become unrecognizable. In this case, if the road sign of the other corresponding point 304 that has become unrecognizable is an important road sign such as a road sign that indicates a speed limit, such an important road sign may not be recognized. End up. Therefore, the sign identifying unit 168 has low reliability as a road sign when the occupied area of an arbitrary corresponding point 304 overlaps with the occupied area of another corresponding point 304 as a second narrowing-down condition. By leaving the way, leave a reliable road sign. The reliability of such road signs is obtained based on the magnitude relationship between the number of votes obtained from the two-dimensional voting table 230a and the number of votes obtained from the three-dimensional voting table 230b.

FIG. 16 is a flowchart showing an example of the label specifying process. As illustrated in FIG. 16, the sign specifying unit 168 sequentially selects two candidates from among road sign candidates (S330). And the label | marker specific | specification part 168 determines whether the occupancy area | region of two selected candidates overlaps (S332). As a result, if the occupancy areas of the two candidates overlap (YES in S332), the sign specifying unit 168 determines that the number of votes C ₁ and D _{1 in} the voting tables 230a and 230b of one candidate is the other candidate. It is determined whether or not both of the vote counts C ₂ and D _{2 in the} voting tables 230a and 230b are larger (S334). As a result, if both are large, that is, if C ₁ > C ₂ and D ₁ > D ₂ (YES in S334), the label specifying unit 168 excludes the other candidate (S336). If the two candidate occupation areas do not overlap (NO in S332), the process proceeds to step S344.

Further, if C ₁ > C ₂ and D ₁ > D _{2 are} not satisfied (NO in S334), the label specifying unit 168 determines that the number of votes C ₁ and D _{1 in} the voting tables 230a and 230b of one candidate is the other candidate. It is determined whether or not the number of votes C ₂ and D _{2 in} the voting tables 230a and 230b is smaller (S338). As a result, if both are small, that is, if C ₁ <C ₂ and D ₁ <D ₂ (YES in S338), the label specifying unit 168 excludes one candidate (S340). As described above, when the number of votes in the candidate voting tables 230a and 230b is large, it is determined that the reliability of the road sign is high, and candidates with small numbers of votes are excluded and only candidates with large numbers of votes are selected. leave.

Further, if C ₁ <C ₂ and D ₁ <D _{2 are} not satisfied (NO in S338), the sign specifying unit 168 moves downward based on the positions of the one candidate and the other candidate on the color image 126. The candidate located at is excluded (S342). As described above, if any of the votes obtained in the candidate voting tables 230a and 230b is large and any of them is small, it is not possible to determine only by the number of votes, so only the candidate located above is adopted and located below. Candidates are excluded. This is because if two road signs are arranged vertically above and below, a relatively important road sign indicating a speed limit is arranged above.

  When two candidates selected in this way are duplicated and it is determined which one to exclude, the label specifying unit 168 determines whether or not all combinations of the two candidates to be selected have been completed. (S344). As a result, if it is finished (YES in S344), the sign specifying process is finished. If it is not finished (NO in S344), the process from Step S330 is repeated. In this way, even if the area occupied by two candidate road signs overlaps, it is possible to appropriately narrow down to candidates with high reliability.

  Next, the sign specifying unit 168 determines whether or not the candidate road sign narrowed down by the first and second narrowing conditions exceeds the candidate upper limit value (here, 3) as the third narrowing condition. Here, if the candidate upper limit value is exceeded, the label specifying unit 168 narrows the candidates to the candidate upper limit value or less, and does not execute the subsequent processing for the other candidates. Specifically, when the road sign candidate exceeds the candidate upper limit value, the horizontal distance x at the three-dimensional position of all candidates is compared, and the candidate upper limit value is ascending in order of the horizontal distance x from the lane of the host vehicle 1. Narrow down to candidates. In this way, it is possible to appropriately extract candidates close to the lane of the own vehicle 1 that are highly likely to be road signs for the own vehicle 1.

  Subsequently, the sign correction unit 170 corrects the position and size of the road sign narrowed down to the candidate upper limit value or less. This is because in this embodiment, template matching is used for recognizing the contents of the road sign, and template matching greatly affects the recognition accuracy due to the positional deviation of the image. Therefore, here, the center position and the radius n derived by the voting unit 166 are corrected, and the road sign occupation area is set again. For this reason, the sign correction unit 170 detects a red frame that exists in four directions, the horizontal direction and the vertical direction, from the center position of each candidate road sign, and a rectangle that circumscribes the red frame (circumferential portion of the road sign). The occupied area is corrected so that Specifically, the occupied area is corrected by the following procedures (1) to (7).

  FIG. 17 is an explanatory diagram for explaining the processing of the sign correction unit 170. (1) First, the label correction unit 170 sets a rectangle centered on the center position as a side having a length twice the radius n as shown in FIG. A component histogram is obtained (voting with the horizontal axis as the V component). If the difference between the maximum value and the minimum value of the V component histogram is equal to or greater than a predetermined value, the marker correction unit 170 determines that the candidate is a red frame. If the difference between the maximum value and the minimum value of the V component histogram is less than the predetermined value, the candidate is determined to be a black frame, and the V component histogram is replaced with a Y component histogram. In the following, the correction process for the occupied area 346 will be described with a candidate for a red frame, but it goes without saying that the process can be applied to a black frame.

  (2) The label correction unit 170 has a predetermined percentage (for example, 30%) from the top of the V component histogram (the ratio of the top: lower area when the histogram is converted into an area is 3: 7) V A component value (threshold value Vthr) is derived. The threshold value Vthr is different from a predetermined value of the V component used for specifying the feature point. This is because an optimum threshold is set for each candidate. When the threshold value Vthr is a predetermined value (for example, −5) or less, the subsequent processing is not performed as an inappropriate threshold value.

  (3) As indicated by an arrow in FIG. 17B, the sign correction unit 170 determines whether or not the V component of each pixel is equal to or higher than the threshold value Vthr from the center position of each candidate road sign. The detection pixel is moved in the four directions of the direction and the vertical direction. Then, when a predetermined number (for example, three pixels) of pixels whose V component is equal to or greater than the threshold value Vthr continues, a pixel in which a V component equal to or greater than the threshold value Vthr is detected is defined as an inner edge 348 of the red frame. Here, the four directions of the horizontal direction and the vertical direction are described as the detection directions. However, the present invention is not limited to this, and four orthogonal radial directions are sufficient, and an oblique direction is added. In addition, the detection accuracy can be increased.

(4) Subsequently, the sign correction unit 170 determines whether or not the position of the inner edge 348 of the red frame is within a predetermined range where it should originally be located. Specifically, for example, when detecting in the horizontal right direction, assuming that the abscissa of the center is J, the x coordinate of the horizontal right end of the occupied area 346 is R, and the coordinates of the inner edge 348 of the obtained red frame are RE, In the case where Expression 3 is satisfied, the coordinate RE of the inner edge 348 of the red frame is set to an inappropriate value and the subsequent processing is not performed.
RE <(R−J) × K + J (Formula 3)

  Here, K is a coefficient taking any value from 0 to 1. For example, 0.6 for horizontal detection and 0.5 for vertical detection. Further, (R−J) × K is a lower limit value (inner edge lower limit value) in the radial direction that the inner edge 348 can take. For example, such processing is a measure for preventing an erroneous inner edge 348 from being taken due to an influence on the V component when the numerical value becomes orange in an electric display type road sign.

  (5) Next, the sign correction unit 170 again sets the center position and the radius n of the image based on the position of each inner edge 348 when all four directions in the horizontal direction and the vertical direction are matched in an arbitrary road sign. To derive. Specifically, of the horizontal inner edge 348, the center position when the left inner edge 348 is LE and the right inner edge 348 is RE is determined by (LE + RE) / 2, and the radius n Is determined by (RE-LE) / 2. The center position and the radius n are determined by the same process for the inner edge 348 in the vertical direction. Thus, the occupied area 346 specified by the center position of the road sign is newly defined.

  (6) Subsequently, the sign correction unit 170 compares the radius n before correction with the radius n after correction for the road sign, and the ratio is within a predetermined range (for example, 0.75 times or more and less than 1.5 times). If it deviates from), the subsequent processing is not performed as an incompatible value.

  (7) Finally, the sign correction unit 170 resizes the corrected occupied area 346 into a rectangle of horizontal predetermined pixels × vertical predetermined pixels, and ends the correction process. As described above, by performing readjustment of the center position and radius of the road sign, it is possible to improve the accuracy of recognition by pattern matching. For resizing, a general method such as arrest neighbor can be used.

(Signal content recognition process S204)
FIG. 18 is a flowchart showing a specific process flow of the sign content recognition process S204. The sign content recognition unit 172 discretizes the brightness of the image corresponding to the occupied area 346 with the image as it is for the road sign corrected by the sign correction unit 170 (S350), and the road sign presents the release of the speed limit. It is determined whether it is a road sign (S352). The sign content recognition unit 172 determines that the road sign is a road sign that presents the speed limit unless the road sign presents the release of the speed limit, and performs vertical alignment (S354). After the matching, horizontal matching is executed (S356). Subsequently, the sign content recognizing unit 172 pays attention to a predetermined part of the content of the road sign, performs template matching of the noticed part (S358), derives a comprehensive evaluation value, and determines which speed limit it is. (S360).

  By the way, as mentioned above, the road sign targeted by the present embodiment includes an electric display type and a non-electric display type. The electric light display type has the characteristics that the content of the road sign, for example, the luminance of the numerical value is higher than the surrounding luminance, and the non-electric light display type has the content of the road sign, for example, the luminance of the numerical value is lower than the peripheral luminance. Have

  In the sign content recognition process S204, since it is not grasped which display type it is, the recognition process is executed on the assumption that both display types are used. For example, the sign content recognition unit 172 executes a series of processes from steps S350 to S360 using the image of the road sign as it is corrected by the sign correction unit 170, and determines whether the contents of the road sign can be recognized effectively. (S362). As a result, if it is determined in any one of steps S350 to S360 that the contents of the road sign could not be recognized effectively (NO in S362), the brightness of the road sign corrected by the sign correction unit 170 is inverted. (S364) A series of processing from steps S350 to S360 is executed again on the road sign (inverted road sign) whose luminance is inverted.

  If it is determined in any of steps S350 to S360 that the content of the road sign can be recognized effectively (YES in S362), the content of the reverse road sign is not performed without reversing the road sign. Without recognizing, the process proceeds to step S366. In this way, it is possible to recognize either the road sign or the reverse road sign corrected by the sign correction unit 170, and appropriately recognize the contents of the road sign regardless of the display type such as lightning / non-lightning. It becomes.

  However, if it is determined in any one of steps S350 to S360 before the inversion that the contents of the road sign cannot be recognized effectively, the processing is interrupted even during the recognition processing, and the subsequent processing is omitted. The process can also be moved to step S362. In this way, unnecessary recognition processing can be avoided and the processing load can be reduced. Since the corrected image and the image whose luminance is inverted are subjected to the same processing, for the sake of convenience of explanation, the processing of the image after correction will be described below, and the image whose luminance is inverted will be described in detail. Omitted.

  Here, after processing of the corrected image, processing of an image with inverted luminance is performed, but this order may be reversed. For example, if the road sign indicating the speed limit is located on the shoulder according to the environment outside the vehicle, the road sign is likely to be a non-lightning display type. In the case where the road sign is located, since it is highly possible that the road sign is an electric display type, the image with the luminance reversed is processed first. In this way, by performing the processing first from the side that is highly likely to be completed in one round (steps S350 to S360), the sign content recognition processing S204 can be made more efficient.

  In addition, if the content of the road sign can be recognized effectively in either the corrected image or the inverted image in which the luminance is inverted (YES in S362), the sign content recognition unit 172 performs all the road signs corrected by the sign correction unit 170. In step S366, it is determined whether the processes in steps S350 to S364 have been executed. As a result, if all the road signs are not completed (NO in S366), the sign content recognition unit 172 repeats the processing from step S350 until it is completed (YES in S366). Hereinafter, the contents of the processing in steps S350 to S360 will be described in detail.

(Luminance Discretization Processing S350)
The sign content recognition unit 172 discretizes the luminance of each pixel with respect to the occupied area 346 of each road sign corrected by the sign correction unit 170. Thus, the image pattern is converted into a recognizable image pattern regardless of the imaging state.

  FIG. 19 is an explanatory diagram for explaining the recognition target area 370. First, the sign content recognition unit 172 sets an area (hereinafter, referred to as a recognition target area) 370 to be recognized in a rectangle of horizontal predetermined pixels × vertical predetermined pixels. The recognition target area 370 is a rectangular area in contact with the inner edge 348 of the red frame in the occupied area 346 shown in FIG. 19. In the subsequent processing, the recognition target area 370 is processed.

  Next, the sign content recognition unit 172 discretizes each pixel in the recognition target area 370 and converts it to N-value. For example, when N = 2, the luminance of each pixel has a value of 0 or 255. Here, N is a value of 2 or more, but in this embodiment, for example, N = 5 in order to suppress the influence on pattern matching when binarization is not successful due to the influence of threshold setting or the like. To do. In the case of quinarization, there are four discretization thresholds, and four predetermined% (for example, 20, 25, 35, 40%) are set from the top in the luminance histogram. Such a predetermined percentage can be set independently and arbitrarily.

  With this configuration, it is possible to appropriately recognize the content of the road sign regardless of the difference in the luminance value distribution. Here, since the upper value of the luminance histogram is used as a reference, quinary can be appropriately performed regardless of the luminance distribution mode of each recognition target region 370, and normalization can be achieved along with discretization. It will be.

(Speed limit release determination process S352)
The sign content recognizing unit 172 performs recognition processing corresponding to each of the road sign indicating the release of the speed limit or the road sign indicating the speed limit on the five-valued recognition target area 370. Since the load can be reduced, first, processing is performed on the premise that the former is the case. If not, processing for the latter is performed. In this way, it is possible to avoid unnecessary recognition processing of the road sign indicating the speed limit.

FIG. 20 is an explanatory diagram for explaining a road sign presenting the release of the speed limit. The sign content recognition unit 172 includes a plurality of line segments L1, L2, L3, and L4 having an inclination angle (inclined) that intersect on the recognition target area 370 in the recognition target area 370 illustrated in FIG. The luminances of the pixels are accumulated, and the accumulated luminance values at the line segments L1, L2, L3, and L4 are denoted by S1, S2, S3, and S4. The inclination angle of the line segment is an angle that matches the display mode of the road sign that indicates the release of the speed limit. Here, the threshold values for determining the accumulated luminance values S1, S2, S3, and S4 are TS1, TS2, TS3, and TS4. However, TS4 is composed of two threshold values having a relationship of TS4a <TS4b. Alternatively, TS2 = TS3. The sign content recognition unit 172 presents the release of the speed limit for the road sign when all of Expression 4 is satisfied based on Expression 4 below, which digitizes the difference in shading in the four line segments L1, L2, L3, and L4. Recognized as a road sign.
S1 <TS1
S2> TS2
S3> TS3
TS4a <S4 <TS4b (Formula 4)
Here, since the deviation of the luminance due to the positional deviation and the brightness is corrected, it becomes possible to recognize the contents of the road sign by a very simple process as shown in Equation 4 above.

(Vertical alignment processing S354)
If it is not determined by the above processing that the road sign indicates that the speed limit is released, the road sign is determined as a road sign that indicates the speed limit. First, the sign content recognition unit 172 performs vertical alignment for a numerical value area occupied by a numerical value in the recognition target area 370. This is because the recognition target area 370 includes a minute positional deviation, and the size, shape, distance between the numerical values, and the like may differ from country to country or from country to country.

  FIG. 21 is an explanatory diagram for explaining the vertical alignment processing. As shown in FIG. 21, the sign content recognition unit 172 accumulates the luminance of each pixel in the recognition target area 370 in the horizontal direction, arranges the accumulated luminance values in the vertical direction, and generates a luminance distribution 372 in the vertical direction. However, in the case of the road sign in FIG. 21, the numerical value portion has a low luminance with respect to the surroundings. Therefore, the luminance distribution 372 in FIG. Looking for. Then, the sign content recognition unit 172 obtains the maximum value 374 of the cumulative luminance value in any of the vertical upper and lower regions from the central portion of the recognition target region 370 and sets a predetermined percentage (for example, 25%) of the maximum value as a threshold value from the central portion. When the detection pixel is moved vertically up and down and a predetermined number (for example, twice) of pixels whose accumulated luminance value falls below the threshold value continues, the locations are set as the upper and lower ends of the numerical value area 376.

  Next, the sign content recognition unit 172 uses the upper end and lower end of the numerical area 376 derived in this way, and normalizes by enlarging or reducing the size in the vertical direction. For example, when the distance between the upper end and the lower end of the numerical area 376 is HI and the distance in the vertical direction of the template is HT, the sign content recognition unit 172 multiplies the numerical area 376 by HT / HI in the vertical direction. In this way, the size of the numerical value region 376 can be adjusted to the size in the vertical direction of the template to be subsequently matched. Correction is performed by a near neighbor.

  By such processing, for example, noise is generated in the accumulated luminance value in the lower part of FIG. 21, but the influence can be excluded, and the normalized numerical area 376 can be appropriately extracted.

(Horizontal matching process S356)
FIG. 22 is an explanatory diagram for explaining a template. The sign content recognition unit 172 recognizes the content of the recognition target area 370 that has been aligned in the vertical direction, that is, the numerical value. Such recognition is performed by matching with a template prepared in advance. As shown in FIG. 22, 13 types of templates are prepared, for example, 10 to 90 (2 digits) and 100 to 130 (3 digits) in 10 increments.

  FIG. 23 is an explanatory diagram for explaining the matching process in the horizontal direction. As shown in FIG. 23, the sign content recognition unit 172 accumulates the pixels of the recognition target region 370 normalized in the vertical direction in the vertical direction, arranges the accumulated luminance values in the horizontal direction, and then distributes the luminance distribution 380 in the horizontal direction. Is generated. In this way, one-dimensionalization of the two-dimensional image is performed. However, since the numerical value portion has low luminance as in the vertical alignment, the luminance distribution 380 in FIG. 23 obtains the accumulated luminance value after inverting the luminance in order to extract the numerical value portion. Further, the cumulative range in the vertical direction is only the range from the upper end to the lower end of the numerical value area 376 derived by the alignment in the vertical direction. Accordingly, it is possible to avoid accumulation of luminance related to unnecessary areas other than the numerical value area in the vertical direction. The sign content recognition unit 172 DP-matches the luminance distribution 380 of FIG. 23 derived in this way with the luminance distribution based on the template, so that the correlation evaluation value with each template (the correlation is higher as the correlation evaluation value is lower). ) Is calculated.

  Here, as in the case of vertical alignment, there are differences in the size of numerical values and the spacing between numerical values, so that sufficient performance cannot be achieved by using a fixed-size template. Therefore, DP matching that allows horizontal expansion and contraction is used. Although it is theoretically possible to perform such DP matching in two dimensions, since the processing amount becomes enormous, one-dimensional DP matching is used in this embodiment.

  FIG. 24 is an explanatory diagram for explaining DP matching. The sign content recognition unit 172 performs DP matching with the template “130”, for example, and obtains a result as shown in FIG. Here, the broken line indicates the luminance distribution 380 of the recognition target region 370 before DP matching, the solid line indicates the luminance distribution 382 of the recognition target region 370 after DP matching, and the alternate long and short dash line indicates the template luminance distribution 384. In DP matching, matching is performed by expanding and contracting the luminance distribution 380 of the recognition target region 370 so as to match the luminance distribution 384 of the template. Therefore, as can be understood with reference to FIG. 24, the luminance distribution 380 of the recognition target region 370 before stretching and the template luminance distribution 384 have a low correlation, but the luminance distribution 382 of the recognition target region 370 after stretching and the template The correlation with the luminance distribution 384 becomes higher.

  However, here, all correlation evaluation values of the luminance distribution 382 of the recognition target region 370 after expansion and contraction and the luminance distributions 384 of a plurality of templates are obtained regardless of the level of the correlation. Specifically, when the luminance distribution 382 of the recognition target area 370 after expansion / contraction is im and the numerical value (limit speed) of the template is T, the sign content recognition unit 172 calculates the correlation evaluation value (square sum of differences) after expansion / contraction. ) DP (im, T) is sequentially derived from DP (im, 10) to DP (im, 130).

  However, subsequent processing is not performed for candidates that are clearly different from the template. For example, since the luminance distribution 382 of the recognition target area 370 is “130”, the two digits “10” to “90” have different numbers in the first place, and therefore DP (10 ”to“ 90 ”corresponding to“ 10 ”to“ 90 ” im, 10) to DP (im, 90) have a low correlation. Therefore, the subsequent processing is omitted for a template in which the value of DP (im, T) exceeds the threshold value (low correlation).

(Attention site matching processing S358)
Here, the correlation evaluation value DP (im, T) is obtained regardless of the number of digits of the numerical value. However, when the change tendency of the numerical value is determined in advance as in the present embodiment, all numerical values of two digits or three digits are used. It is not a good idea to perform matching in. For example, in the numerical values “10” to “90”, the “0” part of the first digit is common to all the numerical values, and “0” in the first digit is common in “100” to “130”. This is because the part and the part “1” in the third digit are common. Accordingly, since all the numerical values are coincident with respect to the common part, if matching is performed for all the digits, it is difficult to cause a difference in the correlation evaluation value.

  Therefore, the sign content recognition unit 172 obtains the correlation evaluation value DP (im, T) as described above, and matches only the second digit in which a difference in numerical shape occurs. However, since the luminance distribution 382 of the recognition target area 370 is expanded or contracted in the horizontal direction, it is necessary to derive which part of the luminance distribution 382 of the recognition target area 370 matches which part of the template luminance distribution 384. I must. Therefore, the sign content recognition unit 172 corresponds to the horizontal coordinate TS (T) of the start position of the second digit of the template, and corresponds to the horizontal coordinate DPR ( im, T). Such horizontal coordinates can be obtained from the history of processing for combining feature points in DP matching. Specifically, information (for example, a route) related to the combination of feature points is stored in advance, and the horizontal coordinates are derived by performing reverse calculation. With such a configuration, the horizontal coordinate can be efficiently obtained by using the midway result of DP matching. Since a specific procedure for such DP matching has already been disclosed through various technical documents, detailed description thereof is omitted here.

  FIG. 25 is an explanatory diagram for explaining the matching process of the target region. When the horizontal coordinate corresponding to the start position of the numerical area of the second digit is obtained as described above, the sign content recognition unit 172 performs simple template matching. Any index may be used for template matching. For example, a sum of absolute values of differences (SAD) is used. The matching target range includes horizontal coordinates DPR (im, T) corresponding to the start position of the luminance distribution 382 of the recognition target region 370 and the second digit of the template, as shown in the example of “130” in FIG. From the horizontal coordinate TS (T) corresponding to the start position of the template, the value in the horizontal direction is the second digit of the template. Here, since alignment in the horizontal direction has already been performed, it is not necessary to perform processing such as optimum value search by matching with shifted positions, and the processing load is greatly reduced.

  However, for example, when the numerical value is 2 digits and when the numerical value is 3 digits, the horizontal length (the horizontal width of the numerical value) is different, and therefore the difference in the horizontal width of the numerical value may affect the matching result. Therefore, the sign content recognition unit 172 adds the horizontal width of the numerical recognition target area 370 with a large number of digits and the numerical recognition target area 370 with a small number of digits to the correlation evaluation value TM (im, T) of DP matching in the second digit. In accordance with the ratio to the horizontal width, a normalization coefficient predetermined for each template is multiplied. For example, if the ratio of the width of a 2-digit numerical value to a 3-digit numerical value is 3: 2, the sign content recognition unit 172 causes the correlation evaluation value TM (im, T) of the second digit from “100” to “130”. ) Is derived, and the result obtained by multiplying the value by 3/2 is defined as TM (im, T). Thus, appropriate evaluation can be performed regardless of the number of digits.

(Evaluation Value Determination Process S360)
Subsequently, the label content recognition unit 172 uses the correlation evaluation value DP (im, T) obtained for each template and the second-digit correlation evaluation value TM (im, T), and uses the following formulas 5 and 6 to generate a template. A comprehensive evaluation value E (im, T) for each is derived.
Overall evaluation value E (im, T) = DP (im, T) × TM (im, T) / F (im) (Equation 5)
F (im) = max (min (TM (im, T)), th) (Formula 6)
Here, since the correlation of the whole numerical value is expressed by DP (im, T), the value is used as it is, and the partial correlation of the second digit is compared with other templates TM (im, T) / F ( im). Here, F (im) is the minimum value min (TM (im, T)) of the correlation evaluation value, but if the value of min (TM (im, T)) becomes too small, the overall evaluation value E (im , T) may diverge. Therefore, when min (TM (im, T)) is smaller than a predetermined value th, the value th is adopted as F (im).

  FIG. 26 is an explanatory diagram for explaining the evaluation determination result. Here, the broken line indicates the correlation evaluation value DP (im, T), the alternate long and short dash line indicates the second-digit correlation evaluation value TM (im, T), and the solid line indicates the overall evaluation value E (im, T). Yes. Referring to FIG. 26, for “100” to “130”, the difference between the original numerical value and the other numerical values is small in the matching of all the numerical values, but by adding only the second digit, It can be understood that the total evaluation value E (im, T) of matching with the template of “130” which is the numerical value of is minimum (correlation is maximum). For convenience of explanation, FIG. 26 also displays templates that are not calculated, and displays templates up to “150”.

  Thus, by performing the matching in all the digits of the numerical value and the matching only in the second digit, the accuracy can be improved and the processing time can be reduced.

(Signal content confirmation processing S206)
In the above, the content of the road sign was recognized. However, as described above, it is not always necessary to recognize the road sign at the moment when it reaches a position where the road sign can be confirmed forward, and it is sufficient if the road sign can be recognized at or after the road sign. Therefore, it is only necessary to recognize the road sign over a plurality of frames and determine the content of the road sign from the information of the plurality of frames. Therefore, the sign content confirmation unit 174 accumulates and confirms the contents of the road sign recognized in one frame in the time direction.

  Here, in order to determine the content of the road sign, four variables are used: a sign accumulation point, a speed limit candidate, a no-marking detection time, and a speed limit output. Here, the sign accumulation point is provided for each of one or a plurality of candidate road signs, and various evaluation values (E (im, T), DP (im, T), TM (im, T) in the sign content recognition process S204. T)) points are shown. The speed limit candidate indicates one speed limit candidate. The sign non-detection time indicates a continuous time in which no road sign is detected. The speed limit output is used to latch a speed limit candidate. When the speed limit output is updated by the speed limit candidate, while the value is held in the speed limit output, the value is notified to the driver or used as a control input of the vehicle control device 130.

The sign content determination unit 174 calculates various evaluation values (E (im, T), DP (im, T), TM (im, T)) derived from the sign content recognition unit 172 for evaluating the probability of the speed limit. The mark accumulation points are accumulated according to the following conditions (1) to (4).
(1) If E (im, T) <ETHR1 & DP (im, T) <DTHR1 & TM (im, T) <TTHR1, the condition of +4 points (2) (1) is not satisfied, and E (im, T) <ETHR2 & DP (im, T) <DTHR2 & TM (im, T) <TTHR2 +2 points provided that ETHR1 <ETHR2, DTHR1 <DTHR2, TTHR1 <TTHR2
(3) Among the templates that satisfy the conditions of (1), if the difference between the minimum value EM of E (im, T) and E (im, T) of all other templates is a predetermined value or more, the minimum The difference between the minimum value EM of E (im, T) and the E (im, T) of one or more other templates among the templates satisfying the condition of +2 points (4) (1) in the template of value EM Is within the predetermined value (ETHR3), +1 point is added to all the templates within the predetermined value in this way. Basic points are added according to the conditions of (1) and (2), and (3) and (4). Points based on the comparison with other templates are added according to the conditions of.

  For example, in the example of FIG. 26, if the conditions (1) to (4) are ETHR1 = 80, ETHR2 = 100, ETHR3 = 5, DTHR1 = 80, DTHR2 = 100, TTHR1 = 100, TTHR2 = 150, “120” and “130” are +4 points based on the condition of 1), “100” and “150” are +2 points based on the condition of (2), and “130” is +2 based on the condition of (3). It becomes a point. In total, “130” is 6 points, “120” is 4 points, “100” and “150” are 2 points, and other numerical values are 0 points. If the road speed sign indicating the speed limit is recognized in the speed limit release determination process, it is uniformly 6 points.

  The sign content determination unit 174 accumulates the contents of the road signs in the time direction as described below based on the accumulated sign points thus obtained, and performs final output.

  FIG. 27 is a time chart showing the flow of the result of road sign notification. The sign content determination unit 174 always accumulates the sign accumulation points for all the detected road sign candidates for each frame. Then, the sign content determination unit 174 compares the sign accumulation point of the current frame with the sign accumulation point of the previous frame, and if the point has not changed, as shown in FIG. Assuming that no label is detected, the label non-detection time is incremented. If the point has changed, as shown in FIG. 27 (2), the sign non-detection time is reset to 0 on the assumption that a road sign is detected in the current frame.

  If a road sign is detected in the current frame, the sign content determination unit 174, as shown in FIG. 27 (3), signs of two road sign candidates from the top having a large accumulated point among a plurality of road sign candidates. Accumulated points and the recognized speed limit are extracted. At this time, since it is determined that the speed limit output is newly updated, the speed limit output currently maintained is reset. Accordingly, the speed limit is not notified. Thus, it is possible to avoid continuously reporting the previous speed limit unnecessarily.

  If the maximum value of the accumulated mark points exceeds a predetermined value (for example, 8 points) and the difference from the second largest accumulated mark point is equal to or greater than the predetermined value (for example, 4 points), As shown in FIG. 27 (4), the speed limit candidate is updated (determined as an output candidate) at the speed limit (for example, 40) of the road sign whose sign accumulation point has the maximum value. In this way, speed limit candidates are extracted. If it is less than the predetermined value, the speed limit candidate is updated to “undefined”. In addition, when the maximum value of the marker accumulation point is equal to or less than the predetermined value, the speed limit candidate is not updated. Therefore, the speed limit candidate maintains “no candidate”.

  Then, after the change of the accumulated point of the sign disappears (passes the road sign), when the sign non-detection time elapses for the output set time (for example, 3 seconds), as shown in FIG. Determines whether there is a speed limit candidate. If there is a speed limit candidate, the speed limit output is updated with the speed limit candidate (for example, 40), and the speed limit notification is resumed. In this way, noisy input of the contents of the road sign can be eliminated.

  Subsequently, after the change in the sign accumulation point has ceased, when a reset time (for example, 5 seconds) that is longer than the output set time has elapsed, as shown in FIG. 27 (6), the sign content confirmation unit 174 In preparation for the next road sign, the sign accumulated point and the speed limit candidate are reset. Thus, preparation for recognition of a new road sign is performed.

  Subsequently, after the change in the accumulated mark point has ceased, when the upper limit time (for example, 10 minutes) of the undetected mark has elapsed, as shown in FIG. 27 (7), the sign content determination unit 174 outputs the speed limit output. Reset. Thus, it is possible to avoid continuously reporting the previous speed limit unnecessarily.

  When it is determined that the vehicle 1 is turning left or right (for example, the absolute value of the steering angle exceeds 360 degrees), the sign content confirmation unit 174 determines the sign accumulation point, the speed limit candidate, the limit Reset all speed outputs. This is because when the host vehicle 1 turns left or right, the traveling road changes, and thus the speed limit of the road that has traveled so far is no longer applied.

  By configuring in this way, the sign content confirmation unit 174 notifies the speed limit 3 seconds after passing the road sign, and the speed limit is detected until 10 minutes have elapsed, left or right turn, or another road sign is detected. Can be maintained. Further, it is possible to eliminate noise-like input of the content of the road sign and increase the accuracy of specifying the content of the road sign.

  In addition, the following processing can be additionally performed in order to increase the practicality. For example, when there are a plurality of lanes of the vehicle, the speed limit may be different for each lane in the road sign arranged at the gate. In the present embodiment, the number of road signs to be recognized is limited to three or less. Therefore, when the speed limit is different for each lane as described above, there are mark accumulation points for different road signs. Assuming that only the correct speed limit is accumulated for each road sign, the sign accumulated point is 6 points, for example, at any speed limit. Then, although the points are appropriately accumulated, the speed limit candidate is updated to “undefined” in the above determination.

  Therefore, in the present embodiment, the sign content determination unit 174 derives each horizontal distance x when there are a plurality of candidate road signs whose sign cumulative points have a significant value at one time in one frame, and any one of them. If the horizontal distance x of the road sign is less than a threshold value (for example, 3 m) and other road signs are greater than or equal to the threshold value, one point is added to one road sign, and one point is subtracted from the other road sign. In this way, the speed limit of the road sign closest to the host vehicle 1 can be preferentially set as a speed limit candidate.

(Differences in road signs by country)
FIG. 28 is an explanatory view showing a display mode of country-specific road signs. As can be understood by comparing the road sign indicating the speed limit for Germany in FIG. 28 (a) with the road sign indicating the speed limit for Switzerland in FIG. 28 (b), the road sign indicating the speed limit is The size, shape, and distance between values may vary from country to country. In addition, as shown in FIG. 28 (c), the road sign indicating the release of the speed limit for Germany is compared with the road sign indicating the release of the speed limit for Italy in FIG. 28 (d). The angle may be different.

  Accordingly, the sign content determination unit 174 determines in which country the road speed candidate is the road sign in parallel with the determination of the road sign contents accumulated and determined in the time direction as described above. Then, the country where the vehicle is currently traveling is correctly grasped, and the speed limit or the like is appropriately recognized using the template of the country.

  FIG. 29 is an explanatory diagram for describing a road sign template. The country determination process is basically executed using a template for each country. Therefore, as shown in FIG. 29, templates arranged in two dimensions are prepared for each country (country A to country Z) and for each speed limit ("10" to "150"). For road signs that indicate the release of the speed limit, the angle information of the diagonal lines is held instead of the template. Information such as the radius n in the sign detection process S202 and the threshold value (predetermined percentage) for N-value conversion in the sign content recognition process S204 is also stored for each country.

  Here, when the environment recognition system 100 outside the vehicle is linked to the navigation system, the template may be switched according to the current country information obtained from the navigation system. The country is determined.

  Since the country determination process has a low real-time requirement, once the image of the recognition target area is acquired, it is temporarily stored in the temporary image memory, and the space after the vehicle environment recognition process for each frame is completed. Process over time and across frames. Here, in order to determine the country, the country-specific accumulated points provided for each of one or more candidates for the country are used as variables. The sign content determination unit 174 initializes the temporary image memory area and the country-specific accumulated points at a predetermined timing.

  In the country determination process, the sign content determination unit 174 determines whether the country determination process has already been executed in the current frame. As a result, if the country determination process has already been executed, the process is continued. If the previous country determination process has been completed, a new country determination process is started. Since the country determination process is performed in the free time as described above, if the specified processing time of the frame is reached during the process, the process is temporarily stopped and continued in the next frame.

  Then, in the above-described sign content confirmation process S206 of the current frame, the sign content confirmation unit 174 determines that only one speed limit (including the same case of a plurality of road signs) is the cumulative number of signs in one or a plurality of road sign candidates. Determine whether you have earned points. As a result, if no road sign is detected, or points for multiple speed limits are obtained with multiple road signs, only the speed limit for one road sign has not obtained the cumulative number of signs, The process is terminated, and the determination of the accumulated mark points is repeated in the next frame.

  Subsequently, when only one speed limit is obtained for the total number of signs, the sign content determination unit 174 stores the image of the recognition target area 370 in the temporary image memory with the recognition result of the speed limit as the speed limit V. To do. Here, when the speed limit V has obtained the total number of sign points for all the plurality of candidate road signs, the comprehensive evaluation value E (im, V) is the lowest (the correlation is maximized). The image of the target area 370 is stored. The image stored here is the occupied area 346 normalized to a rectangle of horizontal predetermined pixels × vertical predetermined pixels after the end of the sign detection process S202.

  Next, the sign content determination unit 174 executes the sign content recognition process S204 on the template of the speed limit V of each country based on the image of the occupied area stored in the image memory. That is, in the above-described sign content recognition processing S204, as shown by the broken line in the template of FIG. 29, the template of each speed limit T for one country is used. Here, as shown by the solid line in the template of FIG. In addition, each country's template for one speed limit V is used. Therefore, when the country identifier is CN and the speed limit is V, the comprehensive evaluation value E (CN, V) is derived instead of the comprehensive evaluation value E (im, T). The evaluation value E (CN, V) for the country prepared as a template can be obtained by such processing.

  However, in the present embodiment, the weights of the currently recognized country and the other evaluation values E (CN, V) are different. For example, the sign content determination unit 174 multiplies only the evaluation value E (CN, V) of the currently recognized country by a weighting coefficient (for example, 0.8 of 1 or less). This is to make the evaluation value E (CN, V) of the currently recognized country relatively low (increase the correlation) so that hunting does not occur in the country determination result. Further, when the country adjacent to the currently recognized country is grasped, the adjacent country may be multiplied by a weighting coefficient (for example, 0.95 between 0.8 and 1).

  The sign content determination unit 174 compares the evaluation values E (CN, V) for all the countries derived in this way to derive the minimum value ECM. Then, if the difference between the minimum value ECM and the evaluation values E (CN, V) of all other templates is equal to or greater than a predetermined value, +1 is added to the cumulative total points of the minimum value ECM template by country.

  Subsequently, the sign content determination unit 174 compares the maximum value of the cumulative points by country with all the cumulative points of other countries, and if the difference is equal to or greater than a predetermined value (for example, 30), the maximum value is obtained. Determine if the country is equal to the currently recognized country. As a result, if the country with the maximum value is equal to the currently recognized country, all country cumulative points are multiplied by ½, and the total cumulative marker points are lowered for fair judgment. Also, if the country with the highest value is different from the currently recognized country, it is determined that the country where the vehicle is traveling has changed, and the currently recognized country is updated to the country with the highest value. Initialize the temporary image memory area and country cumulative points. In addition, if the difference between the maximum value of country cumulative points and all other country cumulative points is less than the predetermined value, the temporary image memory area is initialized, and the marker cumulative points are determined in the next frame. repeat.

  As described above, by appropriately determining the country in which the vehicle is currently traveling, it is possible to increase the accuracy of identifying the content of the road sign. Moreover, it is possible to suppress an increase in processing load by executing the above country determination process in the background of the recognition process of the road sign of one country.

  As described above, in the outside environment recognition device 120 of the present embodiment, it is possible to improve the recognition accuracy of the content of the road sign while suppressing an increase in processing load.

  Also provided are a program that causes a computer to function as the vehicle exterior environment recognition device 120 and a computer-readable storage medium that stores the program, such as a flexible disk, magneto-optical disk, ROM, CD, DVD, and BD. Here, the program refers to data processing means described in an arbitrary language or description method.

  As mentioned above, although preferred embodiment of this invention was described referring an accompanying drawing, it cannot be overemphasized that this invention is not limited to this embodiment. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Is done.

  It should be noted that each step of the vehicle environment recognition processing in the present specification does not necessarily have to be processed in time series in the order described in the flowchart, and may include processing in parallel or by a subroutine.

  INDUSTRIAL APPLICABILITY The present invention can be used for a vehicle environment recognition device that recognizes the contents of road signs installed on the road.

120 Outside-vehicle environment recognition device 160 Image acquisition unit 162 Position information deriving unit 164 Feature point specifying unit 166 Voting unit 168 Sign specifying unit 170 Sign correcting unit 172 Sign content recognizing unit 174 Sign content determining unit

Claims (4)

An image acquisition unit for acquiring images;
Using the image, a sign identifying unit that identifies a circle with a predetermined radius centered on an arbitrary pixel as a road sign,
The identified road sign is converted into an N value based on the luminance distribution, and the content of the road sign is recognized from either the N-valued road sign or an inverted road sign obtained by inverting the luminance of the road sign. A sign content recognition unit;
Equipped with a,
The road sign includes a road sign that indicates the release of the speed limit and a road sign that indicates the speed limit,
The sign content recognizing unit recognizes in order of a road sign presenting the release of the speed limit, followed by a road sign presenting the speed limit, and if the road sign is a road sign presenting the release of the speed limit, A vehicle environment recognition apparatus characterized by not recognizing a road sign indicating speed .   The sign content recognition unit does not invert the brightness of the road sign or does not recognize the content of the inverted road sign when the content of the N-valued road sign can be recognized effectively. Item 1. The vehicle environment recognition device according to Item 1.   When the sign content recognition unit determines that effective recognition cannot be performed during the recognition of the N-valued road sign content, the sign content recognition unit interrupts the recognition of the road sign and recognizes the inverted road sign. The external environment recognition device according to claim 1 or 2. The sign content recognition unit recognizes a road sign that presents the release of the speed limit based on a cumulative luminance value in a plurality of line segments having an inclination angle that intersect on the road sign. The external environment recognition device according to any one of 1 to 3 .

Images