JP6733582B2 - Sign recognition system - Google Patents

Description

The present invention relates to a sign recognition system for recognizing road signs.

Patent Literature 1 discloses a sign recognition system that recognizes a road sign using a front image of a vehicle captured by a camera. The sign recognition system recognizes road signs using pattern matching. In the pattern matching, a predetermined road sign to be recognized is prepared as a template, and the road sign is recognized by matching the template with the road sign in the front image.

JP, 2014-153167, A

The sign recognition system proposed in Patent Document 1 recognizes a road sign using a front image captured by a camera. Even if the road signs are of the same type, if the distance between the camera and the road sign is different for each road sign, the height of the road sign and the size of the road sign in the forward image are different for each road sign. In the above sign recognition system, it is necessary to prepare a plurality of road sign templates having different sizes when performing pattern matching. Then, the sign recognition system matches the road sign in the front image with the template while sequentially changing the plurality of templates having different sizes. At this time, matching is performed while changing the height for each template. Since a plurality of templates having different sizes are changed and the height is changed for each template, there is a problem that the processing load in pattern matching increases.

The present invention has been made in view of the above problems, and an object thereof is to provide a sign recognition system that can reduce processing load.

The invention disclosed herein employs the following technical means in order to achieve the above object. It should be noted that the claims and the reference numerals in parentheses in this section indicate the corresponding relationship with the specific means described in the embodiments described later, and do not limit the technical scope of the invention. ..

A first invention for achieving the object is a sign for recognizing a road sign installed on the side of a road by pattern matching using a front image (31) of a vehicle captured by a camera (10). The recognition system is a template storage unit (50) that stores a template of a road sign used in pattern matching, and a height line (51) showing the same height as the side of the road in the front image. By being partitioned, the side area (53) whose length in the height direction becomes shorter toward the vanishing point is parallel to the direction from the camera toward the vanishing point, and the vertical direction is above and below the imaging surface of the camera. By performing a side-view conversion, which is a conversion in which the heights of the side areas are aligned on the virtual imaging plane (44) parallel to the direction, the side areas in the front image are displayed on the left and right sides of the vehicle. The side-view conversion unit (25) that is a side-view image that is an image viewed from the direction, and a matching area that is a partial area in the side-view image and in which a road sign may exist. Then, a matching unit (28) for performing pattern matching using the template and a sign recognition unit (29) for recognizing the road sign based on the result of the pattern matching are provided.

According to the first invention, when the same object is displayed in the front image, the closer to the vanishing point, the shorter the length in the height direction in the image. This means that if the vanishing point can be determined, the height line representing the same height can be determined in the front image. Therefore, in the present invention, a height line that is the same as the line toward the vanishing point and represents the same height is determined. Then, the lateral area is determined using this height line. The side region is a region in which the side of the road is reflected in the front image and is divided by the height line, so that the length in the height direction becomes shorter toward the vanishing point. The side-view conversion unit converts this side area into an image captured on a virtual image pickup plane that is parallel to the direction from the camera to the vanishing point and whose vertical direction is parallel to the vertical direction of the image pickup plane of the camera. .. A plane which is parallel to the direction from the camera to the vanishing point and whose vertical direction is parallel to the vertical direction of the imaging surface of the camera is a plane orthogonal to the lateral direction of the vehicle. Therefore, by the conversion, the side area can be converted into a side view image which is an image viewed from the left and right direction of the vehicle. In addition, the side-view conversion unit converts the lateral region whose length in the height direction becomes shorter toward the vanishing point so that the length in the height direction becomes uniform. Thereby, in the side view image, the reduction ratio in the height direction is the same regardless of the distance from the vehicle. Therefore, since the heights of the road signs reflected in the side areas are also uniform, it is possible to reduce the range in which the height of the template is changed during pattern matching. Since the range in which the height of the template is changed can be reduced, the processing load at the time of pattern matching can be reduced.

A second invention is a sign recognition system for recognizing a road sign installed above a road by pattern matching using a front image (331) of a vehicle imaged by using a camera (10). The template storage unit (350) that stores the template of the road sign used in the above, and the road above is shown in the front image, and is divided by a pair of horizontal length lines (351) that divide the same horizontal length. By doing so, the upper area (353) whose length in the left-right direction becomes shorter toward the vanishing point is parallel to the direction from the camera toward the vanishing point, and the left-right direction is parallel to the left-right direction of the imaging surface of the camera. By performing bottom-view conversion, which is a conversion in which the horizontal length of the upper region is aligned on the virtual imaging surface (344), the upper region in the front image is viewed from below in the vertical direction of the vehicle. A bottom-view conversion unit (330) that makes a bottom-view image that is an image, and a template for a matching area that is a partial area in the bottom-view image where road signs may exist. A matching unit (328) for performing pattern matching using the pattern matching unit and a sign recognition unit (329) for recognizing a road sign based on the result of the pattern matching are provided.

The first invention is an invention that performs side-view conversion, while the second invention performs bottom-view conversion. When the same object is displayed in the front image, the closer it is to the vanishing point, the shorter the length in the road width direction, that is, the left-right direction in the image. This means that if the vanishing point can be determined, a pair of horizontal length lines that divide the same horizontal length can be determined in the front image. Therefore, in the second aspect of the invention, a pair of left-right direction length lines that delimit the same left-right direction length, which are lines toward the vanishing point, are determined. Then, the upper area is determined using the pair of left-right length lines. The upper region is a region in which the upper part of the road is reflected in the front image and is divided by a pair of left and right length lines, so that the length in the left and right direction becomes shorter toward the vanishing point. The bottom-view conversion unit converts the upper region into an image captured on a virtual imaging plane that is parallel to the direction from the camera to the vanishing point and whose left-right direction is parallel to the left-right direction of the imaging plane of the camera. A plane which is parallel to the direction from the camera toward the vanishing point and whose left-right direction is parallel to the left-right direction of the imaging surface of the camera is a plane orthogonal to the vertical direction of the vehicle. Therefore, by the conversion, it is possible to convert the upper region into a bottom view image which is an image looking up from below. In addition, the bottom-view conversion unit converts the upper region whose length in the left-right direction becomes shorter toward the vanishing point so that the length in the left-right direction becomes uniform. As a result, in the bottom view image, the reduction ratios in the left and right directions are the same regardless of the distance from the vehicle. Therefore, the positions of the road signs reflected in the upper area are aligned in the left-right direction, so that the range in which the position of the template in the left-right direction is changed can be reduced during pattern matching. Since the range in which the position of the template in the left-right direction is changed can be reduced, the processing load at the time of pattern matching can be reduced.

It is a block diagram showing a sign recognition system concerning a 1st embodiment. It is a figure which shows the front image which concerns on 1st Embodiment, and the front image after distortion correction. It is a conceptual diagram for demonstrating the distortion correction process which concerns on 1st Embodiment. It is a conceptual diagram for explaining the camera orientation detection method according to the first embodiment. It is a figure which shows the front image and side view image which concern on 1st Embodiment. It is a conceptual diagram for demonstrating the conversion method which equalizes the height of the road sign which concerns on 1st Embodiment. It is a conceptual diagram for demonstrating the conversion method which equalizes the magnitude|size of the road sign which concerns on 1st Embodiment. It is a figure which shows an example of the template of the road sign which concerns on 1st Embodiment. 6 is a flowchart showing a flow of processing in an image processing unit according to the first embodiment. It is a block diagram which shows the sign recognition system which concerns on 2nd Embodiment. It is a flowchart which shows the process which the sign recognition system which concerns on 2nd Embodiment performs. It is a figure explaining that the size of a road sign changes because a driving lane differs. It is a figure which shows that the shape of a road sign becomes an ellipse rising to the right in the 2nd side view conversion. FIG. 8 is a diagram illustrating a template used in Modification 1. It is a block diagram which shows the structure of the sign recognition system which concerns on 3rd Embodiment. It is an example of a front image. It is a figure explaining an upper area. It is a figure explaining a bottom view conversion. It is a figure which illustrates the 2nd bottom view image. It is a figure which illustrates the template of the road sign which exists above a road.

Hereinafter, a plurality of modes for carrying out the invention will be described with reference to the drawings. In each mode, parts corresponding to the matters described in the preceding mode may be assigned the same reference numerals and overlapping description may be omitted. In each mode, when only a part of the configuration is described, the other mode described earlier can be applied to the other part of the configuration.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a sign recognition system 100 according to this embodiment. As shown in FIG. 1, the sign recognition system 100 includes a camera 10 that captures an image in front of a vehicle, an image processing unit 20 that processes an image output from the camera 10, and an angle of view of the camera that is data necessary for image processing. The camera storage unit 30 stores the focal length f, and the template storage unit 50 stores information about the road sign template used for recognizing the road sign. The sign recognition system 100 is mounted on a vehicle. The sign recognition system 100 is connected to a detection sign utilization system (not shown). The sign recognition system 100 may not be mounted on the vehicle and may transfer an image recorded by a drive recorder or the like to the outside of the vehicle through a recording medium or communication, and perform the recognition processing outside the vehicle. A PC or server may be installed outside the vehicle as a system capable of performing recognition processing outside the vehicle.

The detection sign utilization system utilizes the road sign information that is the information of the road sign recognized by the sign recognition system 100, and alerts or warns based on traffic restrictions such as speed limits. Alternatively, driving diagnosis such as speed diagnosis based on traffic regulation such as speed limit is performed.

The camera 10 is installed, for example, near the windshield of a vehicle. The camera 10 images the front of the vehicle and generates image data. Image data obtained by imaging the front of the vehicle is referred to as a front image 31 (see FIG. 2). As the camera 10, a digital camera using a CMOS (Complementary Metal Oxide Semiconductor) image sensor, a CCD (Charge Coupled Device) image sensor, or the like can be used. Further, the drive recorder may be diverted to generate the front image 31. In the following description, the camera 10 is a camera included in the drive recorder. A wide-angle lens of the drive recorder is used to image the front of the vehicle to generate a front image 31.

The image processing unit 20 includes a distortion correction processing unit 21, a lane marking detection unit 22, a vanishing point detection unit 23, a camera orientation detection unit 24, a side view conversion unit 25, a matching region identification unit 27, and a matching unit. 28 and a sign recognition unit 29.

The distortion correction processing unit 21 corrects the distortion of the front image 31 generated by the camera 10. 2A shows the generated front image 31, and FIG. 2B shows the corrected front image 32 which is the front image 31 after distortion correction. In particular, the front image 31 captured by the drive recorder is significantly distorted because it is captured by the wide-angle lens. If the distortion is large, it may affect the recognition of road signs described later. Therefore, the distortion of the front image 31 is corrected. The distortion here is a phenomenon in which an image is drawn closer to the center of the image as the angle of incidence on the camera increases, and is called distortion. In the case of a wide-angle lens, the distortion is a barrel-shaped image that is drawn closer to the vertical center or horizontal center of the image toward the corners of the image. FIG. 3A is an example of the front image 31 captured by the camera 10. The front image 31 is an original image before correction. The distortion in the wide-angle lens includes orthographic projection and equidistant projection. The distortion correction is a process of performing a linear conversion using an expression representing an orthographic projection or an equidistant projection according to the distortion.

However, image distortion cannot be eliminated only by distortion correction. The distortion image 39, which is an image obtained by performing the distortion correction on the front image 31, becomes an image in which a three-dimensional object projected at the edge of the image is pulled toward the edge of the image as shown in FIG. 3B. Become. This is called a volume distortion image. Due to the volume distortion image, the road sign may be deformed depending on the position where the road sign is displayed in the front image 31, which may affect the pattern matching.

In order to correct the volume distortion image, as shown in FIG. 3C, a formula 48 is used to perform a non-linear conversion in which the edge portion of the image is closer to the center of the image. This conversion is called volume distortion image correction. By performing the volume distortion image correction, as shown in FIG. 2(B), the road and the bridge become substantially straight, and the corrected front image 32, which is an image in which the building reflected in the peripheral portion of the image stands straight can do. The distortion correction processing unit 21 corrects the front image 31 into a post-correction front image 32. Note that the following description will be made using the corrected front image 32.

The marking line detection unit 22 detects a marking line 33 (see FIG. 4) that divides a region in which the vehicle can travel. Based on the corrected front image 32, a grayscale edge in the corrected front image 32 is detected by an edge extraction process which is a known image process. The extracted edges are indicated by a set of points (hereinafter referred to as edge points). Next, the lane marking detection unit 22 detects the lane markings 33 by Hough transforming the set of extracted edge points. The division line 33 is a line written on the road surface of the road, and is, for example, a white line.

The vanishing point detection unit 23 extracts the lane markings 33 extending near the center of the corrected front image 32 from the plurality of lane markings 33 detected by the lane marking detection unit 22. Then, a vanishing point 34 which is an intersection of the lane markings 33 is detected. For example, when there are four lane markings 33 as shown in FIG. 2B, the intersection of the lane markings 33 on the left and right of the traveling vehicle is defined as the vanishing point 34. Although the left and right marking lines 33 of the vehicle are used, any of the four marking lines 33 may be used. The direction from the camera 10 to the vanishing point 34 indicates the traveling direction of the vehicle. This is because the vanishing point 34 is extracted from the lane marking 33 in the corrected front image 32, and the intersection of the lane marking 33 indicating the direction in which the road extends is defined as the vanishing point 34. The vanishing point 34 corresponds to the vanishing point described in the claims.

The camera orientation detector 24 detects the orientation of the camera 10. The orientation of the camera 10 is the same as the orientation of the optical axis 40 (shown in FIG. 7) of the camera 10. Since the camera 10 is included in the drive recorder, it is often installed after purchasing the vehicle, and the orientation of the camera 10 can often be changed. Therefore, the orientation of the camera 10 is different for each camera 10.

A method for detecting the camera orientation will be described with reference to FIG. The equation indicating the yaw angle of the camera 10 is shown below as equation 1. Further, the equation showing the tilt angle of the camera 10 is shown below as equation 2.

The yaw angle and tilt angle of the camera 10 detected from the equations 1 and 2 are the mounting angles of the camera 10 and indicate the orientation of the camera 10. The deviation of the orientation of the camera 10 in the left-right direction with respect to the traveling direction of the vehicle is confirmed by the yaw angle, and the deviation of the orientation of the camera 10 in the up-down direction with respect to the traveling direction of the vehicle is confirmed by the tilt angle.

The orientation of the camera 10 can be detected from the position of the vanishing point 34 of the corrected front image 32, the center position 35 of the camera 10, and the focal length of the camera 10. The center position 35 of the camera 10 is the center of the corrected front image 32. That is, the orientation of the camera 10 here specifically refers to the vanishing point 34, and means the distance between the vanishing point 34 and the center of the image captured by the camera 10. The focal length of the camera 10 is set in advance and stored in the camera storage unit 30.

f: focal length Δx: horizontal distance between vanishing point 34 and center position 35 in corrected front image 32 Δy: vertical distance between vanishing point 34 and center position 35 in corrected front image 32 Side-view conversion unit 25 is a second side-view image 36, which will be described later, which is an image of the corrected front image 32 shown in FIG. 5A viewed from the left and right with respect to the traveling direction of the vehicle shown in FIG. 5B. Convert. The second side-view image 36 is an image in which the size and height of the road sign are aligned. The second side-view image 36 is an image viewed from the left and right with respect to the traveling direction of the vehicle, but the direction of the road sign is changed to a direction that can be recognized by the occupants of the vehicle like the corrected front image 32. To be done. The side view conversion performed by the side view conversion unit 25 will be described with reference to FIGS. 6 and 7. The side-view conversion includes a first side-view conversion described below and a second side-view conversion described below.

In FIG. 6, the first side-view conversion for aligning the heights of the road signs in the corrected front image 32 will be described. FIG. 6A shows a scene to the left of the traveling direction of the vehicle. FIG. 6A is not an image captured by the camera 10 but a simplified actual scene. At this time, it is assumed that the road signs 42 and the road signs 43 lined up in the traveling direction have the same height. Further, in FIG. 6A, a grid is drawn at equal intervals in the vertical and horizontal directions for the purpose of explanation. Note that the grid is shown to illustrate the first side-view transform and need not be drawn in the actual image.

Each line of the lattice extending in the traveling direction has the same height. Each line of the lattice extending in the traveling direction is a height line 51. For example, as shown in FIGS. 6(A) and 6(C), the height line 51 at the upper end of the lateral region 53 cut out to obtain the second side view image 36 is the height line 51a. The vehicle-side end point of the height line 51 at the lower end of the side area 53 is an arbitrary position outside the corrected front image 32 with respect to the lane marking 33 detected on the leftmost side, that is, the lane marking 33 detected closest to the road side. Is. For example, on the left side of the corrected front image 32, the position is set to be a certain distance above the leftmost partition line 33. The line connecting this position and the vanishing point 34 becomes the height line 51 at the lower end of the lateral region 53.

The height line 51 a at the upper end of the side area 53 is located above the image at the height line 51 at the lower end of the side area 53. For example, the vehicle-side end point of the height line 51a is located on the left side of the corrected front image 32 at a position above the vehicle-side end point of the height line 51 at the lower end of the lateral region 53 by a certain distance. The line connecting this position and the vanishing point 34 becomes the height line 51 a at the upper end of the lateral region 53.

Each line of the grid perpendicular to the traveling direction indicates the distance of an object such as the road sign 42 or the road sign 43 from the camera 10. Each line of the lattice perpendicular to the traveling direction is a distance line 52. For example, as shown in FIGS. 6(A) and 6(C), the distance line 52 indicating the farthest distance in the lateral region 53 is defined as the distance line 52a.

FIG. 6B shows the corrected front image 32 to which the lateral region 53 depicted in FIG. 6A is added. The side area 53 depicted in FIG. 6(B) shows the side of the road. The lateral region 53 has a trapezoidal shape, and the distance between the pair of legs connecting the upper bottom and the lower bottom becomes narrower toward the vanishing point 34.

In FIG. 6A, the plurality of height lines 51 and the distance lines 52 are vertically and horizontally equidistant, whereas in the lateral region 53 depicted in FIG. 52 is not vertically and horizontally evenly spaced. This is because FIG. 6B is not the image when the road sign is viewed from the left and right direction with respect to the traveling direction of the vehicle but the corrected front image 32.

The plurality of distance lines 52 represent actual equidistant positions on the road. However, the length of the distance line 52 becomes shorter toward the vanishing point 34, and the distance between the distance lines 52 adjacent to each other is narrowed while maintaining the mutual parallelism. This means that the length of the lateral region 53 in the corrected front image 32 in the height direction becomes shorter toward the vanishing point 34.

If the side area 53 is assumed to be a plane, the relationship between the position in the side area 53 and the distance from the vehicle is determined. Therefore, in the side area 53, the positions of the plurality of distance lines 52 that represent the positions at equal intervals can be determined.

All the height lines 51 are straight lines with the vanishing point 34 as an end point. This means that if the vanishing point 34 can be determined, the height line 51 representing the same height can be determined in the corrected front image 32.

Since all the height lines 51 are straight lines with the vanishing point 34 as an end point, the interval between the height lines 51 above and below becomes narrower toward the vanishing point 34. That is, the lateral area 53 is divided by the height line 51 that represents the same height, but the length in the height direction becomes shorter in the image toward the vanishing point 34. However, the actual length in the height direction is the same regardless of the difference in the height direction in the image. The side area 53 corresponds to the side area described in the claims.

The first side-view transformation is a transformation for aligning the lateral regions 53 in the corrected front image 32 depicted in FIG. 6B so that the lengths in the height direction are aligned as shown in FIG. 6C. It was done. As a result, the height lines 51 are evenly spaced, and the road signs 42 and 43 in the side area 53 are also enlarged in the height direction. The image after the first side-view conversion converts the corrected front image 32 including the road signs 42 and 43 into an image viewed from the left and right with respect to the traveling direction of the vehicle, while the direction of the road signs 42 and 43 is changed. Is the image in which the orientation that can be recognized by the occupant of the vehicle is maintained, like the corrected front image 32.

FIG. 6D is an image after the first side-view conversion. This image is referred to as a first side view image 38. The first side-view image 38 is displayed such that the relative height of the object included in the side area 53 becomes the relative height of the actual object. As a result, the heights of the road signs 42 and the road signs 43 in the first side-view image 38 are the same.

However, when the height lines 51 that partition the lateral regions 53 are enlarged so as to be evenly spaced, the size in the left-right direction of FIG. 6D, that is, the size of the vehicle in the front-rear direction is closer to the vanishing point 34. The bigger it gets. Therefore, in FIG. 6D, the size of the road sign 42 and the size of the road sign 43 differ in the traveling direction of the vehicle.

The side view conversion unit 25 converts the corrected front image 32 into the first side view image 38 by the first side view conversion. Further, the first side-view conversion corresponds to the first side-view conversion described in the claims. Although the road sign is on the left side of the vehicle, the side area 53 on the right side of the vanishing point 34 may be enlarged because the road sign is on the right side.

In FIG. 7, the first side-view conversion and the second side-view conversion in which the sizes of the road signs 42 and 43 having different sizes due to the first side-view conversion are made uniform will be described.

When the road sign 42 and the road sign 43 exist in front of the vehicle, the road signs 42 and 43 are photographed by the camera 10. The corrected front image 32 is an image obtained by correcting the distortion of the original image displayed on the image pickup surface 41 of the road sign 42 and the road sign 43 existing in front of the vehicle through the condenser lens 45 of the camera 10.

The image pickup surface 41 shown in FIG. 7 exists behind the vehicle, but this is because FIG. 7 is an enlarged view of the relationship between the condenser lens 45 of the camera 10 and the image pickup surface 41. It is inside the camera 10 mounted on the.

When the road sign 42 and the road sign 43 have the same size, the corrected front image 32 obtained by correcting the distortion of the original image displayed on the imaging surface 41 is close to the original image as shown in the geometrical relationship shown in FIG. The large road sign 42 is shown in a large size, and the distant road sign 43 is shown in a small size. The lateral area 53 is also displayed on the imaging surface 41. The imaging surface 41 corresponds to the imaging surface described in the claims.

The side view conversion unit 25 projects the side area 53 in the corrected front image 32 on the virtual imaging surface 44. The virtual imaging surface 44 is a virtual plane that is parallel to the direction from the camera 10 toward the vanishing point 34, and the vertical direction is parallel to the vertical direction of the imaging surface 41. The parallelism here does not have to be strictly parallel, and a deviation may occur. For example, the virtual imaging surface 44 may be offset by 5 degrees with respect to the direction from the camera 10 to the vanishing point 34.

Further, the virtual imaging surface 44 parallel to the direction from the camera 10 to the vanishing point 34 is geometrically set based on the relationship between the vanishing point 34 and the optical axis 40 of the camera 10. The relationship between the vanishing point 34 and the optical axis 40 of the camera 10 is the horizontal distance between the vanishing point 34 and the central position 35 detected by the camera orientation detecting section 24, and the vertical direction between the vanishing point 34 and the central position 35. It is calculated from the distance and the value of.

When the side areas 53 are displayed on the virtual imaging surface 44, a process of enlarging the height lines 51 that partition the side areas 53 so as to be equidistant is performed. This processing is the first side-view conversion described above. The virtual imaging surface 44 corresponds to the virtual imaging surface described in the claims.

In other words, the virtual imaging surface 44 is a virtual plane that exists on the side of the vehicle and is parallel to the vertical direction of the vehicle. Therefore, when the side area 53 is projected on the virtual imaging surface 44, an image of the side area 53 viewed from the left and right direction of the vehicle is obtained.

The side-view conversion unit 25 reduces the first side-view image 38 in order to make the size of the road signs 42 and 43 in the first side-view image 38 uniform by using the conversion formula 46. I do. The correction process for reducing the first side-view image 38 is referred to as second side-view conversion. The second side-view conversion is performed after the first side-view conversion. The conversion formula 46 is a curve as shown in FIG. 7. The shape of this curve can be geometrically determined, as shown in FIG. The conversion formula 46 is actually the same size road signs 42 and 43, but the distance from the camera 10 is different. This is a formula that returns road signs of different sizes to the same size.

Therefore, the conversion formula 46 makes the reduction rate higher as it exists farther from the camera 10 on the virtual imaging surface 44. The road sign 42 and the road sign 43 having the same size are displayed on the corrected image pickup surface 47. The side view conversion unit 25 converts the first side view image 38 into the second side view image 36.

The side-view conversion unit 25 performs the first side-view conversion to display the corrected front image 32 illustrated in FIG. 5A on the first side where the heights of the road signs illustrated in FIG. 6D are uniform. The view image 38 is converted. The side-view conversion unit 25 performs the second side-view conversion after the first side-view conversion, so that the first side-view image 38 illustrated in FIG. 6D is illustrated in FIG. 5B. The size and height of the road sign are converted into the second side-view image 36. Although only the second side-view conversion is performed after the first side-view conversion, only the first side-view conversion may be performed.

The matching area specifying unit 27 specifies the matching area in the second side view image 36 from the second side view image 36 converted by the side view converting unit 25. The matching region is a region in the second side view image 36 where pattern matching is performed using the template. In the second side view image 36, since the heights of the road signs are the same, the matching area can be narrowed down to a partial height area of the second side view image 36. Further, since it is not necessary to recognize a road sign that is too far from the vehicle, the matching area is narrowed down to a partial area of the second side-view image 36 also in the left-right direction.

The matching area is obtained based on the angle of view of the camera 10 and the mounting angle of the camera 10. The angle of view of the camera 10 defines the range in which the road sign should be recognized in the left-right direction of the second side-view image 36. The larger the angle of view, the smaller the proportion of the size of the distance area in which the road sign should be recognized in the second side-view image 36. Therefore, the wider the angle of view, the closer the edges of the matching region in the left-right direction are toward the center of the image. On the contrary, when the angle of view of the camera 10 is narrow, the road sign may exist near the edge of the second side view image 36. Therefore, the matching area specifying unit 27 changes the horizontal range of the matching area according to the angle of view of the camera 10.
The above description is intended to alert the driver of the vehicle, and the road sign to be alerted is determined by the distance from the vehicle. Therefore, when the angle of view is wide, a road sign closer to the vehicle is imaged, but it may not be necessary to recognize the road sign closer to the vehicle depending on the distance from the vehicle. In this case, the wider the angle of view, the closer the edges of the matching region in the left-right direction are toward the center of the image.
However, if the purpose is not to alert the driver of the vehicle and it is also necessary to recognize a road sign closer to the vehicle, it is not necessary to center the left and right ends of the matching area.

Further, the vertical position of the road sign in the second side view image 36 is obtained from the tilt angle that is one of the mounting angles of the camera 10. As the tilt angle of the camera 10 tilts toward the ground, the road sign exists above the second side view image 36. As the tilt angle of the camera 10 tilts in the direction opposite to the ground direction, the road sign exists below the second side view image 36. Therefore, the matching area specifying unit 27 changes the vertical position of the matching area according to the tilt angle of the camera 10.
Also, even if the yaw angle, which is one of the camera mounting angles, faces the road side or the road side, the matching region rarely changes. A virtual imaging plane 44 is geometrically set in the direction from the camera 10 to the vanishing point 34, and side-view conversion is performed. That is, the side-view transformation is based on the vanishing point 34. Therefore, even when the yaw angle is facing the side of the road, the position of the road sign in the second side-view image 36 rarely changes depending on the yaw angle. As a result, the matching area specifying unit 27 does not need to change the position of the matching area according to the yaw angle of the camera 10.

Although the matching area in the second side-view image 36 is specified, the matching area in the first side-view image 38 may be specified. Although the matching area is changed according to the angle of view and the tilt angle of the camera 10, the matching area may be changed according to any one of the angle of view and the tilt angle. Further, although the matching area is specified using the angle of view of the camera 10 and the tilt angle which is the mounting angle of the camera 10, an area having a large number of detections is specified during pattern matching, and the area is set as the matching area. May be. The first few times after mounting the camera 10 may be used to identify the matching area. The matching area specifying unit 27 corresponds to the matching area specifying unit described in the claims. Further, this matching area corresponds to the matching area described in the claims.

The matching unit 28 performs pattern matching on the matching area detected by the matching area specifying unit 27 using the template information stored in the template storage unit 50. The template information is a template of road signs installed on the road as shown in FIG. In the second side-view image 36 for which pattern matching is performed in the present embodiment, the size of the road sign is uniform, as described above. Therefore, in pattern matching, once the scaling rate of the template can be determined, it is not necessary to change the scaling rate of the template thereafter. It is also possible to prepare a template that does not need to be scaled up from the beginning. The road sign template is not limited to the one shown in FIG. 8, and various road signs can be used. The matching unit 28 corresponds to the matching unit described in the claims.

The sign recognition unit 29 recognizes the road sign having a high degree of matching as the road sign shown in the front image 31 by the pattern matching performed by the matching unit 28. From the recognized road sign, the type of the road sign and the regulation information are specified. The regulation information is the display information of the regulation indicated by the road sign, and is, for example, the value of the speed limit. The sign recognition unit 29 outputs the specified road sign information to a detection sign utilization system (not shown). The sign recognition unit 29 corresponds to the sign recognition unit described in the claims.

(flowchart)
Hereinafter, the road sign recognition process performed by the image processing unit 20 will be described with reference to the flowchart shown in FIG. 9. The image processing unit 20 always performs this road sign recognition process while the ignition switch of the vehicle is on.

First, in step S1, the front image 31 of the vehicle captured by the camera 10 is input, and the process proceeds to step S2.

In step S2, the distortion correction processing unit 21 performs a process of correcting the distortion of the front image 31 input in S1 to obtain a corrected front image 32, and the process proceeds to step S3.

In step S3, the lane marking detection unit 22 detects a plurality of lane markings 33 from the corrected front image 32 obtained in S2, and the process proceeds to step S4.

In step S4, the vanishing point detection unit 23 detects vanishing points 34 from the plurality of lane markings 33 detected in S3, and the process proceeds to step S5. If the vanishing point 34 has already been determined, steps S3 and S4 may be omitted. If the plurality of lane markings 33 cannot be detected before the vanishing point 34 is detected, the process returns to step S1 without proceeding to step S5.

In step S5, the camera orientation detection unit 24 detects the vanishing point 34 detected in S4, the center position 35 of the camera 10 and the focal length f of the camera 10, and the orientation of the camera 10 from Expressions 1 and 2, Control goes to step S6.

In step S6, the side-view conversion unit 25 determines the lateral region 53 from the corrected front image 32 obtained in step S2 based on the vanishing point 34 detected in step S4, and with respect to the lateral region 53. Perform side view conversion. In the side-view conversion, the first side-view conversion is performed, and after the first side-view conversion, the second side-view conversion is performed. By performing the side view conversion, the lateral region 53 of the corrected front image 32 is converted into the second side view image 36, and the process proceeds to step S7. Note that the side-view conversion unit 25 may perform only the first side-view conversion. When only the first side view conversion is performed, the corrected front image 32 is converted into the first side view image 38, and the process proceeds to step S7.

In step S7, the matching area specifying unit 27 determines whether the second side-view image 36 in the second side-view image 36 is obtained from the second side-view image 36 obtained in S6, the tilt angle of the camera 10 and the angle of view of the camera 10 detected in S5. The matching area is specified, and the process proceeds to step S8. If only the first side-view conversion is performed in step S6, the matching area in the first side-view image 38 is specified, and the process proceeds to step S8. Further, once the matching area is specified, step S7 can be omitted.

In step S8, the matching unit 28 performs pattern matching on the matching area specified in S7, and the process proceeds to step S9.

In step S9, the sign recognition unit 29 identifies the type of the road sign and the regulation information based on the result of the pattern matching, and ends this flow.

As described above, in the present embodiment, the image processing called side view conversion is performed to reduce the processing load at the time of pattern matching. In the first side-view transformation, the road side is reflected in the corrected front image 32, and the road is divided by the height line 51 representing the same height, so that the length in the height direction increases toward the vanishing point 34. The lateral area 53 in which the length becomes shorter is determined. Then, the side area 53 is projected on a virtual imaging surface 44 which is perpendicular to the imaging surface 41 of the camera 10 and whose vertical direction is parallel to the vertical direction of the imaging surface 41. As a result, the first side-view image 38, which is an image of the lateral region 53 viewed from the left-right direction of the vehicle, is obtained. By setting the first side-view image 38, the relative height of the object included in the side area 53 is also projected so as to be the relative height of the actual object. That is, by the first side-view conversion, the image is viewed from the left and right direction of the vehicle, and is converted into the first side-view image 38 in which the heights of the road signs in the side area 53 are uniform.

Since the heights of the road signs are uniform, the matching area can be limited to a partial area of the second side-view image 36 during pattern matching. Therefore, the area in which the height is changed for one template becomes narrow. As a result, the processing load in pattern matching can be reduced.

Further, according to this embodiment, after the first side-view conversion, the second side-view conversion is performed. In the second side-view conversion, the further the distance from the camera 10 to the first side-view image 38, the higher the reduction ratio in the vehicle front-rear direction, and the size of the first side-view image 38 is reduced. As a result, the first side-view image 38 is converted into the second side-view image 36 that is an image in which the heights of the road signs and the sizes of the road signs in the first side-view image 38 are uniform. Since the height and size of the road signs are the same, the number of templates with different sizes that should be prepared for the same type of road sign is reduced (for example, one) or the template is expanded during pattern matching. There is no need to change the reduction ratio. Therefore, the processing load in pattern matching can be further reduced.

Furthermore, the fact that the number of templates used in pattern matching and having different sizes can be reduced has the following advantage. The fact that the number of templates with different sizes used in pattern matching can be reduced means that it is not necessary to perform matching using a template with a small size during pattern matching. If the size of the template is small, it is difficult to determine whether or not the shapes match each other, and thus there is a high possibility that an object that is not a road sign will be erroneously recognized as a road sign. On the other hand, in this embodiment, since it is not necessary to perform matching using a template having a small size, it is possible to improve the recognition accuracy of the road sign.

(Second embodiment)
A second embodiment of the present invention will be described based on the drawings. FIG. 10 is a block diagram showing the sign recognition system 200 according to this embodiment. In the second embodiment, in addition to the configuration of the sign recognition system 100 of the first embodiment, a vehicle position measuring unit 60 that performs vehicle position positioning and travel road identification, a GPS 61, a vehicle speed sensor 62, and a gyro sensor 63. And map data 64. The image processing unit 70 of the second embodiment further includes a traveling lane detection unit 71 and a template selection unit 72 in addition to the configuration of the image processing unit 20 of the first embodiment.

The main difference between the second embodiment and the first embodiment will be described. In the first embodiment, the road sign is specified by pattern matching the second side-view image 36 converted by the side-view conversion with one template. In the second side-view image 36, even if the position of the road sign changes in the front-rear direction of the vehicle, the road sign is displayed in the same size.

However, the distance from the lane in which the vehicle travels to the side where the road sign is provided differs depending on the lane in which the vehicle travels from the roadside. Then, the size of the road sign in the corrected front image 32 changes in accordance with the change in the lateral distance between the lane in which the vehicle is traveling and the road sign. In view of this, in the second embodiment, the template storage unit 50 stores a plurality of types of road lanes of different sizes for one type of road sign depending on the number of lanes the vehicle is traveling from the roadside. Remember the template.

Then, the template storage unit 50 selects a template having a size that is determined according to the number of the driving lane from which the vehicle is traveling. The road sign is specified by pattern matching the second side view image 36 and the selected template.

The vehicle position measuring unit 60 measures the position of the vehicle by the GPS 61. Further, the vehicle position measuring unit 60 measures the position of the vehicle and specifies the traveling road by the known dead reckoning using the vehicle speed sensor 62 and the gyro sensor 63 or the map matching using the map data 64. The vehicle position measuring unit 60 always measures the position of the vehicle and specifies the traveling road while the ignition switch of the vehicle is ON.

The installation position of the road sign detected by the sign recognition system 200 can be specified based on the vehicle position measurement unit 60. This is because the distance between the vehicle and the road sign is known from the position where the road sign is present in the second side view image 36, and the current position of the vehicle is known from the measurement result of the vehicle position measuring unit 60. Further, it is preferable that the vehicle position measuring unit 60 can identify the number of the road on which the vehicle is traveling from the road side. For example, if the current position is specified with high accuracy and the map data 64 includes information about the lane, it is possible to specify on which road the vehicle is traveling from the roadside.

A detection sign utilization system (not shown) outputs the road sign detected by the sign recognition system 200 to which information on the installation position is added. As a result, the detection sign utilization system generates a road map and a road sign database and extracts a change point. The change point is a point where the information on the installation position of the road sign included in the map data is different from the information on the installation position of the road sign detected by the sign recognition system 200. By extracting the change points, the map data can be updated according to the latest road conditions. Further, the detection sign utilization system outputs the road sign detected by the sign recognition system 200 to which the information of the traveling road is added. Thereby, the detection sign utilization system can be utilized for automatic driving such as controlling the vehicle speed according to the speed regulation.

The traveling lane detection unit 71 detects which traveling lane the vehicle is traveling in from the road side (hereinafter, traveling lane position). The traveling lane position detection processing is performed by determining the number of traveling lanes based on, for example, how many lane markings 33 detected by the lane marking detecting unit 22 are present in the corrected front image 32. For example, when the number of the lane markings 33 is 5, it is determined that there are two driving lanes on each side. When there are a plurality of lanes having the same traveling direction, the left lane of the plurality of lanes is changed to the first lane, and the lane numbers are added from the left to the right.

The detection of the traveling lane in which the vehicle exists with respect to the number of lanes is performed based on the number of lane markings 33 on the left and right of the vehicle. For example, if there is one lane marking 33 on the left side of the vehicle, it is determined that there is a vehicle on the first lane. If there are two lane markings 33 on the left side of the vehicle, it is determined that there is a vehicle on the second lane. The traveling lane detector 71 corresponds to the traveling lane detector in the claims.

When the vehicle position measuring unit 60 measures the traveling lane position in addition to detecting the traveling lane position based on the lane marking 33, the traveling lane detecting unit 71 causes the traveling lane measured by the vehicle position measuring unit 60. You can also get the position.

The template selection unit 72 selects the template of the road sign used in the pattern matching performed by the matching unit 28. A plurality of templates having the same type of road signs but different sizes are stored in the template storage unit 50. Then, the template selection unit 72 selects the size of the template of the road sign used in the pattern matching based on the traveling lane position detected by the traveling lane detection unit 71.

As the lane position is farther from the roadside, the template of the road sign used in the pattern matching becomes smaller. For example, as shown in FIG. 12(A), the vehicle is traveling in the first lane of a road having two lanes, and as shown in FIG. 12(B), the vehicle is traveling in the second lane. Compare with the case. 12(A) and 12(B) conceptually show the side view image, the road signs are provided so as to face the road.

Since the distance from the roadside to the vehicle is short when the vehicle is traveling in the first lane, the road signs in the second side view image 36 are shown in FIG. 12(D) as shown in FIG. 12(C). , Is displayed larger than the road sign when the vehicle is traveling in the second lane.

The template selecting unit 72 selects the first lane template when the vehicle is traveling in the first lane. As shown in FIG. 12(E), the first lane template is larger than the second lane template. The first lane template is a template having a size that can match the size of the road sign in the second side-view image 36 when the vehicle is traveling in the first lane. The size of the road sign in the template for each lane is preset based on the distance from the roadside to each lane. The template selection unit 72 corresponds to the template selection unit described in the claims.

(flowchart)
Hereinafter, the processing executed by the sign recognition system 200 according to the second embodiment will be described using the flowchart shown in FIG. 11. The sign recognition system 200 always performs this road sign recognition process while the vehicle ignition switch is ON.

In the second embodiment, after the process of step S6, the process proceeds to step S6A. In step S6A, the traveling lane detection unit 71 detects the traveling lane position based on the lane marking 33 detected in S3, and the process proceeds to step S6B. Note that steps S6A and 6B may be performed in parallel with step S5 after the step S4.

In step S6B, the template selecting unit 72 selects a template of a road sign having a size determined according to the traveling lane position detected in S6A, and the process proceeds to step S7. In the second embodiment, after the process of step S9, the process proceeds to step S10.

In step S10, the vehicle position measurement unit 60 measures the vehicle position measured by the vehicle position measurement unit 60 at the position where the road sign identified in step S9 is installed, and the road sign in the second side view image 36. Specify from the position. Further, the installation position and the map data 64 are collated to identify the traveling road on which the road sign is installed. The information indicating the installation position of these road signs and the traveling road on which the road signs are installed is output to the detection sign utilization system, and this flow ends.

As described above, according to the second embodiment, the template having the size determined according to the traveling lane position is selected. Since the size of the road sign in the second side-view image 36 changes depending on the traveling lane position, it is possible to select a template having an appropriate size by selecting a template having a size determined according to the traveling lane position. it can. Therefore, in pattern matching, it becomes less necessary to sequentially change templates having different sizes. This makes it possible to reduce the processing load in pattern matching as compared with the first embodiment.

(Third Embodiment)
The sign recognition systems 100 and 200 described in the first and second embodiments recognize the road signs provided on the side of the road by performing the side-view conversion on the front image 31. In the side view conversion, the side area 53 of the front image 31 is displayed on a virtual imaging surface 44 that is spatially parallel to the side area 53.

The front image 331 (see FIG. 16) also shows a road sign existing above the road. In the front image 331, a region where a road sign above the road is reflected (hereinafter, the upper region 353 (see FIG. 17)) is a virtual imaging surface 344 (see FIG. 18) spatially parallel to the upper region 353. It can be projected. This is a conversion in which the viewpoint in the side-view conversion is a viewpoint from the vehicle to the side of the road, while the viewpoint is replaced with a viewpoint from the vehicle to the upper side of the road. The conversion of the front image 331 into the image of the viewpoint from the vehicle toward the upper side of the road is hereinafter referred to as bottom view conversion. In the third embodiment, this bottom view conversion will be described.

FIG. 15 shows a configuration diagram of the sign recognition system 300 of the third embodiment. The sign recognition system 300 is different from the sign recognition system 100 of the first embodiment in the configuration of the image processing unit 320 and the template storage unit 350.

The template storage unit 350 stores the template of the road sign installed above the road in addition to the template included in the template storage unit 50 of the first embodiment. The road signs installed above the road include, for example, guide signs.

The image processing unit 320 includes a bottom view conversion unit 330. In addition to having the same function as the matching area specifying unit 27 of the first embodiment, the matching area specifying unit 327 also specifies the matching area from the bottom view image converted by the bottom view converting unit 330. The bottom view image indicates an image obtained by the first bottom view conversion described below and a second bottom view image 336 (see FIG. 19) described below. The matching area specifying unit 327 specifies the matching area based on the tilt angle of the camera 10.

The bottom view image is an image in which the positions of the road signs in the horizontal direction of the image are aligned regardless of the distance between the road sign and the vehicle. However, the position where the road signs are aligned in the left-right direction of the bottom view image changes depending on the yaw angle of the camera 10. Therefore, the matching area for the bottom view image is specified based on the yaw angle of the camera 10 detected by the camera orientation detection unit 24 and a predetermined relationship for determining the matching area from the yaw angle. In addition to this, the matching area may be further limited based on the angle of view, as in the matching area specifying unit 27 of the first embodiment.

The matching unit 328 has the same function as the matching unit 28 of the first embodiment, and in addition, is stored in the template storage unit 350 for the matching region determined by the matching region specifying unit 327 for the bottom view image. Moreover, pattern matching is performed using a template for the road sign above the road.

The sign recognition unit 329 recognizes the road sign having a high degree of matching as the road sign shown in the front image 331 by the pattern matching performed by the matching unit 328.

The bottom-view conversion unit 330 performs the first bottom-view conversion and the second side-view conversion in the same manner as the side-view conversion unit 25 performs the two side-view conversions of the first side-view conversion and the second side-view conversion. Two bottom view transformations, bottom view transformation, are performed.

The region in which the first bottom-view conversion is performed is, for example, the upper region 353 shown in FIG. 17 of the front image 331 shown in FIG. 16. In the forward image 331 shown in FIG. 16, road signs 342 and 343 are shown above the road. The upper area 353 is an area in which the road signs 342 and 343 are included.

As shown in FIG. 17, the upper area 353 is an area defined by a pair of left-right direction length lines 351. The pair of left-right length lines 351 overlap each other at the vanishing point 34. Therefore, the plurality of distance lines 352 drawn so as to intersect the pair of left-right direction length lines 351 have the same distance in the corresponding real space even if the distances in the image are different. It should be noted that the distance on the left-right direction length line 351 from the vanishing point 34 is the same at one end and the other end of each distance line 352.

The upper region 353 is defined by the pair of left-right direction length lines 351 that are drawn toward the vanishing point 34. Therefore, in the front image 331, the length in the left-right direction becomes shorter toward the vanishing point 34. .. However, as described above, the plurality of distance lines 352 drawn so as to intersect the pair of left-right direction length lines 351 have the same distance in the corresponding real space.

FIG. 18 is a diagram illustrating the bottom view conversion. In FIG. 18, a virtual imaging surface 344 is shown. Comparing FIG. 18 and FIG. 7, it can be seen that the drawings are the same except that the orientation of the vehicle is different. From this, it can be understood that the bottom view conversion can be performed based on the same idea as the side view conversion described in the first embodiment. The virtual imaging surface 344 shown in FIG. 18 is perpendicular to the imaging surface 41 of the camera, and the left-right direction is parallel to the left-right direction of the imaging surface 41 of the camera 10.

The first bottom-view conversion is a conversion in which the upper region 353 in the front image 331 is projected on the virtual imaging surface 344. In addition, the first bottom-view conversion is a conversion that makes the lengths in the left-right direction uniform.

In the state of being displayed on the virtual imaging surface 344, as shown in FIG. 18, the road sign 342 and the road sign 343 have different sizes in the vehicle traveling direction due to the difference in the distance from the camera 10. In the second bottom-view conversion, the image obtained by the first bottom-view conversion is converted using the conversion formula 346. Similar to the conversion formula 46 in the side-view conversion, the conversion formula 346 is for aligning road signs having different sizes in the traveling direction due to the difference in the distance from the camera 10 to the same size. It is a conversion formula.

FIG. 19 illustrates a second bottom view image 336 which is an image after the second bottom view conversion. In the second bottom view image 336 shown in FIG. 19, the positions of the road signs in the left-right direction and the sizes of the road signs are aligned. Therefore, the processing load of pattern matching can be reduced. Since the road signs have the same size, the reduction ratios of the road signs in the front-rear direction are also the same.

By the way, in the second bottom view image 336 shown in FIG. 19, the road signs are trapezoidal. The reason for this is that the road signs existing above the road are often larger than the road signs provided on the side of the road and also exist at the center of the road. By the first bottom-view conversion, the grid formed by the horizontal length lines 351 and the distance lines 352 of the upper region 353 shown in FIG. 17 is converted into equal left and right intervals. As a result, the road sign is stretched not only in the horizontal direction but also in the vertical direction with respect to the traveling direction, so that the road sign shown in the second bottom view image 336 becomes a trapezoid. The conversion formula 346 corrects that the size projected on the virtual imaging surface 344 changes due to the difference in the position in the front-rear direction of the road. However, as can be seen from FIG. 18, the vertical size of the road sign also affects the size projected on the virtual imaging surface 344. The conversion formula 346 cannot correct the difference in the size of the road sign caused by the vertical size of the road sign. As a result, as shown in FIG. 19, in the second bottom view image 336, although the upper and lower sides of the road sign are actually the same length, the upper side of the image of the road sign is the lower side of the image. It is shorter than the side of. In consideration of this characteristic, it is preferable that the template of the road sign existing above the road also has a trapezoidal shape as shown in FIG.

Although the preferred embodiments of the invention have been described above, the invention is not limited to the above-described embodiments and can be modified in various ways as illustrated below. Not only the combination of the parts clearly indicating that the respective embodiments can be specifically combined, but also the combination of the embodiments may be partially combined even if not explicitly stated, unless the combination causes any trouble. Is also possible.

(Modification 1)
In FIG. 6D of the first embodiment, the road signs 42 and 43 after the first side-view conversion are shown in an elliptical shape whose major axis is horizontal. Strictly speaking, however, the road signs 42 and 43 after the first side-view conversion become ellipses having an inclination that the major axis goes up toward the right side, as shown in FIG. The reason is that, as shown in FIG. 6C, the road signs 42 and 43 are stretched so as to be pulled upward in the first side-view conversion. As can be seen from FIG. 6C, in the first side-view transformation, the portion of the lateral region 53 above the vanishing point 34 is stretched upward while being pulled below the vanishing point 34. The part at is stretched so that it is pulled downward. Therefore, if the road signs 42 and 43 are below the vanishing point 34, the road signs 42 and 43 are stretched so as to be pulled downward.

Therefore, in the first modification, if it can be estimated that the road sign is above the vanishing point 34, the template is transformed into an ellipse whose major axis extends from the lower left to the upper right (hereinafter, an ellipse that rises to the right) and the pattern. Match. FIG. 14 exemplifies a template obtained by performing this modification.

On the other hand, if it can be estimated that the road sign is below the vanishing point 34, the template is transformed into an ellipse whose major axis extends from the upper left to the lower right (hereinafter, an ellipse that descends to the right) and pattern matching is performed. ..

In FIG. 6B, as can be seen from the fact that the two road signs 42 and 43 are both above the vanishing point 34, the road sign is above the vanishing point 34 in one lateral region 53. It does not exist on both sides. Whether the road sign is above or below the vanishing point 34 depends on the height of the vanishing point 34 in the corrected front image 32, and this height changes depending on the tilt angle. Of course, the height of the road sign from the road surface is also affected, but since the height of the road sign from the road surface is known, it is set in advance. In addition, the installation height of the camera 10 also affects. This is because whether the road sign is above or below the vanishing point 34 depends on the difference between the height of the road sign and the installation height of the camera 10. Therefore, the installation height of the camera 10 is also preferably set in advance.

The tilt angle is detected by the camera orientation detection unit 24. Based on this tilt angle and the difference between the height of the road sign and the installation height of the camera 10, it can be estimated whether the road sign is above or below the vanishing point 34. Then, the template is transformed into an elliptical shape that is determined depending on whether the road sign is above or below the vanishing point 34. In this modification, the orientation of the major axis and the ratio of the major axis to the minor axis may both be constant values set in advance.

If the installation height of the camera 10 is not set, it is sufficient to use both the template having an ellipse that rises to the right and the template that has an ellipse that descends to the right. It can be considered that the installation height of the camera 10 does not change once it is installed in the vehicle. Therefore, after the second time, it is sufficient to perform the deformation on the side where the matching can be performed accurately. From the second time onward, the template is deformed according to the result of estimating whether the road sign is above or below the vanishing point 34.

In the first modification, it can be estimated that the road sign is located above the vanishing point 34 or is located below the circular template that is an ellipse that rises to the right and one that is an ellipse that descends to the right. The template is transformed according to whether it can be estimated. Therefore, the matching accuracy is further improved.

Note that instead of transforming the circular template into an ellipse that rises to the right or an ellipse that descends to the right, a template that is transformed into an ellipse that rises to the right and a template that is transformed into an ellipse that descends to the right are stored in advance. You can leave it. In this case, a template that is determined according to whether it can be estimated that the road sign is above the vanishing point 34 or below the vanishing point 34 is selected.

10: Camera 23: Vanishing point detection unit 24: Camera orientation detection unit 25: Side view conversion unit 27: Matching area identification unit 28: Matching unit 29: Sign recognition unit 31: Front image 32: Front image after correction 33: Section line 34: Vanishing point 36: Second side-view image 38: First side-view image 41: Imaging surface 42: Road sign 43: Road sign 44: Virtual imaging surface 50: Template storage unit 51: Height line 52: Distance line 53: Side area 71: Driving lane detection unit 72: Template selection unit 328: Matching unit 329: Sign recognition unit 330: Bottom view conversion unit 331: Front image 344: Virtual imaging surface 350: Template storage unit 351: Horizontal length line 353: upper area

Claims (7)

A sign recognition system for recognizing a road sign installed on a side of a road by pattern matching using a front image (31) of a vehicle captured by a camera (10),
A template storage unit (50) for storing a template of the road sign used in the pattern matching,
In the front image, the side of the road is reflected and is divided by the height line (51) representing the same height, so that the side area (53) in which the length in the height direction becomes shorter toward the vanishing point. ) Is parallel to the direction from the camera to the vanishing point, and the height of the lateral region is set on a virtual imaging surface (44) whose vertical direction is parallel to the vertical direction of the imaging surface (41) of the camera. By performing side-view conversion, which is a conversion for displaying so that the lengths in the vertical direction are uniform, a side-view conversion is performed in which the side region in the front image is a side-view image that is an image viewed from the left and right direction of the vehicle. Part (25),
A matching unit (28) that performs the pattern matching using the template, with respect to a matching region that is a region in which the road sign may be present, which is a partial region in the side view image;
A sign recognition system (29) for recognizing the road sign based on a result of the pattern matching. The side-view conversion unit performs conversion for aligning the reduction ratios in the front-rear direction of the vehicle, in addition to the conversion for displaying the side regions on the virtual imaging surface so that the lengths in the height direction are aligned. The sign recognition system according to claim 1, wherein the side-view image is generated. The template storage unit has, as the template for the same type of road signs, different sizes depending on the number of the traveling lane on which the vehicle is traveling from the roadside, and the smaller the distance from the roadside is, the smaller the traveling lane is. I remember multiple templates,
A travel lane detection unit (71) for detecting the position of the travel lane in which the vehicle is located, which is the travel lane from the road side;
A template selecting unit (72) for selecting the template used in the pattern matching performed in the matching unit,
The sign recognition system according to claim 1, wherein the template selection unit selects the template that is determined based on the position of the traveling lane detected by the traveling lane detection unit. A camera orientation detection unit (24) that detects the mounting angle of the camera based on the vanishing point, the center position that is the center of the front image, and the focal length of the camera;
The sign recognition system according to any one of claims 1 to 3, further comprising a matching area specifying unit (27) that specifies the matching area based on the mounting angle of the camera. When it can be estimated that the road sign is above the vanishing point, the matching unit uses the template obtained by transforming the circular template into an ellipse rising upward, while the road sign is above the vanishing point. The sign recognition system according to any one of claims 1 to 4, which uses the template obtained by transforming the circular template into a downward-sloping ellipse when it can be estimated that the template is also on the lower side. A sign recognition system for recognizing a road sign installed above a road by pattern matching using a front image (331) of a vehicle captured by a camera (10),
A template storage unit (350) for storing a template of the road sign used in the pattern matching,
In the front image, the upper part of the road is shown and is divided by a pair of left-right length lines (351) that divide the same left-right length, so that the left-right length becomes shorter toward the vanishing point. The upper area (353) is arranged on a virtual imaging plane (344) parallel to the direction from the camera to the vanishing point and whose left-right direction is parallel to the left-right direction of the imaging plane of the camera. By performing bottom-view conversion, which is a conversion for displaying so that the lengths in the directions are aligned, the bottom-view image is a bottom-view image that is an image of the upper region in the front image viewed from the lower side in the vertical direction of the vehicle. A conversion unit (330),
A matching unit (328) that performs the pattern matching using the template on a matching area that is a partial area in the bottom view image and in which the road sign may exist,
A sign recognition system that includes a sign recognition unit (329) that recognizes the road sign based on the result of the pattern matching. The bottom-view conversion unit performs, in addition to the conversion of displaying the upper region on the virtual imaging surface so that the lengths in the left-right direction are uniform, the bottom-view conversion unit performs a conversion that uniformizes a reduction ratio in the front-rear direction of the vehicle. The sign recognition system according to claim 6, which generates a view image.