EP3053097A1

EP3053097A1 - Method and apparatus for identifying road signs

Info

Publication number: EP3053097A1
Application number: EP14777257.8A
Authority: EP
Inventors: Steen Kristensen
Original assignee: Conti Temic Microelectronic GmbH
Current assignee: Conti Temic Microelectronic GmbH
Priority date: 2013-10-01
Filing date: 2014-09-04
Publication date: 2016-08-10
Also published as: DE112014002543A5; US20170017849A1; JP2016538665A; DE102013219909A1; US9697430B2; JP6445024B2; WO2015048954A1

Abstract

The invention relates to a method and an apparatus for identifying a road sign (Hz, Zz) by means of a camera from a travelling vehicle. The method has the following steps: – at least one image of vehicle surroundings is recorded using the camera, – the presence of at least one road sign (Hz, Zz) in the at least one recorded image of the vehicle surroundings is ascertained, – motion blur in the image region that contains the existent road sign (Hz, Zz) is ascertained (also estimated), – the motion blur in this image region is subtracted out, resulting in a sharpened image region, and – the road sign (Hz, Zz) is identified using the sharpened image region.

Description

Method and device for detecting traffic signs

The invention relates to a method and a device for detecting the content of traffic signs according to the preamble of the independent claims.

Modern driver assistance systems are increasingly equipped with an electronic traffic sign recognition system, for example, to warn the driver when exceeding the maximum speed limit. For this purpose, typically takes a camera on the environment in front of the vehicle and delivers corresponding image data to a

Image evaluation device that uses an algorithm to analyze and classify the image data to identify a traffic sign. Such a method is e.g. known from DE 198 52 631 AI.

The information from such detected traffic signs can then be incorporated into a driver assistance function, eg. B. as an indication of the currently permissible maximum speed in the instrument cluster of the vehicle.

WO 2009/135460 Al shows a method for

Exposure control for a camera in a motor vehicle, which makes it possible to realize two driver assistance functions with different exposure requirements, such as a lane detection and traffic character recognition at night, with a common camera. Thus, a sufficiently short exposure time can be used for traffic sign recognition image data, in particular at high driving speeds, in order to avoid or minimize "motion-blur", ie motion blur, so that the statement of the traffic sign can be recognized. In many cases, the traffic signs or signs erected in the road are provided with one or more additional symbols which concretize the meaning of the main character or limit it situationally.

WO 2013/017125 A1 shows a method for recognizing traffic signs in which additional characters are classified as far as possible by means of a pattern recognition method, additional character text is read and interpreted by means of a text recognition method, if the classification was not or not completely possible and a relevance assessment is made the recognized text is compared with situation data.

A fundamental difficulty in reading textual content and, in particular, variable text elements on road signs, is that it requires images that have sufficient resolution and sharpness.

The invention has for its object to meet this difficulty and to provide a method for fast and reliable detection of the content of traffic signs with variable text component.

This object is solved by the features of the independent claims. Advantageous developments of the invention are the subject of the dependent claims, wherein combinations and developments of individual advantageous features are conceivable with each other.

A starting point of the invention are the following considerations: Although cameras today can offer basically a good resolution and sharpness, this is when using cameras in the moving vehicle, with different

Exposure times, especially in the dark, the quality of the optics used or the lens used and the resolution of the image sensor used for the complete detection of traffic signs not always sufficient.

Since it will not be possible for economic reasons in the foreseeable future to increase the image quality, ie the optics and the resolution and sensitivity of the imager, in vehicle cameras so that a reading of traffic signs or signs with a variable text component will be immediately possible the present invention, the approach to determine the motion blur of a traffic sign from an image or from an image sequence.

An inventive method for detecting a traffic sign by means of a camera from a moving vehicle has the following steps:

Recording at least one image of a vehicle environment with the camera,

Determination of the presence of at least one traffic sign in the at least one recorded image of the vehicle surroundings,

Determining a motion blur in the image area in which the existing traffic sign is located,

- Calculating the motion blur in this image area, resulting in a sharpened image area, and

- Recognition of the traffic sign taking into account the sharpened image area. The camera is in particular a forward-facing monocamera, preferably with color-resolving

Image recording sensor (CMOS or CCD), which may be arranged for example behind the windshield approximately in the region of the inner mirror. Alternatively, a stereo camera has the advantage that it provides spatial information about the vehicle environment.

From an image or from an image sequence of the vehicle camera, further camera-based driver assistance functions can preferably be implemented in addition to traffic sign recognition, such as e.g. a lane departure warning, a lane keeping assistance, an automatic high beam control, a collision warning, a precipitation detection, an automatic longitudinal control, a parking assistance, automatic emergency braking and / or emergency steering systems.

The determination of the motion blur comprises the estimation of the motion blur, but also the calculation of the motion blur, in particular one

Motion blur kernels ("blur kernel") or a point spread image function / point spread function.

Subsequently, the influence of the motion blur on the recorded image can be calculated out by equalizing the blurred image or a blurred image area (deconvolution). This leads to a sharpened image (area).

The recognition of the traffic sign taking into account the sharpened image area can take place via pattern recognition, classification or text recognition (OCR).

A method according to the invention has the advantage that the influence of the motion blur on surrounding images of a camera of a moving vehicle, which contain traffic signs, can be attenuated or even eliminated, whereby a reliable and complete detection of traffic signs is possible.

In a preferred embodiment, the step of "recognizing the traffic sign taking into account the sharpened image area" comprises a text recognition in the sharpened

Image area. This allows complete recognition of variable text components in traffic signs. Advantageously, to determine the motion blur, data are evaluated which characterize the proper motion of the vehicle. The motion blur of traffic signs recorded while driving stems only from the camera movement, so typically in a fixed camera in the vehicle only from the vehicle's own movement ago. This cause can be taken into account by knowing about the current proper motion of the vehicle optionally in combination with imaging properties of the camera system and the geometry between the camera and traffic signs. Such data may, for example, provide vehicle motion sensors. Image data evaluations can also provide indications of the vehicle's own motion, in particular in the case of a stereo camera. According to a preferred embodiment, edges in this image area are determined and evaluated in order to determine the motion blur in the image area in which the existing traffic signs are located. One possible way of estimating motion blur from edges in the blurred image is provided by TS Cho, S. Paris, WT Freeman, and B. Horn in "Blur kernel estimation using the radon transform," 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Preferably, a static content of a traffic sign for the partial recognition of the

Traffic sign used from the recorded image. A predefined template traffic sign corresponding to the partially recognized traffic sign is taken into account in determining the motion blur in the image area. Taking into account the sharpened image area, the variable content of the traffic sign is recognized. As a result, the traffic sign, in particular a main character, already partially or even completely recognized at the time of Bewegungsunschärfeschät tion, so its appearance without motion blur (template traffic sign) is known and can be taken into account in determining the motion blur.

In an advantageous development, the static content comprises the form and outline of a main traffic sign. The variable content is then the content included in the image area of the border of the main traffic sign.

Preferably, the image area is formed in such a way that it contains at least one main traffic sign and all auxiliary signs assigned to this main traffic sign. Of course, this can only be a single additional character if only one additional character is assigned to the main traffic sign. As static content, the at least one main traffic sign can then be completely recognized and the contents of the additional sign (s) identified as variable content. This is advantageous, inter alia, since the border (clear edges) of additional characters is often not easily detectable, and additional characters often have variable contents that can not all be trained in advance. In an alternative advantageous embodiment, a separate image area is formed for each individual traffic sign (main character or additional sign) in the image. It is then checked whether at least two traffic signs are assigned to each other, in particular based on their geometric arrangement to each other. When recognized assignment of two traffic signs in a first and a second or further image area (s), the motion blur is determined only for the first image area and then adopted for the second and the further image areas. The first image area can advantageously contain a main traffic sign, in particular a

Main traffic sign, which can be completely recognized without text recognition.

Frequently, the recognition of an additional character from blurred image areas is very difficult. By calculating the motion blur not on the as yet unknown additional character, but on the associated main character (eg a speed limit or a no-overtaking sign), this calculation is much easier and more accurate. Because main characters are larger and very well detectable by their strong red border. Since the main character and the additional character are placed directly above or next to each other, and captured in the same image, the motion blur parameters are almost identical for both characters, and thus can be transferred from the main character to the auxiliary character. Preferably, if the recognition of the traffic sign is not successful or uncertain in consideration of the sharpened image area of a single image, a synthetic high-definition image area is created from a plurality of successive ones with the camera Recorded images extracted sharpened image areas in each of which is the same traffic sign. Thus, with the exception of the step "recognition of the traffic sign taking into account the sharpened image area", all main procedural steps for

Followed images repeated. From this plurality of sharpened image areas representing the same traffic sign, a synthetic high-definition image area is now calculated. Such methods are known per se for preparing higher resolution synthetic images, see e.g. "Improving resolution by image registration" by M. Irani, S. Peleg in CVGIP: Graphical Models and Image Processing, Vol. 53, No. 3, pp. 231-239, 1991. From a series of several, real-world camera shots, eg of Additional characters with low quality can therefore be a high-quality, synthetic image are calculated, which can then be read by means of a text recognition.

The traffic sign is then recognized taking into account the synthetic high-definition image area, which increases the reliability and availability of the

Traffic character recognition continues to increase.

The invention further relates to a device for detecting traffic signs comprising a camera for recording at least one image of a vehicle environment from a moving vehicle and an image evaluation unit, wherein the image evaluation unit is designed such that it detects the presence of one or more traffic signs (Hz, Zz) from the detected motion picture data detects a motion blur in the image area in which the existing traffic sign (Hz, Zz) is located, which calculates motion blur in this image area, resulting in a sharpened image area, the traffic sign (Hz, Zz) recognizes taking into account the sharpened image area and can generate a corresponding output signal.

Further advantages of the invention will become apparent from the description and the drawings. Embodiments are shown in simplified form in the drawing and explained in more detail in the following description.

Show it:

Fig. 1: Examples of traffic signs with variable

Content (a) main and b) additional characters),

Fig. 2: a main character with variable text content, Fig. 3: a speed-limiting

Main sign with an additional sign with French text and

Fig. 4: a main character with associated

Additional character.

1 shows traffic signs with variable contents, in particular variable text contents: FIG. 1a shows main characters with variable text content, namely a prohibition for vehicles (starting from the left) over 5.5 tons actual mass, over 2 meters actual width (including outside mirrors) resp over 8 tons of actual axle load.

Fig. Lb shows additional characters with variable text content, which limit the validity of the associated main character (not shown in Fig. 1), namely to a distance from the main character of 600 meters, a period of 8-11 clock and 16-18 clock or for vehicles with an actual mass of 7.5 tonnes or more. 1, in particular, the numerical values are "variable text contents." Complete recognition of such traffic signs, in particular by means of pattern recognition, is difficult, and requires text recognition (eg, OCR) Images of these traffic signs, which have a sufficient resolution and sharpness. However, images taken with a vehicle camera from a moving car have a blurring that results from the movement of the vehicle and thus the camera relative to the traffic sign. In English, this motion blur is referred to as "motion blur." The shorter the exposure time when taking a picture of a camera, the lower the motion blur is, but in the dark a minimum exposure time is required to see anything from the captured image data.

Especially the traffic signs (Hz, Zz) shown by way of example in FIG. 1 are not always completely recognizable due to the motion blur of the image recordings of a camera-based driver assistance system. In order to counteract this, in an exemplary embodiment according to the invention, an image of the vehicle environment recorded by the camera is searched for existing traffic signs (Hz, Zz). If traffic signs (Hz, Zz) are determined in an image area, the motion blur in this image area is determined (calculated or at least estimated). This calculation or estimation is the subject of current research, especially in hand-held cell phone / photo cameras that often provide shaky images when triggered. The motion blur is typically determined as a "blur kernel" (motion blur core) or as a "point spread function". Only with their help, the blurred image can be rectified targeted (deconvolution), which leads to a sharpened image. One possible way estimates the kernel of edges in the blurred image: "Blur kernel estimation using the radon transform" TS Cho, S. Paris, WT Freeman, B. Horn, 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). The traffic sign recognition here offers two conditions which make this approach advantageous: traffic signs (Hz, Zz) have clear edges and (apart from variable contents (V)), sharp images of the static traffic sign contents (S) are known, eg the general prohibition sign, which can be regarded as a template for all special prohibition signs. Both allow an efficient application, for example, that of Cho et al. Proposed solution for the estimation of the motion blur and the subsequent deconvolution.

Subsequently, the influence of the determined motion blur on this image area is eliminated, which leads to a sharpened image area. The complete recognition of the traffic sign (Hz, Zz), in particular by means of a text recognition for reading the variable text content, then takes place taking into account the thus sharpened image area. With reference to FIGS. 2 to 4 preferred embodiments of the invention are explained in detail.

Fig. 2 shows the first traffic sign of Fig. La: a ban on vehicles with an actual mass of over 5, 5 tons.

This traffic sign can be subdivided into a static component (S), namely its shape and outline, ie essentially the red ring which is indicative of prohibition signs, and a variable component (V), here the text "5,5t" In the context of camera-based traffic sign recognition, it is often possible to recognize from the recorded image data with motion blur that this traffic sign (Hz) is a prohibition sign Image data with motion blur are not immediately recognized because they are too noisy than that a text recognition would be successful. Therefore, the motion blur for the image area in which the traffic sign (Hz) is included is estimated. This is possible because the circular shape and the red border of the prohibition sign are very well detectable, and especially at the time of Bewegungsunschärfeschätzung can already be detected, whereby their appearance is known without motion blur. This motion blur estimate is then used to sharpen the image area within the circular red border (V) by excluding motion blur. Subsequently, a recognition or text recognition of this inner image area (V) is possible, which results in the fact that the characters "5.5 t" are contained there, thereby completely detecting the regulatory content of this traffic sign, namely a prohibition for vehicles with an actual mass of over This can now be communicated to the driver of a vehicle (eg display of a corresponding symbol in the instrument cluster) or with vehicle-specific data such as the current or maximum actual mass of the vehicle adjusted and discarded if not relevant (eg for a car) or even be output from the traffic sign recognition to a vehicle navigation device or an autonomous vehicle control device.

FIG. 3 shows a main character (Hz) which restricts the maximum permissible speed to 50 km / h with an additional character (Zz) with the French text "par temps de pluie", which means that the cruise control of the associated main traffic sign can only be used in the event of rain or rain In the strict sense, this text is not variable in France, but for a traffic sign recognition, the In as many different countries or languages as possible the content of additional characters can be recognized and interpreted, it would be desirable that such texts can be read as single letters without prior knowledge of possible textual content. Therefore, within the scope of the invention, this is the variable text component (V) which it is to be recognized. Upon recognizing this variable text component, it is nevertheless possible to display to the driver this text as additional information about the speed limitation in the instrument cluster, or to search in vehicle databases or in the cloud for the meaning or a translation of the recognized words.

According to one embodiment of the method, the image area is designed such that it comprises the main traffic sign (Hz) and the additional sign (Zz) assigned here individually. Then, the motion blur for this entire image area can be determined from the motion blur for the main character (Hz). The entire main character can be assumed as static content (S), because the possible values for

Speed limits are manageable. Analogous to the procedure in FIG. 2, the variable content (V), ie here the additional character (Zz), can now be recognized or read from the image area sharpened taking into account the determined motion blur.

Fig. 4 shows a main traffic sign (Hz), namely the danger sign "game change" with an associated additional character (Zz), which limits the period of validity of the danger symbol "to 3 km".

According to an advantageous embodiment, the main character (Hz) and the additional character (Zz) are recognized as two separate traffic signs and correspondingly formed two separate image areas. Subsequently, it is checked whether the two traffic signs (Hz, Zz) are assigned to each other. Based on the present arrangement, in which the main character is located directly above the additional character, their assignment is detected. The motion blur is now determined only for the first image area in which the main character (Hz) is located. This can be fully recognized without text recognition: possibly already from the recorded (blurred) image, we assume, however, that the recognition succeeds only after the estimation of the motion blur and subsequent deconvolution. In any case, the calculation / estimation of the

Motion blur for the image area of the main character (Hz) much easier and more accurate. The main character is larger and due to its strong red border, here triangular, very well detectable. Since the main character (Hz) and the additional character (Zz) are placed directly above or next to each other and captured in the same image, the motion blur parameters are almost identical for both characters and can thus be transferred from the main character (Hz) to the auxiliary character (Zz) become. The deconvolution of the image area with the additional character (Zz) is thus directly feasible and then the content of the additional character (Zz) can be recognized or read in the sharpened image detail.

If the additional character (Zz) still can not be fully read, it is appropriate in a further development of the method to create a synthetic high-definition image area from a plurality of successive images taken with the camera as described extracted sharpened image areas, in each case the additional character (Zz) is located. This synthetic high-definition image area then allows complete recognition of the additional sign (Zz) and thus of the control content of the combination of main (Hz) and additional sign (Zz.).

Claims

claims

Method for detecting a traffic sign (Hz, Zz) by means of a camera from a moving vehicle with the steps

Recording at least one image of a vehicle environment with the camera,

Determination of the presence of at least one traffic sign (Hz, Zz) in the at least one recorded image of the vehicle environment,

Determining a motion blur in the image area in which the existing traffic sign (Hz, Zz) is located,

- Detection of the traffic sign (Hz, Zz) under

Consideration of the sharpened image area.

2. The method of claim 1, wherein the recognition of the traffic sign (Hz, Zz) taking into account the sharpened image area comprises a text recognition in the sharpened image area.

3. The method of claim 1 or 2, wherein for determining the motion blur data representing the proper motion of

Characterize vehicle to be evaluated.

4. The method according to any one of the preceding claims, wherein for determining the motion blur in the image area in which the existing traffic sign (Hz, Zz) is located, edges are determined and evaluated.

5. The method according to any one of the preceding claims, wherein a static content (S) of a traffic sign (Hz, Zz) used for the partial recognition of the traffic sign from the recorded image and the partially recognized traffic sign corresponding predetermined template traffic sign in determining the motion blur in Considering the sharpened image area of the variable content (V) of the traffic sign (Hz, Zz) is recognized.

The method of claim 5, wherein the static content (S) comprises the shape and outline of a main traffic sign (Hz).

7. The method according to claim 5, wherein the image area is formed such that at least one main traffic sign (Hz) and all these main traffic signs associated additional character (Zz), wherein as static content at least one entire main traffic sign (Hz) and as variable content of the contents of or the

Additional character (s) (Zz) is detected.

8. The method according to any one of claims 1 to 6, wherein for

Each individual traffic sign (Hz or Zz) in the image is a separate image area is formed, which is then checked whether at least two traffic signs (Hz, Zz) are assigned to each other, and determined association the motion blur determined only for a first image area and then for the is taken over this further assigned image areas.

9. The method according to any one of the preceding claims, wherein if the recognition of the traffic sign (Hz, Zz) under Considering the sharpened image area is unsuccessful or unsure, a synthetic high-definition image area is created from a plurality of successive sharpened image areas extracted from images taken with the camera, in each of which the same traffic sign (Hz, Zz) is located, the traffic sign (Hz , Zz) is recognized in consideration of the synthetic high-definition image area.

10. A traffic sign recognition device comprising a camera for receiving at least one image of a vehicle environment from a moving vehicle and an image evaluation unit, wherein the image evaluation unit is configured to detect the presence of one or more traffic signs (Hz, Zz) from the traffic signals

recorded image data, a motion blur in the image area in which the existing

Traffic sign (Hz, Zz) is located, the

Motion blur in this image area out, resulting in a sharpened image area, the

Recognizes traffic signs (Hz, Zz) taking into account the sharpened image area and can generate a corresponding output signal.