DE102022213066A1 - Machine evaluation of an automatic traffic sign recognition system of a vehicle - Google Patents

Abstract

The invention relates to a system for the automated evaluation of automatic traffic sign recognition of a vehicle, comprising: a first interface (1) for providing system data logged during a past journey of the vehicle and a result of traffic sign recognition with regard to a speed limit; a second interface (3) for providing ground truth data; a computing unit (5) for comparing the ground truth data and the data of the first interface (1) with one another and checking the correctness of the determined speed limit of the vehicle; a display unit (7) for displaying the result of the comparison and/or the check on a graphical user interface by: a map with a position of the vehicle that corresponds to the position of a zone currently viewed on the graphical user interface with a respective speed limit, as well as portions of determinations of the respective speed limits by the vehicle that are consistent with the ground truth data and related to the entire journey or sections of the entire journey of the vehicle.

Description

The invention relates to a system for the automated evaluation of an automatic traffic sign recognition of a vehicle.

• Während in modernen Fahrzeugen mit Fahrerassistenzsystem bei manueller Fahrzeugführung eine automatische Verkehrszeichenerkennung erfolgen kann, um dem Fahrzeugführer per Einspiegelung oder Sprachnachricht beispielsweise das Ergebnis der automatischen Verkehrszeichenerkennung mitzuteilen und insbesondere bei Überschreitung der aktuell erlaubten Höchstgeschwindigkeit den Fahrzeugführer darauf hinzuweisen, können auch automatisierte Fahrzeuge die Information der automatischen Verkehrszeichenerkennung nutzen, um die maschinelle Fahrzeugführung nach den Ergebnissen der automatischen Verkehrszeichenerkennung durchzuführen.

The following information does not necessarily result from a single specific prior art document, but rather from general professional considerations and current but generally known specialist knowledge:

• While modern vehicles with driver assistance systems can use automatic traffic sign recognition when the vehicle is being driven manually, in order to inform the driver of the result of the automatic traffic sign recognition via a display or voice message, for example, and in particular to alert the driver when the currently permitted maximum speed is exceeded, automated vehicles can also use the information from the automatic traffic sign recognition to carry out machine-based vehicle guidance based on the results of the automatic traffic sign recognition.

It is particularly important to verify the traffic sign recognition and ensure its reliability when developing such a driver assistance system or an automated vehicle. The legislature has passed extensive regulations on speed displays. Among other things, the system must show the driver the correct speed sign for ninety percent of the route driven. A large number of field tests and simulations may therefore be necessary to evaluate or confirm the continuous development of traffic sign recognition for the purpose of determining the maximum permitted speed. When adapting or developing a new traffic sign recognition system, a manual check would therefore in principle have to be carried out to determine whether the automatic and therefore machine-based traffic sign recognition system has correctly recorded the local conditions. However, manual evaluation is very time-consuming and, due to the scope, is simply not possible in practice.

In some regions of the world, legal norms determine the criteria by which maximum permitted speeds must be determined on the route driven (for example, if the route is longer than 400 km, divided into towns, country roads and motorways). The requirements set by the legislator also require, for example, that the valid traffic sign is displayed to the driver for at least 90% of the route during a 400 km journey. None of the sections (towns, country roads, motorways) may reach a value of less than 80%. In addition, these limit values must be complied with in all EU countries as well as in Switzerland, Norway and the United Kingdom. The question of whether these requirements were complied with during a test drive or whether adjustments or new developments in traffic sign recognition have achieved the desired effect must be answered quickly and reproducibly.

It is therefore an object of the invention to be able to more easily check compliance with new legal requirements for automatic traffic sign recognition.

The invention results from the features of the independent claims. Advantageous further developments and embodiments are the subject of the dependent claims.

• - eine erste Schnittstelle zum Bereitstellen von während einer vergangenen Fahrt des Fahrzeugs protokollierten Systemdaten, wobei die Systemdaten im Fahrzeug zur elektronischen Verarbeitung mit dem Zweck einer automatischen Verkehrszeichenerkennung während seiner Fahrt erzeugt und aufgezeichnet worden sind, sowie zum Bereitstellen des Ergebnisses der Verkehrszeichenerkennung bezüglich einer jeweils aktuell auf einem befahrenen Straßenabschnitt gültigen Geschwindigkeitsbeschränkung;
• - eine zweite Schnittstelle zum Bereitstellen von Ground-Truth-Daten über die jeweilige auf einem befahrenen Straßenabschnitt gültige und zumindest durch Verkehrszeichen vorgegebene Geschwindigkeitsbeschränkung;
• - eine Recheneinheit, die dazu ausgeführt ist, die Ground-Truth-Daten und die Daten der ersten Schnittstelle miteinander zu vergleichen und auf zu den Ground-Truth-Daten inkonsistente Ermittlung der Geschwindigkeitsbeschränkung durch Verkehrszeichenerkennung durch das Fahrzeug während seiner Fahrt zu überprüfen, und die dazu ausgeführt ist, Informationen über den Vergleich und die Überprüfung an eine Anzeigeeinheit weiterzuleiten;
• - die Anzeigeeinheit, die dazu ausgeführt ist, das Ergebnis des Vergleichs und/oder der Überprüfung jeweils durch die Recheneinheit auf einer grafischen Bedienoberfläche darzustellen durch: eine Karte mit einer Position des Fahrzeugs, die der Position einer aktuell auf der grafischen Bedienoberfläche betrachteten Zone mit einer jeweiligen Geschwindigkeitsbeschränkung entspricht, sowie auf die gesamte Fahrt oder Streckenabschnitte der gesamten Fahrt des Fahrzeugs bezogene Anteile von zu den Ground-Truth-Daten konsistenten Bestimmungen der jeweiligen Geschwindigkeitsbeschränkungen durch das Fahrzeug.

A first aspect of the invention relates to a system for the automated evaluation of an automatic traffic sign recognition of a vehicle, comprising:

• - a first interface for providing system data logged during a past journey of the vehicle, wherein the system data have been generated and recorded in the vehicle for electronic processing for the purpose of automatic traffic sign recognition during its journey, as well as for providing the result of the traffic sign recognition with regard to a speed limit currently valid on a section of road traveled;
• - a second interface for providing ground truth data on the respective speed limit valid on a road section and specified at least by traffic signs;
• - a computing unit designed to compare the ground truth data and the data from the first interface and to check for inconsistencies with the ground truth data in the determination of the speed limit by traffic sign recognition by the vehicle during its journey, and designed to forward information about the comparison and the check to a display unit;
• - the display unit, which is designed to display the result of the comparison and/or the check by the computing unit on a graphical user interface by: a map with a position of the vehicle that corresponds to the position of a zone with a respective speed limit currently viewed on the graphical user interface, as well as proportions of determinations of the respective speed limits by the vehicle that are consistent with the ground truth data and related to the entire journey or sections of the entire journey of the vehicle.

The permitted maximum speeds, also called maximum permissible speeds, can be set either directly by signs with numerical information, or indirectly by signs that specifically indicate a zone, whereby the maximum permissible speed within such a zone is specified by law. Typically in Europe, maximum permissible speeds of 50 km/h are set by place-name signs, but direct numerical information such as a maximum permissible speed of 30 km/h or 80 km/h are more common. Such numerical information is typically given on metal signs, but can also be given by illuminated displays. Typically, it is the job of a manual driver to read such signs and adhere to the permitted maximum speed, but driver assistance systems or even automated vehicles often read the traffic signs mechanically.

The data of the first interface comes from a stored data set of the vehicle, which can be read out in particular from a so-called "trace file". Such a file includes in particular logged data that is generated in the vehicle itself and transmitted via a data bus (such as a CAN bus). The first interface is therefore used in particular to read out the data that was stored while the vehicle was driving and to transfer it to the computing unit.

This requires that the data is saved accordingly when the vehicle is driving, which is the case with separate test drives, but also with data recording that is barely noticeable to the user in so-called "silent testing" procedures. The vehicle does not necessarily have to drive physically, even if physical drives have the advantage of a completely real scenario environment, but the drive can also take place in a physical simulator, for example, by integrating it into "hardware-in-the-loop" structures.

While the ground truth list represents the target state on a route, the recording represents the current actual state of the traffic sign recognition. Such a "ground truth" provides a correct description of reality and can be used to verify machine estimates. This ground truth data was generated in particular by another test vehicle with personal visual control, by highly accurate data from another source, or similar.

It does not matter whether the data was generated from a real journey or from a simulation on a hardware-in-the-loop (HIL) computer. Possible differences in the recording can be regulated by adjusting the input currents.

One advantageous effect of the invention is that a computer-aided solution is provided for determining key figures relating to compliance with new legal requirements. Checking steps are automated and the user is then provided with a comprehensive evaluation with the most important key figures. Compliance with new legal requirements can thus be checked quickly. The user thus receives information on the extent to which the legal requirements that will apply in the future can be complied with. Lengthy manual analyses are no longer necessary. Problems are also highlighted and additional information is prepared for the user in order to analyse these problems as quickly as possible. Selective functional tests by means of simulation tests, often accompanied by a small number of field tests, are therefore no longer the only practical approach, which has proven to be slow and cannot keep up with the necessary development speed. The evaluation process is also standardised and does not allow any scope for human interpretation.

Due to the tightening of legal requirements, such a system for the automatic detection of speed limits, especially from traffic signs, has become more complex. Every update of the input sensors (front camera, navigation device, real-time cloud) and the algorithm used to detect speed limits can thus be more easily checked through a large number of field tests or re-simulations. The legislator requires complete documentation of the results. The evaluations of the system according to the invention can also be used for this and thus replace additional report creation.

According to an advantageous embodiment, the ground truth data comprises a finite List of speed limits referenced for each road section with a road-related path coordinate, or referenced to geodetic position sets.

According to a further advantageous embodiment, the system data logged during the journey of the vehicle comprise sensor data about the environment of the vehicle recorded during the journey of the vehicle.

The input signals from the various sensors (front camera, navigation system and real-time cloud) as well as the output signals from the system are preferably analyzed. The information provided enables an initial root cause analysis.

According to a further advantageous embodiment, the display unit is designed to further display the result of the comparison and/or the check by the computing unit on the graphical user interface by: results of the comparison individually for individual road categories, at least the road categories city, country road, motorway.

According to a further advantageous embodiment, the display unit is designed to further display the result of the comparison and the check by the computing unit on the graphical user interface by at least one of: compass orientation of the vehicle over a plurality of points in time during its journey, assignment of a respective time stamp of the journey duration to the respective position of the vehicle during its journey, type of respective speed limit, actual speed driven when recording the logged system data that was used for traffic sign recognition.

The type of speed limit in question relates in particular to the physical type of the road sign and/or the legal-categorical type of the road sign symbolising the maximum permitted speed. While the physical type of the road sign symbolising the maximum permitted speed may be, for example, “metal” or “illuminated indication”, the legal-categorical type of the road sign relates to categories such as “numerical indication” (for example, a sign with a circled “100” to symbolise a maximum permitted speed of 100 km/h) or a “zone indication” (for example, a sign indicating a traffic-calmed area).

According to a further advantageous embodiment, the display unit is designed to output portions of determinations of the respective speed limits by the vehicle that are consistent with the ground truth data as absolute values or as a percentage, each based on explicit route lengths or based on time periods.

According to a further advantageous embodiment, the display unit is designed to output all determinations of the respective speed limits by the vehicle that are inconsistent with the ground truth data in a separate list and/or to mark them in a color-highlighted list of consistent and inconsistent determinations.

According to a further advantageous embodiment, the computing unit is designed to detect each geodetically referenced position contained in the logged system data and to determine a tolerance range around the respective position, as well as to check whether the respective position is contained in the ground truth data and, if it is contained, to further examine the data associated with this respective position for recognition of the traffic signs by vehicle sensors.

For example, it checks whether the front camera has also detected a new sign in this area and what information is transmitted by the navigation system and the real-time cloud. All information is summarized, saved and displayed graphically in a table.

Further advantages, features and details emerge from the following description, in which - if necessary with reference to the drawing - at least one embodiment is described in detail. Identical, similar and/or functionally identical parts are provided with the same reference numerals.

• 1: Ein System zum automatisierten Auswerten einer automatischen Verkehrszeichenerkennung eines Fahrzeugs gemäß einem Ausführungsbeispiel der Erfindung.

It shows:

• 1 : A system for the automated evaluation of an automatic traffic sign recognition of a vehicle according to an embodiment of the invention.

The representations in the figures are schematic and not to scale.

1 shows a system for the automated evaluation of automatic traffic sign recognition of a vehicle. A computing unit 5 has a first interface 1 and a second interface 3 for data acquisition. The first interface 1 is used to receive and provide data from a previous test drive of the vehicle. logged system data. This system data was generated for electronic processing with the purpose of automatic traffic sign recognition during the journey, transmitted via a CAN bus of the vehicle, and recorded. The results of the traffic sign recognition determined by the vehicle with regard to a speed limit currently in force on a section of road being traveled are also included. The second interface 3 is also used to receive and provide ground truth data about the speed limit that is valid on a section of road being traveled and at least specified by traffic signs. It can therefore be assumed that the computing unit 5 has both the data from the test drive of the vehicle, including the traffic sign recognition carried out therein to determine the speed limits, and the ground truth data that correctly reflect the real situation with regard to the speed limits actually specified by traffic signs. On this basis, the computing unit 5 carries out a comparison between the ground truth data and the data from the first interface 1. This comparison relates to traffic signs detected by a camera on the vehicle - both electronic and permanently physically displayed on metal - in order to be able to test the sensor capacity of the vehicle, as well as the valid speed limits for respective sections of the total distance traveled by the vehicle relative to the speed limits determined by the vehicle, in particular on the basis of the traffic signs detected by the sensors. The computing unit 5 checks the correct determination of the vehicle's speed limits by checking whether these data determined by the vehicle are consistent with the ground truth data of the actually prevailing speed limit loaded via the second interface 3. Information about the comparison and the check is then graphically output on a display unit 7: In the top right corner of the screen of the display unit 7, a map is shown, with the route traveled by the vehicle being marked separately on it (for the sake of simplicity, not shown in the 1 shown); the observed position of the vehicle, which is also shown on the map, correlates with a specific situation of speed limits, which are described in the rectangular field below the map. The middle entry is selected for separate consideration. Accordingly, the geodetic position of the vehicle is shown on the map in WGS-84 coordinates, marked on the map, and according to the ground truth data, the actual speed limit L2 prevailing at this position (from a list L1 to L3) is marked, see the upper left section of the screen of the display unit 7. In the rectangles below, the results of the checks and comparisons between the data from the first interface 1 and the data from the second interface 3 are shown in a table. The order refers to individual sections of the vehicle's route: S for urban area, L for country road, A for motorway section. For each of these sections, the total distance traveled by the vehicle is shown in a table, the associated time stamps at the respective distance, and the percentage share. The absolute values regarding distance and duration of speed limits that match the ground truth data and are determined by the vehicle are also shown. Values that deviate from a given standard are specially marked in color and when the user selects the corresponding fields, all data from the first interface 1 is displayed separately to make troubleshooting easier for the user. The type of sign (e.g. speed limit, town entrance, play street, etc.), the position of the sign (e.g. motorway bridge, left, right or both sides) and the type of sign (e.g. LED sign, temporary sign) are preferably displayed with a correlation to one another.

Although the invention has been illustrated and explained in detail by preferred embodiments, the invention is not limited by the disclosed examples and other variations can be derived therefrom by the person skilled in the art without departing from the scope of the invention. It is therefore clear that a multitude of possible variations exist. It is also clear that embodiments mentioned by way of example really only represent examples that are not to be understood in any way as a limitation of the scope of protection, the possible applications or the configuration of the invention. Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

List of reference symbols

1

first interface

3

second interface

5

Computing unit

7

Display unit

L1-L3

Traffic signs

S

City zone

L

Country road zone

A

Motorway zone

Claims (8)

System for the automated evaluation of automatic traffic sign recognition of a vehicle, comprising: - a first interface (1) for providing system data logged during a past journey of the vehicle, wherein the system data in the vehicle have been generated and recorded for electronic processing with the purpose of automatic traffic sign recognition during its journey, and for providing the result of the traffic sign recognition with regard to a speed limit currently valid on a section of road traveled; - a second interface (3) for providing ground truth data about the respective speed limit valid on a section of road traveled and specified at least by traffic signs; - a computing unit (5) which is designed to compare the ground truth data and the data of the first interface (1) with one another and to check for a determination of the speed limit by traffic sign recognition by the vehicle during its journey that is consistent with the ground truth data, and which is designed to forward information about the comparison and the check to a display unit (7); - the display unit (7), which is designed to display the result of the comparison and/or the check by the computing unit (5) on a graphical user interface by means of: a map with a respective position of the vehicle, as well as parts of the determinations of the respective speed limits by the vehicle that are consistent with the ground truth data and related to the entire journey or sections of the entire journey of the vehicle. System according to Claim 1 , where the ground truth data comprises a finite list of speed limits referenced per road segment with a road-related path coordinate, or referenced to geodetic position sets. A system according to any preceding claim, wherein the system data logged while the vehicle is traveling includes sensor data about the environment of the vehicle recorded while the vehicle is traveling. System according to one of the preceding claims, wherein the display unit (7) is designed to further display the result of the comparison and/or the check by the computing unit (5) on the graphical user interface by: results of the comparison individually for individual road categories, at least the road categories city, country road, motorway. System according to one of the preceding claims, wherein the display unit (7) is designed to further display the result of the comparison and the check by the computing unit (5) on the graphical user interface by at least one of: compass orientation of the vehicle over a plurality of points in time during its journey, assignment of a respective time stamp of the journey duration to the respective position of the vehicle during its journey, type of respective speed limit, actual speed driven when recording the logged system data that were used for traffic sign recognition. System according to one of the preceding claims, wherein the display unit (7) is designed to output portions of determinations of the respective speed limits by the vehicle that are consistent with the ground truth data as absolute values or as a percentage, each based on explicit route lengths, or based on time periods. System according to one of the preceding claims, wherein the display unit (7) is designed to output all determinations of the respective speed limits by the vehicle that are inconsistent with the ground truth data in a separate list and/or to mark them in a color-highlighted list of consistent and inconsistent determinations. System according to one of the preceding claims, wherein the computing unit (5) is designed to detect each geodetically referenced position contained in the logged system data and to determine a tolerance range around the respective position, as well as to check whether the respective position is contained in the ground truth data, and if it is contained, to further process the data associated with this respective position. to examine the detection of traffic signs by vehicle sensors.

Images