DE102019206147A1 - Method for capturing image material for checking image evaluation systems, device and vehicle for use in the method, as well as computer program - Google Patents

Abstract

The proposal relates to a method for capturing image material for checking image evaluation systems, a check being made to determine whether an object was recognized by the image evaluation system correctly in terms of time or location. Such image evaluation systems are used, for example, in vehicles (10). The proposal also relates to the measure that, when a discrepancy in object recognition beyond a tolerance limit is detected, it is determined which images or image parts (17) are to be archived for more precise checking, and these images or image parts (17) are archived. This can be done by forwarding the images or image parts (17) to be archived to a backend server (320). The archived images are subsequently checked and the results can be used to improve the image recognition systems.

Description

The proposal concerns the technical field from driver assistance systems to autonomous vehicles. The proposal relates in particular to a method for capturing image material for checking image evaluation systems. The proposal also relates to a device and a vehicle for use in the method as well as a computer program.

At the moment, intensive work is being done on technologies that will later enable autonomous driving. A first approach is the introduction of various driver assistance systems that relieve the driver of certain tasks. Examples of driver assistance systems are a blind spot warning assistant, an emergency braking assistant, a parking assistant, a turning assistant, a lane departure warning assistant, a cruise control assistant, etc. A further stage of the development can consist in combining several assistance systems. The driver is not completely relieved of his tasks, but it is ensured that the driver can take control of the vehicle at any time. The driver then also performs monitoring functions.

For the near future, it can therefore be assumed that the system will provide comprehensive information about objects through the use of newer technologies (vehicle-to-vehicle communication, use of databases, backend connection, cloud services, server use, vehicle sensors, etc.) (especially vehicles) will be available in the visible and concealed / invisible surroundings of the own vehicle. In the field of vehicle sensors, the following components are mentioned in particular that enable environmental monitoring: RADAR devices corresponding to radio detection and ranging, LIDAR devices corresponding to light detection and ranging, mainly for distance detection / warning, and cameras with corresponding image processing for the Object recognition area. This data about the environment can thus be used as a basis for system-based driving recommendations, warnings, etc. For example, displays / warnings are conceivable about the direction in which (possibly into its own trajectory) another, surrounding vehicle wants to turn. In particular, traffic sign recognition is also mentioned as an application to inform the driver of the legal framework.

Vehicle-to-vehicle communication is now also possible using cellular communication with systems such as LTE in accordance with Long Term Evolution or 5G. As an alternative, systems based on WLAN technology are available for direct vehicle communication, in particular the system based on WLAN p. Such systems are relevant for the field of "autonomous driving". The term “autonomous driving” is sometimes used differently in the literature.

• • Level 0: „Driver only“, der Fahrer fährt selbst, lenkt, gibt Gas, bremst etc.
• • Level 1: Bestimmte Assistenzsysteme helfen bei der Fahrzeugbedienung (u.a. ein Abstandsregelsystem - Automatic Cruise Control ACC).
• • Level 2: Teilautomatisierung. U.a. automatisches Einparken, Spurhaltefunktion, allgemeine Längsführung, Beschleunigen, Abbremsen etc. werden von den Assistenzsystemen übernommen (u.a. Stauassistent).
• • Level 3: Hochautomatisierung. Der Fahrer muss das System nicht dauernd überwachen. Das Fahrzeug führt selbstständig Funktionen wie das Auslösen des Blinkers, Spurwechsel und Spurhalten durch. Der Fahrer kann sich anderen Dingen zuwenden, wird aber bei Bedarf innerhalb einer Vorwarnzeit vom System aufgefordert, die Führung zu übernehmen. Diese Form der Autonomie ist auf Autobahnen technisch machbar. Der Gesetzgeber arbeitet darauf hin, Level 3-Fahrzeuge zuzulassen. Die gesetzlichen Rahmenbedingungen wurden dafür bereits geschaffen.
• • Level 4: Vollautomatisierung. Die Führung des Fahrzeugs wird dauerhaft vom System übernommen. Werden die Fahraufgaben vom System nicht mehr bewältigt, kann der Fahrer aufgefordert werden, die Führung zu übernehmen.
• • Level 5: Kein Fahrer erforderlich. Außer dem Festlegen des Ziels und dem Starten des Systems ist kein menschliches Eingreifen erforderlich.

To clarify this term, the following insert is presented here. Autonomous driving (sometimes also called automatic driving, automated driving or piloted driving) means the locomotion of vehicles, mobile robots and driverless transport systems that are largely autonomous. There are different grades of the term autonomous driving. At certain levels, autonomous driving is also used when there is still a driver in the vehicle who may only be responsible for monitoring the automatic driving process. In Europe the various transport ministries (in Germany the Federal Highway Research Institute was involved) worked together and defined the following levels of autonomy.

• • Level 0: "Driver only", the driver drives himself, steers, accelerates, brakes etc.
• • Level 1: Certain assistance systems help to operate the vehicle (including a distance control system - Automatic Cruise Control ACC).
• • Level 2: partial automation. Automatic parking, lane keeping function, general longitudinal guidance, acceleration, braking, etc. are taken over by the assistance systems (including traffic jam assistant).
• • Level 3: high automation. The driver does not have to constantly monitor the system. The vehicle independently performs functions such as activating the indicator, changing lanes and keeping in lane. The driver can do other things, but if necessary, the system prompts them to take the lead within a warning period. This form of autonomy is technically feasible on motorways. Legislators are working to allow Level 3 vehicles. The legal framework for this has already been created.
• • Level 4: full automation. The system is permanently in charge of the vehicle. If the system can no longer handle the driving tasks, the driver can be asked to take the lead.
• • Level 5: No driver required. No human intervention is required other than setting the destination and starting the system.

Automated driving functions from level 3 relieve the driver of the responsibility for controlling the vehicle. The VDA issued a similar classification of the various levels of autonomy, which can also be used.

Due to the current development towards higher levels of autonomy, where many vehicles are still controlled by the driver, it can be assumed that corresponding additional information can be used for manually operated vehicles in the medium term and not only in the long term for highly automated systems.

For the driver-vehicle interaction, the question arises as to how this information can be presented in such a way that real added value is created for the human driver and he can also locate the information provided quickly, or intuitively. The following solutions in this area are already known from the prior art.

A vision of the future in the automotive industry is to be able to display virtual elements on the windshield of one's own vehicle in order to provide the driver with a number of advantages. The so-called "Augmented Reality" technology (AR) or "Mixed Reality" (MR) technology is used. The corresponding German-language term “augmented reality” or “mixed reality” is less common. The real environment is enriched with virtual elements. This has several advantages: There is no need to look down at displays other than the windshield, as a lot of relevant information appears when looking through the windshield. The driver does not have to take his eyes off the road. In addition, the precise positioning of the virtual elements in the real environment means that less cognitive effort is likely on the part of the driver, since no graphic has to be interpreted on a separate display. With regard to automatic driving, added value can also be generated. In this regard, see the article "3D-FRC: Depiction of the future road course in the Head-Up-Display" by C.A. Wiesner, M. Ruf D. Sirim and G. Klinker in the 2017 IEEE International Symposium on Mixed and Augmented Reality, in which these advantages are explained in more detail.

Since the technological resources are correspondingly limited nowadays, it can be assumed that in the medium term there will be no fully playable windshields in vehicles. Head-up displays are currently being used in vehicles. These also have the advantage that the image of the HUD appears closer to the real environment. These displays are actually projection units that project an image onto the windshield. However, from the driver's point of view, this image is a few meters to 15 meters in front of the vehicle, depending on the type of module.

The “picture” is composed as follows: It is less a virtual display, but rather a kind of “keyhole” into the virtual world. The virtual environment is theoretically superimposed on the real world and contains the virtual objects that support and inform the driver while driving. The limited display area of the HUD means that only a section of it can be seen. So you look through the display area of the HUD at the section of the virtual world. Since this virtual environment supplements the real environment, one speaks in this case of a "mixed reality".

A great advantage of the previously known "Augmented Reality" displays (AR displays) is that the corresponding displays can be displayed directly within or as part of the environment. Relatively obvious examples mostly relate to the area of navigation. While classic navigation displays (in conventional HUDs) usually show schematic representations (e.g. a right-angled arrow pointing to the right as a sign that you should turn right at the next opportunity, AR displays offer much more effective possibilities Environment ”, extremely fast and intuitive interpretations for the user are possible.

In image acquisition systems with image evaluation in the sense of object recognition, it is important to check the correct functioning of the image acquisition and image evaluation during operation. This can be done in a test phase or in normal operation.

From the US 2016/0170414 A1 a vehicle-based system for recognizing traffic signs is known. Environmental sensors such as LIDAR sensors, RADAR sensors and cameras are used. The position of the vehicle is also recorded via GPS. The recognized traffic signs are reported externally with their position and entered there in a database.

From the US 2017/0069206 A1 a vehicle-based system for checking traffic sign recognition is known. Environmental sensors such as LIDAR sensors, RADAR sensors and cameras are also used. The recognized traffic signs and their position are also reported externally and entered there in a database. In order to expose false identifications, various Verification measures proposed, including manual visual inspections by skilled personnel, visual inspections by “crowdsourcing”, computer-aided analysis and statistical analysis.

From the US 2018/0260639 A1 a vehicle-based system and a method for recognizing traffic signs are known. The recognized traffic signs are stored with their position in an image processing unit.

The known solutions have various disadvantages. This was recognized within the scope of the invention. With the image acquisition methods known today for use in the driver assistance area, there is the problem that their correct functioning is not or only insufficiently checked during operation. Incorrect identifications or even incorrect classifications of objects are therefore possible.

There is therefore a need for further improvements in the checking of vehicle-based image acquisition systems, in particular for checking methods that enable a subsequent check in the event of incorrect detection.

The object of the invention is to find such an approach. The effort for archiving the image data as a prerequisite for subsequent checking should also be kept low. This object is achieved by a method for capturing image material for checking image evaluation systems according to claim 1, a device and a vehicle for use in the method according to claims 9 and 14 and a computer program according to claim 15.

The dependent claims contain advantageous developments and improvements of the invention according to the following description of these measures.

As with the US 2018/0260639 A1 the image data are stored for checking incorrect identifications. However, the image data are stored in a device on the vehicle.

According to the proposal, a method for capturing image material for checking image evaluating systems is used to check whether an object was recognized by the image evaluating system correctly in terms of time or location. If so, there is no need to retain any image data for subsequent analysis. If, however, deviations in the temporal or spatial analysis have been recognized that are beyond a tolerance limit, it is determined in one step which images or image parts are to be archived for more precise checking, and these images or image parts are archived. The method offers the advantage that an image evaluation system that supplies safety-relevant data can be checked during operation, the memory expenditure for the check being low. The method can be used particularly advantageously for testing the image evaluation systems that are still in the development stage. However, the process can also be used advantageously in series products. This is advantageous in order to be able to make subsequent improvements to the built-in image evaluation system.

In an advantageous variant, an object recognition algorithm is used for image evaluation and a check is carried out to determine whether an object was recognized correctly. A standard recognition distance is specified for this check. The standard recognition distance indicates the distance from which the object recognition algorithm should deliver the object recognition, i.e. at this standard recognition distance, the image evaluation system should recognize the object in question. It is then determined at what distance to the recognized object the object recognition actually took place. If this distance deviates from the standard recognition distance, the images or image parts that were recorded from the standard recognition distance to the actual object recognition distance are archived for more detailed checking.

The proposal can be used particularly advantageously for testing image evaluation systems in vehicles. These are now equipped with imaging sensors such as cameras, LIDAR or RADAR sensors. They are used to record traffic signs, vehicles ahead and other road users, intersections, turning points, potholes, etc. Such vehicles are also equipped with position detection systems. The determined position of the vehicle can be used to check whether there is a discrepancy in the image recognition. In the simplest case, the vehicle detects the position of the recognized object, even if it passes the location of the object. This can then be used to calculate the distance between the vehicle and the object. In another embodiment, the position of the object can be taken from a highly accurate map. In a further variant, the position can be estimated based on the position of the vehicle.

Satellite navigation and / or odometry are suitable for determining the position data of the vehicle and the object. A generic term for such satellite navigation systems is GNSS, which stands for Global Navigation Satellite System. Existing satellite navigation systems are the Global Positioning System (GPS), Galileo, GLONASS (Globalnaja nawigazionnaja sputnikowaja sistema) or Beidou.

In one embodiment, the images or image parts that were recorded from the standard recognition distance to the object recognition distance are sent to an external archiving location. This has the advantage that only the problematic images have to be archived. Archiving in an external location has the advantage that the images do not have to be archived in the vehicle first. Anyway, the evaluation should be done in the external office by experts who are then also able to improve the image evaluation system.

In another variant, the images or image parts that were recorded from the standard recognition distance to the object recognition distance are archived in a storage unit arranged in the vehicle. For this purpose, the corresponding memory must be provided in the vehicle. The archived images can later be read out by experts. This could be done during a workshop visit. Another advantageous variant consists in transferring the images temporarily stored in the vehicle to the external location when the vehicle is back at the home of the vehicle owner. Modern vehicles are equipped with WiFi. If this is now logged into the vehicle owner's private WLAN network, the archived image data can be transmitted at a high data rate. Compared to direct transmission via a cellular communication system into which the vehicle is registered while driving, this has the advantage that there are fewer costs for the vehicle owner and that the cellular network is less stressed. Its capacity is used by various other applications.

Alternatively, it can be provided that further images or image parts are archived in order to expand the possibilities for checking. It is advantageous if the images or image parts that were recorded from the standard recognition distance to the object recognition distance are archived with high quality, and the other images or other image parts are archived with a lower quality. In this way, the storage effort as well as the transmission effort for archiving further images or image parts can be kept low.

In an extended embodiment it can be provided that an initial analysis of the problematic image data takes place in the vehicle. This can e.g. take place in such a way that it is determined whether an interim concealment of the object by other objects can explain the error detection. This occurs in many ways in public road traffic, e.g. when buses or trucks cover up street signs. In this case, the error identification could be explained and no data may be archived or only a reduced data set (e.g. only a few seconds before the first successful object identification).

• • Genauigkeit der Positionsbestimmung des Fahrzeuges und/oder Objektes
• • Uhrzeit, insbesondere mit Tag/Nacht-Unterscheidung
• • Wetterlage
• • Verkehrslage
• • Fahrbahnbeschaffenheit.

Dies ist immer dann von Vorteil, wenn die Umgebungsbedingungen nicht immer gleich sind. In der Praxis ist es eigentlich immer so, dass die Umgebungsbedingungen ständig wechseln. Wenn immer die gleiche Zahl der Bilder oder Bildteile archiviert werden, könnte es bei wechselnden Umgebungsbedingungen leicht passieren, dass gerade wichtige Szenen verpasst werden und so die Ursachen der Probleme nicht gefunden werden können.In addition, an advantageous variant consists in determining the size of the image sections to be archived and / or the recording period of the images / image sections to be archived as a function of one or more of the following environmental parameters:

• • Accuracy of the position determination of the vehicle and / or object
• • Time, especially with a day / night distinction
• • Weather condition
• • Traffic situation
• • Condition of the road.

This is always an advantage when the ambient conditions are not always the same. In practice it is actually always the case that the ambient conditions change constantly. If the same number of images or image parts are always archived, it could easily happen with changing ambient conditions that important scenes are missed and the causes of the problems cannot be found.

For a device for use in the method according to one of the preceding claims, it is advantageous if the device has an image generation device, a computing device and a storage device. The computing device is designed in such a way that it can perform object recognition on the images supplied by the image generation device. The computing device then also has the task of determining whether the object was recognized correctly in terms of time or location. In this case, the computing device is further designed to determine which images or image parts are to be archived for more precise checking upon detection of a discrepancy in object recognition beyond a tolerance limit, and to issue a command to archive these determined images or image parts.

In a particularly advantageous variant, the device additionally has a communication module and the communication module is designed to send the determined images or image parts to an external archiving point after receiving the command to archive these determined images or image parts. The external archiving agency can be a data center of the manufacturer of the image evaluation system or the vehicle.

In addition, the variant is proposed in which the device has a storage device and the storage device is designed to store the determined images or image parts after receiving the command to archive these determined images or image parts. With this variant, the archiving takes place in the vehicle until the data is read out. As described above, the readout could take place in a workshop or the data could be transmitted to the external archiving center via the home WLAN network on return.

For the intermediate storage of the data in the vehicle, it is advantageous if the storage device is designed as a ring memory in which the oldest previously stored images or image parts are overwritten by the new images or image parts in the event of a memory overflow.

A video camera or a LIDAR or a RADAR sensor can advantageously be used as the image generating device. For the interface for wireless communication it is advantageous to use an interface according to at least one of the communication systems WLAN according to a standard of the IEEE 802.11 standard family or an LTE or 5G mobile radio communication system according to a 3GPP standard.

For a vehicle to be used in the method, it is advantageous if the vehicle is equipped with a device that corresponds to the proposed device.

For a computer program that is processed in a computing device in order to carry out the steps for capturing image material using the method according to the invention, the corresponding advantages as described for the method according to the invention apply.

Embodiments of the invention are shown in the drawings and are explained in more detail below with reference to the figures.

• 1 das typische Cockpit eines Fahrzeuges;
• 2 eine Prinzip-Darstellung der verschiedenen Kommunikations-Möglichkeiten, die in einem Fahrzeug vorgesehen werden;
• 3 ein Blockschaltbild der Bordelektronik des Fahrzeuges;
• 4 eine erste Darstellung einer Fahrsituation zur Erläuterung der Problematik bei der Überprüfung der Funktion eines Bildauswertesystems des Fahrzeuges;
• 5 eine zweite Darstellung einer Fahrsituation zur Erläuterung der Problematik bei der Überprüfung der Funktion eines Bildauswertesystems des Fahrzeuges;
• 6 ein Flussdiagramm für ein Programm zur Erfassung von Bildmaterial für die Überprüfung der Funktion des Bildauswertesystems;
• 7 eine Darstellung der Variation der Bildqualität zur Archivierung von Bildmaterial in Abhängigkeit von den Umgebungsbedingungen; und
• 8 eine Darstellung der Variation der Aufzeichnungsdauer zur Archivierung von Bildmaterial in Abhängigkeit von den Umgebungsbedingungen.

Show it:

• 1 the typical cockpit of a vehicle;
• 2 a principle illustration of the various communication options that are provided in a vehicle;
• 3 a block diagram of the vehicle's on-board electronics;
• 4th a first representation of a driving situation to explain the problem when checking the function of an image evaluation system of the vehicle;
• 5 a second representation of a driving situation to explain the problem when checking the function of an image evaluation system of the vehicle;
• 6th a flow chart for a program for acquiring image material for checking the function of the image evaluation system;
• 7th a representation of the variation in the image quality for archiving image material as a function of the environmental conditions; and
• 8th a representation of the variation of the recording time for archiving image material depending on the environmental conditions.

The present description illustrates the principles of the disclosure of the invention. It is therefore understood that those skilled in the art will be able to design various arrangements which, although not explicitly described here, embody principles of the disclosure according to the invention and are also intended to be protected in their scope.

1 shows the typical cockpit of a vehicle 10 . A passenger car is shown. As a vehicle 10 However, any other vehicles could also be considered. Examples of further vehicles are: buses, utility vehicles, in particular trucks, trucks, agricultural machines, construction machines, rail vehicles, etc. The use of the invention would generally be possible in land vehicles, rail vehicles, water vehicles and aircraft.

Two display units of an infotainment system are shown in the cockpit. It is a touch sensitive screen 30th which is mounted in the center console and the instrument cluster 110 , which is attached to the dashboard. When driving, the center console is not in the driver's field of vision. Therefore, the additional information is not shown on the display unit while driving 30th faded in.

The touch-sensitive screen 30th is used in particular to operate functions of the vehicle 10 . For example, a radio, a navigation system, a playback of stored pieces of music and / or an air conditioning system, other electronic devices or other comfort functions or applications of the vehicle can be used 10 being controlled. In summary, an "infotainment system" is often spoken. In motor vehicles, especially passenger cars, an infotainment system refers to the combination of car radio, navigation system, hands-free system, driver assistance systems and other functions in a central operating unit. The term infotainment is a suitcase word, composed of the words information and entertainment (entertainment). The touch-sensitive screen is mainly used to operate the infotainment system 30th ("Touchscreen") is used, this screen 30th in particular by a driver of the vehicle 10 , but also from a passenger in the vehicle 10 can be easily viewed and operated. Below the screen 30th In addition, mechanical control elements, for example buttons, rotary controls or combinations thereof, such as rotary push-buttons, can be used in an input unit 50 be arranged. Typically, parts of the infotainment system can also be operated using the steering wheel. This unit is not shown separately, but is part of the input unit 50 considered.

1 also shows a head-up display 20th . The head-up display 20th is in the vehicle 10 from the driver's point of view behind the instrument cluster 110 attached in the dashboard area. It is an image projection unit. Additional information is displayed in the driver's field of vision by being projected onto the windshield. This additional information appears as if it were on a projection surface 21st at a distance of 7-15 m in front of the vehicle 10 projected. Through this projection surface 21st but the real world remains visible through it. A virtual environment is created with the additional information displayed. The virtual environment is theoretically superimposed on the real world and contains the virtual objects that support and inform the driver while driving. However, it is only projected onto part of the windshield, so that the additional information cannot be arranged anywhere in the driver's field of vision. This type of representation is also known under the term “augmented reality”.

2 shows a system architecture for vehicle communication using cellular radio. The vehicles 10 come with an on-board communication module 160 Equipped with a corresponding antenna unit so that it can participate in the various types of vehicle communication V2V and V2X. 1 shows that the vehicle 10 with the cellular base station 210 of a mobile network provider can communicate.

Such a base station 210 can be an eNodeB base station from an LTE cellular network provider (Long Term Evolution). The base station 210 and the corresponding equipment is part of a cellular communications network comprising a plurality of cellular cells, each cell from a base station 210 is served.

The base station 210 is positioned near a main road on which the vehicles 10 drive. In LTE terminology, a mobile terminal corresponds to a user equipment UE that enables a user to access network services, connecting to the UTRAN or the Evolved UTRAN via the radio interface. Typically, such user equipment corresponds to a smartphone. Such mobile terminals are used by the passengers in the vehicles 10 used. In addition, the vehicles are 10 each with an on-board communication module 160 fitted. This on-board communication module 160 corresponds to an LTE communication module with which the vehicle 10 can receive mobile data (downlink) and send such data in the uplink direction (uplink). This on-board communication module 160 can also be equipped with a WLAN p-module in order to be able to participate in an ad hoc V2X communication mode. V2V and V2X communication is also supported by the new 5th generation of mobile radio systems. There the corresponding radio interface is referred to as the PC5 interface. With regard to the LTE mobile radio communication system, LTE's Evolved UMTS Terrestrial Radio Access Network E-UTRAN consists of several eNodeBs that provide the E-UTRA user level (PDCP / RLC / MAC / PHY) and the control level (RRC) . The eNodeBs are connected to one another using the so-called X2 interface. The eNodeBs are also connected to the EPC (Evolved Packet Core) via the so-called S1 interface 200 connected.

From this general architecture shows 2 that the base station 210 via the S1 interface with the EPC 200 connected and the EPC 200 with the Internet 300 connected is. A backend server 320 to which the vehicles 10 Being able to send and receive messages is also with the Internet 300 connected. In the case under consideration, the backend server 320 be housed in a data center of the vehicle manufacturer. Finally, there is also a road infrastructure station 310 shown. This can be done, for example, by a roadside unit, often referred to in technical jargon as the Road Side Unit RSU 310 will be illustrated. To simplify the implementation, it is assumed that all components have been assigned an Internet address, typically in the form of an IPv6 address, so that the packets that transport messages between the components can be routed accordingly. The various interfaces mentioned are standardized. In this regard, reference is made to the corresponding LTE specifications that are published.

3 shows a schematic block diagram of the on-board electronics 200 to which also the infotainment system of the vehicle 10 heard. The touch-sensitive display unit is used to operate the infotainment system 30th , a computing device 40 , an input unit 50 and a memory 60 . The display unit 30th comprises both a display area for displaying variable graphic information and a user interface (touch-sensitive layer) arranged above the display area for the input of commands by a user. It can be designed as an LCD touchscreen display.

The display unit 30th is via a data line 70 with the computing device 40 connected. The data line can be designed according to the LVDS standard, corresponding to low voltage differential signaling. Via the data line 70 receives the display unit 30th Control data for controlling the display area of the touch screen 30th from the computing device 40 . Via the data line 70 are also control data of the commands entered from the touchscreen 30th to the computing device 40 transfer. With the reference number 50 the input unit is designated. Associated with it are the control elements already mentioned, such as buttons, rotary controls, slide controls or rotary push-button controls, with the help of which the operator can make entries via the menu navigation. Input is generally understood to mean the selection of a selected menu option, as well as changing a parameter, switching a function on and off, etc.

The storage device 60 is via a data line 80 with the computing device 40 connected. In the memory 60 a list of pictograms and / or symbols is stored with the pictograms and / or symbols for the possible insertion of additional information.

The other parts of the infotainment system camera 150 , Radio 140 , Navigation device 130 , Phone 120 and instrument cluster 110 are via the data bus 100 connected to the device for operating the infotainment system. As a data bus 100 the high-speed variant of the CAN bus follows ISO standard 11898-2 into consideration. Alternatively, for example, a bus system based on Ethernet technology such as BroadR-Reach could also be used. Bus systems in which data is transmitted via fiber optics can also be used. Examples are the MOST bus (Media Oriented System Transport) or the D2B bus (Domestic Digital Bus). To the data bus 100 is also a vehicle measuring unit 170 connected. This vehicle measuring unit 170 is used to record the movement of the vehicle, in particular to record the accelerations of the vehicle. It can be designed as a conventional IMU unit, corresponding to the inertial measurement unit. In an IMU unit there are typically acceleration sensors and rotation rate sensors such as a laser gyroscope or a magnetometer gyroscope. The vehicle measuring unit 170 can be used as part of the vehicle's odometry 10 be considered. But this also includes the wheel speed sensors.

It is also mentioned here that the camera 150 can be designed as a conventional video camera. In this case, it records 25 frames / s, which is the interlace recording mode 50 Corresponds to fields / s. Alternatively, a special camera can be used that records more images / s in order to increase the accuracy of the object recognition in faster moving objects, or that records light in a spectrum other than the visible. Several cameras can be used to monitor the surroundings. In addition, the RADAR or LIDAR systems already mentioned could also be used in addition or as an alternative to carry out or expand the observation of the surroundings. The vehicle is responsible for wireless internal and external communication 10 with the communication module 160 equipped, as already mentioned.

The camera 150 is mainly used for object recognition. Typical objects to be recognized are traffic signs, vehicles driving ahead / surrounding / parked and other road users, intersections, turning points, potholes, etc. If an object with a potential meaning has been recognized, information can be output via the infotainment system. Typically, for example, symbols for the recognized traffic signs are displayed. It can also be a warning if the object poses a hazard. About the HUD 20th the information is projected directly into the driver's field of vision. As an example of an object that poses a danger, the case is mentioned that the image evaluation is carried out by the camera 150 The images provided showed that a deer is standing on the edge of the road. The image evaluation takes place in the processing unit 40 instead of. Known algorithms for object recognition can be used for this purpose. As a result, a danger symbol is displayed at the position of the deer. The display should be done in such a way that the display of the danger symbol does not cover the deer, otherwise the driver cannot see exactly what the danger is.

The object recognition algorithms are used by the processing unit 40 processed. It depends on the performance of this processing unit depends on how many images can be analyzed per second.

With the reference number 151 is called an engine control unit. The reference number 152 corresponds to an ESP control unit and the reference number 153 refers to a transmission control unit. Other control units, such as an additional driving dynamics control unit (for vehicles with electrically adjustable dampers), airbag control units, etc., can be present in the vehicle. The networking of such control devices, all of which are assigned to the drive train category, typically takes place with the CAN bus system (Controller Area Network) 104, which is standardized as an ISO standard, mostly as ISO 11898-1 . For different sensors 161 to 163 In motor vehicles, which are no longer to be connected only to individual control units, it is also provided that they are connected to the bus system 104 connected and whose sensor data are transmitted to the individual control units via the bus. Examples of sensors in motor vehicles are wheel speed sensors, steering angle sensors, acceleration sensors, yaw rate sensors, tire pressure sensors, distance sensors, knock sensors, air quality sensors, etc. In particular, the wheel speed sensors and the steering angle sensors are included in the odometry of the vehicle. The acceleration sensors and yaw rate sensors can also be connected directly to the vehicle measuring unit 170 be connected.

The 4th and 5 show the basic functionality of an image evaluation system as it is used in vehicles. What is shown is the camera image from the front camera 150 of the vehicle 10 is recorded. The example of traffic sign recognition was chosen. 4th shows the camera image from a close distance to the traffic sign 15th . 5 shows the camera image from a further distance to the traffic sign 15th . It is a traffic sign 15th , with which an absolute no overtaking is indicated. For the design of the traffic sign recognition system it is prescribed that when the vehicle approaches 10 at the traffic sign 15th Under standard conditions, recognition should take place at a standard recognition distance of 80 m from the traffic sign. 4th shows the camera image for the situation in which the approach took place up to a distance of 20 m. In 5 is the situation shown where the vehicle 10 40 m from the traffic sign 15th away. The image evaluation shows that the traffic sign recognition only took place at a distance of 40 m. The checking function of the image evaluation system determines that the traffic sign recognition occurred too late under the given conditions. The checking function then carries out a test with which a preliminary evaluation of the recorded images takes place. As shown, this shows that a truck driving ahead 12th can be seen in the camera image. As in 5 can be seen is the road sign 15th immediately to the right of the truck 12th to see. From this, the verification function concludes that the traffic sign 15th was covered by the truck at a further distance. For this reason, the images recorded are archived with lower quality at a distance of 80 m up to the actual traffic sign recognition at a distance of 40 m.

The 4th and 5 also serve to explain another false detection situation. The traffic sign was initially recognized at a distance of 40 m 15th for a restricted no-overtaking ban that applies to trucks. At a distance of 20 m from the traffic sign 15th was only then recognized that it was the traffic sign 15th acts for an absolute ban on overtaking. In this situation, the verification function reacts as follows: The area marked by the background 17th of the recorded camera image is archived with high quality for the images to be archived. The remaining parts of the images are recorded with lower quality. This applies to all pictures taken from a distance of 40 m up to a distance of 20 m.

6th shows a flow diagram for a program with which the checking function is implemented. This variant of the program is intended for the second variant explained, in which the road sign was initially identified incorrectly and the road sign was only correctly identified when the vehicle was approached further. With the reference number 310 the program start is designated. In the program step 312 the processing of the algorithm for traffic sign recognition takes place. It is an object recognition algorithm that recognizes patterns based on patterns stored in a table. All valid traffic signs are known and their patterns can be saved in a table. Typically, the pattern recognition is improved by a convolution operation in which the recorded images are folded with the known patterns. Such algorithms are known to the person skilled in the art and are available. If a traffic sign has been recognized in this way, the recognition distance to the traffic sign is also determined and stored.

In the program step 314 it is checked whether the traffic sign recognition was standardized or not. In the case under consideration, the traffic sign recognition was defective for two reasons. Initially, the detection did not take place within the standard detection distance of 80 m from the traffic sign 15th . Furthermore, a restricted overtaking ban was initially recognized, which was revised as it got closer to an absolute no-overtaking ban. If in this program step a standard-compliant traffic sign recognition is determined, no images need to be archived for control and the program ends in step 322 . In the case under consideration, however, incorrect traffic sign recognition was recognized. Then the program with program step 316 continued. In program step 316 the number of problematic images is calculated. Because there is a deviation from the standard recognition distance, the calculation can be carried out as follows. The first detection took place at a distance of 40 m. The correct detection only took place at a distance of 20 m. We are therefore interested in the images from the standard recognition distance of 80 m up to a distance of 20 m.

In the program step 316 an additional check is made as to whether the error identification can be explained. It is determined by image analysis that a truck 12th drives ahead and this is probably the traffic sign 15th covered up to the first detection. It is therefore concluded that the images in the distance range from 80 m to 40 m are less important for the subsequent verification. These images are therefore saved with a lower quality.

In the program step 318 the problematic image details are calculated. In the case of traffic sign recognition, the 4th and 5 marked image section around the traffic sign 15th around selected as relevant. This image section is therefore selected with high image quality in the range from a distance of 40 m to 20 m.

In the program step 320 the problematic images and the problematic image details are then transmitted from the memory 60 to the communication module 160 and the communication module 160 sends this image data to the backend server via cellular network 320 . They are archived there. The archived images are later evaluated by experts in the data center or by machines with artificial intelligence. The purpose of the check is to identify possible problems in the image evaluation system. The results of the checks can be used to improve the evaluation algorithms. Ideally, the improved evaluation algorithm can be downloaded to the vehicle via OTA (Over the Air) 10 transferred back and installed there. However, the result can also be that the check reveals a systematic error due to incorrect adjustment of the camera 150 is based. In that case a message could be sent to the vehicle 10 to inform the driver to go to the workshop. The program ends in the program step 322 .

The number and quality of the images to be archived can be influenced by various factors. In addition to the cases already presented, the following influencing factors are mentioned in particular:

Weather conditions

In adverse weather conditions, visibility is severely restricted. This can go so far that object detection is no longer possible. However, the other environment detection sensors RADAR sensor and LIDAR sensor can deliver better results here. In any case, it can be provided that, in adverse weather conditions, the number of images or image parts to be archived is restricted because the corresponding objects cannot be detected from the standard recognition distance.

Positional accuracy

The accuracy of the position determination based on GNSS signals and odometry signals can also depend on the weather. But it can also depend on other influencing factors. The environment in which the vehicle is moving is cited as an example. In the city, the reception of satellite signals can be restricted due to heavy buildings. This can also be the case when driving overland. The satellite signals can be weakened in forests. Due to the tectonics, poor reception can also prevail in the mountains. In such cases, it is therefore suggested that more images or parts of images be recorded. Then it is more likely that the relevant route sections were recorded despite the positional inaccuracy.

Road surface condition

The road surface can also have a similar influence. There are strong vibrations on cobblestones, so that the recorded images may be blurred. When it is slippery due to ice, snow or rain, the drive slip control determines an estimate of the coefficient of friction. If the slip is correspondingly large, the odometry data are no longer as reliable and this influence should be taken into account as with the influence due to position inaccuracies.

Time

It can be seen that the image quality will vary greatly with the time of day. A distinction should at least be made between daytime and nighttime. At night, the data from RADAR or LIDAR sensors are recorded as the camera image data.

Traffic conditions

A distinction can be made here as to whether the vehicle is moving in city traffic, on the motorway, on the country road, etc. In city traffic, it is particularly important to record the traffic signs precisely. The number of images to be recorded can be increased here. In a traffic jam on the motorway, however, the number of images to be recorded can be reduced.

The images or parts of images to be archived are temporarily stored in the vehicle 17th . The memory 60 can be used. In practical implementation, in the memory 60 a ring buffer set up. This is managed in such a way that the newly recorded images are written one after the other into the free memory area of the ring buffer. When the end of the free memory area is reached, the part of the ring buffer already written is overwritten. By managing the allocated memory area as a ring memory, the oldest part of the memory is always overwritten. The pictures or parts of the picture 17th can be saved uncompressed or compressed. However, it is recommended to use a lossless compression process so that any relevant image content cannot be lost. The FFmpeg codec is mentioned as an example of a lossless compression method. The image data are saved in a corresponding file format. Various data container formats come into question here. The ADTF format, which was developed for the automotive sector, and the TIFF format are cited as examples. Other container formats for storing image and sound data are MPEG, Ogg, Audio Video Interleave, DIVX, Quicktime, Matroska, etc. The images become the backend server during the journey 320 transmitted, it may be sufficient to design the ring buffer for a recording duration of 20 s. A separate memory can also be provided in the vehicle on which the images are archived. A USB hard drive, for example, is ideal here, which can be removed in order to connect it to a computer that is used to evaluate the archived images.

7th also shows the effect of how the amount of data to be archived can be reduced by reducing the image quality. From top to bottom, the amount of image data for the various image qualities is displayed: Full HD with a resolution of 1920 X 1080 pixels, SD with a resolution of 640 X 480 pixels and VCD with a resolution of 352 X 288 pixels.

The 8th then shows the effect of the variation in the number of images or image parts 17th depending on the position accuracy or the weather conditions, etc. If the recording time is increased to 160 m due to position inaccuracy, twice as much data is required for archiving. When reducing to a distance of 40 m, only half as many.

All examples mentioned herein as well as conditional formulations are to be understood without restriction to such specifically cited examples. For example, it will be appreciated by those skilled in the art that the block diagram presented herein is a conceptual view of exemplary circuitry. In a similar way, it can be seen that a depicted flowchart, state transition diagram, pseudocode and the like represent different variants for the representation of processes that can essentially be stored in computer-readable media and thus executed by a computer or processor. The object named in the patent claims can expressly also be a person.

It should be understood that the proposed method and associated devices can be implemented in various forms of hardware, software, firmware, special purpose processors, or a combination thereof. Specialty processors can include application-specific integrated circuits (ASICs), reduced instruction set computers (RISC), and / or field programmable gate arrays (FPGAs). The proposed method and the device are preferably implemented as a combination of hardware and software. The software is preferably installed as an application program on a program storage device. Typically, it is a computer platform-based machine that includes hardware such as one or more central processing units (CPU), random access memory (RAM), and one or more input / output (I / O) interfaces. An operating system is also typically installed on the computer platform. The various processes and functions described here can be part of the application program or a part that is executed by the operating system.

The disclosure is not restricted to the exemplary embodiments described here. There is room for various adaptations and modifications which those skilled in the art, based on their expert knowledge as well as belonging to the disclosure, would consider.

The invention is explained in more detail in the exemplary embodiments using the example of use in vehicles. This also applies to the Possible use in airplanes and helicopters, for example during landing maneuvers or search missions etc.

The invention can also be used for remote-controlled devices such as drones and robots, where image evaluation is very important. Other possible uses concern a smartphone, a tablet computer, a personal assistant or data glasses.

List of reference symbols

10

vehicle

12

truck

15th

Traffic sign

17th

Image part

20th

Head-Up Display

30th

touch-sensitive display unit

40

Arithmetic unit

50

Input unit

60

Storage unit

70

Data line to the display unit

80

Data line to the storage unit

90

Data line to the input unit

100

Data bus

110

Instrument cluster

120

phone

130

navigation device

140

radio

150

camera

160

Communication module

170

Vehicle measuring unit

200

Evolved Packet Core

310-322

different program steps of a computer program

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• US 2016/0170414 A1 [0014]
• US 2017/0069206 A1 [0015]
• US 2018/0260639 A1 [0016, 0021]

Non-patent literature cited

• ISO Standard 11898-2 [0052]
• ISO 11898-1 [0056]

Claims (15)

A method for capturing image material for checking image evaluation systems, whereby it is checked whether an object was recognized by the image evaluation system correctly in terms of time or location, characterized in that when a deviation beyond a tolerance limit is recognized, it is determined which images or image parts (17) are to be used should be archived for a more precise examination and the archiving of these determined images or image parts (17) is carried out. Procedure according to Claim 1 , wherein an object recognition algorithm is used for image evaluation and wherein, to check whether an object recognition has taken place correctly, it is checked at what distance to the recognized object (15) the object recognition took place, a standard recognition distance being established which indicates the distance from which the object recognition algorithm is Object recognition should deliver and, if there is a deviation from this standard recognition distance, the images or image parts (17) that were recorded from the standard recognition distance to the object recognition distance are archived for more detailed checking. Procedure according to Claim 1 or 2 , the image evaluation system being an image evaluation system for a vehicle (10) and the position data of the vehicle (10) and the position data of the object (15) being evaluated to determine a distance between the vehicle (10) in order to detect a discrepancy in the image recognition and object (15). Procedure according to Claim 3 , the position data of the vehicle (10) and of the object (15) being determined by satellite navigation and / or odometry. Method according to one of the preceding claims, wherein the images or image parts (17) which were recorded from the standard recognition distance to the object recognition distance are sent to an external archiving location (320). Method according to one of the preceding claims, wherein the images or image parts (17) recorded from the standard recognition distance to the object recognition distance are archived in a storage unit (60) arranged in the vehicle (10). Method according to one of the preceding claims, wherein further images or image parts (17) are archived and wherein the images or image parts (17) that were recorded from the standard recognition distance to the object recognition distance are archived with high quality and the further images or other image parts are archived as well of a lower quality can be archived. Method according to one of the preceding claims, wherein the size of the image sections to be archived and / or the recording period of the images / image parts (17) to be archived is determined as a function of one or more of the following environmental parameters: • Accuracy of the position determination of the vehicle and / or object • Time, especially with a day / night distinction • Weather condition • Traffic conditions • Condition of the road. Device for use in the method according to one of the preceding claims, comprising an image generation device, a computing device (40) and a storage device (60), the computing device (40) being designed to perform and determine object recognition in the images supplied by the image generation device whether the object recognition was carried out correctly in terms of time or location, characterized in that the computing device (40) is designed to determine which images or image parts (17) are to be archived for more precise checking when a discrepancy in object recognition is recognized beyond a tolerance limit To give the command to archive these identified images or image parts (17). Device according to Claim 9 , wherein the device has a communication module (160) and the communication module (160) is designed to send the determined images or image parts (17) to an external archiving location (320) after receiving the command to archive these determined images or image parts (17) . Device according to Claim 9 wherein the device has a storage device (60) and the storage device (60) is designed to archive the determined images or image parts (17) after receiving the command to archive these determined images or image parts (17). Device according to Claim 11 , wherein the storage device (60) is designed as a ring memory in which the oldest previously stored images or image parts (17) are overwritten by the new images or image parts (17) in the event of a memory overflow. Device according to one of the Claims 8 to 12th , wherein the image generation device is a video camera (150) or a LI DAR or a RADAR sensor and wherein the communication module (160) is for wireless communication according to at least one of the communication systems WLAN is designed according to a standard of the IEEE 802.11 standard family or an LTE or 5G mobile radio communication system according to a 3GPP standard. Vehicle for use in the method according to one of the Claims 1 to 8th , characterized in that the vehicle (10) with a device according to one of Claims 8 to 13 Is provided. Computer program, characterized in that the computer program is designed, when processed in a computing device (40), to carry out the steps of the method according to one of the Claims 1 to 8th perform.

Images