DE102022201730A1 - Method and assistance system for supporting vehicle guidance based on natural language processing and motor vehicles - Google Patents

Abstract

The invention relates to a method and an assistance system (7) for supporting vehicle guidance and a corresponding motor vehicle (1). In the method, a camera (6) of the respective vehicle (1) records image data. A sign (12) on which traffic-guiding text (13) is depicted is then identified by processing this image data. The text (13) is processed using natural language processing and its semantic content is thereby determined. Based on this, the relevance for driving the vehicle is determined. If the sign (12) is relevant for driving the vehicle, a driver of the vehicle (1) is informed and/or the driving of the vehicle is adjusted accordingly.

Description

The present invention relates to a method and an assistance system for supporting vehicle guidance. The invention further relates to a correspondingly equipped motor vehicle.

In large cities, for example, complex driving or traffic situations can arise. This may be the case, for example, at blind intersections, in areas with changed traffic routing, in combination with tram or bus traffic, potentially distracting commercial signage on a development along a respective street and/or the like. Such situations can require a particularly high degree of concentration, attention and prudence on the part of a driver of a vehicle and in particular can overwhelm drivers who are unfamiliar with the area and lead to errors in the correct choice of route and accordingly to accidents. In order to counteract this problem, corresponding traffic-directing signs are often set up, which can also be problematic with a large number of such signs, since there is not always enough time in the flow of traffic to reliably read and understand or interpret all signs and in good time respond to information conveyed by the signs. This can be particularly problematic when the signs are in a language foreign to the driver. If vehicle guidance can be supported automatically in such situations, this can result in a reduction in stress for the driver and an overall improvement in traffic safety.

As one approach, the U.S. 7,386,437 B2 a system and method for providing translated information to a driver of a vehicle with a navigation system. In the method, surroundings of the vehicle are monitored with a video camera to detect informational signs. An input language is determined based on the current position of the vehicle. When a detected sign is placed along a route of the vehicle, it is translated from the input speech for the driver.

As a further approach, the DE 20 2011 100 069 U1 a traffic sign recognition system for use in a motor vehicle. The system is set up to record images with traffic signs to be recognized through a windshield of the motor vehicle, to compensate for distortions in the recorded image caused by this, and finally to recognize traffic signs in the recorded images. In this way, traffic-related information is to be offered to the vehicle occupants of the motor vehicle, with the extensive use of components that are generally and inexpensively available.

The object of the present invention is to automatically support vehicle guidance.

According to the invention, this object is achieved by the subject matter of the independent patent claims. Possible refinements and developments of the present invention are disclosed in the dependent patent claims, in the description and in the figure.

The method according to the invention is used to support vehicle guidance. This means that at least partially automated guidance of the vehicle can be supported or a driver or guide of the vehicle can be supported in its manual guidance. Such a vehicle can be a motor vehicle, in particular a passenger car, although the present invention is not limited to this. The method according to the invention or its method steps explained below can be carried out automatically, for example by an assistance system set up for this purpose.

In a method step of the method according to the invention, image data are recorded by means of a camera of the vehicle, which depict a respective environment of the vehicle lying ahead in the direction of travel of the vehicle. In the present sense, the environment ahead in the direction of travel includes not only areas that the vehicle will actually drive through if it continues to move, but can also extend conically forward in the direction of travel from the current position of the vehicle, i.e. also in the direction of travel ahead and to the side of include areas lying on the vehicle or its trajectory. The image data can be or include camera images, an image stream or a video stream and accordingly map traffic signs, signs and the like arranged in the area surrounding the vehicle. During operation of the vehicle, such image data can be recorded continuously or regularly, for example with a predetermined recording frequency, which can be processed as described below.

In a further method step of the method according to the invention, at least one sign is identified by processing the image data—if present, i.e. depicted therein, on which a traffic-directing, ie traffic-relevant or relevant for driving the vehicle text is depicted. Such a text is to be understood broadly in the present case, so it can, for example comprise one or more complete or incomplete sentences or be a discontinuous text or the like. Likewise, a single word or several single words, which, for example, conveys a reference or the like, can be regarded as text in the present sense. In particular, however, a text in the present sense can include at least one word, ie, for example, not only consist of numbers or arrows or the like. However, such numbers, arrows or other symbols that are shown on the respective sign together with at least one word can be understood or processed as part of the text.

In particular, a temporary, regular or standardized traffic sign or sign supplementing or non-standard sign can be identified here. Such signs may often contain text or information that is temporary or only specific to the location or relevant. Such signs may be unfamiliar to drivers and accordingly may require more time and attention to interpret, or may not be as intuitive or as quick to grasp as standard road signs. Thus, in situations with such signs, there may be an increased need for support in order to achieve or enable the safest possible vehicle or traffic guidance.

In a further method step of the method according to the invention, the text is processed by means of natural language processing, ie by means of automated or machine processing of natural language (English: Natural Language Processing, NLP). As a result or in the process, a semantic content, i.e. the meaning of the text, is determined. In other words, an automatic recognition or extraction of the meaning indicated by the respective label can be carried out. Based on this, the relevance of the sign or the text or its semantic content for driving the vehicle in the current situation is determined. For example, the sign can only be relevant or valid at certain times, for certain vehicles, for certain people, for one of several possible routes starting from the location of the sign and/or the like. Here, not only is information given by the sign determined, but it is also processed further semantically or checked with regard to its relevance. Predetermined data which, for example, characterize the respective vehicle, its driver and/or a current situation can be taken into account. Such data can be stored, for example, in a computer-readable data memory of the vehicle or of the assistance system or the like used to carry out the method.

If or when the sign or the information it conveys is relevant for driving the respective vehicle at the time the vehicle is in the area of the sign, the driver of the vehicle is informed and/or the vehicle guidance is adjusted accordingly . This can therefore be done automatically or performed when the sign is appropriate for the vehicle according to its route, type and/or occupants, according to the current situation and/or the like or - depending on the implementation of the present invention - for example, when the determined relevance is greater than a predefined relevance threshold value.

In order to draw the driver's attention to the sign or its meaning or semantic content, for example, a corresponding display can take place in an instrument cluster, infotainment display or head-up display (HUD) of the vehicle and/or a corresponding voice output. An image or a representation of the sign, the text given on it or the semantic content can be output, optionally in a processed, adapted or abbreviated form. A corresponding adaptation of the vehicle guidance can mean or include, for example, a functional consideration of the semantic content, ie a reaction thereto in the form of a corresponding control or control measure or the like. In other words, according to the semantic content of the sign, for example, a navigation device for situation-specific route guidance, an assistance system for finding a suitable parking or parking space for the vehicle, a lighting device, a radio, a speed of the vehicle and/or the like can be controlled automatically or be adjusted.

Informing the driver about the sign or its semantic content can enable him to record the meaning of the sign in question in a particularly reliable and simplified manner. This can be supported, for example, by translating or converting the sign or its semantic content in a modified form, for example into predefined symbols familiar to the driver, a language familiar to the driver or an alphabet familiar to the driver, or the like, a corresponding display for one is displayed for a longer period of time and can therefore be studied by the driver longer than the respective actual sign is possibly visible, a voice output capturing the semantic content or the meaning of the sign also allows when the driver does not have the sign in view or cannot see it and/or the like. In addition, the present invention relieves the driver of the task or trouble of detecting whether a sign he has seen is relevant at all in the current situation or not. This allows the driver to focus on the relevant signs, or capturing and considering their central semantic content or meaning. Overall, the present invention can thus support the driver in driving the vehicle in particularly stressful or high-risk situations. In this way, the driver can be relieved, especially in particularly complex traffic situations, since he can be made aware of important, ie relevant, information in a targeted manner. Likewise, the reliability, robustness and safety of an at least partially automated vehicle guidance, ie a corresponding function of one or more further assistance systems of the vehicle, can be supported or improved.

The present invention makes it possible in particular to also take account of traffic-related information that is not displayed or conveyed via classic or standardized signs or traffic signs. Such information is often not reliably recognized by conventional assistance systems for traffic sign recognition. Another advantage is that the vehicle guidance can be supported here in a way that is particularly adapted to the situation, for example even if the information conveyed by the respective sign is not or not yet available via other channels, for example not in a navigation map or navigation database or via TMC Signal are transmitted or detected and / or the like more. This can occur, for example, when the sign is new or is only valid for a relatively short period of time, for example when the road is temporarily closed, when a traffic area is kept free for loading or unloading work for a short time, during construction work and/or the like.

The method according to the invention can be implemented relatively easily and with little effort and can also be retrofitted or used in existing vehicles, for example. For example, a camera that is already present in the vehicle for other functionalities can be used to record the image data and an existing control unit of the vehicle can be used to run appropriate software, ie a computer program that implements the method according to the invention. Likewise, for example, a mobile electronic device such as a smartphone or the like can be attached in or on the respective vehicle and used to carry out the method according to the invention. The method according to the invention can therefore also be used, for example, to support the guidance of vehicles that are not equipped with a camera or a suitable control device or the like, non-motorized vehicles, bicycles, handcarts and/or the like.

It can be provided that even if the respective sign is classified as not relevant, this is also displayed, represented or conveyed to the driver, for example by a corresponding visual and/or acoustic message or the like. This can also relieve the driver, since he then knows whether he has to pay attention to the sign or not, for example if he has overlooked the relevant information that is decisive for the relevance.

Likewise, if the respective sign is initially not classified as relevant, for example when it is first identified, data determined for this sign can be stored or temporarily stored. For example, such data can be or include the position of the sign, its area of validity, the determined text, its semantic content and/or the like. This data can, for example, be stored or temporarily stored, ie kept available, for at least a predetermined period of time and/or at least until the vehicle has moved at least a predetermined minimum distance from the sign. Furthermore, a position, a movement or a behavior of the vehicle or the driver can be checked or monitored continuously or regularly. Based on this, the relevance of the sign or its importance can then also be checked continuously or regularly. This is based on the recognition that the relevance of a sign can change depending on future driver decisions or future unforeseen changes in vehicle handling or with the passage of time. A sign that was initially classified as irrelevant can become relevant at a later point in time. At this later point in time, however, the sign may already be outside of the camera's field of vision or detection, so that it would then possibly not be identified again and then classified as relevant. Whenever a previously identified sign becomes relevant, an automatic response can be made as described, for example the driver can be informed or the vehicle guidance can be adjusted accordingly. For example, a sign next to a parking area may indicate only certain types of vehicles, such as Taxis or the like, declare reserved. If the vehicle is of a different type, the sign may not be considered relevant because it does not indicate a parking space that can be used by the vehicle, especially if a current navigation route of the vehicle passes the parking area and/or the vehicle or driver drives past the sign without braking and/or without driving into the parking area. However, for example, the driver may then unexpectedly brake and reverse in order to head for the parking area. Since this can or would lead to the vehicle being illegally parked in the parking lot, the sign then becomes more relevant. This can be detected automatically. Accordingly, the driver can then be made aware of the sign or its semantic content. A further improved support of the vehicle guidance can thus be achieved.

In one possible embodiment of the present invention, the processing of the image data and the text is carried out locally in the vehicle by means of a predetermined data processing device set up for this purpose. This data processing device can in particular be or include a control unit of the vehicle. Likewise, a mobile electronic device carried in the vehicle, for example coupled to the vehicle, such as a smartphone, a tablet, a computer or the like, can be used as the data processing device or as part thereof. The former can enable a particularly reliable and convenient use of the method according to the invention, while the latter can enable, for example, simpler software updating or the use of more or additional computing power for the method. Due to the local data processing proposed here in the vehicle, no data transmission, for example to a vehicle-external server device, is required. The method according to the invention can thus be used independently of a corresponding connectivity, for example in areas without mobile network coverage, in tunnels, abroad and the like. As a result, the reliability and availability of the method according to the invention or the corresponding assistance functionality can be ensured or improved. In addition, due to the local processing of the image data and the text in the vehicle, the method according to the invention can be carried out with less latency than when corresponding data has to be exchanged between the vehicle and a server device external to the vehicle, for example a backend, a cloud server, a data center or the like. As a result, the information to the driver or the adaptation of the vehicle guidance can take place particularly quickly, ie particularly early, which means that more time can be available for a reaction or implementation.

Image processing based on an optical flow can be used, for example, in order to reduce the computing effort, ie the need for hardware resources, for the application of the method according to the invention. This can be used particularly advantageously for local processing of the image data in the vehicle, since this requires a series or temporal sequence of image data and data that describes a trajectory of the vehicle can also be taken into account, for example, which would otherwise be sent to the server device external to the vehicle would increase the amount of data to be transmitted.

Nevertheless, the data processing described can alternatively or additionally be carried out entirely or partially by the vehicle-external server device. A greater computing power or computing capacity can be available there than in the vehicle, which can possibly enable faster, more precise or more robust processing of the corresponding data. In addition, this can make it possible to use the data in a fleet or a swarm of vehicles, to update server-side or central map data and/or the like.

In a further possible embodiment of the present invention, the image data is recorded as a time sequence, ie for example as a large number of individual images, as an image stream or as a video stream. For each newer part of the image data, it is then checked whether the text for a sign imaged therein has already been processed. If this is the case, no re-processing of the text is performed for this sign based on the newer parts of the image data. In other words, signs that have already been recognized and semantically processed can be filtered out, so that the corresponding double or multiple data processing effort, such as for letter recognition and for semantic text analysis, can be saved. However, other data or properties of the respective sign, such as position data or pixel coordinates, can continue to be determined, processed or updated again or again. This may, for example, enable correct continuous highlighting of the respective shield, such as displaying a frame or the like at the position of the shield in a video display, in a HUD and/or the like. It can be provided here that the processing of the text of a sign that has been identified several times or permanently can be repeated for so long fetch is carried out or attempted until a corresponding processing result, for example the automatic recognition of the text or the semantic content of the text, has been determined with at least a predetermined minimum confidence or the same processing result has been determined in at least a predetermined number of corresponding consecutive processing operations or analyses. It can thereby be taken into account, for example, that depending on the position of the sign relative to the vehicle or depending on the recording angle at which the sign or the corresponding image data is recorded, text recognition can have different degrees of accuracy or reliability. In this way, a particularly efficient application of the method or a compromise between efficiency and robustness can be implemented overall.

In a further possible embodiment of the present invention, when the semantic content is determined, a check is made as to whether the respective text contains a time specification which specifies or restricts a period of validity for the tag. If this is the case, the sign is only classified as relevant for driving the vehicle if the current time falls within this period of validity. Additionally or alternatively, when determining the semantic content, a check is made as to whether the text specifies a vehicle feature that restricts the validity of the sign to vehicles with this vehicle feature. If this is the case, the plate is only classified as relevant for driving the vehicle if the vehicle has or fulfills the specified vehicle characteristic. In the first case, the recognized text can be interpreted with regard to a date or a time, for example. For example, the text can indicate that a road or a passage or the like is only closed or passable between a first date or a first time and a second date or a second time, a stopping ban or a detour only applies in a certain period of time and/or or something like that. In the second case, corresponding vehicle features can relate to or specify, for example, a total weight, a vehicle type and/or the like. For example, the sign can only apply to cars, trucks, buses, motorcycles, bicycles or the like. For example, the sign can indicate or specify that a road or bridge or the like, in particular temporarily, for example during a repair or maintenance period or the like, is blocked for certain vehicles, such as correspondingly heavy vehicles, or for example during a possibly only temporarily changed traffic routing a passage is blocked for certain vehicles, such as bicycles, an overtaking ban only applies to certain vehicle types or only above a certain total vehicle weight and/or the like. Such detailed information that supplements, restricts or specifies the main information on the sign can be overlooked relatively easily by a driver of the vehicle, particularly in hectic, stressful or unclear traffic or driving situations. The driver would then possibly only record the main information on the sign without the supplement or specification specifying the period of validity or the vehicle feature relevant to the validity and then possibly act in violation of the rules or unnecessarily restrict or adapt his behavior.

In a further possible embodiment of the present invention, when the semantic content is determined, it is checked whether the text contains a direct, ie explicit, or indirect, ie implicit instruction. If this is the case and the sign is relevant to vehicle guidance, vehicle guidance is then automatically adjusted according to the instruction. In other words, the vehicle or a device of the vehicle can then be controlled automatically in order to follow, take into account or implement the instruction. A corresponding adjustment can be, for example, a navigation route, a setting for a windshield wiper, an air conditioning device, a de-icing system, a surface-specific setting or activation of a chassis, a brake system, a steering device or the like, for example for loose or slippery surfaces, a setting or selection of a radio station Activating or deactivating a lighting device and/or the like relate to, mean or include. This can enable a forward-looking execution of a corresponding adjustment, for example in contrast to conventional reactive sensors or assistance systems, which react accordingly later, for example when the ambient brightness - for example after entering a tunnel - has already fallen, a temperature change has already taken place, a braking or steering maneuver has not shown the desired success, the vehicle is already threatening to swerve and/or the like. In contrast, the embodiment of the present invention proposed here enables an earlier, anticipatory reaction, since corresponding signs typically indicate a corresponding situation or danger even before it occurs. It can be provided that the vehicle guidance is automatically adapted only if the semantic content was recognized with at least a predetermined minimum confidence. This can be surprising, unnecessary or unnerving to the driver avoid or reduce automatic interventions in vehicle guidance.

In a further possible embodiment of the present invention, repeated identical reference information is recognized in a number of signs identified one after the other. Based on this, an identification or indication of a route, in particular a detour or bypass, is then determined as its semantic content. Such reference notes can be or include, for example, diversion designations such as “U1” or a blue dot or the like. Corresponding signs can then be relevant because the driver or the vehicle appears to be following the corresponding route. The driver may then be alerted to the reference cues to make it easier for them to follow the route, or a navigation route may be automatically adjusted at a time. The embodiment of the present invention proposed here can be particularly useful since corresponding reference or route guidance information often occurs along a route that is unfamiliar to the driver or can be overlooked particularly easily in an unfamiliar environment.

In a further possible embodiment of the present invention, when the text is processed, its language and/or alphabet is identified, that is to say automatically recognized. For the notification to the driver, an automatic translation into a predefined language and/or a replacement of characters in the text with corresponding characters of a predefined alphabet is then carried out. This can make it easier for the driver, for example abroad, to grasp or check the meaning of the sign. For this purpose, the driver can specify the appropriate language or the appropriate alphabet, for example via an appropriate user interface of the vehicle or the appropriate assistance system. The text translated into the specified language and/or the specified alphabet can be displayed, for example, using a display device in the vehicle. This can give the driver the opportunity to read and understand the text in its entirety or, for example, to check his or her understanding of the meaning of the sign. Overall, this can reduce the driver's uncertainty when he is in a region with signs that are in a foreign language for him.

In a possible development of the present invention, the text is translated by means of a server device external to the vehicle. For this purpose, the image data or text data extracted therefrom in the original language of the sign can be sent automatically to the vehicle-external server device via an at least partially or partially wireless data connection, for example a mobile phone connection or the like. The automatic translation can then be carried out there. A corresponding translation result, ie in particular the translated text, can then be sent back to the vehicle from the vehicle-external server device via the data connection. The vehicle can then receive and output the translation result, ie in particular the translated text, or use it locally in order to draw the driver's attention to the respective sign or its semantic content or meaning. The translation of the text by means of or in the server device external to the vehicle can enable an improved translation quality, since more resources can then typically be used than are available in the vehicle.

Another aspect of the present invention is an assistance system for a vehicle. The assistance system according to the invention has a data processing device for image processing, text recognition and natural language processing. The assistance system according to the invention is set up to carry out the method according to the invention, in particular automatically. For this purpose, the assistance system according to the invention or its data processing device can have a processing device, for example a microchip, microcontroller or microprocessor or the like, and a computer-readable data memory coupled thereto. An operating or computer program can be stored in this data memory, which encodes or implements the method steps, sequences or measures or corresponding control instructions mentioned in connection with the method according to the invention. This operating or computer program can then be executed or be executable by the processor device in order to execute the corresponding method or to bring about its execution. The assistance system according to the invention can in particular be the assistance system mentioned in connection with the method according to the invention or correspond to it. Accordingly, the assistance system according to the invention can therefore be designed, for example, as a vehicle control unit or the like. The assistance system according to the invention can also include the camera mentioned, for example, and/or have a corresponding interface, for example for connection to an on-board network of a vehicle or the like.

Another aspect of the present invention is a motor vehicle, the at least one, ins has a special camera, which is aligned forward in the longitudinal direction of the vehicle, for recording or imaging the respective environment of the motor vehicle and an assistance system according to the invention. The motor vehicle according to the invention can in particular be or correspond to the vehicle mentioned in connection with the method according to the invention and/or in connection with the assistance system according to the invention. Accordingly, the motor vehicle according to the invention can have some or all of the properties and/or features mentioned in these contexts. For example, the motor vehicle can have an output device for outputting instructions to a driver of the motor vehicle and/or a vehicle guidance device for at least partially automated driving of the motor vehicle, which is more relevant for driving or controlling the motor vehicle in accordance with or depending on the semantic content of texts for driving the vehicle signs is set up.

Further features of the invention can result from the following description of the figures and from the drawing. The features and feature combinations mentioned above in the description and the features and feature combinations shown below in the description of the figures and/or in the figures alone can be used not only in the combination specified in each case, but also in other combinations or on their own, without going beyond the scope of the invention to leave.

In the single figure, the drawing shows a partial schematic representation to illustrate a method for semantic-based support of vehicle guidance.

In large cities in particular, there are sometimes very complex driving situations, for example at confusing intersections, which require a high degree of concentration in order to choose the right route and avoid accidents. In addition to the well-known street signs, there are often other signs that indicate detours, road closures or other traffic-related information. Seeing and mentally processing these signs also complicates the driving task. If these are written in another language or not at all with Latin characters, it becomes even more complicated.

It would therefore be useful if a vehicle could "read" these signs and texts and specifically point them out to the driver or process the information directly. This can be achieved through the use of Natural Language Processing (NLP).

For illustration shows 1 a partial schematic representation of a motor vehicle 1 and its surroundings. A dashboard 2 of the motor vehicle 1 with an instrument cluster 3 and an infotainment screen 4 are indicated here. A windshield 5 of the motor vehicle 1 is indicated above the dashboard 2 . The motor vehicle 1 also has a camera 6, which is oriented forward in the direction of travel, for detecting the surroundings. The camera 6 can record image data of the surroundings, which can then be processed by an assistance system 7 of the motor vehicle 1, which is also indicated here schematically, to support the vehicle guidance. The assistance system 7 has a processor 8 and a computer-readable data memory 9 and is set up to carry out a method for supporting vehicle guidance, which is explained below by way of example.

A section of an area surrounding the motor vehicle 1 with a road 10 is shown here through the windshield 5 . For example, there is a parking area 11 and a sign 12 on street 10.

The information sign 12 has a text 13 directing traffic. This can indicate, for example, that the road 10 and/or the parking area 11 is closed at certain times and/or for certain vehicles, or the like.

Information sign 12 can be depicted in the image data recorded by camera 6 and can be identified by assistance system 7 by processing it. The assistance system 7 can also recognize the text 13 and process it using natural language processing in order to determine its semantic content. The assistance system 7 can then determine whether or not the information sign 12 or its meaning is relevant to driving the motor vehicle 1 in the current situation. If the information sign 12 is classified as relevant, a driver of the motor vehicle 1 can then be made aware of the information sign 12 and/or its semantic content. For this purpose, for example, a highlighting 14 of the information sign 12, for example in the form of a colored frame around the information sign 12 or the like, can be displayed, for example by means of a HUD of the motor vehicle 1. Additionally or alternatively, for example in the instrument cluster 3 and/or in the infotainment screen 4 a corresponding note 15 on the sign 12 or its meaning, ie the semantic content of the text 13 are displayed. Likewise, the assistance system 7 can output a corresponding spoken instruction and/or a vehicle controller or at least partially automated vehicle guidance of the motor vehicle 1 auto automatically adapt according to the semantic content of the text 13.

In large cities in particular, but also in other places, such a situation can occur in which, in addition to known or standardized signs, traffic lights, lane markings and other street equipment, other signs such as information sign 12 can also be present. Such information signs 12 can be situation- and/or location-specific, non-standardized and/or only temporarily present or relevant. For example, the information sign 12 can provide information in text form about traffic-related information, such as detours, road closures, other important traffic information and the like. Since such information or notices may only be available temporarily or are relevant, they are often not shown or contained in navigation systems or navigation maps. In addition, such information signs 12 in different regions or countries can also contain other nouns and/or verbs in addition to place names, which can also be shown in different languages or with different alphabets or characters.

Such text-heavy information signs 12 can be recognized here automatically by the motor vehicle 1, in particular by the assistance system 7. Information conveyed by the respective information sign 12 can then be automatically interpreted semantically in order to support or relieve the driver of the motor vehicle 1, in particular in a complex traffic situation.

For this purpose, an already existing multifunction camera of the motor vehicle 1 can be used as the camera 6, for example, which can also be used for one or more other functions, such as for a masking permanent high beam, standard traffic sign recognition, lane or lane marking recognition and obstacle recognition and/or or something like that.

The assistance system 7 can be connected or coupled to the camera 6 . Likewise, the camera 6 and the assistance system 7 can be combined with one another or integrated.

By processing the image data recorded by the camera 6, the assistance system 7 can find corresponding texts 13, digitize them and analyze or interpret them semantically. A method for processing natural language can be used, with which texts 13 that are not standardized for traffic signs can also be interpreted semantically. In other words, it is therefore possible to react to almost any traffic-guiding information or text 13 according to its semantic content.

For this purpose, a respective image can first be recorded by the camera 6, pre-processed and then processed further. Finally, the text 13 recognized in the process can be processed and a corresponding output can be made, for example in the form of the highlighting 14, the information 15 and/or a control signal for the motor vehicle 1 or another assistance device of the motor vehicle 1.

In the first of these steps, the camera 6 can continuously film what is happening in front of the motor vehicle 1 in the direction of travel. One or more other surroundings cameras, such as a laterally aligned camera 6 or a reversing camera, can also be used. However, the latter will most likely not be able to detect any information signs 12 or texts 13 relevant to the respective driving situation. Corresponding data processing effort for processing images recorded with it can therefore be saved. The camera 6 can work with a predetermined frame rate, that is, for example, record 20 to 30 frames per second, so that a new image can then be available every 33 ms to 50 ms, for example. These images can, for example, be pre-processed directly in the camera 6 and then transmitted to the assistance system 7 for further processing.

In the second step, the corresponding pre-processing of the respective image takes place, which can mean or include, for example, the application of one or more predefined algorithms for image processing. For example, automatic brightness or contrast correction or adjustment, compensation for motion blur in combination with previous images, transformation of the image for rectification, for example due to distorting optics or an oblique recording angle, conversion to a different color range, e.g a conversion to a black-and-white representation, a gray value representation or to a specified HSV range (HSV: Hue-Saturation-Value) or a specified HSV coding or the like can be carried out. An improved, more precise or more efficient further processing of the respective image can thus be made possible. Such pre-processing can be done by using appropriate software algorithms or in hardware.

In the third step, the respective image is further processed, with the text 13 being extracted. A multi-stage approach can be used for this. In a first stage, a segmentation can be applied or performed, in which regions of the image that contain or may contain relevant text 13 can be identified. An image recognition method, for example based on deep learning, ie machine learning or an artificial neural network, can be used for this purpose. Classic image processing methods can also be used, such as edge detection, for example canny edge detection, in order to find areas surrounded by connected edges. Areas identified in this way can then be classified with regard to their shape, if necessary taking into account possible distortions. Since information signs 12 are typically rectangular, rhombus-shaped or round, only such shapes, ie surfaces shaped in this way, can then be filtered out and further processed. Another approach is to use deep learning or AI-based methods to identify regions with letters. A so-called bounding box can then be calculated in each case, which surrounds the recognized region containing letters and can be further processed as a corresponding area in the subsequent stage or further method steps.

In a second stage, all potentially relevant regions or areas identified in the respective image can be analyzed. For this purpose, for example, optical character recognition (OCR) can be applied to these regions or areas. In this case, digitized texts 13, in particular including their pixel coordinates, can be stored in a corresponding predetermined data structure, for example in the data memory 9, for further processing. It can then also be saved or stored there, whether it was recognized whether these texts 13 were each on a sign.

In the fourth step, the recognized texts 13 can be analyzed semantically and their relevance for the respective traffic situation can be checked. For this purpose, a predetermined pipeline of processing steps can first be executed or run through in order to normalize the respective recognized text 13 . This can mean, for example, that the text 13 is converted to lower-case letters, sentences are extracted by separating the text 13 at punctuation marks, words are mapped to their root word, for example by means of stemming or lemmatization, a so-called part-of-speech detection used to recognize parts of speech, and/or the like.

After such a normalization of the respective text 13, further processing or analysis steps can be applied. For example, a predefined rule-based text analysis can be carried out in order to extract, ie recognize, location and time information or information referring to specific vehicle functions or vehicle features and/or the like. If, for example, a location, such as a place or street name or the like, is recognized, it can then be checked whether a current navigation route of motor vehicle 1 includes the corresponding location, ie, for example, leads through the corresponding street. If an indication of the time is recognized, it can be checked whether the respective sign 12 is relevant at the current time or not. In addition, it can be checked, for example, whether the respective text 13 contains a diversion recommendation, possibly combined with a reference note or diversion abbreviation, such as “U2” or the like. If this is the case, the navigation route can be automatically adjusted accordingly. Likewise, it can then be set or caused that the driver is explicitly informed if the same reference or diversion abbreviation is recognized again on a later identified sign 12 during the current journey of the motor vehicle 1 . Further and/or other processing and/or analysis steps are also possible.

In the fifth step, the text 13 processed in this way or its semantic content or a corresponding note 15 or a corresponding control signal can be output. In view of the complexity of the semantic text analysis proposed here, it can be useful to display the recognized information signs 12 to the driver, in particular if they are relevant to the driver or the respective situation in terms of location and/or time. As already indicated, this can be done, for example, by a prominent display in the instrument cluster 3, the infotainment screen 4, the HUD of the motor vehicle 1 and/or the like in order to draw the driver's attention to it.

If there are interpretations that indicate, for example, a speed adjustment to be carried out or recommended, a light setting, a windshield wiper setting, an air conditioning setting and/or the like, a corresponding setting can be carried out automatically or a corresponding control signal can be output. This can be done at least when the corresponding recognition or interpretation has at least a predetermined minimum or minimum confidence.

Likewise, the assistance system 7 can translate instructions conveyed by the respective information sign 12, for example regarding detours or the like, into a predefined language or cause such a translation to be carried out by a server device external to the vehicle. Similarly, characters that are potentially unknown to the driver, for example that deviate from a predefined alphabet or character set, can be replaced in the respective message 15 by characters from a predefined alphabet or character set. For example, an automatic conversion of Cyrillic, Arabic or Chinese characters into equivalents, that is to say corresponding characters or character groups of the Latin alphabet, can be carried out. The reference 15 cannot then use, ie display, the actual information sign 12 or the characters used for its text 13, but the characters replacing them, for example Latin characters here.

Furthermore, the assistance system 7 can determine a change in a camera image or the images recorded by the camera 6 via an optical flow and/or vehicle trajectory data which indicate or describe a current trajectory of the motor vehicle 1 . Based on this, information signs 12 that have already been recognized and whose text 13 has already been semantically processed can be filtered out, so that, for example, no renewed OCR application and semantic text analysis is then carried out. If a predetermined minimum confidence or minimum accuracy was not yet achieved in a previous text recognition or text analysis, a new corresponding processing, i.e. for example a new OCR application and semantic text analysis, can be carried out for the same information sign 12 when at least one new image becomes available.

Overall, the examples described show how a method for using Natural Language Processing (NLP), i.e. for processing natural language, can be implemented and applied to improve a driving experience.

Reference List

1

motor vehicle

2

dashboard

3

instrument cluster

4

infotainment screen

5

windshield

6

camera

7

assistance system

8th

processor

9

data storage

10

Street

11

parking area

12

information sign

13

text

14

emphasis

15

Notice

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• US7386437B2 [0003]
• DE 202011100069 U1 [0004]

Claims (10)

Method for supporting vehicle guidance, in which automatically - using a camera (6) of the vehicle (1), image data are recorded which depict a respective area ahead of the vehicle (1) in the direction of travel of the vehicle (1), - A sign (12) on which traffic-directing text (13) is depicted is identified by processing the image data, - The text (13) is processed by means of natural language processing and its semantic content is thereby determined and the relevance for driving the vehicle is determined based thereon, and - if the sign (12) is relevant for driving the vehicle, a driver of the vehicle (1) is informed and/or the driving of the vehicle is adjusted accordingly. procedure after claim 1 , characterized in that the image data and the text (13) are processed locally in the vehicle (1) by means of a data processing device (7, 8, 9). Method according to one of the preceding claims, characterized in that the image data are recorded as a temporal sequence and it is checked for newer parts of the image data whether the text (13) for a sign (12) depicted therein has already been processed and if this is the case is, no re-processing of the text (13) is performed for this sign (12) based on the newer parts of the image data. Method according to one of the preceding claims, characterized in that when determining the semantic content, it is checked whether - the text (13) contains a time specification which specifies a period of validity for the sign (12), and then only if the current time in each case is within this validity period falls, the plate (12) is classified as relevant for driving the vehicle, and/or - the text (13) indicates a vehicle feature by which the validity of the plate (12) is restricted to vehicles (1) with this vehicle feature, and the plate (12) is only classified as relevant for driving the vehicle if the vehicle (1) has the specified vehicle feature. Method according to one of the preceding claims, characterized in that when determining the semantic content, it is checked whether the text (13) contains an instruction and if this is the case and the sign (12) is relevant for the vehicle guidance, the vehicle guidance automatically according to the instruction is adjusted. Method according to one of the preceding claims, characterized in that repeated identical reference information is recognized in a number of signs (12) identified one after the other and, based on this, an identification of a route, in particular a diversion or bypass, is determined as its semantic content. Method according to one of the preceding claims, characterized in that when the text (13) is processed, its language and/or alphabet is identified and an automatic translation into a predetermined language and/or a replacement of characters of the text ( 13) is performed by corresponding characters of a given alphabet. procedure after claim 7 , characterized in that the translation is carried out by means of a vehicle-external server device. Assistance system for a vehicle (1), having a data processing device (8, 9) for image processing, text recognition and natural language processing, wherein the assistance system (7) is set up to carry out a method according to one of the preceding claims. Motor vehicle (1), having a camera (6) for imaging the surroundings of the motor vehicle (1) and an assistance system (7). claim 9 .

Images