DE102019009062A1 - Method for operating an at least partially automated motor vehicle and a computer program product - Google Patents

Abstract

A method is described for operating an at least partially automated motor vehicle (2), wherein the motor vehicle (2) drives in an automated driving mode and wherein a driver (4) of the motor vehicle (2) does not intervene in control devices of the motor vehicle (2), whereby a secondary activity of the driver (4) is determined with the aid of a driver monitoring device (10), the type of secondary activity of the driver (4) being recognized and classified, with a categorization of the secondary activity based on an assignment of permitted and / or non-permitted secondary activity classes to the automated driving mode of the driver (4) takes place, an action perceptible to the driver being carried out depending on the secondary activity class. A computer program product and a motor vehicle (2) are also described.

Description

A method for operating an at least partially automated motor vehicle and a computer program product are described.

Methods for operating a motor vehicle that can be driven at least partially automatically; computer program products of the type mentioned at the beginning are known in the prior art.

The aim of semi-automated or fully automated driving or highly automated driving of motor vehicles is to relieve the driver and also to increase road safety so that the driver does not have to deal with driving the motor vehicle on an ongoing basis. The driver will and may, depending on the available driving mode, more or less turn away from the traffic situation and the vehicle control. Depending on the current legislation and the driving mode, it may be necessary for the driver to be able to take control of the vehicle again within specified parameters. Depending on the country and driving mode, this can be immediately or within a specified time. To take over the driving task, an action that can be perceived by the driver, in particular a takeover request, is output to the driver on the vehicle side, in particular as a function of an existing driving situation. Such a driving situation can be, for example, that automated driving is not possible on a route section ahead of the motor vehicle, for example due to prevailing weather conditions, or when the motor vehicle is driving in a region or entering a region in which an automated driving mode is not permitted.

Depending on the secondary activity carried out by the driver of the motor vehicle, however, it takes a certain amount of time until the driver has released himself from his secondary activity, has recorded the traffic situation and can react as planned. Accordingly, it can be foreseen that certain secondary activities will not be permitted at certain levels of automation, at least in certain regions or under certain conditions, for example sleeping.

From the DE 10 2016 207 353 A1 a method for detecting use of an electronic device by a driver for a vehicle is known. The method includes a reading-in step in which at least one sensor signal is read in via an interface to a sensor arranged in the vehicle, the sensor signal representing at least one current parameter of the electronic device and / or of the driver in the vehicle. The method further comprises a step of comparing, in which the sensor signal is compared with at least one stored use signal that represents the use of the electronic device by the driver while driving the vehicle. Finally, the method comprises a step of outputting, in which a detection signal is output if a predetermined relationship exists between the sensor signal and the use signal, in order to detect the use of the electronic device.

Thus, the task arises of developing methods for operating an at least partially automated motor vehicle and computer program products of the type mentioned at the outset such that the motor vehicle helps the driver to comply with the legal requirements in order to ensure road safety.

The object is achieved by a method for operating an at least partially automated motor vehicle according to claim 1, a computer program product according to the independent claim 10. Further embodiments and developments are the subject matter of the dependent claims.

A method for operating an at least partially automated motor vehicle is described, the motor vehicle driving in an automated driving mode and wherein a driver of the motor vehicle does not intervene in control devices of the motor vehicle, with the aid of a driver monitoring device a secondary activity of the driver is determined, the type of secondary activity of the driver is recognized and classified, with the driver's ancillary activity being categorized based on an assignment of permissible and / or non-permissible secondary activity classes to the automated driving mode, with an action perceptible to the driver being carried out depending on the secondary activity class. Secondary activities are activities of the driver that do not relate to the driving of the vehicle by the driver himself.
Depending on the type of secondary activity currently being carried out, the driver is more or less absorbed in the secondary activity currently being carried out and is thus less or more distracted from the current events in the area, inside and outside the motor vehicle. Reading a book, playing a computer game or watching a video film requires significantly more concentration than just looking at pictures or interacting with a mobile phone. The perception of events in the driver's or vehicle environment therefore depends on the type of secondary activity currently being carried out.

This also applies in the case of a planned, system-related driver takeover request to the driver. The ability to take over - the ability of the driver to perceive a takeover request from the vehicle and to react to it with a sufficiently high takeover quality - in the case of demanding secondary activities, e.g. B. reading a book, playing a computer game or watching a video film can be significantly more restricted than simply viewing a picture. Another aspect is the willingness of the driver himself to cancel his currently performed secondary activity in order to immediately comply with the takeover request from the vehicle, so that the driver's reaction time for such secondary activities may take longer, which means that a warning or takeover request is issued earlier must become.

The method makes it possible to identify a secondary activity that is not permitted in a current driving situation and to at least inform the driver who is performing the not permitted secondary activity of this fact. By monitoring the driver's secondary activities, an active contribution to driving safety controlled by the motor vehicle can be made.

In a first further refinement, it can be provided that the driver monitoring device has a driver monitoring camera, the type of secondary activity of the driver being recorded on the basis of at least one object actuated by the driver.

An object operated by the driver can, for example, be an object that the driver holds in his hands. Such an object can be, for example, a mobile phone, a magazine, a book, a drink or some other food or an operating device of the motor vehicle.
In another further development, depending on the type of secondary activity, a plausibility check may be useful or necessary.

In a further refinement, it can be provided that the perceptible action is a warning signal and / or a takeover request.

If the driver drives the motor vehicle in question more frequently, he will subconsciously associate the action in question with the non-permitted secondary activity, which shortens the reaction times and helps the driver to react more quickly.

In a further refinement, it can be provided that the perceptible action is a deactivation of at least one function controlled by the motor vehicle.

A relevant function can be an automatic driving function, for example, so that a driver is requested to take control of the motor vehicle. It can thus be achieved that the driver has to turn away from his secondary activity, which is not permitted in the situation in question, and thus adequately monitors the traffic.

In a further refinement, it can be provided that the secondary activities or secondary activity classes each have at least one attention index that is correlated with an acquisition parameter.

With the help of a corresponding attention index, it is possible to sort and categorize secondary activities in groups. Corresponding attention indices can, for example, be given by recommendations issued by experts, organizations or authorities. By assigning certain secondary activities or secondary activity classes to an attention index, local adaptation can also be achieved and secondary activities that are permitted in one country and prohibited in another can be classified accordingly.

The takeover parameter can, for example, define a time within which the driver must have recorded the traffic situation and taken control of the motor vehicle.

In a further refinement, provision can be made for the type of secondary activity to be classified by comparison with at least one secondary activity stored in a memory.

A classification of the type of secondary employment is useful because not all secondary employment can be conclusively recorded. Furthermore, it does not make sense in practice to specify for each individual secondary activity whether the secondary activity should or should not be permitted. It is therefore more practicable to group them together in secondary activity classes.

In a further refinement, it can be provided that further attention parameters of the driver that are correlated with the takeover parameters are recorded and evaluated.

Because the secondary activity is stored in the memory, the classification can take place in the motor vehicle without having to access data that is stored in a computer center. This means that the method can also work in regions where there is no cell phone reception.

In a further refinement, it can be provided that a time during which the driver's secondary activity is carried out is recorded and evaluated.

The longer the driver is busy with his secondary activity, the more likely it is that he is very absorbed in his current secondary activity, which increases the time until the driver has sufficiently grasped a traffic situation.

By detecting further attention parameters, for example a direction of the driver's gaze, pupil movements, blinking times and the like, further factors can be derived. These allow a detection of the current state of the driver and a prediction of how quickly the driver can take control of the motor vehicle. If, for example, it can be seen that the driver is looking very closely at his newspaper, book or computer, it can be assumed that he is very absorbed in his respective secondary activity. If it can be seen that the driver is blinking more slowly, this can indicate tiredness, so that it is to be expected that the driver will need longer to take control of the motor vehicle.

In another further refinement, it can be provided that the determined secondary activity is used to check whether the secondary activity is available via a vehicle's own system, for example the infotainment system, in which case a message is issued to the driver to inform him of this possibility to inform.

E.g. a text can be shown in a display of a man-machine interface or an acoustic message can be triggered in the vehicle or a haptic message can be transmitted directly to the driver, for example via a driver's seat or a driver's belt.

A first independent subject relates to a computer program product with a computer-readable storage medium on which instructions are embedded which, when they are executed by at least one processing unit, have the effect that the at least processing unit is set up to carry out the method of the type described above.

The method can be carried out distributed on one or more processing units, so that certain method steps are carried out on one processing unit and other method steps are carried out on at least one further processing unit, with calculated data being able to be transmitted between the processing units if necessary.

Another independent subject matter relates to a motor vehicle with a computer program product of the type described above.

Further advantages, features and details emerge from the following description in which at least one exemplary embodiment is described in detail - possibly with reference to the drawing. Identical, similar and / or functionally identical parts are provided with the same reference symbols.

• 1 eine Draufsicht auf ein Kraftfahrzeug;
• 2 eine Tabelle, die Nebentätigkeiten je nach Fahrsituation in zulässige und nicht zulässige Nebentätigkeiten klassifiziert;
• 3 ein erstes Ablaufdiagramm, sowie
• 4 ein zweites Ablaufdiagramm.

They show schematically:

• 1 a plan view of a motor vehicle;
• 2 a table that classifies secondary activities into permissible and non-permissible secondary activities depending on the driving situation;
• 3 a first flowchart, as well
• 4th a second flow chart.

1 shows a plan view of a motor vehicle 2 .

In the motor vehicle 2 a driver sits in a driver's seat 4th who is in his hands 6th a newspaper 8th holds and reads. The car 2 drives in a partially automated, highly automated or fully automated driving mode so that the driver 4th can devote himself to other permitted secondary activities.

Depending on the respective circumstances, certain secondary activities or secondary activity classes are permitted in addition to monitoring processes in road traffic, others are not permitted. This is because, depending on the country / region, driving mode and driving condition of the driver 4th Must be able to take control of the motor vehicle 2 to take over in a given time. While the driver is in the driving mode of partially automated driving 4th must permanently monitor the processes in road traffic and thus take control of the motor vehicle 2 can take place immediately, the takeover in higher automated driving modes requires a correspondingly sufficient time. Taking control of the motor vehicle 2 In the driving modes of highly or fully automated driving, the steps require the driver 4th to draw attention to the required takeover (takeover request), the request by the driver 4th to perceive them as such, to get an overview of what is happening on the road, to bring hands and feet to the steering wheel and pedals, and to control the vehicle 2 take control of the same.

The time that is required to carry out all these steps depends on the one hand on the deepening in the respective current secondary activity, but on the other hand also on the logistics of removing the object with which the driver is currently busy. A computer may have to be put away from one's lap, a newspaper has to be folded up, a book has to be folded up, a mobile phone has to be put down, etc. In this respect there are certain secondary activities or secondary activity classes that are permitted in certain driving situations and others that are in the driving situations and driving mode are not permitted, e.g. those in which an object has to be put away.
The secondary activity classes can record the interaction with a certain object, for example a general ban on working with a laptop on your lap, or a restricted ban, for example dealing too long with a certain object, for example reading a magazine, as there is a risk that the driver 4th is too absorbed in reading and it would take too long to return sufficient attention to what is happening on the road.

The driver 4th is made using a driver monitoring camera 10 supervised. The driver monitoring camera 10 is with a controller 12 connected to a processor 14th and a memory 16 contains, on which a computer program product is stored, which when it is processed by the processor 14th is loaded and executed, follow the procedure described below. The control 12 is also with a loudspeaker 18th connected, with the help of which a takeover request, for example by a tone sequence or an announcement or both to the driver 4th can be output.

With the help of the driver monitoring camera, 10 are the hands 6th of the driver 4th recorded and both the position of the hands 6th as well as the one in the hands 6th objects located. The control 12 uses image processing techniques to determine where the hands are 6th and what item 8th straight from the driver 4th is held or operated. The controller can use this 12 identify a secondary job or at least a secondary job class (reading, writing, working with a computer, etc.).

With the help of the driver monitoring camera 10 In addition, additional information can be recorded, for example the direction of view of the driver 4th as well as fatigue indicators, for example based on blinking frequency or blinking speed. This additional information can also provide information about the secondary activity and the condition of the driver 4th allow.

In certain embodiments, additional driver monitoring devices, not shown in the figure, such as at least one additional camera or at least one additional microphone, can be used to enable the driver to perform operating actions 4th or voice and sound information are recorded in the vehicle as additional information, for example in connection with a mobile phone to differentiate between reading a text and a video conference. In addition, at least one additional camera can be used to record screen contents of a mobile device or a laptop and to classify the screen contents as an image, video, text or computer game.

• • Lesen von E-Mails, E-Books, Nachrichten, Berichten, Zeitungen und Zeitschriften (z.B. Objekt: Buch; Zeitschrift; Zeitung. Zusätzliche Information: Blickrichtung)
• • Schreiben von Texten und Nachrichten (z.B. Objekt: Mobilgerät. Zusätzliche Information: Blickrichtung)
• • Teilnahme an Videokonferenzen (z.B. Objekt: Handy. Zusätzliche Information: Blickrichtung und/oder Sprache und/oder Ton)
• • Telefonieren (z.B. Objekt: Handy. Zusätzliche Information: Sprache)
• • Spielen von Computerspielen auf mobilen Endgeräten (z.B. Objekt: Computer, Handy. Zusätzliche Information: Blickrichtung und/oder Bedienhandlung und/oder Ton)
• • Betrachtung von Videos/Filmen oder Hören von Podcasts/Musik (z.B.: Objekt: Mobilgerät. Zusätzliche Information: Blickrichtung und/oder Ton)
• • Essen oder Trinken (z.B. Objekt: Flasche. Zusätzliche Information: bestimme Bewegung des Objekts)
• • Kind auf Schoß fahren lassen (Objekt: Erkennen von 2 Köpfen im Fahrersitzbereich)
• • Schminken (Objekt: Makeup. Zusätzliche Information: Blickrichtung und/oder bestimmte Bewegung des Objekts)
• • Rauchen (Objekt: Zigarette. Zusätzliche Information: bestimmte Bewegung des Objekts)

Corresponding secondary activities can include:

• • Reading e-mails, e-books, news, reports, newspapers and magazines (e.g. object: book; magazine; newspaper. Additional information: viewing direction)
• • Writing texts and messages (eg object: mobile device. Additional information: viewing direction)
• • Participation in video conferences (e.g. object: mobile phone. Additional information: viewing direction and / or language and / or sound)
• • Telephoning (e.g. object: cell phone. Additional information: language)
• • Playing computer games on mobile devices (eg object: computer, cell phone. Additional information: viewing direction and / or operating action and / or sound)
• • Viewing videos / films or listening to podcasts / music (eg: object: mobile device. Additional information: viewing direction and / or sound)
• • Eating or drinking (e.g. object: bottle. Additional information: determine movement of the object)
• • Let the child ride on the lap (object: recognition of 2 heads in the driver's seat area)
• • Make-up (object: makeup. Additional information: direction of view and / or certain movement of the object)
• • Smoking (object: cigarette. Additional information: certain movement of the object)

2 shows a table showing the admissibility of a certain secondary activity or Represents secondary activity class depending on the current driving situation.

The respective driving modes are specified for the driving situations city, motorway heavy traffic, motorway normal traffic and motorway approved route, for example according to the 5 levels of automated driving, from Level 1 (assisted driving) to level 5 (autonomous driving). Depending on the driving mode and / or the driving situation, different driving modes may be allowed in a region, partially automated in the city (level 2 ), highly automated in dense motorway traffic (level 3 ), fully automated in normal motorway traffic (level 4th ) and autonomously on approved routes on the autobahn (level 5 ).

Certain takeover parameters are assigned to each of these driving modes, for example a takeover time in seconds. In the city the driver must be able to take control of the motor vehicle at any time, in dense motorway traffic the driver must be able to take control within e.g. 10 seconds to take over, in normal motorway traffic within e.g. 20 seconds and on approved route sections on the motorway within e.g. 60 seconds if, for example, a change to another area that requires a different driving mode is imminent.

Accordingly, certain secondary jobs or secondary job classes are permitted or not permitted. In level autonomous driving mode 5 All secondary activities, including sleeping, are permitted on approved sections of the motorway. In the partially automated driving mode Level 2 there are certain sideline jobs that are permitted and others that are not permitted. In normal motorway traffic, for example, objects handled by the driver have to be put away before control of the motor vehicle is taken over. In contrast, this may not be acceptable in heavy traffic. However, secondary activities may still be permitted there, in which no object has to be put down, for example an interaction with the infotainment system, regardless of the type of interaction (for example using a navigation system or playing computer games offered via the infotainment system.

3 shows a first flow chart.

In a first step, an operated or in the hand or in the hands 6th held object 8th detected.

Then the secondary employment is classified.

• • Beispiel 1: Der Fahrer hält eine Zeitung in der Hand, schaut jedoch nicht auf die Zeitung. Dies bedeutet der Blickvektor des Fahrers ist nicht in Richtung der Zeitung gerichtet. Die Plausibilisierung ergibt, dass der Fahrer die Zeitung gerade nicht liest. Die Nebentätigkeit „Lesen“ wird verworfen.
• • Beispiel 2: Der Fahrer telefoniert mit dem Handy in der Hand. In diesem Fall ist die Nebentätigkeit „Telefonieren“ gültig auch wenn er nicht auf sein Handy schaut.
• • Beispiel 3: Der Fahrer hält zumindest ein Objekt 8 in der Hand, das er mehrfach in Richtung des Kopfes/Mundes führt. Die Kamera erfasst diesen Gegenstand und weist ihm das Attribut trinkbarer oder essbarer Gegenstand zu. Das System klassifiziert die aktuelle Nebentätigkeit als Nebentätigkeit „Essen und Trinken“ und berücksichtigt hierbei die Handbewegung (Führen des Objektes in Richtung des Kopfes/Mundes) zur Plausibilisierung der Nebentätigkeit „Essen und Trinken“.

At the same time, additional information is recorded, for example the direction in which the driver is looking 4th . This additional information can be used to check the plausibility of the secondary activity. This means that based on the position of a captured and recognized object 8th and a secondary activity classified therefrom and the additional information about a direction of the driver's gaze, it is checked whether the driver's gaze vector in the direction of the object 8th is directed. This results in the information whether the driver actually does the secondary activity.
If this is not the case, this secondary activity will be discarded.
The following non-exhaustive list of examples for plausibility checks is intended to illustrate the inventive method.

• • Example 1: The driver holds a newspaper but does not look at the newspaper. This means that the driver's gaze vector is not directed towards the newspaper. The plausibility check shows that the driver is not reading the newspaper at the moment. The secondary activity "reading" is discarded.
• • Example 2: The driver is talking on the phone with the mobile phone in hand. In this case, the secondary activity "Telephoning" is valid even if he does not look at his cell phone.
• • Example 3: The driver is holding at least one object 8th in the hand, which he leads several times towards the head / mouth. The camera detects this object and assigns it the attribute drinkable or edible object. The system classifies the current secondary activity as a secondary activity “eating and drinking” and takes into account the hand movement (moving the object towards the head / mouth) to check the plausibility of the secondary activity “eating and drinking”.

The plausibility-checked secondary job is then checked with the in 2 the list of secondary jobs described. If it turns out that a certain secondary activity is not permitted, a warning is issued. If, on the other hand, it is permissible, an attention index and / or a maximum permissible duration is assigned to it and a warning or takeover request is generated on the basis of the previously made allocations as a function of the attention index and / or the maximum permissible duration.

4th shows a second flow chart.

The first step is one in the hands 6th held object 8th detected.

Then the secondary employment is classified.

At the same time, additional information is recorded, for example the direction in which the driver is looking 4th . This information is used to check the plausibility of the secondary activity.

This is followed by a comparison of the secondary employment with a list of the secondary employment available in the infotainment system. If a certain secondary activity is available, this is offered in the infotainment system or a message is given to the driver that this possibility exists. In this way, a certain secondary activity can be transferred from a separate device, for example a mobile phone, to the infotainment system, which in certain circumstances makes it permissible to carry out a secondary activity that would not be permissible in the current situation with an additional device.

Although the invention has been illustrated and explained in more detail by preferred exemplary embodiments, the invention is not restricted by the disclosed examples and other variations can be derived therefrom by the person skilled in the art without departing from the scope of protection of the invention. It is therefore clear that there is a multitude of possible variations. It is also clear that embodiments cited by way of example really only represent examples that are not to be interpreted in any way as a limitation, for example, of the scope of protection, the possible applications or the configuration of the invention. Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without leaving the scope of protection that is defined by the claims and their legal equivalents, such as a more detailed explanation in the description.

List of reference symbols

2

Motor vehicle

4th

driver

6th

hands

8th

newspaper

10

Driver surveillance camera

12

control

14th

processor

16

Storage

18th

speaker

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• DE 102016207353 A1 [0005]

Claims (10)

A method for operating a motor vehicle (2) that can be driven at least partially automatically, the motor vehicle (2) driving in an automated driving mode and wherein a driver (4) of the motor vehicle (2) does not intervene in control devices of the motor vehicle (2), characterized in that a secondary activity of the driver (4) is determined with the aid of a driver monitoring device (10), the type of secondary activity of the driver (4) being recognized and classified, a categorization of the secondary activity based on an assignment of permitted and / or non-permitted secondary activity classes to the automated driving mode of the driver (4) takes place, with an action perceptible to the driver being carried out as a function of the secondary activity class. Procedure according to Claim 1 , characterized in that the driver monitoring device has a driver monitoring camera (10), the type of secondary activity of the driver (4) being detected on the basis of at least one object (8) operated by the driver. Procedure according to Claim 1 or 2 , characterized in that the perceptible action is a warning signal and / or a takeover request. Method according to one of the preceding claims, characterized in that the perceptible action is a deactivation of at least one function controlled by the motor vehicle (2). Method according to one of the preceding claims, characterized in that the secondary activity classes each have at least one attention index which is correlated with an acceptance parameter. Method according to one of the preceding claims, characterized in that the classification of the type of secondary activity is carried out by comparison with at least one secondary activity stored in a memory. Method according to one of the preceding claims, characterized in that further attention parameters of the driver (4) that are correlated with the takeover parameters are recorded and evaluated. Method according to one of the preceding claims, characterized in that the recognized and classified secondary activity is checked for plausibility by the acquisition of additional information. Method according to one of the preceding claims, characterized in that a time for performing the secondary activity of the driver (4) is recorded and evaluated. Computer program product with a computer-readable storage medium (16) on which instructions are embedded which, when executed by at least one processing unit (14), have the effect that the at least processing unit (14) is set up to implement the method according to one of the preceding claims execute.

Images