DE102017204983B4 - Method for specifying a driving movement in a machine learning-based autopilot device of a motor vehicle and a control device, motor vehicle and training device for an autopilot device - Google Patents

Abstract

A method for specifying a driving route (24 ') in a machine learning-based autopilot device (14) of a motor vehicle (10), the autopilot device (14) being external to the vehicle based on sensor data (22) from a sensor device (11) of the motor vehicle (10) Detects obstacle objects (20, 27) in an environment (21) of the motor vehicle (10) and guides the motor vehicle (10) past the obstacle objects (20, 27) without collision by generating at least one control signal (19) for longitudinal guidance and / or lateral guidance, characterized in that, in the method, a control device (12) inserts respective display data (23) of at least one virtual obstacle object (28) and / or at least one virtual following object (31) to be tracked into the sensor data (22) and thereby the at least one virtual obstacle object (27) and / or the at least one virtual following object (31) is shown in each of the sensor data (22), di e display data (23) are generated as a function of the route (24 ').

Description

The invention relates to a method for specifying a driving route in a machine learning-based autopilot device of a motor vehicle. Such an autopilot device can automatically guide a motor vehicle past obstacle objects without collision by longitudinal guidance (acceleration and braking) and / or transverse guidance (steering). The invention also includes a control device for a motor vehicle and the motor vehicle and a training device for an autopilot device in order to carry out machine learning.

Autonomous driving, that is to say automated driving of a motor vehicle by means of an autopilot device, can be implemented on the basis of machine learning. For this purpose, an artificial neural network can be operated on a computer architecture, which receives sensor data from a sensor device and, as a function of the sensor data, generates a control signal for longitudinal guidance and / or lateral guidance of the motor vehicle. A maximum expansion stage here is the so-called end-to-end model of autonomous driving, in which an autopilot device, for example on the basis of a so-called deep artificial neural network, uses image data from a camera and / or at least one other sensor to send the control signal directly to set a steering angle and / or an accelerator pedal position / brake pedal position generated.

Machine learning results in an autopilot device that cannot be controlled in such a way that it can be given a route to a destination. That is, the motor vehicle drives safely on the road, thus avoiding a collision with obstructing objects such as other road users, but the autopilot device cannot be influenced to determine which turn and / or intersection or which direction of travel should be selected in general. This is decided by the autopilot device depending on the training that it has undergone. One consequence would be, for example, to provide a separate artificial neural network for each desired driving maneuver, which, however, affects the computational effort and the manageability of such an autopilot device.

A motor vehicle with an autonomous driving function and an artificial neural network for processing signals from a camera is from the DE 10 2013 016 488 A1 known, by means of the artificial neural network, however, only a degree of driver fatigue is recognized.

The determination of a driving strategy by means of an artificial neural network is in the DE 103 58 498 A1 known. In this way, a motor vehicle can be driven past traffic obstacles without collision.

Also from the DE 10 2015 115 666 A1 it is known how an augmented reality environment can be created in a motor vehicle.

From the DE 10 2015 215 918 A1 a parking system with interactive trajectory optimization is known, in which a user can check on a screen whether an automatic parking system has recognized all obstacles that have to be taken into account during an upcoming parking maneuver. In order to correct the parking trajectory, the user can manually insert virtual obstacles interactively in order to trace or mark the obstacles not recognized by the system.

From the US 2016/0327953 A1 It is known, in a motor vehicle with an autopilot, to insert virtual obstacles into the environment map, which describes the immediate surroundings of the motor vehicle, in those areas that could not be viewed or detected by means of the motor vehicle's sensors because they are shielded, for example the autopilot is supposed to prepare prophylactically if it later turns out that they are actually present in the area when the motor vehicle reaches the area.

From the EP 2 863 177 A1 it is known to represent virtual obstacles for an aircraft, which can represent the boundaries of the flight corridor.

From the US 2013/0050260 A1 A navigation system is known in which a following vehicle, which the driver is to follow, can be shown to a driver in a head-up display for specifying a route.

The invention is therefore based on the object of causing an autopilot device based on machine learning to follow a predetermined driving route.

The object is achieved by the subjects of the independent claims. Advantageous further developments of the invention are described by the dependent claims, the following description and the figures.

The invention provides a method by means of which a driving route can be specified for an autopilot device of a motor vehicle based on machine learning. The route can be determined, for example, by a navigation device of the motor vehicle. The autopilot device is set up to detect obstacle objects outside the vehicle in the surroundings of the motor vehicle on the basis of sensor data from a sensor device of the motor vehicle and to guide the motor vehicle past the obstacle objects without collision by specifying or generating at least one control signal for longitudinal guidance and / or lateral guidance. The autopilot device is based on machine learning, ie it can have, for example, an artificial neural network (ANN) and / or a support vector machine (SVM), by means of which the at least one control signal can be wholly or partially generated from the sensor data.

In order to specify the driving route to be followed or to be driven to such an autopilot device based on machine learning, the invention provides that in the method, a control device provides display data of at least one virtual obstacle object, for example a virtual road user, and / or at least one virtual one to be followed Object to be inserted into the sensor data. The sensor data are thus supplemented by display data of at least one virtual object (obstacle object and / or following object). In this way, the at least one obstacle object and / or the at least one follow-up object is shown in each case in the sensor data, that is to say in the detection range of the autopilot device. In other words, the autopilot device “sees” not only the actual obstacle objects external to the vehicle, but at least one virtual obstacle object and / or at least one virtual following object. The autopilot device then reacts to the at least one virtual obstacle object in the same way as to an actual obstacle object external to the vehicle, i.e. the at least one control signal ensures that the virtual obstacle object is also passed collision-free by means of the longitudinal guidance and / or lateral guidance. In contrast, the autopilot device can follow the at least one following object by means of the control signal by setting a corresponding longitudinal guide and / or transverse guide.

The invention has the advantage that it is not necessary to manipulate or control the autopilot device itself in order to train the autopilot device to follow a route in a targeted manner. This is achieved by “simulating” at least one object in the vicinity of the motor vehicle by inserting the corresponding display data into the sensor data. If the sensor device is a camera and the sensor data thus include at least one camera image, this can be implemented to the effect that the autopilot device is presented with a so-called augmented reality environment (AR) on the basis of the sensor data, in which the real environment is superimposed on the at least one virtual object.

In order to guide the motor vehicle along a travel route by means of the at least one obstacle object and / or the at least one follow-up object, the display data are generated as a function of the travel route. The general direction of travel and / or the road to be selected can be specified on the basis of the route.

The invention also includes further developments, the features of which result in additional advantages.

In order to guide the motor vehicle along a driving route by means of the at least one obstacle object and / or the at least one following object, the display data are preferably generated as a function of a predetermined driving trajectory and / or as a function of the course of the road. A driving maneuver (for example overtaking and / or changing lanes) can be specified for the autopilot unit by means of the driving trajectory. The driving trajectory can be formed, for example, on the basis of maneuver planning, as is known per se from the prior art in connection with automated driving. On the basis of the course of the road, the motor vehicle can be stopped or guided along a road. The course of the road itself can be determined, for example, using a digital map, that is, using digital map data.

A delimiting object or a delimitation of a driving corridor encompassing the driving route and / or a blocking object in the area of a driving direction to be avoided, which can be displayed in the sensor data, has proven to be a suitable respective virtual obstacle object. The driving corridor is in particular wider than the motor vehicle itself. It is therefore not a matter of specifying a specific driving trajectory, but only a corridor within which the autopilot device can maneuver itself, for example avoiding an obstacle object located in the driving corridor or an intruding obstacle. A driving corridor can be formed, for example, by displaying walls as obstacle objects. A blocking object can, for example, be displayed at an intersection on all those streets into which the motor vehicle should not turn. As a result, the at least one blocking object blocks all driving alternatives, except for the one which the autopilot device is to select. An example of a blocking object is a tree or a stationary motor vehicle.

A floor covering or strip that is to be driven on and encompasses the route can be faded in as a respective virtual following object. The strip is wider than the vehicle width of the motor vehicle in order to enable the autopilot device to maneuver, for example, evasive, within the lane in the manner described. Additionally or alternatively, a guide object moved in front of the motor vehicle along the route can be displayed as at least one virtual follow-up object. For example, a ball flying or rolling in front of the motor vehicle or a lead vehicle can be displayed.

It is preferably provided that the autopilot device generates the at least one control signal from the sensor data by means of an artificial neural network (ANN), in particular a deep artificial neural network. An artificial neural network has proven to be an advantageous solution when processing sensor data for generating the at least one control signal.

The display data can be inserted in image data of a camera of the sensor device and / or in radar data of a radar of the sensor device and / or in ultrasound data of an ultrasound sensor of the sensor device. An augmented reality representation is thus generated in each case in image data and / or radar data and / or ultrasound data by simulating the presence or presence of at least one object. The procurement of the display data results from the property of the at least one sensor used. An image of an object (obstacle object and / or following object) can be displayed in image data of a camera. Simulated or prepared reflection data of a virtual object and / or ultrasound data can be displayed in radar data.

In order to carry out the method according to the invention, that is to say to influence an autopilot device in such a way that it guides the motor vehicle along a driving route, the invention provides a control device for a motor vehicle. Such a control device can be designed, for example, as a control unit of the motor vehicle. The control device has a processor device which is set up to carry out an embodiment of the method according to the invention. For this purpose, the processor device can have at least one microprocessor and / or at least one microcontroller. The processor device can comprise a program code which is set up to carry out the embodiment of the method according to the invention when executed by the processor device. The program code can be stored in a data memory of the processor device.

The invention also includes a motor vehicle which has an embodiment of the control device according to the invention. The motor vehicle can thus have a sensor device and an autopilot device which are coupled via the control device. Alternatively, the autopilot device can be part of the control device.

The motor vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger vehicle or truck.

In the motor vehicle, the control device can display or insert the display data of at least one virtual obstacle object and / or at least one follow-up object in the sensor data of the sensor device in the manner described.

If the at least one obstacle object and / or at least one subsequent object is one that could also be present in the sensor data itself as a real object, for example another road user (motor vehicle) and / or a wall or a tree as Obstacle object, the autopilot device does not have to be further adapted. It will react to the display data, that is to say the artificially superimposed images or photos of such real objects, in the same way as to the actual obstacle objects recognized in the vicinity.

However, in order to be able to fade in a fantasy object in the sensor data that the autopilot device cannot encounter in the real environment, for example a texture for identifying a driving corridor or a lane, or a floating guide object such as a ball, the autopilot device must be trained to do so will also be able to recognize such a virtual fantasy object and to generate a predetermined control signal in order to avoid such a virtual obstacle object and / or to follow such a follow-up object. An autopilot device must also first be trained to track a follow-up object.

For this purpose, the invention provides a training device for an autopilot device, the training device being set up to use a machine learning method to provide the autopilot device with training data that contain real and / or simulated prefabricated sensor data and label data for identifying obstacle objects depicted in the prefabricated sensor data . This is the conventional training of an autopilot device in order to react to real obstacle objects existing in an environment in the manner described. The training device is also part of it set up to also insert display data of at least one virtual obstacle object and / or of at least one follow-up object into the training data and provide the at least one virtual obstacle object and / or the at least one virtual follow-up object in each case identifying label data. As a result, the at least one virtual obstacle object and / or the at least one virtual follow-up object need not have the shape and / or the appearance of an obstacle object or follow-up object that is actually to be expected in road traffic. The autopilot device is trained on the appearance of artificial obstacle objects and / or follow-up objects, that is to say on fantasy objects.

The training device can be, for example, a computing device that does not have to be located in the motor vehicle. The training device is set up in particular to use the training data to train an artificial neural network of the autopilot.

At least one method of machine learning can be used as training method, for example supervised learning, unsupervised learning, reinforcement learning.

• 1 eine schematische Darstellung einer Ausführungsform des erfindungsgemäßen Kraftfahrzeugs; und
• 2 eine schematische Darstellung zur Veranschaulichung einer Ausführungsform des erfindungsgemäßen Verfahrens, das durch eine Steuervorrichtung des Kraftfahrzeugs von 1 durchgeführt werden kann.

Exemplary embodiments of the invention are described below. This shows:

• 1 a schematic representation of an embodiment of the motor vehicle according to the invention; and
• 2 a schematic representation to illustrate an embodiment of the method according to the invention, which is carried out by a control device of the motor vehicle from 1 can be carried out.

The exemplary embodiments explained below are preferred embodiments of the invention. In the exemplary embodiments, the described components of the embodiments each represent individual features of the invention that are to be considered independently of one another, which also develop the invention independently of one another and are thus also to be regarded as part of the invention individually or in a combination other than the one shown. Furthermore, the described embodiments can also be supplemented by further features of the invention that have already been described.

In the figures, functionally identical elements are each provided with the same reference symbols.

1 shows a motor vehicle 10 , which can be a motor vehicle, in particular a passenger car or truck. A sensor device is shown 11th , a control device 12th , a navigation device 13th , an autopilot device 14th and an abstract representation of vehicle guidance components 16 who, for example, have a steering system 17th , a motor controller 18th and / or a brake control 19th can include.

The autopilot facility 14th can for the vehicle components 16 generate a control signal to thereby control the vehicle 10 automated driving in road traffic. To at least one obstacle object 20th , for example another road user, so for example another motor vehicle, in an environment 21 to be able to recognize and then the motor vehicle by means of the control signal 19th collision-free at the obstacle object 20th To be able to drive past, uses the autopilot device 14th from the sensor device 11th provided sensor data 22nd . The sensor data 22nd can be camera data, for example.

However, the autopilot will be 14th the sensor data 22nd not provided directly. The control device 12th can the sensor device 11th and the autopilot device 14th be interposed. Alternatively, the autopilot device 14th Part of the control device 12th be. The control device 12th inserts in the tax data 22nd additional display data 23 a, so that supplemented or manipulated sensor data 22 ' result. The presentation data 23 can through the control device 12th depending on the route data 24 can be generated, which one for example by the navigation device 13th determined route 24 ' describe or represent.

In this context, in 2 shown as the control device 12th from the sensor data 22nd by inserting the display data 23 the manipulated sensor data 22 ' can generate. A camera image is an example of this 25th and a manipulated camera image 26th shown. The camera image can be taken by a camera of the motor vehicle 10 be generated, the one in the forward direction in front of the motor vehicle 10 recorded road traffic. In camera image 25th can have multiple obstacle objects 27 be shown, for example the other road user 20th .

The autopilot facility 14th can, for example, based on an artificial neural network KNN the control signal 19th generate such that the motor vehicle 10 collision-free at the obstacle objects 27 is passed. However, it cannot target the route 24 ' according to the route data 24 follow.

The control device 12th can therefore depending on the travel route data 24 at least one artificial object in the camera image 25th fade in so that the manipulated camera image 26th results. This is done by inserting the display data 23 into the sensor data 22nd carried out. For example, there can be at least one virtual obstacle object 28 which, for example, a driving corridor 29 can limit within which the motor vehicle 10 should be maneuvered. The driving corridor 29 is wider than the motor vehicle 10 so one in the corridor 29 an urgent obstacle object 30th by means of an evasive maneuver by the autopilot device 14th can be bypassed. Additionally or alternatively to a virtual obstacle object 28 can be a virtual follow-up object 31 are displayed, in the direction of which the autopilot device 14th the driving maneuver can align to thereby the following object 31 to follow. In 2 is an example of a floating ball as a follow-up object 31 into the manipulated camera image 26th shown or inserted. The follow-up object 31 can for example be faded in as if it were flying in front of the motor vehicle 10 along the route 24 ' here. As a follow-up object 31 can additionally or alternatively be a driving corridor 29 be marked, for example as colored and / or textured lanes.

The manipulated camera image 26th thus contains both images of real obstacle objects 27 and / or the environment 21 as well as artificial, virtual objects 28 , 31 . This represents the manipulated camera image 26th a proximity augmented reality.

By integrating a camera into a neural network, in particular a deep neural network, it is possible, by suitable augmentation of the camera image 26th a control of the artificial neural network KNN take place.

Augmentation means in particular the integration of virtual elements or objects in a camera image 26th . Correspondingly, a driver's hose or a driving corridor can be created through such an integration 29 or a line affect the artificial neural network in such a way that a desired route 24 ' is departed. The artificial neural network KNN can use a training device to respond to a desired behavior in relation to a virtual obstacle object in a training process 28 and / or a virtual follow-up object 31 be trained or configured by defining or specifying appropriate label data. Thus, a driving corridor 29 by the course of the road itself and the navigation destination through which the route 24 ' defined, to be determined. The artificial neural network KNN then learns the driving corridor 29 not to leave or traverse. Depending on the navigation destination, the driving corridor then becomes 29 customized.

Because the augmentation is directly on the sensor image or camera image 25th occurs, this does not affect that of the driver of the motor vehicle 10 perceived environment.

This means that automated or autonomous driving processes on a route can also be carried out 24 ' a navigation device 13th follow.

Should the motor vehicle 10 can turn into the camera image 25th based on the ego position or the vehicle's own position 10 a virtual element (obstacle object or following object) shown from this perspective can be displayed. The artificial neural network KNN is taught by training in this virtual objects or elements not to penetrate them and according to the driving corridor 29 or to follow the route path. The exact design of the virtual elements can be freely selected, as they can be taught to the artificial neural network ANN in a training phase.

Overall, the examples show how the invention can provide a method for route tracking and navigation during an autonomous journey by means of artificial neural networks.

Claims (10)

A method for specifying a driving route (24 ') in a machine learning-based autopilot device (14) of a motor vehicle (10), the autopilot device (14) being external to the vehicle based on sensor data (22) from a sensor device (11) of the motor vehicle (10) Detects obstacle objects (20, 27) in an environment (21) of the motor vehicle (10) and guides the motor vehicle (10) past the obstacle objects (20, 27) without collision by generating at least one control signal (19) for longitudinal guidance and / or lateral guidance, characterized in that, in the method, a control device (12) inserts respective display data (23) of at least one virtual obstacle object (28) and / or at least one virtual following object (31) to be tracked into the sensor data (22) and thereby the at least one virtual obstacle object (27) and / or the at least one virtual following object (31) is shown in each case in the sensor data (22), d ie display data (23) are generated as a function of the route (24 '). Procedure according to Claim 1 , the display data (23) being generated as a function of a predetermined driving trajectory and / or as a function of the course of the road. Method according to one of the preceding claims, wherein as a respective virtual obstacle object (28) a delimitation of a driving corridor (29) comprising the driving route (24 '), which is wider than the motor vehicle (10), and / or a blocking object in the area of a closed avoiding direction of travel is displayed. Method according to one of the preceding claims, wherein as a respective virtual follow-up object (31) a lane to be driven, encompassing the driving route, which is wider than the motor vehicle (10), and / or a lane in front of the motor vehicle (10) along the driving route (24 ') moving master object is displayed. Method according to one of the preceding claims, wherein the autopilot device (14) generates the at least one control signal (19) by means of an artificial neural network (ANN), in particular a deep artificial neural network. Method according to one of the preceding claims, wherein the display data (23) are inserted into image data from a camera of the sensor device (11) and / or into radar data from a radar from the sensor device (11) and / or into ultrasound data from an ultrasound sensor of the sensor device (11). Control device (14) for a motor vehicle (10), wherein the control device (14) has a processor device which is set up to carry out a method according to one of the preceding claims. Motor vehicle (10) with a control device (14) according to Claim 7 . Training device for an autopilot device, the training device being set up to use a machine learning method in a training phase of the autopilot device to specify training data that contain real and / or simulated prefabricated sensor data and label data for identifying obstacle objects mapped in the prefabricated sensor data, characterized in that that the training device is set up to insert representation data of at least one virtual follow-up object into the training data and to provide label data that characterize the at least one virtual follow-up object. Exercise device according to Claim 9 , wherein the training device is set up to use the training data to train an artificial neural network of the autopilot.

Images