DE102017207810B4 - Method for automated driving of a motor vehicle - Google Patents

Abstract

Method for the automated guidance of a motor vehicle (1), wherein environmental information relating to the area surrounding the motor vehicle (1) is recorded by a sensor system (2) on the motor vehicle and at least one driving intervention to be carried out as part of the automated guidance is determined as a function of the environmental information, wherein a Image data relating to the surroundings of the motor vehicle (1) are captured by a user (7) portable, vehicle-external capture device (6), the image data being transmitted from the capture device (6) to the motor vehicle (1), after which the driving intervention is carried out as a function of the image data is determined, and/or wherein, on the other hand, driving intervention information relating to the driving intervention to be carried out is transmitted from the motor vehicle (1) to the detection device (6), after which the detection device (6) superimposes an image representation (15) generated as a function of the image data with least ns a graphic element (17, 18) generated depending on the driving intervention information for the user (7), wherein in both cases a relative position and / or orientation of the motor vehicle (1) to the detection device (2) or vice versa is determined, wherein the driving intervention is determined as a function of the relative position and/or orientation and/or the graphic element (17, 18) is generated and/or the superimposition of the graphic element (17, 18) on the image representation (15) is parameterized, with the image data being recorded in this way that at least a portion of the motor vehicle (1) is imaged, wherein the relative position and / or orientation is determined by evaluating the image data.

Description

The invention relates to a method for the automated guidance of a motor vehicle, wherein environmental information relating to the area surrounding the motor vehicle is detected by a vehicle-side sensor system and at least one driving intervention to be carried out as part of the automated guidance is determined as a function of the environmental information.

Automated guidance of motor vehicles, in which a vehicle-side or an external system relieves the driver of the lateral and/or longitudinal guidance of the motor vehicle for certain periods of time or for certain driving tasks, is becoming increasingly relevant as a comfort function. Different degrees of automation are discussed, for example, in the article "Legal consequences of increasing vehicle automation" in the publication Research Compact 11/12 of the Federal Highway Research Institute. In the case of such guidance, it is essential to have as complete an environment model as possible of the vehicle environment in the control device that controls the driving operation, in order to enable robust automated guidance. At the same time, in many applications, for example when a vehicle automatically parks when the driver is not in the vehicle, a possibility should be created to inform a user about the intentions of the motor vehicle or to give him the opportunity to intervene in the driving operation.

One possible approach to informing a user and other road users of driving intentions in automated driving is to project corresponding information onto a road surface. This approach is discussed in the references U.S. 2015/0 336 502 A1 and DE 10 2011 112 577 A1 discussed. A corresponding projection is technically relatively complex and not necessarily clearly visible in all lighting conditions.

From the pamphlet DE 10 2014 200 611 A1 a method for autonomous parking of a vehicle is known. An operator can use a mobile device to take a digital image of the target position at the desired parking location. After wireless transmission of this image to a vehicle, it can be used together with current environment data from an environment detection system installed in the vehicle in order to determine a trajectory to the target position and to carry out an autonomous parking process along this trajectory.

A method for carrying out an automated journey of a vehicle along a trajectory provided from a map is known from the publication DE 10 2016 211 182 A1 famous. Here, trafficability information for the environment is taken into account, which can be determined, for example, from a visual image of the environment that was captured by a camera of a smartphone.

The pamphlet DE 10 2016 208 214 A1 proposes linking image data received from a vehicle-external sensor with image data recorded by an on-board sensor in order to support a maneuvering process of a vehicle.

A method for connecting a mobile camera device to a vehicle-mounted display device is known from the publication DE 10 2014 222 190 A1 famous.

The pamphlet DE 10 2014 218 921 A1 proposes to monitor automatic parking of a vehicle, with at least one video image of the parking process being recorded and sent to a mobile terminal device.

The invention is therefore based on the object of improving automated guidance of a motor vehicle with little technical effort with regard to involving a user and/or with regard to determining environmental information that can be used within the framework of automated guidance.

The object is achieved according to the invention by a method of the type mentioned at the outset, in which image data relating to the surroundings of the motor vehicle are acquired by a user-portable, vehicle-external acquisition device, with the image data being transmitted from the acquisition device to the motor vehicle, after which the driving intervention in is determined as a function of the image data, and/or wherein, on the other hand, driving intervention information relating to the driving intervention to be carried out is transmitted from the motor vehicle to the detection device, after which the detection device superimposes an image representation generated as a function of the image data with at least one graphic element generated as a function of the driving intervention information for is issued to the user, in both cases a relative position and/or orientation of the motor vehicle to the detection device or vice versa is determined, depending on the rel ative position and / or orientation of the driving intervention is determined and / or the graphic element is generated and / or the overlay of the graphic element is parameterized to the image display, the image data are recorded in such a way that at least one section of the motor vehicle is imaged, the relative position and/or orientation being determined by evaluating the image data.

According to the invention, it is proposed to use a vehicle-external detection device that can be worn by a user in order to provide additional sensor data to the motor vehicle that are used as part of the automated guidance and/or to inform a user about planned driving interventions. This exploits the fact that the detection devices that can be used in the method according to the invention are now often carried along by users anyway. For example, a mobile phone with an integrated camera can be used as a detection device. The use of so-called augmented reality devices, in particular corresponding glasses, which can also be used as detection devices in the method according to the invention, is also expected to increase.

The procedure described can thus be implemented with very little additional technical effort, since only communication between the detection device and a communication device on the motor vehicle must be enabled. However, since devices for wireless communication are typically already provided in motor vehicles, cell phones and augmented reality devices, the method according to the invention can often already be implemented by pure software adaptations. For example, a corresponding application or app can be installed on a mobile phone and the vehicle-side software components can be offered as a free or chargeable update.

The superimposition of the image representation with the generated graphic element can in particular be displayed on a screen, which can be, for example, the screen of a mobile phone or data glasses. In principle, however, it would also be possible to project this overlay directly into the user's eye or onto a projection surface in the vicinity of the user.

In the method according to the invention, the driving intervention can preferably be planned by a vehicle-side control device and carried out by controlling a corresponding actuator system in the motor vehicle. In principle, it would also be possible to plan the driving intervention using a vehicle-external control device. The vehicle-external detection device is to be understood as meaning a detection device that is configured separately from the motor vehicle and is located outside of the motor vehicle. When carrying out the method, it can be arranged at a distance from the motor vehicle and can be carried by a user, for example. For example, it can be held in the user's hand or attached to his body.

In the method according to the invention, a position and/or orientation of the motor vehicle relative to the detection device or vice versa is determined, with the driving intervention being determined and/or the graphic element generated and/or the overlay of the graphic element on the image representation depending on the relative position and/or orientation is parameterized. After the relative position and/or orientation has been determined, a common coordinate system for the planned driving intervention and/or the environmental information recorded by the sensor system of the motor vehicle and the image data can be determined, whereby the image data can be used to plan the driving intervention or to adapt the superimposed representation can. For example, the graphic element can thereby represent a trajectory or a trajectory section that is to be traversed due to the driving intervention with a correct position, orientation and/or scaling with respect to the motor vehicle. Expressed in general terms, the graphic element can be positioned, scaled, rotated and/or distorted as a function of the determined relative position and/or orientation with respect to the image representation. A distortion can occur, for example, by compressing, stretching and/or shearing the graphic element.

The position and/or orientation of the motor vehicle relative to the detection device or vice versa can be determined by determining absolute positions and/or orientations of the motor vehicle and the detection device. For this purpose, for example, satellite-based position determination systems, acceleration and/or magnetic field sensors or the like can be used. A relative position or orientation can be calculated from the respective absolute positions or orientations.

According to the invention, the image data is recorded in such a way that at least one section of the motor vehicle is imaged, with the relative position and/or orientation being determined by evaluating the image data. An image recognition algorithm can be used for this purpose, which can preferably be carried out on the acquisition device side or by a control device of the motor vehicle. A large number of methods are known in the prior art for recognizing specific objects in image data and for determining their relative position and/or orientation with respect to the capturing camera. For example, the motor vehicle or the illustrated section of the force vehicle can be recognized based on scale-invariant characteristics. Since individual pixels of the image data are associated with specific spatial angles in conventional camera sensors, the spatial angle in which the motor vehicle is located with respect to the detection device can be determined directly from the position of the motor vehicle or the corresponding features in the image. The size of the image of the motor vehicle and thus also the distance between different features, for example, varies with the distance of the motor vehicle from the camera, as a result of which this can also be determined from the image data.

In order to further improve recognition of the motor vehicle in the image data or determination of the relative position and/or orientation based on this information, it is advantageous if known features of the motor vehicle can be used in the context of image data processing. It is therefore possible for at least one item of motor vehicle information relating to the motor vehicle to be stored or received by the detection device, with the motor vehicle or the section of the motor vehicle being recognized on the basis of the motor vehicle information in the image data. The motor vehicle information can, for example, describe specific patterns or scale-invariant characteristics of the motor vehicle. Corresponding information can be provided by the motor vehicle to the detection device, for example when the detection device is paired with the motor vehicle. Such a pairing often takes place anyway for mobile phones, for example, which can serve as a detection device, in order to be able to use a hands-free device in the motor vehicle, for example. Alternatively, it would be possible, for example, for a user to be instructed as part of a preceding calibration to record the motor vehicle from specific perspectives, after which this data can then be used as motor vehicle information.

At least one piece of detection device information relating to the detection device can be stored or received in the motor vehicle, after which the driving intervention information is transmitted and/or the driving intervention is determined as a function of the detection device information. For example, different protocols or information representations can be supported on the motor vehicle side in order to provide the driving intervention information to the detection device. Depending on a type of acquisition device as acquisition device information, a corresponding protocol or a corresponding information display can thus be selected. However, the procedure explained is particularly advantageous in order to provide recording parameters of the detection device or a camera of the detection device to the motor vehicle in order to promote the processing of the image data there. For example, a focal length and/or a width and/or height of the image sensor can be present or received as detector information. It is also possible to use information about the distortion of the image data, for example due to lens distortions, as acquisition device information, as a result of which the fusion of the image data with the data from the vehicle-side sensor system can be further improved.

The image display can show the area surrounding the motor vehicle from the perspective from which the image data is recorded. In particular, the motor vehicle is also depicted from this perspective. In particular, the perspective cannot be converted, for example for display from a bird's-eye view, but an essentially unchanged or natural image display can take place. If, for example, a screen is used that is located on the opposite side of the housing from the camera, as is the case with many cameras or mobile phones, the detection device appears more or less transparent, which means that an augmented reality display can be implemented by overlaying the graphic element. A corresponding representation can be grasped particularly intuitively by a user.

The graphic element can be generated and superimposed with the image representation in such a way that it visualizes at least a section of a trajectory along which the motor vehicle is to be guided by the driving intervention. The trajectory can be represented, for example, by a central line along which the motor vehicle is to move, by boundary lines that characterize an area traveled by the motor vehicle, by at least one arrow or the like. In addition or as an alternative, specific symbols can be displayed as graphic elements, which visualize individual driving interventions, for example braking, accelerating or steering. Corresponding symbols can be displayed in particular at corresponding positions of a trajectory of the motor vehicle.

If a driving intervention or a specific sequence of driving interventions is visualized for a user, it is advantageous if he can also interact with the motor vehicle by means of appropriate operating inputs on the detection device. In the simplest case, a driving intervention or a group of driving interventions can only be carried out when the execution has been confirmed or released at the detection device. It is particularly beneficial However, if the driving intervention or driving interventions can be adapted by a user directly or by other specifications. It is therefore possible for at least one operator input from the user to be recorded on the acquisition device, after which the driving intervention is determined or adjusted as a function of the operator input. The operator input or information determined therefrom can be transmitted to the motor vehicle.

In a particularly advantageous variant of the described procedure, a preliminary driving intervention is first determined, the corresponding driving intervention information is transmitted to the detection device and the preliminary driving intervention is visualized there as explained above. If the visualized driving intervention is confirmed by the user or if a certain intervention interval elapses, this driving intervention is carried out. However, if the user carries out appropriate operating inputs, this information can be transmitted to the motor vehicle and the driving intervention can be adjusted accordingly.

At least one blocked area can be defined by the operator input, after which the driving intervention is determined in such a way that the motor vehicle is guided exclusively outside the blocked area. The driving intervention or the trajectory of the motor vehicle can thus be planned in such a way that the motor vehicle does not enter the restricted area, ie the restricted area is not driven over within the scope of the automated guidance. For example, the restricted area may be an obstacle detected by the user. However, it is also possible to mark areas that would in principle be passable, but which should not be driven on, such as, for example, unclear junctions with roads or paths, footpaths or the like.

In some cases it can also be advantageous to define restricted areas by the user selecting at least one drivable area, after which all non-selected areas are interpreted as restricted areas. In this way, the maneuvering range for the automated guidance can be specifically restricted to certain areas.

The driving intervention can be planned as part of an automated parking process or exit process. With these processes, it is possible that the user is in the vicinity of the motor vehicle, but outside of the vehicle. In this case, it is possible that image data of relevant areas that cannot be seen by the sensor system on the motor vehicle can be captured using the capturing device, for example using a mobile phone. On the other hand, a future planned behavior of the motor vehicle can be visualized for the user by a previously explained image representation of the motor vehicle environment with at least one superimposed graphic element, so that the user can monitor the motor vehicle better or intervene in a targeted manner in the future driving operation.

The sensing device may be or comprise a mobile phone, or the sensing device may comprise an output device worn on the user's head for outputting the overlay. For example, the detection device can be data or VR glasses.

In addition to the method according to the invention, the invention relates to a motor vehicle with a sensor system on the motor vehicle for detecting environmental information relating to the area surrounding the motor vehicle and a control device, by means of which at least one driving intervention to be carried out as part of automated guidance of the motor vehicle can be determined as a function of the environmental information, the motor vehicle having a by the control device controllable communication device for communication with a vehicle-external detection device, wherein the control device is set up to carry out the method according to the invention.

The invention also relates to an acquisition device with a communication device, at least one camera for acquiring image data and a control device for controlling the communication device and the camera, the control device being set up to carry out the method according to the invention. The detection device preferably includes a display device that can be controlled by the control device, for example a display or a projection device, in order to display the superimposition of the image representation with the graphic element. The detection device can, for example, be a cell phone on which an application for carrying out the method according to the invention has been installed. It is also possible for the detection device to be data or VR glasses, through which an augmented reality display is possible directly in the user's field of vision.

The motor vehicle according to the invention or the detection device according to the invention can be further developed with the features explained for the method according to the invention with the advantages mentioned there.

• 1 eine Fahrsituation, in der ein Ausführungsbeispiel des erfindungsgemäßen Verfahrens durch ein Ausführungsbeispiel einer erfindungsgemäßen Erfassungseinrichtung und ein Ausführungsbeispiel eines erfindungsgemäßen Kraftfahrzeugs durchgeführt wird,
• 2 eine Detailansicht der in 1 gezeigten Erfassungseinrichtung, und
• 3 eine Bildschirmdarstellung auf einem Bildschirm dieser Erfassungseinrichtung.

Further advantages and details of the invention result from the following description examples and the associated drawings. Here show schematically:

• 1 a driving situation in which an exemplary embodiment of the method according to the invention is carried out by an exemplary embodiment of a detection device according to the invention and an exemplary embodiment of a motor vehicle according to the invention,
• 2 a detailed view of the in 1 detection device shown, and
• 3 a screen display on a screen of this detection device.

1 shows a driving situation in which a method for the automated guidance of a motor vehicle 1 is carried out. A sensor system 2 on the motor vehicle detects environmental information relating to the environment of motor vehicle 1 , and a control device 3 determines a driving intervention to be carried out as a function of this environmental information. In the driving situation shown, the driving intervention is intended to serve to park the motor vehicle 1 in the parking position 12 . In order to carry out the driving intervention, the control device 3 can activate an actuator system 4 of the motor vehicle in order to intervene in the longitudinal and lateral guidance of the motor vehicle 1 .

It is known that automatic parking maneuvers can also be carried out in situations in which a user 7 is outside the motor vehicle 1, as is the case in 1 is shown. Preferably, however, the user 7 should still be given the opportunity to monitor the driving operation of the motor vehicle 1 and, in particular, to obtain information about them even before automated driving interventions are carried out.

This is realized in that a detailed in 2 shown, carried by the user 7, vehicle-external detection device 6 is used, which communicates via a communication device 10 of the detection device and a communication device 5 of the motor vehicle 1 with the motor vehicle 1. The detection device includes a control device 9 which controls the communication device 10 and a screen 11 of the detection device 6 and which can record image data of the area surrounding the detection device via a camera 8 . A cell phone, for example, can be used as a corresponding detection device.

After a driving intervention to be carried out has been determined by the control device 3 of the motor vehicle 1 , driving intervention information relating to this can be transmitted from the motor vehicle 1 to the detection device 6 . The control device 9 can then control the screen 11 in order to visualize the driving intervention to be carried out for the user 7 . As will be explained in detail below, this takes place as an augmented reality display.

The image representation on the screen 11 of the detection device 6 is in 3 shown. This screen image includes several overlaid components. The screen display is based on an image display 15 of the detection area 16 detected by the camera 8. The objects in the area surrounding the motor vehicle 1 that can be detected by the camera 8 can therefore be seen on the screen 11, of which a parked additional motor vehicle 13 and an obstacle 14 are shown as an example are. In addition, a section of the motor vehicle 1 is displayed. Based on the image of the motor vehicle 1, a relative position of the motor vehicle 1 to the detection device 6 is determined by feature recognition in the image data, so that information about the planned driving interventions can be displayed in a correct positional relationship to the motor vehicle 1.

A plurality of graphic elements 17, 18, which were determined as a function of the driving intervention information received from motor vehicle 1, are superimposed on this image representation. In the example shown, the graphic elements 17, 18 represent the trajectory that the motor vehicle 1 travels when carrying out the planned driving interventions, in the form of arrows. Of course, any other representations of the driving interventions would also be possible. The user 7 thus receives a preview of the driving interventions as they would appear from his viewing position. If he agrees to the determined driving interventions, he can authorize the implementation of these driving interventions by actuating the control element 19 that is additionally displayed in the image display on the screen 11 designed as a touchscreen.

In a further development of the exemplary embodiment explained, it would be possible for the obstacle 14 not to be recognized as an area that cannot be driven over. For example, the obstacle 14 could be an unplanted bed that the control device 3 of the motor vehicle 1 initially classifies as passable. In this case, for example, the alternative trajectory shown by the dashed arrow 20 for the motor vehicle 1 could be calculated and visualized for the user 7 . The user 7 can now see that this trajectory 20 would not be advantageous since it could lead to damage to the bed and potentially also to the motor vehicle 1 . The user 7 can now, instead of the previously explained release of the driving interventions, an operator input on the Activate detection device 6, in particular on screen 11 designed as a touchscreen, in order to adjust the driving intervention. It would in principle be possible here to adjust the trajectory 20 manually, for example by dragging its end point. However, it is particularly advantageous if at least one blocking area 21 is defined by the operator input. For example, the user can draw in a rectangle that encompasses the obstacle 14 so that the area of the obstacle 14 is marked as impassable. Corresponding information can be transmitted back to the motor vehicle 1, after which an updated trajectory or new driving interventions are determined by the control device 3. With regard to these new driving interventions, driving intervention information can then be transmitted again to the detection device 6 in order to visualize this for the user 7 .

The image data from the detection device can also be used to provide additional information about the surroundings of motor vehicle 1 to control device 3 in order to improve planning of the driving interventions. For example, it would be possible that in the in 1 The exemplary embodiment shown, the obstacle 14 partially covers the area of the parking position 12 or the maneuvering area used for parking for the sensor system 2 of the motor vehicle 1 . In order to at least partially compensate for this, the acquisition device 6 can transmit the image data acquired via the camera 8 to the motor vehicle 1 so that they can be taken into account as an additional source of information in the context of determining the driving intervention to be carried out. The use of an additional detection device external to the vehicle can thus also serve to enable more robust planning of driving interventions.

Claims (11)

Method for the automated guidance of a motor vehicle (1), wherein environmental information relating to the area surrounding the motor vehicle (1) is recorded by a sensor system (2) on the motor vehicle and at least one driving intervention to be carried out as part of the automated guidance is determined as a function of the environmental information, wherein a Image data relating to the surroundings of the motor vehicle (1) are captured by a user (7) portable, vehicle-external capture device (6), with the image data being transmitted from the capture device (6) to the motor vehicle (1), after which the driving intervention is carried out as a function of the image data is determined and/or wherein, on the other hand, driving intervention information relating to the driving intervention to be carried out is transmitted from the motor vehicle (1) to the detection device (6), after which the detection device (6) superimposes an image display (15) generated as a function of the image data with least ns a graphic element (17, 18) generated as a function of the driving intervention information for the user (7), wherein in both cases a relative position and/or orientation of the motor vehicle (1) to the detection device (2) or vice versa is determined, wherein the driving intervention is determined as a function of the relative position and/or orientation and/or the graphic element (17, 18) is generated and/or the superimposition of the graphic element (17, 18) on the image display (15) is parameterized, with the image data being recorded in this way that at least a portion of the motor vehicle (1) is imaged, wherein the relative position and / or orientation is determined by evaluating the image data. procedure after claim 1 , characterized in that at least one piece of motor vehicle information relating to the motor vehicle (1) is stored or received by the detection device (6), the motor vehicle (1) or the section of the motor vehicle (1) being recognized on the basis of the motor vehicle information in the image data. Method according to one of the preceding claims, characterized in that at least one piece of detection device information relating to the detection device (6) is stored or received on the motor vehicle side, after which the driving intervention information is transmitted and/or the driving intervention is determined as a function of the detection device information. Method according to one of the preceding claims, characterized in that the image representation (15) shows the surroundings of the motor vehicle (1) from the perspective from which the image data are recorded. Method according to one of the preceding claims, characterized in that the graphic element (17, 18) is generated and overlaid with the image representation (15) in such a way that it visualizes at least a section of a trajectory along which the motor vehicle (1) is guided by the driving intervention shall be. Method according to one of the preceding claims, characterized in that at least one operator input from the user (7) is recorded on the acquisition device (6), after which the driving intervention is determined or adapted as a function of the operator input. procedure after claim 6 , characterized in that by the operator input at least one blocked area (21) is defined, after which the driving intervention is determined in such a way that the motor vehicle (1) is guided exclusively outside the blocked area (21). Method according to one of the preceding claims, characterized in that the driving intervention is planned as part of an automated parking process or parking space exit process. Method according to one of the preceding claims, characterized in that the detection device (6) is or comprises a mobile telephone or that the detection device (6) comprises an output device worn on the user's head for outputting the overlay. Motor vehicle with an on-board sensor system for detecting environmental information relating to the area surrounding the motor vehicle (1) and a control device (3) by means of which at least one driving intervention to be carried out as part of automated guidance of the motor vehicle (1) can be determined, characterized in that that it comprises a communication device (5) controllable by the control device (3) for communication with a vehicle-external detection device (6), wherein the control device (3) is set up to carry out the method according to one of the preceding claims. Recording device with a communication device (10), at least one camera (8) for recording image data and a control device (9) for controlling the communication device (10) and the camera (8), characterized in that the control device (9) for carrying out a Procedure according to one of Claims 1 until 9 is set up.

Images