DE102017205194A1 - Method and device for providing 3D image information in a driver assistance system of a motor vehicle - Google Patents

Abstract

The invention relates to a method for operating a system with data glasses and a driver assistance system in a motor vehicle, comprising the following steps:
- Detecting a vehicle camera image of the vehicle environment using at least one vehicle camera;
- acquiring an environment data camera image from the perspective of the goggle wearer using a data camera;
- determining a pose of the data glasses;
Determining a common overlap area of the data camera image and the vehicle camera image;
Determining depth information for the overlap area depending on the pose of the data glasses; and
Performing an application function based on the depth information of the overlap area.

Description

Technical area

The invention relates to driver assistance systems with display systems for data glasses, in particular for use in a motor vehicle. The invention further relates to camera systems in motor vehicles and the provision of 3D image information based on recordings in vehicle cameras of the camera system.

Technical background

Driver assistance systems are electronic auxiliary devices in motor vehicles to assist the driver in certain driving situations. Here, safety aspects and the enhancement of driving comfort are often in the foreground.

Driver assistance systems can be in communication with data glasses, also known as head-mounted displays (HMD). Such data glasses can display a picture on one or two display surfaces in the field of vision of the wearer of the data glasses by means of a display device. The display surfaces correspond to reflection surfaces that direct images into the eye of the wearer of the data glasses. The viewing apertures of the data glasses are transparent, so that the data glasses can be used to perceive the real environment in the usual way. The display surfaces are located in the viewing apertures and may be semitransparent, so that information to be displayed, such as text, symbols, graphics, video displays and the like, can be superimposed on the perception of the environment. In addition, data glasses are also known which comprise a non-transparent display.

The information is usually presented to the wearer of the data glasses contact-analogous, i. represented such that the information is superimposed on or oriented towards a specific object in the real environment or that the information to be displayed is displayed in a specific orientation of the data glasses or their wearer. Furthermore, it is desirable to display the information so that it appears perspectively correct with respect to the object in the real environment, i. the illusion arises that the object of the real environment has actually been supplemented by the additional feature of visual information.

In order to display the information corresponding to the contact analog on the display surfaces of the data glasses, it is necessary to determine the position of the object in the environment and the pose of the data glasses, i. the 3D position and the 3D orientation of the data glasses, to know. To determine the pose of the data glasses, a pose detection unit can be provided in the data glasses. The pose detection unit typically includes a data camera and computing device, e.g. in the form of a microprocessor, on. Using the camera recorded images of the environment of the wearer of the data glasses can be determined based on deposited images or structures of the vehicle interior, the pose of the data glasses in the vehicle interior. This process is also called tracking.

This is from the publication DE 10 2014 206 623 A1 a device for determining the pose of data goggles is known, comprising a display and a camera. The device is designed to take pictures of the surroundings of the data glasses with the aid of the camera, to detect the image of a stored and predefined area of the surroundings in the recordings of the camera, to detect a feature in the recognized image of the area and to reflect the pose of the data glasses Consideration of the particular feature in the recordings.

For certain driver assistance functions, a three-dimensional detection of objects in the vehicle environment is necessary. These include, for example, a three-dimensional spatial representation of the vehicle environment in the data glasses and other functions based on the position of environment objects. An environment detection device, such as is provided in motor vehicles, has a camera system with a plurality of vehicle cameras, which detect the surroundings of the vehicle. However, often only a two-dimensional detection of the vehicle environment is provided because the equipment of the environment detection device with stereo cameras or the like is expensive. Also, in motor vehicles in which the environment detection device is formed with a camera system for spatial detection, often the arrangement of the vehicle cameras so that the accuracy of the three-dimensional detection depends strongly on their positioning.

It is an object of the present invention to provide an improved method and detection system for the three-dimensional optical detection of environmental objects of a motor vehicle.

Disclosure of the invention

This object is achieved by the method for operating a driver assistance system with data glasses in a motor vehicle according to claim 1 and by the device and the system according to the independent claims.

Further embodiments are specified in the dependent claims.

• - Erfassen eines Fahrzeugkamerabilds der Fahrzeugumgebung mithilfe mindestens einer Fahrzeugkamera;
• - Erfassen eines Datenbrillenkamerabilds der Umgebung aus Sicht des Trägers der Datenbrille mithilfe einer Datenbrillenkamera;
• - Bestimmen eines gemeinsamen Überlappungsbereichs des Datenbrillenkamerabilds und des Fahrzeugkamerabilds;
• - Bestimmen einer Tiefeninformation für den Überlappungsbereich abhängig von einer Pose der Datenbrille; und
• - Durchführen einer Anwendungsfunktion basierend auf der Tiefeninformation des Überlappungsbereichs.

According to a first aspect, a method is provided for operating a system with data goggles and a driver assistance system in a motor vehicle, comprising the following steps:

• - Detecting a vehicle camera image of the vehicle environment using at least one vehicle camera;
• - acquiring an environment data camera image from the perspective of the goggle wearer using a data camera;
• Determining a common overlap area of the data camera image and the vehicle camera image;
• Determining a depth information for the overlap area depending on a pose of the data glasses; and
• Performing an application function based on the depth information of the overlap area.

One idea of the above method is to enable stereoscopic spatial detection of environmental objects by the functional combination of the at least one vehicle camera and the data-reading camera used, for example, for pose detection. Due to the spatial offset of the vehicle camera and the data camera, an overlapping area can arise, in particular in the direction of view of the wearer, in which a section of the vehicle surroundings is detected by both cameras. In the overlap area, surrounding objects are then acquired from different perspectives, so that a depth information of the surrounding objects can be determined therefrom.

In this way, it is possible to obtain a spatial representation of the vehicle surroundings only with a few vehicle cameras provided in the motor vehicle in conjunction with data glasses with a data camera in the direction of the data glasses.

Furthermore, the application function can be performed in the data glasses or in the driver assistance system.

It can be provided that the pose of the data glasses is determined by means of the data glasses camera image in the data glasses or in the driver assistance system.

In particular, the pose of the data glasses can be determined by means of a pattern recognition method for recognizing distinctive structures in the vehicle interior in the data glasses. This represents a possibility to determine the pose of the data glasses in the data glasses themselves.

• - Ermitteln eines räumlichen Abstands zwischen der Position der Datenbrille und einer bekannten Position der Fahrzeugkamera;
• - Bestimmen der Tiefeninformation basierend auf den korrespondierenden Überlappungsbereichen und dem räumlichen Abstand zwischen der Position der Datenbrille und der bekannten Position der Fahrzeugkamera.

According to one embodiment, determining depth information for the overlap region may include, depending on the pose of the data goggles:

• Determining a spatial distance between the position of the data glasses and a known position of the vehicle camera;
• Determining the depth information based on the corresponding overlap areas and the spatial distance between the position of the data glasses and the known position of the vehicle camera.

The position and orientation of the at least one vehicle camera of the camera system are constant and known in the vehicle coordinate system. The data camera is usually used for detecting the pose of the data glasses and it is therefore always a current pose of the data glasses with respect to the vehicle coordinate system before. From the pose of the data glasses, this results in a spatial distance to the at least one vehicle camera. Thus, by merging the image information in conjunction with an indication of the pose of the data glasses, a three-dimensional depth map can be determined on the basis of the simultaneously recorded images of the vehicle cameras and the data glasses cameras and the pose of the data camera.

Further, determining the common overlap area of the data camera image and the vehicle camera image may include identifying those vehicle environment imaging areas recorded by both the vehicle camera and the data camera. This may be done by a pattern or image recognition method that identifies an identical image area in the data camera image and the vehicle camera image as the common overlap area. Alternatively, by the pose of the data glasses, the area of the environment imaged by the data camera image can be identified and, given a known orientation of the vehicle camera, the overlapping area of the images recorded by both cameras can be determined by calculation.

• - eine Lokalisierung von Umgebungsobjekten für eine Gefährdungs- oder Kollisionserkennung, und
• - eine Erkennung und Anzeige von Entfernungen einzelner Umgebungsobjekte.

Performing the application function based on the depth information may include at least one of the following functions:

• - a localization of environment objects for hazard or collision detection, and
• - Detection and display of distances of individual environment objects.

In particular, performing the application function based on the depth information may include a plausibility check of a depth information determined by means of vehicle cameras.

• - ein Fahrzeugkamerabild der Fahrzeugumgebung mithilfe mindestens einer Fahrzeugkamera zu erfassen;
• - ein Datenbrillenkamerabild der Umgebung aus Sicht des Trägers der Datenbrille von einer Datenbrillenkamera zu empfangen;
• - eine Pose der Datenbrille zu erfassen oder zu empfangen;
• - einen gemeinsamen Überlappungsbereich des Datenbrillenkamerabilds und des Fahrzeugkamerabilds zu bestimmen;
• - eine Tiefeninformation für den Überlappungsbereich abhängig von der Pose der Datenbrille zu bestimmen; und
• - eine Anwendungsfunktion basierend auf der Tiefeninformation des Überlappungsbereichs auszuführen.

According to a further aspect, a driver assistance system for a display system with data glasses in a motor vehicle is provided, wherein the driver assistance system is designed to:

• to detect a vehicle camera image of the vehicle environment using at least one vehicle camera;
• to receive a data-goggle camera image of the environment from the perspective of the wearer of the data goggles from a data-reading camera;
• to capture or receive a pose of the data glasses;
• to determine a common overlap area of the data camera image and the vehicle camera image;
• - To determine a depth information for the overlap area depending on the pose of the data glasses; and
• to execute an application function based on the depth information of the overlap area.

According to a further aspect, a display system with data glasses and the above driver assistance system is provided.

list of figures

• 1 eine schematische Darstellung eines Anzeigesystems mit einem Fahrerassistenzsystem und einer Datenbrille zum Einsatz in einem Kraftfahrzeug; und
• 2 ein Flussdiagramm zur Veranschaulichung eines Verfahrens zum Erfassen einer Tiefeninformation für optisch zu erfassende Umgebungsobjekte.

Embodiments are explained below with reference to the accompanying drawings. Show it:

• 1 a schematic representation of a display system with a driver assistance system and a data glasses for use in a motor vehicle; and
• 2 a flowchart illustrating a method for detecting a depth information for optically detected environment objects.

Description of embodiments

1 shows a schematic representation of a display system 1 in particular for use in a motor vehicle. The display system 1 includes a driver assistance system 2 that in communication connection 4 with a data glasses 3 stands. The communication connection 4 is designed as a data transmission channel 4, for example in the form of a wireless communication connection or a wired communication connection. The communication connection 4 is capable of any kind of data and information between the driver assistance system 2 and the data glasses 3, for example, based on a packet-bound data transmission.

The data glasses 3 is with two transparent lenses 32 , which are enclosed in a frame 31 in a conventional manner. The frame 31 is provided with eyeglass temple 33, so that the data glasses 3 can be worn on the head of a user in a conventional manner.

The lenses 32 are each further provided with a transparent display surface 35, in which by a suitable device, such as one on the frame 31 arranged display device 36 , a display image can be displayed. The display device 36 may comprise a microprocessor or a comparable computing unit and a display unit, such as a projection device or the like. The display unit may be configured to display an electronically generated display image on the display surface 35 display.

Due to the transparent design of the display surface 35 The electronically generated image can pass through the display area 35 Superimposed perceptible real environment. Using the display device 36 may display information such as a text, a symbol, video information, graphics or the like on one or both display surfaces 35 being represented.

The data glasses 3 can be worn like a typical visual aid on the user's head, the data glasses 3 with the frame 31 rests on the nose of the user and the hanger 33 abut laterally on the user's head.

The viewing direction of the user in the straight-ahead direction then takes place through the viewing windows 32 essentially through the transparent display surfaces 35 , so that the user's line of sight of the orientation of the data glasses 3 corresponds or can be derived from it.

Furthermore, the data glasses 3 with a control unit 37 be provided with the display device 36 is coupled to the display on the display surfaces 35 to determine. Furthermore, the control unit is used 37 to detect a pose of the data glasses 3 in a vehicle interior. The Pose of data glasses 3 herein designates the spatial position of the data glasses 3 in a vehicle-fixed coordinate system and their three-dimensional orientation. The control unit 37 can do this with a data camera 38 connected in the direction of the wearer of the data glasses 3 is aligned and has a fixed relation to this. The pose of the data glasses 3 is determined to use pattern recognition techniques to detect distinctive structures within the cockpit and from this the pose of the data glasses 3 in a conventional manner using a known localization algorithm to determine.

In particular, the localization algorithm detects the position of several distinctive structures in the vehicle interior and can via their positions in by the data camera 38 recorded camera image the orientation of the data glasses 3 recognize and their distances in the camera image to each other the distance of the data glasses 3 be determined by the distinctive structures. Since the distinctive structures have a known position in the vehicle coordinate system, a position of the data goggles can take their distances in the camera image 3 be determined in the vehicle coordinate system.

The driver assistance system 2 is with a camera system 21 connected to the vehicle cameras 22 to record a portion of the vehicle environment. For example, one of the vehicle cameras 22 in the region of the rearview mirror with an orientation in the advance direction of the motor vehicle and / or another vehicle camera 22 be arranged in the range of one or both side mirrors with a lateral orientation to the motor vehicle.

The driver assistance system 2 needed for the visual representation of real with the camera system 21 recorded environmental objects, a depth information, ie a stereoscopic detection of these environmental objects, in particular those environment objects that are located in the immediate vicinity of the motor vehicle. This allows a variety of application functions, which can be realized in particular in connection with displays in the data glasses 3, such as a localization of environment objects for a hazard or collision detection, detection and display of distances of individual environment objects and the like.

The driver assistance system 2 performs together with the control unit 37 of the data glasses 3, a method by which one can obtain depth information about surrounding objects. This method will be described with reference to the flowchart of FIG 2 explained in more detail.

For this purpose, in step S1, first with the help of the at least one vehicle camera 22 a vehicle camera image of the vehicle environment detected. Essentially at the same time in step S2 by the data camera 38 a data camera image of the environment from the perspective of the wearer of the data glasses 3 detected.

In step S3, a pose of the data glasses 3 is determined on the basis of the data glasses camera image, so that their spatial position in the vehicle interior and viewing direction can be detected.

In step S4, the data camera image and the pose in which it has been recorded are communicated via the communication link 4 transmitted to the driver assistance system 2.

In the driver assistance system 2 be in a subsequent step S5 from the predetermined fixed position of the vehicle camera 22 as well as their orientation and the orientation of the data glasses 3 Overlaps between the vehicle camera image of the vehicle camera 22 and the data glasses camera image of the data glasses 3 determined. This results in at least one overlapping region of the data camera image and a respective overlapping region of the vehicle camera image.

In step S6, a spatial distance between the position of the data glasses 3 and the known position of the respective vehicle camera 22 determined.

For the corresponding overlapping areas in the data-frame camera image and in the vehicle camera image, depth information of the objects located thereon is determined in step S7. This can be done by a known per se evaluation of recorded from different perspectives camera images

The depth information determined in step S7 can now be used in the driver assistance system 2 for carrying out suitable application functions.

In an alternative embodiment, the vehicle camera image can also be transmitted to the data glasses 3, so that the detection of the depth information is not in the driver assistance system 2 but in the control unit 37 the data glasses 3 can be performed according to the method described above.

Overall, the procedure described above makes it possible to provide stereoscopic vehicle cameras or vehicle cameras overlapping with the detection area to give up to obtain a three-dimensional detection of the vehicle environment. By using the data glasses camera image, it is possible, at least in the direction of the wearer of the data glasses 3 the corresponding depth information for objects in the field of vision of the wearer of the data glasses 3 to investigate.

In camera systems with vehicle cameras, which permit a three-dimensional detection of the surroundings of the motor vehicle, the data camera image can also be used to specify already obtained depth information by selecting one of the vehicle camera images and using the data spectacle camera image.

LIST OF REFERENCE NUMBERS

1

display system

2

Driver assistance system

21

Environment detection device

22

vehicle cameras

3

data glasses

31

frame

32

visors

33

Temples

35

transparent display area

36

display

37

control unit

38

Data POV

4

Data transmission channel

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102014206623 A1 [0006]

Claims (10)

Method for operating a system (1) with data goggles (3) and a driver assistance system (3) in a motor vehicle, comprising the following steps: - detecting (S1) a vehicle camera image of the vehicle surroundings using at least one vehicle camera (22); - acquiring (S2) an environment's environmental camera image from the perspective of the goggle wearer using a data camera (38); Determining (S3) a pose of the data glasses; - determining (S5) a common overlap area of the data camera image and the vehicle camera image; - determining (S7) a depth information for the overlapping area depending on the pose of the data glasses (3); and - performing (S8) an application function based on the depth information of the overlapping area. Method according to Claim 1 , wherein the application function in the data glasses (3) or in the driver assistance system (2) is performed. Method according to Claim 1 or 2 in which the pose of the data goggles (3) is determined by means of the data glasses camera image in the data goggles (3) or in the driver assistance system (2). Method according to Claim 3 in which the determination of the pose of the data goggles (3) is carried out by means of a pattern recognition method for recognizing prominent structures in the vehicle interior in the data goggles (3). Method according to one of Claims 1 to 4 wherein determining depth information for the overlap region depends on the pose of the data glasses (3): - determining a spatial distance between the position of the data glasses (3) and a known position of the vehicle camera (22); - Determining the depth information based on the corresponding overlap areas and the spatial distance between the position of the data glasses (3) and the known position of the vehicle camera (22). Method according to one of Claims 1 to 5 wherein determining the common overlap area of the data camera image and the vehicle camera image comprises identifying those vehicle environment imaging areas recorded by both the vehicle camera (229 and the data camera (38)). Method according to one of Claims 1 to 6 wherein performing the application function based on the depth information comprises at least one of the following functions: - localization of environment objects for threat or collision detection, and - detection and display of distances of individual environment objects. Method according to one of Claims 1 to 7 wherein performing the application function based on the depth information comprises plausibility checking a depth information determined using one or more vehicle cameras (22). Driver assistance system for a display system (1) with data goggles (3) in a motor vehicle, wherein the driver assistance system (2) is designed to: to detect a vehicle camera image of the vehicle environment using at least one vehicle camera (22); - To receive a data glasses camera image of the environment from the perspective of the wearer of the data glasses (3) from a data camera; - to capture or receive a pose of the data glasses (3); to determine a common overlap area of the data camera image and the vehicle camera image; - To determine a depth information for the overlap area depending on the pose of the data glasses (3); and to execute an application function based on the depth information of the overlap area. Display system (1) with a data glasses (3) and a driver assistance system (2) according to Claim 9 ,

Images