DE102017215023A1 - A method, apparatus and computer readable storage medium having instructions for providing information for controlling a display of a head-up display device for a motor vehicle - Google Patents

Abstract

A method, an apparatus and a computer-readable storage medium having instructions for providing information for controlling a display of a head-up display device for a motor vehicle. In a first step, an optical flow from image data of a camera of the motor vehicle is determined (10). From the optical flow micro-movements of the motor vehicle are then determined (11). Information about the micromotion is then output to a controller of the head-up display device (12). The display of the head-up display device is finally adapted according to the micro-movements determined (13).

Description

The present invention relates to a method, a device and a computer-readable storage medium with instructions for providing information for controlling a display of a head-up display device for a motor vehicle. The invention further relates to a motor vehicle in which a method according to the invention or a device according to the invention is used.

With the continuous development of virtual and augmented reality technologies and applications, these are finding their way into the automobile. Augmented Reality (AR), augmented reality, is the enrichment of the real world by virtue of virtual elements that are registered in three-dimensional space with the correct location and allow real-time interaction. Since the term "augmented reality" has prevailed in the German-speaking world compared with the term "augmented reality", the former is used in the following.

A possible technical realization to enrich the driver's workplace with perspectively correct virtual extensions is provided by the Head-Up-Display (HUD). The light rays of a built-in dashboard display are folded over several mirrors and lenses and mirrored on a projection screen in the eye of the driver so that he perceives a virtual image outside the vehicle. In the automotive sector, the front screen, whose curved shape must be taken into account in the illustration, often serves as the projection surface. As an alternative, an additional disc made of glass or plastic is used in part, which is arranged between the driver and the windshield on the dashboard. Due to the optical overlay of the display and the driving scene, fewer head and eye movements are required to read the information. In addition, the adaptation effort for the eyes is reduced since, depending on the virtual distance of the display, less, if at all, does not have to be accommodated. Current serial HUDs, however, are limited to displays just below the driver's primary field of view and contain redundant information, such as information on the driver's primary viewing area. the speedometer, which can also be found elsewhere in the cockpit. Although this display technology reduces street glare, it still has the disadvantage that the information presented has to be interpreted and transferred to the real situation since it is not registered in the real scene. In complex situations, this can be a mentally demanding cognitive process. By marking objects and displaying information at their real reference location, i. Through a contact-analogue representation, environmental information can be displayed directly in the driver's field of vision. This direct visual enrichment of the environment in the form of augmented reality can significantly reduce cognitive transfer requirements.

In this context, the DE 10 2014 214 516 A1 a device, in particular a data glasses or a head-up display, for reproducing data in an augmented reality for an occupant of a vehicle. The device is set up to display data, in particular visual media contents, in an image section of the field of view of the occupant, which currently and probably also in the future does not contain any environmental information essential for traffic around the vehicle.

Augmented Reality offers a wide range of possible applications to support the driver by contact-analogous marking of lanes and objects. Relatively obvious examples mostly relate to the field of navigation. While classic navigation displays in conventional HUDs typically display schematic representations, e.g. a right arrow pointing to the right as a sign that you want to turn right at the next opportunity offers AR ads much more effective options. Since the ads can be viewed as "part of the environment", extremely fast and intuitive interpretations by the user are possible. In modern motor vehicles, a wide variety of data is captured by environmental sensors. The aim of current developments is to meaningfully integrate this data into augmented reality representations.

Registration of the virtual information in the driver's real field of vision in the sense of augmented reality places very high demands on the technical implementation. In order to be able to represent virtual contents in the right place and perspective in the real world, very detailed knowledge about the environment and proper movement of the vehicle is necessary.

Against this background describes the DE 10 2013 226 969 A1 a head-up display device for a vehicle. The head-up display device utilizes an augmented reality representation and is capable of correcting a transverse error between an actual front vehicle position in front of the vehicle and its position in a graphical image when the front vehicle is on a curved road moves and the graphic image of the front vehicle is generated as an AR representation.

A major problem in the generation of augmented reality representations is the Detection and compensation of micro-movements of the motor vehicle, which can be caused for example by bumps. Usually, special motion sensors are installed for the detection of micro-movements, for example, gyroscopes or accelerometers. However, the installation of such sensors is associated with additional costs.

It is an object of the invention to provide alternative solutions for providing information for controlling a display of a head-up display device.

This object is achieved by a method having the features of claim 1, by a computer-readable storage medium with instructions according to claim 7, by an apparatus having the features of claim 8 and by a system having the features of claim 10. Preferred embodiments of the invention are the subject of the dependent claims.

• - Bestimmen eines optischen Flusses aus Bilddaten einer Kamera des Kraftfahrzeugs;
• - Ermitteln von Mikrobewegungen aus dem optischen Fluss; und
• - Ausgeben von Informationen zu den Mikrobewegungen an ein Steuergerät der Head-up-Display-Vorrichtung.

According to a first aspect of the invention, a method for providing information for controlling a display of a head-up display device for a motor vehicle comprises the steps of:

• Determining an optical flow from image data of a camera of the motor vehicle;
• - Detecting micro-movements from the optical flow; and
• - Output information about the micro-movements to a control unit of the head-up display device.

• - Bestimmen eines optischen Flusses aus Bilddaten einer Kamera des Kraftfahrzeugs;
• - Ermitteln von Mikrobewegungen aus dem optischen Fluss; und
• - Ausgeben von Informationen zu den Mikrobewegungen an ein Steuergerät der Head-up-Display-Vorrichtung.

In accordance with another aspect of the invention, a computer-readable storage medium includes instructions that, when executed by a computer, cause the computer to perform the following steps to provide information for controlling a display of a head-up display device for a motor vehicle:

• Determining an optical flow from image data of a camera of the motor vehicle;
• - Detecting micro-movements from the optical flow; and
• - Output information about the micro-movements to a control unit of the head-up display device.

The term computer is to be understood broadly. In particular, it also includes controllers and other processor-based data processing devices.

• - Eine Bildverarbeitungseinheit zum Bestimmen eines optischen Flusses aus Bilddaten einer Kamera des Kraftfahrzeugs;
• - Eine Analyseeinheit zum Ermitteln von Mikrobewegungen aus dem optischen Fluss; und
• - Einen Ausgang zum Ausgeben von Informationen zu den Mikrobewegungen an ein Steuergerät der Head-up-Display-Vorrichtung.

According to a further aspect of the invention, a device for providing information for controlling a display of a head-up display device for a motor vehicle comprises:

• An image processing unit for determining an optical flow from image data of a camera of the motor vehicle;
• An analysis unit for determining micro-movements from the optical flow; and
• - An output for outputting information about the micro-movements to a control unit of the head-up display device.

• - Ein Steuergerät für die Head-up-Display-Vorrichtung;
• - Eine Bildverarbeitungseinheit zum Bestimmen eines optischen Flusses aus Bilddaten einer Kamera des Kraftfahrzeugs;
• - Eine Analyseeinheit zum Ermitteln von Mikrobewegungen aus dem optischen Fluss; und
• - Einen Ausgang zum Ausgeben von Informationen zu den Mikrobewegungen an das Steuergerät der Head-up-Display-Vorrichtung.

According to another aspect of the invention, a system for controlling a display of a head-up display device for a motor vehicle comprises:

• - A control device for the head-up display device;
• An image processing unit for determining an optical flow from image data of a camera of the motor vehicle;
• An analysis unit for determining micro-movements from the optical flow; and
• - An output for outputting information about the micro-movements to the control unit of the head-up display device.

According to the invention, an image flow or optical flow detected from the image data of a camera of the motor vehicle is used to determine relative micro-movements of the motor vehicle. The optical flow is already being used for different functions, for example for the collision detection by a collision avoidance assistant, and can therefore be used for a single cost. At the same time eliminates the need for special additional motion sensors.

According to one aspect of the invention, the determination of the optical flow and the determination of the micro-movements are carried out in the camera or in a control device of the camera. This approach has the advantage that the micromovements can be determined as close as possible to the source of the image data and thus without additional latencies by transmitting the image data via a data network of the motor vehicle. The camera is preferably a front camera of the motor vehicle. Such a camera is often part of an existing environment sensor of a motor vehicle, for example, for a traffic sign recognition or as an image source for a lane assistant, so that the inventive solution can be realized inexpensively.

According to one aspect of the invention, the micro-movements determined comprise rotational and translational motion data. These Types of movement are the cause of the typical display errors in head-up display devices. Their capture is therefore usually sufficient to correct display errors.

According to one aspect of the invention, the display of the head-up display device is adapted according to the micro-movements determined. This ensures that annoying display errors are corrected for the viewer and that the overall impression appealing to the viewer is achieved.

According to one aspect of the invention, the head-up display device generates an augmented reality display, and the display of the head-up display device is adapted such that visible objects are optically superimposed on displayed objects. For example, if an object is highlighted in advance with a box and the car, e.g. due to a bump, tilts down, the presentation is adjusted by the box is shifted on the presentation accordingly upwards. In this way, an improved contact analog representation is achieved.

A method according to the invention or a device according to the invention in a vehicle, in particular a motor vehicle, is used particularly advantageously.

• 1 zeigt schematisch ein Verfahren zum Bereitstellen von Informationen zur Steuerung einer Anzeige einer Head-up-Display-Vorrichtung für ein Kraftfahrzeug;
• 2 zeigt eine erste Ausführungsform einer Vorrichtung zum Bereitstellen von Informationen zur Steuerung einer Anzeige einer Head-up-Display-Vorrichtung für ein Kraftfahrzeug;
• 3 zeigt eine zweite Ausführungsform einer Vorrichtung zum Bereitstellen von Informationen zur Steuerung einer Anzeige einer Head-up-Display-Vorrichtung für ein Kraftfahrzeug;
• 4 stellt schematisch ein Kraftfahrzeug dar, in dem eine erfindungsgemäße Lösung realisiert ist;
• 5 zeigt schematisch den allgemeinen Aufbau einer Head-up-Display-Vorrichtung für ein Kraftfahrzeug;
• 6 zeigt ein Beispiel einer Augmented-Reality-Darstellung einer Head-up-Display-Vorrichtung;
• 7 zeigt einige Beispiele für einen optischen Fluss; und
• 8 zeigt einen Systementwurf der erfindungsgemäßen Lösung zum Bereitstellen von Informationen zur Steuerung einer Anzeige einer Head-up-Display-Vorrichtung für ein Kraftfahrzeug.

Further features of the present invention will become apparent from the following description and the appended claims, taken in conjunction with the figures.

• 1 schematically shows a method for providing information for controlling a display of a head-up display device for a motor vehicle;
• 2 shows a first embodiment of a device for providing information for controlling a display of a head-up display device for a motor vehicle;
• 3 shows a second embodiment of a device for providing information for controlling a display of a head-up display device for a motor vehicle;
• 4 schematically represents a motor vehicle in which a solution according to the invention is realized;
• 5 schematically shows the general structure of a head-up display device for a motor vehicle;
• 6 shows an example of an augmented reality representation of a head-up display device;
• 7 shows some examples of an optical flow; and
• 8th shows a system design of the inventive solution for providing information for controlling a display of a head-up display device for a motor vehicle.

For a better understanding of the principles of the present invention, embodiments of the invention will be explained in more detail below with reference to the figures. It is understood that the invention is not limited to these embodiments, and that the features described can also be combined or modified without departing from the scope of the invention as defined in the appended claims.

1 schematically shows a method for providing information for controlling a display of a head-up display device for a motor vehicle. In a first step, an optical flow from image data of a camera of the motor vehicle is determined 10, for example a front camera of the motor vehicle. Micro-movements of the motor vehicle are subsequently determined from the optical flow. The determined micro-movements may include, for example, rotational and translatory movement data. Information on the micro-movements are then output to a control device of the head-up display device 12. The display of the head-up display device is finally adapted according to the micro-movements determined 13. For example, the head-up display device generates a augmented reality display. The display of the head-up display device is then adapted 13 so that visible objects are superimposed optically correct with displayed objects. Preferably, the determination takes place 10 of optical flow and detecting 11 the micro-movements in the camera or in a control unit of the camera.

2 shows a simplified schematic representation of a first embodiment of a device 20 for providing information for controlling a display of a head-up display device for a motor vehicle. The device 20 has an entrance 21 via which image data of a camera can be received, for example from a front camera of the motor vehicle. The device 20 also has an image processing unit 22 for determining an optical flow from the image data. An analysis unit 23 determined 11 relative micro-movements from the optical flow, such as rotational and translational movements. About an exit 26 the device 20 information about the micro-movements to a control unit 42 the head-up display device output. The control unit 42 can then display adjust the head-up display device according to the detected micro-movements. For example, the head-up display device generates an augmented reality display. The display of the head-up display device is then adapted such that visible objects are visually superimposed with displayed objects.

The image processing unit 22 and the analysis unit 23 can be from a control unit 24 to be controlled. Via a user interface 27 If necessary, settings of the image processing unit 22 , the analysis unit 23 or the control unit 24 be changed. The in the device 20 accumulating data can be stored in memory as needed 25 the device 20 are stored, for example, for later evaluation or for use by the components of the device 20 , The image processing unit 22 , the analysis unit 23 as well as the control unit 24 may be implemented as dedicated hardware, such as integrated circuits. Of course, they may also be partially or fully combined or implemented as software running on a suitable processor, such as a GPU. The entrance 21 and the exit 26 may be implemented as separate interfaces or as a combined bidirectional interface. Preferably, the device is 20 integrated into the camera or into a control unit for the camera.

3 shows a simplified schematic representation of a second embodiment of a device 30 for providing information for controlling a display of a head-up display device for a motor vehicle. The device 30 has a processor 32 and a memory 31 on. For example, the device is 30 to a computer or a controller. In the storage room 31 are filed instructions that the device 30 when executed by the processor 32 to perform the steps according to one of the described methods. The in the store 31 filed instructions thus embody one by the processor 32 executable program that realizes the inventive method. The device 30 has an entrance 33 for receiving information, such as camera data. From the processor 32 generated data will be via an output 34 provided. In addition, they can be stored in memory 31 be filed. The entrance 33 and the exit 34 can be combined to a bidirectional interface.

The processor 32 may include one or more processing units, such as microprocessors, digital signal processors, or combinations thereof.

The stores 25 . 31 The embodiments described may have both volatile and non-volatile memory areas and include a wide variety of memory devices and storage media, for example hard disks, optical storage media or semiconductor memories.

4 schematically represents a motor vehicle 40 in which a solution according to the invention is realized. The car 40 has a head-up display device 41 with an associated controller 42 on. Furthermore, the motor vehicle 40 a camera 43 , a control unit 44 for the camera 43 and a device 20 for providing information for controlling a display of the head-up display device 41 on. The device 20 For example, in the camera 43 or in the control unit 44 for the camera 43 be integrated. Other components of the motor vehicle 40 are a navigation system 45 , a data transmission unit 46 as well as a number of assistance systems 47 of which one is shown as an example. By means of the data transmission unit 46 can be established a connection to service providers. To store data is a memory 48 available. The data exchange between the various components of the motor vehicle 40 via a network 49 ,

Hereinafter, preferred embodiments of the invention based on the 5 to 8th to be discribed.

5 schematically shows a head-up display device 41 for a motor vehicle 40 , with their help content on a projection screen 52 of the motor vehicle 40 can be displayed, for example, on the windshield or on an additional disc of glass or plastic, which is arranged between the driver and the windscreen on the dashboard. The contents presented are by an imaging unit 50 generated and with the help of an optical module 51 on the projection screen 52 projected. Typically, the projection takes place in an area of the windshield above the steering wheel. The imaging unit 50 For example, it can be an LCD TFT display. The augmented reality head-up display device 41 is usually in a dashboard of the motor vehicle 40 installed.

6 shows an example of an augmented reality representation of a head-up display device. The actual environment is superimposed on a number of elements, some in contact-analog representation. In the example in 6 These are a current speed, a distance to a vehicle in front, navigation information, a pointer for a possible or upcoming overtaking maneuver and a "virtual wall" on the right side line as a safety note. The elements shown are mainly based on information collected by vehicle sensors. For the concrete design of the presentation a variety of design options are available to the skilled person.

7 shows some examples of an optical flow. The optical flow is understood to mean a vector field which assigns to each pixel of an image a velocity vector with which the pixel moves over the image over time. The optical flux can be determined from the shift of the grayscale or color dots between two or more temporally successive images. The resulting vector fields are characteristic of different types of movements. 7 shows by way of example three vector fields. Each arrow represents an optical flux vector. The length and direction of an arrow indicate the speed and direction of movement of an image feature at the origin of the arrow, respectively. For the sake of simplicity, it is assumed that all visible objects are the same distance from the camera. In 7a) The camera moves sideways to the left. In this case, a uniform vector field results, all flow vectors are horizontal and have the same length. In 7b) the camera moves downhill. Here, too, results in a uniform vector field, all flow vectors are vertical and have the same length. In 7c) shows an approach to a scene, ie a forward movement of the camera. The result is an "expanding" vector field with a central point from which the flow vectors appear to expand. The flow velocity increases with the distance to the central point, but the quotient of speed and distance is constant. 7d) finally shows a rotation of the camera about an axis perpendicular to the image plane.

In real world environments, where objects that move independently of one another are moving at different speeds or objects are at different distances from the camera, more complex flow fields result, with different image areas showing different patterns. The optical flow can be determined with different approaches from the image data. Examples of applicable algorithms are the methods of Horn-Schunck [1] or Lucas-Kanade [2].

8th shows a schematic system design of the solution according to the invention. In this example, the determination of the optical flow and the detection of micro-movements are made from the optical flow in the camera 43 , An image sensor 60 The camera captures image data and passes it to an image processing unit 22 further. The image processing unit 22 determines from the received image data an optical flow, for example in the form of a vector field. The determined optical flow is sent to an analysis unit 23 passed, with the aid of micro-movements of the motor vehicle, in particular relative translational and rotational movements are determined. The motion information is sent to the controller 42 passed the head-up display device 41. The control unit 42 uses this data directly for image compensation for the head-up display, which is connected to the head-up display device 41 is issued.

LIST OF REFERENCE NUMBERS

10

Determining an optical flow from image data of a camera

11

Determine micro-movements from the optical flow

12

Output information about the micromotion

13

Adjusting an indication of a head-up display device

20

contraption

21

entrance

22

Image processing unit

23

analysis unit

24

control unit

25

Storage

26

output

27

User interface

30

contraption

31

Storage

32

processor

33

entrance

34

output

40

motor vehicle

41

Head-up display device

42

Control unit of the head-up display device

43

camera

44

Control unit of the camera

45

navigation system

46

Data transfer unit

47

assistance system

48

Storage

49

network

50

Imaging unit

51

Optical module

52

projection

60

image sensor

references

1. [1] Horn et al .: "Determining Optical Flow", Artificial Intelligence, Vol. 17 (1981), pp. 185-203 ,
2. [2] Lucas et al .: "An iterative image registration technique with an application to stereo vision", Proceedings of Imaging Understanding Workshop (1981) 121-130 ,

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102014214516 A1 [0004]
• DE 102013226969 A1 [0007]

Cited non-patent literature

• Horn et al .: "Determining Optical Flow", Artificial Intelligence, Vol. 17 (1981), pp. 185-203 [0036]
• Lucas et al .: "An iterative image registration technique with an application to stereo vision", Proceedings of Imaging Understanding Workshop (1981) 121-130 [0036]

Claims (11)

A method for providing information for controlling a display of a head-up display device (41) for a motor vehicle (40), comprising the steps of: - determining (10) an optical flow from image data of a camera (43) of the motor vehicle (40); - determining (11) micro-movements from the optical flow; and - Outputting (12) information about the micro-movements to a control unit (42) of the head-up display device (41). Method according to Claim 1 in which the determination (10) of the optical flow and the determination (11) of the micromovements take place in the camera (43) or in a control device (44) of the camera (43). Method according to Claim 1 or 2 , wherein the determined micro-movements include rotational and translational motion data. Method according to one of the preceding claims, wherein the display of the head-up display device (41) is adapted (13) in accordance with the determined micro-movements. Method according to Claim 4 in that the head-up display device (41) generates an augmented reality display and the display of the head-up display device (41) is adapted (13) in such a way that optically correct superimposes visible objects with displayed objects become. Method according to one of the preceding claims, wherein the camera (43) is a front camera of the motor vehicle (40). A computer readable storage medium having instructions that, when executed by a computer, execute the computer to perform the steps of a method according to any one of Claims 1 to 6 for providing information for controlling a display of a head-up display device (41) for a motor vehicle (40). Device (20) for providing information for controlling a display of a head-up display device (41) for a motor vehicle (40), with An image processing unit (22) for determining (10) an optical flow from image data of a camera (43) of the motor vehicle (40); - an analysis unit (23) for determining (11) micro-movements from the optical flow; and - An output (26) for outputting (12) information about the micro-movements to a control unit (42) of the head-up display device (41). Device (20) according to Claim 8 , wherein the device (20) is part of the camera (43) or a control device (44) of the camera (43). System for controlling a display of a head-up display device (41) for a motor vehicle (40), having a control device (42) for the head-up display device (41), characterized in that the system comprises a device (20) according to Claim 8 or 9 or is arranged, a method according to one of Claims 1 to 6 for providing information for controlling a display of the head-up display device (41). Motor vehicle (40) with a head-up display device (41), characterized in that the motor vehicle (40) comprises a device (20) according to Claim 8 or 9 or is arranged, a method according to one of Claims 1 to 6 for providing information for controlling a display of the head-up display device (41).

Images