DE102013206435A1 - Visualization system and device for generating and displaying a virtual image of a vehicle environment - Google Patents

Abstract

The present invention relates to a visualization system for a vehicle environment. In the case of corresponding visualization systems, it proves problematic to combine the video information from several cameras in such a way that a consistent image results. Furthermore, the display of camera images in the field of view of an observer (eg by insertion in a pair of glasses) can lead to inconsistencies that are due to the differing optical center. The present invention proposes a visualization system comprising: at least one image capture device (30) for acquiring video image information; - At least one position determining means (40) for determining at least one position information of at least one object (1, 2); - processing means (50) for processing the video image information and position information and for determining optical information; A modeling device (60) for generating a three-dimensional model of at least part of the vehicle environment; - a visualization device (70) for generating at least one two-dimensional image using the model in dependence on an optical center; and - a display device (80) for displaying the at least one two-dimensional image.

Description

The invention relates to a visualization system and a method for visualizing a vehicle environment of a vehicle.

• 1. Perspektive der Anzeige relativ zur Kameraperspektive;
• 2. Anzahl der Bilder, die in einer Anzeige einer Anzeigeeinrichtung dargestellt werden.

Corresponding visualization systems often use camera systems in order to display information from the environment to the driver of a vehicle. These systems can be described by the following properties:

• 1. Perspective of the display relative to the camera perspective;
• 2. Number of images displayed in a display of a display device.

The perspective of the ad is differentiated between systems that display in the camera perspective and those in which the display is in a virtual perspective that is different from the camera's perspective.

The number of pictures is different between singleview systems and multiview systems. In singleview systems, the image of a camera is displayed in exactly one display. Known single-view systems with display in camera perspective are reversing cameras as well as cameras, which respectively replace the outside and rear mirrors, whereby there is a one-to-one correspondence between the individual camera and the associated display.

The multiview system combines images or views of multiple cameras in one display. Here, the picture can be greatly abstracted. For example, an indication of the vehicle environment can be made from a bird's eye view. A well-known multiview system that visualizes from a virtual perspective is the BMW Topview system. With the multiview system, it proves to be complicated to combine the multitude of views into a single representation. When combining several images, ghost objects can be created at the interfaces, which can be attributed to the different recording perspectives and / or different orientations of the image capture devices and / or different settings (eg zoom).

From the field of panoramic photography, there are algorithms that allow to combine images that were created by rotating the camera around the vertical axis between the images. By known rotation angles or by using image features on the cut edges of the individual images and corresponding image processing operations (eg cutting, scaling, rotation, distortion, etc.), these can usually be joined together seamlessly for the eye. However, this approach assumes that the optical center of the image capture device is on one axis for each shot. For many applications in the automotive sector, this approach is therefore unsuitable.

From the DE 10 2012 010 156 A1 For example, a visualization system is known in which cameras on the sides of a moving vehicle are used to capture a plurality of images of the environment. These images are used to create a virtual view of an environment, for example, as a replacement for a rearview mirror. It is obvious that this approach is not suitable for some application scenarios (the car stands still).

Starting from the described prior art, it is an object of the present invention to provide an improved visualization system. In particular, the visualization system should present the vehicle environment essentially error-free, be very flexible with regard to the perspectives to be displayed and ensure a high degree of user comfort.

This object is achieved by a visualization system according to claim 1 and a method according to claim 10.

• – mindestens eine Bilderfassungseinrichtung zum Erfassen von Videobildinformationen;
• – mindestens eine Positionsbestimmungseinrichtung zum Bestimmen mindestens einer Positionsinformation mindestens eines Objekts;
• – eine Verarbeitungseinrichtung zur Verarbeitung der Videobildinformationen und Positionsinformationen des wenigstens einen Objektes und zur Bestimmung von optischen Informationen, insbesondere Farb- und/oder Kontur- und/oder Oberflächeninformationen, des mindestens einen Objekts;
• – eine Modellierungseinrichtung zum Erzeugen eines dreidimensionalen Modells mindestens eines Teils der Fahrzeugumgebung unter Verwendung der optischen Informationen und der Positionsinformationen;
• – eine Visualisierungseinrichtung zum Erzeugen mindestens eines zweidimensionalen Abbilds des wenigstens einen Objektes unter Verwendung des Modells in Abhängigkeit von einem vorzugsweise frei wählbaren optischen Zentrum; und
• – eine Anzeigeeinrichtung zum Darstellen des mindestens einen zweidimensionalen Abbilds.

In particular, the object is achieved by a vehicle environment visualization system, the system comprising:

• At least one image capture device for acquiring video image information;
• - At least one position determining means for determining at least one position information of at least one object;
• A processing device for processing the video image information and position information of the at least one object and for determining optical information, in particular color and / or contour and / or surface information, of the at least one object;
• Modeling means for generating a three-dimensional model of at least a part of the vehicle environment using the optical information and the position information;
• A visualization device for generating at least one two-dimensional image of the at least one object using the model in dependence on a preferably freely selectable optical center; and
• - A display device for displaying the at least one two-dimensional image.

An essential point of the invention is that for generating the visualization a three-dimensional model of the vehicle environment is calculated. Such a model of the vehicle environment with a freely selectable optical center allows visualization from the perspective of different positions or perspectives. By creating the model, bad results in representing the vehicle environment can be avoided. In particular, it is possible to present information relating to the vehicle environment and taken from a first perspective without error from the perspective of a given optical center. Further, the system allows a variety of differently captured video and position information to be combined into a single model of the vehicle environment so that this model can be determined very accurately. All information obtained can then be considered and displayed in the production of the two-dimensional image. The user of the system, therefore, it is possible to capture the vehicle environment much more accurate.

It is conceivable to display the vehicle environment in two dimensions using the visualization system. In one embodiment of the visualization system, the vehicle environment for the user is displayed in three dimensions. For example, the display device may include an autostereoscopic display device and / or stereoscopic display device with eye separation, such as a shutter and / or polarization glasses and / or a head-mounted display. A three-dimensional visualization has the advantage that the already existing three-dimensional information of the model can be passed on to the user of the system.

In one embodiment, the visualization system may include a stereo camera system having at least two cameras for providing position information and / or video image information. Stereo camera systems are capable of taking over the function of the image capture device and the position determination device. It is conceivable to use several stereo camera systems and to combine the information obtained in the said three-dimensional model. This results in a very detailed model that can be provided to the user. In addition, the computational effort to create the model can be reduced because it is not necessary to correlate positions and orientations of individual sensors.

In a further embodiment of the visualization system, this can be designed as an augmented reality system. For example, the display device may include a transparent display area for displaying the at least one two-dimensional image. In this respect, it is possible for the user of the system that captures the real world to display additional information that he can not grasp from his perspective. In one embodiment, the transparent display surface may be a vehicle window. In another embodiment, the transparent display area is integrated in a head-up display of a user.

In a further embodiment of the visualization system, the visualization system uses a person detection device for detecting a viewing direction and / or a position of a person or a user. This person may be, for example, a driver of a vehicle. The visualization device can determine the optical center based on information about the viewing direction and / or position. In another embodiment, the position of the person is statically predetermined or configurable. With a dynamic detection of the direction of vision and / or the position of the person, images of the vehicle environment can be displayed that need no interpretation, since they coincide with the usual perception of the person. A misinterpretation can be avoided, so that the visualization system allows a much better representation of the vehicle environment.

The position determination device and the image acquisition device need not necessarily detect optical information. With regard to the position determination device, it is conceivable that a lidar device and / or a radar device be used. Furthermore, TOF cameras (time-of-flight cameras) can be used.

A TOF camera is a 3D camera system that uses distance to measure distances. For this purpose, the vehicle environment is illuminated by means of a light pulse and the camera measures for each pixel the time it takes for the light to reach the object and back again. The time required is directly proportional to the distance.

In another embodiment of the visualization system, the visualization device and / or modeling device uses information provided by other vehicle components. These other vehicle components may be common components used in vehicles. It is conceivable, for example, the use of distance sensors as a position-determining device.

The above object is further achieved by a vehicle having a visualization system configured as already described.

The vehicle may comprise a plurality of image acquisition devices and / or position determination devices arranged on the vehicle, wherein the modeling device is designed to take into account the position and / or orientation of the image acquisition devices and / or the position determination devices when generating the three-dimensional model.

As already explained, the at least one image capture device and / or at least one position determination device can be designed to provide information for a driver assistance system. For example, the corresponding sensors can be used in conjunction with a parking aid, a distance warning and / or a lane change assistant. In this respect, it is not necessary to install redundant systems or sensors in the vehicle.

In another embodiment, the visualization system is configured to create a plurality of models of at least a portion of the vehicle environment in consideration of the time. When creating a new model, information from older models can then be considered. Thus, at a certain time non-existent information, eg. For example, because a particular object is obscured by a larger and closer object, it can be filled in using previous models. Furthermore, it is possible to implement a motion detection device that detects changes in the model, such as a moving cyclist. Corresponding changes can be displayed in a special manner (eg flashing, outlined in red), so that they can be easily detected by the user of the system. Alternatively, the individual components of the visualization system, for. B. the image capture device and / or the position determination device, provide corresponding movement information. Thus, it is conceivable that ultrasonic sensors not only provide position information but also motion information.

In one embodiment of the visualization system, the image capture devices are at least partially monocameras that capture the video image information of the vehicle environment. For example, these monocameras can replace conventional side mirrors of the vehicle. These monocameras can be combined with dedicated position-determining devices, for example lidar device and / or radar device and / or TOF cameras. In this respect, it is conceivable simply to convert existing systems in such a way that the visualization system according to the invention can be implemented.

• a) Erfassen von Videobildinformationen;
• b) Bestimmen mindestens einer Positionsinformation mindestens eines Objekts;
• c) Verarbeiten der Positionsinformationen des mindestens einen Objekts mit den Videobildinformationen und Bestimmen von optischen Informationen, insbesondere Farb- und/oder Kontur- und/oder Oberflächeninformationen, des mindestens einen Objekts;
• d) Erzeugen eines dreidimensionalen Modells mindestens eines Teils der Fahrzeugumgebung unter Verwendung der optischen Informationen und der Positionsinformationen;
• e) Erzeugen mindestens eines zweidimensionalen Abbilds des wenigstens einen Objektes unter Verwendung des Modells in Abhängigkeit von einem optischen Zentrum;
• f) Anzeigen des mindestens einen zweidimensionalen Abbilds auf einer Anzeigeeinrichtung.

The object mentioned at the outset is furthermore achieved by a method comprising the following steps:

• a) capturing video image information;
• b) determining at least one position information of at least one object;
• c) processing the position information of the at least one object with the video image information and determining optical information, in particular color and / or contour and / or surface information, of the at least one object;
• d) generating a three-dimensional model of at least a portion of the vehicle environment using the optical information and the position information;
• e) generating at least one two-dimensional image of the at least one object using the model in dependence on an optical center;
• f) displaying the at least one two-dimensional image on a display device.

This is preferably a method for visualizing a vehicle environment by means of a vehicle, as has been described above. Preferably, a visualization system can be used, as has already been described.

The method has similar advantages as have already been explained in connection with the devices.

The method may include determining and / or determining viewer information and using the viewer information to generate the at least one two-dimensional image. The observer information can be information relating to a viewing direction and / or a field of view and / or a position of a user. Preferably, the optical center is determined based on this information. This allows the vehicle environment to be represented in an easy-to-understand manner.

The step e) may include generating at least a first and a second image, wherein the first and the second two-dimensional image differ on the basis of the underlying viewer information and / or settings made by a user. Thus, the method preferably displays two-dimensional images taking into account different optical centers. The optical centers can be manually input or be automatically made by the provision of appropriate detection devices. For example, it is conceivable to output a bird's eye view and a side view of certain objects or the entire vehicle environment. These different two-dimensional illustrations may facilitate the understanding of the vehicle environment for the user.

Furthermore, the aforementioned object can be achieved by a computer-readable storage medium comprising executable machine instructions which cause a computer to implement the described method.

Further advantageous embodiments will be apparent from the dependent claims.

The invention will be described by means of several embodiments, which are explained in more detail with reference to figures. Hereby show:

1 a vehicle with a rear stereo camera based visualization system;

2 individual components of the visualization system 1 ;

3 a vehicle having a visualization system based on ultrasonic sensors;

4 a vehicle having a visualization system based on a front stereo camera; and

5 a flowchart of a method for visualization of a vehicle environment.

In the following description, the same reference numerals are used for the same and like parts.

1 shows a schematic plan view of a vehicle 10 with a vehicle front 12 , a passenger compartment 13 and a vehicle rear 16 , On the sides of the vehicle 10 There are each a left side camera 31 and a right side camera 33 , With the two side cameras 31 . 33 of the embodiment according to 1 These are mono cameras that can optionally be used as virtual side mirrors. At the rear of the vehicle 16 is another camera, namely a rear camera 35 attached, which detects the rear of the vehicle.

In the described embodiment, it is the rear camera 35 around a stereo camera. The side cameras 31 . 33 and the rear camera 35 cover a rearward (relative to the vehicle) directed field of view of 180 degrees. This field of view is composed of a left field of view 32 the left side camera 31 and a right field of view 34 the right side camera 33 and a rearview camera field of view 36 the rear camera 35 , The rear camera field of view 36 overlaps both with the left field of view 32 as well as with the right field of view 34 , In a section of the right field of view 34 and the rearview camera field of view 36 There are two objects, namely a first pillar 1 and a second pillar 2 ,

In conventional visualization systems, they try to have a continuous panorama image based on the side cameras 31 . 33 and the rear camera 35 There are visualization of the first and second pillar 1 . 2 Problems, since the distance of the columns 1 . 2 to the vehicle as well as the different orientation and positioning of the right side camera 33 and the rear camera 35 lead to an optical effect, depending on the camera, the first column 1 to the left of the second pillar 2 (Rear camera 35 ) or the second column 2 on the left of the first pillar 1 (right side camera 33 ) appears. In this respect, it proves very difficult to create a continuous panoramic picture.

According to the invention, positional information is provided by the rear-view camera designed as a stereo camera 35 are used to produce a continuous panoramic image and by means of a display device 80 the driver 15 display.

Individual components of the visualization system are in the 2 shown. This is an image capture device 30 , a position determination device 40 , a processing device 50 , a modeling device 60 , a visualization device 70 and a display device 80 ,

In the basis of 1 described embodiment, the left side camera 31 , the right side camera 33 and the rear camera 35 as an image capture device 30 be understood. They all provide video image information by the processing device 50 can be processed to determine particular optical information of individual objects in the vehicle environment. This optical information can be, for example, the color, the exact contour and / or the surface texture of the objects.

The stereo camera has a dual function in the described embodiment. It also serves as a position-determining device 40 and provides positional information of the individual in the vehicle environment objects. Specifically, it provides position information of the objects that are in the rear camera field of view 36 are located.

The processing device 50 uses the video image information and the position information to determine the positions and optical nature of the detected objects. Based on the information obtained, the modeling device generates 60 a three-dimensional model of the vehicle environment, where each object is modeled with the associated optical information. After the generation of this three-dimensional model is through the visualization device 70 generates a two-dimensional view of the model (called a "view" or "2D view"). This two-dimensional view is taken as an image by means of a display device 80 for the driver 15 displayed.

In the in 1 the embodiment shown is the display device 80 in the rear of the passenger compartment 13 , According to the invention, an optical center is taken into account when generating the two-dimensional view. In the in 1 embodiment shown, the position of the head of the driver 15 are detected and the optical center can be selected so that it is at this position. The driver looks 15 So over his shoulder, he sees in the display device 80 a two-dimensional image that essentially coincides with reality. In that sense it is for the driver 15 very easy to understand the picture shown. By generating the three-dimensional model, it is also possible to avoid the formation of errors in the image. A continuous panoramic image containing information from multiple cameras can be created. With the visualization system according to the invention, not only transitions between several images of the environment, but also between the reality and the virtual representations can be represented without errors.

The named components of the processing device 50 Modeling device 60 and visualization device 70 can be implemented as separate physical units. Preferably, the individual components in a single physical unit, comprising at least one CPU and a memory device, such as an on-board computer 17 , implemented.

In the embodiment of the 1 are available for the detected vehicle environment only partially position information. So only the rear camera 35 used stereo camera to determine positions of the detected objects. The modeling device 60 is therefore adapted to model some areas of the three-dimensional model without positional information. In these areas, for example, the model lacks the depth information, ie the knowledge about the distance of the individual objects to the respective camera.

In a further exemplary embodiment, in order to produce a corresponding panoramic image on the three-dimensional modeling of the exclusively by the left and right side camera 31 . 33 be omitted. The model captures only the rear camera field of view 36 and provides a two-dimensional image that blends seamlessly with the images of the side cameras 31 . 33 followed.

In another embodiment, all cameras are 31 . 33 and 35 to stereo cameras, so that a three-dimensional model can be created for the entire field of view.

In the embodiment according to 3 are the left side camera 31 , the right side camera 33 and the rear camera 35 Monocameras and are only used to video information for the processing device 50 provide. As in the embodiment according to 1 can the left side camera 31 and the right side camera 33 be used as a rear view mirror replacement. According to the invention, the vehicle 10 of the 3 at the rear of the vehicle 16 additional ultrasonic sensors 37 on that as position-determining devices 40 serve and provide corresponding position information of the objects. As in the embodiment according to 1 For example, the video information and the position information may be used to create an at least partially three-dimensional model of the vehicle environment and the driver 15 display. In the embodiment according to 3 is the display device 80 in the dashboard of the vehicle 10 ,

It is possible to use the embodiment of 3 for more ultrasonic sensors 37 on the sides of the vehicle 10 to extended. Likewise, the signals of the ultrasonic sensors 37 and the rear camera 35 combined with those from side-mounted stereo cameras.

In the embodiment of 4 The side mirrors are designed as conventional side mirrors. Also, the vehicle points 10 no rear camera 35 on. It's just a front camera 39 provided, which allows video image information and position information of the vehicle environment in front of the vehicle 10 to deliver. The processing device 50 and the modeling means 60 use this information to generate a three-dimensional model of this area. The visualization device 70 is used, for example, to extract contours of the generated model and to obtain a contour image from this. This contour image is in the windscreen of the vehicle 10 faded in, so this as a display device 80 serves. The driver sees through the windscreen, the real world, with the contours of individual objects are highlighted. For a representation covering the real world, it is necessary that the optical center used to generate the contour representation coincides with the driver's optical center 15 coincides. In one embodiment, this can be ensured by first taking a three-dimensional model from the perspective of the front camera 39 is generated. The visualization device 70 knows the relative position of the driver's head 15 to the rear camera 35 and can determine the position of the head within the generated virtual world. The head position then serves as the optical center to create the contour image. The path over the three-dimensional model is particularly advantageous when the sensors used from the desired optical center far, z. B. more than 50 cm, are removed.

5 describes a method according to the invention for visualizing a vehicle environment. Here are at the same time in one step 20 and 21 Captured video image information or position information. The information gained will be in one step 22 processed. In this step, optical information, for example color and / or contour information, is extracted. In a subsequent step 23 From the obtained information a three-dimensional model is generated, which then in one step 24 for generating a two-dimensional image. This two-dimensional image is in one step 25 displayed.

In a further embodiment of the visualization method, after the acquisition of the video image information and the position information, a three-dimensional model is created directly on the basis of the position information. In a later step, this model can be "enriched" (eg colored) based on the video image information.

LIST OF REFERENCE NUMBERS

1

Column 1

2

Column 2

10

vehicle

12

vehicle front

13

Vehicle host cells

15

driver

16

vehicle rear

17

board computer

20-25

steps

30

Image capture device

31

Left side camera

32

Left field of view

33

Right side camera

34

Right field of view

35

rear view camera

36

Rear camera field of view

37

Ultrasonic sensors

39

front camera

40

Location facility

50

processing device

60

modeling means

70

visualiser

80

display

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102012010156 A1 [0007]

Claims (14)

A vehicle environment visualization system comprising: - at least one image capture device ( 30 ) for capturing video image information; At least one position determining device ( 40 ) for determining at least one position information of at least one object ( 1 . 2 ); A processing device ( 50 ) for processing the video image information and position information of the at least one object ( 1 . 2 ) and for determining optical information, in particular color and / or contour and / or surface information, of the at least one object ( 1 . 2 ); A modeling device ( 60 ) for generating a three-dimensional model of at least a part of the vehicle environment using the optical information and the position information; A visualization device ( 70 ) for generating at least one two-dimensional image of the at least one object using the model in dependence on an optical center; and - a display device ( 80 ) for displaying the at least one two-dimensional image. Visualization system according to claim 1, characterized in that the display device ( 80 ) comprises an autostereoscopic display device and / or stereoscopic display device with eye separation, such as a shutter and / or polarization glasses and / or a head-mounted display. Visualization system according to one of the preceding claims, characterized by at least one stereo camera system having at least two cameras for providing position information and / or video image information. Visualization system according to one of the preceding claims, characterized in that the display device ( 80 ) comprises a transparent display area for displaying the at least one two-dimensional image. Visualization system according to one of the preceding claims, characterized by a person detection device for detecting a viewing direction and / or a position of a person, in particular a driver ( 15 ), the visualization device ( 80 ) determines the optical center based on information about the viewing direction and / or position. Visualization system according to one of the preceding claims, characterized in that the position-determining device ( 40 ) comprises at least one lidar device and / or radar device and / or TOF camera. Vehicle ( 10 ), characterized by a visualization system according to one of the preceding claims. Vehicle ( 10 ) according to claim 7, characterized by a plurality of image capture devices ( 30 ) and / or position determining devices ( 40 ), wherein the modeling device ( 50 ) is adapted to generate the three-dimensional model, the position and / or orientation of the image capture devices ( 30 ) and / or the position-determining devices ( 40 ). Vehicle ( 10 ) according to one of claims 7 or 8, characterized in that at least one image capture device ( 30 ) and / or at least one position determining device ( 40 ) is adapted to provide information for a driver assistance system, for example for a parking assistance system, a distance warning and / or a lane change assistant. Method for visualizing a vehicle environment of a vehicle ( 10 ), in particular by means of a visualization system according to one of claims 1 to 6, wherein the method comprises the following method steps: a) acquisition of video image information; b determining at least one position information of at least one object ( 1 . 2 ); c) processing the position information of the at least one object ( 1 . 2 ) with the video image information and determining optical information, in particular color and / or contour and / or surface information, of the at least one object ( 1 . 2 ); d) generating a three-dimensional model of at least a portion of the vehicle environment using the optical information and the position information; e) generating at least one two-dimensional image of the at least one object ( 1 . 2 ) using the model as a function of an optical center; f) displaying the at least one two-dimensional image on a display device ( 80 ). A method according to claim 10, characterized by setting and / or determining observer information, in particular information relating to a viewing direction and / or a field of view and / or a position of a user; and using the viewer information to generate the at least one two-dimensional image. Method according to one of claims 10 or 11, in particular according to claim 11, characterized in that the step e) comprises: generating at least a first two-dimensional image and at least one second two-dimensional image, wherein the first and the second two-dimensional image due to Differs from basic viewer information and / or user-made settings. Method according to one of claims 10 to 12, characterized in that the detection of the video image information and / or the determination of the position information is carried out at least by a first and a second means; and step d) comprises: correlating the position information of the at least one object ( 1 . 2 ) with the video image information taking into account at least the relative positions of the first and second devices relative to each other and / or taking into account the orientation of the first and second device. A computer readable storage medium having executable machine instructions that cause a computer to implement the method of any one of claims 10 to 14 when executed.

Images