Classifications

• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
  • B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
  • B60R1/20—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
  • B60R1/22—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle
  • B60R1/23—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view
  • B60R1/27—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view providing all-round vision, e.g. using omnidirectional cameras
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
  • B60K35/20—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor
  • B60K35/28—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
  • B60K35/85—Arrangements for transferring vehicle- or driver-related data
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
  • B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
  • B60R1/20—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
  • B60R1/22—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle
  • B60R1/23—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view
  • B60R1/24—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view in front of the vehicle
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
  • B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
  • B60R1/20—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
  • B60R1/22—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle
  • B60R1/28—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with an adjustable field of view
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S11/00—Systems for determining distance or velocity not using reflection or reradiation
  • G01S11/12—Systems for determining distance or velocity not using reflection or reradiation using electromagnetic waves other than radio waves
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
  • G01S17/88—Lidar systems specially adapted for specific applications
  • G01S17/93—Lidar systems specially adapted for specific applications for anti-collision purposes
  • G01S17/931—Lidar systems specially adapted for specific applications for anti-collision purposes of land vehicles
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
  • G06T19/006—Mixed reality
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
  • B60K2360/16—Type of output information
  • B60K2360/176—Camera images
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
  • B60K2360/592—Data transfer involving external databases
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
  • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
  • B60R2300/20—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of display used
  • B60R2300/207—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of display used using multi-purpose displays, e.g. camera image and navigation or video on same display
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
  • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
  • B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
  • B60R2300/304—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing using merged images, e.g. merging camera image with stored images
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
  • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
  • B60R2300/60—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
  • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
  • B60R2300/80—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement
  • B60R2300/802—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement for monitoring and displaying vehicle exterior blind spot views
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
  • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
  • B60R2300/80—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement
  • B60R2300/806—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement for aiding parking
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
  • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
  • B60R2300/80—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement
  • B60R2300/8086—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement for vehicle path indication
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
  • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
  • B60R2300/80—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement
  • B60R2300/8093—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement for obstacle warning

Definitions

• the present invention relates to an expanded display of the environment of a vehicle.
• augmented reality location-specific additional information is superimposed into the reality perceived by the viewer.
• this is usually accomplished by having a camera generate a real-time video image of the vehicle environment that is shown to the driver via a corresponding output device.
• This screen will then display additional information to assist the driver. This can be, for example, the visualization of the probable route with the current steering angle or the highlighting of an obstacle.
• Close range of the vehicle can often not be detected with such a system.
• an accurate environmental model of the vehicle at close range is desirable in order to be able to communicate information to the driver here as well.
• the proximity of a vehicle is detected and generated from the data obtained an environment map in which the vehicle is virtually moving.
• the visualization is the display of the generated space, for example from a bird's-eye view. Furthermore, important information such as protruding collisions or the display of the driving tube can be visualized therein. However, there is no visualization of the far end of the environment of the vehicle.
• the document DE 10 2009 005 566 A1 describes a method for
• Short-range sensors such as ultrasonic sensors.
• objects are detected in the vicinity of the vehicle with the help of the video camera, which can be measured with the post-range sensors.
• Object be entered in a three-dimensional environment map of the vehicle. This card can then be used for various applications, such as helping the driver to park. However, in this document is not closer to the visualization
• DE 10 2009 005 505 A1 describes a method for generating an environment image of the vehicle from a virtual camera position, the image of a camera perspective being converted to the virtual camera position. This is done with multiple cameras of the near range of
• the 3D model is also obtained with the mentioned cameras by using the stereo-from-motion principle. This allows the driver a video image of the entire vicinity of his vehicle, for example from the
• the cited prior art has the disadvantage that only the near zone of the vehicle is visualized, while the far-end zone is ignored. Furthermore, as in the first two cases, the driver no
• the third model discloses the use of multiple cameras, increasing the manufacturing cost and cost of the vehicle.
• none of the cited references discusses the possibility of a perspective continuous transition between artificially rendered Use ambient models and image data of an optical environment sensor advantageous.
• the inventive method for visualizing the vehicle environment addresses the disadvantages of the prior art by the combination of the features of the independent claim 1.
• a bipartite display of synthetically generated environment model is a bipartite display of synthetically generated environment model and
• Real-time video image generated To generate the environment model, first a current distance between the vehicle and obstacles in the environment is detected. At the same time, a current position of the vehicle is determined, which is temporarily stored together with the distance information. From several stored position and distance data can now be a
• a two-dimensional or three-dimensional model can be calculated.
• the advantage of this environment model is that it can be viewed from different virtual locations. If a specific virtual observer position is determined (for example by the driver), then a virtual view of the environment model is calculated from this observer position. In this way, it is possible, for example, to create a top-view view of the environment.
• part of the environment is defined by means of at least one, preferably by means of exactly one,
• Video camera captured The resulting video presentation is conveniently integrated into the virtual view so that the driver can see a full picture of the surroundings of his vehicle: in the video camera's detection area, he sees a real-time video image of the environment while synthesizing it outside the scope of the video camera Environment model sees. This puts the driver in addition to the
• Video presentation of the environment also provides information about obstacles that are currently not within the scope of the video camera.
• the virtual view comprises a two-part representation, according to which, at the level of the transition between the engine compartment passenger compartment, an artificial horizon is laid out, which allows a division into different views.
• the artificial horizon can be the image of a
• the distance between the vehicle and obstacles is determined by means of optical sensors.
• These sensors can be either passive sensors (mono and stereo video) or active sensor systems (laser scanner, laser imager, PMD (Photonic Mixing Device, optical sensor according to the time-of-flight method), flush lidar).
• the distance data relating to obstacles in the surroundings of the vehicle is obtained from the signals of the video camera.
• the video camera has to fulfill two tasks. On the one hand, it serves as a distance sensor (for example, according to the structure-from-motion method or in that the video camera is a stereo camera) and, on the other hand, as a supplier of the video signal to be displayed. Therefore, in addition to the video camera no additional sensors are needed to determine the environment model.
• the position of the vehicle is measured as an absolute position or as a relative position to an already stored position.
• the odometry offers itself what the
• a differential GPS can be used here.
• the virtual vehicle Positioning procedure to pay attention.
• a differential GPS can be used here.
• the virtual vehicle can be used here.
• the virtual vehicle can be used here.
• Observer position of the environment model set by the driver of the vehicle This can thus customize the visualization of the obstacles by varying the views and perspectives in the environment of the vehicle.
• Arithmetic unit are automatically set and displayed in the display device. Finally, the virtual viewer view of the selected virtual viewer position is used to calculate the virtual view of the environment model and adjust the video presentation to fit seamlessly into the virtual view.
• Coordinate system such additional information can also be seamlessly placed over both areas. In this way, for example, even a detected post can be "monitored” during a parking operation, even if it has left the visible area of the camera.
• the invention relates to a device for visualizing the
• Vehicle environment with the features of claim 7. This comprises at least one sensor for determining a distance of the vehicle to obstacles and at least one sensor for determining a current position of the vehicle. Furthermore, there is a storage device in which position data and distance data are stored. In addition, a video camera is used which detects at least part of the environment and forwards this data to a computing device. Based on the stored position data of the vehicle and distance data of the obstacles as well as the signals of the video camera, the computing device generates a visualization of the environment, which is displayed on an output device to the driver. For this purpose, the computing device calculates an environment model from the stored position and distance data. Wrd a virtual Viewer position, the computing device further determines a virtual view of that environment model from the selected virtual one
• the computing device integrates the video camera signals into the virtual view. Therefore, with the device according to the invention, the inventive method can be ideally carried out.
• the at least one sensor for determining the distance between the vehicle and the obstacles is either a passive optical sensor (mono and stereo video) or an active optical sensor system (laser scanner, laserimager, PMD, flush lidar). is working.
• the senor for determining a distance of the vehicle from obstacles to the video camera which must thus fulfill two tasks.
• it provides distance information about obstacles in the environment of the
• the video signal to be displayed for example, according to the structure-from-motion method or the stereo video method.
• a vehicle which comprises a device according to the invention or is set up in such a way to carry out a method according to the invention. Furthermore, it is advantageous if the recorded by the camera
• Video presentation for the display representation is edited. In this way, it can be adapted to the perspective from which the display representation shows the virtual environment, so that no perspective disturbances arise.
• Figure 1 is a representation of the visualization of the environment of a
• FIG. 2 shows a representation of a first visualization variant of FIG
• FIG. 3 shows a representation of a second visualization variant of FIG
• FIG. 4 shows the visualization of the vehicle environment when passing through a narrow lane at a first time according to a second embodiment of the invention
• Figure 5 shows the visualization of the vehicle environment in passing a narrow lane at a second time according to the second embodiment of the invention.
• FIG. 1 shows a conventional, perspective video representation 20 of an environment captured by a camera 2 with a restricted viewing angle.
• Figures 2 and 3 illustrate a first embodiment of the invention. It is considered a vehicle 1, which is equipped with a camera 2.
• the camera fulfills two functions: on the one hand, only a video display 20 of the surroundings can be displayed, on the other hand, the distance to obstacles 6 in the surroundings of the vehicle can be determined by means of the stereo-from-motion method. Therefore, the camera also serves as a distance sensor, so that a three-dimensional environment model of
• Vehicle can be calculated.
• the environment model in turn makes it possible to define a virtual observer position 3 (see FIGS. 2 and 3) from which the environment is to be viewed. Therefore, it is possible, for example, to generate a display representation 30 that provides a continuous transition from a top-view perspective (view from above) into the display
• a first part 40 of the display representation is based on a reconstruction from the environment model. This corresponds to the area 4 of the environment, which is not visible from the camera 2.
• a second part 50 of the display is based on the video camera signals. This corresponds to Area 5 of the environment, which is visible to the camera.
• the video representation 20 is correspondingly transformed to the environment so that it can be seamlessly attached to the view calculated from the environment model without perspective distortion.
• Figures 4 and 5 show a vehicle 1 driving through a narrow lane.
• This lane is formed by three vehicles 60, 62 and 64 on the left side and by three vehicles 61, 63 and 65 on the right side and by some more containers 66.
• Embodiment visualized The overall process is explained by way of example on the basis of three images, a first image 21, a second image 22 and a third image 23, which were taken by the camera 2 at different times.
• the display representation 30 shown to the driver is also divided into two. In the upper area 31, a perspective is shown, as well as the driver sees. Here, the video presentation of the camera 2 is integrated, which does not need to be further transformed.
• the lower area 32 shows the
• the upper area 31 corresponds to the visualization of the long range of the vehicle 1
• the lower area 32 corresponds to the visualization of the vicinity of the vehicle.
• the vehicle is in the position in which the camera 2 takes the second image 22. At this point, the rear left vehicle 60 and the rear right vehicle 61 of the lane are no longer visible.
• the camera 2 is also used to determine a
• the video presentation of the camera 2 is integrated into the display representation 30 in the upper area 31 (driver's perspective). This will be seamlessly inserted into the rest of the ad and can provide additional environmental information from the
• the video image shown in the upper area 31 of the display 30 can be extended by the environment model
• FIG. 3 shows a second time step of the passage through which the camera 2 picks up the third image 23. No longer relevant objects, such as the above-mentioned container 66, are no longer displayed. In addition, it becomes clear on the basis of the middle right vehicle 63 how objects from the upper region 31 of the display representation 30 can change into the lower region 32. Such a change corresponds to the transition from the far end

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Physics & Mathematics (AREA)
• Mechanical Engineering (AREA)
• General Physics & Mathematics (AREA)
• Electromagnetism (AREA)
• Radar, Positioning & Navigation (AREA)
• Remote Sensing (AREA)
• Computer Hardware Design (AREA)
• General Engineering & Computer Science (AREA)
• Software Systems (AREA)
• Theoretical Computer Science (AREA)
• Computer Graphics (AREA)
• Computer Networks & Wireless Communication (AREA)
• Chemical & Material Sciences (AREA)
• Combustion & Propulsion (AREA)
• Transportation (AREA)
• Traffic Control Systems (AREA)
• Closed-Circuit Television Systems (AREA)

Priority Applications (5)

Applications Claiming Priority (2)

Publications (1)

Family

ID=47429749

Family Applications (1)

Country Status (7)

Families Citing this family (24)

Citations (6)

Family Cites Families (15)

• 2012
  • 2012-01-19 DE DE102012200731A patent/DE102012200731A1/de not_active Withdrawn
  • 2012-11-28 US US14/371,827 patent/US9862316B2/en active Active
  • 2012-11-28 CN CN201280067596.6A patent/CN104204847B/zh active Active
  • 2012-11-28 KR KR1020147019760A patent/KR102034189B1/ko not_active Expired - Fee Related
  • 2012-11-28 EP EP12805962.3A patent/EP2805183B1/de active Active
  • 2012-11-28 WO PCT/EP2012/073835 patent/WO2013107548A1/de not_active Ceased
  • 2012-11-28 IN IN5777DEN2014 patent/IN2014DN05777A/en unknown

Patent Citations (7)

Also Published As

Similar Documents

Legal Events