Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N7/00—Television systems
  • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
  • H04N7/183—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
  • B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
  • B60R1/20—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
  • B60R1/22—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle
  • B60R1/23—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view
  • B60R1/27—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view providing all-round vision, e.g. using omnidirectional cameras
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T3/00—Geometric image transformations in the plane of the image
  • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
  • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
  • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
  • B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
  • B60R2300/301—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing combining image information with other obstacle sensor information, e.g. using RADAR/LIDAR/SONAR sensors for estimating risk of collision
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
  • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
  • B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
  • B60R2300/303—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing using joined images, e.g. multiple camera images
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
  • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
  • B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
  • B60R2300/304—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing using merged images, e.g. merging camera image with stored images
  • B60R2300/305—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing using merged images, e.g. merging camera image with stored images merging camera image with lines or icons
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
  • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
  • B60R2300/60—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective
  • B60R2300/607—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective from a bird's eye viewpoint
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
  • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
  • B60R2300/80—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement
  • B60R2300/806—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement for aiding parking

Definitions

• the present invention relates to a method for generating an image of a vehicle environment of a vehicle with a camera by recording a first partial image in a first position or orientation of the camera and recording at least one second partial image in at least one second position or
• the present invention relates to a
• An imaging device for generating an image of a vehicle environment of a vehicle with a corresponding camera.
• Vehicles often have driver assistance systems to assist the driver in operating the vehicle.
• Such assistance systems include z. B. a
• Parking assistance system an ACC system, a lane keeping system, a lane change system, a high beam assistant and the like.
• some cameras built into the vehicle provide the driver with information about the surroundings of the vehicle.
• either the current image of the camera or current images of cameras are displayed individually or in combination, or the individual images are processed so that a virtual image emerges from a bird's eye view.
• Systems are also known which use different sensors (usually ultrasound-based) to scan the surroundings of the vehicle and calculate the distances between the vehicle itself and the obstacles used to warn the driver.
• Systems are also known which calculate a trajectory in recognized parking spaces and can maneuver the vehicle by driving the steering semi-automatically into the parking space. If such systems are operated separately, it may be difficult for the driver to associate the warning or event with the camera image. In addition, it is not always clear from the picture which obstacle is critical and which is not.
• a navigation device connects image data acquired by an image data acquisition unit to a position where the Image data are recorded, and stores the data as a previous image data in an image memory.
• the control unit further determines whether the vehicle has entered at least in a parking target area. Is determined that the vehicle in the
• Parking set area then an earlier image based on the earlier image data taken by the parking target area and a current image based on the image data obtained at the current position of the vehicle are displayed on a direction display.
• US Pat. No. 7,212,653 B2 discloses an image processing system for a vehicle for taking pictures from the surroundings of the vehicle. The images are stored and a processed image obtained by processing a previous image taken before reaching a current position is superimposed on a portion of a current image that can not be captured by the camera.
• the system has one
• An imaging member for recording with a single camera during the transition of the vehicle from a first to a second location, first and second images around the vehicle at a first and second location and at a first and second time.
• the system has an object distance calculating part for calculating a distance from the vehicle to the 3D object by using the position of the 3D object in each of the first and second images and the transition data of the vehicle.
• an image generator is provided for generating a third image based on the images and the acquired data.
• the object of the present invention is to obtain more meaningful images of a vehicle environment and to assist the driver
• a method for generating an image of a vehicle surroundings of a vehicle with a camera
• Each field corresponds to the entire detection range of the camera
• the first field and the second field are assembled to the image of the vehicle environment so that the image shows a larger area than each field.
• a camera for capturing a first partial image in a first position or orientation and for capturing at least one second partial image in at least one second position or orientation of the camera, wherein
• Each field corresponds to the entire detection range of the camera, further comprising
• An image synthesis device with which the first field and the second field to the image of the vehicle environment is composable so that the image shows a larger area than each field.
• the driver can thereby avoid the situation in the
• the generated image may have a virtual viewpoint above the vehicle. It is thus provided a bird's eye view image. Such a picture further makes it easier for the driver to assess the current situation.
• the picture shows a parking space in at least one of the fields
• the driver is presented in a normal mode one or more of the partial images and in an extended mode, when the parking space is recognized as free, the entire composite image.
• the parking space can be measured by means of an ultrasonic sensor.
• ultrasonic sensors are robust and reliable. You can capture limitations of a parking space for surveying or visual representation with sufficient accuracy.
• an artificial picture element which represents a dimension of the parking space, an obstacle or a calculated parking path is superimposed in the picture.
• obstacles that are outside the field of view of the camera system can also be displayed. With this information, the driver can lead the vehicle much safer.
• a picture element representing an automatically proposed parking position can be faded into the picture and the parking position can be changed by manually shifting the picture element.
• the driver can thus one
• Parking process of the vehicle carried out semi-automatically or fully automatically. Of the Parking process can thus be better assessed or tracked by the driver.
• the image may represent multiple parking spaces in the vehicle environment, and the driver may select one of them for parking.
• the image is not only used to visualize a parking operation in a given parking space, but also to select one of several parking spaces meaningful.
• the imaging device can have a plurality of cameras. Then, the images taken in parallel with the cameras can be used together with images taken earlier by the cameras to produce an overall image. It may be sufficient if each of the individual cameras each have a detection range corresponding to a radius of less than 5 m. This is usually sufficient to register the critical environment of a vehicle, in particular for a parking operation.
• a vehicle equipped with such a driver assistance system can be guided by a driver very comfortably.
• Fig. 1 is a bird's eye view of a parking situation with parallel
• Fig. 2 is a block diagram of a controller with connected
• Fig. 3 is a principle block diagram of the controller of Fig. 2;
• Fig. 5 is a map of the environment in a schematic representation
• Fig. 6 is a map view, as of a invention
• Imaging device is generated.
• Embodiments of the present invention In contrast to the known systems for generating images of a
• Vehicle environment the consecutively obtained in different vehicle positions or orientations or positions of the camera images so put together that an image of the vehicle environment can be displayed, which has a larger area than the detection range of a single camera.
• Camera system can be used in particular to take in the measurement of a parking space successively multiple images, store them and then using the images taken and by means of displacement sensors (wheel sensors on the vehicle, possibly a steering angle sensor) determined travel path to generate an expanded environment image that the parking space and covering the parking space limiting obstacles.
• displacement sensors wheel sensors on the vehicle, possibly a steering angle sensor
• partially overlapping images can be combined to form a new image, the new image combining image information from the individual images and resulting in a larger image.
• the general approach is to combine the display and environmental sensing functions.
• the resulting HMI output is particularly understandable for the driver.
• the environment of the vehicle is scanned by means of suitable sensors and an image of the environment is stored in an environment map (see FIG.
• the information stored in the area map is further processed so that individual obstacles can be identified and classified.
• the risk of collision is also determined and, if necessary, values for a distance-to-collision and time-to-collision calculated.
• the information obtained using the environment map is used when generating HMI images.
• the detected obstacles are displayed directly at the right places in the image (obstacle highlighting).
• the dangerous obstacles are highlighted by a special presentation.
• distances can also be displayed as bars or numeric values.
• the predicted or planned trajectory, for example, of a semi-automatic parking process of the vehicle is also displayed. This type of presentation can be used both for images generated from a natural perspective (live view) and for images created as a bird's-eye view.
• Obstacles 3, 4, 5, the target position 6 in the parking space 2 and the currently planned trajectory are displayed.
• Such a representation can also be used for an interactive correction of, for example, the target position by the driver.
• a corresponding input means is then provided to move the picture element, which is shown as a superposition element (target position 6) in the overall picture.
• the system calculates the
• the camera or the cameras of the vehicle 1 take a plurality of partial images, from which a current image 7 (bird's-eye view in FIG. 1 or enlarged live-view image) is generated.
• Image 7 is thus the result of a combination of images taken at partially different times, these images individually covering only a part of the image 7, if necessary with overlapping. That generated
• Whole image can be expanded with virtual objects that have not been seen with the camera, but z. B. already detected by ultrasound.
• a camera image 7 possibly composed of several partial images, is available up to half of the gap 2.
• a curb 4 or a wall could be detected.
• Curb 4 may already be displayed above the camera image 7 in the image presented to the driver. If necessary, the curb 4 is even displayed behind the parked vehicles 3 and 5 in the image, if it was registered by other sensors or interpolated computationally. All these aspects help to better connect the camera image with reality.
• the camera image 7 extends to the limit of the detection range of the camera system. In the example of FIG. 7, however, the camera image 7 does not extend to the curb 4.
• the area between the border of the camera image 7 and the curb 4 can be shown in color for better orientation of the driver.
• the parking space 2 or the target position 6 could be marked green, which conveys to the driver that the parking space is sufficiently large.
• the area from the detection limit of the camera system to the curb could be grayed out. In particular, it can be represented with that color which possesses, for example, the roadway or the optically detected partial area of the parking space.
• the curb 4 au outside the detection range of the camera system is detected by the ultrasonic sensors in position and size. From these measured quantities, a virtual object and its position are calculated true to scale in the enlarged image displayed to the driver.
• the extended image is thus composed of a camera image (possibly several sub-images) and the virtual objects, with intervening areas possibly being suitably colored.
• the "camera image" can be calculated from one or more live-view images.
• This fade-out is always dynamically adapted to the current image representation or the current position of the vehicle.
• the camera images are processed so that objects can be recognized. In doing so, the information from the area map for the calculation of interesting areas and
• Another illustration possibility relates to the fading in of the information about the environment into the viewing devices used by the driver taking into account the
• Such vision devices include mirrors (side mirrors, rearview mirrors) and head-up display's front and
• Rear window or a hologram presentation unit which moves the static or dynamic, recognized objects in the field of view of the driver.
• the driver moves in the direction of an obstacle to be warned, he can there recognize the object or obstacle being warned, real and marked, and he understands the warning during the parking operation or the driving movement.
• Particularly advantageous is a representation in combination with the electronic, camera-based mirrors, in which the image is generated purely electronically and suggests a mirror image.
• the map information can be blended into all the mirror images so that the driver, no matter where he looks, can always perceive the system view of the obstacles, trajectory, and so forth.
• control unit 10 which is networked with several peripheral devices 1 1 to 16.
• This controller 16 is capable of producing an image according to the present invention. For this purpose it receives corresponding image information or partial images from a camera 1 1 or more cameras.
• the controller 10 the camera 1 1 drive to capture images.
• the orientation of the camera 1 1 can be varied via the control unit 10.
• control unit 10 has as further suppliers of
• Input information ultrasonic sensors 12 These provide, for example
• an electric power steering are connected to the control unit 10, for example via a CAN bus. It delivers the current steering angle to the
• an ESP 14 (Electronic Stabilization Program) is connected to the control unit via the CAN-BUS.
• the ESP 14 provides motion data to the controller 10. This includes, for example, the wheel speed, the roll direction, the yaw rate, the acceleration, and the like.
• control unit 10 is connected here to a head unit 15.
• head unit 15 any other optical output device
• control unit 10 (eg screen). It can also be controlled by the control unit 10, several optical display devices.
• a speaker 16 or more speakers is used as an acoustic wave
• the structure and the data flows of a system implementing the solution according to the invention will now be explained with reference to FIG.
• the camera 1 1 is in a
• Video processing unit 20 embedded. It supplies video raw data 21 to
• Video controller 22 This will, as indicated by the rectangle on the right edge of the box of the video controller 22, result in a processing loop
• an ultrasonic processing 30 is provided. It comprises the ultrasonic sensor 12. This supplies raw data 31 to a signal filter 32. The filtered sensor data 33 become a
• Obstacle detection unit 34 supplied. This continuously performs a data analysis to detect obstacles in a processing loop.
• a position detection unit 40 (eg, odometry, ESP, EPS, etc.) constantly supplies current position data 41 to a memory unit 42. From this memory unit 42, the video control unit 22 and the obstacle detection unit 34 call
• the current position on request 43 is made available to a memory unit 44 in which the local map of the immediate vicinity of the vehicle (see Fig. 6) is stored.
• the surrounding area refers to a radius of 30 m around the vehicle.
• an object tracking 45 is constantly performed. In this case, it is calculated, for example, in which direction an object moves relative to the vehicle, so that it can also be displayed when it is outside the detection range of the camera 11 or the ultrasonic sensors 12.
• a functional logic 50 receives on request 51 map data or
• Obstacle information from the memory unit 44 is from an ultrasonic parking assistant 53, which also the desired
• a continuous processing loop constantly checks and calculates the distances. For example, he issues corresponding warnings 55 to the driver via a loudspeaker.
• segment data 56 of the ultrasonic parking assistant 53 are stored in a memory unit 57 of the functional logic 50. For example, this data relates to bars that need to be superimposed on an image.
• a display control unit 58 of the function logic 50 receives video data 25 processed by the video control unit 22. Further, the display control unit 58 obtains map data or obstacle information 46 from the memory unit 44 on request 59. Further, the display control unit 58 reads the ultrasonic segment data from the display unit
• Display control unit 58 is constantly running a fitting and generating process 61 for overlay elements.
• the display control unit 58 makes an output 62, for example to a screen or a head unit, each of which forms an optical interface to the driver.
• An interaction 63 takes place between the driver and the display control unit 58.
• the driver can, for example, change the display or, for example, move an element in the image (for example, shift parking position marking).
• FIG. 4 shows a live view representation, which may be composed of several sub-images.
• a roadway 70 to recognize on the right side parking 71 are located.
• one of the parking spaces 71 has a detected obstacle 72, 73.
• the obstacle 73 is not in the detection area of the camera, it is displayed as a virtual object because it was detected in a previous image or by another sensor. It was taken for the current presentation.
• the system has detected a wall 74 as an obstacle. Between the two vehicles, d. H.
• a parking space 75 which detects the system and overlays with a rectangular overlay element in perspective exactly in the live view image.
• the system computes a target position 76, which is also marked by a rectangular overlay element (overlay) in the live view image. If necessary, a trajectory for a parking process is displayed in the image.
• the image shown in Fig. 4 may be composed of a plurality of partial images obtained in succession in different vehicle positions.
• the image can also be enlarged or widened on the right-hand edge so that the obstacle 73 can then also be seen in order to facilitate orientation for the driver.
• FIG. 5 first shows a real situation in a schematic representation.
• a vehicle 80 parks in accordance with a trajectory 81 between two vehicles 82 and 83.
• In the parking lot are numerous other obstacles such as a wall 84, a motorcycle 85, other vehicles 86 and garbage cans 87.
• the imaging device of the vehicle 80 registers these obstacles and makes a map therefrom, as shown in FIG 6 is shown.
• Each obstacle is blocked by a block 82 'to 87' d.
• H. virtual object symbolized.
• the motorcycle 85 is symbolized by the block 85 '.
• the image presented to the driver FIG.
• Conventional camera systems comprise at least one reversing camera and side cameras (eg in the exterior mirrors or corner regions of the bumpers). These camera systems have a detection range of approx. 2 m to 4 m range around the vehicle. This detection range is usually not sufficient to represent the parking space over its entire extent along the guideway.
• the camera images are therefore combined in the present example to a total image of a virtual viewpoint above the vehicle (bird-view).
• the successively recorded images can be used, in particular, as shown here, to measure a parking space.
• the recorded images are stored and used to generate an expanded environmental image on the basis of the travel sensors (wheel sensors on the vehicle, optionally a steering angle sensor) determined by means of displacement sensors. This image represents the parking space and the parking space limiting obstacles, for example, as shown in FIG. 6 again.
• the camera system according to the invention particularly knows a normal mode
• the parking process can then be semi-automatic (steering only) or fully automated (steering and braking).

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Mechanical Engineering (AREA)
• Traffic Control Systems (AREA)
• Closed-Circuit Television Systems (AREA)
• Image Processing (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=45001710

Family Applications (1)

Country Status (5)

Families Citing this family (59)

Citations (1)

Family Cites Families (20)

• 2010
  • 2010-11-12 DE DE102010051206A patent/DE102010051206A1/de not_active Ceased
• 2011
  • 2011-10-26 CN CN2011800545421A patent/CN103328261A/zh active Pending
  • 2011-10-26 WO PCT/EP2011/068728 patent/WO2012062573A1/de active Application Filing
  • 2011-10-26 US US13/884,573 patent/US9544549B2/en active Active
  • 2011-10-26 EP EP11785353.1A patent/EP2637898A1/de not_active Withdrawn

Patent Citations (1)

Also Published As

Similar Documents

Legal Events