DE102011082881A1 - Method for representing surroundings of vehicle e.g. motor vehicle e.g. car, involves transforming primary image information into secondary image information corresponding to panoramic view using spatial information - Google Patents

Abstract

The method involves receiving (100,110) primary image information of environment of a motor vehicle using vehicle cameras of vehicle camera system. The spatial information is determined (120) from received primary image information. The primary image information is transformed (130) into secondary image information corresponding to panoramic view using spatial information. The surroundings of vehicle in panoramic view are represented (140) based on secondary image information. An independent claim is included for system for representing surroundings of vehicle.

Description

The invention relates to a method for representing the environment of a motor vehicle in a particular view. Furthermore, the invention relates to a corresponding system for displaying the environment.

Camera-based systems for displaying the vehicle surroundings are known from the prior art, which transform and display image information of a camera or several cameras installed at different positions of the vehicle by means of image processing in a view which is particularly clear to the driver, for example in a view from above. When using multiple cameras, the image information of these cameras is transformed in perspective and then these partial views are combined to form an overall view. For example, the two images on the right and left side of the vehicle exterior mirror cameras and the image of a rear camera are transformed perspective to a plan view and then joined together by the resulting partial views to form an overall view, for example, to a 270 ° or 360 ° of the surrounding comprehensive view , The joining of individual images to an overall view is also referred to as image stitching.

In order to combine images from different camera positions without error, the perspective for all camera images should typically be converted to an identical projection center as part of the image transformation, for example to a projection center directly above the vehicle.

In the transformation of image information, conventional systems generally adopt the simplifying assumption that all objects within the camera image are in one plane, typically the roadway plane. Sublime objects, which in reality emerge from the plane or are above the plane, are increasingly displaced with increasing height during the transformation due to this simplification, and are therefore mapped in the original image in the wrong position. This artifact becomes particularly noticeable when joining images of several cameras together (image stitching). In the transition area between two camera images, raised objects can disappear completely or be displayed twice.

From the publication EP 1 775 952 A2 a system is known, which converts images of several vehicle cameras to a total image in a bird's-eye view. In this document, for example, it is proposed to set an example comb-shaped boundary line in an overlap area between two images in a bird's eye view, wherein one image is used to generate the overall image above the boundary line and the other image is used below the boundary line. This prevents a raised object from disappearing when projected onto the floor in the overall picture.

From the publication DE 10 2008 029 181 A1 is a system known with two cameras, which creates a single image of an overall picture that represents the environment in a bird's eye view. The overall image is composed of partial regions of individual images of the two cameras in an overall overlapping region, whereby in each case one subregion of the individual image of the first camera is arranged parallel to a subregion of the individual image of the second camera.

Indian publication DE 10 2009 036 200 A1 a method for monitoring an environment of a vehicle is described, wherein the environment and existing objects in this by means of at least two image acquisition units, the coverage areas at least partially overlap and form an overlap area are detected, wherein from using the image acquisition units captured individual images based on an image processing unit an overall image is generated, which shows the vehicle and its surroundings from a bird's eye view. Depending on determined positions of the objects in the overall image between a first individual image area and a second individual image area, a profile of at least one boundary line extending from an origin to the image edge of the overall image is predefined such that the boundary line extends away from the objects.

From the publication DE 100 59 900 A1 For example, the bird's eye view of pictorial environmental information is known, with raised or moving objects detected within the image data. In this case, recourse can be had to methods known from stereo image processing. The location parameters of the detected objects can be made available to a system for further processing. It is conceivable that the non-drivable space described by its location parameters consists of a contiguous area or is formed of several non-contiguous subareas. To emphasize the non-passable spaces, the corresponding areas in the image data can be hidden or masked in the image data.

It is an object of the invention to provide an alternative method for representing the environment of a motor vehicle, which allows the most error-free representation of raised objects.

The object is solved by the features of the independent claims.

One aspect of the invention relates to a method for representing the surroundings of the motor vehicle in a specific view in a motor vehicle, for example for representing the motor vehicle in a view from above.

In the method, first image information of the surroundings of a motor vehicle is detected with at least one vehicle camera of a vehicle camera system comprising one or more cameras. For example, two cameras are used for this purpose.

In the method, spatial information, such as depth information (i.e., distance information) or spatial coordinates, is determined with respect to the captured first image information. The spatial information is preferably generated by the camera system itself. The spatial information is generated, for example, by a stereo triangulation method, the images of at least two cameras in which the image fields overlap being evaluated. Instead, however, other methods, such as an active triangulation method, can be used.

Using the spatial information, the first image information is transformed into second image information corresponding to the desired view

Finally, the environment of the motor vehicle is displayed in the desired view based on the second image information. In particular, second image information for the plurality of directions can also be put together and this combination displayed.

By considering spatial information, the invention enables a realistic perspective transformation of the image information. This allows a faultless joining together when creating an overall view of several partial views.

The view shown may be a view from above (for example, an orthogonal view from above with a virtual camera direction orthogonal to the road plane, which is also referred to as a plan view, or alternatively from obliquely above). Here you can combine several partial views from a virtual camera view from the top to a general view from the top.

Alternatively, an overall view can also be generated from a plurality of partial views, in which the virtual camera is arranged parallel to the road surface, and the partial views from different directions of the surroundings are joined together to form a panoramic view.

It is advantageous if at least one camera or even several cameras of the camera system are used to determine the spatial information.

Preferably, a passive triangulation method is used to determine the spatial information, which is based on the evaluation of at least two images from at least two different camera positions. For this purpose, a first image from a first camera position is recorded to record the first image information. In addition, to capture the first image information, a second image is taken from a second camera position. The recorded surrounding area of the first image and the recorded surrounding area of the second image overlap at least partially. Preferably, the first image is taken with a first camera and the second image is taken with a second camera. It would also be conceivable to use a single camera for recording the two images, which can be moved, for example.

In an alternative embodiment, an active triangulation method may be used in which, for example, with a camera of the camera system functioning as a sensor, a light signal reflected by the scene is picked up by an additional light source, for example a laser or an LED light source.

The spatial information may be, for example, depth information (i.e., distance information) or spatial coordinates.

Preferably, spatial coordinates for pixels of the first image information are determined when determining the spatial information. The spatial coordinates can be determined directly, for example, using a triangulation method. Alternatively, depth information (i.e., distance information) is first determined by a triangulation method, and spatial information for pixels of the first image information is then calculated using the distance information.

The linking of the first image information with the spatial coordinates corresponds to three-dimensional image information, ie a three-dimensional image. The three-dimensional image information is typically confined to a certain area in space (this is often referred to in the literature as 2½-dimensional information) and it is no three-dimensional image information for all spatial points of space as in a computer tomograph available.

In the transformation of the first image information into second image information, the position of the camera (i.e., the projection center) is changed virtually according to the desired view, preferably by means of three-dimensional projection.

For transforming the first image information into second image information, the three-dimensional image information is projected into the second image information by changing the projection center. The three-dimensional image information is thus projected into a two-dimensional image plane associated with a new projection center. The resulting second image information corresponds to a two-dimensional image in a two-dimensional plane. The projection is implemented, for example, in such a way that the position of the object points in the scene is related relative to the new projection center (for example above the vehicle) and an existing rotation of the coordinate systems relative to one another by means of rotation matrices is taken into account. Depending on the position of the observer to this new projection center then the three-dimensional scenery (which is now in coordinates relative to the projection center) is projected into a two-dimensional plane.

In a preferred embodiment, in which an overall view is generated by image stitching from a plurality of partial views, first image information is recorded in each case for a plurality of partial views. At least for a partial view, spatial information regarding the respective first image information is determined. For each of the plurality of partial views, in each case the first image information is transformed into second image information, this taking place at least for a partial view using the corresponding spatial information by a three-dimensional projection. The three-dimensional projection can be performed several times for all partial views. However, this is not absolutely necessary; For example, the three-dimensional projection can only take place for one or more partial views that comprise a raised object. The projection takes place for the individual partial views in such a way that the resulting image information has the same projection center for all partial views. The second image information assigned to the individual partial views is joined together and the joining is displayed.

Since the projection center is preferably selected identically for all partial views that are subsequently joined together to form an overall view, the individual partial views can be joined together in an artifact-free manner to form an overall view.

A second aspect of the invention relates to a system for representing the environment of the motor vehicle in a particular view, for example in a view from above. The system includes a vehicle camera system with one or more vehicle cameras. The vehicle camera system is set up to record first image information of the surroundings of a motor vehicle with at least one vehicle camera. Furthermore, there are means for determining spatial information with regard to the recorded first image information. The system is further adapted to transform the first image information using the spatial information in second image information appropriate for the particular view. Further, the system includes means for displaying the environment of the motor vehicle in the particular view based on the second image information.

The above statements on the method according to the invention according to the first aspect of the invention also apply correspondingly to the system according to the invention according to the second aspect of the invention.

The invention will be described below with reference to the accompanying drawings with reference to an embodiment. In these show:

1 an example of an artificial change in the camera perspective;

2 the resulting error when changing the camera perspective;

3 a built in a vehicle, exemplary camera system with at least two front cameras;

4 exemplary processing of the frames of the front cameras;

5 the joining of the processed individual images of the front cameras into an overall view;

6 a flowchart for an embodiment of the method according to the invention; and

7 an example of a camera system with eight cameras.

1 and 2 show an example of an artificial change of the camera perspective and the resulting display error in one sublime object. In transforming the image information to change the perspective of an actual camera position 1 to a virtual camera position 2 (here in the camera position of a supervisor) is generally taken the simplistic assumption that all objects 3 within the camera image in one plane 4 , typically the carriageway plane. Sublime objects 3 that in reality is out of the plane 4 come out or above the plane 4 is located in the transformation due to this simplification with increasing height h increasingly offset and thus mapped in the original image at the wrong position. This is in 2 to see what the erroneous offset Δ of the raised object 2 in the transformed image taking into account the virtual camera position 2 shows. Actually, the object would have to 4 when changing the camera perspective from the real camera position 1 to the virtual camera position 2 in the correct position 5 being represented. In fact, the endpoint of the object becomes 2 but at the faulty position 6 represented by the offset Δ relative to the position 5 is offset. For the offset Δ, Δ = h / tanβ, where the angle β is the viewing angle between the actual camera position 1 and the plane 4 describes.

This misrepresentation becomes particularly obvious when joining images of several cameras that have been transformed in their perspective. In the transition region between two camera images, raised objects can disappear completely or be displayed twice, since the error described above is opposite for both camera images. This will be explained below with reference to an example.

3 shows a built-in vehicle camera system with at least two front cameras 10 and 11 , The captured surrounding area of the first image and the captured surrounding area of the second image partially overlap.

The pictures of the two cameras 10 and 11 be like in 4 shown, processed and connected, as in 5 shown, joined together. 4a1 ) shows the recorded image of the camera 10 and 4a2) shows the captured image of the camera 11 , Both pictures are taken in a fish eye perspective. In 4a1) and 4a2) On the raised object a closure is centrally arranged, which is marked by a black circle and is located above the ground. In 4a1) This closure is located between the 3rd and 4th dark floor line to the right of the bright floor line 20 , In 4a2) the shutter is between the 3rd and 4th dark floor line to the left of the bright floor line 21 , 4b1) and 4b2) show the picture of the camera 10 or the picture of the camera 11 after a correction of the fish eye perspective. 4c1) and 4c2) show the picture of the camera 10 or the picture of the camera 11 after a subsequent image transformation, changing the camera perspective to a view from above. After the image transformation is in 4c1) the shutter is laterally approximately at the position of the 5th dark line to the right of the light line 20 , After the image transformation is in 4c2) the shutter is sideways approximately at the position of the 5th dark line to the left of the light line 21 , By changing the perspective without the use of three-dimensional information, it has thus come to a faulty displacement of the position of the shutter.

In 5 is the juxtaposition of image information 4c1) and 4c2) to an overall view. When joining together is not the entire image information of both images in 4c1) and 4c2) instead, part of the image information is truncated. Left of the dashed line in 5 becomes image information 4c1) used to the right of the dashed line in 5 becomes image information 4c2) used. When joining in 5 Due to the erroneous shift, object areas are lost with increasing height. Unlike the 4c1) and 4c2) is in 5 the shutter in the middle of the object is no longer visible.

Errors in the representation of raised objects, which are particularly obvious when joining several images together, are avoided according to the invention in that spatial information of the scene, in particular via the camera system itself, is generated, and this information is used in the image transformation. The determination of the spatial information can be done in particular by providing a sufficient overlap of the image fields of at least two cameras. The overlap can be used to generate a depth map of the scenery or the three-dimensional coordinates for the captured pixels using common passive stereo triangulation techniques. Likewise, using an additional illumination unit, an active triangulation technique could be used to generate the spatial information. In addition, the spatial information could be determined by a time of flight method or other method.

In 6 a flow chart for an embodiment of the method according to the invention is shown. This in 6 The illustrated embodiment essentially comprises the following steps:

• 1. Take two two-dimensional images from different camera positions (see steps 100 and 110 )
• 2. Determining spatial information, in particular determining spatial coordinates, by means of a stereo triangulation method (see step 120 )
• 3. associating the acquired two-dimensional image information (ie the brightness and / or color information) with spatial information, in particular with three-dimensional space coordinates, so that the linkage results in three-dimensional image information (see step 130 )
• 4. Projection of the three-dimensional image information on a two-dimensional projection plane starting from a new Prejektionszentrum (see step 140 )

In the steps 100 and 110 a dimensional image acquisition takes place by means of two two-dimensional cameras of the camera system, which are installed at different points of the vehicle. The cameras can be color cameras or black-and-white cameras that provide a color value or brightness value for the recorded pixels. These are, for example, the cameras 10 and 11 in 3 , The distance of the cameras 10 and 11 can also be significantly lower than this in 3 is shown. The two cameras can basically be integrated anywhere in the vehicle; In addition, the two cameras can only represent a subset of all cameras installed in the vehicle. When taking pictures in the steps 100 and 110 It is assumed that the scene area in the step 100 captured by a camera, with the scene area in step 110 is covered by the other camera, at least partially covered, as for example in 4a1) and 4a2) the case is.

Look in the steps 100 and 110 recorded image information can be triangulated by a stereo triangulation method spatial information (see step 120 ). For example, knowing the position of two corresponding two-dimensional pixels in the image of the first camera and the image of the second camera and the distance between the two cameras, the three-dimensional spatial coordinate of the corresponding to the two pixels point of space can be determined. To determine the spatial coordinates, the actual distance to the associated spatial point can first be determined from the displacement (disparity) between corresponding pixels via a disparity estimation. The determination of the three-dimensional spatial coordinates can be carried out for all two-dimensional pixels in the overlapping region of the cameras, so that the spatial coordinates can be determined for these two-dimensional pixels. The calculation of the spatial coordinates is, for example, under the Wikipedia website " http://en.wikipedia.org/wiki/Triangulation_(computer_vision) "Or in the book "Digital Image Processing", Bernd Jähne, Springer Berlin Heidelberg, 6th edition , the disclosure of which is hereby incorporated by reference in the present disclosure.

Alternatively, an alternative method for generating spatial information can be used to determine spatial information instead of a stereo triangulation method, for example an active triangulation method or eleven time-of-flight methods. In an active triangulation method, a specific light pattern (for example strips) is emitted with a light source, for example a laser or an LED light source, the reflection of which is detected by a sensor, for example a camera of the camera system, which in turn results in known object distance can be triangulated. Time-of-flight, on the other hand, is a transit time method, ie the transit time of the light emitted by the sensor and reflected by the object is determined in order to determine the distance therefrom. Both methods are described in the current literature, for example in "Handbook of Computer Vision and Applications," Bernd Jähne, Academic Press Inc, October 9, 2009 ,

In step 130 Each pixel in the overlapping region of the recorded two-dimensional images is assigned the respective spatial information, for example the respective three-dimensional spatial coordinates (for example x, y, z coordinates). The system is thus aware of which color or brightness value is located at the corresponding position relative to the vehicle.

In step 140 For example, the camera position (ie the projection center) is virtually changed by means of three-dimensional projection, for example in a plan view of the vehicle surroundings. In three-dimensional projection, the three-dimensional image information (ie the respective color or brightness value at the respective spatial points) is projected onto a two-dimensional image plane. This projection is implemented, for example, by relating the position of the image objects relative to the new projection center and taking into account any existing rotation of the coordinate systems relative to each other by means of rotation matrices. Depending on the position of the observer to this new projection center then the three-dimensional scenery (which is now present in three-dimensional coordinates relative to the new projection center) is projected into the two-dimensional image plane. In essence, this step involves cutting the visual beams with an image plane. The method of three-dimensional projection is on the Wikipedia website " http://en.wikipedia.org/wiki/3D_projection "Or in the book "Multiple View Geometry in Computer Vision," Richard Hartley et al., Carmbridge University Press, 2nd Edition , the disclosure of which is hereby incorporated by reference in the present disclosure.

In the foregoing, a perspective transformation of overlapping image information of two cameras using spatial information has been described. To generate an overall view from a plurality of partial views, the above-described steps for the individual partial views can be carried out, ie, for each partial view, the overlapping image information of two cameras is transformed into two-dimensional image information with a changed projection center using spatial information. Here, the projection center is selected identically for all partial views. For the partial views, however, a different tilt of the virtual camera can be used. When using the same projection center, the correctly projected partial views can be joined together in an artifact-free manner to form an overall view. For example, as in 7 shown, per camera monitored vehicle side one pair of cameras each with two mutually offset cameras are provided. When monitoring all sides of the vehicle so for example, eight cameras 10 - 17 be used. The dashed lines in 7 each give the field of view of the cameras 10 - 17 at. The picture information of the cameras 10 . 11 is perspectively transformed using spatial information into a first partial view. The picture information of the cameras 12 . 13 is perspectively transformed using spatial information to a second partial view. The picture information of the cameras 14 . 15 is perspectively transformed using spatial information into a third partial view. The picture information of the cameras 16 . 17 is perspectively transformed using spatial information into a fourth partial view. The four partial views are then combined to form an overall view.

In the absence of spatial information, the image transformation can only be performed with a change in the camera position if simplified assumptions are made. It is then typically assumed that the scene is completely in a plane, such as the roadway plane. Sublime objects, such as posts, are then incorrectly transformed into the two-dimensional plane. Since the error manifestation depends on the actual camera position, the error is different and is thus perceived as extremely disturbing, in particular when joining several images.

The invention described allows due to the use of spatial information, a realistic perspective transformation of captured camera images and, associated with an error-free joining of several partial views to a realistic display of the vehicle environment. To obtain the spatial information, the camera system itself is preferably used by, for example, evaluating the camera information of a camera pair by means of a stereo triangulation method. In this case, since the spatial information is obtained by using the camera system which is installed anyway for displaying the vehicle environment, the additional cost is low. For the realization of the function according to the invention, for example, only changes to the image field are made and additional computing capacities are made available on the control unit.

Likewise, the spatial information allows deriving further new functions and can be seen as a supplement to existing ultrasound sensors.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• EP 1775952 A2 [0005]
• DE 102008029181 A1 [0006]
• DE 102009036200 A1 [0007]
• DE 10059900 A1 [0008]

Cited non-patent literature

• http://en.wikipedia.org/wiki/Triangulation_(computer_vision) [0047]
• "Digital Image Processing", Bernd Jähne, Springer Berlin Heidelberg, 6th Edition [0047]
• "Handbook of Computer Vision and Applications," Bernd Jähne, Academic Press Inc, October 9, 2009 [0048]
• http://en.wikipedia.org/wiki/3D_projection [0050]
• "Multiple View Geometry in Computer Vision", Richard Hartley et al., Carmbridge University Press, 2nd Edition [0050]

Claims (11)

A method of representing the environment of the motor vehicle in a particular view in a motor vehicle, comprising the steps of: - picking up ( 100 . 110 ) first image information of the surroundings of a motor vehicle with at least one vehicle camera ( 10 - 17 ) a vehicle camera system comprising one or more cameras; - Determine ( 120 ) of spatial information regarding the captured first image information; - using the spatial information, transform ( 130 ) the first image information into second image information corresponding to the determined view; and - representing the environment of the motor vehicle in the particular view based on the second image information. The method of claim 1, wherein the particular view is a top view. Method according to one of the preceding claims, wherein - the step of recording first image information comprises: - recording ( 100 ) of a first image from a first camera position; and - recording ( 110 ) a second image from a second camera position, the captured surrounding area of the first image and the captured surrounding area of the second image at least partially overlapping; and the spatial information is determined by a triangulation method processing the first and the second image. Method according to claim 3, wherein the first image with a first camera ( 10 - 17 ) of the vehicle camera system and the second image with a second camera ( 10 - 17 ) of the vehicle camera system. Method according to one of the preceding claims, wherein the spatial information is determined using a camera ( 10 - 17 ) of the camera system. Method according to one of the preceding claims, wherein the spatial information is depth information or space coordinate information. Method according to one of the preceding claims, wherein in the step of determining spatial information spatial coordinates for the first image information are determined. The method of claim 7, wherein the step of transforming the first image information into second image information comprises: Projecting three-dimensional image information into a two-dimensional image plane, the three-dimensional image information corresponding to the association of the first image information with the spatial coordinates. Method according to one of the preceding claims, wherein For each of a plurality of partial views, first image information is recorded; - At least for a partial view spatial information is determined with respect to the respective first image information; For each of the plurality of partial views, in each case the first image information is transformed into second image information, this taking place at least for a partial view using the corresponding spatial information and the second image information for the plurality of partial views each having the same projection center; and - Combining second image information for the plurality of partial views and the assembly of second image information is displayed. The method of claim 9, wherein spatial information is determined for each of the plurality of subviews and the transformation is performed for each of the plurality of subviews using the spatial information. System for a motor vehicle for representing the surroundings of the motor vehicle in a specific view, comprising: a vehicle camera system with one or more vehicle cameras ( 10 - 17 ), wherein the vehicle camera system is adapted to receive first image information of the surroundings of a motor vehicle with at least one vehicle camera; - means for determining spatial information regarding the captured first image information; - means for transforming the first image information using the spatial information into second image information according to the determined view; and means for representing the environment of the motor vehicle in the particular view based on the second image information.

Images