EP2791895A1

EP2791895A1 - Method for improving the object detection in multicamera systems

Info

Publication number: EP2791895A1
Application number: EP12794902.2A
Authority: EP
Inventors: Tobias Ehlgen; Leo VEPA
Original assignee: Robert Bosch GmbH
Current assignee: Robert Bosch GmbH
Priority date: 2011-12-13
Filing date: 2012-11-19
Publication date: 2014-10-22
Also published as: IN2014DN03173A; DE102011088332B4; WO2013087362A1; DE102011088332A1; CN103999122A

Abstract

The invention relates to a method for reproducing raised objects (68) which are in a critical region (78) or are moving toward the latter. The critical region (78) is situated at interfaces (52, 54, 56, 58) assembled from visual ranges (30, 34, 38, 42) that are captured by a plurality of individual cameras (28, 32, 36, 40). The following method steps are performed: visual ranges (30, 34, 38, 42) are captured using a respective individual camera (28, 32, 36, 40) on a vehicle (10). Next, the visual ranges (30, 34, 38, 42) are assembled along interfaces (52, 54, 56, 58) to form a transformed bird's‑eye view (24). Finally, at least one interface (52, 54, 56, 58) between adjoining visual ranges (30, 34, 38, 42) is displaced upon detection of at least one raised object (68) that is situated within the critical region (78) at at least one interface (52, 54, 56, 58) or that is moving toward the latter.

Description

Description title

Method for improving object detection in multicamera systems. Prior art

The invention relates to a method for improving the object detection in multi-camera systems, in particular to an automatic displacement of seams in object detection in overlapping areas of images of a multi-camera system, a device for implementing the method and a computer program for performing the method on a computer device.

DE 10 2009 036 200 A1 discloses a method for monitoring the surroundings of a vehicle. The environment and objects located therein, in particular raised objects, are captured by means of image capture units in individual images and the individual images are used to create a bird's-eye view of the overall image. When creating the overall image, boundary lines are defined in the overlapping areas between the individual images. In order to be able to display complete objects in the overlapping regions of the individual images on the overall image, a boundary line is provided as a function of the determined position of the objects in the said image. This boundary line is set so variable that it runs away from the objects. Dividing lines between the individual images are defined, wherein the dividing line is designed such that raised objects can be displayed in the overlapping region.

DE 10 2006 003 538 B3 discloses a method for assembling several

Photographs of an overall bird's eye view. In particular, moving objects are displayed reliably. The images are captured from different positions and have an overlap area. The image recordings are transformed into a bird's eye view, which takes place as part of an image-data processing, and image sections of the transformed image recordings are combined to form a bird's-eye view of the overall picture. The transition between two adjacent ones Image sections take place along a straight or curved boundary line which runs asymmetrically between the image sections in the overall image.

KR 2010 0113 959 A discloses a method for monitoring the surroundings of a vehicle, wherein the image areas of several cameras partially overlap. In an image processing system, the individual images are combined to form an overall image in which the vehicle and its surroundings are shown from a bird's eye view. In order to image the environment as completely as possible, the cameras are arranged and aligned in such a way that their respective imaged areas are partially overlaid. Accordingly, the overall picture is expressed throughout the entire overlay area shown

Subareas of the individual images of the two cameras together. Alternately, a partial area of the single image of the first camera is arranged in each case to a partial area of the single image of the second camera. DE 10 2008 029 181 A1 discloses a device to visualize the environment to the driver and to represent obstacles in the vehicle environment. When an obstacle is detected by a detection element other than the camera, the

Image processing unit the direction and slope of the virtual view depending on the obstacle.

Today in many vehicles built multi-camera systems, such as a

Front camera, comprising at least one side camera in the exterior mirror and a rear camera, calculate from images a view from above of the vehicle from a bird's eye view and reflect the vehicle environment (Bird's Eye View, Top View or Area View). This allows the driver, by a single glance at a

Display unit, such as a display in a center console to see the entire closer vehicle environment. Thus, the driver can overlook blind spots due to the bird's eye view. To get a complete picture of the vehicle environment, the cameras are mounted in different positions. The front camera is usually located in the front or in the windshield, the rear camera either above the rear window or at the level of the trunk lid, while side cameras are usually with a vertical orientation on the road orientation in the exterior mirrors on the driver and passenger side of the vehicle. Some of these cameras have one Common field of view, which is referred to below as the overlap area. In current systems, the images that deliver these cameras, by a

perspective transformation (also called homography) on a plane. In order to join the pictures transformed on the plane into a common picture, the pictures are "stitched together", which is also called "stitching" in the jargon. The overlapping areas of individual juxtaposed images contain image data taken by several cameras. such

Overlap areas of two adjacent images or their image data define. Stitching points. In this case, each camera image is assigned an image area within the bird's-eye view today, i. a representation of

Vehicle environment and the vehicle from above. In common systems, a permanently implemented and unchangeable seam (stitching point) within the

Overlap area of images defined. Since the seam is usually optimized with respect to the plane, a blind wedge is created for raised objects near the seam in which these raised objects are no longer or hardly recognizable. The images taken by each of the cameras installed in the vehicle can not be stitched together along the camera lines, as is usually the case

envisaged installation positions of the cameras on the vehicle do not allow this. Presentation of the invention

According to the invention, it is proposed to modify the individual overlapping areas resulting between the cameras, in which the stitching points of the individual images are located, such that, on the basis of the

Presence of a raised object, the stitching point within the overlap area is shifted such that the at least one detected raised object always remains visible. Following the method proposed according to the invention, it is ensured that upon detection of a raised object within an overlapping area or upon detection of a raised object in the vicinity of a stitching point, the visualization of the at least one detected raised object in this area is prioritized prior to training Overlapping areas enjoys. By the method proposed according to the invention, the significant disadvantage of methods known from the prior art is now excluded, that in overlapping areas or in their vicinity sensitive Sublime objects are simply swallowed and can not be visualized by the driver despite their presence.

The detection of the at least one raised object, whether in the overlapping area of two individual images, be it in the vicinity of a stitching point, can be achieved by sensors such as e.g. Ultrasound, laser scanner, radar or lidar applications and the like are reliably detected today.

In the case where a plurality of raised objects are within the overlapping area of two frames or in the vicinity of a stitching point of frames, the stitching point is shifted so that the most relevant object is not within a non-displayable area but it is ensured that the detection point (Stitching point) on detection of at least one raised object includes this at least one raised object, ie this is located in the visualization area. In the above context, the term "most relevant object" means that at least one raised object, which is within the overlap region or which is in the region of a

Stitching point closest to the vehicle and / or at least one raised object moving at the highest speed.

Advantages of the invention

The method proposed according to the invention ensures that detected raised objects which are located in said areas, i. either in the overlap area of two frames or in the vicinity of a stitching point, do not disappear in a blind area. As a result, on the one hand, it is possible to overcome the disadvantages of the solutions known from the state of the art. Furthermore, the driver of such a vehicle is offered a significantly improved all-round view, in which all objects are and remain constantly visible. This not least leads to a considerable gain in security.

Furthermore, the solution proposed according to the invention makes it possible to color the detected object when displayed on a display within the vehicle in order to attract the driver's attention. Furthermore, it is ensured that the driver of the vehicle sees the object at any time, since this is not within a Interface area disappears, in particular not in a transition region of two adjacent viewing areas of the cameras. Furthermore, the solution proposed by the invention makes it possible to present raised objects in a much more natural manner, which results in a clear advantage with regard to the perception by the driver.

Brief description of the drawings

With reference to the drawing, the invention will be described below in more detail. It shows:

FIG. 2 shows an image of the vehicle, calculated from individual images, from the image taken by cameras mounted in the vehicle

Bird's eye view,

Figure 3, the viewing areas arranged at different positions in the vehicle

Cameras as well as the overlapping areas resulting from the individual viewing areas of the cameras,

FIG. 4 shows firmly defined stitching locations for an image on the image

Vehicle from a bird's eye view, Figure 5 is located in front of a vehicle (shown here as a semi-trailer)

Person,

FIG. 6 shows one in the region of an assembly point (stitching point)

person, in this case in the middle of the vehicle and thus no longer clearly visible,

Figure 7 shows a resulting blind wedge along that indicated in Figure 6

Assembly point (stitching point), FIG. 8 shows a detected raised object approaching a stitching point

FIG. 9 shows a displacement of the stitching point within the overlapping area in order to visualize the detected, at least one raised object for the driver of the vehicle.

Embodiments The illustration according to FIG. 1 shows a vehicle which has four individual cameras. It should be noted that the vehicle may also have more or fewer cameras, as long as they have at least one common

Have overlap area. As shown in FIG. 1, a vehicle 10 is equipped with a 4-camera system 12. The 4-camera system 12 includes a camera that captures an image of a

Front image area 14 receives, further includes a camera that receives a rear image area 16. The front image area 14 and the rear image area 16 of the vehicle 10 are identified with respect to a direction of travel 18. Furthermore, the 4-camera system 12 of the vehicle 10 comprises a side camera which receives a left image area 20 and a further side camera, which is mounted, for example, in an exterior mirror of the vehicle 10, which receives a right image area 22 next to the vehicle 10.

The representation according to FIG. 2 shows that the front image areas, rear image areas and left and right image areas 14, 16, 20 and 22 recorded according to FIG. 1 are transformed into a bird's eye view 24. The perspective resulting from the bird's-eye view 24 from above the vehicle 10 is shown in FIG.

The representation according to FIG. 3 shows that the vehicle 10 comprises a front camera 28. The front camera 28 of the vehicle 10 receives a viewing area 30, which is located substantially in front of the vehicle 10. In addition, the vehicle 10 has a rear camera 32, the viewing area 34 of which extends substantially behind the vehicle 10. In addition, the vehicle 10 includes a right side camera 36 whose viewing area 38 is located on the right side of the vehicle 10. Furthermore includes the vehicle 10, a left side camera 40, whose viewing area is identified by reference numeral 42 and which is located substantially on the left side of the vehicle 10.

The two side cameras 36 and 40 may be arranged, for example, in the exterior mirrors of the vehicle 10.

As can be seen from the illustration according to FIG. 3, due to the overlapping of the individual viewing regions 30, 34, 38 and 42 with one another, overlapping regions 44, 46, 48 and 50 respectively result.

This results in a first overlap region 44, resulting from an overlap of the field of view 30 of the front camera 28 with the front part of the field of view 38 of the right side camera 36. Similarly, a second overlap region 46 arises, resulting from the overlap of the field of view 30 of the front camera 28 with the viewing area 42 of the left side camera 40 of the vehicle 10 results. Furthermore, a third overlapping region 48 results, which results from the overlap of the viewing region 42 of the left side camera 40 of the vehicle with the front visual region 30 of the front camera 28. A fourth overlapping area 50 is formed by overlapping the rear part of the viewing area 42 of the left side camera 40 with the viewing area 34 of the rear camera 32.

Because of these overlapping regions 44, 46, 48 and 50, there is a need to provide individual bird's eye views of the vehicle 10 as will be described in greater detail in connection with Figure 4, to provide stitches of the frames for stitching together the frames.

FIG. 4 shows the individual stitching locations along which the individual images are assembled when the vehicle 10 is viewed from above. From the position of the seams 52, 54, 56, 58, which define a first, a second, a third and a fourth seam, it follows that the seams 52, 54, 56, 58 in a comparison with Figure 3, the individual Overlap areas 44, 46, 48, 50

correspond. Thus, according to FIG. 4, a modification of the field of view 30 of the front camera 28 as well as of the first overlapping area 44 and the second one takes place

Overlap 46 in the form of the first seam 52 on the right side of the Vehicle 10 seen in the direction of travel and the second interface 54 on the left side of the vehicle 10 seen in the direction of travel.

A comparison between the representations according to FIGS. 3 and 4 shows that the

Viewing area 30 of the front camera 28 is transformed to the first modified viewing area 60, which is a trapezoidal compared to the viewing area 30 of the front camera 28

Appearance. The same results from the transformation of the second

Overlap area 46 and the third overlap area 48 (see illustration according to Figure 3), which are to a second modified field of view 62 on the left side of the vehicle 10. The second modified viewing area 62 is bounded by the second seam 54 and by the third seam 56. A third modified vision area 64 located behind the vehicle 10 is defined by the third seam 56 and the fourth

Interface 58 which lead to the third modified field of view 64 on the back of the vehicle 10. Finally, a fourth modified field of vision 66 is formed on the right side of the vehicle 10, as shown in FIG. 4, bounded by the fourth seam 58 and the first seam 52. In this case, the first seam 52 corresponds to the first overlapping region 44 on the right front side of the vehicle 10 and the fourth seam 58 corresponds to the fourth overlapping region 50 on the right rear side of the vehicle 10-seen in the direction of travel.

The figure sequence of Figures 5, 6 and 7 are the presentation conditions in front of a vehicle - here a semi-trailer - in forming a blind wedge on the

Remove the front of the vehicle. Figure 5 shows that in front of a vehicle 10, here the tractor of a

Semitrailer, the vehicle center is designated by reference numeral 70, a raised object 68 is in the form of a person. The distortion in the vertical direction of the person representing a raised object 68 results from the vertical distortion that is traversed during the image transformation in order to arrive at a bird's eye view of the vehicle 10.

As can be seen in FIG. 6, the raised object 68 illustrated here as a point is located at a central interface 72 between the viewing areas on the front side of the

Vehicle 10. The central interface 72 corresponds to the vehicle center 70. The central interface 72 (stitching point) is located directly in front of the vehicle 10, so that there located raised objects 68 are no longer clearly visible to the driver of the vehicle 10.

In the illustration according to FIG. 7, the blind wedge resulting for the driver on the basis of the image processing with respect to the central interface 72 is designated by reference numeral 74. This blind wedge 74 is located in front of the vehicle front 76 of the vehicle 10, here designed as a semitrailer. The at least one raised object 68 is shown distorted in an oblique section as shown in FIG. This distorted representation within the blind wedge 74, which can only be shown imperfectly in FIG. 7 - leads to an unfavorable reproduction of the raised object 68, be it as a distorted point 68 in FIG. 6, or as a distorted person 68 as in FIG see. The blind wedge 74, as shown in FIG. 7, results from the circumstance that in the vehicle 10 according to FIGS. 5, 6 and 7 cameras are arranged in the exterior mirrors of the vehicle, which faces the central interface 72 (stitching position) extending in the direction of the vehicle front 76 of the vehicle

Detect visible range, whereby the area immediately before the vehicle front 76 just forms those blind wedge 74. This is indicated in FIG.

The illustration according to FIG. 8 can be seen on the basis of FIG. 4 that the modified viewing areas 60, 62, 64 and 66 are located around the vehicle 10. 4, the first modified viewing area 60 is bounded by the first and second stitching sites 52, 54, while the second modified viewing area 62 on the left side of the vehicle is defined by the second interface 54 and the third seam 56 is limited. Furthermore, it can be seen from the illustration according to FIG. 8 that the third modified viewing area 64 is delimited by the second interface 54 and the third interface 56. The fourth modified viewing area 66 located on the right side of the vehicle 10 is bounded by the fourth interface 58 and the first interface 52. From the plan view according to FIG. 8 it follows that within the first modified

Field of view 60 lying in front of the vehicle 10, a raised object 68 of in

Direction of travel of the vehicle 10 seen, the left side, i. starting from the second interface 54, from left to right in front of the vehicle 10 in the direction of the first

Interface 52 moves. From the illustration according to FIG. 8, it is also clear that here-shown in broken lines in FIG. 8 -a critical one around the first seam 52 Area 78 is defined. The raised object 68, which moves in the direction of movement 80 from left to right in front of the vehicle 10, moves toward this critical area 78 of the first interface 52. Reference numeral 82 designates an opening angle of this critical region 78, which in the illustration according to FIG. 8 is in the order of magnitude of 10 ° -20 °, for example 15 °.

9 shows that the at least one raised object 68 has moved in the direction of movement 80 as far as the first seam 52, and that a displacement 84 of the first seam 52 has taken place in the direction of the fourth modified visible region 66. As FIG. 9 shows, the first modified viewing area 60 and the fourth modified viewing area 66 are affected by the displacement 84 of the first interface 52. In the representation of the at least one raised object 68, the first modified viewing area 60 in front of the vehicle 10 is enlarged, whereas the fourth modified viewing area 66 lying to the right of the vehicle 10 is reduced. This is due to the visualization of the at least one raised object 68 at the first proximity point 52, whose original position is shown in dashed lines in FIG. 9.

The displacement 84 of the first interface 52 shown in FIG

Approaching the at least one raised object 68 to the first seam 52 can analogously also with a corresponding movement direction 80 of the at least one raised object 68 to a shift of the second interface 54 and the third interface 56 and finally the fourth, behind the vehicle 10th lying

Lead seam 58. The raised object 68 shown in FIGS. 8 and 9 may be one which moves very fast or which is in the greatest proximity to the vehicle 10. This is to be understood in the above context under a relevant object. From the bird's-eye view of the vehicle 10 shown in FIGS. 8 and 9, it can be seen that, in the context of image processing, the modification of the individual viewing areas 60, 62, 64 and 66 depends on the occurrence of the at least one in the direction of movement 80

moving raised object 68 to the respective displacement of the modified viewing areas 60, 62, 64 and 66 connecting seams 52, 54, 56, 58 leads. The invention further relates to a computer program, according to which one of the methods described above can be performed when the computer program is executed on a programmable computer device. The computer program can be, for example, a module for implementing a driver assistance system or a subsystem thereof in a vehicle, or an application for driver assistance functions that can be executed on a smartphone. The

Computer program can be stored on a machine-readable storage medium, such as on a permanent or rewritable storage medium or in association with a computer device or on a removable CD-ROM, DVD or USB stick. Additionally or alternatively, the computer program on a

Computer equipment, such as provided on a server for downloading, for example via a data network, such as the Internet or a

Communication connection, such as a telephone line or a wireless connection. By means of the solution described above in particular with reference to FIGS. 3 to 9, the position of the seams 52, 54, 56 and 58, which is firmly implemented in current systems and can not be changed, can be shifted and modified in accordance with the representation requirements. The solution proposed by the invention advantageously avoids the blind wedge 74, in which raised objects 68 are no longer recognizable or hardly recognizable. Due to the modification of the blind wedge 74 in front of the vehicle 10, raised objects 68 located there will not disappear in the blind wedge 74 in the short term - as shown in FIG. 7 - but the seams 52, 54, 56, 58 will be displaced depending on the object. The inventively proposed solution can be achieved in a vehicle equipped with multiple cameras that in the range of

Seams 52, 54, 56 and 58 existing raised objects 68 in improved

can be reproduced more realistic representation and it can be achieved in an advantageous manner that these raised objects 68 are always visible, although they may also be detected by cameras in a lower resolution. The safety aspect that the raised objects 68 are always visible to the driver in any case, weighs the disadvantage of possibly lower quality

Far lower resolution representation by far.

The invention is not limited to the embodiments described herein and the aspects highlighted therein. Rather, it is within the bounds of the pending Claims specified range a variety of modifications possible, which are within the scope of expert action.

Claims

claims

A method of reproducing raised objects (68) that is in the range of

Connected seams (52, 54, 56, 58) made up of from several cameras (28, 32, 36, 40) detected viewing areas (30 34, 38, 42) are located, with subsequent

Method steps: a) detection of viewing areas (30, 34, 38, 42), each with a camera (28, 32, 36, 40) on a vehicle (10), b) joining the viewing areas (30, 34, 38, 42) at seams (52, 54, 56, 58) to a transformed bird's-eye view (24), c) moving at least one seam (52, 54, 56, 58) between adjacent ones

Viewing regions (30, 34, 38, 42) upon detection of at least one raised object (68) located in the region of at least one seam (52, 54, 56, 58) or moving thereon.

2. The method according to claim 1, characterized in that the at least one

raised object (68) is detected by an external sensor or image processing algorithms.

3. Method according to the preceding claim, characterized in that the at least one raised object (68) is detected by ultrasound, laser radar, radar or lidar.

4. The method according to any one of the preceding claims, characterized in that the at least one interface (52, 54, 56, 58) within at least one

Overlap area (44, 56, 48, 50) of overlapping view areas (30, 34, 38, 42) is moved.

5. The method according to one or more of the preceding claims, characterized

in that, in the case of a plurality of raised objects (68) located in or approaching at least one overlapping area (44, 46, 48, 50), the at least one seam (52, 54, 56, 58) is displaced such that the most relevant of the raised objects (68) is detected.

Method according to the preceding claim, characterized in that the most relevant raised object (68) is the next object to the vehicle (10).

Method according to the two preceding claims, characterized in that the most relevant raised object (68) is the most mobile object in the vicinity of the vehicle (10).

Computer program for implementing the method according to one of Claims 1 to 7 on a computer-programmable device.

A driver assistance system for a vehicle (10) for displaying raised objects (68), which are located in the region of seams (52, 54, 56, 58) of joined viewing areas (30, 34, 40) detected by a plurality of cameras (28, 32, 36, 40) , 38, 42) are located with

- Cameras (28, 32, 36, 40) which detect a plurality of viewing areas (30, 34, 38, 42), a component for joining the viewing areas (30, 34, 38, 42) along seams (52, 54, 56, 58) to a transformed bird perspective view (24), a component for moving the at least one interface (52, 54, 56, 58) between adjacent viewing areas (30, 34, 38, 42) upon detection of at least one in a critical area (78) located on at least one interface (52, 54, 56, 58) or moving towards this raised object (68).