WO2018060409A1 - Method for reducing disturbing signals in a top view image of a motor vehicle, computing device, driver assistance system as well as motor vehicle - Google Patents

Method for reducing disturbing signals in a top view image of a motor vehicle, computing device, driver assistance system as well as motor vehicle Download PDF

Info

Publication number
WO2018060409A1
WO2018060409A1 PCT/EP2017/074750 EP2017074750W WO2018060409A1 WO 2018060409 A1 WO2018060409 A1 WO 2018060409A1 EP 2017074750 W EP2017074750 W EP 2017074750W WO 2018060409 A1 WO2018060409 A1 WO 2018060409A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
input
filter
motor vehicle
top view
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/EP2017/074750
Other languages
French (fr)
Inventor
Philippe Lafon
Vladimir Zlokolica
Akhil Kudlu
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Valeo Schalter und Sensoren GmbH
Original Assignee
Valeo Schalter und Sensoren GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Valeo Schalter und Sensoren GmbH filed Critical Valeo Schalter und Sensoren GmbH
Publication of WO2018060409A1 publication Critical patent/WO2018060409A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation

Definitions

  • the invention relates to a method for reducing disturbing signals in a top view image showing a motor vehicle and an environmental region of the motor vehicle.
  • the invention relates to an image processing device, to a driver assistance system for a motor vehicle as well as to a motor vehicle.
  • a third person perspective shows the environmental region of the motor vehicle as well as the motor vehicle itself in the form of a model of the motor vehicle from the view of a viewer external to vehicle, a so-called virtual camera.
  • a third person view can be a top view, in which the viewer looks from above to the motor vehicle.
  • This top view image can be displayed on the display device and shows an upper side of the modelled vehicle comprising the vehicle roof as well as the environmental region surrounding the motor vehicle.
  • the top view image can be created by warping and merging several input images captured by the cameras of the camera system.
  • the top view image gives the impression as if it had been captured by a real camera in a position of the virtual camera.
  • the merged top view image comprises some disturbing signals like artificial flickering effects also referred to as aliasing effect.
  • Those flickering effects degrade the image quality displayed to a driver and, hence, may disturb the driver.
  • this object is solved by a method, an image processing device, a driver assistance system as well as a motor vehicle having the features according to the respective independent claims.
  • Advantageous embodiments of the invention are the subject matter of the dependent claims, of the description as well as of the figures.
  • a respective filter strength can be determined as a function of the associated input coordinates and, depending on the filter strength, a respective filter for interpolating the output pixel can be determined.
  • a respective input coordinates of a pixel with subpixel accuracy within the at least one input image are determined for an output pixel to be interpolated of the top view image.
  • a respective filter strength is determined as a function of the associated input coordinates and, depending on the filter strength, a respective filter for interpolating the output pixel is determined.
  • the method serves for generating high quality top view images showing the motor vehicle and the environmental region surrounding the motor vehicle that can be displayed to a driver of the motor vehicle in the form of a video sequence, in particular a real-time video.
  • the top view images are created based on the input images or raw images captured by the at least one vehicle-side camera, e.g. by a vehicle-side image processing device.
  • the multi-camera system can be formed as a surround view camera system comprising four vehicle mounted cameras.
  • the surround view camera system can comprise a front camera for capturing input images of the environmental region in front of the motor vehicle, a rear camera for capturing input images of the environmental region behind the motor vehicle and two side cameras for capturing input images of the environmental region besides the motor vehicle.
  • the cameras can comprise fisheye lenses to enlarge a field of view of the cameras.
  • the input images distorted by the fisheye lenses can be dewarped and combined to the top view image during the same operation. Overlapping image areas taken from two different cameras and showing the same portion of the environmental area can be considered when determining the top view image
  • the input images as well as the top view images can be displayed on a display device, for example according to a view selected by the driver.
  • a display device for example according to a view selected by the driver.
  • the surround view camera system and the display device form a camera-monitor-system (CMS) which can replace mirrors of the motor vehicle.
  • CMS camera-monitor-system
  • the top view image which has a grid-like array of output pixels is generated from the at least one input image by scaling.
  • image resolution is changed. That means that from a pre-given raster graphics, a new image is generated with a higher or lower number of pixels.
  • a colour value can be associated with each output pixel or output picture element of the top view image, which colour value is calculated from the input image's pixels situated nearby. Then a filter is applied, which determines which pixels of the input image are used for calculation, and how their colour values are weighted.
  • the disturbing signals are particularly removed during interpolation of the output pixels for the top view image from the input image/the input images, approximately in real time.
  • the output pixels to be interpolated are positioned in the top view image in grid form, in rows or columns, and, thereby, have a respective position or respective output coordinates in the top view image.
  • the output coordinates are in particular integer, two-dimensional coordinates, each having a horizontal component or x-coordinate and a vertical component or y- coordinate.
  • an input image window or an input image area with input pixels is identified within the at least on input image, based on which the output pixel can be calculated.
  • the input image window describes a neighbourhood or vicinity of the output pixel in the at least one input image.
  • the input position or the input coordinates of the pixel with subpixel accuracy (hereinafter referred to as subpixel) corresponding to the output pixel is determined within at least one input image.
  • the input coordinates of the subpixel or partial image element usually have a non-integer value and describe the coordinates of a fictitious input pixel or input image element in the input image, which represents the output pixel at the associated input position.
  • real input pixels in the input image window especially their colour values, are used.
  • a specific filter strength is determined for the, especially for each, output pixel as a function of the input coordinates of the associated subpixel.
  • the output pixels are generated by the filter having the corresponding filter strength, for example by low pass filtering.
  • a horizontal filter strength value corresponding with a respective horizontal input coordinate of the subpixel and a vertical filter strength value corresponding with a respective vertical input coordinate of the subpixel are determined.
  • the invention is based on the knowledge that the output pixels coordinates have a variable spacing within the input image. This spacing information is therefore used to adjust the strength of the interpolation filter.
  • the filter strength can be a two-dimensional parameter having a horizontal value and a vertical value respectively to each output pixel that is interpolated.
  • the plan view image can be separately filtered in horizontal and in vertical direction, ie in columns and in rows, with customized, specific filters.
  • the customized filter strengths particularly high-quality top view images can be generated, which can be displayed to the driver on the display device.
  • the selection of the filter strength is performed for each output pixel and/or in regions on the output image areas having at least two output pixels in the horizontal and/or vertical direction.
  • the interpolation is performed pixel-by-pixel for each output pixel.
  • An output pixel-specific filter can be determined for each output pixel in the top view image for interpolating the respective output pixel. Therefore, it can be ensured that each output pixel is interpolated with an adapted filter optimized for that output pixel filter and, thus, particularly high quality and noise-reduced top view images can be produced.
  • a region-specific filter strength can be determined for certain output image areas, within which a strength of the disturbing signals hardly varies. Using area- based selection of the filter strength, computational time and computational complexity can be reduced in an advantageous manner, so that the top view images can be determined very quickly.
  • a combination of pixel-based and area-based selection of filter strength is possible. This means that, for example, a pixel-specific horizontal filter strength value is associated with each output pixel and a region-specific vertical filter strength value is associated with a group of output pixels (or vice versa). Due to these mixing forms, a trade-off between quality and computational time can be provided during the
  • a filter bank comprising at least two filter sets is predetermined, each filter set is assigned to a bandwidth corresponding with a respective filter strength and, for interpolating the respective output pixel, one of the filter sets is selected depending on the associated filter strength.
  • each filter set comprises at least two filters with different filter phases corresponding to subpixel values from the coordinates, wherein one of the filters is selected as a function of the determined input coordinates of the subpixel.
  • the filters of the filter bank are determined to be low pass filter curves. Thereby, a cut-off frequency of each low pass filter curve corresponds to a bandwidth
  • the filter bank can be stored in a storage unit of the image processing device and comprises the at least two groups of filters or filter sets.
  • Each filter set can be assigned an index, over which the respective filter set with the associated bandwidth is addressable or identifiable.
  • a filter set is provided for each filter strength.
  • the filter strength is, for example, compared to the indices of the filter sets, and that filter set is selected, wherein a deviation between the index and the filter strength is minimal.
  • the filters are not determined for each interpolation operation specifically, but one of the filter sets is selected based on the determined filter strength. Therefore, computing time can be saved in an advantageous manner.
  • the top view image can quickly and easily be determined based on the at least one input image.
  • At least one lookup table readable for interpolating output pixels for the top view image is predetermined, in which the respective input coordinates of the subpixels and the corresponding filter strengths are associated with predetermined output coordinates of output pixels.
  • the lookup table may also be stored in the storage unit of the image processing device to which the interpolation unit can access.
  • a respective input position of the subpixel as well as the filter strength is associated with an output position of an output pixel within the lookup table (LUT).
  • This lookup table is pre-defined. This means that the assignments are, for example, determined based on at least one reference input image and a reference top view image and are stored in the lookup table.
  • This lookup table can then be read during operation of the driver assistance system by identifying the input positions as well as the corresponding filter strengths in the lookup table for each output pixel to be generated. Both, a pixel based and a region- based filter strength can be determined.
  • the information in the lookup table are used to quickly and easily determine the associated filter by, for example, identifying them in the deposited filter bank.
  • this already stored lookup table can be read in real time so that, by means of the lookup table, and especially in combination with the predetermined filter bank, the corresponding filter parameters, ie the filter band and the filter phase, can quickly and easily be identified.
  • the corresponding filter parameters ie the filter band and the filter phase
  • an output image window comprising the respective output coordinates of the output pixel to be interpolated and an associated input image window comprising the respective input coordinates of the subpixel in the at least one input image are determined.
  • a spacing ratio is determined based on the input image window and the output image window
  • the spacing ratio here describes a local sampling ratio between the output image and the input image.
  • the samples as well as the output pixels in the output image are usually arranged regularly spaced from one another.
  • the positions within the input image may appear not regularly spaced because of the geometrical transform (warping) applied.
  • the spacing ratio is thus determined based on the input coordinates of the irregularly spaced subpixels which correspond to the output pixels within an output image window.
  • input coordinates which are located within the input image window are additionally deposited in the at least one lookup table. Based on these input pixels, for example, based on their colour values, the output pixel, for example, its colour value, can be calculated.
  • the lookup table the input pixels within the at least one input image can be identified, based on which the corresponding output pixel is interpolated.
  • a rule is stored in the lookup table, based on which a top view image can be determined from at least one input image captured by a camera.
  • the method is particularly efficiently designed so that high-quality top view images can be generated, approximately in real time.
  • the lookup table is determined as a camera-specific lookup table for the at least one camera, wherein the preparation of the camera-specific lookup table is performed during a calibration process of the at least one camera.
  • the lookup table can be determined every time the camera is recalibrated. Therefore, for example, an input image is captured and used as a reference input image. Thus, a reliable generation of high quality top view images can permanently be ensured.
  • a contrast enhancement and/or an edge enhancement for the at least one input image in case of the camera equipped with integrated enhancement functions are supressed or mitigated and the top view image is determined based on the at least one input image images without the contrast enhancement and/or edge enhancement.
  • integrated enhancement functions or enhancement features of the cameras are turned off or reduced.
  • Most cameras or camera systems include built-in image processing units with built-in contrast sharpening and/or edge enhancement. Even if the noise is not generated by the built-in sharpening functions, these sharpening functions exacerbate the strength of the disturbing signals in the top view image.
  • the integrated image processing unit of the camera can be controlled by the vehicle-side image processing device, whereupon the sharpening functions are not applied to the input images and unprocessed input images or raw images are provided to the image processing unit for rendering the top view image.
  • the sharpening functions of the cameras By suppressing or at least attenuating the sharpening functions of the cameras and determining the plan view images on the basis of unprocessed raw images, the introduction of disturbing signals into the top view image can be reduced in a simple manner.
  • the edge enhancement and/or contrast enhancement can be performed in the top view image after the interpolation of the top view image.
  • the invention also relates to an image processing device for a driver assistance system of a motor vehicle, which is adapted to perform a method according to the invention or an embodiment thereof.
  • the image processing device may be integrated, for example, in an in-vehicle control unit.
  • the image processing device can comprise an interpolation unit and a storage unit, wherein the interpolation unit can access the memory unit.
  • a driver assistance system for a motor vehicle comprises at least one camera for capturing at least one input image and an image processing device according to the invention.
  • the driver assistance system comprises at least four cameras forming a surround view camera system, wherein the image processing device is adapted to generate the output image based on the input images captured by the surround view camera system.
  • the invention also relates to a motor vehicle with a driver assistance system according to the invention.
  • the vehicle is particularly formed as a passenger car.
  • the cameras are in particular arranged on the motor vehicle in a distributed manner, so that the
  • Fig. 1 a schematic representation of an embodiment of a motor vehicle according to the invention
  • Fig. 2 a schematic representation of a top view image with disturbing signals
  • Fig. 3 a schematic representation of a general image pipeline
  • Fig. 4 a schematic representation of an interpolation of an output image area based on an input image area
  • Fig. 5 a schematic representation of the operation of an embodiment of an image processing device according to the invention.
  • Fig. 6 a schematic representation of a top view image with boundary artefacts.
  • Fig. 1 shows a motor vehicle 1 , which is formed as a passenger car in the present case.
  • the motor vehicle 1 includes a driver assistance system 2, which can assist a driver of the motor vehicle 1 in driving the motor vehicle 1 .
  • the driver assistance system 2 comprises a surround view camera system 3 for monitoring an environmental region 4 of the motor vehicle 1 .
  • the driver assistance system 2 comprises four vehicle mounted cameras 5a, 5b, 5c, 5d.
  • a first camera 5a is attached to a front area 6 of the motor vehicle 1 and serves for capturing input images showing the environmental region 4 in front of the motor vehicle 1 .
  • a second camera 5b is attached to a rear area 7 of the motor vehicle 1 and serves for capturing input images showing the environmental region 4 behind the motor vehicle 1 .
  • a third camera 5c is attached to a left side 8 of the motor vehicle 1 and serves for capturing input images showing the environmental region 4 to the left besides the motor vehicle 1 and a fourth camera 5d is attached to a right side 9 of the motor vehicle 1 and serves for capturing input images showing the environmental region 4 to the right besides the motor vehicle 1 .
  • the cameras 5a, 5b, 5c, 5d can comprise fisheye lenses in order to enlarge a field of view of the cameras 5a, 5b, 5c, 5d.
  • the input images or input video frames captured by the cameras 5a, 5b, 5c, 5d can be displayed on a display device 10 of the driver assistance system 2 in the form of a video.
  • the input images captured by the cameras 5a, 5b, 5c, 5d of the surround view camera system 3 can be warped and merged, e.g. by an image processing device 1 1 of the driver assistance system 2, to determine a top view image 12 of the motor vehicle 1 and the environmental region 4.
  • Fig. 2 shows a top view image 12 afflicted with disturbing signals 13 and generated from input images of the surround view camera system 3.
  • the top view images 12 or top view video frames can be displayed on the display device 10.
  • the environmental region 4 is shown from the perspective of a viewer above the motor vehicle 1 looking down to the motor vehicle 1 .
  • the top view image 12 is suggestive of being captured by a camera, a so-called virtual camera, positioned above the motor vehicle 1 . Since the motor vehicle 1 itself cannot be captured by the cameras 5a, 5b, 5c, 5d of the surround view camera system 3 a model 14 of the motor vehicle 1 is inserted into the top view image 12.
  • Fig. 3 shows a schematic representation of a general image pipeline 17 or video pipeline represented by a set of components 18, 19, 20, 21 , 22.
  • a customer view image 23 like the top view image 12, can be generated based on input images or raw images 24, like the raw images captured by the cameras 5a, 5b, 5c, 5d of the surround view camera system 3.
  • the image sensor unit 19 Based on image quality settings 20 of the cameras 5a, 5b, 5c, 5d the image sensor unit 19 generates the raw images 24 as an output.
  • an image processing device 21 e.g. the vehicle-side image processing device 1 1 , generates a virtual view, for example the top view.
  • the customer view image 23 can be determined for display on the display device 10.
  • the aliasing problem can be solved by limiting all frequencies in a signal, here a two-dimensional image signal, to half of the sampling frequency. This frequency limited to half of the sampling frequency is also called Nyquist frequency.
  • this problem is solved by low pass filtering the input images, which may be optimal for constant resampling cases, but not for variable resampling cases, since the low pass filter does not consider local variations.
  • a method for creating top view images 12 with reduced noise 13 is presented wherein a real-time processing as well as a limited computational complexity required in automotive applications can additionally be met.
  • the method for reducing the disturbing signals 13 can be performed by the vehicle-side image processing device 1 1 when determining the top view image 12 from the input images of the cameras 5a, 5b, 5c, 5d.
  • the image processing device 1 1 can comprise an interpolation engine 29 (see Fig. 5) which is adapted to interpolate an output pixel P (see Fig. 4) or an output picture element from any non- integer input coordinates Cx(P), Cy(P).
  • the interpolation engine 29 or the interpolator can determine a two-dimensional input position Cx(P), Cy(P) or 2D input coordinates and a two-dimensional local strength for each output pixel P to be generated.
  • the 2D input position Cx(P), Cy(P) (in sub-pixel resolution) is a position of an output pixel P to be interpolated within the input image (or a partial area of the input image in the case of a region-based approach).
  • the two-dimensional local strength or 2D filter strength corresponds to an index indexO, index2 (see Fig. 5) of a low pass filter applied for interpolation of the output pixel P.
  • Each index indexO, index2 corresponds with an appropriate filter band BandO, Bandl , Band2, Band3 of the low pass filter.
  • the 2D filter strength is in particular a two-dimensional index with a horizontal filter strength and a vertical filter strength. The horizontal and the vertical filter strength can be determined by a local estimation of the resampling as a function of pixels P' in a neighbourhood of the interpolated output pixel P.
  • the filter strengths can be determined as two- dimensional image area-specific filter strengths. Therefore, the interpolation can be performed in areas, if the resampling ratio hardly varies within the processed image area.
  • the filter strengths are, however, determined as the local pixel-specific filter strengths, so that the filter strength is adapted pixel-by-pixel. Thereby, an optimal adaptation or selection of the bandwidth of the filter is guaranteed, whereas, at the same time, the quality of the top view image 12 is maximized and boundary artefacts 32 (see Fig. 6) can be prevented at edges 31 or transitions of the image regions 15 resulting from a too coarse filter change.
  • the filter strength is provided for a group of two output pixels P, respectively, whereby a specific filter strength can be selected for a YUYV output image.
  • YUYV describes a colour model which uses two components, luminance "Y” and chrominance, which in turn consists of the two sub-components "U” and "V” for representation of colour information. It can be provided that the interpolation process is applied only to the luminance samples.
  • a chrominance can be determined, for example by averaging of the input chrominance considering the input luminance. The averaging can be carried out both in the horizontal and in the vertical direction.
  • the interpolation unit 29 is adapted to read a filter bank 30 with predefined low pass curves. By varying the filter bandwidth an adjustment near the required filter strength value is possible.
  • a number of filter curves depends on the overall resampling region. The number of filters is for example from four to eight filters.
  • An oversampling factor depends on the subpixel accuracy, which is provided by the interpolation engine 29. A subpixel accuracy of 1 ⁇ 4 pixel, ie a 1 ⁇ 4-pixel accuracy, at least requires a fourfold oversampling.
  • the interpolator 29 requires appropriate, in particular, for horizontal and vertical direction x, y, separate, filter curves that provide different filter strengths for both the horizontal and the vertical direction x, y.
  • a number of delay elements ("taps") of the low pass filter is dimensioned dependent on the narrowest filter band required or the narrowest bandwidth.
  • the filters can be formed, in consideration of the real-time processing required, with four delay elements, wherein an interpolation in this case is equivalent to a 4x4 interpolation filtering operation. Due to the limited bandwidth capacity (for example, 0:25 with four taps) it can be possible to downsample or pre-decimate the input images in advance to extend the overall filter range.
  • a pre-decimation of 1 ⁇ 4 extends the resampling range to 1 /16.
  • the input coordinates Cx(P), Cy(P) and the subpixel SP are appropriately recalculated to be adapted to the new, pre-decimated input image.
  • a resampling factor is calculated on pixel basis, which corresponds to a local ratio of an output distance to an input distance. Since the output samples are by definition regularly spaced in the top view image 12, ie the top view image 12, comprising one output pixel P, P per output sample, the output distance can be normalized to "1 ". Therefore, the resampling factor can be determined as 1 /input distance.
  • Fig. 4 an input image area 24 of an input image and an output image area 25 of a top view image 12 with a plurality of output pixels P, P' is shown.
  • a local, centred output image window 26 with a width Wx in the horizontal direction x and a height Wy in vertical direction y is defined around the output pixel P to be interpolated.
  • the output pixel P to be interpolated is located in the centre of the output image window 26.
  • the output pixels P +W x/2, P-wx/2, P + w y /2, P + wy/2 are located on edges of the output image window 26 at a distance of two output pixels P 'along the horizontal and vertical directions x, y from the output pixel P to be interpolated.
  • a rectangular input image window 27 with the dimension values or dimension parameters L, R, B, T can be defined.
  • the input image includes a plurality of input samples 28, which are indicated by crosses in the input image area 24.
  • the input coordinates Cx(P), Cy(P) of the output pixel P are located within the input image window 27, wherein the input coordinates Cx(P), Cy(P) are the coordinates of a so- called subpixel SP or partial picture element.
  • the input coordinates Cx(P'), Cy(P') of the output pixels P' in particular the input coordinates Cx(P +W x/2), Cy(P +Wx /2) of the output pixel P +W x/2, the input coordinates Cx(P- Wx /2), Cy(P- Wx /2) of the output pixel P. wx/2, the input coordinates Cx(P +Wy /2), Cy(P +Wy /2) of the output pixel P +Wy /2 and the input coordinates Cx(P- Wy /2), Cy(P- Wy /2) of the output pixel P- Wy /2 on the edges of the input image window 27, are shown
  • R maX (Cx(P+Wx/2) , Cx(P-Wx/2), Cx(P+Wy/2) , Cx(P+Wy/2)) ,
  • T min (Cy(P +W x/ 2 ) , Cy ⁇ P. Wy 2 ), Cy(P +W y/ 2 ) , Cy(P +W y/ 2 )) ,
  • the local resampling factor R is, for example, defined as:
  • the width Wx and the height Wy correspond to dimensions of the image area with the respective pixels to be interpolated.
  • the output pixel is not located in the centre of the output image window 26 determined for this image area.
  • the filter strength can be determined.
  • the filter bandwidth is the closest available value to R.
  • the bandwidth to be selected from a given filter set ⁇ 1 .0, 0.7, 0.5, 0.36, 0.25 ⁇ is the filter 0.36 (the third index in the list).
  • the division 1 /R can be prevented on implementation-side, so that the filter strength is determined from (R - L) and (T - B).
  • at least one lookup table LUT1 , LUT2 can be determined in advance and can, for example, be saved in the image processing device 1 1 .
  • the lookup table LUT1 is used for the pixel-based interpolation and the lookup table LUT2 is used for the image area-based interpolation.
  • the lookup table LUT1 includes input coordinates XB, YB of the input image pixels used for interpolation of an output pixel P, the input coordinates Cx(P), Cy(P) of the sub-pixel SP, which are also referred to as interpolation phases Ph1 , Ph2, and the filter strengths, which are characterized by the respective indices indexO index2. As already indicated, only one filter strength is provided for two output samples.
  • the preparation of the lookup tables LUT1 , LUT2 can be performed at certain times, in particular periodically, for example at a calibration of the camera 5a, 5b, 5c, 5d.
  • At least one of the lookup tables LUT1 , LUT2 is read by the interpolation unit 29 and the coordinates XB, YB of the input pixels or input image data to be used are identified. Respective filter strengths indexO, index2 and phases Ph2, Ph1 are associated with these input coordinates XB, YB, based on which the corresponding filter bands BandO, Bandl , Band2, Band3 with the
  • corresponding phases PhO, Ph1 , Ph2, Ph3 can be extracted from the filter bank 30 by the interpolation unit 29. Thereafter, the corresponding coefficients Px, Py can be loaded for the interpolation of the output pixel P.
  • a range based index can be stored in the lookup table LUT2.
  • a top view image 12 is shown which has been interpolated in image regions.
  • boundary artefacts 32 are visible at the edges 31 of the image regions 15 which result from a sudden change in the applied filter strength or the applied filter bandwidth.
  • the interpolation is performed pixel-by-pixel so that a top view image 12 can be generated with at least reduced disturbing signals 13 and no boundary artefacts 32.
  • a method which reduces the real-time complexity by adjusting the local Nyquist frequency, which is applied to each output pixel P.
  • indices indexO, index2 corresponding with certain filter bands are deposited in the lookup tables LUT1 , LUT2 so that the low pass filtering can be varied in real-time. This is in particular separately provided both for the horizontal and for the vertical direction x, y.
  • the method comprises the advantage that aliasing artefacts can be reduced, wherein an overall sharpness of the top view image 12 remains.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

The invention relates to a method for reducing disturbing signals (13) in a top view image showing (12) a motor vehicle (1) and an environmental region (4) of the motor vehicle (1), wherein for interpolation of the top view image (12) based on at least one input image captured by at least one vehicle-side camera (5a, 5b, 5c, 5d) - for an output pixel (P) to be interpolated of the top view image (12), respective input coordinates (Cx(P), Cy(P)) of a pixel (SP) with subpixel accuracy within the at least one input image are determined, - for the output pixel (P) to be interpolated a respective filter strength is determined as a function of the associated input coordinates (Cx(P), Cy(P)), and - depending on the filter strength a respective filter for interpolating the output pixel (P) is determined. In addition, the invention relates to an image processing device (119; to a driver assistance system (2) as well as to a motor vehicle (1).

Description

Method for reducing disturbing signals in a top view image of a motor vehicle, computing device, driver assistance system as well as motor vehicle
The invention relates to a method for reducing disturbing signals in a top view image showing a motor vehicle and an environmental region of the motor vehicle. In addition, the invention relates to an image processing device, to a driver assistance system for a motor vehicle as well as to a motor vehicle.
It is already known from the prior art to monitor an environmental region of a motor vehicle by capturing the environmental region by vehicle mounted cameras of a camera system, for example a surround view camera system, and displaying the images captured by the cameras on a display device. Therein, three-dimensional representations of the environmental region are increasingly also displayed on the display device from a so-called third person perspective. Such a third person perspective shows the environmental region of the motor vehicle as well as the motor vehicle itself in the form of a model of the motor vehicle from the view of a viewer external to vehicle, a so-called virtual camera. Such a third person view can be a top view, in which the viewer looks from above to the motor vehicle. This top view image can be displayed on the display device and shows an upper side of the modelled vehicle comprising the vehicle roof as well as the environmental region surrounding the motor vehicle. The top view image can be created by warping and merging several input images captured by the cameras of the camera system. The top view image gives the impression as if it had been captured by a real camera in a position of the virtual camera.
Therein, it may occur that the merged top view image comprises some disturbing signals like artificial flickering effects also referred to as aliasing effect. Those flickering effects degrade the image quality displayed to a driver and, hence, may disturb the driver.
It is the object of the present invention to provide a solution, how high quality top view images showing a motor vehicle and an environmental region of the motor vehicle can be generated for being displayed to a driver of the motor vehicle.
According to the invention, this object is solved by a method, an image processing device, a driver assistance system as well as a motor vehicle having the features according to the respective independent claims. Advantageous embodiments of the invention are the subject matter of the dependent claims, of the description as well as of the figures. In an embodiment of a method according to the invention for reducing disturbing signals in a top view image showing a motor vehicle and an environmental region of the motor vehicle, for interpolation of the top view image based on at least one input image captured by at least one vehicle-side camera, in particular, respective input coordinates of a pixel with subpixel accuracy within the at least one input image are determined for an output pixel to be interpolated of the top view image. For the output pixel to be interpolated, a respective filter strength can be determined as a function of the associated input coordinates and, depending on the filter strength, a respective filter for interpolating the output pixel can be determined.
In a preferred embodiment of a method according to the invention for reducing disturbing signals in a top view image showing a motor vehicle and an environmental region of the motor vehicle, for interpolation of the top view image based on at least one input image captured by at least one vehicle-side camera, respective input coordinates of a pixel with subpixel accuracy within the at least one input image are determined for an output pixel to be interpolated of the top view image. For the output pixel to be interpolated, a respective filter strength is determined as a function of the associated input coordinates and, depending on the filter strength, a respective filter for interpolating the output pixel is determined.
The method serves for generating high quality top view images showing the motor vehicle and the environmental region surrounding the motor vehicle that can be displayed to a driver of the motor vehicle in the form of a video sequence, in particular a real-time video. The top view images are created based on the input images or raw images captured by the at least one vehicle-side camera, e.g. by a vehicle-side image processing device.
In particular, several input images from different perspectives are used for generating the top view image. The images are particularly captured by an automotive multi-camera systems. For instance, the multi-camera system can be formed as a surround view camera system comprising four vehicle mounted cameras. The surround view camera system can comprise a front camera for capturing input images of the environmental region in front of the motor vehicle, a rear camera for capturing input images of the environmental region behind the motor vehicle and two side cameras for capturing input images of the environmental region besides the motor vehicle. Thereby, the cameras can comprise fisheye lenses to enlarge a field of view of the cameras. The input images distorted by the fisheye lenses can be dewarped and combined to the top view image during the same operation. Overlapping image areas taken from two different cameras and showing the same portion of the environmental area can be considered when determining the top view image
The input images as well as the top view images can be displayed on a display device, for example according to a view selected by the driver. By means of the display of the top view images on the vehicle-side display device the driver can be supported in
manoeuvring the motor vehicle. The surround view camera system and the display device form a camera-monitor-system (CMS) which can replace mirrors of the motor vehicle. Thus, by means of the CMS, a mirrorless car can be provided since the driver can capture the environmental region by looking at the display device.
The top view image which has a grid-like array of output pixels is generated from the at least one input image by scaling. When scaling raster graphics, image resolution is changed. That means that from a pre-given raster graphics, a new image is generated with a higher or lower number of pixels. This corresponds to a resampling operation, wherein the top view image is scaled from the input image by means of interpolation. For interpolating the top view image, for example by means of an interpolation engine of the image processing device, a colour value can be associated with each output pixel or output picture element of the top view image, which colour value is calculated from the input image's pixels situated nearby. Then a filter is applied, which determines which pixels of the input image are used for calculation, and how their colour values are weighted.
When rendering or generating the top view image, disturbing signals or aliasing effects are introduced, which appear as low frequency variations in the top view images. This flickering is annoying for the driver and shall therefore be reduced. The disturbing signals are particularly removed during interpolation of the output pixels for the top view image from the input image/the input images, approximately in real time. The output pixels to be interpolated are positioned in the top view image in grid form, in rows or columns, and, thereby, have a respective position or respective output coordinates in the top view image. The output coordinates are in particular integer, two-dimensional coordinates, each having a horizontal component or x-coordinate and a vertical component or y- coordinate.
For interpolating the output pixels, for example, for determining colour values of the output pixels, an input image window or an input image area with input pixels is identified within the at least on input image, based on which the output pixel can be calculated. The input image window describes a neighbourhood or vicinity of the output pixel in the at least one input image. For this purpose, the input position or the input coordinates of the pixel with subpixel accuracy (hereinafter referred to as subpixel) corresponding to the output pixel is determined within at least one input image. The input coordinates of the subpixel or partial image element usually have a non-integer value and describe the coordinates of a fictitious input pixel or input image element in the input image, which represents the output pixel at the associated input position. In particular, for interpolating the output pixel, real input pixels in the input image window, especially their colour values, are used.
After determination of the associated input coordinates and output coordinates a specific filter strength is determined for the, especially for each, output pixel as a function of the input coordinates of the associated subpixel. The output pixels are generated by the filter having the corresponding filter strength, for example by low pass filtering. In particular, a horizontal filter strength value corresponding with a respective horizontal input coordinate of the subpixel and a vertical filter strength value corresponding with a respective vertical input coordinate of the subpixel are determined.
The invention is based on the knowledge that the output pixels coordinates have a variable spacing within the input image. This spacing information is therefore used to adjust the strength of the interpolation filter. Advantageously, the filter strength can be a two-dimensional parameter having a horizontal value and a vertical value respectively to each output pixel that is interpolated. Thus, the plan view image can be separately filtered in horizontal and in vertical direction, ie in columns and in rows, with customized, specific filters. By means of the customized filter strengths, particularly high-quality top view images can be generated, which can be displayed to the driver on the display device.
It can be provided that the selection of the filter strength is performed for each output pixel and/or in regions on the output image areas having at least two output pixels in the horizontal and/or vertical direction. The interpolation is performed pixel-by-pixel for each output pixel. An output pixel-specific filter can be determined for each output pixel in the top view image for interpolating the respective output pixel. Therefore, it can be ensured that each output pixel is interpolated with an adapted filter optimized for that output pixel filter and, thus, particularly high quality and noise-reduced top view images can be produced. Also, a region-specific filter strength can be determined for certain output image areas, within which a strength of the disturbing signals hardly varies. Using area- based selection of the filter strength, computational time and computational complexity can be reduced in an advantageous manner, so that the top view images can be determined very quickly.
Moreover, a combination of pixel-based and area-based selection of filter strength is possible. This means that, for example, a pixel-specific horizontal filter strength value is associated with each output pixel and a region-specific vertical filter strength value is associated with a group of output pixels (or vice versa). Due to these mixing forms, a trade-off between quality and computational time can be provided during the
determination of the top view image.
Particularly preferably, a filter bank comprising at least two filter sets is predetermined, each filter set is assigned to a bandwidth corresponding with a respective filter strength and, for interpolating the respective output pixel, one of the filter sets is selected depending on the associated filter strength. In particular, each filter set comprises at least two filters with different filter phases corresponding to subpixel values from the coordinates, wherein one of the filters is selected as a function of the determined input coordinates of the subpixel. The filters of the filter bank are determined to be low pass filter curves. Thereby, a cut-off frequency of each low pass filter curve corresponds to a bandwidth
For instance, the filter bank can be stored in a storage unit of the image processing device and comprises the at least two groups of filters or filter sets. Each filter set can be assigned an index, over which the respective filter set with the associated bandwidth is addressable or identifiable. Here, a filter set is provided for each filter strength. In order to identify the filter sets, the filter strength is, for example, compared to the indices of the filter sets, and that filter set is selected, wherein a deviation between the index and the filter strength is minimal. This means that the filters are not determined for each interpolation operation specifically, but one of the filter sets is selected based on the determined filter strength. Therefore, computing time can be saved in an advantageous manner. Thus, the top view image can quickly and easily be determined based on the at least one input image.
Preferably, at least one lookup table readable for interpolating output pixels for the top view image is predetermined, in which the respective input coordinates of the subpixels and the corresponding filter strengths are associated with predetermined output coordinates of output pixels. For instance, the lookup table may also be stored in the storage unit of the image processing device to which the interpolation unit can access. Here, a respective input position of the subpixel as well as the filter strength is associated with an output position of an output pixel within the lookup table (LUT). This lookup table is pre-defined. This means that the assignments are, for example, determined based on at least one reference input image and a reference top view image and are stored in the lookup table. This lookup table can then be read during operation of the driver assistance system by identifying the input positions as well as the corresponding filter strengths in the lookup table for each output pixel to be generated. Both, a pixel based and a region- based filter strength can be determined. In particular, the information in the lookup table are used to quickly and easily determine the associated filter by, for example, identifying them in the deposited filter bank.
By storing the lookup table processing power can be saved, since the input positions of the subpixel and the filter strengths do not need to be determined during the real time interpolation of a top view image. In addition, this already stored lookup table can be read in real time so that, by means of the lookup table, and especially in combination with the predetermined filter bank, the corresponding filter parameters, ie the filter band and the filter phase, can quickly and easily be identified. Thus, high-quality plan view images with reduced noise can be determined in real time operation.
In a development of the invention, for determining the at least one lookup table, an output image window comprising the respective output coordinates of the output pixel to be interpolated and an associated input image window comprising the respective input coordinates of the subpixel in the at least one input image are determined. A spacing ratio is determined based on the input image window and the output image window
surrounding each pixel and the filter strength is determined based on the spacing ratio. The spacing ratio here describes a local sampling ratio between the output image and the input image. Therein, the samples as well as the output pixels in the output image are usually arranged regularly spaced from one another. The positions within the input image may appear not regularly spaced because of the geometrical transform (warping) applied. The spacing ratio is thus determined based on the input coordinates of the irregularly spaced subpixels which correspond to the output pixels within an output image window.
It proves advantageous if, for the output pixel to be interpolated, input coordinates which are located within the input image window are additionally deposited in the at least one lookup table. Based on these input pixels, for example, based on their colour values, the output pixel, for example, its colour value, can be calculated. Thus, by means of the lookup table, the input pixels within the at least one input image can be identified, based on which the corresponding output pixel is interpolated. In other words, a rule is stored in the lookup table, based on which a top view image can be determined from at least one input image captured by a camera. Thus, the method is particularly efficiently designed so that high-quality top view images can be generated, approximately in real time.
In one development of the invention, the lookup table is determined as a camera-specific lookup table for the at least one camera, wherein the preparation of the camera-specific lookup table is performed during a calibration process of the at least one camera. In other words, this means that for each camera its own lookup table is determined by means of which the input images captured by this camera can be processed. The lookup table can be determined every time the camera is recalibrated. Therefore, for example, an input image is captured and used as a reference input image. Thus, a reliable generation of high quality top view images can permanently be ensured.
It proves advantageous if a contrast enhancement and/or an edge enhancement for the at least one input image in case of the camera equipped with integrated enhancement functions are supressed or mitigated and the top view image is determined based on the at least one input image images without the contrast enhancement and/or edge enhancement. In other words, integrated enhancement functions or enhancement features of the cameras are turned off or reduced. Most cameras or camera systems include built-in image processing units with built-in contrast sharpening and/or edge enhancement. Even if the noise is not generated by the built-in sharpening functions, these sharpening functions exacerbate the strength of the disturbing signals in the top view image. For example, the integrated image processing unit of the camera can be controlled by the vehicle-side image processing device, whereupon the sharpening functions are not applied to the input images and unprocessed input images or raw images are provided to the image processing unit for rendering the top view image. By suppressing or at least attenuating the sharpening functions of the cameras and determining the plan view images on the basis of unprocessed raw images, the introduction of disturbing signals into the top view image can be reduced in a simple manner. For instance, the edge enhancement and/or contrast enhancement can be performed in the top view image after the interpolation of the top view image.
The invention also relates to an image processing device for a driver assistance system of a motor vehicle, which is adapted to perform a method according to the invention or an embodiment thereof. The image processing device may be integrated, for example, in an in-vehicle control unit. For instance, the image processing device can comprise an interpolation unit and a storage unit, wherein the interpolation unit can access the memory unit.
A driver assistance system for a motor vehicle according to the invention comprises at least one camera for capturing at least one input image and an image processing device according to the invention. In particular, the driver assistance system comprises at least four cameras forming a surround view camera system, wherein the image processing device is adapted to generate the output image based on the input images captured by the surround view camera system.
The invention also relates to a motor vehicle with a driver assistance system according to the invention. The vehicle is particularly formed as a passenger car. The cameras are in particular arranged on the motor vehicle in a distributed manner, so that the
environmental area surrounding the motor vehicle can be monitored.
The preferred embodiments presented with respect to the method according to the invention and the advantages thereof correspondingly apply to the driver assistance system according to the invention as well as to the motor vehicle according to the invention.
With indications of "in front of", "behind", "besides", "above", "left", "right", "besides", etc., the positions and orientations given with an observer standing in front of the motor vehicle and looking in a direction of a longitudinal axis of the motor vehicle are specified.
Further functions of the invention are apparent from the claims, the figures and the description of figures. The functions and function combinations mentioned above in the description as well as the functions and function combinations mentioned below in the description of figures and/or shown in the figures alone are usable not only in the respectively specified combination, but also in other combinations or alone without departing from the scope of the invention. Thus, implementations are also to be considered as encompassed and disclosed by the invention, which are not explicitly shown in the figures and explained, but arise from and can be generated by separated function combinations from the explained implementations. Implementations and function combinations are also to be considered as disclosed, which thus do not have all of the functions of an originally formulated independent claim. Moreover, implementations and function combinations are to be considered as disclosed, in particular by the implementations set out above, which extend beyond or deviate from the function combinations set out in the relations of the claims.
Now, the invention is explained in more detail based on preferred embodiments as well as with reference to the attached drawings.
These show in:
Fig. 1 a schematic representation of an embodiment of a motor vehicle according to the invention;
Fig. 2 a schematic representation of a top view image with disturbing signals;
Fig. 3 a schematic representation of a general image pipeline;
Fig. 4 a schematic representation of an interpolation of an output image area based on an input image area;
Fig. 5 a schematic representation of the operation of an embodiment of an image processing device according to the invention; and
Fig. 6 a schematic representation of a top view image with boundary artefacts.
In the figures, identical as well as functionally identical elements are provided with the same reference characters.
Fig. 1 shows a motor vehicle 1 , which is formed as a passenger car in the present case. The motor vehicle 1 includes a driver assistance system 2, which can assist a driver of the motor vehicle 1 in driving the motor vehicle 1 . The driver assistance system 2 comprises a surround view camera system 3 for monitoring an environmental region 4 of the motor vehicle 1 . Herein, the driver assistance system 2 comprises four vehicle mounted cameras 5a, 5b, 5c, 5d. A first camera 5a is attached to a front area 6 of the motor vehicle 1 and serves for capturing input images showing the environmental region 4 in front of the motor vehicle 1 . A second camera 5b is attached to a rear area 7 of the motor vehicle 1 and serves for capturing input images showing the environmental region 4 behind the motor vehicle 1 . A third camera 5c is attached to a left side 8 of the motor vehicle 1 and serves for capturing input images showing the environmental region 4 to the left besides the motor vehicle 1 and a fourth camera 5d is attached to a right side 9 of the motor vehicle 1 and serves for capturing input images showing the environmental region 4 to the right besides the motor vehicle 1 . The cameras 5a, 5b, 5c, 5d can comprise fisheye lenses in order to enlarge a field of view of the cameras 5a, 5b, 5c, 5d. The input images or input video frames captured by the cameras 5a, 5b, 5c, 5d can be displayed on a display device 10 of the driver assistance system 2 in the form of a video.
The input images captured by the cameras 5a, 5b, 5c, 5d of the surround view camera system 3 can be warped and merged, e.g. by an image processing device 1 1 of the driver assistance system 2, to determine a top view image 12 of the motor vehicle 1 and the environmental region 4. Fig. 2 shows a top view image 12 afflicted with disturbing signals 13 and generated from input images of the surround view camera system 3. The top view images 12 or top view video frames can be displayed on the display device 10. In the top view image 12, the environmental region 4 is shown from the perspective of a viewer above the motor vehicle 1 looking down to the motor vehicle 1 . The top view image 12 is suggestive of being captured by a camera, a so-called virtual camera, positioned above the motor vehicle 1 . Since the motor vehicle 1 itself cannot be captured by the cameras 5a, 5b, 5c, 5d of the surround view camera system 3 a model 14 of the motor vehicle 1 is inserted into the top view image 12.
When creating or scaling the top view image 12 partial environmental areas of the environmental area 4 close to the motor vehicle 1 , especially near of side mirrors of the motor vehicle 1 , tend to be highly downsized. In particular, image areas 15 within the top view image 12, showing this partial environmental areas, are afflicted with the disturbing signals 13 or aliasing effects, which are visible in the top view image 12 in the form of flicker effects. Partial environmental areas far from the motor vehicle 1 do not show this problem and appear too soft or blurred in the plan view image 12 instead. The overall impression of the displayed top view image 12 is uncomfortable for the driver of the motor vehicle 1 and non-homogeneous. The aliasing effect is annoying, when the motor vehicle 1 is moving, as low frequency moving waves (aliasing) combine with high-frequency texture details of a road 16 of the motor vehicle 1 .
This problem typically occurs in every digital image processing device, wherein the processing require downsampling operations, and is usually visible when a resampling varies across the image. Then, for example, some areas are more decimated or downsampled than other areas of the image. For example, this is the case in applications of geometric corrections, for example, for removing fish-eye distortions, or, more generally, in image distortion operations. Fig. 3 shows a schematic representation of a general image pipeline 17 or video pipeline represented by a set of components 18, 19, 20, 21 , 22. By means of the video pipeline 17 a customer view image 23, like the top view image 12, can be generated based on input images or raw images 24, like the raw images captured by the cameras 5a, 5b, 5c, 5d of the surround view camera system 3. Therein, light is projected from the environmental region 4 of the cameras 5a, 5b, 5c, 5d via lenses 18 to an image sensor unit 19 of the cameras 5a, 5b, 5c, 5d comprising an imager and a microprocessor, e.g. a companion chip. Based on image quality settings 20 of the cameras 5a, 5b, 5c, 5d the image sensor unit 19 generates the raw images 24 as an output. Based on the raw images 24 an image processing device 21 , e.g. the vehicle-side image processing device 1 1 , generates a virtual view, for example the top view. Based on virtual view settings and calibration output 22 the customer view image 23 can be determined for display on the display device 10.
According to the Nyquist-Shannon theorem, the aliasing problem can be solved by limiting all frequencies in a signal, here a two-dimensional image signal, to half of the sampling frequency. This frequency limited to half of the sampling frequency is also called Nyquist frequency. Usually this problem is solved by low pass filtering the input images, which may be optimal for constant resampling cases, but not for variable resampling cases, since the low pass filter does not consider local variations. In order to reduce the disturbing signals 13, a method for creating top view images 12 with reduced noise 13 is presented wherein a real-time processing as well as a limited computational complexity required in automotive applications can additionally be met.
For instance, the method for reducing the disturbing signals 13 can be performed by the vehicle-side image processing device 1 1 when determining the top view image 12 from the input images of the cameras 5a, 5b, 5c, 5d. Thereto, the image processing device 1 1 , for example, can comprise an interpolation engine 29 (see Fig. 5) which is adapted to interpolate an output pixel P (see Fig. 4) or an output picture element from any non- integer input coordinates Cx(P), Cy(P). The interpolation engine 29 or the interpolator can determine a two-dimensional input position Cx(P), Cy(P) or 2D input coordinates and a two-dimensional local strength for each output pixel P to be generated. The 2D input position Cx(P), Cy(P) (in sub-pixel resolution) is a position of an output pixel P to be interpolated within the input image (or a partial area of the input image in the case of a region-based approach). The two-dimensional local strength or 2D filter strength corresponds to an index indexO, index2 (see Fig. 5) of a low pass filter applied for interpolation of the output pixel P. Each index indexO, index2 corresponds with an appropriate filter band BandO, Bandl , Band2, Band3 of the low pass filter. The 2D filter strength is in particular a two-dimensional index with a horizontal filter strength and a vertical filter strength. The horizontal and the vertical filter strength can be determined by a local estimation of the resampling as a function of pixels P' in a neighbourhood of the interpolated output pixel P.
As an approximation and simplification, the filter strengths can be determined as two- dimensional image area-specific filter strengths. Therefore, the interpolation can be performed in areas, if the resampling ratio hardly varies within the processed image area. Preferably, the filter strengths are, however, determined as the local pixel-specific filter strengths, so that the filter strength is adapted pixel-by-pixel. Thereby, an optimal adaptation or selection of the bandwidth of the filter is guaranteed, whereas, at the same time, the quality of the top view image 12 is maximized and boundary artefacts 32 (see Fig. 6) can be prevented at edges 31 or transitions of the image regions 15 resulting from a too coarse filter change.
For input images with a 4:2:2 format, the filter strength is provided for a group of two output pixels P, respectively, whereby a specific filter strength can be selected for a YUYV output image. YUYV describes a colour model which uses two components, luminance "Y" and chrominance, which in turn consists of the two sub-components "U" and "V" for representation of colour information. It can be provided that the interpolation process is applied only to the luminance samples. A chrominance can be determined, for example by averaging of the input chrominance considering the input luminance. The averaging can be carried out both in the horizontal and in the vertical direction.
For resampling, ie for example for oversampling, the interpolation unit 29 is adapted to read a filter bank 30 with predefined low pass curves. By varying the filter bandwidth an adjustment near the required filter strength value is possible. A number of filter curves depends on the overall resampling region. The number of filters is for example from four to eight filters. An oversampling factor depends on the subpixel accuracy, which is provided by the interpolation engine 29. A subpixel accuracy of ¼ pixel, ie a ¼-pixel accuracy, at least requires a fourfold oversampling. Therefore, the interpolator 29 requires appropriate, in particular, for horizontal and vertical direction x, y, separate, filter curves that provide different filter strengths for both the horizontal and the vertical direction x, y. Ideally, a number of delay elements ("taps") of the low pass filter is dimensioned dependent on the narrowest filter band required or the narrowest bandwidth. For example, the filters can be formed, in consideration of the real-time processing required, with four delay elements, wherein an interpolation in this case is equivalent to a 4x4 interpolation filtering operation. Due to the limited bandwidth capacity (for example, 0:25 with four taps) it can be possible to downsample or pre-decimate the input images in advance to extend the overall filter range. A pre-decimation of ¼ extends the resampling range to 1 /16. In this case, the input coordinates Cx(P), Cy(P) and the subpixel SP are appropriately recalculated to be adapted to the new, pre-decimated input image.
For the interpolation of the output pixel P, a resampling factor is calculated on pixel basis, which corresponds to a local ratio of an output distance to an input distance. Since the output samples are by definition regularly spaced in the top view image 12, ie the top view image 12, comprising one output pixel P, P per output sample, the output distance can be normalized to "1 ". Therefore, the resampling factor can be determined as 1 /input distance.
In Fig. 4 an input image area 24 of an input image and an output image area 25 of a top view image 12 with a plurality of output pixels P, P' is shown. Initially, a local, centred output image window 26 with a width Wx in the horizontal direction x and a height Wy in vertical direction y is defined around the output pixel P to be interpolated. In other words, the output pixel P to be interpolated is located in the centre of the output image window 26. In the present case, the output image window 26 is square with equal width Wx and height Wy, for example, Wx = Wy = W = 4 (pixels). The output pixels P+Wx/2, P-wx/2, P+wy/2, P+wy/2 are located on edges of the output image window 26 at a distance of two output pixels P 'along the horizontal and vertical directions x, y from the output pixel P to be interpolated.
Based on the output pixels P, P', a rectangular input image window 27 with the dimension values or dimension parameters L, R, B, T can be defined. The input image includes a plurality of input samples 28, which are indicated by crosses in the input image area 24. The input coordinates Cx(P), Cy(P) of the output pixel P are located within the input image window 27, wherein the input coordinates Cx(P), Cy(P) are the coordinates of a so- called subpixel SP or partial picture element. Moreover, the input coordinates Cx(P'), Cy(P') of the output pixels P', in particular the input coordinates Cx(P+Wx/2), Cy(P+Wx/2) of the output pixel P+Wx/2, the input coordinates Cx(P-Wx/2), Cy(P-Wx/2) of the output pixel P. wx/2, the input coordinates Cx(P+Wy/2), Cy(P+Wy/2) of the output pixel P+Wy/2 and the input coordinates Cx(P-Wy/2), Cy(P-Wy/2) of the output pixel P-Wy/2 on the edges of the input image window 27, are shown
The parameters L, R, B, T are determined as follows:
L = min (Cx(P+Wx/2) , Cx(P-Wx/2), Cx(P+Wy/2) , Cx(P+Wy/2)) ,
R = maX (Cx(P+Wx/2) , Cx(P-Wx/2), Cx(P+Wy/2) , Cx(P+Wy/2)) ,
T = min (Cy(P+Wx/2) , Cy{P.Wy 2), Cy(P+Wy/2) , Cy(P+Wy/2)) ,
B = max (Cy(P+Wx/2), Cy(P-Wx/2) , Cy(P+Wy/2), Cy(P+Wy/2)).
For a given output pixel P the local resampling factor R is, for example, defined as:
R(x) = Wx / (R - L) or 1 / R(x) = (R - L) » 2 (mit Wx = 4)
R(y) = Wy / (B - T) or 1 / R(y) = (B - T) » 2 (mit Wy = 4).
"»" represents a bitwise operator.
Using an image area based interpolation, the width Wx and the height Wy correspond to dimensions of the image area with the respective pixels to be interpolated. In this case, the output pixel is not located in the centre of the output image window 26 determined for this image area.
Based on the resampling factor, the filter strength can be determined. As mentioned above, the filter strength can comprise two components which correspond to filter bandwidths for the horizontal and vertical directions x, y, respectively. Due to the Nyquist- Shannon theorem, the filter is selected such that the bandwidth is limited in accordance with bandwidth <= R. If
- R >= 1 , the filter bandwidth = 1 (full bandwidth interpolation),
- R < 1 , the filter bandwidth is the closest available value to R.
When the resampling factor is R = 0.34, for example, the bandwidth to be selected from a given filter set {1 .0, 0.7, 0.5, 0.36, 0.25} is the filter 0.36 (the third index in the list). For simplicity, the division 1 /R can be prevented on implementation-side, so that the filter strength is determined from (R - L) and (T - B). In order to control the interpolation at least one lookup table LUT1 , LUT2 (see Fig. 5) can be determined in advance and can, for example, be saved in the image processing device 1 1 . The lookup table LUT1 is used for the pixel-based interpolation and the lookup table LUT2 is used for the image area-based interpolation. The lookup table LUT1 includes input coordinates XB, YB of the input image pixels used for interpolation of an output pixel P, the input coordinates Cx(P), Cy(P) of the sub-pixel SP, which are also referred to as interpolation phases Ph1 , Ph2, and the filter strengths, which are characterized by the respective indices indexO index2. As already indicated, only one filter strength is provided for two output samples. The preparation of the lookup tables LUT1 , LUT2 can be performed at certain times, in particular periodically, for example at a calibration of the camera 5a, 5b, 5c, 5d.
During a real-time generation of the top view images 12, at least one of the lookup tables LUT1 , LUT2 is read by the interpolation unit 29 and the coordinates XB, YB of the input pixels or input image data to be used are identified. Respective filter strengths indexO, index2 and phases Ph2, Ph1 are associated with these input coordinates XB, YB, based on which the corresponding filter bands BandO, Bandl , Band2, Band3 with the
corresponding phases PhO, Ph1 , Ph2, Ph3 can be extracted from the filter bank 30 by the interpolation unit 29. Thereafter, the corresponding coefficients Px, Py can be loaded for the interpolation of the output pixel P. In order to save memory space within the lookup tables LUT1 , LUT2, alternatively or in combination with the local pixel-based index in the lookup table LUT1 , a range based index can be stored in the lookup table LUT2.
In Fig. 6, a top view image 12 is shown which has been interpolated in image regions. Here, boundary artefacts 32 are visible at the edges 31 of the image regions 15 which result from a sudden change in the applied filter strength or the applied filter bandwidth. In order to prevent this boundary artefacts 32, the interpolation is performed pixel-by-pixel so that a top view image 12 can be generated with at least reduced disturbing signals 13 and no boundary artefacts 32.
In summary, a method is provided, which reduces the real-time complexity by adjusting the local Nyquist frequency, which is applied to each output pixel P. In addition, indices indexO, index2 corresponding with certain filter bands are deposited in the lookup tables LUT1 , LUT2 so that the low pass filtering can be varied in real-time. This is in particular separately provided both for the horizontal and for the vertical direction x, y. The method comprises the advantage that aliasing artefacts can be reduced, wherein an overall sharpness of the top view image 12 remains.

Claims

Claims
1 . Method for reducing disturbing signals (13) in a top view image showing (12) a
motor vehicle (1 ) and an environmental region (4) of the motor vehicle (1 ), wherein for interpolation of the top view image (12) based on at least one input image captured by at least one vehicle-side camera (5a, 5b, 5c, 5d)
- for an output pixel (P) to be interpolated of the top view image (12), respective input coordinates (Cx(P), Cy(P)) of a pixel (SP) with subpixel accuracy within the at least one input image are determined,
- for the output pixel (P) to be interpolated a respective filter strength is determined as a function of the associated input coordinates (Cx(P), Cy(P)), and
- depending on the filter strength a respective filter for interpolating the output pixel (P) is determined.
2. Method according to claim 1 ,
characterized in that
a filter bank (30) comprising at least two filter sets is predetermined, each filter set is assigned to a bandwidth (BandO, Bandl , Band2, Band3) corresponding with a respective filter strength and, for interpolating the respective output pixel (P), one of the filter sets is selected depending on the associated filter strength.
3. Method according to claim 2,
characterized in that
each filter set comprises at least two filters with different filter phases (PhO, Ph1 , Ph2, Ph3) corresponding to respective input coordinates (Cx (P), Cy (P)) of the pixel (SP) with subpixel accuracy, wherein one of the filters is selected as a function of the determined input coordinates (Cx(P), Cy(P)) of the pixel (SP) with subpixel accuracy.
4. Method according to claim 2 or 3,
characterized in that
the filters of the filter bank (30) are determined to be low pass filter curves.
5. Method according to any one of the preceding claims,
characterized in that
at least one lookup table (LUT 1 , LUT 2) readable for interpolating output pixels (P) for the top view image (12) is predetermined, in which the respective input coordinates (Cx(P), Cy(P)) of the pixels (SP) with subpixel accuracy and the corresponding filter strengths are associated with predetermined output coordinates of the output pixels (P).
6. Method according to claim 5,
characterized in that
for determining the at least one lookup table (LUT1 , LUT2), an output image window (26) comprising the respective output coordinates of the output pixel (P) to be interpolated and an input image window (27) comprising the respective input coordinates (Cx(P), Cy(P)) of the pixel (SP) with subpixel accuracy within the at least one input image are determined, a spacing ratio is determined based on the input image window (27) and the output image window (27), and the filter strength is determined based on the spacing ratio.
7. Method according to claim 6,
characterized in that
for the output pixel (P) to be interpolated, input coordinates (ΧΒ,ΥΒ) which are located within the input image window (27) are additionally deposited in the at least one lookup table (LUT1 , LUT2).
8. Method according to claim 6 or 7,
characterized in that
the at least one lookup table (LUT1 , LUT2) is determined as a camera-specific lookup table (LUT1 , LUT2) for the at least one camera (5a, 5b, 5c, 5d), wherein the preparation of the camera-specific lookup table (LUT1 , LUT2) is performed during a calibration process of at the least one camera (5a, 5b, 5c, 5d).
9. Method according one of the preceding claims,
characterized in that
a horizontal filter strength value corresponding with a respective horizontal input coordinate (Cx(P)) of the pixel (SP) with subpixel accuracy and a vertical filter strength value corresponding with respective vertical input coordinate (Cy(P)) of the pixel (SP) with subpixel accuracy are determined.
10. Method according to any one of the preceding claims,
characterized in that
the filter strength selection is performed for each output pixel (P) and/or in regions for output image areas having at least two output pixels (P) in the horizontal and/or vertical direction (x, y).
1 1 . Method according to any one of the preceding claims,
characterized in that
a contrast enhancement and/or an edge enhancement for the at least one input image in case of the camera (5a, 5b, 5c, 5d) equipped with integrated enhancement functions are supressed or mitigated and the top view image (12) is determined based on the at least one input image images without the contrast enhancement and/or edge enhancement.
12. Image processing device (1 1 ) for a driver assistance system (2) of a motor vehicle
(I ) which is adapted to perform a method of any one of the preceding claims.
13. Driver assistance system (2) for a motor vehicle (1 ) comprising at least one camera (5a, 5b, 5c, 5d) for capturing at least one input image and an image processing device (1 1 ) according to claim 12.
14. Driver assistance system (2) according to claim 13,
characterized in that
the driver assistance system (2) comprises at least four cameras (5a, 5b, 5c, 5d) forming a surround view camera system (3), wherein the image processing device
(I I ) is adapted to generate the top view image (12) based on the input images captured by the surround view camera system (3).
15. Motor vehicle (1 ) comprising a driver assistance system according to claim 13 or 14.
PCT/EP2017/074750 2016-09-29 2017-09-29 Method for reducing disturbing signals in a top view image of a motor vehicle, computing device, driver assistance system as well as motor vehicle Ceased WO2018060409A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
DE102016118465.9A DE102016118465A1 (en) 2016-09-29 2016-09-29 Method for reducing interference signals in a top view image of a motor vehicle, image processing device, driver assistance system and motor vehicle
DE102016118465.9 2016-09-29

Publications (1)

Publication Number Publication Date
WO2018060409A1 true WO2018060409A1 (en) 2018-04-05

Family

ID=60138331

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2017/074750 Ceased WO2018060409A1 (en) 2016-09-29 2017-09-29 Method for reducing disturbing signals in a top view image of a motor vehicle, computing device, driver assistance system as well as motor vehicle

Country Status (2)

Country Link
DE (1) DE102016118465A1 (en)
WO (1) WO2018060409A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11508043B2 (en) * 2020-06-18 2022-11-22 Connaught Electronics Ltd. Method and apparatus for enhanced anti-aliasing filtering on a GPU

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112906515B (en) * 2021-02-03 2024-07-19 珠海研果科技有限公司 In-vehicle abnormal behavior identification method, system, electronic equipment and storage medium

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120257009A1 (en) * 2009-12-14 2012-10-11 Etu System, Ltd. Image Processing Device and Method for Matching Images Obtained from a Plurality of Wide-Angle Cameras
US9317895B1 (en) * 2012-08-09 2016-04-19 Xilinx, Inc. Non-linear image mapping

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100510685B1 (en) 2003-04-22 2005-08-31 엘지전자 주식회사 Apparatus for anti-aliasing in backward mapping algorithm
US7619639B1 (en) 2005-09-12 2009-11-17 Nvidia Corporation Adaptive scaling using a programmable video engine
US9215440B2 (en) 2012-10-17 2015-12-15 Disney Enterprises, Inc. Efficient EWA video rendering

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120257009A1 (en) * 2009-12-14 2012-10-11 Etu System, Ltd. Image Processing Device and Method for Matching Images Obtained from a Plurality of Wide-Angle Cameras
US9317895B1 (en) * 2012-08-09 2016-04-19 Xilinx, Inc. Non-linear image mapping

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
KIM DONGGYUN ET AL: "Lens distortion correction and enhancement based on local self-similarity for high-quality consumer imaging systems", IEEE TRANSACTIONS ON CONSUMER ELECTRONICS, IEEE SERVICE CENTER, NEW YORK, NY, US, vol. 60, no. 1, 1 February 2014 (2014-02-01), pages 18 - 22, XP011544334, ISSN: 0098-3063, [retrieved on 20140330], DOI: 10.1109/TCE.2014.6780920 *
PETER HANSEN ET AL: "Scale-Invariant Features on the Sphere", ICCV 2007. IEEE 11TH INTERNATIONAL CONFERENCE ON COMPUTER VISION, 2007, IEEE, 1 October 2007 (2007-10-01), pages 1 - 8, XP031194382, ISBN: 978-1-4244-1630-1 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11508043B2 (en) * 2020-06-18 2022-11-22 Connaught Electronics Ltd. Method and apparatus for enhanced anti-aliasing filtering on a GPU

Also Published As

Publication number Publication date
DE102016118465A1 (en) 2018-03-29

Similar Documents

Publication Publication Date Title
KR102327762B1 (en) A method of generating output images showing a car and its environment areas in a predefined target view, a camera system and a car
US7529424B2 (en) Correction of optical distortion by image processing
US8233744B2 (en) Image enlargement apparatus, method, integrated circuit, and program
KR101077584B1 (en) Apparatus and method for processing images obtained by a plurality of cameras
US9369693B2 (en) Stereoscopic imaging device and shading correction method
US9317895B1 (en) Non-linear image mapping
CN110517206B (en) Method and device for eliminating color moire
EP3035664B1 (en) Method for processing a video stream
JP5195841B2 (en) On-vehicle camera device and vehicle
KR20090050223A (en) Image processing device with distortion image correction
US10068314B2 (en) Image processing apparatus, image processing method, and computer-readable storage medium
JP2019185817A (en) Image processing device, image processing method, and computer program
CN114928698A (en) Image distortion correction device and method
US20210398256A1 (en) Method and apparatus for enhanced anti-aliasing filtering on a gpu
JP2011044891A (en) Image processor and method for the same
JP4862321B2 (en) In-vehicle camera device
US11145093B2 (en) Semiconductor device, image processing system, image processing method and computer readable storage medium
DE102016118465A1 (en) Method for reducing interference signals in a top view image of a motor vehicle, image processing device, driver assistance system and motor vehicle
KR101230909B1 (en) Apparatus and method for processing wide angle image
JP2008060814A (en) Imaging device
CN107979715B (en) Image pickup apparatus
Weerasinghe et al. Digital zoom camera with image sharpening and suppression
CN103154999B (en) Image signal processing apparatus, image-signal processing method
KR20110088680A (en) An image processing apparatus having a correction function for a composite image obtained by synthesizing a plurality of images
JPH0795595A (en) Color imaging device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17786833

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17786833

Country of ref document: EP

Kind code of ref document: A1