WO2018087348A1

WO2018087348A1 - Method for determining a composite image of an environmental region of a motor vehicle with adjusting brightness and/or colour, camera system as well as motor vehicle

Info

Publication number: WO2018087348A1
Application number: PCT/EP2017/078985
Authority: WO
Inventors: Vladimir Zlokolica; Brian Michael Thomas DEEGAN; Patrick Eoghan Denny
Original assignee: Connaught Electronics Ltd
Current assignee: Connaught Electronics Ltd
Priority date: 2016-11-14
Filing date: 2017-11-13
Publication date: 2018-05-17
Anticipated expiration: 2019-05-14
Also published as: DE102016121755A1

Abstract

The invention relates to a method for determining a composite image (14) of an environmental region (9) of a motor vehicle (1), wherein at least two input images (FV, RV, ML, MR) from respective cameras (4a, 4b, 4c, 4d) of the motor vehicle (1) are received and the composite image (14) is determined from the input images (FV, RV, ML, MR), wherein for determining the composite image (14), a brightness and/or colour of at least one of the input images (FV, RV, ML, MR) is adjusted, wherein histograms are determined, which describe a luminance and/or a chrominance of the respective input images (FV, RV, ML, MR) and the brightness and/or colour of the at least one input image (FV, RV, ML, MR) is adjusted on the basis of histograms.

Description

Method for determining a composite image of an environmental region of a motor vehicle with adjusting brightness and/or colour, camera system as well as motor vehicle

The invention relates to a method for determining a composite image of an environmental region of a motor vehicle. In addition, the invention relates to an image processing device, to a camera system for a motor vehicle as well as to a motor vehicle.

It is already known from the prior art to monitor an environmental region of a motor vehicle by capturing the environmental region by vehicle mounted cameras of a camera system, for example a surround view camera system, and displaying the images captured by the cameras on a display device. Therein, three-dimensional representations of the environmental region are increasingly also displayed on the display device from a so-called third person perspective. Such a third person perspective shows the environmental region of the motor vehicle as well as the motor vehicle itself from the view of a viewer external to vehicle, a so-called virtual camera. Such a third person view can be a top view, in which the viewer looks from above to the motor vehicle. This top view image can be displayed on the display device and shows an upper side of the vehicle comprising the vehicle roof as well as the environmental region surrounding the motor vehicle. The top view image can be created by warping and merging several input images captured by the cameras of the camera system. The top view image gives the impression as if it had been captured by a real camera in a position of the virtual camera.

A camera system for determining a composite image, in particular a top view image, comprises a plurality of cameras. The cameras are typically mounted in the front of the motor vehicle, in the rear of the motor vehicle and in the areas of the wing mirrors of the motor vehicle. In many implementations, each camera has independent image signal processing chain. Therefore, the cameras can have a different exposure or gain control, a different white balance or the like. As each camera is facing in different directions and is seeing different areas of the vehicle environment, the brightness and colour levels will be slightly different for each camera. These slight differences in colour and brightness negatively affect the illusion of a virtual camera.

US 6 672 796 B2 describes an image pickup system for concurrently picking up several images and merging those into one for display. The system comprises determination means for determining whether the image quality of the merged image is consistent on said display device. The system further comprises image quality control means for controlling the image quality parameter set in each of the image pickup devices so that the image quality of the merged image becomes consistent on said display device.

It is an object of the present invention to provide a solution, how a composite image based on input images by the respective cameras of a motor vehicle may be determined easier and more reliable.

According to the invention, this object is solved by a method, by a camera system as well as by a motor vehicle having the features according to the respective independent claims. Advantageous developments of the present invention are the subject matter of the dependent claims.

According to an embodiment of a method for determining a composite image of an environmental region of a motor vehicle, at least two input images are preferably received from respective cameras of the motor vehicle. Further, from the input images, preferably, the composite image is determined, wherein for determining the composite image preferably a brightness and/or colour of at least one of the input images is adjusted.

Moreover, it is preferred that histograms, which describe a luminance and/or a

chrominance of the respective input images, are determined and the brightness and/or colour of the at least one input image is adjusted based on the histograms.

A method according to the invention is used to determine a composite image of an environmental region of a motor vehicle, wherein at least two input images are received from respective cameras of the vehicle and from the input images, the composite image is determined, wherein for determining the composite image a brightness and/or a colour of at least one of the input images is adjusted. Further, histograms which describe a luminance and/or a chrominance of the respective input images are determined and the brightness and/or colour of the at least one input image is adjusted based on the histograms.

By means of the method, a composite image should be provided, showing an

environmental region or an environment of the motor vehicle. This composite image is generated from a plurality of input images, which are provided by means of respective cameras of a camera system of the motor vehicle. In particular, as the composite image a top view image is provided showing the motor vehicle from a bird's eye view. The composite image may further comprise an image of the motor vehicle, which represents the motor vehicle. This composite image or the top view image can be displayed to the driver of the motor vehicle on a display device. In this way, for example, the

environmental region around the motor vehicle can be represented the driver. Thus, the driver can be supported while driving the motor vehicle. The respective input images provided by the cameras can be transmitted to an image processing unit, by means of which then the composite image can be determined. The respective input images can preferably be high dynamic range images (HDRI). In the provision of the composite image, it is necessary that the brightness and/or colour of the input images is matched.

According to an essential aspect of the present invention, it is now provided that histograms describing a luminance and/or chrominance of the respective input image of each of the input images are determined. On the basis of the histograms, which describe the luminance, then an adjustment of the brightness of the input images can be performed. On the basis of the histograms which describe the chrominance, then an adjustment of the colour of the respective input images can be performed. The adjustment of brightness and/or colour represents a post-processing of the input images in which the respective pixels of the input images can be rendered. Therefore, a simple and reliable adjustment of brightness and/or colour of the individual images can be performed. The use of histograms provides the advantage that the computational effort in the adjustment of brightness and/or colour can be significantly reduced. The respective histograms, which are determined by the input images, represent quasi one-dimensional data. Compared to processes in which two-dimensional image data of the individual images are matched with each other, thus the effectiveness can be significantly increased.

Preferably, on the basis of the histograms respective correction values for the

components Y, U and/or V are determined in a YUV colour space of the at least one input image and an input image, which is received at a later instance in time, will be corrected by the respective correction values. For each of the input images histograms can be determined, which describe the Y, U and/or V component of each input image. On the basis of a comparison of the respective components Y, U and/or V of the respective input images the correction values can then be determined. It may further be provided that a tone mapping is performed on the basis of the correction values. In particular, it is envisaged that appropriate tone mapping tables are determined based on the correction values. The correction values or the tone mapping tables are determined at a first point in time. At a second point in time, subsequent to the first point in time, then the respective input images are corrected based on the correction values. Thus, an adjustment of brightness and/or colour in the successive input images can be performed.

According to one embodiment at least two regions of interest are determined in the input images and the histograms are determined for the respective regions of interest. The respective input images can thus be divided into regions of interest. This makes it possible that the brightness and/or colour of the respective regions of interest can be adapted to each other. In this way, histograms can be determined for each region of interest, which describe the brightness and/or colour in the region of interest. Thus, it is possible to adapt adjacent regions of interest with respect to the brightness and/or colour to each other in a simple manner.

Preferably, it is provided that the histograms are determined dependent of input parameters of the respective cameras. For example, the histograms can be adapted based on a gamma correction of the respective cameras. Thus, the histograms can be adapted to the specific camera settings. This enables a reliable adjustment of brightness and/or colour of the input images to each other.

It is further advantageous if an overlapping area is determined, in which the at least two input images overlap at least in portions when determining the composite image and the histograms are determined for the respective input images depending on the overlapping area. The overlapping areas describe the areas of the respective input images, which represent the same area in the environment of the motor vehicle. It is in particular provided that the histogram for each input image is segmented in dependence of the overlapping area and the segments of the histograms of the respective input images, which belong to the overlapping area, are weighted. The respective histograms of the input images can thus be segmented or divided. Thus, the segments of the respective histograms of the input images that belong to the overlapping area, are adapted to each other. This can be done by determining an appropriate weighting value for the segments of the histograms. It is important to take into account that the brightness and/or colour of the input images is adjusted so that it matches in the overlap region.

According to a further embodiment, a plurality of input images is received, on the basis of the histograms of luminance values are determined for each of the input images and a respective difference between the luminance values is determined. Based on the histograms, which are determined for the respective input images, then estimates of the luminance of the respective input images can be determined. Here, in particular the histogram of the component Y of the respective input images can be determined. The respective luminance values, which have been determined based on the histograms for the respective input images, can then be compared. On the basis of the differences of the luminance values the adjustment of the brightness of the input images can then be effected. It is particularly provided that the input images are adjusted only minimally. If during the determination of the composite image the adaptation of single input images is too large, this can lead to unwanted side effects. For example, if an input image has a high brightness value but a low average brightness and this input image should be adapted to the other input images, the average brightness can be increased. This may for example lead to a saturation of brighter areas of the input image. Further, the texture and colour adjustments may be negatively affected.

Furthermore, it is advantageous if one of the input images, based on which the adjustment of the brightness of the input images is started, and/or an order of the input images, along which the adjustment of the brightness of the input images is performed, is determined based on the respective differences. As explained above, differences between the luminance values of the respective input images can be determined. For example, the input image having the largest difference of the luminance value with respect to the other input images can be used as so-called master image or master camera. Starting from this master camera the adaption of the brightness can then be started. For adjusting the brightness of the input images, a predetermined direction or order can be provided. If as the composite image a top view image is provided, the order can take place along a circle or a closed path. Again, the direction can be determined depending on the respective differences between the luminance values. Basically, the colour matching can also be carried out in such a way that it is started from any of the input images. This allows reliable adjustment of the colour.

According to a further embodiment, the respective differences are determined for two different orders along the input images for each of the input images and a correction value for the at least one input image is determined based on the respective differences. The brightness correction can determined for the case that each camera is a master camera. As already explained, an input image can be selected, starting from which the determination of the respective differences is started. In particular, it is envisaged that the determination of the respective differences is determined from each of the input images. For each input image, the differences between the luminance values can be determined in two different directions. The respective differences determined for each direction can then be weighted. If four input images are received, the brightness correction can be calculated for each of the input images clockwise and counter clockwise. In this case, eight different brightness correction values for each master camera or input image can be determined. After that, a weighted average of the brightness correction values can be calculated to determine the final correction value. The brightness correction values for the respective input images can thus be precisely determined.

According to another embodiment, at least four input images are received and as the composite image a top view image is determined. This can for example be done using a camera system, which has at least four cameras. Here, a camera can be arranged at a front of the motor vehicle, a camera can be arranged in a rear of the motor vehicle, and two cameras can be arranged in portions of the side mirror of the motor vehicle. These four input images can then be composed to the top view image, in which case an image of the motor vehicle can be displayed. In this way, the vehicle can be shown from the bird's eye view.

According to another embodiment, an absolute correction value for the colour is estimated for each input image, a relative correction value for the colour is determined for respective overlapping areas of the input images on the basis of the absolute correction values and the colour adjustment is performed on the basis of the absolute and relative correction values. For each input image or each region of interest, a colour hue can be estimated independently. This absolute colour correction can be applied to mean colour estimates in each region of interest within the overlapping area. The overlapping areas can be assigned to the corners of the top view image. The corrected colour hue values can be used to determine if a relative colour correction should be added to the absolute colour correction. In this way, the remaining relative colour correction that would arise even after applying the absolute colour correction can be simulated or estimated. This can also be referred to as horseshoe approach. Therefore, even if the real colour hue cannot be estimated, the colour in the composite image can be harmonized so that a human cannot see the difference from one input image to another.

Moreover, it is advantageous if the colour adjustment is performed based on two- dimensional Gaussian curves describing a colour cast. The histograms can be weighted in a first instance. Here, the weighting can be performed based on a Gaussian curve. In particular, it is envisaged that a plurality of two-dimensional Gaussian curves is used which describe respective colour casts in a U-V space. Especially those two-dimensional curves are taken into account, which describe a yellow cast and/or a blue cast. This takes into account that a colour cast in the images affects usually as a yellow cast or a blue cast. Thus, a reliable and simple adjustment of colour are possible.

According to a further embodiment, for the adjustment of the brightness and/or colour of the at least one input image a brightness and/or colour of a road surface is determined in the environmental region of the motor vehicle and used as a reference. The road surface in the environment of the motor vehicle or the road surface, on which the motor vehicle is located, usually has a grey colour. This grey colour is used as a reference. If this grey colour of the road is used as a reference, colour casts can be reliably detected.

The invention also relates to an image processing device for a camera system of a motor vehicle, which is adapted to perform a method according to the invention or an

embodiment thereof. The image processing device may be integrated, for example, in an in-vehicle control unit.

A camera system for a motor vehicle according to the invention comprises at least two cameras for capturing at least one input image and an image processing device according to the invention. In particular, the camera system comprises at least four cameras forming a surround view camera system, wherein the image processing device is adapted to generate the output image based on the input images captured by the surround view camera system. In addition, the camera system can have a display device for displaying the top view image.

The invention also relates to a motor vehicle with a camera system according to the invention. The vehicle is particularly formed as a passenger car. The cameras are in particular arranged on the motor vehicle in a distributed manner, so that the

environmental area surrounding the motor vehicle can be monitored.

The preferred embodiments presented with respect to the method according to the invention and the advantages thereof correspondingly apply to the camera system according to the invention as well as to the motor vehicle according to the invention.

Further features of the invention are apparent from the claims, the figures and the description of figures. The features and feature combinations mentioned above in the description as well as the features and feature combinations mentioned below in the description of figures and/or shown in the figures alone are usable not only in the respectively specified combination, but also in other combinations or alone without departing from the scope of the invention. Thus, implementations are also to be considered as encompassed and disclosed by the invention, which are not explicitly shown in the figures and explained, but arise from and can be generated by separated feature combinations from the explained implementations. Implementations and feature combinations are also to be considered as disclosed, which thus do not have all of the features of an originally formulated independent claim. Moreover, implementations and feature combinations are to be considered as disclosed, in particular by the implementations set out above, which extend beyond or deviate from the feature combinations set out in the relations of the claims.

Now, the invention is explained in more detail based on preferred embodiments as well as with reference to the attached drawings.

There show:

Fig. 1 a motor vehicle according to an embodiment of the invention, which

comprises a camera system;

Fig. 2 a top view image which is provided with the camera system;

Fig. 3 input images, which are provided with cameras of the camera system and which are divided into regions of interest;

Fig. 4 a schematic flow diagram of a method for determining a top view image;

Fig. 5 the top view image, which is composed of the input images;

Fig. 6 histogram values of a histogram of an input image;

Fig. 7 the top view image according to another embodiment;

Fig. 8 a top view image, in which the brightness of the input images is adapted;

Fig.9, 10 areas of the top view images, in which the colour is adapted; and Fig. 1 1 regions of interest in a U-V-space for the adaption of a colour cast of the input images.

In the figures, identical and functionally identical elements are provided with the same reference characters.

Fig. 1 shows a motor vehicle 1 according to an embodiment of the present invention in a plan view. Presently, the motor vehicle 1 is formed as a passenger car. The motor vehicle 1 includes a camera system 2, which includes an electronic control unit 3. In addition, the camera system 2 comprises a plurality of cameras 4a, 4b, 4c and 4d.

Herein, the camera system 2 comprises four vehicle-mounted cameras 4a, 4b, 4c, 4d. A first camera 4a is attached to a front area 7 of the motor vehicle 1 and serves for capturing input images FV showing the environmental region 9 in front of the motor vehicle 1 . A second camera 4b is attached to a rear area 5 of the motor vehicle 1 and serves for capturing input images RV showing the environmental region 9 behind the motor vehicle 1 . A third camera 4c is attached to a left side 8 of the motor vehicle 1 and serves for capturing input images ML showing the environmental region 9 to the left besides the motor vehicle 1 and a fourth camera 4d is attached to a right side 6 of the motor vehicle 1 and serves for capturing input images MR showing the environmental region 9 to the right besides the motor vehicle 1 . The cameras 4a, 4b, 4c, 4d can comprise fisheye lenses in order to enlarge a field of view of the cameras 4a, 4b, 4c, 4d. The input images FV, RV, ML and MR or input video frames captured by the cameras 4a, 4b, 4c, 4d can be displayed on a display device 10 of the camera system 2 in the form of a video. The input images FV, RV, ML and MR captured by the cameras 4a, 4b, 4c, 4d of the camera system 2 can be warped and merged, e.g. by the electronic control unit 3 to determine a top view image 1 1 of the motor vehicle 1 and the environmental region 9.

Fig. 2 shows a top view image 1 1 generated from input images FV, RV, ML and MR of the camera system 2. The top view image 1 1 is a composite image 14, which is determined from the input images FV, RV, ML and MR. The top view image 1 1 can be displayed on the display device 10. In the top view image 1 1 , the environmental region 9 is shown from the perspective of a viewer above the motor vehicle 1 looking down to the motor vehicle 1 . The top view image 1 1 is suggestive of being captured by a camera, a so-called virtual camera, positioned above the motor vehicle 1 . Since the motor vehicle 1 itself cannot be captured by the cameras 4a, 4b, 4c, 4d of the camera system 2 a model 12 of the motor vehicle 1 is inserted into the top view image 1 1 . In this top view image 1 1 the input images FV, RV, ML and MR are not harmonized in terms of brightness and colour. As can be noticed in Fig. 2, the input image FV from the front camera 4a and therefore an area 13 in the top view image 1 1 is much darker than the other parts of the top view image 1 1 (in the present case shown by hatching).

For providing the top view image 1 1 the input images FV, RV, ML and MR should be matched in terms of brightness and colour. For this purpose, the input images FV, RV, ML and MR are divided into respective regions of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2 and MR3. This is illustrated in Fig. 3. Here it can be seen that the input image FV is divided into the regions of interest FVO and FV2. The input image RV is divided into the regions of interest RVO and RV2. Further, the input image ML is divided into the regions of interest MLO, ML1 , ML2 and ML3. Finally, the input image MR is divided into the regions of interest MRO, MR1 , MR2 and MR3. In this case, overlapping areas 17 between the regions of interest MLO and FVO, between the regions of interest FV2 and MRO, between the regions of interest RV2 and MR3 as well as between the regions of interest ML3 and RVO arise.

Fig. 4 shows a schematic flow diagram of a method for providing the top view image 1 1 . In a step S1 the components Y, U and V are determined in the YUV color space for the regions of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2 and MR3. Furthermore, histograms for the components Y, U and V are determined. The histograms are fed to a harmonization algorithm in a step S2. In order to harmonize the input images FV, RV, ML, MR, the statistics from the regions of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3 are used as input to the harmonization algorithm. Here, the histograms for the components Y, U and V for each of the four cameras 4a, 4b, 4c 4d and their corresponding regions of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2 and MR3 are computed. The region of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3 statistics are computed in a form of three histograms corresponding to their Y, U and V values. The histogram representation provides much higher flexibility in terms of image segmentation and post processing operations rather than representing the whole region with a single number, for example as a mean value. The harmonization algorithm takes computed histograms as an input and generates correction values for each camera 4a, 4b, 4c, 4d and each Y, U and V channel based on which tone mapping tables are determined (step S3). The determined tone mapping tables are then applied in the next time instance t+1 for tone mapping (step S4). Thus, the in a step S5 determined input images FV, RV, ML, MR can be edited. For the brightness and colour cast estimate the values for the histograms for the components Y, U and V are quantized in order to reduce large number values and remove contribution of histogram values, which occur relatively rare. The quantization can be done by simple division of histogram values by a constant value, which in general can be different for the components Y, U and V. The strength of histogram quantization can be used as a tuning parameter for a particular case scenario or specific camera image settings. One example of histogram quantization is to enhance stability of histogram based brightness and colour cast estimates. Depending on the camera settings, YUV histograms will be differently spread across the input images FV, RV, ML, MR and change from frame to frame. After histogram quantization, a simple histogram smoothing through neighbouring histogram bins with an MR filter kernel can be applied. This can in some cases necessary when the histogram oscillates considerably from frame to frame even though the three dimensional scene is the same. This may be the case when the motor vehicle 1 is moving.

In the following, the brightness balancing and colour correction will be described separately.

For the brightness balancing the input histograms of the component Y for the regions of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3 are first transformed using a particular transfer function for the specific camera image, prior to histogram quantization. This histogram transformation is used to modify histogram shape according to camera parameters and/or some other input statistics or parameters. In the proposed implementation, gamma correction values from the camera registers are used to define a gamma un-correction transfer curve that is applied to the input histogram. After that, a weighted histogram integration within boundaries defined by specific parameters is performed. By weighting certain histogram bins a higher importance to certain luminance values or luma values is given. For example, histogram bins that most probable correspond to the road surface that is used as reference can be weighted. By introducing lower and upper boundaries for histogram segmentation the influence of certain histogram parts that are assumed to be not related to brightness can be completely cut off. This includes pedestrian crossing white lines, black objects, e.g. a dark vehicle parked next to the motor vehicle 1 , or the like.

The weighted and bound histogram integration is used to generate the initial luma estimates for each regions of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3. These initial luma estimates provide information of rough brightness levels corresponding to the camera image. There are two main options or schemes for luma estimation: a non-segmentation based approach and a histogram-segmentation based approach.

In the non-segmentation based approach, the histogram values are computed as weighted average of the histogram values between a minimum value and a maximum value. These values will depend on camera settings, such as the high dynamic range images, the gamma correction, sharpness, or the like. Additionally, a weighting of each histogram bin value can be determined in two possible ways: On the one hand, a

Gaussian curve weighting, which is centred around a most common luma value can be used. Here, the width and the slope can be adapted using a standard deviation. On the other hand, a weighting using histogram values that are experimentally found can be performed. These histogram values can be adjusted depending on the histogram quantization, which is performed before this step, or changed camera settings.

Preferably, the histogram-segmentation based approach is used. The main idea in the histogram segmentation-based approach is to determine luminance histograms for each of overlapping regions in the regions of interests FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3. The overlapping of the regions of interests FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3 are shown in Fig. 5. The main reason for introducing histogram segmentation is to reduce the influence of surrounding objects that do not correspond to road surface. The road surface is assumed as a reference feature for the overlapping regions. As can be noticed, the overlapping regions do not overlap completely, but there are certain parts that are different, for example MLO and FVO. Therefore, it is important to segment histograms and weight certain parts in order to obtain the final luma estimates that can be compared. Based on this comparison, the brightness correction for each input image FV, RV, ML, MR can be determined.

In the present case, a bi-modal histogram segmentation is used, which will be explained in connection with Fig. 6. This means that the histogram is segmented only in two regions. This is done mainly for the sake of simplicity, which in practice provides sufficiently accurate and robust results. First, a mean value Hm is determined for each region of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3 as a weighted average value of the histogram. Then a first histogram value H1 and a second histogram value H2 for each region of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3 are determined. The mean value Hm is used as a threshold to estimate the first and the second histogram value H1 , H2. The first histogram value H1 and the second histogram value H2 describe the luminance of different parts of the region of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3. For example the first histogram value H1 can describe the luminance in the left part of the region of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3 and the second histogram value H2 can describe the luminance in the right part of the region of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3.

Based on the first histogram value H1 and the second histogram value H2 a weighting coefficient wC is determined to calculated the final brightness estimate Y_est for each region of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3:

Yes, = ((m - wC) ^* H2 + wC ^* H1 )/m, wherein m is a constant multiplication parameter that defines the precision of the weighting coefficient wC. In order to determine the weighting coefficient wC for each region of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3, a low weighting value and high weighting value are determined that correspond to the first and second histogram value H1 , H2, respectively. It is important to note that two overlapping region of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3 must have same resulting weighting coefficients wC. In this way, the histogram values that should be matched, are weighted in the same manner.

An overlapping area 17 of the regions of interests FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3 results for example in the upper left corner of the top view image 1 1 . There, the regions of interest FVO and MLO overlap. Here, the minimum of the first histogram values H1 from the two regions of interest FVO and MLO is determined. Based on the minimum the lower weighting value is determined. In the same manner, the higher weighting value is determined. For the computation of the lower weighting value a black level is used as a reference and for the computation of the higher weighting value a white level is used as a reference. The black level and the white level are thresholds corresponding to black and white objects, respectively. Therefore, a factor can be used to determine the lower and higher weighting value. Moreover, based on computed low and high weighting values a normalization function to obtain the weighting coefficient is performed. In this manner, both the distance of the first histogram value H1 to the black level and the distance of the second histogram value H2 to the white level are taken into account. Additionally, in order to avoid oscillations of the weighting coefficients from one image to another, as a final step a recursive temporal filtering of the weighting coefficients is applied.

Besides the weighted histogram averaging, a temporal filtering of luma estimate values is performed. This is again to ensure robustness against small oscillations from image to image, which in some particular cases can cause reduced algorithm performance and hence annoying visual artefacts. From this point, the luma estimate values can be used as an input to a brightness harmonization algorithm that determines amount of brightness correction to be applied to each input image FV, RV, ML, MR. The main idea in the brightness balancing approach is to use overlapping regions for harmonizing estimated brightness values between the different regions of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3. In the present case, the overlapping regions are located in the four corners of the topview image 1 1 . There is an overlapping between the input images FV and ML, between the input images FV and MR, between the input images RV and ML as well as between the input images RV and MR.

In order to determine the luminance matching error between two neighboring input images FV, RV, ML, MR the respective differences between the pairs of input images FV, RV, ML, MR are determined. In this way, the difference between the overlapping input images FV, RV, ML, MR in the corners can be determined. Furthermore, the difference between the all pairs of input images FV, RV, ML, MR can be determined to provide the matching error for the cameras 4a, 4b, 4c, 4d. In this way the "worse" camera can be determined, which corresponds to the maximum paired matching error. This camera can also be referred to as master camera. Consequently, this can be used as an indicator from which input image FV, RV, ML, MR the brightness correction should start in a predetermined direction. This predetermined direction can extend along a circle. It may further be decided whether the direction is traversed clockwise or counter clockwise. This is illustrated by the arrows 16 and 16' in Fig. 6.

The two important decisions, namely the determination of the master camera and the correction direction, can vary from image to image. In general, this binary kind of decision for choosing the master camera and correction direction can be found not be reliable enough and to produce instability in the final algorithm output. Consequently, the correction values can be calculated for all possible master cameras and correction directions. Then a weighted averaging can be performed in order to determine final brightness correction values. The computed difference values are then used to form tone mapping tables for each input image FV, RV, ML, MR within the rendering process. These values are additionally temporally recursively averaged to obtain the final brightness correction values. A transfer curve for tone mapping can be a linear curve, which is shifted up and down. Furthermore, a gamma correction curve can be included to the table so that the final luma values can be adapted to the display device 10.

Fig. 8 shows a top view image 1 1 , wherein the brightness of the input images FV, RV, ML, MR was adjusted. Compared to Fig. 2 can be clearly seen that the top view image 1 1 has a uniform brightness.

Hereinafter, the colour correction of the input images FV, RV, ML, MR will be explained. Similarly to the case of brightness balancing, the aim of the colour correction algorithm is to estimate correction values for each camera, except that in this case chroma U and V values are modified. Therefore, the outputs of the algorithm are correction values for both U and V for each camera, which are subsequently used for building chroma tone mapping tables.

In comparison to the brightness balancing where the problem to be solved was harmonizing brightness of particular parts of the top view image 1 1 from different cameras 4a, 4b, 4c, 4d, in the colour correction algorithm the harmonization (relative correction) and the absolute colour correction should be performed in order to generate a top view image 1 1 with correct colours. The latter refers to the colour cast problem in general camera imaging systems where output camera image can have certain colour cast due to specific 3D content and radiating light in the environmental region 9. Although sometimes small colour casts are not so easily perceivable by humans in case of only single image view, when two images corresponding from two cameras are put next to each other - as it is the case in top view image 1 1 generation - even relatively small colour cast differences can be visible and annoying. Consequently, colour correction aims at both correcting absolute and relative colour cast between different camera images.

Since in general the true colour values of particular objects surrounding the motor vehicle 1 are not known, the information coming from the colour of the object cannot be efficiently taken into account for colour correction. Here the fact is considered that in automotive application a road surface is in most cases present in environmental region 9 and that the road surface is in most cases colour free. In cases when any relevant information around the motor vehicle cannot be found, performing of the colour correction is avoided. This can be the case when the motor vehicle 1 is not on the road or the road surface is not grey, for example close to crossings where it can be yellow, blue, red, or the like. Another important aspect is that in case when grey road surface cannot be used as a reference for colour cast correction, in most cases it is not even important since the colour cast is mostly visible on objects that are either by default grey or some objects, such as human face for example, that human eye is used to its reference colour. In other words, very colourful objects incur colour casts that are often not so visible.

The input for the colour correction algorithm are U and V histograms for the regions of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3, which are quantized prior to further processing. These histograms are further weighted using a Gaussian function of a specific standard deviation value. The central or mean value of the Gaussian curve is set on the basis of the reference chroma values for grey. In this respect two different colour correction algorithms are proposed, a first and a second colour correction algorithm.

Within the first colour correction algorithm, the U and V histograms are weighted with a Gaussian curve that is centred at U = V = 128 and has a predetermined standard deviation. The standard deviation can be set depending on different 3D content and camera settings. The Gaussian curve is applied as weighting to the histogram in the form of table or array. Then, the final colour cast estimated values are determined. Here, a threshold can be defined, which represents the minimum amount of computed weighted histogram values that is considered necessary for colour cast estimation for the region of interest. The optimal value depends on the application, the camera setting and other algorithm parameters such as histogram quantization. The absolute colour cast estimated values for each camera 4a, 4b, 4c, 4d are the determined as average colour cast value corresponding to the same camera 4a, 4b, 4c, 4d and different regions of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3. After the absolute colour casts have been determined, the relative colour harmonization is applied. This is used to balance colour cast so that the viewer gets the feeling that the whole top view image 1 1 is of single colour cast.

The colour harmonization is done in two-step method, the so-called "horse shoe" approach. This is schematically described in Fig. 9 and Fig.10. The first step of the colour harmonization that corresponds to Fig. 9 comprises the measuring of differences of the coulor cast in the four corners of the top view image 1 1 . In this way, miss-matches are determined that are subsequently used for correcting the colour cast values of the input images ML and MR. First, the miss-match values are used to correct the colour casts of the input images MR and ML relatively to the input images FV and RV (shown in Fig. 9). The corrected colour cast values are then used in second step to determine the miss- match values for the corners. Finally, the colour cast correction values for the input images FV and RV are adjusted relatively to the input images MR and ML (Fig. 10).

In the second colour correction algorithm, the U and V histograms are weighted with multiple Gaussian curves as opposed to being weighted by only one curve. The position of the multiple two dimensional Gaussian weighting curves in the U-V-space is shown in Figure 1 1 .

Fig. 1 1 shows different regions of interest UV_ROI_0, UV_ROI_1 , UV_ROI_2, UV_ROI_3 and UV_ROI_4 in the U-V-space. The region of interest UV_ROI_0 corresponds to the central position of the two dimension Gaussian curve and this is the one actually only used in the first colour correction algorithm. In the second colour correction algorithm in addition the regions of interest UV_ROI_2 and UV_ROI_4 are used, which correspond to yellowish and bluish casts, respectively. The two additional regions of interest UV_ROI_2 and UV_ROI_4 are used since the colour cast, introduced and corrected in the camera 4a, 4b, 4c, 4d, is in most cases moving from UV_ROI_2 to UV_ROI_4 and in the opposite direction. The other two regions UV ROM and UV_ROI_3 are assumed to belong to the colour information coming from coloured objects and are not used for colour correction. It is aimed to search for presence of colour casts in the grey objects corresponding preferably to the road surface. The regions of interest UV_ROI_2 and UV_ROI_4 are further used to determine the amount of absolute colour correction for each region of interest FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3 and subsequently for each camera 4a, 4b, 4c, 4d.

For the colour correction, first the indices for the Gaussian weighting curves are determined. Therefore, the indices are placed in filter kernel array structure. The filter kernel is generated only once for a particular standard deviation. In the current implementation, it is set to one value for all two-dimensional Gaussian weighting curves. The only difference is shift value, which can be adjusted. For example, it can be used for adjusting the centre of Gaussian kernel array to the centre value, which is then shifted to the left. After determining the indices, the weighted histogram averaging is performed by weighted histogram integration. Also in this case a threshold is used to verify if there is a sufficient number of collected chrominance values or chroma values in the accumulated and weighted histogram values. Furthermore, the weightings for the regions of interest UV_ROI_2 and UV_ROI_4 are determined. These weightings will provide a soft decision, which out of these two should be used for estimating final chroma value. Moreover, the distance between the points in second and the forth quadrant in the U-V-space is determined. If the distance is small and close to zero, both weighting coefficients are equal and thus give the same influence to the final chroma estimate. The idea is to determine the distance of the point to the centre estimate that corresponds to UV_ROI_0. The closer it is to the centred estimate the more weighting will be provided in the final calculation. This is based on the fact that in most cases the colour of the object moves between the second and fourth quadrant (due to a colour correction algorithm of the camera) and that one that is closer to the centre is actually colour cast while the other one corresponds to the coloured object.

The colour correction is performed using the horse shoe approach for further chroma harmonization between the input images FV, RV, ML, MR. This is applicable for both, the first and the second colour correction algorithm. Finally, the U and V colour cast outputs are further subtracted from the reference value of 128. These values are used to generate tone mapping tables for U and V chroma components, which are used for adjusting the colour cast of the display device 10.

Claims

1 . Method for determining a composite image (14) of an environmental region (9) of a motor vehicle (1 ), wherein at least two input images (FV, RV, ML, MR) from respective cameras (4a, 4b, 4c, 4d) of the motor vehicle (1 ) are received and the composite image (14) is determined from the input images (FV, RV, ML, MR), wherein for determining the composite image (14), a brightness and/or colour of at least one of the input images (FV, RV, ML, MR) is adjusted,

characterized in that

histograms are determined, which describe a luminance and/or a chrominance of the respective input images (FV, RV, ML, MR) and the brightness and/or colour of the at least one input image (FV, RV, ML, MR) is adjusted on the basis of histograms.

2. Method according to claim 1 ,

characterized in that

on the basis of the histograms respective correction values for the components Y, U and/or V in a YUV colour space of the at least one input image (FV, RV, ML, MR) are determined and an input image (FV, RV, ML, MR), which is received at a later instance in time (t+1 ), will be corrected on the basis of the respective correction values.

3. Method according to claim 1 or 2,

characterized in that

in the input images (FV, RV, ML, MR) at least two regions of interest (FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3) are determined and the histograms are determined for the respective regions of interest (FVO, FV2, RVO, RV2, MLO, ML1 , ML2, ML3, MRO, MR1 , MR2, MR3).

4. Method according to any one of the preceding claims,

characterized in that

an overlapping area (15) is determined, in which the at least two input images (FV, RV, ML, MR ) overlap at least in portions when determining the composite image

(14) and the histograms are determined for the respective input images (FV, RV, ML, MR) depending on the overlapping area (14).

5. Method according to claim 4,

characterized in that

the histogram for respective input image (FV, RV, ML, MR) is segmented in dependence of the overlapping area (15) and the segments of the histograms of the respective input images (FV, RV, ML, MR), which belong to the overlapping area

(15) , are weighted.

6. Method according to any one of the preceding claims,

characterized in that

a plurality of input images (FV, RV, ML, MR) is received, luminance values are determined on the basis of the histograms for each of the input images (FV, RV, ML, MR) and a respective difference between the luminance values is determined.

7. Method according to claim 6,

characterized in that

the respective differences are determined for two different orders along the input images (FV, RV, ML, MR) for each of the input images (FV, RV, ML, MR) and a correction value for the at least one input image (FV, RV, ML, MR) is determined based on the respective differences.

8. Method according to any one of the preceding claims,

characterized in that

at least four input images (FV, RV, ML, MR) are received and as the composite image (14) a top view image (1 1 ) is determined.

9. Method according to claim 8,

characterized in that

an absolute correction value for the colour is estimated for each input image (FV, RV, ML, MR), a relative correction value for the colour is determined for respective overlapping areas (15) of the input images (FV, RV, ML, MR) on the basis of the absolute correction values and the colour adjustment is performed on the basis of the absolute and relative correction values.

10. Method according to any one of the preceding claims,

characterized in that

the colour adjustment is performed based on two-dimensional Gaussian curves describing a colour cast.

1 1 . Method according to any one of the preceding claims,

characterized in that

when adjusting the brightness and/or colour of the at least one input image (FV, RV, ML, MR), a brightness and/or colour of a road surface is determined in the environmental region (9) of the motor vehicle (1 ) and used as a reference.

12. Image processing device (3) for a camera system (2) of a motor vehicle (1 ), which is adapted to perform a method according to any one of the preceding claims.

13. Camera system (2) for a motor vehicle (1 ) comprising at least two cameras (4a, 4b, 4c, 4d) for capturing a respective input image and an image processing device (3) according to claim 12.

14. Camera system (2) according to claim 13,

characterized in that

the camera system (2) comprises at least four cameras (4a, 4b, 4c, 4d), wherein the image processing device (3) is adapted to generate the top view image (1 1 ) as the composite image (14) based on the input images (FV, RV, ML, MR)

15. Motor vehicle (1 ) comprising a camera system (2) according to claim 13 or 14.