EP4078941A2

EP4078941A2 - Converting input image data from a plurality of vehicle cameras of a surround-view system into optimised output image data

Info

Publication number: EP4078941A2
Application number: EP20841885.5A
Authority: EP
Inventors: Christian Scharfenberger; Michelle Karg
Original assignee: Continental Autonomous Mobility Germany GmbH
Current assignee: Continental Autonomous Mobility Germany GmbH
Priority date: 2019-12-19
Filing date: 2020-12-09
Publication date: 2022-10-26
Also published as: DE102019220171A1; WO2021121491A2; WO2021121491A3; US20230342894A1; DE112020006216A5

Abstract

The invention relates to a machine learning method, to a method and to a device for converting input image data (Ini) from a plurality of vehicle cameras (2-i) of a surround-view system into optimised output image data (Opti). The method for converting input image data from a plurality of vehicle cameras (2-i) of a surround-view system into optimised output image data comprises the following steps: a) input image data (Ini) captured by the vehicle cameras (2-i) and having a current brightness or colour distribution are provided to a trained artificial neural network (CNN1, CNN10, CNN11, CNN12); b) the trained artificial neural network (CNN1, CNN10, CNN11, CNN12) is configured to convert the input image data (Ini) having the current brightness or colour distribution into optimised output image data (Opti) having different output brightness or colour distribution; and c) the trained artificial neural network (CNN1, CNN10, CNN11, CNN12) is configured to output the output image data (Opti).

Description

Conversion of input image data from a plurality of vehicle cameras

All-round vision system in optimized output image data

The invention relates to a machine learning method, a method and a device for converting input image data from a plurality of vehicle cameras of an all-round vision system into optimized output image data.

Today's vehicles are increasingly equipped with surround view and / or assistance systems that monitor the areas in front of, next to or behind the vehicle. This is used either to recognize objects to avoid collisions, to recognize road boundaries, to keep the vehicle within the lane or simply to display the surroundings to assist with a parking process.

These systems work with high-resolution cameras, which today have an ever higher dynamic range. Display and recognition functions in particular benefit from the latter in situations with different levels of brightness and contrast.

DE 102014210323 A1 shows a device and a method for adaptive image correction of at least one image parameter of a camera image with: several cameras for generating camera images, the camera images from adjacent cameras each having overlapping image areas; and with an image processing unit which combines the camera images generated by the cameras to form a composite overall image; wherein the image processing unit has an image correction component that calculates a plurality of average image parameter levels of the image parameter in the overlapping image areas of the camera image for each received camera image and sets the respective image parameter as a function of the calculated average image parameter levels.

The systems mentioned above show a very good performance in scenarios that are sufficiently illuminated by daylight, street lighting or headlights of a vehicle. However, degradation both in the detection of objects and in the representation of the environment or objects occurs as soon as in In a situation there is little or no ambient light available to illuminate the scenario.

A prominent example is driving on an unlit country road at night. The vehicle is equipped with a surround view system, which is supposed to offer both assistance and display functions while driving. While the vehicle illuminates the front and rear area through the front and rear headlights, the area next to the vehicle is almost unlit.

Another example is parking a vehicle in a dark corner of a parking garage. Here, too, the case occurs especially in parking positions next to walls or other vehicles in which too little or no light is available for the side cameras.

This can be remedied by additional lamps, which are installed in the side areas of the vehicle and illuminate the critical areas next to the vehicle. For complete illumination, however, a large number of lamps is necessary, which, in addition to critical design restrictions, can lead to considerable additional costs in the vehicle.

Well-known algorithmic processes such as gamma correction, automatic white balance or histogram equalization can algorithmically brighten and improve the images. The latter, however, show significant performance losses, especially in the dark, due to a lack of color information in the image.

A system would therefore be desirable which algorithmically enables good upgrading of the unlit areas without additional lighting.

It is the object of the present invention to provide solutions for this.

The object is achieved by the subjects of the independent claims. Advantageous embodiments are the subject matter of the dependent claims, the following description and the figures. A method for machine learning of the conversion of input image data from several vehicle cameras of a panoramic system into optimized output image data by means of an artificial neural network provides that the learning is carried out with a large number of training image pairs in such a way that at the input of the artificial neural network one first image of a first brightness or color distribution and a second image of the same scene with a different second brightness or color distribution is provided as the target output image. The artificial neural network can be, for example, a convolutional neural network (“folding neural network”, CNN). The vehicle cameras are preferably arranged and configured in such a way that, taken together, they capture and image the area of the vehicle environment surrounding the vehicle.

In one embodiment, the training image pairs are generated by recording a first image with a first and a second image with a second brightness at the same time or immediately following one another with different exposure times. A first, shorter exposure time leads to a darker training image and a second, longer exposure time to a lighter training image. For example, the respective vehicle camera is stationary (immobile) in relation to the surroundings to be recorded during the generation of the training data. For this purpose, the training data can be recorded with at least one vehicle camera of a stationary vehicle, for example. The scene captured by the vehicle camera can, for example, contain a static environment, that is to say without moving objects.

In one embodiment (only) one artificial neural network is trained jointly or simultaneously for all vehicle cameras.

A sequence of successive images can be used for each individual camera for joint training.

The time correlation of images can be profitably taken into account during training and / or when using the trained network.

Information about image features and their desired output image data can be used, which are recorded at a point in time t by a front camera and at a later point in time by a side camera or the rear camera. In this way it can be trained that an object with certain image features has an identical brightness and color in the output images of all individual cameras. According to one embodiment, at least one factor d is determined as a measure for the difference between the second and the first brightness or color distribution of a training image pair and is provided to the artificial neural network as part of the training.

The factor d can be determined, for example, as the ratio of the second brightness or color distribution to the first brightness or color distribution. The brightness can in particular be determined as the mean brightness of an image or on the basis of a luminance histogram of an image.

In one embodiment, the artificial neural network has a common input interface for two separate output interfaces. The common input interface has shared feature representation layers. Converted image data are output at the first output interface. ADAS-relevant detections of at least one ADAS detection function are output at the second output interface. ADAS stands for advanced systems for assisted or automated driving (English: Advanced Driver Assistance Systems). ADAS-relevant detections are e.g. objects, objects, road users, which represent important input variables for ADAS / AD systems. The artificial neural network includes ADAS detection functions, e.g. lane recognition, object recognition, depth recognition (3D estimation of the image components), semantic recognition, or the like. As part of the training, the outputs of both output interfaces are optimized.

A method for converting input image data from a plurality of vehicle cameras of an all-round vision system into optimized output image data comprises the steps: a) input image data of a current brightness or color distribution recorded by the vehicle cameras are provided to a trained artificial neural network, b) the trained artificial one The neural network is configured to convert the input image data with the current brightness or color distribution into output image data with a different output brightness or color distribution, and c) the trained artificial neural network is configured to output the output image data. The output image data optimized in terms of their brightness or color distribution advantageously enable the images from the individual vehicle cameras to be better combined to form a combined image which can be displayed to the driver.

In one embodiment, in step a) a factor d is additionally provided to the trained artificial neural network and in step b) the (strength or degree of) conversion is controlled as a function of factor d.

According to one embodiment, the conversion in step b) takes place in such a way that an improvement in vision with regard to overexposure is achieved. For example, during the training course, people learned to reduce the brightness of overexposed images or to adjust the color distribution.

In one embodiment, in step b) the input image data with the current brightness are converted into output image data with a longer (virtual) exposure time. This offers the advantage of avoiding motion blur.

According to one embodiment, the factor d is estimated and in the estimation the brightness or color distribution of the current captured image data (e.g. illuminance histogram or average brightness) or the previously captured image data or the history of the factor d is taken into account.

For example, too high a brightness indicates overexposure, while too low a brightness indicates underexposure. Both can be determined by means of corresponding threshold values and corrected by a corresponding conversion

In one embodiment, a separate factor d is estimated or determined for each of the vehicle cameras. This enables the individual conversion for image data from the individual vehicle cameras, in particular as a function of the current brightness or color distribution of the image from the respective vehicle camera.

In one embodiment, after a detection that at least two image regions of a currently captured image have (significantly) different image brightness, a different factor d is estimated or determined for each of the image regions. If there are image regions with different illumination intensities, the factor d can thus vary within an image and image regions with different factors d are determined via brightness estimates. The brightness improvement can thus be adapted to individual image regions.

According to one embodiment, a development of factor d over time can be taken into account when determining or estimating factor d.

For this purpose, the development of factor d over time and a sequence of input images are included in the estimate. Information about the temporal development of the brightness can also be used for image regions with different factors d.

According to one embodiment, information about the current surroundings of the vehicle is taken into account when determining the factor d. The estimate of the factor d can take into account further scene information, such as information about the surroundings (country road, city, motorway, tunnel, underpass), which is obtained via image processing from the sensor data or data from a navigation system (e.g. GPS receiver with digital map).

For example, the factor d can be estimated based on information about the surroundings and from the chronological sequence of images as well as from the history of the factor d.

The estimation of the factor d when using a trained artificial neural network can thus take place dynamically.

In one embodiment, the converted image data is output to at least one ADAS detection function which determines and outputs ADAS-relevant detections. ADAS detection functions can include known edge or pattern recognition methods as well as recognition methods which can recognize and optionally classify relevant image objects by means of an artificial neural network.

In an alternative embodiment, the approach can be expanded and the artificial neural network for converting the image data can be combined with a neural network for ADAS detection functions, for example lane detection, object detection, depth detection, semantic detection. This means that there is hardly any additional effort in computing time. After the training, the (first) output interface for outputting the converted image data can be eliminated so that only the (second) output interface for the ADAS detections is available when used in the vehicle.

The invention further relates to a device with at least one data processing unit configured to convert input image data from a plurality of vehicle cameras of a panoramic vision system into optimized output image data. The device comprises: an input interface, a trained artificial neural network and a (first) output interface.

The input interface is configured to receive input image data of a current brightness or color distribution from the vehicle cameras. The trained artificial neural network is configured to convert the input image data, which have a first brightness or color distribution, into output image data with a different output brightness or color distribution.

The (first) output interface is configured to output the converted image data.

The device or the data processing unit can in particular be a microcontroller or processor, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable) Gate array) and the like, as well as software for carrying out the corresponding method steps.

According to one embodiment, the data processing unit is implemented in a hardware-based image preprocessing stage (Image Signal Processor, ISP).

In one embodiment, the trained artificial neural network for converting input image data into output image data with optimized brightness or color distribution is part of an ADAS detection neural network in the vehicle, e.g. for semantic segmentation, lane detection or object detection, with a divided input interface (Input or feature representation layers), and two separate output interfaces (output layers). The first output interface is for Output of the converted output image data and the second output interface is configured to output the ADAS detections (image recognition data).

The invention further relates to a computer program element which, when a data processing unit is programmed with it, instructs the data processing unit to carry out a method for converting input image data from the vehicle cameras into optimized output image data.

The invention also relates to a computer-readable storage medium on which such a program element is stored.

The invention also relates to the use of a method for machine learning conversion of input image data from a plurality of vehicle cameras of a panoramic vision system into optimized output image data for training an artificial neural network of a device with at least one data processing unit.

The present invention can thus be implemented in digital electronic circuitry, computer hardware, firmware, or software.

Exemplary embodiments and figures are described in more detail below. 1 shows a first schematic representation of a device according to the invention in one embodiment;

2: a second schematic representation of a device according to the invention in an embodiment in a vehicle;

3: schematically, a general overview of a system for converting or improving the visibility of vehicle camera images;

4: a system with a first neural network for improving visibility;

5 shows a neural network for improving the view of an input image, which shares feature representation layers with a second network for the detection functions and has two outputs; and FIG. 6 shows a modified approach based on FIG. 5.

As can be seen from FIG. 1, a device according to the invention for 1 for converting input image data from several vehicle cameras of an all-round viewing system into optimized output image data can be several units or Have circuit components. In the exemplary embodiment shown in FIG. 1, the device for adaptive image correction has a plurality of vehicle cameras 2-i which each generate camera images or video data. In the exemplary embodiment shown in FIG. 1, the device 1 has four vehicle cameras 2-i for generating camera images. The number of vehicle cameras 2-i can vary for different applications. The device 1 according to the invention has at least two vehicle cameras for generating camera images. The camera images from neighboring vehicle cameras 2-i typically have overlapping image areas.

The device 1 contains a data processing unit 3 which combines the camera images generated by the vehicle cameras 2-i to form a composite overall image. As shown in FIG. 1, the data processing unit 3 has a system for image conversion 4. The image conversion system 4 uses the input image data (Ini) of the vehicle cameras (2-i) to generate output or output image data (Opti) which have an optimized brightness or color distribution. The optimized output image data from the individual vehicle cameras are combined to form a composite overall image (so-called stitching). The overall image composed of the optimized image data (Opti) by the image processing unit 3 is then displayed to a user by a display unit 5. In one possible embodiment, the image conversion system 4 is formed by an independent hardware circuit which converts the brightness or the color distribution. In an alternative embodiment, the system executes program instructions when performing an image conversion method.

The data processing unit 3 can have one or more image processing processors, converting the camera images or video data received from the various vehicle cameras 2-i and then putting them together to form a composite overall image (stitching). In one possible embodiment, the image conversion system 4 is formed by a processor provided for this purpose, which converts the brightness or the color distribution in parallel with the other processor or processors of the data processing unit 3. The parallel data processing reduces the time required to process the image data. Figure 2 shows a further schematic representation of a device 1 according to the invention in one embodiment. The device 1 shown in FIG. 2 is used in a surround view system of a vehicle 10, in particular a passenger car or a truck. The four different vehicle cameras 2-1, 2-2, 2-3, 2-4 can be located on different sides of the vehicle 10 and have corresponding viewing areas (dashed lines) in front of V, behind H, left L and right R the or . of the vehicle (s) 10.

For example, the first vehicle camera 2-1 is located on a front side of the vehicle 10, the second vehicle camera 2-2 is located on a rear side of the vehicle 10, the third vehicle camera 2-3 is located on the left side of the vehicle 10, and the fourth vehicle camera 2-4 is on the right side of the vehicle 10. The camera images from two adjacent vehicle cameras 2-i have overlapping image areas VL, VR, HL, HR. In one possible embodiment, the vehicle cameras 2-i are so-called fisheye cameras which have a viewing angle of at least 185 °. The vehicle cameras 2 - i can transmit the camera images or camera image frames or video data in one possible embodiment to the data processing unit 3 via an Ethernet connection. The data processing unit 3 calculates a composite surround view camera image from the camera images of the vehicle cameras 2-i, which is displayed on the display 5 of the vehicle 10 to the driver and / or a passenger. In some cases, the light conditions in the surroundings deviate from each other in the front vehicle camera 2-1 and in the rear vehicle camera 2-2 while driving, for example when entering a vehicle tunnel or when driving into a vehicle garage.

When the surroundings of vehicle 10 are dark, the activated front headlights illuminate the front area V in front of vehicle 10 with white light and relatively high intensity, and the rear headlights illuminate the rear area H behind the vehicle with red light and medium intensity. In contrast, the areas to the left L and right R next to the vehicle 10 are virtually unlit.

With a simultaneous or joint training of an artificial neural network with dark images (for example for the side cameras 2-3, 2-4) and bright images (for example for the front 2-1 and rearview cameras 2-2), the neural network learns optimally Parameter. In the joint training for a plurality of vehicle cameras 2-i, ground truth data are preferably used in a first application, which have a brightness and balance used for all target cameras 2-1, 2-2, 2-3, 2-4. In other words, the ground truth data for all target cameras 2-1, 2-2, 2-3, 2-4 are balanced in such a way that, for example, in a surround view application, no differences in brightness are discernible in the ground truth data. With this ground truth data as a reference and the input data from the target cameras 2-1, 2-2, 2-3, 2-4, which can have different brightnesses, a neural network CNN1, CNN10, CNN11, CNN12 is created with regard to an optimal parameter set trained for the network. This data record can consist, for example, of images with white and red headlights for the front cameras 2-1 and rear cameras 2-2, and dark images for the side cameras 2-3, 2-4. Data with differently illuminated side areas L, R are also conceivable, for example when the vehicle 10 is located next to a street lamp or the vehicle 10 has an additional light source on one side.

In a further application, the neural network for the common cameras 2-i can be trained to the effect that even in the case of missing training data and ground truth data for a camera, for example a side camera 2-3 or 2-4, the network can set the parameters for this camera 2-3 or 2-4 is trained and optimized with the missing data based on the training data of the other cameras 2-1, 2-2 and 2-4 or 2-3. This can be achieved, for example, as a restriction (or constraint) in the training of the network, for example as an assumption that the correction and training must always be the same due to similar lighting conditions of the side cameras 2-3 and 2-4.

In a final example, the neural network uses training and ground truth data that differ over time and that are correlated with the cameras 2-i and that were recorded by the various cameras 2-i at different points in time. For this purpose, information from features or objects and their ground truth data can be used which, for example, were recorded at a point in time t by the front camera 2-1 and at a point in time t + n by the side cameras 2-3, 2-4. These features or objects and their ground truth data can replace missing information in the training and basic truth data of the respective other cameras if they are used in the images of the other cameras 2-i and then by the network as training data. In this way the network can set the parameters for Optimize all side cameras 2-3, 2-4 and, if necessary, compensate for missing information in the training data.

When using several vehicle cameras 2-i, this leads to an adapted brightness and balance for all vehicle cameras 2-i, since the individual lighting profiles in the exterior space are explicitly recorded and trained in the overall network.

3 schematically shows a general overview of a system for converting or improving the view of camera images. An essential component is an artificial neural network CNN1, which learns in a training phase to assign a set of corresponding vision-improved images Out (Out1, Out2, Out3, ...) to a set of training images In (In1, In2, In3, ...) . Assigning means here that the neural network CNN1 learns to generate an image with improved vision. A training image (In1, In2, In3, ...) can contain, for example, a street scene at dusk, on which the human eye can only see another vehicle immediately in front of the vehicle and the sky. On the corresponding image with improved visibility (Out1, Out2, Out3, ...), the contours of the other vehicle, a sidewalk as a lane boundary and adjacent buildings can also be seen.

A factor d is preferably used as an additional input variable for the neural network CNN1. The factor d is a measure of the degree of visibility improvement. During training, the factor d for an image pair consisting of the training image and the image with improved vision (In1, Out1; In2, Out2; In3, Out3; ...) can be determined in advance and made available to the neural network CNN1. When using the trained neural network CNN1, a factor d can be used to control how strongly the neural network CNN1 "brightens" or "darkens" an image - one can also think of factor d as an external regression parameter (not just bright - dark, but with any gradation). Since the factor d can be subject to possible fluctuations in the range of +/- 10%, this is taken into account during training. The factor d can be noisy during the training by approx. +/- 10% (e.g. during the different epochs of the training of the neural network), in order to be robust against incorrect estimates of the factor d in the range of approx. +/- during the inference in the vehicle. To be 10%. In other words, the necessary accuracy of factor d is in the range of +/- 10% - thus the neural network CNN1 is robust against deviations in estimates from this parameter. One possibility for generating the training data (training images (In1, In2, In3, ...) and associated vision-enhanced images (Out1, Out2, Out3, ...)) is to record image data of a scene, each with a short and simultaneous or . in immediate succession with a long exposure time. In addition, image pairs (In1, Out1; In2, Out2; In3, Out3; ...) with different factors d can be recorded for a scene in order to learn a continuous spectrum for the improvement of vision depending on the parameter or factor d. The vehicle camera 2-i is preferably stationary (immobile) with respect to the surroundings to be recorded during the generation of the training data. For example, the training data can be recorded by means of a vehicle camera 2 - i of a stationary vehicle 10. The scene captured by the vehicle camera 2-i can in particular contain a static environment, that is to say without moving objects.

When the neural network CNN1 has been trained, the vision is improved according to the following scheme:

Input image Factor d - CNN1

CNN1 visually enhanced source / output image.

4 shows a system with a trained neural network CNN1 for improving visibility. The trained neural network CNN1 receives original input image data (Ini) from the multiple cameras 2-i as input. A factor d can be specified or determined by the neural network CNN1 itself on the basis of the input image data (Ini). The factor d specifies (controls) how strongly the input image data should be converted. The neural network calculates image data (Opti) with improved visibility from the multiple cameras 2-i. The optimized image data (Opti) of the multiple cameras 2-i are output.

FIGS. 5 and 6 show exemplary embodiments for possible combinations of a first network for improving visibility with one or more networks of the functions for driver assistance functions and automated driving.

5 shows a neural network CNN10 for improving the view of an input image (Ini), possibly controlled by a factor d, which Share feature representation layers (as input or lower layers) with a network for detection functions (fn1, fn2, fn3, fn4). The detection functions (fn1, fn2, fn3, fn4) are image processing functions that detect objects, structures, properties (generally: features) in the image data that are relevant for ADAS or AD functions. Many such detection functions (fn1, fn2, fn3, fn4), which are based on machine learning, have already been developed or are the subject of current developments (e.g. traffic sign classification, object classification, semantic segmentation, depth estimation, lane marking recognition and localization). Detection functions (fn1, fn2, fn3, fn4) of the second neural network CNN2 deliver better results on visually enhanced images (Opti) than on the original input image data (Ini) in poor visibility conditions. In the feature representation layers of the neural network CNN 10, common features for the improvement of vision and for the detection functions are learned.

The neural network CNN10 with divided input layers and two separate outputs has a first output CNN 11 for outputting the visually enhanced output / output image (Opti) and a second output CNN 12 for outputting the detections: objects, depth, trace, semantics, etc. .

Because the feature representation layers are optimized during training with regard to both the vision improvement and the detection functions (fn1, fn2, fn3, fn4), an optimization of the vision improvement also leads to an improvement in the detection functions (fn1, fn2, fn3, fn4).

If an output of the visually improved image (Opti) is not desired or not required, the approach can be varied further, as is explained with reference to FIG. 5.

FIG. 6 shows an approach based on the system of FIG. 5 for neural network-based vision improvement by optimizing the features. In order to save computing time, the features for the detection functions (fn1, fn2, fn3, fn4) are optimized during training with regard to improved visibility and with regard to the detection functions (fn1, fn2, fn3, fn4).

During runtime, ie when using the trained neural network (CNN10, CNN11, CNN 12), no visually enhanced images (Opti) are calculated. Nevertheless, the detection functions (fn1, fn2, fn3, fn4) - as already explained - are improved by the joint training of vision improvement and detection functions compared to a system with only one neural network (CNN2) for detection functions (fn1, fn2, fn3, fn4) , in which only the detection functions (fn1, fn2, fn3, fn4) have been optimized during training.

In the training phase, an additional output interface (CNN11) outputs the improved brightness image (Opti) and compares it with the ground truth (the corresponding training image with improved visibility). This output (CNN11) can continue to be used in the test phase or during runtime, or it can be cut off to save computing time. The weights for the detection functions (fn1, fn2, fn3, fn4) are modified in this training with the additional output (CNN11) so that they take into account the brightness improvements for the detection functions (fn1, fn2, fn3, fn4). The weights of the detection functions (fn1, fn2, fn3, fn4) thus implicitly learn the information about the brightness improvement.

Further aspects and embodiments of an assistance system based on surround view cameras, which algorithmically converts the image data, despite darkness and lack of color information, into a representation that corresponds to a recording with illumination or daylight are set out below. The converted image can either be used for display purposes only or as input for feature-based recognition algorithms.

1a) In a first embodiment, the calculation in a system is based, for example, on a neural network which, upstream of a detection or display unit, converts a very dark input image with little contrast and color information into a, for example, daylight representation. For this task, the neural network was trained with a data set consisting of “dark input images” and the associated “daylight images”.

This training can be carried out individually for each vehicle camera, so that a conversion takes place individually for each individual vehicle camera. Individual training for each vehicle camera offers the advantage of redundancy in the overall system. Depending on the type of training, the neural network can reproduce processes such as white balancing, gamma correction and histogram equalization in a very ideal way, and additional ones stored in the network structure Use information to automatically add missing color or contrast information. In this way, very dark images can be converted into a representation which is advantageous for feature-based recognition and viewing.

1b) In an alternative training, all vehicle cameras can also be trained at the same time. This would have the advantage that the network learns optimal parameters when training with dark images (for example for the side cameras) and bright images (for example for the front and rearview cameras) at the same time. When using several vehicle cameras, this leads to an adapted brightness and balance for all vehicle cameras, since the individual lighting profiles in the exterior are explicitly recorded and trained in the overall network.

During training, the specification of corresponding target output images (ground truth data) enables the network to learn to convert the different images from the individual vehicle cameras into images with optimized brightness and / or color distribution. For example, it can be considered optimal that the brightness or color distribution in the overlapping field of view of adjacent vehicle cameras are the same or almost the same. By using ground truth data for all vehicle cameras, which all correspond to the optimized image of the front camera in terms of brightness or color distribution, the converted images from all vehicle cameras are always ideal later when the trained neural network is used, in order to use stitching to create a composite image for the display unit to create.

In one embodiment, the training can provide ground truth data which are the result of an image simulation or an image synthesis. For example, the target output images from the side vehicle cameras can be simulated or synthesized taking real front and rear images into account.

2) In a further embodiment, this method can be integrated in a hardware-based image preprocessing stage, the ISP (Image Signal Processor). On the hardware side, this ISP is supplemented by a small, trained neural network, which carries out the corresponding conversion and provides the processed information with the original data for possible detection or display methods. 3) This method can be used in a further application in such a way that, as with a surround view system, it calculates images with different lighting profiles to form an overall image with balanced lighting. An example is the display of the vehicle surroundings on a display on an unlit country road, where the areas of the front and rear vehicle cameras are illuminated by headlights, but the lateral areas are not illuminated by headlights.

4) In a further embodiment, the system with the neural network can be trained to use information from the better-lit areas in order to further improve the conversion for the unlit areas. Here, the network is trained less with individual images for each vehicle camera, but rather as an overall system consisting of several camera systems.

5) In a further application, in addition to lighting information and image data, information on image quality can be made available to the network for training purposes. To this end, the system and the method can be optimized in such a way that it calculates image data optimized for computer vision and human vision.

6) In a further embodiment, the computing efficiency is optimized. No separate neural network is required for the reconstruction of the night images with improved visibility; instead, one of the operational networks in the vehicle is used for this purpose, e.g. a network for semantic segmentation, lane recognition, object recognition or a multi-task network.

One or more further output layer (s) is added to this network, which is responsible for the reconstruction of the vision-enhanced night images. During the training, the training data for the night vision improvement is used for the calculation of this initial layer.

During the runtime in the vehicle, two different versions can be implemented: a) The reconstruction of the visually enhanced recordings is made available to additional functions: Here, the output of the extended output layers is required. This output is calculated and the visually improved images are made available to the functions, e.g. for a display for the driver. b) The direct reconstruction of the vision-enhanced recordings is not required: Since a common network is used for the function (s) and the vision enhancement, in this case a significant improvement in the feature representation of the individual layers for the functions takes place during the training. This is achieved by the following network structure:

- the network layers are achieved both for the functions and for the improvement of the visibility,

- only the output layers of the functions and the vision enhancements contain separate neurons for the function or the visual enhancement, and

- The training of the network includes data for the function and the visibility improvement.

This setup makes it possible for the feature representation of the commonly used layers to contain information about the visibility improvement and for this information to be made available to the functions. At runtime, there is thus the possibility of using the network only for the calculation of the functions which for this purpose work on visualized feature representations. This is a computation-time-efficient implementation that is particularly suitable for operation on embedded systems.

The additional computational effort at runtime in this embodiment is either only the calculation of the initial layer (s) if the vision-enhanced night images are made available to other functions in the vehicle, or no additional computing effort if the vision enhancement is integrated into the algorithms of the functions and only the Output of these functions is further used, for example, lane recognition, object recognition, semantic segmentation and / or depth estimation.

7) For use in road traffic, the visibility improvement can be extended with regard to overexposure. With the methods described here, a common network for improving visibility can be learned, which enhances both the quality of overexposed and underexposed images. A fusion of these two applications in a network enables a computationally efficient implementation in the vehicle. Furthermore, the computational efficiency can be increased if this network fusion is also expanded to include functions such as object recognition, lane recognition, depth estimation, semantic segmentation.

8) At runtime, only the recordings of the vehicle cameras are required and optionally an estimate of the ratio / factor d, which describes the improvement in visibility achieved between the input image and the output image. This can be described, for example, by the ratio of the exposure time between the input and output image. Other ratios for measuring such a ratio are also conceivable.

In other words, based on the input data (= a picture with a short exposure time), the network calculates an image with improved visibility, which corresponds to a picture with a longer exposure time.

In the vehicle, the exposure time is limited, especially for night photos, so that the quality of the recording is not impaired by the driving speed or in curves (including motion blur). In combination with the proposed network for improving visibility, images with longer exposure times are calculated without being affected by e.g. the driving speed.

The estimate of the ratio (or factor d) can be obtained from the current image, the previous image or a sequence of images. An example of this is the change from an illuminated city center to the country road.

For this purpose, a control loop can be set up in which the properties of the luminance histogram are tracked. If there is a deviation from the mean expected distribution, the ratio can be increased or decreased.

This adjustment of the ratio is relevant during runtime in the vehicle. During training, the relationships between the exposure times are known, and the mean expected distribution for improved visibility can be calculated from the ground truth images of the training data for various scenes. The scene types can be obtained from the functions during runtime in the vehicle, e.g. semantic segmentation.

Thus, the ratio to the improvement in visibility during runtime a) can be a constant, b) depending on the luminance histogram, c) depending on the street scene, or d) depending on the luminance histogram and the street scene.

In summary, deviations from a fixed visibility improvement factor d can occur in dynamic scenes such as road traffic scenes. By regulating this factor, it is possible to improve the visibility for a large number of different traffic scenes during runtime in the vehicle.

The main advantages are: - Very efficient method for improving the image quality when it is insufficient

lighting

- Significant improvement in the image quality when displaying night images

- No additional lighting is required, which brightens areas of the vehicle such as the side areas with no lighting. This can represent a unique selling proposition for ADAS.

- Generation of an image data stream for human and computer vision from a network

In addition to motor vehicles, alternative areas of application are, for example, airplanes, ships, drones, buses and trains.

Claims

1. A method for machine learning of a conversion of input image data from a plurality of vehicle cameras (2-i) of an all-round vision system into optimized output image data by means of at least one artificial neural network (CNN1, CNN10, CNN11, CNN12), the learning with a plurality of training image pairs (In1, Out1; In2, Out2; In3, Out3; ...) takes place in such a way that at the input of the artificial neural network (CNN1, CNN10) a first image (In1, In2, In3, ... ) a first brightness or color distribution and a second image (Out1, Out2, Out3, ...) of the same scene with a different second brightness or color distribution is provided as the target output image.

2. The method according to claim 1, wherein the training image pairs (In1, Out1; In2, Out2; In3, Out3; ...) are generated for each of the vehicle cameras (2-i) by a first image (In1, In2 , In3, ...) with a first and a second image (Out1, Out2, Out3, ...) with a second brightness can be recorded simultaneously or in direct succession with different exposure times.

3. The method according to claim 1 or 2, wherein an artificial neural network (CNN1; CNN10, CNN11, CNN12) for all vehicle cameras (2-i) is trained together.

4. The method according to any one of the preceding claims, wherein at least one factor d is determined as a measure of the difference between the second and the first brightness of a training image pair (In1, Out1; In2, Out2; In3, Out3; ...) and the artificial neural network (CNN1, CNN10, CNN11, CNN12) is provided.

5. The method according to any one of the preceding claims, wherein the artificial neural network (CNN1, CNN10, CNN11, CNN12) has a common input interface for two separate output interfaces (CNN11, CNN 12), the common input interface having shared feature representation layers, with the first Output interface (CNN11) brightness-converted image data (Opti) are output, whereby ADAS-relevant detections of at least one ADAS detection function (fn1, fn2, fn3, fn4) are output at the second output interface (CNN12) and the outputs of both output interfaces (CNN11, CNN12) are optimized as part of the training.

6. A method for converting input image data from a plurality of vehicle cameras (2-i) of an all-round vision system into optimized output image data with the following steps: a) Input image data (Ini) recorded by the vehicle cameras (2-i) of a current brightness or Color distribution are provided to a trained artificial neural network (CNN1, CNN 10, CNN11, CNN 12), b) the trained artificial neural network (CNN1, CNN10, CNN11, CNN12) is configured to match the input image data (Ini) with the current To convert brightness or color distribution into optimized output image data (Opti) with a different output brightness or color distribution, and c) the trained artificial neural network (CNN1, CNN10, CNN11, CNN12) is configured to output the optimized output image data (Opti).

7. The method according to claim 6, wherein in step a) a factor d is additionally provided to the trained artificial neural network (CNN1, CNN10, CNN11, CNN12) and in step b) the conversion is controlled as a function of the factor d.

8. The method according to claim 7, wherein the factor d is estimated and, in the estimation, the brightness or color distribution of the current captured image data, the previously captured image data and / or the history of the factor d is taken into account.

9. The method according to claim 7 or 8, wherein a separate factor d is estimated or determined for each of the vehicle cameras (2-i).

10. Device (1) with at least one data processing unit (3) configured for converting input image data (Ini) from a plurality of vehicle cameras (2-i) of an all-round vision system into optimized output image data (Opti) comprising:

- an input interface which is configured to receive the input image data (Ini) of a current brightness or color distribution from the vehicle cameras (2-i),

- a trained artificial neural network (CNN1, CNN10, CNN11, CNN12) which is configured to convert the input image data (Ini) with the current brightness or color distribution into output image data (Opti) with a different output brightness or color distribution and - A first output interface (CNN1, CNN11) which is configured to output the optimized output image data (Opti).

11. The device (1) according to claim 10, wherein the data processing unit (3) is implemented in a hardware-based image preprocessing stage.

12. Device (1) according to claim 10 or 11, wherein the trained artificial neural network (CNN1, CNN10, CNN11) for brightness or color distribution conversion is part of a vehicle-side ADAS detection neural network (CNN2, CNN 12) with a shared input interface , and two separate output interfaces, the first output interface (CNN11) being configured to output the optimized output image data (Opti) and the second output interface (CNN12) being configured to output the ADAS-relevant detections.

13. Computer program element which, when a data processing unit is programmed with it, instructs the data processing unit to carry out a method according to one of claims 6 to 9.

14. Computer-readable storage medium on which a program element according to claim 13 is stored.

15. Use of a method according to claim 1 to 5 for training an artificial neural network (CNN1, CNN10, CNN11, CNN12) of a device (1) according to one of claims 10 to 12.