DE102020125232A1 - Color correction method for a camera system and a camera system - Google Patents

Abstract

Method for color correction for a camera system of a vehicle (1), in which the camera system has several cameras (3a, 3b, 3c, 3d) arranged on/in the vehicle (1) for detecting the surroundings, the cameras (3a, 3b, 3c, 3d) each have a field of view (6a, 6b, 6c, 6d) for generating a camera image, with the camera images being color corrected by selecting a reference value that is in at least one field of view (6a, 6b, 6c, 6d) of the cameras (3a, 3b, 3c, 3d), and the color of the camera images is corrected using the reference value.

Description

The present invention relates to a method for color correction for a camera system, in particular a surround view camera system, for a vehicle and a corresponding camera system, in particular surround view camera system, which achieves improved color correction or color harmonization.

Technological background

Vehicles are increasingly being equipped with driver assistance systems that support the driver in carrying out driving maneuvers. In addition to radar sensors, lidar sensors, ultrasonic sensors and/or camera sensors, these driver assistance systems also include, in particular, surround view camera systems that allow the vehicle environment to be displayed to the driver of the vehicle. Such surround view camera systems generally include several vehicle cameras in the front, rear and side areas, which provide real images of the vehicle environment, which are combined in particular by a data processing unit of the surround view camera system to form an image of the vehicle environment. The image of the vehicle surroundings can then be displayed to the driver on a display unit (such as the display of the navigation system). In this way, the driver can be assisted during a vehicle maneuver, for example when reversing the vehicle or during a parking maneuver.

In the area of surround view systems, there are functions such as "Bowl" and "Top View" ("bird's eye view" or "top view"), in which images or textures from the surround view cameras are combined or lined up seamlessly (stitching). The images or textures of the surround view cameras generally have overlapping regions or overlapping areas. However, the color in the overlapping regions does not merge or blend seamlessly when stitched. In these cases, color harmonization approaches are typically used to ensure a seamless transition of color from one camera texture to the next, as such color changes in the display are often perceived as distracting, which can also affect driver action. Accordingly, color correction is of particular importance for the development and operation of surround view systems.

Conventional color correction approaches in surround view systems use white balance correction, i. H. separately for each surround view camera. This approach is successful in many cases, since the color temperature of the respective scene is the same with the different cameras. However, there are cases where auto white balance can produce different results on different cameras, such as B. when the color temperature varies in the scene or the image statistics do not match the cameras due to scene variation and camera placement.

Printed state of the art

From the DE 10 2004 020 682 A1 an image recording system for a vehicle with a camera with white balance is known, in which vehicle data and/or environment data, the origin of which does not lie in the video image captured by the color camera, for the color correction or the white balance of the camera image. This produces a color image that is free from color variations caused by onboard lighting and/or ambient lighting.

the DE 11 2012 004 641 T5 discloses an imaging system for a vehicle that includes multiple cameras for detecting surroundings and performs color correction using a color correction matrix algorithm with a 3×3 color correction matrix and a histogram algorithm for the captured image data.

the DE 11 2017 008 125 T5 discloses an image synthesizer having multiple cameras whose images are combined into one image. Based on the difference between the calculated mean values of brightness and hue of the images from different cameras to be joined, a brightness and color correction is performed on all pixels in the areas of the junction.

the DE 10 2018 113 281 A1 relates to a method for image harmonization of at least two images of a camera system, in which a brightness and color correction is applied for multi-camera brightness and color harmonization in order to achieve improved harmonization with respect to brightness and/or color and of images taken from different Cameras of a motor vehicle were recorded to allow.

From the DE 10 2018 118 996 A1 a surround-view camera system is known in which the color is harmonized using corresponding color correction values for the respective cameras. In order to harmonize or to determine corresponding color correction values, the camera images are compared with regard to their chromi nanz analyzed, whereby the UV color components of the camera images are analyzed.

Object of the present invention

The present invention is therefore based on the object of making available a method for a surround view system and a corresponding surround view system, by means of which improved color correction can be achieved.

solution of the task

The above object is achieved by the entire teaching of claim 1 and the independent claims. Expedient developments of the invention are claimed in the dependent claims.

In the method according to the invention for color correction for a (surround view) camera system of a vehicle (1), the camera system has a number of cameras (3a, 3b, 3c, 3d) or surround view cameras arranged on/in the vehicle (1) for detecting the surroundings, each of the cameras (3a, 3b, 3c, 3d) having a respective field of view (6a, 6b, 6c, 6d) for generating a camera image. In this case, the camera images are color corrected in a simple manner by selecting a reference value (as a kind of calibration point, so to speak, whose properties or color values are known), which is located in at least one field of view of the cameras. Following this, the camera images are then corrected using the reference value. This results in the advantage that improved color correction or color harmonization can be achieved. Furthermore, such an implementation can be done with low complexity and real-time application without extensive use of processing and memory resources.

The reference value is preferably the color of the vehicle or a color sample; in particular, parts of the vehicle body or the vehicle body, the vehicle doors, side mirrors, hood and the like can be used as the origin for the reference. This has the advantage that the color is "known to the system", i. H. that their parameters can be used as a kind of zero setting or reset setting, since they should not change. Provision can also be made for the system to be calibrated afterwards, in that the parameters of the reference value, e.g. B. be coordinated as part of a maintenance or a self-test. Furthermore, in the case of strong deviations, there can also be an indication of soiling, so that the driver z. B. the degree of contamination can also be communicated.

Furthermore, a number of reference values can also be provided, in which case the corrections are then carried out either in parallel with one another or one after the other (so to speak, jointly or step by step). Alternatively or additionally, the reference value can also be a color marking and/or a color of a traffic sign and/or a color of a traffic light sign and/or a lane marking.

According to a preferred embodiment of the invention, the correction includes an adjustment of the color temperature, i. H. especially a white balance. However, other color values or brightness or contrast can also be adjusted/corrected.

Expediently, the fields of view can at least partially overlap in so-called overlapping areas. This ensures that there are no "blind" areas between the camera fields of view, so that an all-round view or at least a continuous view in a certain area is guaranteed.

In a preferred manner, the camera images of the cameras in whose field of view the reference value is located can first be corrected using the reference value. For example, the cameras arranged on the side, as their field of view usually also includes part of the vehicle body (e.g. door or fender).

A first histogram is preferably created from the corrected camera images of the cameras in whose field of view the reference value is located, and a second histogram for the cameras in whose field of view the reference value is not visible.

The second histogram for the overlapping areas can then advantageously be adjusted using the first histogram for color correction.

In addition or subordinately, the invention also claims a method for color correction for a camera system of a vehicle, which is characterized by the following method steps: color correction of the camera images of the cameras in whose field of view the reference value, in particular the vehicle body of the vehicle, is visible; creating a first histogram from the corrected camera images; Creating a second histogram for the cameras in whose field of view the reference value, in particular the vehicle body, is not visible, and adjusting the second histogram for overlapping areas of the fields of view using the first histogram. The result of this is that the camera images in which the reference value (preferably the vehicle body) is visible are first corrected be adjusted, in particular white-balanced, and then the camera images in which the vehicle body is not visible are corrected or compared using the comparison values or the histogram of the initially corrected camera images. As a result, a particularly simple method for color correction that can be implemented with little effort is proposed, which can improve the coloring of the camera images to a particular extent using a histogram-based approach. Due to the fact that the second or the second histograms are only corrected for overlapping areas of the fields of view using the first histogram, computing capacity can be saved to a particular extent.

According to an advantageous embodiment of the invention, the first histogram and/or the second histogram can also only be created for the overlapping areas. As a result, computing capacity can be saved to a particular extent.

Furthermore, the present invention claims a camera system for a vehicle for generating an image of a vehicle environment, which comprises a plurality of (surround view) cameras for detecting the environment and at least one means for controlling the cameras. The camera system is preferably a physical object, with the individual components (cameras and control means) being wired to one another, e.g. B. can communicate via signal lines or wirelessly. The control means can be designed as a control device (ECU—Electronic Control Unit, computer or the like) provided specifically for the cameras or as part of an overall control system (e.g. integrated circuit, chip or microcontroller), so to speak, as part of a higher-level unit. The camera system is characterized in particular in that a color correction is carried out using a method according to the invention.

character list

• 1 eine vereinfachte schematische Darstellung eines Fahrzeuges mit einem erfindungsgemäßen Surroundview-System zur Erzeugung eines Bildes der Fahrzeugumgebung;
• 2 eine stark vereinfachte schematische Darstellung einer Ausgestaltung der Sichtfelder der Surroundview-Kameras aus 1, sowie
• 3 eine Darstellung einer vorteilhaften Ausgestaltung eines Verfahrensablaufs zur Identifizierung von Fahrbahnmarkierungen bzw. Fahrbahnrandmarkierungen.

In the following, the invention is described in more detail with reference to expedient exemplary embodiments. Show it:

• 1 a simplified schematic representation of a vehicle with a surround view system according to the invention for generating an image of the vehicle surroundings;
• 2 shows a highly simplified schematic representation of an embodiment of the fields of view of the surround view cameras 1 , as
• 3 a representation of an advantageous embodiment of a method sequence for identifying road markings or road edge markings.

Reference number 1 in 1 refers to a vehicle with a control device 2 (ECU, Electronic Control Unit or ADCU, Assisted and Automated Driving Control Unit), which can access various actuators (e.g. steering, engine, brake) of vehicle 1 in order to control processes of vehicle 1 to be able to execute. Furthermore, the vehicle 1 has several surround view cameras in the front area (camera 3a), on the sides (camera 3b, 3c) and in the rear area (camera 3d) for detecting the surroundings. In addition, vehicle 1 has an additional camera sensor 4 and a lidar sensor 5 in the area of the windscreen. In addition, other sensors such. B. radar or ultrasonic sensors may be provided (not shown for clarity). The sensor data from camera sensor 4, lidar sensor 5 and the other sensors that can be optionally provided can then be used for environment and object recognition. As a result, various assistance functions, such as As parking assistants, emergency brake assistant (EBA, Electronic Brake Assist), distance following control (ACC, Automatic Cruise Control), lane keeping control or a lane departure warning system (LKA, Lane Keep Assist) or the like can be realized. In a practical manner, the assistance functions can also be carried out via the control device 2 or a separate control device.

The (surround view) cameras 3a-3d are part of a surround view camera system, which is preferably controlled by the control device 2 (alternatively, a separate control can be provided, for example) and which provides a full 360-degree view all around Vehicle 1 offers by changing the fields of view of the individual surround view cameras, e.g. B. 120 degrees, to form an overall view or an overall picture. As a rule, four cameras 3a-3d are used, as in 1 shown, the z. B. are arranged in the front and rear area and on the side mirrors. Alternatively, however, three, six, eight, ten or more surround view cameras can also be provided. In addition, fish-eye cameras or cameras with fish-eye lenses are preferably used in order to span the widest possible detection angle. Due to the simple environmental monitoring, e.g. B. the blind spot, such a camera system has numerous advantages in many everyday situations. Through the camera system, the driver can thus see different angles of the vehicle 1, z. B. on a display unit or a display, shown or displayed to z. B. to monitor a blind spot when changing lanes or parking.

The individual fields of view 6a-6d of the cameras 3a-3d are in 2 shown in a highly simplified form (rectangular representation of the fields of view, which can also have other shapes such as elliptical, semicircular or may have a different expanding shape). The fields of view 6a-6d overlap in some areas (overlapping areas 7a-7d), in the area of which undesirable effects such as e.g. B. color changes. For this purpose, for example, color harmonization is used to minimize or eliminate these effects. A problem with the color harmonization of independently white-balanced surround view cameras according to the prior art is that it is not possible to individually select which of the cameras is used as the reference image. A favorable and reliable approach here is the inventive placement of an object with a known color in the scene as a neutral reference. The details of the color can then be passed as input to the white balance algorithm, allowing it to correct the color accordingly. As a neutral reference, the color of the vehicle body or the vehicle body is defined as input for the correction of the white balance of the side cameras.

The method according to the invention accomplishes this using the following method steps: First, the white balance is corrected for the side cameras in which the vehicle body is visible. A histogram is then calculated from the corrected side camera texture for the overlapping regions. The histogram can then be calculated from the front and rear camera texture for the overlapping regions. Subsequently, the histograms of the side camera textures (particularly the left and right camera textures) can then be matched to the histograms of the front and rear camera textures for the overlapping regions, e.g. B. by applying histogram matching techniques known in the art.

In a first step, white balance is carried out first. For this purpose, the cameras are first selected in whose field of view there is a suitable reference value. As a rule, the vehicle body or the vehicle body is visible in the cameras 3b, 3c arranged on the side. A color sample can be selected from the expected vehicle body surface. This can be done, for example, based on the vehicle geometry and the extrinsic camera parameters, so that z. B. a region in the vehicle door can be selected as a reference. The reference value can be determined from a mean value or the median of the collected values.

• - Festlegung von Beispielpunkten in den Fahrzeugkoordinaten,
• - Konvertieren der Beispielpunkte in Kamerakoordinaten bzw. Bildkoordinaten mithilfe einer Sichtmatrix (camera extrinsic parameter),
• - Erstellen von Kamerastrahlen, um die konvertierten Beispielpunkte zu normalisieren, und
• - Abbilden der Kamerastrahlen auf die Bildkoordinaten, z. B. mithilfe eines Fischaugen-Kameramodells (camera intrisic parameter).

In the next step, the histogram is then calculated from regions of the input image that correspond to the overlapping regions or overlapping areas. This can be done using the following process steps:

• - definition of example points in the vehicle coordinates,
• - Converting the sample points into camera coordinates or image coordinates using a view matrix (camera extrinsic parameter),
• - create camera rays to normalize the converted sample points, and
• - Mapping of the camera beams to the image coordinates, e.g. B. using a fisheye camera model (camera intrinsic parameter).

Alternatively or additionally, it is also conceivable to use other neutral input values for the white balance, such as e.g. B. Street signs, lane markings, directional arrows, traffic light colors or color targets to be set manually. These additional inputs can make the method even more robust and reliable. In 3 an advantageous embodiment for identifying roadway markings or roadway edge markings (e.g. lateral roadside strips) is shown, which can be used to determine the reference value. The identification algorithm is first started (start 10), with all cameras being queried or addressed in succession, ie the image data from the cameras are analyzed in succession (query for one of the surround view cameras 11). The analysis is carried out as to whether a ground sign or a lane marking was recognized (identification of a lane marking 12) In the event that query 13 shows that there is a lane marking (answer YES 14), in a further step (query 16) the type of the road marking (e.g. based on a comparison with patterns or measurement parameters stored in the system) to determine whether it is a road marking. If it is a lane edge marking (answer YES 17), an attempt is then made to identify the lane edge marking with another of the surround view cameras, i.e. the algorithm is ended (END 19) and starts for another of the surround view cameras from in front. In addition, the algorithm is also started again from query 11 if queries 13 and 16 show that no lane markings or lane edge markings were detected (response: NO 15). If a lane marking or lane edge marking has been detected, this can now be used as a reference value (eg the white or yellow color of the marking). This identification approach assumes that the ground signs at the roadside are visible on all cameras and that the extrinsic camera parameters are given in each camera image or frame will give. Furthermore, it is a particularly cost-effective identification approach that can also be used in the same way for other reference values to be identified (street signs, body color, color markings to be defined manually, and the like).

Reference List

1

vehicle

2

control device

3a

camera (front)

3b

Camera (left side)

3c

Camera (right side)

3d

camera (rear)

4

camera sensor

5

lidar sensor

6a

Field of view (from camera 3a)

6b

Field of view (from camera 3b)

6c

Field of view (from camera 3c)

6d

Field of view (from 3d camera)

7a

Overlap area (of viewport 6a and 6b)

7b

Overlap area (of view fields 6a and 6c)

7c

Overlap area (of view fields 6b and 6d)

7d

Overlap area (of viewport 6c and 6d)

10

begin

11

Query for one of the cameras

12

Identification of a lane marking

13

Query: Was a lane marking detected?

14

YES

15

NO

16

Query: Is it a lane edge marking?

17

YES

18

Identification of the lane marking (using another of the cameras)

19

END

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 102004020682 A1 [0005]
• DE 112012004641 T5 [0006]
• DE 112017008125 T5 [0007]
• DE 102018113281 A1 [0008]
• DE 102018118996 A1 [0009]

Claims (11)

Method for color correction for a camera system of a vehicle (1), in which the camera system has several cameras (3a, 3b, 3c, 3d) arranged on/in the vehicle (1) for detecting the surroundings, the cameras (3a, 3b, 3c, 3d) each have a field of view (6a, 6b, 6c, 6d) for generating a camera image, wherein the color correction of the camera images is done by a reference value is selected which is located in at least one field of view (6a, 6b, 6c, 6d) of the cameras (3a, 3b, 3c, 3d), and the color correction of the camera images is based on the reference value. procedure after claim 1 , characterized in that the reference value is the color of the vehicle body. procedure after claim 1 or 2 , characterized in that the reference value is a color marking and/or a color of a traffic sign and/or a color of a traffic light sign and/or a lane marking. Method according to one of the preceding claims, characterized in that the color correction includes an adjustment of the color temperature. Method according to one of the preceding claims, characterized in that the fields of view (6a, 6b, 6c, 6d) overlap at least partially in overlapping areas (7a, 7b, 7c, 7d). Method according to one of the preceding claims, characterized in that the camera images of the cameras (3b, 3c) in whose field of view (6b, 6c) the reference value is located are first corrected using the reference value. procedure after claim 6 , characterized in that a first histogram from the corrected camera images of the cameras (3b, 3c) in whose field of view (7b, 7c) the reference value is located, and a second histogram for the cameras (3a, 3d) in whose field of view ( 7a, 7d) the reference value is not visible. procedure after claim 7 , characterized in that the second histogram for overlapping areas (7a, 7b, 7c, 7d) is adjusted based on the first histogram for color correction. Method for color correction for a camera system of a vehicle (1), in particular according to one of the preceding claims, characterized in that the method comprises the following method steps: correction of the camera images of the cameras (3b, 3c) in their field of view (6b, 6c) the reference value of the vehicle (1) is visible, creating a first histogram from the corrected camera images, creating a second histogram for the cameras (3a, 3d) in whose field of view (6a, 6d) the reference value is not visible, and tuning the second histogram, in particular for Overlapping areas (7a, 7b, 7c, 7d) of the fields of view (6a, 6b, 6c, 6d) based on the first histogram. procedure after claim 6 and one of the Claims 7 or 8th or after claim 9 , characterized in that the first histogram and/or the second histogram is/are created only for the overlapping areas (7a, 7b, 7c, 7d). Camera system for a vehicle (1) for generating an image of a vehicle environment, having a plurality of cameras (3a, 3b, 3c, 3d) for detecting the environment, and at least one control means, characterized in that the camera system is designed to carry out a color correction using a Carry out method according to any one of the preceding claims.

Images