EP2839662A1 - Method for white balance of an image considering the colouring of the motor vehicle - Google Patents
Method for white balance of an image considering the colouring of the motor vehicleInfo
- Publication number
- EP2839662A1 EP2839662A1 EP13716288.9A EP13716288A EP2839662A1 EP 2839662 A1 EP2839662 A1 EP 2839662A1 EP 13716288 A EP13716288 A EP 13716288A EP 2839662 A1 EP2839662 A1 EP 2839662A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- colour
- motor vehicle
- image
- white balance
- offset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
- H04N23/88—Camera processing pipelines; Components thereof for processing colour signals for colour balance, e.g. white-balance circuits or colour temperature control
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
- H04N23/85—Camera processing pipelines; Components thereof for processing colour signals for matrixing
Definitions
- the invention relates to a method for white balance of an image of an environmental region of a motor vehicle captured by means of a camera of the motor vehicle. A region of the motor vehicle is depicted in the image.
- the white balance is performed by means of an electronic image processing device considering colour information related to a colouring of the motor vehicle.
- the invention relates to a camera system for a motor vehicle as well as to a motor vehicle with a camera system.
- Such a cast to the blue/green is based on the fact that the so-called grey-world assumption is used in the known automatic white balance methods, which implies that the averages of all of the colour channels - thus of the red, the green and the blue colour channel - are to result in a grey value on average.
- the colour components of the image - thus the red, green and blue components - can therefore be affected such that the averages of all of the colour channels are equal.
- the colour of the presented image then rather corresponds to reality.
- the printed matter JP 2007 189 595 A describes a method for white balance of an image considering the information about the colour of the motor vehicle, which is depicted in certain regions in the captured images.
- the colour of the motor vehicle is dynamically determined "online”, thus in the operation of the camera system based on the captured images.
- this type of white balance is associated with substantial disadvantages in particular with motor vehicles. Namely, the white balance is usually performed by means of an electronic image processing device in a vehicle camera, which is incorporated in the camera and disposed next to the image sensor. In vehicle cameras, it is sought to configure it particularly compact because relatively little installation space is available in motor vehicles - for instance in the side-view mirror.
- JP 2007 189 595 A requires a relatively high computational power and therefore a correspondingly large image processing device. Namely, the dynamic determination of the colouring of the motor vehicle based on the captured images is particularly
- a further solution of the above mentioned problem could be configured such that an image region without a depicted vehicle region is first determined in the captured image of the camera and the parameters of the white balance are determined exclusively based on this image region.
- the averages of the colour channels could be determined exclusively based on the image region, and the white balance of the entire image could be performed based on these averages.
- this solution is also relatively computationally intensive. This in particular for the reason that the position and the orientation of the camera on the motor vehicle can change in the operation of the motor vehicle due to mechanical as well as thermal deformations. Therefore, the position and the size of the depicted vehicle region in the image can also change. It is an object of the invention to demonstrate a solution how in a method of the initially mentioned type the white balance of the image can be particularly effectively performed without much computational effort.
- this object is solved by a method, by a camera system as well as by a motor vehicle having the features according to the respective independent claims.
- Advantageous implementations of the invention are the subject matter of the dependent claims, of the description and of the figures.
- a method according to the invention serves for white balance of an image captured by means of a camera of a motor vehicle. Besides an environmental region of the motor vehicle, a region of the motor vehicle itself is also depicted in the image.
- the white balance of the image is performed by means of an electronic image processing device considering colour information with respect to a colouring of the motor vehicle. According to the invention, it is provided that as the colour information for performing the white balance exclusively information stored in the image processing device is used.
- the colouring of the motor vehicle is determined in the operation of the camera - in the method according to the invention, the white balance is effected depending on stored colour information.
- This approach has diverse advantages over the prior art: the white balance can be performed without much computational effort and thus particularly fast. Because a lower computational power is required for the white balance compared to the prior art, a smaller image processing device can also be used, which can be incorporated in the camera. The entire camera can thus also be constructed in compact manner, which proves particularly advantageous in use in a motor vehicle.
- the camera can therefore be incorporated in an exterior mirror without much effort, in which only very little installation space is available, as is known.
- the decrease of the required computational power has the advantage that less heat is released by the image processing device, which could adversely affect the operation of the image sensor.
- the invention is based on the realization that a reliable white balance of the image can also be performed based on stored and thus constant colour information with respect to the colouring of the motor vehicle and dynamic determination of the colour of the motor vehicle in the operation of the camera is not required, because the actual colouring of the motor vehicle substantially remains constant over the entire lifetime of the motor vehicle.
- the method according to the invention proves particularly advantageous also with lower- priced camera systems, which do not have the required computational power.
- the method according to the invention can also represent a mere software solution such that the employment of additional hardware components for the white balance with the
- colour information related to the colouring of the motor vehicle is stored in the image processing device.
- This colour information is preferably stored in the image processing device in a development phase or a manufacturing process of the camera or of the motor vehicle, namely in particular in a non-volatile memory of the image processing device.
- this colour information is determined and stored in a calibration phase of the camera.
- a data value is understood by the colour information, in which the colouring of the motor vehicle is taken into account.
- the colour information can for example be a data value specifying the exact colour of the motor vehicle.
- the colour information can also be a data value used in the white balance and only indirectly including information about the colour of the motor vehicle.
- the colour information can for example be an offset applied to the average of at least one of the colour channels of the image within the scope of the white balance.
- the performance of the white balance of the image can include that respective colour averages of colour channels of the image - namely in particular of the entire image - are determined in a predetermined colour space. Then, an in particular negative offset can be stored as the colour information in the image processing device for at least one of the colour channels, which is applied to the colour average of this at least one colour channel within the scope of the white balance.
- This embodiment is based on the fact that usually the grey-world assumption is used for performing the white balance, in which it is assumed that all of the colour channels of the image are to result in a grey value on average.
- the colour average of at least one of the colour channels is increased compared to an image without the vehicle region - according to colouring of the motor vehicle - such that the white balance too provides an unnatural result.
- the present embodiment now takes the way to provide the average of at least one of the colour channels of the image with an associated offset and thus to computationally affect the determined colour average of the at least one colour channel.
- the at least one offset represents the stored colour information.
- an offset is applied to the colour average of that colour channel, which is included in the colour of the depicted region of the motor vehicle.
- the respective colour averages of both the red colour channel and the green colour channel are each provided with an offset. Because the yellow colour includes both red and green colour components.
- the respective offsets can also be selected differently dimensioned. Therein, the respective offset can for example be calculated during a calibration phase of the camera - for example in the manufacture of the motor vehicle.
- a further development includes that an offset table based on calibration results of the camera is determined and stored for the at least one colour average.
- different offset values can each be determined and stored for at least one colour channel for different conditions (for instance for different light conditions) in a calibration phase of the camera.
- These offset values can be stored in a so-called look-up table.
- the image processing device selects one of the offset values, namely depending on the current conditions.
- the grey-world assumption (thus the grey-world method) can be used.
- respective gain factors are determined for the colour channels, by which the colour averages of all of the colour channels each provided (optionally) with an offset are balanced.
- this is based on the assumption that the colour averages of all of the colour channels are to result in a grey value.
- an image with natural colours can be produced, which correspond to the colours of the actual environmental scenario.
- the averages of the colour channels are determined in the RGB colour space (Red, Green, Blue) and the at least one offset is applied in the RGB colour space.
- the image sensors of cameras usually produce images in the RGB colour space such that the corrections of the colour average of at least one of the colour channels with the offset can also already be performed in the RGB colour space.
- the white balance can already be performed by means of the internal image processing device of the camera such that the camera outputs images already corrected in colour. Before the images are output, they can be translated from the RGB colour space to the YUV colour space in the camera.
- one or two of the offsets is/are zero. Namely, the offset is applied to the colour average of that colour channel that is included in the colour of the motor vehicle - as already explained. If one of the offsets is unequal to zero, thus, it is in particular a negative value such that the magnitude of the offset is subtracted from the respective average.
- the above mentioned formula is based on the grey-world assumption, and by the gain factors it is allowed that all of the colour channels of the image result in a grey value on average and additionally all of the colour channels are equal. Especially thereby, it can be achieved that the colours of the image presented on a display correspond to the natural colours of the environment.
- the gain factors are preferably dynamically adjusted in the operation of the camera, thus in real time, such that the above mentioned equation is satisfied.
- This adjustment of the gain factors is thus performed in real time, while the simultaneously captured images of the camera are for example displayed on a display.
- the adjustment of the gain factors can be configured such that first all of the gain factors are set to one and then adjusted to the final values via several iterations.
- the colour information namely in particular the at least one offset
- the colour information can be determined and stored in the image processing device in a manufacturing process of the motor vehicle or else in a development phase of the camera.
- this determination is effected in a calibration session of the camera.
- the determination of the offset is effected considering the size of the region of the motor vehicle depicted in the image of the camera. Namely, the size of the depicted region of the motor vehicle has influence on the colour average of the associated colour channel such that a precise offset can be determined considering the size of the depicted vehicle region, which provides for effective white balance of the image.
- the image can be translated from the RGB colour space to the YUV colour space.
- Image sensors usually capture images in the RGB colour space, which are then translated to the YUV colour space and output by the camera in this YUV colour space.
- the so-called colour matrix is used for translating the image.
- at least one matrix component of the colour matrix is determined depending on the stored colour information within the scope of the white balance.
- an offset can also be stored as the colour information, which is applied to at least one component of the colour matrix.
- the colour is green or blue, while it causes a red colour cast with a positive V value.
- the V value can be increased with the aid of a corresponding offset in the colour matrix. This approach too provides for a natural white balance without much computational effort, because only one offset has to be defined and applied to the at least one component of the colour matrix. This embodiment can even be combined with the above described compensation in the RGB colour space.
- the invention relates to a camera system for a motor vehicle.
- the camera system includes a camera for capturing an image of an environmental region of the motor vehicle, wherein a region of the motor vehicle itself is depicted in the image.
- the camera system also includes an electronic image processing device formed for performing a white balance of the image considering colour information related to a colouring of the motor vehicle.
- the image processing device can also be incorporated in the camera.
- the colour information is stored in the image processing device, and the image processing device is configured to use solely the stored information as the colour information for performing the white balance.
- a motor vehicle according to the invention has a camera system according to the invention.
- FIG. 1 in schematic illustration a motor vehicle with a camera system according to an embodiment of the invention.
- Fig. 2 in schematic illustration an image of an environmental region of the motor vehicle captured by means of a camera of the camera system.
- a motor vehicle 1 illustrated in Fig. 1 is for example a passenger car.
- the motor vehicle 1 includes a camera system 2 having a plurality of cameras 3 in the embodiment, which are disposed distributed on the motor vehicle 1 . Therein, the cameras 3 each capture an environmental region of the motor vehicle 1 and overall for example the entire
- the number as well as the arrangement of the cameras 3 are only exemplarily illustrated in Fig. 1 ; the number as well as the
- respective cameras 3 are incorporated in the two exterior mirrors of the motor vehicle 1 .
- a camera 3 can be disposed in a front region of the motor vehicle 1 , and a camera 3 can also be disposed in the rear region of the motor vehicle 1 .
- All of the cameras 3 are electrically coupled to a controller 4, which is a central signal processor of the camera system 2.
- the controller 4 therefore receives the images of all of the cameras 3 and can process these images.
- the controller 4 is electrically coupled to a display 5 disposed in the interior of the motor vehicle 1 . Images can be displayed on the display 5, which are based on the images of the cameras 3.
- the controller 4 can produce an image from the received images of all of the cameras 3, which shows the motor vehicle 1 and its environment from a bird's eye view. Such a plan view of the motor vehicle 1 and its environment is also referred to as a bird eye view.
- the cameras 3 are video cameras, which are each able to capture a plurality of images per unit time. Thus, the cameras 3 can each capture a temporal sequence of images and output it to the controller 4.
- the cameras 3 can also be so-called fish-eye cameras, and they can have a relatively wide capturing angle, for instance in a range of values from 150 ° to 200 °.
- each camera 3 also has an internal image processing device (not illustrated in the figures), which conditions the captured images and transmits them to the controller 4. For example, translation of the image from the RGB colour space to the YUV colour space is performed by means of the internal image processing device. In addition, a white balance of the images is performed by means of the internal image processing device, which is described in more detail below. In addition, the internal image processing device can also perform the so-called black level correction of the image.
- the central controller 4 can also take the function of the internal image processing device.
- FIG. 2 An exemplary image 6 of one of the cameras 3 (for example a camera 3 disposed in the exterior mirror) is schematically illustrated in Fig. 2. As is apparent from Fig. 2, a region 7 of the own motor vehicle 1 is also depicted in the captured image. In addition, a road 8 as well as another vehicle 9 are depicted.
- the internal image processing device of the camera 3 can perform the white balance according to the grey-world method. This implies that first respective colour averages F ge, G avg e and B avg e of the colour channels R, G, B of the image 6 are determined. According to the grey-world assumption, the colour channels R, G, B are to result in a grey value on average. Based on this, the white balance is performed according to the following equation:
- R ga i n , G ga in and B ga i n denote respective gain factors of the red, green and blue colour channel, respectively.
- the gain factors R gain , G ga in and B ga i n are determined, by which the respective colour channels are multiplied.
- the displayed image can have very different colour casts in the white balance according to the grey-world assumption, namely according to the colouring of the motor vehicle.
- This is attributable to the fact that the region 7 of the motor vehicle 1 is also depicted in the captured image 6 and therefore the performance of the white balance is adversely affected, because at least one of the colour averages R avge , G avg e, B avge is influenced by the colouring of the motor vehicle 1 .
- the colour average R avge of the red colour channel is unnaturally increased due to the red colouring of the motor vehicle 1 , which results in a blue and a green colour cast being seen in the presented images 6.
- an offset for the colour averages is stored in the image processing device of the camera 3, which is applied to at least one of the colour averages in performing the white balance.
- the above equation is complemented as follows:
- At least one of the offsets R 0 ff S et, G 0 ff S et, Boffset can also be equal to zero.
- two offsets in the above equation can be zero.
- this is dependent on the colour of the motor vehicle 1 .
- offsets for those colour channels are used, which are really included in the colour of the motor vehicle 1 as basic colours. This for example implies that with a red motor vehicle 1 , only an offset FUset for the red colour channel is used.
- respective offsets FUset, G 0 ff Se t are used both for the red colour channel and for the green colour channel.
- the motor vehicle 1 is blue, thus, only an offset B ga i n for the blue colour channel is used.
- an offset is used for that colour component, which is required for emergence of the colour of the motor vehicle 1 .
- the offsets FUset, G 0 ff S et, B 0 ff Se t are negative values. This implies that the magnitude of the offset is subtracted from the respective colour average R avge , G avge , B avge . Thus, the colour cast due to the colouring of the own motor vehicle 1 can be compensated for.
- the respective colour averages R avge , G avg e and B avge are extracted from the images of the camera 3 and thus determined in the operation of the camera 3.
- the gain factors R gain , Ggain and B gain too are determined suonline" in the operation of the camera 3. This is effected in real time and thus also at the same time with the presentation of the images on the display 5.
- the gain factors R gain , G ga in and B ga i n are determined. This determination of the gain factors is described in more detail below:
- R' gain , G' gain , B' gain denote the respective gain factors from the last iteration.
- H max denotes the maximum value from all of the colour averages R avge , G avge , B avge . For example, if the colour average R avg e is larger than the other colour averages G a vge, B avge , then it applies:
- the offsets R 0 ff S et, G 0 ff S et, B 0 ff Se t represent colour information in terms of the present invention and are therefore stored in the camera.
- an offset can also be stored as the colour information, with which a component of the colour matrix is provided, by means of which the image 6 is translated from the RGB colour space to the YUV colour space.
- the image 6 is translated from the RGB colour space to the YUV colour space.
- V MI R + M2 G + M3 B
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Color Television Image Signal Generators (AREA)
Abstract
Description
Claims
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| DE102012007982A DE102012007982A1 (en) | 2012-04-20 | 2012-04-20 | Method for white balance of an image taking into account the color of the motor vehicle |
| PCT/EP2013/057741 WO2013156417A1 (en) | 2012-04-20 | 2013-04-12 | Method for white balance of an image considering the colouring of the motor vehicle |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| EP2839662A1 true EP2839662A1 (en) | 2015-02-25 |
Family
ID=48095856
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| EP13716288.9A Ceased EP2839662A1 (en) | 2012-04-20 | 2013-04-12 | Method for white balance of an image considering the colouring of the motor vehicle |
Country Status (3)
| Country | Link |
|---|---|
| EP (1) | EP2839662A1 (en) |
| DE (1) | DE102012007982A1 (en) |
| WO (1) | WO2013156417A1 (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DE102014222780A1 (en) | 2014-11-07 | 2016-05-12 | Robert Bosch Gmbh | Camera system and vehicle comprising a camera system |
| JP6354578B2 (en) * | 2014-12-26 | 2018-07-11 | 株式会社Jvcケンウッド | Imaging system |
| DE102020125232A1 (en) | 2020-09-28 | 2022-03-31 | Continental Automotive Gmbh | Color correction method for a camera system and a camera system |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2011000392A1 (en) * | 2009-07-02 | 2011-01-06 | Hi-Key Limited | Method and camera system for improving the contrast of a camera image |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH07162890A (en) * | 1993-12-08 | 1995-06-23 | Matsushita Electric Ind Co Ltd | White balance adjustment device |
| JP5073207B2 (en) | 2006-01-16 | 2012-11-14 | 株式会社日立アドバンストデジタル | In-vehicle camera device |
| CN102342089A (en) * | 2009-03-02 | 2012-02-01 | 弗莱克斯电子有限责任公司 | Calibration techniques for camera modules |
-
2012
- 2012-04-20 DE DE102012007982A patent/DE102012007982A1/en active Pending
-
2013
- 2013-04-12 EP EP13716288.9A patent/EP2839662A1/en not_active Ceased
- 2013-04-12 WO PCT/EP2013/057741 patent/WO2013156417A1/en not_active Ceased
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2011000392A1 (en) * | 2009-07-02 | 2011-01-06 | Hi-Key Limited | Method and camera system for improving the contrast of a camera image |
Also Published As
| Publication number | Publication date |
|---|---|
| DE102012007982A1 (en) | 2013-10-24 |
| WO2013156417A1 (en) | 2013-10-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP4325642B2 (en) | Mobile camera system | |
| JP5971207B2 (en) | Image adjustment apparatus, image adjustment method, and program | |
| US10166921B2 (en) | Brightness and color matching video from multiple-camera system | |
| US7969480B2 (en) | Method of controlling auto white balance | |
| JP2009017020A (en) | Image processing apparatus and display image generation method | |
| US20140036106A1 (en) | Image processing apparatus and image processing method | |
| CN103986875A (en) | Image acquiring device, method and terminal and video acquiring method | |
| US9214034B2 (en) | System, device and method for displaying a harmonized combined image | |
| JP2012054827A (en) | Image processor, image processing method and program | |
| KR20110016812A (en) | Imaging Device and Imaging Method | |
| JP5195841B2 (en) | On-vehicle camera device and vehicle | |
| CN110915204B (en) | Image processing apparatus, image processing method, storage medium, and imaging apparatus | |
| US11831995B2 (en) | Image capturing apparatus, image capturing method, and carrier means | |
| EP2839662A1 (en) | Method for white balance of an image considering the colouring of the motor vehicle | |
| WO2014118167A1 (en) | Method for white balance of an image presentation and camera system for a motor vehicle | |
| CN101510955B (en) | Image pickup device | |
| JP2010154478A (en) | Compound-eye imaging apparatus and method for generating combined image thereof | |
| JP4862321B2 (en) | In-vehicle camera device | |
| CN111355942B (en) | Semiconductor device, image processing system, method, and computer-readable storage medium | |
| US20180069998A1 (en) | Imaging apparatus, imaging system, and vehicle | |
| JP2006173988A (en) | In-vehicle camera image presentation apparatus and in-vehicle camera image presentation method | |
| KR20180028354A (en) | Method for displaying image in multiple view modes | |
| JP2009017301A (en) | Signal processing device | |
| JP2009506722A (en) | Improved color difference filter for white balance statistics calculation | |
| EP4622285A1 (en) | Method for synchronization of a multi camera device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
| 17P | Request for examination filed |
Effective date: 20141014 |
|
| AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
| AX | Request for extension of the european patent |
Extension state: BA ME |
|
| DAX | Request for extension of the european patent (deleted) | ||
| 17Q | First examination report despatched |
Effective date: 20151221 |
|
| REG | Reference to a national code |
Ref country code: DE Ref legal event code: R003 |
|
| APAF | Appeal reference modified |
Free format text: ORIGINAL CODE: EPIDOSCREFNE |
|
| APBK | Appeal reference recorded |
Free format text: ORIGINAL CODE: EPIDOSNREFNE |
|
| APBN | Date of receipt of notice of appeal recorded |
Free format text: ORIGINAL CODE: EPIDOSNNOA2E |
|
| APBT | Appeal procedure closed |
Free format text: ORIGINAL CODE: EPIDOSNNOA9E |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION HAS BEEN REFUSED |
|
| 18R | Application refused |
Effective date: 20171114 |