EP2839662A1 - Method for white balance of an image considering the colouring of the motor vehicle - Google Patents

Method for white balance of an image considering the colouring of the motor vehicle

Info

Publication number
EP2839662A1
EP2839662A1 EP13716288.9A EP13716288A EP2839662A1 EP 2839662 A1 EP2839662 A1 EP 2839662A1 EP 13716288 A EP13716288 A EP 13716288A EP 2839662 A1 EP2839662 A1 EP 2839662A1
Authority
EP
European Patent Office
Prior art keywords
colour
motor vehicle
image
white balance
offset
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
EP13716288.9A
Other languages
German (de)
French (fr)
Inventor
Patrick Eoghan Denny
Brian Michael Thomas DEEGAN
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Connaught Electronics Ltd
Original Assignee
Connaught Electronics Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Connaught Electronics Ltd filed Critical Connaught Electronics Ltd
Publication of EP2839662A1 publication Critical patent/EP2839662A1/en
Ceased legal-status Critical Current

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80Camera processing pipelines; Components thereof
    • H04N23/84Camera processing pipelines; Components thereof for processing colour signals
    • H04N23/88Camera processing pipelines; Components thereof for processing colour signals for colour balance, e.g. white-balance circuits or colour temperature control
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80Camera processing pipelines; Components thereof
    • H04N23/84Camera processing pipelines; Components thereof for processing colour signals
    • H04N23/85Camera processing pipelines; Components thereof for processing colour signals for matrixing

Definitions

  • the invention relates to a method for white balance of an image of an environmental region of a motor vehicle captured by means of a camera of the motor vehicle. A region of the motor vehicle is depicted in the image.
  • the white balance is performed by means of an electronic image processing device considering colour information related to a colouring of the motor vehicle.
  • the invention relates to a camera system for a motor vehicle as well as to a motor vehicle with a camera system.
  • Such a cast to the blue/green is based on the fact that the so-called grey-world assumption is used in the known automatic white balance methods, which implies that the averages of all of the colour channels - thus of the red, the green and the blue colour channel - are to result in a grey value on average.
  • the colour components of the image - thus the red, green and blue components - can therefore be affected such that the averages of all of the colour channels are equal.
  • the colour of the presented image then rather corresponds to reality.
  • the printed matter JP 2007 189 595 A describes a method for white balance of an image considering the information about the colour of the motor vehicle, which is depicted in certain regions in the captured images.
  • the colour of the motor vehicle is dynamically determined "online”, thus in the operation of the camera system based on the captured images.
  • this type of white balance is associated with substantial disadvantages in particular with motor vehicles. Namely, the white balance is usually performed by means of an electronic image processing device in a vehicle camera, which is incorporated in the camera and disposed next to the image sensor. In vehicle cameras, it is sought to configure it particularly compact because relatively little installation space is available in motor vehicles - for instance in the side-view mirror.
  • JP 2007 189 595 A requires a relatively high computational power and therefore a correspondingly large image processing device. Namely, the dynamic determination of the colouring of the motor vehicle based on the captured images is particularly
  • a further solution of the above mentioned problem could be configured such that an image region without a depicted vehicle region is first determined in the captured image of the camera and the parameters of the white balance are determined exclusively based on this image region.
  • the averages of the colour channels could be determined exclusively based on the image region, and the white balance of the entire image could be performed based on these averages.
  • this solution is also relatively computationally intensive. This in particular for the reason that the position and the orientation of the camera on the motor vehicle can change in the operation of the motor vehicle due to mechanical as well as thermal deformations. Therefore, the position and the size of the depicted vehicle region in the image can also change. It is an object of the invention to demonstrate a solution how in a method of the initially mentioned type the white balance of the image can be particularly effectively performed without much computational effort.
  • this object is solved by a method, by a camera system as well as by a motor vehicle having the features according to the respective independent claims.
  • Advantageous implementations of the invention are the subject matter of the dependent claims, of the description and of the figures.
  • a method according to the invention serves for white balance of an image captured by means of a camera of a motor vehicle. Besides an environmental region of the motor vehicle, a region of the motor vehicle itself is also depicted in the image.
  • the white balance of the image is performed by means of an electronic image processing device considering colour information with respect to a colouring of the motor vehicle. According to the invention, it is provided that as the colour information for performing the white balance exclusively information stored in the image processing device is used.
  • the colouring of the motor vehicle is determined in the operation of the camera - in the method according to the invention, the white balance is effected depending on stored colour information.
  • This approach has diverse advantages over the prior art: the white balance can be performed without much computational effort and thus particularly fast. Because a lower computational power is required for the white balance compared to the prior art, a smaller image processing device can also be used, which can be incorporated in the camera. The entire camera can thus also be constructed in compact manner, which proves particularly advantageous in use in a motor vehicle.
  • the camera can therefore be incorporated in an exterior mirror without much effort, in which only very little installation space is available, as is known.
  • the decrease of the required computational power has the advantage that less heat is released by the image processing device, which could adversely affect the operation of the image sensor.
  • the invention is based on the realization that a reliable white balance of the image can also be performed based on stored and thus constant colour information with respect to the colouring of the motor vehicle and dynamic determination of the colour of the motor vehicle in the operation of the camera is not required, because the actual colouring of the motor vehicle substantially remains constant over the entire lifetime of the motor vehicle.
  • the method according to the invention proves particularly advantageous also with lower- priced camera systems, which do not have the required computational power.
  • the method according to the invention can also represent a mere software solution such that the employment of additional hardware components for the white balance with the
  • colour information related to the colouring of the motor vehicle is stored in the image processing device.
  • This colour information is preferably stored in the image processing device in a development phase or a manufacturing process of the camera or of the motor vehicle, namely in particular in a non-volatile memory of the image processing device.
  • this colour information is determined and stored in a calibration phase of the camera.
  • a data value is understood by the colour information, in which the colouring of the motor vehicle is taken into account.
  • the colour information can for example be a data value specifying the exact colour of the motor vehicle.
  • the colour information can also be a data value used in the white balance and only indirectly including information about the colour of the motor vehicle.
  • the colour information can for example be an offset applied to the average of at least one of the colour channels of the image within the scope of the white balance.
  • the performance of the white balance of the image can include that respective colour averages of colour channels of the image - namely in particular of the entire image - are determined in a predetermined colour space. Then, an in particular negative offset can be stored as the colour information in the image processing device for at least one of the colour channels, which is applied to the colour average of this at least one colour channel within the scope of the white balance.
  • This embodiment is based on the fact that usually the grey-world assumption is used for performing the white balance, in which it is assumed that all of the colour channels of the image are to result in a grey value on average.
  • the colour average of at least one of the colour channels is increased compared to an image without the vehicle region - according to colouring of the motor vehicle - such that the white balance too provides an unnatural result.
  • the present embodiment now takes the way to provide the average of at least one of the colour channels of the image with an associated offset and thus to computationally affect the determined colour average of the at least one colour channel.
  • the at least one offset represents the stored colour information.
  • an offset is applied to the colour average of that colour channel, which is included in the colour of the depicted region of the motor vehicle.
  • the respective colour averages of both the red colour channel and the green colour channel are each provided with an offset. Because the yellow colour includes both red and green colour components.
  • the respective offsets can also be selected differently dimensioned. Therein, the respective offset can for example be calculated during a calibration phase of the camera - for example in the manufacture of the motor vehicle.
  • a further development includes that an offset table based on calibration results of the camera is determined and stored for the at least one colour average.
  • different offset values can each be determined and stored for at least one colour channel for different conditions (for instance for different light conditions) in a calibration phase of the camera.
  • These offset values can be stored in a so-called look-up table.
  • the image processing device selects one of the offset values, namely depending on the current conditions.
  • the grey-world assumption (thus the grey-world method) can be used.
  • respective gain factors are determined for the colour channels, by which the colour averages of all of the colour channels each provided (optionally) with an offset are balanced.
  • this is based on the assumption that the colour averages of all of the colour channels are to result in a grey value.
  • an image with natural colours can be produced, which correspond to the colours of the actual environmental scenario.
  • the averages of the colour channels are determined in the RGB colour space (Red, Green, Blue) and the at least one offset is applied in the RGB colour space.
  • the image sensors of cameras usually produce images in the RGB colour space such that the corrections of the colour average of at least one of the colour channels with the offset can also already be performed in the RGB colour space.
  • the white balance can already be performed by means of the internal image processing device of the camera such that the camera outputs images already corrected in colour. Before the images are output, they can be translated from the RGB colour space to the YUV colour space in the camera.
  • one or two of the offsets is/are zero. Namely, the offset is applied to the colour average of that colour channel that is included in the colour of the motor vehicle - as already explained. If one of the offsets is unequal to zero, thus, it is in particular a negative value such that the magnitude of the offset is subtracted from the respective average.
  • the above mentioned formula is based on the grey-world assumption, and by the gain factors it is allowed that all of the colour channels of the image result in a grey value on average and additionally all of the colour channels are equal. Especially thereby, it can be achieved that the colours of the image presented on a display correspond to the natural colours of the environment.
  • the gain factors are preferably dynamically adjusted in the operation of the camera, thus in real time, such that the above mentioned equation is satisfied.
  • This adjustment of the gain factors is thus performed in real time, while the simultaneously captured images of the camera are for example displayed on a display.
  • the adjustment of the gain factors can be configured such that first all of the gain factors are set to one and then adjusted to the final values via several iterations.
  • the colour information namely in particular the at least one offset
  • the colour information can be determined and stored in the image processing device in a manufacturing process of the motor vehicle or else in a development phase of the camera.
  • this determination is effected in a calibration session of the camera.
  • the determination of the offset is effected considering the size of the region of the motor vehicle depicted in the image of the camera. Namely, the size of the depicted region of the motor vehicle has influence on the colour average of the associated colour channel such that a precise offset can be determined considering the size of the depicted vehicle region, which provides for effective white balance of the image.
  • the image can be translated from the RGB colour space to the YUV colour space.
  • Image sensors usually capture images in the RGB colour space, which are then translated to the YUV colour space and output by the camera in this YUV colour space.
  • the so-called colour matrix is used for translating the image.
  • at least one matrix component of the colour matrix is determined depending on the stored colour information within the scope of the white balance.
  • an offset can also be stored as the colour information, which is applied to at least one component of the colour matrix.
  • the colour is green or blue, while it causes a red colour cast with a positive V value.
  • the V value can be increased with the aid of a corresponding offset in the colour matrix. This approach too provides for a natural white balance without much computational effort, because only one offset has to be defined and applied to the at least one component of the colour matrix. This embodiment can even be combined with the above described compensation in the RGB colour space.
  • the invention relates to a camera system for a motor vehicle.
  • the camera system includes a camera for capturing an image of an environmental region of the motor vehicle, wherein a region of the motor vehicle itself is depicted in the image.
  • the camera system also includes an electronic image processing device formed for performing a white balance of the image considering colour information related to a colouring of the motor vehicle.
  • the image processing device can also be incorporated in the camera.
  • the colour information is stored in the image processing device, and the image processing device is configured to use solely the stored information as the colour information for performing the white balance.
  • a motor vehicle according to the invention has a camera system according to the invention.
  • FIG. 1 in schematic illustration a motor vehicle with a camera system according to an embodiment of the invention.
  • Fig. 2 in schematic illustration an image of an environmental region of the motor vehicle captured by means of a camera of the camera system.
  • a motor vehicle 1 illustrated in Fig. 1 is for example a passenger car.
  • the motor vehicle 1 includes a camera system 2 having a plurality of cameras 3 in the embodiment, which are disposed distributed on the motor vehicle 1 . Therein, the cameras 3 each capture an environmental region of the motor vehicle 1 and overall for example the entire
  • the number as well as the arrangement of the cameras 3 are only exemplarily illustrated in Fig. 1 ; the number as well as the
  • respective cameras 3 are incorporated in the two exterior mirrors of the motor vehicle 1 .
  • a camera 3 can be disposed in a front region of the motor vehicle 1 , and a camera 3 can also be disposed in the rear region of the motor vehicle 1 .
  • All of the cameras 3 are electrically coupled to a controller 4, which is a central signal processor of the camera system 2.
  • the controller 4 therefore receives the images of all of the cameras 3 and can process these images.
  • the controller 4 is electrically coupled to a display 5 disposed in the interior of the motor vehicle 1 . Images can be displayed on the display 5, which are based on the images of the cameras 3.
  • the controller 4 can produce an image from the received images of all of the cameras 3, which shows the motor vehicle 1 and its environment from a bird's eye view. Such a plan view of the motor vehicle 1 and its environment is also referred to as a bird eye view.
  • the cameras 3 are video cameras, which are each able to capture a plurality of images per unit time. Thus, the cameras 3 can each capture a temporal sequence of images and output it to the controller 4.
  • the cameras 3 can also be so-called fish-eye cameras, and they can have a relatively wide capturing angle, for instance in a range of values from 150 ° to 200 °.
  • each camera 3 also has an internal image processing device (not illustrated in the figures), which conditions the captured images and transmits them to the controller 4. For example, translation of the image from the RGB colour space to the YUV colour space is performed by means of the internal image processing device. In addition, a white balance of the images is performed by means of the internal image processing device, which is described in more detail below. In addition, the internal image processing device can also perform the so-called black level correction of the image.
  • the central controller 4 can also take the function of the internal image processing device.
  • FIG. 2 An exemplary image 6 of one of the cameras 3 (for example a camera 3 disposed in the exterior mirror) is schematically illustrated in Fig. 2. As is apparent from Fig. 2, a region 7 of the own motor vehicle 1 is also depicted in the captured image. In addition, a road 8 as well as another vehicle 9 are depicted.
  • the internal image processing device of the camera 3 can perform the white balance according to the grey-world method. This implies that first respective colour averages F ge, G avg e and B avg e of the colour channels R, G, B of the image 6 are determined. According to the grey-world assumption, the colour channels R, G, B are to result in a grey value on average. Based on this, the white balance is performed according to the following equation:
  • R ga i n , G ga in and B ga i n denote respective gain factors of the red, green and blue colour channel, respectively.
  • the gain factors R gain , G ga in and B ga i n are determined, by which the respective colour channels are multiplied.
  • the displayed image can have very different colour casts in the white balance according to the grey-world assumption, namely according to the colouring of the motor vehicle.
  • This is attributable to the fact that the region 7 of the motor vehicle 1 is also depicted in the captured image 6 and therefore the performance of the white balance is adversely affected, because at least one of the colour averages R avge , G avg e, B avge is influenced by the colouring of the motor vehicle 1 .
  • the colour average R avge of the red colour channel is unnaturally increased due to the red colouring of the motor vehicle 1 , which results in a blue and a green colour cast being seen in the presented images 6.
  • an offset for the colour averages is stored in the image processing device of the camera 3, which is applied to at least one of the colour averages in performing the white balance.
  • the above equation is complemented as follows:
  • At least one of the offsets R 0 ff S et, G 0 ff S et, Boffset can also be equal to zero.
  • two offsets in the above equation can be zero.
  • this is dependent on the colour of the motor vehicle 1 .
  • offsets for those colour channels are used, which are really included in the colour of the motor vehicle 1 as basic colours. This for example implies that with a red motor vehicle 1 , only an offset FUset for the red colour channel is used.
  • respective offsets FUset, G 0 ff Se t are used both for the red colour channel and for the green colour channel.
  • the motor vehicle 1 is blue, thus, only an offset B ga i n for the blue colour channel is used.
  • an offset is used for that colour component, which is required for emergence of the colour of the motor vehicle 1 .
  • the offsets FUset, G 0 ff S et, B 0 ff Se t are negative values. This implies that the magnitude of the offset is subtracted from the respective colour average R avge , G avge , B avge . Thus, the colour cast due to the colouring of the own motor vehicle 1 can be compensated for.
  • the respective colour averages R avge , G avg e and B avge are extracted from the images of the camera 3 and thus determined in the operation of the camera 3.
  • the gain factors R gain , Ggain and B gain too are determined suonline" in the operation of the camera 3. This is effected in real time and thus also at the same time with the presentation of the images on the display 5.
  • the gain factors R gain , G ga in and B ga i n are determined. This determination of the gain factors is described in more detail below:
  • R' gain , G' gain , B' gain denote the respective gain factors from the last iteration.
  • H max denotes the maximum value from all of the colour averages R avge , G avge , B avge . For example, if the colour average R avg e is larger than the other colour averages G a vge, B avge , then it applies:
  • the offsets R 0 ff S et, G 0 ff S et, B 0 ff Se t represent colour information in terms of the present invention and are therefore stored in the camera.
  • an offset can also be stored as the colour information, with which a component of the colour matrix is provided, by means of which the image 6 is translated from the RGB colour space to the YUV colour space.
  • the image 6 is translated from the RGB colour space to the YUV colour space.
  • V MI R + M2 G + M3 B

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Color Television Image Signal Generators (AREA)

Abstract

The invention relates to methods for white balance of an image (6) of an environmental region of a motor vehicle (1) captured by means of a camera (3) of the motor vehicle (1). A region (7) of the motor vehicle (1) is also depicted in the captured image (6). The white balance is performed by means of an electronic image processing device considering colour information (Roffset, Goffset, Boffset) related to a colouring of the motor vehicle (1). For performing the white balance, exclusively information stored in the image processing device is used as the colour information (Roffset, Goffset, Boffset).

Description

Method for white balance of an image considering the colouring of the motor vehicle
The invention relates to a method for white balance of an image of an environmental region of a motor vehicle captured by means of a camera of the motor vehicle. A region of the motor vehicle is depicted in the image. The white balance is performed by means of an electronic image processing device considering colour information related to a colouring of the motor vehicle. In addition, the invention relates to a camera system for a motor vehicle as well as to a motor vehicle with a camera system.
Methods for performing a white balance of an image are already prior art. Presently, the interest is directed to a camera of a motor vehicle, which captures an environmental region of the motor vehicle. Usually, the cameras are mounted on the motor vehicle such that a region of the motor vehicle is also depicted in the captured images. This is in particular the case if cameras with a wide capturing angle (for instance 190°) are used such as for example so-called fish-eye cameras. If the motor vehicle has an intense colouring, thus, the performance of the automatic white balance is adversely affected. For example, with a red motor vehicle, there is the problem that the image has a slightly enhanced blue and green shade of colour after performing the automatic white balance. Such a cast to the blue/green is based on the fact that the so-called grey-world assumption is used in the known automatic white balance methods, which implies that the averages of all of the colour channels - thus of the red, the green and the blue colour channel - are to result in a grey value on average. The colour components of the image - thus the red, green and blue components - can therefore be affected such that the averages of all of the colour channels are equal. The colour of the presented image then rather corresponds to reality.
If the white balance of an image is now performed, in which the motor vehicle is depicted in certain regions with a relatively intense colouring, thus, corruption of the colour of the depicted environmental region occurs. As already explained, the image has a slight cast to the blue/green with a red colouring of the motor vehicle, because the determined average of the red colour channel of the image with the depicted vehicle region is considerably higher than the actual average of the red colour channel of the depicted vehicle environment. In performing the automatic white balance according to the grey- world method, namely, it is attempted to balance the averages of all of the colour channels such that with a high average of the red colour channel, correspondingly, the averages of the green and the blue colour channel are also automatically increased. Therefore, in the prior art, methods are proposed, in which the colouring of the motor vehicle is taken into account in performing the white balance of the images. Thus, for example, the printed matter JP 2007 189 595 A describes a method for white balance of an image considering the information about the colour of the motor vehicle, which is depicted in certain regions in the captured images. In this prior art, the colour of the motor vehicle is dynamically determined "online", thus in the operation of the camera system based on the captured images. However, this type of white balance is associated with substantial disadvantages in particular with motor vehicles. Namely, the white balance is usually performed by means of an electronic image processing device in a vehicle camera, which is incorporated in the camera and disposed next to the image sensor. In vehicle cameras, it is sought to configure it particularly compact because relatively little installation space is available in motor vehicles - for instance in the side-view mirror. However, the performance of the white balance according to the document
JP 2007 189 595 A requires a relatively high computational power and therefore a correspondingly large image processing device. Namely, the dynamic determination of the colouring of the motor vehicle based on the captured images is particularly
computationally intensive and thus demands much computational power, which is not always available in vehicle cameras. Accordingly, the construction of a compact camera is not possible in the prior art.
A further solution of the above mentioned problem could be configured such that an image region without a depicted vehicle region is first determined in the captured image of the camera and the parameters of the white balance are determined exclusively based on this image region. Here, for example, the averages of the colour channels could be determined exclusively based on the image region, and the white balance of the entire image could be performed based on these averages. However, this solution is also relatively computationally intensive. This in particular for the reason that the position and the orientation of the camera on the motor vehicle can change in the operation of the motor vehicle due to mechanical as well as thermal deformations. Therefore, the position and the size of the depicted vehicle region in the image can also change. It is an object of the invention to demonstrate a solution how in a method of the initially mentioned type the white balance of the image can be particularly effectively performed without much computational effort.
According to the invention, this object is solved by a method, by a camera system as well as by a motor vehicle having the features according to the respective independent claims. Advantageous implementations of the invention are the subject matter of the dependent claims, of the description and of the figures.
A method according to the invention serves for white balance of an image captured by means of a camera of a motor vehicle. Besides an environmental region of the motor vehicle, a region of the motor vehicle itself is also depicted in the image. The white balance of the image is performed by means of an electronic image processing device considering colour information with respect to a colouring of the motor vehicle. According to the invention, it is provided that as the colour information for performing the white balance exclusively information stored in the image processing device is used.
Unlike the subject matter according to document JP 2007 189 595 A - there, the colouring of the motor vehicle is determined in the operation of the camera - in the method according to the invention, the white balance is effected depending on stored colour information. This implies that exclusively the stored information is used as colour information without dynamically determining the colour information in the operation of the camera. This approach has diverse advantages over the prior art: the white balance can be performed without much computational effort and thus particularly fast. Because a lower computational power is required for the white balance compared to the prior art, a smaller image processing device can also be used, which can be incorporated in the camera. The entire camera can thus also be constructed in compact manner, which proves particularly advantageous in use in a motor vehicle. The camera can therefore be incorporated in an exterior mirror without much effort, in which only very little installation space is available, as is known. Not least, the decrease of the required computational power has the advantage that less heat is released by the image processing device, which could adversely affect the operation of the image sensor. Accordingly, the invention is based on the realization that a reliable white balance of the image can also be performed based on stored and thus constant colour information with respect to the colouring of the motor vehicle and dynamic determination of the colour of the motor vehicle in the operation of the camera is not required, because the actual colouring of the motor vehicle substantially remains constant over the entire lifetime of the motor vehicle. The method according to the invention proves particularly advantageous also with lower- priced camera systems, which do not have the required computational power. The method according to the invention can also represent a mere software solution such that the employment of additional hardware components for the white balance with the
disadvantages associated therewith is unnecessary. The method according to the invention is therefore more cost-effective than complex hardware solutions for the white balance.
Thus, colour information related to the colouring of the motor vehicle is stored in the image processing device. This colour information is preferably stored in the image processing device in a development phase or a manufacturing process of the camera or of the motor vehicle, namely in particular in a non-volatile memory of the image processing device. Preferably, this colour information is determined and stored in a calibration phase of the camera. Therein, in particular a data value is understood by the colour information, in which the colouring of the motor vehicle is taken into account. Thus, the colour information can for example be a data value specifying the exact colour of the motor vehicle. However, additionally or alternatively, the colour information can also be a data value used in the white balance and only indirectly including information about the colour of the motor vehicle. Thus, the colour information can for example be an offset applied to the average of at least one of the colour channels of the image within the scope of the white balance.
The performance of the white balance of the image can include that respective colour averages of colour channels of the image - namely in particular of the entire image - are determined in a predetermined colour space. Then, an in particular negative offset can be stored as the colour information in the image processing device for at least one of the colour channels, which is applied to the colour average of this at least one colour channel within the scope of the white balance. This embodiment is based on the fact that usually the grey-world assumption is used for performing the white balance, in which it is assumed that all of the colour channels of the image are to result in a grey value on average. If this grey-world method is now applied to the image, which also shows the own motor vehicle in certain regions, thus, the colour average of at least one of the colour channels is increased compared to an image without the vehicle region - according to colouring of the motor vehicle - such that the white balance too provides an unnatural result. The present embodiment now takes the way to provide the average of at least one of the colour channels of the image with an associated offset and thus to computationally affect the determined colour average of the at least one colour channel. Therein, the at least one offset represents the stored colour information. This embodiment has the advantage that the white balance thus provides a natural image, the colours of which correspond to those of the real environment without having to use much computational power. Namely, because the offset only has to be added to the colour average, the computational effort is minimal.
In an embodiment it is provided that an offset is applied to the colour average of that colour channel, which is included in the colour of the depicted region of the motor vehicle. This implies that - in particular exclusively - the colour averages of those colour components (basic colours) are provided with an offset, which really constitute the colour of the motor vehicle. With a red motor vehicle for example, this is exclusively the red colour channel, the colour average of which is provided with an associated offset. By contrast, with a yellow vehicle, the respective colour averages of both the red colour channel and the green colour channel are each provided with an offset. Because the yellow colour includes both red and green colour components. If a negative offset is each added both to the colour average of the red colour channel and to the colour average of the green colour channel with a yellow motor vehicle, thus, it is avoided that too high averages of the red and the green colour channel are taken as a basis for the white balance and the final image has a too intense blue shade of colour. According to colouring of the motor vehicle, the respective offsets can also be selected differently dimensioned. Therein, the respective offset can for example be calculated during a calibration phase of the camera - for example in the manufacture of the motor vehicle.
A further development includes that an offset table based on calibration results of the camera is determined and stored for the at least one colour average. This implies that a plurality of offset values is stored for the at least one colour channel, from which an offset is used depending on the current scenario, respectively. Thus, different offset values can each be determined and stored for at least one colour channel for different conditions (for instance for different light conditions) in a calibration phase of the camera. These offset values can be stored in a so-called look-up table. In the operation of the camera, the image processing device then selects one of the offset values, namely depending on the current conditions.
As already explained, for performing the white balance, the grey-world assumption (thus the grey-world method) can be used. Then, respective gain factors are determined for the colour channels, by which the colour averages of all of the colour channels each provided (optionally) with an offset are balanced. As explained - this is based on the assumption that the colour averages of all of the colour channels are to result in a grey value. Exactly then, an image with natural colours can be produced, which correspond to the colours of the actual environmental scenario.
It proves advantageous if the averages of the colour channels are determined in the RGB colour space (Red, Green, Blue) and the at least one offset is applied in the RGB colour space. Namely, the image sensors of cameras usually produce images in the RGB colour space such that the corrections of the colour average of at least one of the colour channels with the offset can also already be performed in the RGB colour space. Thus, the white balance can already be performed by means of the internal image processing device of the camera such that the camera outputs images already corrected in colour. Before the images are output, they can be translated from the RGB colour space to the YUV colour space in the camera.
If the grey-world assumption is used, thus, the above mentioned gain factors are selected such that they satisfy the following criterion:
(^avge + ^ offset ) ' ^gain = (^avge + ^offset ) ' ^gain = (^avge + ^ offset ) " Bgain , wherein:
- Ravge, GaVge and Bavge denote the respective colour averages of the red, green and blue colour channel of the image, respectively, which are calculated based on the image,
- Rgain Ggain and Bgain denote the respective gain factors of the red, green and blue colour channel, respectively, and
- Roffset, Goffset and Boffset denote the respective offsets.
In the above mentioned formula, it can also occur that one or two of the offsets is/are zero. Namely, the offset is applied to the colour average of that colour channel that is included in the colour of the motor vehicle - as already explained. If one of the offsets is unequal to zero, thus, it is in particular a negative value such that the magnitude of the offset is subtracted from the respective average. The above mentioned formula is based on the grey-world assumption, and by the gain factors it is allowed that all of the colour channels of the image result in a grey value on average and additionally all of the colour channels are equal. Especially thereby, it can be achieved that the colours of the image presented on a display correspond to the natural colours of the environment. Therein, the gain factors are preferably dynamically adjusted in the operation of the camera, thus in real time, such that the above mentioned equation is satisfied. This adjustment of the gain factors is thus performed in real time, while the simultaneously captured images of the camera are for example displayed on a display. For example, the adjustment of the gain factors can be configured such that first all of the gain factors are set to one and then adjusted to the final values via several iterations.
As already explained, the colour information, namely in particular the at least one offset, can be determined and stored in the image processing device in a manufacturing process of the motor vehicle or else in a development phase of the camera. Preferably, this determination is effected in a calibration session of the camera. Particularly preferred, the determination of the offset is effected considering the size of the region of the motor vehicle depicted in the image of the camera. Namely, the size of the depicted region of the motor vehicle has influence on the colour average of the associated colour channel such that a precise offset can be determined considering the size of the depicted vehicle region, which provides for effective white balance of the image.
As already explained, the image can be translated from the RGB colour space to the YUV colour space. Image sensors usually capture images in the RGB colour space, which are then translated to the YUV colour space and output by the camera in this YUV colour space. For translating the image, the so-called colour matrix is used. In an embodiment it is provided that at least one matrix component of the colour matrix is determined depending on the stored colour information within the scope of the white balance. For example, an offset can also be stored as the colour information, which is applied to at least one component of the colour matrix. By such an approach, the colour cast due to the intense colour of the motor vehicle can be compensated for. As already explained, the intense red colour of the motor vehicle causes a blue cast in the image, which entails a negative value of the V colour channel. If the V value is negative, the colour is green or blue, while it causes a red colour cast with a positive V value. In order to compensate for the blue and green colour cast with a red vehicle, the V value can be increased with the aid of a corresponding offset in the colour matrix. This approach too provides for a natural white balance without much computational effort, because only one offset has to be defined and applied to the at least one component of the colour matrix. This embodiment can even be combined with the above described compensation in the RGB colour space.
In addition, the invention relates to a camera system for a motor vehicle. The camera system includes a camera for capturing an image of an environmental region of the motor vehicle, wherein a region of the motor vehicle itself is depicted in the image. The camera system also includes an electronic image processing device formed for performing a white balance of the image considering colour information related to a colouring of the motor vehicle. The image processing device can also be incorporated in the camera. According to the invention, the colour information is stored in the image processing device, and the image processing device is configured to use solely the stored information as the colour information for performing the white balance.
A motor vehicle according to the invention has a camera system according to the invention. The preferred embodiments presented with respect to the method according to the invention and the advantages thereof correspondingly apply to the camera system according to the invention as well as to the motor vehicle according to the invention.
Further features of the invention are apparent from the claims, the figures and the description of figures. All of the features and feature combinations mentioned above in the description as well as the features and feature combinations mentioned below in the description of figures and/or shown in the figures alone are usable not only in the respectively specified combination, but also in other combinations or else alone.
Now, the invention is explained in more detail based on a preferred embodiment as well as with reference to the attached drawings. At this point, it is to be emphasized that the embodiment described below only represents an exemplary embodiment of the invention and the invention is therefore not restricted to this exemplary embodiment.
There show:
Fig. 1 in schematic illustration a motor vehicle with a camera system according to an embodiment of the invention; and
Fig. 2 in schematic illustration an image of an environmental region of the motor vehicle captured by means of a camera of the camera system.
A motor vehicle 1 illustrated in Fig. 1 is for example a passenger car. The motor vehicle 1 includes a camera system 2 having a plurality of cameras 3 in the embodiment, which are disposed distributed on the motor vehicle 1 . Therein, the cameras 3 each capture an environmental region of the motor vehicle 1 and overall for example the entire
environment around the motor vehicle 1 . The number as well as the arrangement of the cameras 3 are only exemplarily illustrated in Fig. 1 ; the number as well as the
arrangement of the cameras 3 can be different according to embodiment. For example, respective cameras 3 are incorporated in the two exterior mirrors of the motor vehicle 1 . In addition, a camera 3 can be disposed in a front region of the motor vehicle 1 , and a camera 3 can also be disposed in the rear region of the motor vehicle 1 .
All of the cameras 3 are electrically coupled to a controller 4, which is a central signal processor of the camera system 2. The controller 4 therefore receives the images of all of the cameras 3 and can process these images. The controller 4 is electrically coupled to a display 5 disposed in the interior of the motor vehicle 1 . Images can be displayed on the display 5, which are based on the images of the cameras 3. For example, the controller 4 can produce an image from the received images of all of the cameras 3, which shows the motor vehicle 1 and its environment from a bird's eye view. Such a plan view of the motor vehicle 1 and its environment is also referred to as a bird eye view.
The cameras 3 are video cameras, which are each able to capture a plurality of images per unit time. Thus, the cameras 3 can each capture a temporal sequence of images and output it to the controller 4. The cameras 3 can also be so-called fish-eye cameras, and they can have a relatively wide capturing angle, for instance in a range of values from 150 ° to 200 °.
Besides an image sensor, each camera 3 also has an internal image processing device (not illustrated in the figures), which conditions the captured images and transmits them to the controller 4. For example, translation of the image from the RGB colour space to the YUV colour space is performed by means of the internal image processing device. In addition, a white balance of the images is performed by means of the internal image processing device, which is described in more detail below. In addition, the internal image processing device can also perform the so-called black level correction of the image. Optionally, the central controller 4 can also take the function of the internal image processing device.
An exemplary image 6 of one of the cameras 3 (for example a camera 3 disposed in the exterior mirror) is schematically illustrated in Fig. 2. As is apparent from Fig. 2, a region 7 of the own motor vehicle 1 is also depicted in the captured image. In addition, a road 8 as well as another vehicle 9 are depicted. The internal image processing device of the camera 3 can perform the white balance according to the grey-world method. This implies that first respective colour averages F ge, Gavge and Bavge of the colour channels R, G, B of the image 6 are determined. According to the grey-world assumption, the colour channels R, G, B are to result in a grey value on average. Based on this, the white balance is performed according to the following equation:
wherein Rgain, Ggain and Bgain denote respective gain factors of the red, green and blue colour channel, respectively. Thus, for performing the white balance, the gain factors Rgain, Ggain and Bgain are determined, by which the respective colour channels are multiplied.
It has turned out that the displayed image can have very different colour casts in the white balance according to the grey-world assumption, namely according to the colouring of the motor vehicle. This is attributable to the fact that the region 7 of the motor vehicle 1 is also depicted in the captured image 6 and therefore the performance of the white balance is adversely affected, because at least one of the colour averages Ravge, Gavge, Bavge is influenced by the colouring of the motor vehicle 1 . For example, with a red motor vehicle 1 , the colour average Ravge of the red colour channel is unnaturally increased due to the red colouring of the motor vehicle 1 , which results in a blue and a green colour cast being seen in the presented images 6. In order to prevent this, an offset for the colour averages is stored in the image processing device of the camera 3, which is applied to at least one of the colour averages in performing the white balance. Generally speaking, the above equation is complemented as follows:
(^avge + ^ offset ) ' ^gain = (^avge + ^offset ) ' ^gain = (^avge + ^ offset ) " Bgain ' wherein R0ffSet, G0ffSet, B0ffSet denote the respective offsets of the red, green and blue colour channel, respectively. These offsets are now determined depending on the colouring of the motor vehicle 1 as well as depending on the size of the depicted region 7 of the motor vehicle 1 . This determination is already effected in the development phase of the camera 3 or in a manufacturing process of the motor vehicle 1 . Preferably, the determination of the offsets is effected during a calibration of the camera 3.
Depending on the colouring of the motor vehicle 1 , at least one of the offsets R0ffSet, G0ffSet, Boffset can also be equal to zero. However, at the maximum, two offsets in the above equation can be zero. As already explained, this is dependent on the colour of the motor vehicle 1 . Therein, offsets for those colour channels are used, which are really included in the colour of the motor vehicle 1 as basic colours. This for example implies that with a red motor vehicle 1 , only an offset FUset for the red colour channel is used. By contrast, with a mixture of red and green (yellow), respective offsets FUset, G0ffSet are used both for the red colour channel and for the green colour channel. By contrast, if the motor vehicle 1 is blue, thus, only an offset Bgain for the blue colour channel is used. Generally speaking, an offset is used for that colour component, which is required for emergence of the colour of the motor vehicle 1 .
Preferably, the offsets FUset, G0ffSet, B0ffSet are negative values. This implies that the magnitude of the offset is subtracted from the respective colour average Ravge, Gavge, Bavge. Thus, the colour cast due to the colouring of the own motor vehicle 1 can be compensated for.
The respective colour averages Ravge, Gavge and Bavge are extracted from the images of the camera 3 and thus determined in the operation of the camera 3. The gain factors Rgain, Ggain and Bgain too are determined„online" in the operation of the camera 3. This is effected in real time and thus also at the same time with the presentation of the images on the display 5. In order to be able to perform the white balance, thus, the gain factors Rgain, Ggain and Bgain are determined. This determination of the gain factors is described in more detail below:
The gain factors Rgain, Ggain and Bgain are determined in several iterations. In the first step, all of the gain factors are set to one such that it applies: Rgain = Ggain = Bgain = 1 . Then, the colour averages Ravge, Gavge and Bavge are read out from the register of the image sensor of the camera 3, and the respective offsets R0ffSet, G0ffSet and B0ffSet are applied. The white balance, thus the determination of the gain factors, is then effected according to the following equations:
R' gain -H max
avge
' gain -H
avge
wherein R'gain, G'gain, B'gain denote the respective gain factors from the last iteration. Thus, it applies in the first iteration: R'gain = G'gain = B'gain = 1 . By contrast, Hmax denotes the maximum value from all of the colour averages Ravge, Gavge, Bavge. For example, if the colour average Ravge is larger than the other colour averages Gavge, Bavge, then it applies:
Umax = Ravge-
The iterations are performed until the white balance is completely terminated, thus until the following equation is satisfied:
Vnavge ~*~ noffset / ' ngain V^avge ^offset / ' °gain VDavge ~*~ D offset / ' Dgain
Therein, the offsets R0ffSet, G0ffSet, B0ffSet represent colour information in terms of the present invention and are therefore stored in the camera.
Additionally or alternatively, an offset can also be stored as the colour information, with which a component of the colour matrix is provided, by means of which the image 6 is translated from the RGB colour space to the YUV colour space. In the translation of the image, there applies:
U
= (colour _ matrix
V
wherein U, V denote the colour channels (thus the colour components) of the YUV colour space and colour matrix denotes the colour matrix. The colour matrix can be represented as follows:
M4 M5 M6
colour matrix =
M1 M2M3 such that:
U = M4 R + M5 G + M6 B and
V = MI R + M2 G + M3 B By adding an offset to the components of the colour matrix, the colour casts due to the colouring of the motor vehicle 1 can be compensated for. For example, a red colouring of the motor vehicle 1 causes a blue and green colour cast in the image 6. This corresponds to a negative value of the V colour channel. Namely, if the V value is negative, the colour cast is blue or green, while the colour cast is rather red with a positive V value. Thus, it is possible to compensate for the blue and green colour cast, namely by adding an offset. It is apparent from the above equation that by increase of the component M1 , the V value and thus the red component can also be increased. By increasing the component M1 , thus, with a red motor vehicle 1 , the blue/green colour cast can be corrected.

Claims

Claims
1 . Method for white balance of an image (6) of an environmental region of a motor vehicle (1 ) captured by means of a camera (3) of the motor vehicle (1 ), comprising the steps of:
- capturing the image (6) by means of the camera (3), wherein a region (7) of the motor vehicle (1 ) is depicted in the image (6), and
- performing the white balance by means of an electronic image processing device considering colour information (FUset, G0ffSet, B0ffSet) related to a colouring of the motor vehicle (1 ),
characterized in that
for performing the white balance, exclusively information stored in the image processing device is used as the colour information (FUset, G0ffSet, B0ffSet)-
2. Method according to claim 1 ,
characterized in that
for performing the white balance, respective colour averages (Ravge, Gavge, Bavge) of colour channels of the image (6) are determined in a predetermined colour space, and for at least one of the colour channels an, in particular negative, offset is stored as colour information (F set, G0ffSet, B0ffSet) in the image processing device, which is applied to the colour average (Ravge, Gavge, Bavge) of this at least one colour channel within the scope of the white balance.
3. Method according to claim 2,
characterized in that
an offset (F et, G0ffSet, Boffset) is applied to the colour average (Ravge, Gavge, Bavge) of that colour channel, which is included in the colour of the depicted region of the motor vehicle (1 ).
4. Method according to claim 2 or 3,
characterized in that for performing the white balance, the grey-world assumption is used by determining respective gain factors (Rgai n, Ggai n, Bgain) for the colour channels, by which the colour averages (Ravge, Gavge, Bavge) of all of the colour channels each provided with an offset (Roffset, Goffset, B0ffSet) are balanced.
5. Method according to any one of claims 2 to 4,
characterized in that
the colour averages (Ravge, Gavge, Bavge) of the colour channels are determined in the RG B colour space and the at least one offset (R0ffSet, G0ffSet, B0ffSet) is applied in the RG B colour space.
Method according to claim 4 and 5,
characterized in that
the gain factors (Rgai n, Ggai n, Bgai n) satisfy the following criterion:
(^avge + R offset ) ' ^gain = (^avge + ^offset ) ' ^gain = feavge + ^ offset ) " Bgain ' wherein Ravge, Gavge, Bavge denote the respective colour averages (Ravge, Gavge, B of the red, green and blue colour channel of the image (6) and Rgai n, Ggai n, Bgai n denote the respective gain factors (Rgai n, Ggai n, Bgain) of the red, green and blue colour channel as well as R0ffSet, G0ffSet, B0ffSet denote the respective offsets.
7. Method according to anyone of the preceding claims,
characterized in that
the colour information (R0ffSet, G0ffSet, B0ffSet) , in particular the at least one offset (R0ffSet, Goffset, Boffset) , is determined considering the size of a region of the motor vehicle (1 ) depicted in the image (6) of the camera (3) and stored in the image processing device in a manufacturing process of the motor vehicle (1 ).
8. Method according to anyone of the preceding claims,
characterized in that
the image (6) is translated from the RGB colour space to the YUV colour space with the aid of a colour matrix (colour matrix), and at least one component (M1 to M6) of the colour matrix (colour matrix) is determined depending on the stored colour information (R0ffSet, G0ffSet, B0ffSet) within the scope of the white balance.
9. Camera system (2) for a motor vehicle (1 ), including a camera (3) for capturing an image (6) of an environmental region of the motor vehicle (1 ), wherein a region (7) of the motor vehicle (1 ) is depicted in the image (6), and including an electronic image processing device for performing a white balance of the image (6) considering colour information (FUset, G0ffSet, B0ffSet) related to a colouring of the motor vehicle (1 ),
characterized in that
the colour information (FUset, G0ffSet, B0ffSet) is stored in the image processing device and the image processing device is configured to use as the colour information (Roffset, Goffset, B0ffSei) solely the stored information for performing the white balance.
10. Motor vehicle (1 ) including a camera system (2) according to claim 9.
EP13716288.9A 2012-04-20 2013-04-12 Method for white balance of an image considering the colouring of the motor vehicle Ceased EP2839662A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
DE102012007982A DE102012007982A1 (en) 2012-04-20 2012-04-20 Method for white balance of an image taking into account the color of the motor vehicle
PCT/EP2013/057741 WO2013156417A1 (en) 2012-04-20 2013-04-12 Method for white balance of an image considering the colouring of the motor vehicle

Publications (1)

Publication Number Publication Date
EP2839662A1 true EP2839662A1 (en) 2015-02-25

Family

ID=48095856

Family Applications (1)

Application Number Title Priority Date Filing Date
EP13716288.9A Ceased EP2839662A1 (en) 2012-04-20 2013-04-12 Method for white balance of an image considering the colouring of the motor vehicle

Country Status (3)

Country Link
EP (1) EP2839662A1 (en)
DE (1) DE102012007982A1 (en)
WO (1) WO2013156417A1 (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102014222780A1 (en) 2014-11-07 2016-05-12 Robert Bosch Gmbh Camera system and vehicle comprising a camera system
JP6354578B2 (en) * 2014-12-26 2018-07-11 株式会社Jvcケンウッド Imaging system
DE102020125232A1 (en) 2020-09-28 2022-03-31 Continental Automotive Gmbh Color correction method for a camera system and a camera system

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011000392A1 (en) * 2009-07-02 2011-01-06 Hi-Key Limited Method and camera system for improving the contrast of a camera image

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07162890A (en) * 1993-12-08 1995-06-23 Matsushita Electric Ind Co Ltd White balance adjustment device
JP5073207B2 (en) 2006-01-16 2012-11-14 株式会社日立アドバンストデジタル In-vehicle camera device
CN102342089A (en) * 2009-03-02 2012-02-01 弗莱克斯电子有限责任公司 Calibration techniques for camera modules

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011000392A1 (en) * 2009-07-02 2011-01-06 Hi-Key Limited Method and camera system for improving the contrast of a camera image

Also Published As

Publication number Publication date
DE102012007982A1 (en) 2013-10-24
WO2013156417A1 (en) 2013-10-24

Similar Documents

Publication Publication Date Title
JP4325642B2 (en) Mobile camera system
JP5971207B2 (en) Image adjustment apparatus, image adjustment method, and program
US10166921B2 (en) Brightness and color matching video from multiple-camera system
US7969480B2 (en) Method of controlling auto white balance
JP2009017020A (en) Image processing apparatus and display image generation method
US20140036106A1 (en) Image processing apparatus and image processing method
CN103986875A (en) Image acquiring device, method and terminal and video acquiring method
US9214034B2 (en) System, device and method for displaying a harmonized combined image
JP2012054827A (en) Image processor, image processing method and program
KR20110016812A (en) Imaging Device and Imaging Method
JP5195841B2 (en) On-vehicle camera device and vehicle
CN110915204B (en) Image processing apparatus, image processing method, storage medium, and imaging apparatus
US11831995B2 (en) Image capturing apparatus, image capturing method, and carrier means
EP2839662A1 (en) Method for white balance of an image considering the colouring of the motor vehicle
WO2014118167A1 (en) Method for white balance of an image presentation and camera system for a motor vehicle
CN101510955B (en) Image pickup device
JP2010154478A (en) Compound-eye imaging apparatus and method for generating combined image thereof
JP4862321B2 (en) In-vehicle camera device
CN111355942B (en) Semiconductor device, image processing system, method, and computer-readable storage medium
US20180069998A1 (en) Imaging apparatus, imaging system, and vehicle
JP2006173988A (en) In-vehicle camera image presentation apparatus and in-vehicle camera image presentation method
KR20180028354A (en) Method for displaying image in multiple view modes
JP2009017301A (en) Signal processing device
JP2009506722A (en) Improved color difference filter for white balance statistics calculation
EP4622285A1 (en) Method for synchronization of a multi camera device

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20141014

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAX Request for extension of the european patent (deleted)
17Q First examination report despatched

Effective date: 20151221

REG Reference to a national code

Ref country code: DE

Ref legal event code: R003

APAF Appeal reference modified

Free format text: ORIGINAL CODE: EPIDOSCREFNE

APBK Appeal reference recorded

Free format text: ORIGINAL CODE: EPIDOSNREFNE

APBN Date of receipt of notice of appeal recorded

Free format text: ORIGINAL CODE: EPIDOSNNOA2E

APBT Appeal procedure closed

Free format text: ORIGINAL CODE: EPIDOSNNOA9E

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN REFUSED

18R Application refused

Effective date: 20171114