AU2013374190B2 - Imaging apparatus with scene adaptive auto exposure compensation - Google Patents

Imaging apparatus with scene adaptive auto exposure compensation Download PDF

Info

Publication number
AU2013374190B2
AU2013374190B2 AU2013374190A AU2013374190A AU2013374190B2 AU 2013374190 B2 AU2013374190 B2 AU 2013374190B2 AU 2013374190 A AU2013374190 A AU 2013374190A AU 2013374190 A AU2013374190 A AU 2013374190A AU 2013374190 B2 AU2013374190 B2 AU 2013374190B2
Authority
AU
Australia
Prior art keywords
image
pixel
pixels
saturation
histogram
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
AU2013374190A
Other versions
AU2013374190A1 (en
Inventor
Sina Afrooze
Guoqian Sun
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Motorola Solutions Inc
Original Assignee
Motorola Solutions Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Motorola Solutions Inc filed Critical Motorola Solutions Inc
Publication of AU2013374190A1 publication Critical patent/AU2013374190A1/en
Application granted granted Critical
Publication of AU2013374190B2 publication Critical patent/AU2013374190B2/en
Assigned to MOTOROLA SOLUTIONS, INC. reassignment MOTOROLA SOLUTIONS, INC. Request for Assignment Assignors: AVIGILON CORPORATION
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/73—Circuitry for compensating brightness variation in the scene by influencing the exposure time
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/40—Image enhancement or restoration by the use of histogram techniques
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/71—Circuitry for evaluating the brightness variation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/76—Circuitry for compensating brightness variation in the scene by influencing the image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/30—Transforming light or analogous information into electric information
    • H04N5/33—Transforming infrared radiation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20072—Graph-based image processing

Abstract

A method adjusts an auto exposure target in an auto-exposure operation on a sequence of images, such as a sequence of infrared images. The method comprises: obtaining a histogram of at least one of the images; applying a weighted histogram table to the histogram to obtain weighted histogram bins wherein at least some bins in the histogram containing saturated pixels are assigned a higher weighting value and at least some bins in the histogram containing unsaturated pixels are assigned a lower weighting value, and summing the weighted histogram bins to obtain a saturation score and decreasing an auto exposure target for an auto exposure operation when the saturation score exceeds a first threshold value, and increasing the auto-exposure target when the saturation score is below the first threshold value and the image is underexposed.

Description

PCT/CA2013/050060 WO 2014/110654
Imaging Apparatus With Scene Adaptive Auto Exposure Compensation
Field
This disclosure relates generally to a scene adaptive auto-exposure compensation method and an imaging apparatus operable to perform this method.
Background
It is often desirable to use an autonexposure algorithm in a camera to ensure that the image is acquired at a balanced shutter-time and gain under different lighting conditions. Often the objective of the auto-exposure algorithm is to maintain a constant global luminance for the image by adjusting the shutter-time and gain. The problem with this approach is that if the scene contains objects that are very small (<1% of image pixels) but much brighter than the rest of the image, these objects will be over-exposed and appear saturated in the image. This problem is specifically severe for a camera equipped with an infrared ("IR") illuminator because certain small objects of interest (e.g. license plates) have a much higher IR reflectivity than the rest of the scene. However if the camera ensures that no area of the image is saturated, a small background light in the image can result in the majority of the image being permanently underexposed.
There are several cameras in the market that try to solve the problems of saturated objects as a result of IR-lllumination by simply reducing the intensity of the IR illuminator. Controlling IR intensity to control saturation will only work for a camera with built-in IR illuminator which can be directly controlled. In particular, this approach will not work with cameras that use an external IR illuminator wherein the illumination intensity cannot be directly controlled.
Summary
According to one aspect of the invention, there is provided a method for adjusting an auto exposure target in an auto-exposure operation on a sequence of images, such as 1 PCT/CA2013/050060 WO 2014/110654 a sequence of infrared images. The method comprises obtaining a histogram of an image; applying a weighted histogram table to the histogram to obtain weighted tonal value bins wherein at least some bins in the histogram containing oversaturated pixels are assigned a higher weighting value and at least some bins in the histogram 5 containing non-saturated pixels are assigned a lower weighting value, and summing the weighted histogram bins to obtain a saturation score and decreasing an auto exposure target for an auto exposure operation when the saturation score indicates that the image is oversaturated, and increasing the auto exposure target when the saturation score indicates that the image is non-saturated and when the image is underexposed. 10 Prior to obtaining a histogram of at the image, the static or mostly static background saturated pixels in the image can be masked by identifying pixels in the image that have exceeded a saturation threshold a selected number of times over a time period; and applying a mask to the identified pixels such that the masked pixels are not included in the histogram. The step of identifying pixels as being static or mostly static background 15 pixels can comprise reading a saturation level of a pixel and incrementing a counter when the pixel exceeds the saturation threshold and identifying the pixel as being a static or mostly static saturated background pixel when the counter exceeds a background saturated pixel threshold value. The method can further comprise unmasking masked pixels by reading a saturation level of a masked pixel and 20 decrementing a counter when the masked pixel falls below the saturation threshold, and removing the mask from the masked pixel when the counter fails below a background saturated pixel threshold value.
The histogram can be obtained from only one out of every selected number of images in the sequence of images, wherein the selected number of images is greater than the 25 number of images in the sequence of images to which the auto-exposure operation is applied, The weighted histogram table can have a weighting value that linearly increases from the bins assigned the lower weighting value to the bins assigned the higher weighting value, wherein the linear increase occurs over a selected range of bins. The higher weighting value can be 1 and the lower weighting value can be 0. 2 PCT/CA2013/050060 WO 2014/110654
The method can further comprise decreasing the auto exposure target by a first amount when the image is highly oversaturated, and decreasing the auto exposure target by a second amount when the image is slightly oversaturated, wherein the saturation score when the image is highly saturated is higher than the saturation score when the image 5 is slightly saturated. The first amount can be a function of the original auto exposure target, and the second amount can be a function of the global mean exposure level of the image. The images can be infrared images.
According to another aspect of the invention, there is provided an imaging apparatus comprising: an imager; and processing circuitry communicative with the imager to 10 receive an image captured by the imager. The processing circuitry comprises a processor and a memory having encoded thereon program code executable by the processor to perform the aforementioned method.
According to another aspect of the invention, there is provided a method for generating a mask of background saturated objects in an image, comprising: identifying pixels in 15 the image that exceed a saturation threshold a selected number of times over a time period as being static or mostly static background saturated pixels; and assigning a mask flag to each pixel in the image identified as being a mostly static background saturated pixel. The method can further comprise identifying masked pixels in the image that have fallen below a saturation threshold a selected number of times over a 20 time period as being non-saturated pixels, and removing the mask flag from masked pixels that are non-saturated.
The method can further comprise two saturation thresholds, namely an upper saturation threshold above which a pixel Is considered saturated and a lower saturation threshold below which a pixel is considered non-saturated. The step of identifying pixels as 25 being static or mostly static background pixels can comprise reading a saturation level of a pixel and incrementing a counter when the pixel exceeds the saturation threshold and identifying the pixel as being a static or mostly static saturated background pixel when the counter exceeds a background saturated pixel threshold value. The step of identifying masked pixels that are non-saturated can comprises reading a saturation 3 2013374190 11 Oct 2016 level of a masked pixel and decrementing a counter when the masked pixel falls below the saturation threshold and identifying the masked pixel as being non-saturated when the counter falls below a background saturated pixel threshold value.
In another aspect there is provided a method for adjusting an auto exposure target in an auto-exposure operation on a sequence of images, comprising: (a) prior to obtaining a histogram of an image, masking static or mostly static background saturated pixels in the image by identifying pixels in the image that have exceeded a saturation threshold a selected number of times over a time period, and applying a mask to the identified pixels such that the masked pixels are not included in the histogram; (b) obtaining the histogram of the image; (c) applying a weighted histogram table to the histogram to obtain weighted tonal value bins wherein at least some bins in the histogram containing oversaturated pixels are assigned a higher weighting value and at least some bins in the histogram containing non-saturated pixels are assigned a lower weighting value; and (d) summing the weighted histogram bins to obtain a saturation score and decreasing an auto exposure target for an auto exposure operation when the saturation score indicates that the image is oversaturated, and increasing the auto exposure target when the saturation score indicates that the image is non-saturated and when the image is underexposed.
In another aspect there is provided an imaging apparatus comprising: (a) an imager; and 4 2013374190 11 Oct 2016 (b) processing circuitry communicative with the imager to receive an image captured by the imager, the processing circuitry comprising a processor and a memory having encoded thereon program code executable by the processor to perform a method of adjusting an auto exposure target in an auto-exposure operation on a sequence of images, comprising: (i) prior to obtaining a histogram of an image, masking static or mostly static background saturated pixels in the image by identifying pixels in the image that have exceeded a saturation threshold a selected number of times over a time period, and applying a mask to the identified pixels such that the masked pixels are not included in the histogram; (ii) obtaining the histogram of the image; (iii) applying a weighted histogram table to the histogram to obtain weighted tonal value bins wherein at least some bins in the histogram containing oversaturated pixels are assigned a higher weighting value and at least some bins in the histogram containing non-saturated pixels are assigned a lower weighting value; and (iv) summing the weighted histogram bins to obtain a saturation score and decreasing an auto exposure target for an auto exposure operation when the saturation score indicates that the image is oversaturated, and increasing the auto exposure target when the saturation score indicates that the image is non-saturated and when the image is underexposed.
In another aspect there is provided a method for generating a mask of background saturated pixels in an image, comprising: (a) identifying pixels in the image that exceed a saturation threshold a selected number of times over a time period as being static or mostly static background saturated pixels; and
4A 2013374190 11 Oct 2016 (b) masking a pixel by assigning a mask flag to each pixel in the image identified as being a static or mostly static background saturated pixel.
In another aspect there is provided an imaging apparatus comprising: (a) an imager; and (b) processing circuitry communicative with the imager to receive an image captured by the imager, the processing circuitry comprising a processor and a memory having encoded thereon program code executable by the processor to perform a method of generating a mask of background saturated pixels in an image, comprising: (i) identifying pixels in the image that exceed a saturation threshold a selected number of times over a time period as being static or mostly static background saturated pixels; and (ii) masking a pixel by assigning a mask flag to each pixel in the image identified as being a static or mostly static background saturated pixel.
Brief Description of Drawings FIGS. 1A, 1B and 1C are schematic block diagrams of three embodiments of an imaging apparatus, wherein a first embodiment comprises an imaging apparatus with an IR illuminator having a pair of infrared emitting diode (IRED) sets, a second embodiment comprises an imaging apparatus having an IR illuminator with one IRED set, and a third embodiment comprises an imaging apparatus that uses an external IR illuminator to provide IR illumination. Each embodiment also comprises a memory having encoded thereon a program executable to perform a scene adaptive auto-exposure compensation method for an auto exposure operation. FIG. 2 is a perspective view of the imaging apparatus embodied as a security camera.
4B 2013374190 11 Oct 2016 FIG. 3 is an exemplary histogram weighting table to be applied to a histogram of an image captured by the imaging apparatus in carrying out the scene adaptive autocompensation method. FIG. 4 is a flow chart showing steps of a first component of the scene adaptive autocompensation method, namely steps for generating a background saturation mask. FIGS. 5A and 5B are a flow chart showing steps of a second component of the scene adaptive auto-compensation method, namely steps for controlling saturation level in an auto-exposure operation.
Detailed Description
The embodiments described herein provide a scene adaptive auto-exposure compensation method used in an auto-exposure operation of an imaging apparatus, such as a security camera. The scene adaptive auto-exposure compensation method comprises two components, namely a first component which applies a mask to saturated objects in the background of the image (“background saturation mask
4C 2013374190 22 Sep 2016 generator”), and a second component that provides a bias value to different parts of an image histogram when calculating an auto-exposure target value for use in an image-mean based auto-exposure operation (“auto-exposure companion for controlling saturation level”). The first component causes the auto-exposure operation to ignore background saturated objects, which otherwise could cause the auto-exposure operation to underexpose objects of interest in the image. The second component controls the saturation levels in the image by reducing the auto exposure target value of the auto-exposure operation when certain parts of the image are oversaturated, and returning the auto-exposure target value back to the original auto-exposure target value when no area in the image is oversaturated or the oversaturated areas are masked by the backgrounds saturation mask.
In other words, the method embodied by the present embodiments controls exposure (shutter time and gain) to prevent saturation in an IR-illuminated scene, instead of conventional approaches which prevent saturation by controlling the intensity of the IR illuminator. This enables the method to be used in imaging apparatuses that have built-in or external IR illuminators, since controlling IR illumination intensity is not used to control saturation levels. Also, the method, by preventing or at least reducing saturation in the image and ignoring saturated background objects, is expected to improve image quality and in particular preserve image detail for objects of interest in the image. A further benefit of preventing or reducing saturation by controlling exposure is the ability to reduce motion-blur for moving objects in the image by reducing the exposure of the image.
The method is embodied as program code stored on a memory of an imaging apparatus and executed by a processor in that apparatus. The imaging apparatus can feature an on-board IR illuminator like that shown in FIGS. 1A and 1B, or use an external IR illuminator like that shown in FIG. 1C.
Referring now to FIG. 1A, an imaging apparatus 10 according to one embodiment comprises the following major components: a zoom lens 12, an imager 14 optically coupled to the zoom lens 12, a lens driver 16 mechanically coupled to the zoom lens 12 5 2013374190 22 Sep 2016 and operable to change the focal length of the zoom lens, an IR illuminator 18 comprising a pair of IR emitters 18(a), 18(b) each producing an IR illumination beam with a different linear profile (respectively, “wide angle IR emitter” 18(a) and “narrow angle IR emitter” 18(b)), a current driver 20(a), 20(b) for each IR emitter 18(a), 18(b), and control and processing circuitry 22 communicative with the imager 14, lens driver 16, and current drivers 20(a), 20(b).
The imaging apparatus 10 can be embodied as a security camera such as that shown in FIG. 2. The security camera 10 has a housing 30 which houses the aforementioned major components of the imaging apparatus, and a movable mount 32 for mounting the camera 10 to a surface such as a ceiling. The zoom lens 12 is mounted at the front of the camera 10, and a printed circuit board (“PCB”, not shown) is also mounted at the front of the camera 10 around the zoom lens 12; the wide angle IR emitter 18(a) and narrow angle IR emitter 18(b) are respectively mounted on this PCB and face the same direction as the zoom lens 12 and serve to illuminate the field of view of the zoom lens with infrared light.
Each IR emitter 18(a), 18(b) in this embodiment comprises a set of infrared emitting diodes (IRED) 34. Such IRED sets are known in the art; one suitable such IRED set comprises a pair of Osram SFH4715S IREDs. Each IR emitter 18(a), 18(b) also comprises a lenslet 36 for each IRED 34; the lenslet 36 is configured to shape the IRED emission into an IR beam having a particular illumination pattern and a particular linear profile. In particular, the lenslets 36 for the wide angle IR emitter 18(a) will produce an IR beam with a linear profile that is relatively widely dispersed (hereinafter referred to as the “wide beam component”), and the lenslets 36 for the narrow angle IR emitter 18(b) will produce an IR beam with a linear profile that is relatively narrowly dispersed, i.e. (hereinafter referred to as the “narrow beam component”). Such lenslets are known in the art; one suitable such lenslet can be provided by Ledil.
The current drivers 20(a), 20(b) are designed to regulate the current delivered to the IR emitters 18(a), 18(b). The current drivers 20(a), 20(b) can be controlled to deliver all of the total available power to one or the other of the IR emitters 18(a), 18(b), or vary the 6 2013374190 22 Sep 2016 power ratio between the two emitters 18(a), 18(b). Such current drivers are known in the art; one suitable such current driver is the AL8805 Buck LED Driver by On Semiconductor. The current drivers 20(a), 20(b) are communicatively coupled to a general purpose input/output (GPIO) pin 38 on a circuit board inside the housing which contains the processing circuitry 22 (otherwise known as main system on chip (SoC)) of the surveillance camera 10. The processing circuitry 22 comprises a interface bus with pins 42, 44 that are communicatively coupled to the lens driver 16 and imager 14. The imager 14 is configured to capture light in the infrared spectrum, and can be for example, a digital sensor such as a complementary metal-oxide-semiconductor (CMOS) sensor. The specifications of the imager 14 and the zoom lens 12 can be selected based on an operator’s requirements and performance expectations. Operation of zoom lenses and imaging sensors in a surveillance camera are well known in the art and thus the operation of the imager 14, lens driver 16 and zoom lens 12 are not described in further detail here.
The processing circuitry 22 also includes a processor and a memory (CPU) 40 having encoded thereon program code that is executed by the processor to operate the security camera 10. This program code includes instructions for sending a control signal from the GPIO pin 38 to the current drivers 20(a), 20(b) to produce the IR beam. The program code can also include instructions for combining the wide beam component and narrow beam component in one or more different power ratios to produce a combined IR beam having different linear profiles. As will be described in further detail below, the program code also includes instructions for performing an auto-exposure operation and the scene adaptive auto-exposure compensation method that can be used in the auto-exposure operation.
Referring now to FIG. 1B, the second embodiment of the imaging apparatus 10 is the same as the first embodiment except that this imaging apparatus only consists of a single IRED set, to produce a fixed-width (i.e. non-varying linear profile) IR illumination beam instead of a variable-width IR illumination beam (i.e. variable linear profile). Like the first embodiment, this second embodiment also comprises processing circuitry 22 having a memory encoded with program code that includes instructions for performing 7 2013374190 22 Sep 2016 an auto-exposure operation and the scene adaptive auto-exposure compensation method that can be used in the auto-exposure operation.
Referring now to FIG. 1C, the third embodiment of the imaging apparatus 10 is the same as the first and second embodiments except that this imaging apparatus does not have a built-in IR illuminator but instead uses an external IR illuminator to illuminate a scene. Like the first and second embodiments, this third embodiment also comprises processing circuitry 22 having a memory encoded with program code that includes instructions for performing an auto-exposure operation and the scene adaptive autoexposure compensation method that can be used in the auto-exposure operation.
Scene Adaptive Auto-exposure Compensation Method
Referring now to FIGS. 3, 4, 5A and 5B, the scene adaptive auto-exposure compensation method will now be described in detail. As noted above, the method comprises two components, namely (1) a scene adaptive background saturation mask generator; and (2) an auto-exposure companion for controlling saturation level. The first component will create a mask that identifies regions of the image that are saturated but can be considered static or mostly static and part of the background; this mask can be used by the second component to ignore background saturated regions when controlling the saturation level of the image. The second component ensures that the saturation level of the image stays below a threshold level (typically 0) by processing a histogram of the image using a weighted histogram table, and setting new autoexposure targets for the auto-exposure operation when the processed histogram values do not meet a target threshold range.
The auto-exposure operation can be based on a conventional image-mean based autoexposure algorithm that adjusts the exposure of a series of images (frames) taken over a time period. Such conventional auto-exposure algorithms are well known in the art and thus the auto-exposure operation is not described in detail here.
Referring particularly to FIG. 4, the scene-adaptive background saturation mask comprises a series of instructions executed by the processor to identify saturated pixels 8 2013374190 22 Sep 2016 in the image that are static or mostly static and therefore can be considered part of the background of the image (the assumption being that at least for security surveillance purposes, objects of interests will be moving in the image). Static and mostly static objects that are found to be saturated will be masked, i.e. ignored by the auto-exposure companion for controlling saturation level of the image. This ensures that the exposure will not stay at very small values if the image background contains small bright objects such as street lights.
The background saturation mask component of the method operates by processing each pixel i of every single captured image (“image[ij’) on a pixel by pixel basis (from /=0 to NumberOfPixels). Starting with the first pixel /=0 (step 100), the saturation level of each pixel /' is read (102); assuming that saturation level is represented by one byte of data, there can be 256 measurable saturation levels ffrom t = 0 to 255.
The read saturation level of each pixel / is compared to an upper saturation threshold T1 above which the pixel is considered saturated (step 104), and a lower saturation threshold TO below which the pixel is considered not-saturated (step 106). The values of these thresholds being dependent on the type of sensor used; typical values can be for example, T1=200 and 70=190 for a saturation range having 256 saturation levels.
Alternatively but not shown in FIG. 4, only one saturation threshold T can be provided above which the pixel is considered saturated, and below which the pixel is considered not saturated.
To determine whether the pixel represents a static or mostly static saturated object, a counter (“counter[iJ’) is provided which is increased by increment U when the read saturation level exceeds the upper saturation threshold T1 (step 108), and decreased by decrement D when the read saturation level is below the lower saturation threshold TO (step 110). A clipping function is applied to keep the counter value between 0 and a selected maximum (“maxCount’) (step 112); when the counter fall below 0, counter[i] will be set to 0, and when the counter exceeds maxCount, counter[i] will be set to maxCount. If the read saturation level is equal to or between the upper and lower saturation thresholds T1, TO, the counter is not changed.. 9 2013374190 22 Sep 2016 the field of view of the pixel, and will decrement the counter when the bright object has moved out of the field of view of the pixel).
It can also be seen that the method can be calibrated to capture “mostly static” objects like blinking lights as background saturated objects, by adjusting the increments U and D appropriately. For example, U can be assigned a higher value than D in which case the counter will slowly increment over a number of image frames until counter[i] is greater than threshold M. For example, U can be assigned a value of 2 and D assigned a value of 1.
Referring now to FIGS. 3, 5A and 5B, the auto-exposure companion for controlling saturation level component of the method (“auto exposure companion component”) involves creating a histogram of an image after background saturated objects have been masked by the background saturated masking generator, then applying a weighted histogram table to the histogram to obtain a “saturation score” which can be considered to be the degree of which the image is saturated, and which is used to determine whether new auto-exposure target values should be used by the autoexposure operation when the image is found to be oversaturated or non-saturated but underexposed. It is expected that when the auto-exposure operation utilizes the autoexposure companion of the present method, that the saturation level of the image can be kept below a threshold value thereby improving image quality especially in parts of the image that are of interest.
The auto-exposure companion component of the method starts by assigning a current auto exposure target value (“aeTrgf) to be equal to the original auto-exposure target associated with the auto-exposure operation (“origAeTrgf) (step 200). The original auto-exposure value can be defined by the user or preset in the auto-exposure operation program code. Then a series of steps (steps 204-208) are carried out to execute the auto-exposure companion component on only one out of every selected number K of image frames in order to ensure that the auto-exposure companion component loop and the auto-exposure operation loop do not interact in way that causes image instability. For example, K can be four, in which case a counter 11 2013374190 22 Sep 2016 S aturatUmS core (“imgCounf) is deployed to count each image frame (step 206) and ensure that the auto-exposure companion is performed on every fourth image frame in sequence (step 208).
The background saturation mask generator is applied to the images including the Kth image to which the auto-exposure companion component is applied (step 210), in the manner as described above. Then, an image histogram of the masked image is obtained in a manner that is well known in the art (step 212). An image histogram is a type of histogram that acts as a graphical representation of the tonal distribution in a digital image. It plots the number of pixels for each tonal value bin across a range of tonal values.
Then, a histogram weighting table like the one shown in FIG. 3 is applied to the histogram to obtain the saturation score of the image (step 214). The histogram weighting table assigns a weighting value WVto each bin of the histogram; for example, for an image histogram having a tonal range of one byte, there will be 256 separate bins (bin[0] to bin[255]) each representing a different tonal value. The histogram weighting table assigns a weighting value of 0 for all tonal value bins that are considered to be non-saturated, from the lowest tonal value (bin[0]) to a first selected tonal value /c(bin[k- 1]), and assigns a weighting value of 1 for all tonal value bins that are considered to be oversaturated, from a second selected tonal value m [bin m+1] to the highest tonal value (bin[255]). The weighting value linearly increases from 0 to 1 from bin[k] to bin[m] in other to provide a smooth transition during exposure changes. The first and second tonal values k, m can be selected based on what an operator would consider to be acceptable amount of saturation in the image, and can for example be 200 and 230 respectively. The saturation score (“SaturationScore”) is the sum of the multiplication of each histogram bin (“Historgam[i]’) with its corresponding weighting value (“ WeightingTable[i]’): y- Histogram\i\ * WeigktingTable[i] 12 PCT/CA2013/050060 WO 2014/110654
Once the saturation score has been determined for the image, the saturation score is compared to an upper threshold value maxScore (step 216) and a lower threshold value minScore (step 218). When the saturation score is higher than the upper threshold value maxScore the image is considered highly oversaturated, and when the saturation score is below the lower threshold value minScore the image is considered not saturated. When the saturation score is between maxScore and minScore the image is considered slightly oversaturated, As will be discussed below, steps are taken to lower the auto exposure target when the image is considered oversaturated in order to reduce the degree of saturation in the image, and to increase the auto exposure target when the image is considered not saturated and under exposed to improve the exposure in the image without causing the image to be oversaturated.
The values of maxScore and minScore are determined empirically and will depend on the imager used and operator preferences. A typical value for maxScore can be 20 pixels and a typical value for minScore can be 0.
When the saturation score is above maxScore, the current auto-exposure target aeTrgt is reduced by a large constant factor L from its original value origAeTrgt (step 220) in order to cause the exposure to reduce relatively rapidly, wherein
aeTrgt = origAeTrgt* L
The large constant factor L is empiricaliy determined and can depend on operator preferences. A typical value for L can be 0.1, The auto exposure target for the auto exposure operation (AutoExposureTarget) is then set to the current auto exposure target aeTrgt (step 232), the auto exposure operation is executed to adjust the exposure to meet the new auto exposure target and the method returns to step 202,
When the saturation score is between minScore and maxScore, the current auto exposure target aeTrgt is reduced from the current global mean exposure level of the image (GloballmageMean) by a small constant factor S, in order to cause the exposure to reduce relatively slowly (step 222):
aeTrgt = aeMean*S 13 PCT/CA2013/050060 WO 2014/110654 wherein aeMean = GiobailmageMean (step 221)
The small constant factor S has a higher value than the large constant factor L and can be a factor that is linearly interpolated between values S1 and S2 based on the saturation score and determined by: 5 S = (S2+(S2-S1)*(saturationscore-minScore)/{maxScore-minScore)) wherein S1 and S2 represent the degree of change in the auto-exposure target. S1 and S2 are empirically determined and can depend on user preferences. S1 is larger than S2 and can for example be 0,9 and 0.5 respectively. The auto exposure target for the auto exposure operation (AutoExposureTarget) is then set to the current auto exposure 10 target aeTrgt (step 232), the auto exposure operation is executed to adjust the exposure to meet the new auto exposure target and the method returns to step 202.
When the saturation score is smaller than minScore, the image is considered to be non-saturated and the auto-exposure target aeTrgt is increased if the image is underexposed; underexposed in this context means that the pixels having the highest 15 tonal values in the image have a mean tonal value that is below a tonal value threshold above which a pixels is considered oversaturated. Alternatively, underexposed can mean when the pixel having the highest tonal value in the image is less than the tonal value threshold. Assuming the current auto exposure target aeTrget is smaller than the original auto exposure target (step 224), a series of steps are carried out to increase the 20 auto exposure target, provided the image is in fact underexposed. In order to determine if the image is underexposed, the mean tonal value of a selected number of upper pixels n of the histogram hiMean is determined (Step 226). In other words, the n pixels in the image with the highest tonal values are identified and the mean of their tonal values is determined. The value of n is empirically determined: a typical value for n can 25 be 10. Then, a constant value maxMean is selected which represents the tonal value threshold above which a pixel is considered oversaturated; this value can be empirically determined and will be in part dependent on the imaging sensor used to capture the image. A typical value for maxMean is a value that is less than or equal to the second selected total value m used in the histogram weighting table, which in this case is 230. 14 2013374190 22 Sep 2016
Since the saturation score is less than minScore, hiMean should be close to maxMean. The auto-exposure target is then set to the current global mean exposure level of the image (aeMean = GloballmageMean) (step 225) scaled by the ratio between hiMean and maxMean (step 228): aeTrgt = aeMean*{maxMean/hiMean)
Therefore, when hiMean is less than maxMean the ratio is greater than 1 and the auto exposure target will be set at a value that is greater than the current global mean exposure level of the image. It can be seen that when hiMean equals maxMean, the image is not considered to be underexposed and thus the auto exposure target is set to the global mean exposure level of the image.
The auto exposure target should not be increased above the original auto exposure targetjf the above scaling results in the autoexposure target aeTrgt being greater than the original auto exposure target origAeTrgt then the auto exposure target is set to equal the original auto exposure target (step 230): aeTrgt = MIN(origAeTrgt, aeTrgt)
Finally the auto exposure target for the auto exposure operation (AutoExposureTarget) is set at the current auto exposure target aeTrgt and the auto exposure operation is executed, and the method returns to step 202.
While the present invention has been described herein by the preferred embodiments, it will be understood to those skilled in the art that various changes may be made and added to the invention. The changes and alternatives are considered within the spirit and scope of the present invention.
Throughout this specification and the claims which follow, unless the context requires otherwise, the word "comprise", and variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated integer or step or group of integers or 15 2013374190 22 Sep 2016 steps but not the exclusion of any other integer or step or group of integers or steps.
The reference in this specification to any prior publication (or information derived from it), or to any matter which is known, is not, and should not be taken as, an acknowledgement or admission or any form of suggestion that that prior publication (or information derived from it) or known matter forms part of the common general knowledge in the field of endeavour to which this specification relates.
15A

Claims (18)

  1. The claims of the present invention are as follows:
    1. A method for adjusting an auto exposure target in an auto-exposure operation on a sequence of images, comprising: (a) prior to obtaining a histogram of an image, masking static or mostly static background saturated pixels in the image by identifying pixels in the image that have exceeded a saturation threshold a selected number of times over a time period, and applying a mask to the identified pixels such that the masked pixels are not included in the histogram; (b) obtaining the histogram of the image; (c) applying a weighted histogram table to the histogram to obtain weighted tonal value bins wherein at least some bins in the histogram containing oversaturated pixels are assigned a higher weighting value and at least some bins in the histogram containing non-saturated pixels are assigned a lower weighting value; and (d) summing the weighted histogram bins to obtain a saturation score and decreasing an auto exposure target for an auto exposure operation when the saturation score indicates that the image is oversaturated, and increasing the auto exposure target when the saturation score indicates that the image is non-saturated and when the image is underexposed.
  2. 2. A method as claimed in claim 1 wherein the step of identifying pixels as being static or mostly static background pixels comprises reading a saturation level of a pixel and incrementing a counter when the pixel exceeds the saturation threshold and identifying the pixel as being a static or mostly static saturated background pixel when the counter exceeds a background saturated pixel threshold value.
  3. 3. A method as claimed in claim 1 further comprising unmasking masked pixels by reading a saturation level of a masked pixel and decrementing a counter when the masked pixel falls below the saturation threshold, and removing the mask from the masked pixel when the counter falls below a background saturated pixel threshold value.
  4. 4. A method as claimed in claim 1 wherein the histogram is obtained from only one out of every selected number of images in the sequence of images, wherein the selected number of images is greater than the number of images in the sequence of images to which the auto-exposure operation is applied.
  5. 5. A method as claimed in claim 4 wherein the weighted histogram table has a weighting value that linearly increases from the bins assigned the lower weighting value to the bins assigned the higher weighting value, wherein the linear increase occurs over a selected range of bins.
  6. 6. A method as claimed in claim 5 wherein the higher weighting value is 1 and the lower weighting value is 0.
  7. 7. A method as claimed in claim 6 further comprising decreasing the auto exposure target by a first amount when the image is highly oversaturated, and decreasing the auto exposure target by a second amount when the image is slightly oversaturated, wherein the saturation score when the image is highly saturated is higher than the saturation score when the image is slightly saturated.
  8. 8. A method as claimed in claim 7 wherein the first amount is a function of the original auto exposure target, and the second amount is a function of a global mean exposure level of the image.
  9. 9. A method as claimed in claim 1 wherein the images are infrared images.
  10. 10. A method as claimed in claim 1 wherein the image is underexposed when a selected number of pixels with highest tonal values in the image have a mean tonal value that is below a tonal value threshold above which a pixel is considered oversaturated.
  11. 11. A method as claimed in claim 10 wherein the auto exposure target is increased by the global mean exposure level of the image multiplied by the ratio of the tonal value threshold above which a pixel is considered oversaturated to the mean tonal value of the selected number of pixels having the highest tonal values in the image.
  12. 12. An imaging apparatus comprising: (a) an imager; and (b) processing circuitry communicative with the imager to receive an image captured by the imager, the processing circuitry comprising a processor and a memory having encoded thereon program code executable by the processor to perform a method of adjusting an auto exposure target in an auto-exposure operation on a sequence of images, comprising: (i) prior to obtaining a histogram of an image, masking static or mostly static background saturated pixels in the image by identifying pixels in the image that have exceeded a saturation threshold a selected number of times over a time period, and applying a mask to the identified pixels such that the masked pixels are not included in the histogram; (ii) obtaining the histogram of the image; (iii) applying a weighted histogram table to the histogram to obtain weighted tonal value bins wherein at least some bins in the histogram containing oversaturated pixels are assigned a higher weighting value and at least some bins in the histogram containing non-saturated pixels are assigned a lower weighting value; and (iv) summing the weighted histogram bins to obtain a saturation score and decreasing an auto exposure target for an auto exposure operation when the saturation score indicates that the image is oversaturated, and increasing the auto exposure target when the saturation score indicates that the image is non-saturated and when the image is underexposed.
  13. 13. A method for generating a mask of background saturated pixels in an image, comprising: (a) identifying pixels in the image that exceed a saturation threshold a selected number of times over a time period as being static or mostly static background saturated pixels; and (b) masking a pixel by assigning a mask flag to each pixel in the image identified as being a static or mostly static background saturated pixel.
  14. 14. A method as claimed in claim 13 wherein step (a) comprises reading a saturation level of a pixel and incrementing a counter when the pixel exceeds the saturation threshold and identifying the pixel as being a static or mostly static saturated background pixel when the counter exceeds a background saturated pixel threshold value.
  15. 15. A method as claimed in claim 13 further comprising unmasking a masked pixel by reading a saturation level of a masked pixel and decrementing a counter when the masked pixel falls below the saturation threshold, and removing the mask flag from the masked pixel when the counter falls below a background saturated pixel threshold value.
  16. 16. A method as claimed in claim 13 further comprising two saturation thresholds, namely an upper saturation threshold above which a pixel is considered saturated and a lower saturation threshold below which a pixel is considered non-saturated.
  17. 17. An imaging apparatus comprising: (a) an imager; and (b) processing circuitry communicative with the imager to receive an image captured by the imager, the processing circuitry comprising a processor and a memory having encoded thereon program code executable by the processor to perform a method of generating a mask of background saturated pixels in an image, comprising: (i) identifying pixels in the image that exceed a saturation threshold a selected number of times over a time period as being static or mostly static background saturated pixels; and (ii) masking a pixel by assigning a mask flag to each pixel in the image identified as being a static or mostly static background saturated pixel.
  18. 18. An apparatus as claimed in claim 17 wherein the imager is an infrared imager.
AU2013374190A 2013-01-15 2013-01-28 Imaging apparatus with scene adaptive auto exposure compensation Active AU2013374190B2 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201361752889P 2013-01-15 2013-01-15
US61/752,889 2013-01-15
PCT/CA2013/050060 WO2014110654A1 (en) 2013-01-15 2013-01-28 Imaging apparatus with scene adaptive auto exposure compensation

Publications (2)

Publication Number Publication Date
AU2013374190A1 AU2013374190A1 (en) 2015-07-23
AU2013374190B2 true AU2013374190B2 (en) 2016-11-10

Family

ID=51164846

Family Applications (1)

Application Number Title Priority Date Filing Date
AU2013374190A Active AU2013374190B2 (en) 2013-01-15 2013-01-28 Imaging apparatus with scene adaptive auto exposure compensation

Country Status (14)

Country Link
US (2) US9536292B2 (en)
EP (1) EP2946249B1 (en)
JP (1) JP5981053B2 (en)
KR (1) KR101883037B1 (en)
CN (1) CN105190424B (en)
AU (1) AU2013374190B2 (en)
BR (1) BR112015016525B8 (en)
CA (1) CA2896825C (en)
HK (1) HK1216192A1 (en)
IL (1) IL239679A0 (en)
MX (1) MX349155B (en)
NZ (1) NZ709731A (en)
SG (1) SG11201505509RA (en)
WO (1) WO2014110654A1 (en)

Families Citing this family (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104702849B (en) * 2014-01-10 2018-03-30 杭州海康威视数字技术股份有限公司 A kind of thermal camera and its infrared lamp luminance regulating method
WO2015106138A1 (en) * 2014-01-10 2015-07-16 Avigilon Corporation Camera housing for reducing internal reflections
JP6345500B2 (en) * 2014-06-20 2018-06-20 株式会社ソシオネクスト Digital camera exposure control method and digital camera
WO2016009729A1 (en) * 2014-07-18 2016-01-21 ソニー株式会社 Imaging control device, imaging apparatus, and imaging control method
CN107454344B (en) * 2015-06-15 2019-05-24 Oppo广东移动通信有限公司 A kind of acquisition parameters setting method and user terminal
CN104954700B (en) * 2015-06-17 2018-09-14 浙江宇视科技有限公司 A kind of overexposure control method and device
TWI581632B (en) * 2016-06-23 2017-05-01 國立交通大學 Image generating method and image capturing device
TW201822709A (en) * 2016-12-30 2018-07-01 曦威科技股份有限公司 Real-time heart rate detection method and real-time heart rate detection system therefor
KR20180078961A (en) 2016-12-30 2018-07-10 삼성전자주식회사 Image capturing device and methos of operating the same
CN110476416B (en) * 2017-01-26 2021-08-17 菲力尔系统公司 System and method for infrared imaging in multiple imaging modes
US11238281B1 (en) * 2017-02-27 2022-02-01 Amazon Technologies, Inc. Light source detection in field of view
CN108668088A (en) * 2017-04-02 2018-10-16 田雪松 Sense image formation control method and device
US10929955B2 (en) 2017-06-05 2021-02-23 Adasky, Ltd. Scene-based nonuniformity correction using a convolutional recurrent neural network
US10511793B2 (en) * 2017-06-05 2019-12-17 Adasky, Ltd. Techniques for correcting fixed pattern noise in shutterless FIR cameras
CN110892322A (en) 2017-06-05 2020-03-17 威智伦公司 Ball type camera
US11012594B2 (en) * 2017-06-05 2021-05-18 Adasky, Ltd. Techniques for correcting oversaturated pixels in shutterless FIR cameras
WO2018226437A1 (en) 2017-06-05 2018-12-13 Adasky, Ltd. Shutterless far infrared (fir) camera for automotive safety and driving systems
US10699386B2 (en) * 2017-06-05 2020-06-30 Adasky, Ltd. Techniques for scene-based nonuniformity correction in shutterless FIR cameras
US10491808B1 (en) 2017-06-27 2019-11-26 Amazon Technologies, Inc. Detecting sunlight in images
RU2667790C1 (en) 2017-09-01 2018-09-24 Самсунг Электроникс Ко., Лтд. Method of automatic adjustment of exposition for infrared camera and user computer device using this method
TWI697867B (en) * 2018-12-12 2020-07-01 晶睿通訊股份有限公司 Metering compensation method and related monitoring camera apparatus
KR102622112B1 (en) * 2019-01-11 2024-01-05 한화비전 주식회사 Thermo image processing method, and apparatus thereof
EP3709628B1 (en) * 2019-03-14 2020-12-23 Axis AB Control of an illuminator
WO2020219873A1 (en) * 2019-04-24 2020-10-29 Hubbell Incorporated System and method for integrated surveillance and communication into lighting equipment
KR20210123602A (en) * 2020-04-03 2021-10-14 에스케이하이닉스 주식회사 Image sensing device and operating method of the same
CN114071026B (en) * 2022-01-18 2022-04-05 睿视(天津)科技有限公司 Automatic exposure control method and device based on red component characteristic detection
CN114885104A (en) * 2022-05-06 2022-08-09 北京银河方圆科技有限公司 Method for self-adaptive adjustment of camera, readable storage medium and navigation system

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070009253A1 (en) * 2003-06-30 2007-01-11 Jarno Nikkanen Method and system in digital imaging for adjusting exposure and a corresponding device
US20080117318A1 (en) * 2006-11-20 2008-05-22 Sony Corporation Image processing apparatus, image processing method, and program
US20080158430A1 (en) * 2006-12-29 2008-07-03 Hu Shane C Method, apparatus and system using hierarchical histogram for automatic exposure adjustment of an image

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6816200B1 (en) * 1998-08-31 2004-11-09 Neostar, Inc. Method and apparatus for detecting camera sensor intensity saturation
JP3465632B2 (en) * 1999-06-04 2003-11-10 日本電気株式会社 Object detection device and object detection method
US20040100565A1 (en) * 2002-11-22 2004-05-27 Eastman Kodak Company Method and system for generating images used in extended range panorama composition
US7327504B2 (en) 2002-12-06 2008-02-05 Eastman Kodak Company Method of detecting clipped image pixels
US7538801B2 (en) * 2003-09-15 2009-05-26 Micron Technology, Inc. Region-based auto gain control and auto exposure control method and apparatus
JP4580820B2 (en) * 2005-06-02 2010-11-17 オリンパスイメージング株式会社 Image display device
US20070024721A1 (en) * 2005-07-29 2007-02-01 Rogers Sean S Compensating for improperly exposed areas in digital images
KR100793230B1 (en) * 2006-07-24 2008-01-10 엘지전자 주식회사 Apparatus and method for compensating a partial back light in camera
MX2010000931A (en) * 2007-07-25 2010-04-21 Candela Microsystems S Pte Ltd Exposure control for an imagning system.
WO2009018647A1 (en) * 2007-08-08 2009-02-12 Tony Mayer Non-retro-reflective license plate imaging system
JP4877319B2 (en) * 2008-12-24 2012-02-15 カシオ計算機株式会社 Image generating apparatus, program, image display method, and imaging method
JP5469527B2 (en) * 2010-05-10 2014-04-16 パナソニック株式会社 Imaging device
JP2013005017A (en) 2011-06-13 2013-01-07 Sony Corp Image pickup apparatus, image pickup apparatus control method, and program
JP5950196B2 (en) * 2011-08-30 2016-07-13 株式会社リコー Imaging apparatus, and image analysis apparatus and moving apparatus using the same
JP2013197892A (en) * 2012-03-19 2013-09-30 Fujitsu Ltd Object recognition apparatus, object recognition method, and computer program for object recognition

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070009253A1 (en) * 2003-06-30 2007-01-11 Jarno Nikkanen Method and system in digital imaging for adjusting exposure and a corresponding device
US20080117318A1 (en) * 2006-11-20 2008-05-22 Sony Corporation Image processing apparatus, image processing method, and program
US20080158430A1 (en) * 2006-12-29 2008-07-03 Hu Shane C Method, apparatus and system using hierarchical histogram for automatic exposure adjustment of an image

Also Published As

Publication number Publication date
CN105190424A (en) 2015-12-23
CA2896825C (en) 2019-09-10
EP2946249A4 (en) 2016-11-02
AU2013374190A1 (en) 2015-07-23
KR101883037B1 (en) 2018-07-27
EP2946249B1 (en) 2018-05-23
WO2014110654A1 (en) 2014-07-24
NZ709731A (en) 2016-12-23
KR20150139826A (en) 2015-12-14
US9536292B2 (en) 2017-01-03
US20140198218A1 (en) 2014-07-17
BR112015016525A2 (en) 2017-07-11
HK1216192A1 (en) 2016-10-21
MX2015009082A (en) 2016-03-11
BR112015016525B8 (en) 2023-02-28
CN105190424B (en) 2018-06-01
MX349155B (en) 2017-07-14
BR112015016525B1 (en) 2021-09-14
JP5981053B2 (en) 2016-08-31
US20160142608A1 (en) 2016-05-19
JP2016510532A (en) 2016-04-07
IL239679A0 (en) 2015-08-31
SG11201505509RA (en) 2015-08-28
US9615032B2 (en) 2017-04-04
CA2896825A1 (en) 2014-07-24
EP2946249A1 (en) 2015-11-25

Similar Documents

Publication Publication Date Title
AU2013374190B2 (en) Imaging apparatus with scene adaptive auto exposure compensation
RU2565343C2 (en) Imaging device and control method
US10349486B2 (en) Method and apparatus for generating an infrared illumination beam with a variable illumination pattern
US10084967B1 (en) Systems and methods for regionally controlling exposure time in high dynamic range imaging
US11418730B2 (en) Use of IR pre-flash for RGB camera&#39;s automatic algorithms
WO2012093519A1 (en) Imaging device and light emission amount control method
KR20180086646A (en) Adaptive exposure control apparatus for a camera
KR101971535B1 (en) Apparatus and method for adjusting auto focus in image taking device
KR100978675B1 (en) Method of controlling white balance at flashlight photographing
JP2020101624A (en) Imaging apparatus, electronic apparatus, and control method and program of imaging apparatus
CN117278867B (en) Image sensor chip imaging system, exposure method and device thereof and storage medium
JP5546381B2 (en) Imaging apparatus and control method thereof
AU2013374191B2 (en) Method and apparatus for generating an infrared illumination beam with a variable illumination pattern
JP6148577B2 (en) Imaging apparatus and control method
KR20120117155A (en) Video camera with adjustable brightness of light and method for adjusting of the same
JP2009055174A (en) Image processor and imaging apparatus
JP2015005948A (en) Imaging apparatus, control method of the same, and control program

Legal Events

Date Code Title Description
FGA Letters patent sealed or granted (standard patent)
PC Assignment registered

Owner name: MOTOROLA SOLUTIONS, INC.

Free format text: FORMER OWNER(S): AVIGILON CORPORATION