WO2015105643A1

WO2015105643A1 - System and method for tone mapping of images

Info

Publication number: WO2015105643A1
Application number: PCT/US2014/070568
Authority: WO
Inventors: Huai Dong LI
Original assignee: Marvell World Trade Ltd
Current assignee: Marvell World Trade Ltd
Priority date: 2014-01-13
Filing date: 2014-12-16
Publication date: 2015-07-16
Anticipated expiration: 2016-07-13
Also published as: US20150201109A1; US9420145B2

Abstract

A method and imaging device are provided. An image is divided into N zones, and a histogram is generated for each of the N zones. A tone curve is determined for each of the N zones, where tone curve parameters including the controlling of shape and transition point of the curve are based on the histogram of each zone. A tone -mapped value for each pixel of the image is determined based on a sum of M weighted values determined by applying tone curves of M zones to a value of the pixel.

Description

SYSTEM AND M ETHOD FOR TONE MAPPING OF IMAGES

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This disclosure claims priority to U.S. Provisional Patent Application No. 61/926,673, filed on January 13, 2014, which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

[0002] The technology described in this document relates generally to image signal processing (ISP) methods and more particularly to systems and methods for tone mapping of images.

BACKGROUND

[0003] Semiconductor image sensors are used to sense radiation that includes, for example, visible light. Complementary metal-oxide-semiconductor (CMOS) image sensors and charge- coupled device (CCD) sensors are widely used in digital cameras and mobile phone cameras. These sensors utilize an array of pixels located in a substrate, where the pixels include photodiodes and transistors. The pixels absorb radiation projected toward the substrate and convert the absorbed radiation into electrical signals, and these electrical signals are used in forming digital images. Tone mapping techniques may be used in transforming digital images to produce images that are more visually appealing to a viewer. Tone mapping techniques are used, generally, to map one set of colors or image characteristics to another set of values.

SUMMARY

[0004] The present disclosure is directed to an imaging device and a method of processing an image. In an example method of processing an image, an image is divided into N zones. A histogram is generated for each of the N zones. A tone curve is determined for each of the N zones, where each of the tone curves is determined based on the histogram for the zone. A tone-mapped value for each pixel of the image is determined based on a sum of M weighted values determined by applying tone curves of M zones to a value of the pixel.

[0005] In another example, an imaging device includes a pixel array and a processor for processing pixel output signals received from the pixel array. The processor is configured to divide an image into N zones and generate a histogram for each of the N zones. The processor is further configured to determine a tone curve for each of the N zones, where each of the tone curves is determined based on the histogram for the zone. The processor is also configured to determine a tone-mapped value for each pixel of the image based on a sum of weighted values determined by applying tone curves of M zones to a value of the pixel.

BRIEF DESCRIPTION OF THE FIGURES

[0006] Fig. 1A depicts an image divided into a plurality of zones.

[0007] Fig. I B depicts a determining of a tone-mapped value for a pixel of the image based on a sum of M weighted values.

[0008] Figs. 2A and 2B depict example S-shaped tone curves.

[0009J Fig. 3 depicts an example luma histogram for a zone that is used in determining a tone curve for the zone.

[0010] Fig. 4 is a flowchart of an example process for determining tone curves for each of the N zones of an image.

[0011] Fig. 5 A depicts an image including pixels, where tone-mapped values are determined for each of the pixels based on a sum of M weighted values.

[0012] Fig. 5B depicts an image divided into an 8 zone x 8 zone matrix, where 5 zone x 5 zone neighborhoods are used in determining the tone-mapped values for pixels of the image.

9 [0013] Fig. 6 depicts a dividing of an image into N zones and a detection of a face zone included in the N zones.

[0014] Fig. 7 is a block diagram of an example imaging device that includes an image processing unit configured to perform image processing according to the approaches described herein.

[0015] Fig. 8 is a flowchart illustrating an example method of processmg an image according to the approaches described herein.

DETAILED DESCRIPTION

[0016] Fig. 1A depicts an image 100 divided into a plurality of zones 154- 162. In an example, the image 100 is a digital image captured with a personal electronic device, such as a digital camera, digital video camera, smartphone, mobile device (e.g., tablet computer, etc.), or other imaging device. In capturing a scene, these devices may produce digital images having a linear mapping between scene radiance and pixel intensity. For this reason and others, the digital images produced by these devices may be visually unappealing. To render these digital images more visually appealing, the tone mapping techniques described herein are applied. In general, tone mapping is a technique used in image processing to map one set of colors or image characteristics to another set of values.

[0Θ17] The tone mapping techniques described herein utilize a local tone mapping method, rather than a global tone mapping method. Global tone mapping methods apply a single tone curve to all pixels of an image. A tone curve is a function that maps original pixel values of an image to new pixel values and may be based on, for example, a power function, a logarithmic function, a sigmoidal (i.e., S-shaped) function, or another function (e.g., a function that is image dependent). In the global tone mapping methods, one input value results in one and only one output value. By contrast, in the local tone mapping method described herein, one input value may result in multiple different tone -mapped output values, with the tone -mapped output values varying based on a spatial location of a pixel being transformed.

[0018] As an example of the local processing used in the local tone mapping method, a first pixel is located at a first location of the image 100, and a second pixel is located at a second location of the image 100, The second location is different from the first location. The first and second pixels each have a same pixel value xj. The local tone mapping method is applied to the first pixel, with the pixel value xi being processed to generate a tone-mapped value vi for the first pixel. The local tone mapping method is further applied to the second pixel, with the pixel value xj being processed to generate a tone-mapped value y₂ for the second pixel. Due to the local processing methods applied, the tone-mapped value v._? is different from v_/.

[001 ] To apply the local tone mapping method to the image 100 of Fig. 1A, the image 100 is divided into N zones 154-162. In the example of Fig. 1 A, N is equal to nine, and the image 100 is thus divided into the nine non-overlapping zones 154-162 depicted in the figure. For each of the N zones 154-162, a histogram of the luma components of the pixels that form the zone is generated. A luma component for a pixel indicates an intensity or brightness of the pixel. The resulting luma histogram for each zone is a representation of the frequency of occurrence of each intensity or brightness value for pixels in the zone. An example luma histogram 300 is depicted in Fig. 3 and described in further detail below. It should be understood that the luma histogram generated for each of the N zones 154-1 62 need not include a graphical representation. In an example, the luma histogram for a zone includes only data (e.g. , luma values and corresponding pixel counts) from which statistics of the histogram are calculated. [0020] For each of the N zones 154- 162, a local tone curve is determined. As explained above, a tone curve is a function that maps original pixel values of an image to new pixel values. In the image 100, a first zone 154 is associated with a first local tone curve Tn, a second zone 155 is associated with a second local tone curve T₁₂, and so on. Each of the different tone curves Tn-T ₃ associated with the zones 154-162 may have different parameters that are based on the local image statistics for each of the zones 154-162. Specifically, as described in further detail below with reference to Fig. 3, each of the tone curves is determined based on the luma histogram for the zone. Thus, for the first zone 154, the first local tone curve Tn is based on the luma histogram for the first zone 154, and for the second zone 155, the second local tone curve Ti2 is based on the luma histogram for the second zone 155, and so on.

[0021] After determining the tone curves for each of the zones 154-162, tone-mapped values for each of the pixels in the image 100 are determined. As described in further detail below with reference to Figs. 5A and 5B, the determining of a tone-mapped value for a pixel of the image 100 is based on a sum of M weighted values. The M weighted values are determined by applying tone curves of M zones to a value of the pixel . Determining a single weighted value of the M weighted values includes i) applying to the value of the pixel a tone curve for a particular zone to determine a non-weighted value, and ii) weighting the non-weighted value based on a distance from the pixel to a center point of the particular zone and an intensity difference between the value of the pixel and an average pixel value of the particular zone. The applying and weighting steps are performed for each of the M zones to determine the M weighted values for the pixel. The tone-mapped value for the pixel is based on the sum of the M weighted values, with the sum being normalized as described below with reference to Figs. 5A and 5B. The M zones include a first zone of the N zones 154-162 in which the pixel is located and one or more other zones of the N zones 154-162 that are within a neighborhood of the first zone,

[0022] To illustrate the determining of a tone-mapped value for a pixel of the image 100, Fig. I B depicts a pixel 152 located within a zone 158 of the image. The determining of a tone- mapped value for the pixel 152 is based on a sum of M weighted values. In the example of Fig. IB, M is equal to N, but it should be understood that in other examples, M is less than N. Such other examples are illustrated in Figs, 5 A and 5B and described in detail below. The M weighted values are determined by applying tone curves of M zones to a value of the pixel 152. In the example of Fig. IB, with M equal to N, tone curves for all of the N zones 154-162 are applied to the value of the pixel 152. To determine a first weighted value of the M weighted values, a tone curve for the zone 158 is applied to the value of the pixel 152 to determine a first non-weighted value. The first non-weighted value is weighted based on i) a distance from the pixel 152 to a center point of the zone 158, and ii) an intensity difference between the value of the pixel 152 and an average pixel value of the zone 158. The weighting of the first non- weighted value yields the first weighted value of the M weighted values.

[0023] Similarly, to determine a second weighted value of the M weighted values, a tone curve for the zone 154 is applied to the value of the pixel 152 to determine a second non- weighted value. The second non-weighted value is weighted based on a distance from the pixel 152 to a center point of the zone 154 and an intensity difference between the value of the pixel 152 and an average pixel value of the zone 154. The rest of the M weighted values for the pixel 152 are determined by repeating these steps for the remaining zones (i.e. zones 155- 157 and 159-162). The tone-mapped value for the pixel 152 is determined based on a sum of the M weighted values. [0024] In determining the tone-mapped value for the pixel 152, tone curves for a neighborhood of zones are applied to the value of the pixel 152. Although the example of Fig. IB utilizes a "3 zone x 3 zone" neighborhood, it should be understood that other types of neighborhoods are used in other examples. In another example, the image 100 is divided into an 8 zone x 8 zone matrix, and neighborhoods used in determining the tone-mapped values are 3 zone x 3 zone neighborhoods or 5 zone x 5 zone neighborhoods, for example. Such an 8 x 8 zone matrix with 3 x 3 and 5 x 5 zone neighborhoods are illustrated in Figs. 5A and 5B, respectively.

[0025] The local tone mapping method described herein is used for a variety of purposes, such as reproducing color images captured with a normal sensor for a visually pleasing display. Although the described systems and methods are said to implement a "local tone mapping method," it should be understood that if the image 100 is considered to comprise a single zone (i.e., the image 100 is not divided into multiple zones), then the systems and methods implement a global tone mapping method.

[0026] Figs. 2A and 2B are graphs 200, 250, respectively, showing example S-shaped tone curves having different parameter values. As explained above, a tone curve applied in a tone mapping method may be based on, for example, a power function, a logarithmic function, a sigmoidal function, or another function. In an example of the approach described herein, an S- shaped curve function is used. An example S-shaped tone curve for a particular zone is

(Equation 1)

where x is a value of a pixel to which the tone curve is applied, y is an output value of the tone curve, and a and γ are parameters of the tone curve. In the systems and methods described herein, each zone of an image has its own tone curve with a and γ parameters determined specifically for the zone. In consi dering a particular zone of the image, the parameters a and γ of the tone curve are set based on a determination that the particular zone is over-exposed, underexposed, or of normal exposure. This is described in further detail below with reference to Fig. 3.

[0027] In Equation 1, the parameter a controls the transition point in the piecewise function T, and the parameter γ controls the shape of the tone curve. If γ is equal to 1.0, then the tone curve is a linear curve with the output value y being equal to the value x of the pixel. Fig. 2 A. is a graphical depiction of the S-shaped tone curve described by Equation 1. In Fig. 2 A, the value γ is equal to 2.0 for all curves, and a first tone curve 402 has a value of a equal to 0.2, a second tone curve 404 has a value of a equal to 0.4 , a third tone curve 406 has a value of a equal to 0.6, and a fourth tone curve 408 has a value of a equal to 0.8. Fig. 2B is another graphical depiction of the S-shaped tone curve described in Equation 1. In Fig. 2B, the value a is equal to 0.5 for ail curves, and a first tone curve 452 has a value of γ equal to 0.5, a second tone curve 454 has a value of γ equal to 1.0, a third tone curve 456 has a value of γ equal to 1.5, and a fourth tone curve 458 has a value of γ equal to 2.0.

[0028] Fig. 3 depicts an example luma histogram 300. In the local tone mapping method described herein, an input image is divided into N zones, and each of the N zones has its own tone curve. As explained above with reference to Figs. 2A and 2B, parameters a and γ for a tone curve of a particular zone are set based on a determination that the particular zone is overexposed, under-exposed, or of normal exposure. This determination is made based on a luma histogram for the particular zone. Fig. 3 depicts the example luma histogram 300 that is associated with a particular zone of an input image. An x-axis of the luma histogram 300 represents different luma values (i.e., brightness or intensity values), and a y-axis represents a number of pixels. Thus, the Junta histogram 300 shows a count of the pixels in the zone having each of the different luma values represented on . the x-axls. A luma value Mj_ama on the x-axis- represents a maximum luma value of the input data (i.e., a maximum luma value of the entire input image, as opposed to a maximum, luma value of the particular zone), and the luma histogram 300 shows thai the particular zone includes only a small number of pixels having this maximum luma value,

[ΘΘ29] The determination of whether the particular zone is over-exposed, under-exposed, or of normal exposure includes determining a normal feed average luma value Y ^ for the particular zone based on the luma histogram 300. The value YNj,*₈ is equal to an average luma value Y_av§ of the zone normalized by the maximum luma value M[_;,_mi, of the input data;

Y avg ^w V_aVg / ^luraa- {Equation ^.2)

The average luma value Y_8Vg -used in determining the value YN is determined from the luma histogram 300 for the particular zone.

[0030} if YN_m is greater than a first threshold value (e.g., a threshold "overEx thre^" referred to herein), the particular zone is determined to be over-exposed. Then, in setting the parameters a and y for the tone curve of the particular zone, the parameter a is set equal to- approximately 1 ^,0 (e.g., 0.9), and the parameter γ is set equal to a value greater than 1 .0 (e.g., 1.1 ). in an example, the parameter a is approximately equal to 1.0 if the parameter a has a value between 0J0 and L20,

[0031] If YH s less than a second threshold value (e.g.,. a threshold "underBxjhre" referred to herein), the particular zone is determined o be under-exposed. Then, in setting the parameters a and γ for the tone curve of the particular zone, the parameter a is se equal to approximately 0,0 (e.g., 0.02), and the parameter y is set equal to a value greater than 1 ,0 (e.g., 1 , ί .), In an example, the parameter a is approximately equal to 0.0 if the parameter a has a value between 0.00 and 0.20.

[0032] The particular zone is determined to be of normal exposure if the zone is not determined to be over-exposed, and the zone is not determined to be under-exposed. If the particular zone is of normal exposure, the parameter a for the tone curve of the zone is set equal to (H_lialf / M_lun,_a). Hhaif is a 50-percentile luma value, where fifty percent of the pixels in the particular zone have a luma value that is less than or equal to the 50-percentile luma value {i.e., Hhaif is a luma value where the histogram reaches percentile 50%). M_luma is the maximum luma value for the whole input data, as noted above. The value Hhaif is determined from the luma histogram 300 for the particular zone.

[0033] Additionally, if the particular zone is of normal exposure, it is determined, based on the luma histogram 300 for the particular zone, whether the zone has a wide dynamic range, a narrow dynamic range, or a normal dynamic range. The parameter γ is set based on this determination. Specifically, if the particular zone is of normal exposure, a value DR is determined from the luma histogram 300, and the determining of whether the particular zone has the wide dynamic range, the narrow dynamic range, or the normal dynamic range is based on the value DR. The value DR is a difference between a low-end histogram point, Hi_ow, and a high- end histogram point, Hwg_h. Hi_ow s & ^-percentile luma value where the histogram 300 reaches a predefined percentile q% (i.e., Hi_ow is the ^-percentile luma value, where q percent of the pixels in the particular zone have a luma value that is less than or equal to the ^-percentile luma value). An example Hi_ow value is depicted on the example luma histogram 300 of Fig. 3, which shows the Hj_cw luma value that is less than the ¾_aif luma value. ¾i_gh is a (100 - ^-percentile luma value where the histogram 300 reaches a predefined percentile (100 - q)% {i.e., H_hi_gh is the {100 - #)-percentile !uroa value, where (100 -- q) percent of the pixels in the particular zone have a luma value that is less than or equal to the (100 - ^ e centile luma value).

|0034] Using the DR value, it is determined whether the particular zone has a wide dynamic range, a narrow dynamic range, or a norma! dynamic range, and based on this determination, the tone curve parameter γ for the particular zone is set. Specifically, the particular zone is determined to have a wide dynamic range and γ is set equal to y-mm if DR h greater than R,_Tax. ^■y_iTilfi is a predetermined minimum value for y,_.. and D max is a predetermined maximum threshold value for DR., The particular zone is determined to have a narrow dynamic range and γ is set equal- to > _: if DR is less than DR_ro;„. γ_η¾χ is a predetermined ma imum value for y, and DR_m„_: is a pr detemiine minimum threshold value for DR. Setting γ equal to y_!!m: avoids the over- aniplillcatiorf of noise. lit an example, for a flat area like .a wail y„_tax is set to 1 ,0. Further, in another example, y_fna_X is set based on an analog gain applied so that ] ; is a small value for a low light scene with high gain.

ftK)35] The particular zone is determined to have a normal dynamic range if the gone is not determined to have the wide dynamic range, and the zone is not determined to have the narrow dynamic range. If the particular zone is determined to have the normal dynamic range, y is set equal to

0R-DR_max)^*(Yma ^"¾in

+ y_mj_n. (Equation.3}

tDRimax-DR * min

^■r mmm n)!

[0036] Fig. 4 is a flowchart 400 of an example process for determining tone curves for each of the N zones of an image, At 402, the input image is divided into N zones, and at 404, a iuma histogram is generated for each of ihe zones. To determine a tone curve for each of the zones, subsequent steps of the flowchart 400 are performed. Specifically, for a first zone of the N zones, at 406, a determination is made as to whether the first zone is over -exposed. If the zone

I I is over-exposed, at 408, a value a for a tone curve for the: first zone is set to a value approximately equal to 1.0 (e.g., 0.9) and γ is set to a suitable value (e.g., 1 , 1). It should he understood that the parameters a and γ discussed herein with reference to Fig. 4 are the parameters for the tone curve included in Equation I . At 10, a determination is made as to whether all zones of the N zones have been processed. If ail zones have been processed, the process ends, and i ali zones have not been processed, the flowchart returns to the determination at 406. and a determination is made as to whether a second zone of the. N^' zones is over-exposed.

[0037] If it is determined at 406 that the first zone is not over-exposed, at 4 12, a determination Is made as to whether the first zone is under-exposed. If the zone is under-exposed, at 14, the value tor the tone curve for the first: zone is set to a value approximately equal to 0.0 (e.g., 0.02) and y is set to a suitable value (e.g., 1 , 1 ), At 416, a determination is made as to whether all zones of the N zones have been processed. If ail zones have been processed, the process ends, and if all zones have not been processed, the flowchart returns to the determination at 406,_. and a determination is made as to whether the second zone of the N zones is over-exposed.

|0038| if it is ^'determined at. 412 that the first zone is not under-exposed, at 418. the first zone is ^'determined to be of normal exposure. Consequently, at 418, the parameters a and γ of the tone curve for th first zone are set as follows;

a - Hhalf ' M\_um, (Equation 4) x - DR_min) + jy _n else (Equation 5)

Variables included in Equations 4 and 5 above (e.g., H^f, Μι_ϋη1¾. y_m»j, y_m , DR, DR_ulii:(} D ^,^,} are described above with reference to Fig, 3. At 420, a determination is made as to whether allzones of the N zones have been processed. If all zones have been processed, the process ends, and if all zones have not been processed, the flowchart returns to the determination at 406, and a determination is made as to whether a second zone of the N zones is over-exposed.

[0039] Fig. 5 A depicts an image 500 including pixels 502, 512, 526, 546, where tone-mapped values are determined for each of the pixels 502, 512, 526, 546 based on a sum of M weighted values. The image 500 is divided into zones, and in the example of Fig. 5A, the image 500 is divided up into a matrix of 8 zones x 8 zones. Each of the zones of the image 500 has its own tone curve with corresponding a and γ parameters, with the a and γ parameters being set based on each zone's luma histogram statistics, as described above. After determining the tone curves for each of the zones of the image 500, tone-mapped values for each of the pixels in the image 500 are determined.

[0040] For each pixel in the image 500, the pixel's tone-mapped value is based on a sum of M weighted values. The M weighted values are determined by applying tone curves of M zones to a value of the pixel. Determining a single weighted value of the M weighted values includes i) applying to the value of the pixel a tone curve for a particular zone to determine a non- weighted value, and ii) weighting the non-weighted value based on a distance from the pixel to a center point of the particular zone and an intensity difference between the value of the pixel and an average pixel value of the particular zone. The applying and weighting steps are performed for each of the M zones to determine the M weighted values for the pixel. The M zones include a first zone of the 8 zone x 8 zone matrix in which the pixel is located and one or more other zones of the 8 zone x 8 zone matrix th at are located wi thin a neighborhood of the first zone.

[0041] To illustrate this approach to finding a tone-mapped output value for a pixel, Fig. 5A depicts a pixel 502 located within a zone 504 of the image 500. The determining of a tone- mapped value for the pixel 502 is based on a sum of M weighted values. For the pixel 502, M is equal to four, based on the neighborhood of zones 505 used in the -calculation- The M weighted values are determined by applying tone carves, of M ¾ones to a. value of the pixel 502. With. M equal to four for the pixel 502, tone curves for the zones 504, 506, 508, 510 are applied to the value of the pixel 502. To determine a first weighted valise of the M weighted values, a tone curve for the zone 504 is applied to the value of the pixel 502 to deiennine a first non-weighted value. The first non-weighted value is weighted based on i) a distance from the pixel 502 to a center point of the zone 504, and si) an intensity difference between the value of the pixel 502 and an average pixel value of the. zone 504. The weighting of {he first non-weighted value yields the first weighted value of the- M weighted values. Similarly, to determine -a second weighted value of the M weighted values, a tone curve for the zone 50.6 is applied to the value of the pixel 502 to determine a second non-weighted value. The second non-weighted value is. weighted based on a distance from the pixel 502 to a center point of the zone 506 and an intensity difference between the value of the pixel 502 and an average pixel value of the zone 506. The rest of the M weighted values for the pixel 502 are. determined by repeating these steps for the remaining zones (i.e. zones 508 and 510). The tone-mapped value for the pixel 502 is determined based on a sum of the M weighted values.

{0042] In an example, the determining of the tone-mapped value for the pixel 502 is based on '(¾^") (Equation 6)

where y is the tone-mapped value for the pixel 502, x is the original value of the pixel 502, T„ is a tone curve for a zone n of the M zones 504. 506, 50$, 5 1.0 defining the neighborhood 505·, wi* is a distance weighting function based on a distance from the pixel 502 to a center point of the zone n, w2„ is a similarity weighting function based on an intensity difference between the value of the pixel 502 and an average pixel value of the zone n. [0043] In an example, the distance weighting function wl_« is based on a first Gaussian function with values that vary based on the distance, and the similarity weighting function w2„ is based on a second Gaussian function with values that vary based on the intensity difference. The first Gaussian function has values that decrease wit increasing distance, such that a value of the distance weighting function wl„ decreases as the distance between the pixel 502 and the center point of the zone n increases. Similarly, the second Gaussian function has values that decrease with increasing intensity difference, such that a value of the similarity weighting function w2_n decreases as the intensity difference between the value of the pixel 502 and the average pixel value of the zone n increases. In other examples, the distance weighting function wl_» and the similarity weighting function w2^' _n are not Gaussian functions and are instead other functions.

[0044] Tone-mapped values for other pixels of the image 500 are determined in a similar manner using Equation 6. When considering different pixels of the image 500, a neighborhood of zones used in making the calculation varies. For example, in calculating the tone-mapped value for the pixel 502, four zones are used, where the four zones include the zone 504 in which the pixel 502 is located and the three zones 506, 508, 510 located within the neighborhood 505 of the zone 504. By contrast, in calculating the tone -mapped value for the pixel 526, nine zones are used (i.e., M is equal to nine in Equation 6), where the nine zones include the zone 536 in which the pixel 526 is located and the eight zones 528, 530, 532, 534, 538, 540, 542, 544 located within a neighborhood 503 of the zone 536. In calculating the tone-mapped values for the pixels 512 and 546, six zones included within neighborhoods 507 and 509, respectively, are used.

[0045] Each of the neighborhoods of zones 503, 505, 507, 509 illustrated in Fig. 5A is considered to be a 3 zone x 3 zone neighborhood. For the pixel 526 located within an interior of the image 500, all nine zones of the 3 zone x 3 zone neighborhood 503 are used. By contrast, for the pixels 502, 512 , 546 located near a periphery of the image 500, less than all nine zones of the 3 zone x 3 zone neighborhood are used. Thus, special treatment is used in applying the 3 zone x 3 zone neighborhood to pixels located near the edges of the image 500.

[0046] To further illustrate the variations in neighborhood size for calculating tone-mapped values for different pixels of an image, Fig. 5B depicts an image 560 with neighborhoods 568, 570, 572 used in calculating tone-mapped values for pixels 562, 564, 566, respectively. The image 560 is divided into an 8 zone x 8 zone matrix, and each of the neighborhoods of zones 568, 570, 572 is considered to be a 5 zone x 5 zone neighborhood. For the pixel 566 located within an interior of the image 560, ail twenty-five zones of the 5 zone x 5 zone neighborhood 572 are used (i.e., M is equal to twenty-five in Equation 6). By contrast, for the pixel 562 located near a corner of th e image 560, only nine zones of the 5 zone x 5 zone neighborhood 568 are used. Similarly, for the pixel 564 located near a top edge of the image 560, only fifteen zones of the 5 zone x 5 zone neighborhood 570 are used. It should be understood that the features of Figs. 5A and 5B are exemplary only. For example, although both Figs. 5A and 5B illustrate dividing an image into an 8 zone x 8 zone matrix, fewer zones or additional zones are used in other examples. Similarly, the 3 x 3 zone and 5 x 5 zone neighborhoods illustrated in these figures are examples only, and other neighborhood systems are used in other examples (e.g., neighborhoods of different sizes and/or shapes). In Figs. 5 A and 5B, the arrows extending from the pixels illustrate distances between the pixels and center points of the relevant zones.

[0047] Fig. 6 depicts an image 600 divided into N zones, where one of the zones is determined to be a face zone 602. Determining a tone curve for each of the N zones of the image 600 includes determining whether the zone is over-exposed, under-exposed, or of normal exposure, where the determination is based on a normalized average luma value YN_avs for the zone. Specifically, the value Y N_avg is compared to first and second threshold values overEx__thre and underEx thre, respectively, to determine if the zone is over-exposed, under-exposed, or of normal exposure, as described above. As referred to herein, the threshold value overEx thre is the over-exposed threshold value used in determining if the zone is over-exposed, and the threshold value underEx_thre is the under-exposed threshold value used in determining if the zone is under-exposed. However, if the zone is determined to include a face of a human being, then the overEx thre and underEx thre threshold values are modified in order to modify the tone curve parameters for the zone. Thus, in an example, the tone mapping methods described above are supplemented with face detection methods.

[0Θ48] In implementing the face detection methods, it is determined, for each of the N zones of the image 600, if the zone includes a face of a human being. For each zone including the face, the under-exposed threshold value underEx__thre is selected differently to cause an increase in the brightness of the zone including the under-exposed face. Similarly, for each zone including the face, the over-exposed threshold value overEx thre is selected differently based on the inclusion of the face in the zone. The YN_avg is calculated for the zone based on the skin-tone pixel values included in the zone and compared to the modified threshold values for the zone. Additionally, for each zone including the detected face, the γ parameter of the tone curve (e.g., as used in the S-shaped tone curve defined in Equation 1) is set to approximately 1.0 to avoid the enhancement of the contrast in the face.

[0049] Fig. 7 is a block diagram of an example imaging device 700 that includes an image processing unit 704 configured to perform image processing according to the approaches described herein. Embodiments of the example imaging device shown in Fig. 7 are included in stand-alone digital cameras and in digital cameras embedded in other electronic devices, such as cellular phones, laptop computers, and hand-held personal computing devices. As shown in Fig. 7, example imaging device 700 includes an optical sensing unit 702, an image processing unit 704, a data storage unit 706, and an input/output interface 708. In other examples, the example imaging device 700 further includes memory (e.g., as part of the data storage unit 706, the input/output interface 708, or as a standalone component) and a bus that connects one or more of the components 702, 704, 706, 708.

[0050] The components of the imaging device 700 are configured to provide image acquisition and tone mapping. In providing the image acquisition, the optical sensing unit 702 includes a pixel array or other components used to form a complementary metal-oxide-semiconductor (CMOS) image sensor or charge-coupled device (CCD) image sensor. In providing the tone mapping, the image processing unit 704 includes one or more processors for processing image pixel output signals that are generated by the optical sensing unit 702. The one or more processors of the image processing unit 704 obtain the pixel output signals and perform procedures to adjust the pixel output signals as necessary for the tone mapping.

[0051 J The data storage unit 706 and memory are configured to hold persistent and non- persistent copies of computer code and data. The computer code includes instructions that when accessed by the image processing unit 704 result in the imaging device 700 performing the tone mapping operations as described above. The data includes data to be acted upon by the instructions of the code, and in an example, the data includes stored pixel output signals and/or digital images formed by the pixel output signals. The processing unit 704 includes one or more single-core processors, multiple-core processors, controllers, or application-specific integrated circuits (ASICs), among other types of processing components. The memory includes random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), or dual-data rate RAM (DDRRAM), among other types of memory.

[0052] The data storage unit 706 includes integrated or peripheral storage devices, suc as, but not limited to, disks and associated drives (e.g., magnetic, optical), USB storage devices and associated ports, flash memory, read-only memory (ROM), or non-volatile semiconductor devices, among others. In an example, data storage unit 706 is a storage resource physically part, of the imaging device 700, and in another example, the data storage unit 706 is accessible by, but not a part of, the imaging device 700. The input/output interface 708 includes interfaces designed to communicate with peripheral hardware (e.g., remote optical imaging sensors or other remote devices). In various embodiments, imaging device 700 has more or less elements or a different archi tecture .

[0053] Fig. 8 is a flowchart 800 illustrating an example method of processing an image according to the approaches described herein. At 802, an image is divided into N zones. At 804, a limia histogram is generated for each of the N zones. At 806, a tone curve for each of the N zones is determined, where each of the tone curves is determined based on the luma histogram for the zone. At 808, a tone-mapped value is determined for each pixel of the image based on a sum of M weighted values determined by applying tone curves of M zones to a value of the pixel. Determining the M weighted values includes, at 810, applying to the value of the pixel a tone curve for a zone to determine a non-weighted value. Determining the M weighted values further includes, at 812, weighting the non-weighted value based on i) a distance from the pixel to a center point of the zone, and ii) an intensity difference between the value of the pixel and an average pixel value of the zone. At 814, the applying and weighting steps are performed for each of the M zones to determine the M weighted values for the pixel. The M zones include a first zone in which the pixel is located and one or more other zones located within a neighborhood of the first zone.

[0054] This written description uses examples to disclose the invention, including the best mode, and also to enable a person skilled in the art to make and use the invention. The patentable scope of the invention includes other examples. Additionally, the methods and systems described herein may be implemented on many different types of processing devices by program code comprising program instructions that are executable by the device processing subsystem. The software program instructions may include source code, object code, machine code, or any other stored data that is operable to cause a processing system to perform the methods and operations described herein. Other implementations may also be used, however, such as firmware or even appropriately designed hardware configured to carry out the methods and systems described herein,

[0055] The systems' and methods' data (e.g., associations, mappings, data input, data output, intermediate data results, final data results, etc.) may be stored and implemented in one or more different types of computer-implemented data stores, such as different types of storage devices and programming constructs (e.g., RAM, ROM, Flash memory, flat fi les, databases, programming data structures, programming variables, IF-THEN (or similar type) statement constructs, etc.). It is noted that data structures describe formats for use in organizing and storing data in databases, programs, memory, or other computer-readable media for use by a computer program.

[0056] The computer components, software modules, functions, data stores and data structures described herein may be connected directly or indirectly to each other in order to allow the flow of data needed for their operations. It is also noted that a module or processor includes but is not limited to a unit of code that performs a software operation, and can he implemented for example as a subroutine unit of code, or as a software function unit of code, or as an object (as in an object-oriented paradigm), or as an applet, or in a computer script language, or as another type of computer code. The software components and/or functionality may be located on a single computer or distributed across multiple computers depending upon the situation at hand.

[0Θ57] It should be understood that as used in the description herein and throughout the claims that follow, the meaning of "a," "an," and "the" includes plural reference unless the context clearly dictates otherwise. Also, as used in the description herein and throughout the claims that follow, the meaning of "in" includes "in" and "on" unless the context clearly dictates otherwise. Further, as used in the description herein and throughout the claims that follow, the meaning of "each" does not require "each and every" unless the context clearly dictates otherwise. Finally, as used in the description herein and throughout the claims that follow, the meanings of "and" and "or" include both the conjunctive and disjunctive and may be used interchangeably unless the context expressly dictates otherwise; the phrase "exclusive of may be used to indicate situations where only the disjunctive meaning may apply.

Claims

It is claimed:

1 , A method of processing an image, the method comprising:

dividing an image into N zones and generating a histogram for each of the N zones;

determining a tone c urve for each of the N zones, wherein each of th e tone curves is determined based on the histogram for the zone; and

determining a tone-mapped value for each pixel of the image based on a sum of

M weighted values determined by applying tone curves of M zones to a value of the pixel,

2, The method of claim 1 , wherein determining the M weighted values comprises:

applying to the value of the pixel a tone curve for a zone to determine a non- weighted value; weighting the non-weighted value based on i) a distance from the pixel to a center point of the zone, and ii) an intensity difference between the value of the pixel and an average pixel value of the zone; and

performing the applying and weighting steps for each of the M zones to determine the M weighted values for the pixel, wherein the M zones include a first zone in which the pixel is located and one or more other zones within a neighborhood of the first zone,

3, The method of claim 1, wherein the determining of the tone curve for each of the N zones includes:

determining, based on the histogram for the zone, whether the zone is over-exposed, under-exposed, or of normal exposure; and

based on the determination that the zone is over-exposed, under-exposed, or of normal exposure, setting parameters of the tone c urve for the zone.

4. The method of claim 3, wherein the determining of whether the zone is over-exposed. under-exposed, or of normal exposure includes:

determining, based on the histogram for the zone, a normalized average !uma value YN_avs for the zone, YN_av¾ being equal to an average luma value of the zone normalized by a maximum iuma value of the image;

determining that the zone is over-exposed if YN_avg is greater than a first threshold value; determining that the zone is under-exposed if YN_avg is less than a second threshold value; and

determining that the zone is of normal exposure if the zone is not determined to be overexposed, and the zone is not determined to be under-exposed.

5 The method of claim 4, wherein the tone curve for the zone is based on

where x is the value of the pixel to which the tone curve is applied, y is an output value of the tone curve, and a and γ are the parameters of the tone c urve set based on the deterrmnation that the zone is over-exposed, under-exposed, or of normal exposure.

6 The method of claim 5 comprising:

if the zone is of normal exposure:

determining a value D from the histogram, wherein DR is a difference between H_low H_hi_gh, wherein H_!ow is a ^-percentile luma value that is less than H_half, and is a (100 - percentile luma value, and wherein the determining of whether the zone has the wide dynamic range, the narrow dynamic range, or the normal dynamic range is based on the value DR;

determining that the zone has the wide dynamic range and setting γ equal to y_mi„ if DR is greater than DR_max, DR_max being a maximum threshold value for DR and ,„m being a minimum value for y;

determining that the zone has the narrow dynamic range and setting γ equal to y_max if DR is less than DR_mj_n, DRmia being a minimum threshold value for DR and y_max being a maximum value for γ; and

determining that the zone has a normal dynamic range and setting γ equal to

if the zone is not determined to have the wide dynamic range and not determined to have the narrow dynamic range.

7. The method of claim 1 comprising:

determining, for each of the N zones, if the zone includes a face of a h um an being; and modifying the tone curve for each of the N zones determined to include the face.

8. The method of claim 1 , wherein the determining of the tone-mapped value for each pixel is based on

^~ ∑£₌₁0vl_n * w2_n) '

where y is the tone-mapped value, x is the value of the pixel, T„ is a tone curve for a zone n of the M zones, wl„ is a distance weighting function based on a distance from the pixel to a center point of the zone n, w2„ is a similarity weighting function based on an intensity difference between the value of the pixel and an average pixel value of the zone n, wherem the distance weighting function and the similarity weighting function are based on Gaussian functions,

9. The method of claim 1,

wherein in the determining of the tone -mapped value for a first pixel of the image, M is equal to N, and

wherein in the determining of the tone-mapped value for a second pixel of the image, M is less than N.

10. The method of claim 1 ,

wherein in the determining of the tone-mapped value for a first pixel of the image, M is equal to a first value, the first pixel being in a zone that is located at a boundary of the image, wherein in the determining of the tone-mapped value for a second pixel of the image, M is equal to a second value, the second pixel being in a zone that is not located at the boundary of the image, and wherein the second value is not equal to the first value.

11. An imaging device comprising:

a pixel array; and

a processor for processing an image received from the pixel array, the processor being configured to:

divide the image into N zones and generate a histogram for each of the N zones; determine a tone curve for each of the N zones, wherein each of the tone curves is determined based on the histogram for the zone; and

determine a tone-mapped value for each pixel of the image based on a sum of

M weighted values determined by applying tone curves of M zones to a value of the pixel.

12. The imaging device of claim 11, wherein the processor is configured to determine the M weighted values, the determining of the M weighted values comprising:

applying to the val ue of the pixel a tone curve for a zone to determine a non-weighted v alue; weighting the non-weighted value based on i) a distance from the pixel to a center point of the zone, and ii) an intensity difference between the value of the pixel and an average pixel value of the zone; and

performing the applying and weighting steps for each of the M zones to determine the M weighted values for the pixel, wherein the M zones mclude a first zone in which the pixel is located and one or more other zones within a neighborhood of the first zone.

13. The imaging device of claim 1 1 , wherein the determining of the tone curve for each of the N zones includes:

based on the determination that the zone is over-exposed, under-exposed, or of normai exposure, setting parameters of the tone curve for the zone.

14. The imaging device of claim 13, wherein the determining of whether the zone is overexposed, under-exposed, or of normal exposure includes:

detennining, based on the histogram for the zone, a nonnalized average luma value YN_Wj for the zone, YN_avg being equal to an average luma value of the zone normalized by a maximum luma value of the image;

15. The imaging device of claim 14, wherein the tone curve for the zone is based on

where x is the value of the pixel to which the tone curve is applied, y is an output value of the tone curve, and a and γ are the parameters of the tone curve set based on the determination that the zone is over-exposed, under-exposed, or of normal exposure.

16. The imaging device of claim 15, wherein the processor is further configured to:

if the zone is of normal exposure:

determine a value DR from the histogram, wherein DR is a difference between H_low and H_hi_g , wherein H_low is a ^-percentile luma value that is less than H_half, and H_hj_gh is a (100 - q)- percentile luma value, and wherein the determining of whether the zone has the wide dynamic range, the narrow dynamic range, or the normal dynamic range is based on the value DR; determine that the zone has the wide dynamic range and setting γ equal to γ,_η;„ if DR is greater than DR_raa>;, DR_ma>; being a maximum threshold value for DR and _{m n} being a minimum value for y;

determine that the zone has the narro w dynamic range and setting γ equal to y_max if DR is less than DR,„j_n, DR_m;_n being a minimum threshold value for DR and y_max being a maximum value for γ; and

determine that the zone has a normal dynamic range and setting γ equal to

(DR-DK_{m ax})*(y_max- y_min )

(DR_max-DR_mln) Yrnirn if the zone is not determined to have the wi de dynamic range and not determined to have the narrow dynamic range,

17. The imaging device of claim 11, wherein the processor is configured to:

determine, for each of the N zones, if the zone includes a face of a human being; and modify the tone curve for each of the N zones determined to include the face,

18. The imaging device of claim 1 1 , wherein the determining of the tone -mapped value for each pixel is based on

where y is the tone-mapped value, x is the value of the pixel, T_n is a tone curve for a zone n of M zones, wl„ is a distance weighting function based on a distance from the pixel to a center point of the zone n, w2_n is a similarity weighting function based on an intensity difference between the value of the pixel and an average pixel value of the zone n, wherein the distance weighting function and the similarity weighting function are based on Gaussian functions.

19. The imaging device of claim 11,

wherein in the determining of the tone-mapped value for a first pixel of the image, M is equal to N, and

20. The imaging device of claim 11 ,

wherein in the determining of the tone-mapped value for a first pixel of the image, M is equal to a first value, the first pixel being in a zone that is located at a boundary of the image, wherein in the determining of the tone-mapped value for a second pixel of the image, M is equal to a second value, the second pixel being in a zone that is not located at the boundary of the image, wherein the second value is not equal to the first value.