EP2372686A1 - A method for local dimming boost using salient features - Google Patents

Abstract

The local backlight intensity in the LED backlight displays is commonly estimated based on the intensity levels of the image. Present invention offers a post processing method in which estimation is improved using salient features of the image. The method mainly boosts up backlight in low intensity regions where a salient area occurs. This post processing improves image quality and perception by means of preventing local contrast loss due to high level of compensation where salient regions are present. As a result, high contrast and power efficiency can be achieved with an improved human perception of the image.

Description

Technical Field
• The present invention offers a method for local dimming in direct-lit and edge lit panels using salient features for required boost up in LED driving levels.
Prior Art
• LED (Light Emitting Diode) backlight displays have recently become popular in the market due to their performance and energy saving characteristics. Moreover, the advantages of LED backlight can be used for improving the image quality and motion picture dynamics of the displays. Several methods have been developed in order to combine conventional LCD (Liquid Crystal Display) characteristics with LED backlight technology for image enhancement.
• The perceived brightness of a pixel on the LCD depends on the cumulative effect of backlight intensity, filter transparency and LCD level. For an LCD with constant backlight intensity, the range of brightness level depends only on LCD setting. However, when a controllable backlight is used, the brightness can be increased by boosting the backlight and similarly darker regions can be achieved by backlight dimming. In this manner, if a locally controllable backlight is used with a conventional LCD, the contrast ratio (the ratio of the brightness of the brightest pixel to that of darkest pixel) can be boosted up to much higher levels. The control algorithm and methodology of backlight intensity are critical issues at this point.
• Several control methods and systems are proposed to control light intensity of LED backlight. The patent document CN 201315147 proposes a method in which maximal gray value and density of this value is used to obtain the backlight driving value. The method enables use of backlight dynamics. However the proposed method is insufficient as it aims to process the whole display as a single unit and does not provide solution for locally controllable LED backlights.
• The utility model document CN 201315148 offers a similar method to the utility model document CN 201315147 . However, CN 201315148 offers a solution for locally controllable LED backlights in which, maximum of local gray value and its density is used to control the LED backlight. This method is better than CN 201315147 in terms of locality issues but still the information used for processing is inadequate.
• The method in CN 101354875 proposes to utilize brightness statistics of different zones of the picture and to adjust corresponding local backlight intensity accordingly. The document also proposes to process the LCD control data in order to smooth the transition regions between backlight locals. The method is useful in improving image quality, yet it's insufficient as brightness statistic is inadequate for a full perception of the image.
• Several literature works have also been published for control methods used for LED backlights. The methods depend on the histogram of the related zone to adjust light intensity of the region. The histogram based methods give better results however; they are still inadequate in terms of dark region perception.
Brief Description of the Invention
• Common methods used for the control of intensity of the LED backlight displays employ intensity or histogram of the image in which human visual system is not fully regarded. The invention proposes a method where a post processing is performed in order to compensate dimming of LED backlight based on intensity based method using salient features. The method first calculates LED driving values according to regional intensities. Meanwhile a saliency map is extracted. Then, histogram analysis of saliency map is realized and intensity based LED driving values are updated according to saliency map histogram. With the utilization of salient features, possible loss of detail in regions of the image, where intensity is low but the region is salient, are eliminated. In this manner, histogram based backlight adjustment errors are compensated and an image closer to the original is reflected, which is also more adequate for human visual perception.
Object of the Invention
• The object of the invention is to compensate possible loss of detail originating from intensity based LED backlight adjustment techniques, using salient features of the image.
• Another object of the invention is to improve overall contrast of the image.
• Another object of the invention is to decrease power consumption while delivering a more perceptible image.
Brief Description of the Drawings

Figure 1;

shows the saliency map of a synthetically generated image.

Figure 2;

shows how the means of the small blocks are calculated hierarchically for computational efficiency.

Figure 3;

shows how the mean values for the pixels that area not located at the center of the blocks are calculated by bilinear interpolation for computational efficiency.

Figure 4;

shows the block diagram of algorithmic flow.

Figure 5;

shows the histogram of the saliency map and effect of weighting on histogram.

Figure 6;

shows the linear weighting function of the histogram.

Figure 7;

shows how the amount of LED driving value boost is limited according to a parametric function depending on the block saliency level.

• The reference numbers as used in figures may possess the following meanings.

  The background region of the image which has a gray level of 120	(1)
  The region of the image which has a gray level of 200	(2)
  The region of the image which has a gray level of 50	(3)
  The region of the image which has a gray level of 100	(4)
  The background region of saliency map which has a gray level of 10	(5)
  The region of saliency map which has a gray level of 200	(6)
  The region of saliency map which has a gray level of 190	(7)
  The region of saliency map which has a gray level of 20	(8)
  The center points of 4 neighboring regions of LCD	(9, 10, 11, 12)
  Saliency calculation point	(13)

Detailed Description of the Invention
• The invention proposes a better control method for LED backlight displays in which intensity based and saliency based backlight level adjustment is performed, improving image details, contrast ratio and power efficiency.
• Saliency concept is related with the response of the human visual system and defined by a map as response to original image. The saliency map is based on the deviation of a pixel from its surrounding pixels. If the intensity of a specific pixel is brighter than its surrounding pixels, the saliency map value of the said specific pixel is positive. The human visual perception directly depends on difference levels. A region deviating from surroundings in terms of light intensity takes visual attention regardless of local average light intensity.
• Saliency is widely used in computer vision area and has applications such as coding, watermarking, feature matching, video abstraction and summarization. Various methods are proposed to extract salient features of an image or a video. Image enhancement based on saliency is used in computer graphics and biomedical engineering. In computer graphics area, computational complexity is decreased by processing the graphic such that, salient regions are processed in detail whereas regions with less saliency are processed with less detail. Biomedical engineering uses salient features to process an image to attract user attention to specific regions of the image.
• An example of a frame and its corresponding saliency map is given in Figure 1. A synthetic view with 4 different 8 bit gray levels with a background level of 120 (1) and 3 regions with (2, 3 and 4) gray levels of 200, 50 and 100 respectively are given. The human visual system responds more to dissimilarity; hence a region deviating from the general background will be more attractive. In other words, attention is given to the objects diverging from the mean intensity level. On the right side of the Figure 1, the saliency map is given, where bright regions (6, 7) indicate high saliency; and corresponding area to the background (5) and region having gray level of 100 (8) are dark, that is saliency is low.
• In most saliency extraction methods, Lab color space, which is a color space especially designed to approximate human vision, is widely utilized due to its uniformity in perceptual domain. In this work, since the fundamental aim is to determine LED backlight intensity based on the local properties of the pixels with respect to intensity, the saliency extraction is realized in RGB color space. In that manner, for each pixel, differences with the mean value of certain window sizes are calculated for three channels (R, G and B) independently. At that point, five different window sizes are utilized in order to determine the center-surround differences and how much the pixel is different from the surround. The window sizes are chosen to be according to equation (14). As can be seen in said equation (14), the window sizes are considerably large; the reason behind this choice is that small window sizes do not give enough information about the surrounding and provide edge detection instead of attraction detection which results in noisier feature maps. $$\frac{\mathit{frame\; width}}{16}\le \mathit{window\; width}\le \frac{\mathit{<figure-callout\; id="1"\; label="frame\; width"\; filenames="imgf0001.png,imgf0003.png"\; state="\{\{state\}\}">frame\; width</figure-callout>}}{1}$$

  
• Once the mean values within the specified windows of changing size are determined, the saliency of a pixel (C_k) is calculated as in the equation (15); where k corresponds to the index of the utilized window size, I_RGB is the RGB image; w and h are the window width and height respectively. $$C_k\left(xy\right)=I_{\mathit{RGB}}\left(xy\right)-\frac{1}{w*h_i}{\displaystyle \sum _{i=-w/2}^{w/2}}{\displaystyle \sum _{j=-h/2}^{h/2}}{I}_{\mathit{RGB}}\left(x+i,y+j\right)$$

  
• Large window sizes are utilized during the estimation of the saliency map, thus calculating the mean values of the window for each pixel independently requires huge processing. However, calculating mean for specific windows and performing bilinear interpolation drastically decreases the computation complexity. To achieve computationally simple processing, the input frame is divided into blocks determined by the smallest window ((frame width)/16 through the invention), and the calculated mean values of the blocks are assigned to the center pixels of these blocks as given in the following Figure 2 as dotted points. As the blocks get larger, the mean values are determined by mean operation through the smallest blocks in the corresponding large block. Once the mean values are determined for block centers of different sizes, bilinear interpolation is utilized to determine the mean values for each pixel (13) as illustrated in Figure 3. For the crossed pixel (13) which is between the block centers of (9, 10, 11 and 12) the mean value for a window size of ((frame_width)/4, (frame_height)/4) is calculated by interpolation of 4 neighboring block means determined bilinearly by the window size of ((frame_width)/4, (frame_height)/4). This operation is iterated for each window size defined in the equation (14), and the closest block centers to the corresponding pixel are utilized at each resolution. The calculated mean values for 5 different resolutions are subtracted from the corresponding pixel intensity value. Finally, these differences (C_i) are added to each other to determine the saliency of the pixel as in the equation (16). $$\mathit{SaliencyMap}\left(xy\right)={\displaystyle \sum _{i=1}^5}{C}_i\left(xy\right)$$

  
• In that way, the measure of how the pixels are different from their surroundings is obtained; in addition the darkness or brightness is also regarded such that, for darker pixels, saliency values are negative and for the brighter pixels, saliency is positive. This is an important fact and will be crucial in the refinement stage. Finally the salient values are mapped to a range of between (0, 255) by scaling the minimum and maximum values for convenience.
• The general flow of the method is shown in Figure 4. Original image is taken as input and LED driving values are estimated through intensity based LED backlight algorithm. Simultaneously saliency levels are computed using saliency based algorithm. Then both outputs are used to decide final LED driving value. Saliency Map Extraction block computes the saliency map of the input image as explained previously. Saliency map is displayed as an intensity map where 0-255 levels show the amount of salient information contained in the image. 0 level represents the less salient region whereas 255 level corresponds to the most salient region. The histogram of the saliency map corresponding to the LED backlight blocks are computed in the next step. The number of individually controllable LED regions varies due to the manufacturer. The histogram of the saliency map for each region is calculated and weighted according to a parametric weighting formula (17). An exemplary block histogram of saliency map for a LED region is shown in Figure 5. A typical weighting function is given in figure 6. A linear mapping is used for convenience, other mapping functions may be used depending on the settings. The weighted histogram shown in Figure 5 is used to decide on saliency level for each block. $$\mathit{BlockSaliency}=\frac{\sum \mathit{weight}\left[\mathrm{i}\right]\times \mathrm{histogram}\left[\mathrm{i}\right]}{\mathit{Pixelsize}}$$

  
• The final block of Boost Region Classification decides on the regions where LED driving values will be increased. The blocks that are previously assigned below a threshold (by the intensity based method) should not be boosted. Said blocks are important parts of the image where the intensity of the regions is really low. Said blocks supply mainly the contrast improvement. Increasing the intensity levels at these regions will bring no benefit since the proposed idea targets the medium intensity level regions where salient features are present. At this stage we have two sets of useful information. One is the estimated LED driving levels; other one is the weighted saliency levels for each region. The amount of boost is calculated in this block as follows. $$\mathit{LedBlockValueUpdate}=\left(k\times \mathit{BlockSaliency}\right)+\left(\left(1-k\right)\times \mathit{LedBlockValue}\right)$$

  
• As can be seen in the equation (18), Final Led Driving Level (LedBlockValueUpdate) is sum of product of parameter k and block saliency (BlockSaliency) and product of 1-k and intensity based calculated LED block Value (LedBlockValue).
• The LED driving value estimated by the intensity based method is simply weighted with the estimated saliency level of the block. The parameter "k" is used as the ratio of saliency level, 0.9 is chosen for an effective boost-up. It is important to note that if the saliency level of the block is less than the estimated LED driving level, the equation (18) will cause a decrease in the final LED driving level. Saliency is meant to be used only for boosting, hence a decrease due to saliency is no means allowed. The final step in LED driving value decision is to check the limits of boost in order to prevent high temporal and spatial changes. Huge changes in intensity may cause lack of uniformity in time and space; hence a limiting idea has been utilized. The limits of boost in LED driving levels are shown in Figure 7. Limiting the boost level is realized depending on the value of block saliency. The higher the estimated block saliency, the higher the limit is. The limiting curve for two cases where block saliency is 255 (19) and 128 (20) are illustrated. The low threshold is the level below which no boost shall be performed.

Claims (14)

1. A method for local dimming boost in LED backlight displays using salient features, characterized in that it comprises the steps of;

   - Calculating LED driving values according to regional intensities of an image,

   - Dividing said image into windows with certain sizes and calculating mean intensity values of said windows,

   - Extracting a saliency map of said image using said mean intensity values of said windows,

   - Calculating histogram of said saliency map for each locally controllable LED region,

   - Updating intensity based LED driving values according to said calculated saliency map histogram.
2. A method according to Claim 1 wherein said saliency extraction is realized in RGB color space.
3. A method according to Claim 1 wherein saliencies of pixels of the said image are calculated for three channels (R, G and B) independently.
4. A method according to Claim 1 wherein bilinear interpolation is performed for calculating mean for said windows.
5. A method according to Claim 1 wherein said saliency map of said image are mapped to a range between (0, 255) by scaling the minimum and maximum values of said saliency map.
6. A method according to Claim 1 wherein said saliency map is realized as an intensity map.
7. A method according to Claim 1 wherein a weighted saliency is calculated by weighting histogram of saliency map for each locally controllable LED region according to a parametric weighting formula.
8. A method according to Claim 7 wherein said histogram of saliency map weighting for each locally controllable LED region is performed according to formula; $$\mathit{BlockSaliency}=\frac{\sum \mathit{weight}\left[\mathrm{i}\right]\times \mathrm{histogram}\left[\mathrm{i}\right]}{\mathit{Pixelsize}}$$

   
9. A method according to Claim 1 wherein boost in the driving value of said locally controllable LED region is inhibited, if corresponding said mean intensity value of said window is below a defined threshold.
10. A method according to Claim 1 and 7 wherein said intensity based calculated LED driving values are weighted with said weighted saliency of said window using a simple weighting formula.
11. A method according to Claim 10 wherein said weighting is performed with the formula; $$\mathit{LedBlockValueUpdate}=\left(k\times \mathit{BlockSaliency}\right)+\left(\left(1-k\right)\times \mathit{LedBlockValue}\right)$$

    
12. A method according to Claim 10 wherein result of the said weighting is checked whether a decrease in said LED driving values occurs or not in order to eliminate occurrence of said decrease.
13. A method according to Claim 1 wherein boost in said LED driving values are controlled ensuring that said LED driving values are below a defined limit.
14. A method according to Claim 13 wherein said limit is defined for said boost in LED driving values depending on the value of histogram of saliency map of said corresponding window.

Images