JP5114494B2 - Contrast enhancement of video data with sharpness - Google Patents

Description

  Embodiments of the claimed invention generally relate to image or video information enhancement schemes, and more particularly to image or video information contrast enhancement schemes.

  An image in a video sequence can be represented, for example, by three signals (eg, red / green / blue (RGB) or luminance and chrominance (YUV)) sampled in both the horizontal and vertical directions. In those images in YUV color space, contrast can also be conceptualized as a feature that indicates the energy distribution between different luminance (Y) values in the image. In general, contrast enhancement increases the dynamic range of an image, while the energy in the image can be redistributed across the luminance value range (eg, the highest and lowest values).

  The sharpness of an image is a perceptual impression caused by a relatively high energy concentration in the high frequency components of the image. Some contrast enhancements enhance perceived sharpness by enhancing image details.

  However, if the contrast is enhanced too much, the details of the image disappear and the sharpness can be lost accordingly. This is because energy may be redistributed to the luminance value of the image without being redistributed to the high frequency components of the image.

  The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate, together with the description, one or more embodiments consistent with the principles of the invention. The drawings are not necessarily to scale, and may be emphasized to illustrate the principles of the invention.

1 shows an example of a contrast enhancement system.

An example of a contrast enhancement transfer function is shown.

An example of the image before and behind frequency extraction is shown. An example of the image before and behind frequency extraction is shown.

2 shows an example of an image contrast enhancement process.

  In the following detailed description, references are made to the accompanying drawings that illustrate specific embodiments in which the invention may be practiced. Where the same reference number is used in different drawings, it indicates the same or similar component. In the following description, for purposes of explanation and not limitation, specific details are set forth as specific structures, designs, interfaces, techniques, etc., in order to provide a thorough understanding of various aspects of the claimed invention. Is done. However, one of ordinary skill in the art having the benefit of this disclosure will appreciate that the various aspects of the claimed invention can be practiced in other examples that depart from these detailed descriptions. In certain instances, descriptions of well-known devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

  FIG. 1 shows an example of a contrast enhancement system 100. System 100 receives video information from any suitable medium including, but not limited to, various transmission and / or storage media. Although illustrated as separate functional components for ease of explanation, any or all of the components of system 100 may be co-located and / or a common group of gates and / or transistors. May be implemented. Further, system 100 may be implemented by software, firmware, hardware, or any suitable combination thereof.

  The contrast enhancement system 100 includes a first memory 110, a contrast enhancer 120, a second memory 130, a frequency extractor 140, a third memory 150, a fourth memory 160, and comparison logic 170. Although shown as individual functional components in the figure, certain of the components 110-170 may be implemented in common. For example, in some embodiments, the memories 110, 130, 150, 160 may be implemented as part of a single memory with or without reuse. Also, in some embodiments, contrast enhancer 120, frequency extractor 140, and / or comparison logic 170 may be implemented by a general purpose or special purpose processor that executes software and / or firmware. Other specific implementations for components 110-170 are possible and are contemplated.

  The first memory 110 receives and stores reference image data in various formats. The format of the data is MPEG-1, MPEG-2, MPEG-4, Advanced Video Coding (AVC) (eg, MPEG-4, part 10 and ITU-T Recommendation, H.264), Windows (registered trademark) Media Video 9 (WMV-9), SMPTE VC-1, and the like. In some implementations, the first memory 110 may store luminance values (such as Y) of the entire reference image. However, in some implementations, the first memory 110 may store a portion of the entire image, or the brightness values of two or more images. In some implementations, the first memory 110 may also include a random access memory (RAM) that can overwrite individual pixels in the stored reference image as needed.

  The contrast enhancer 120 may enhance the contrast of the reference image in the first memory 110 and output an enhanced image having the enhanced contrast to the second memory 130. Contrast enhancer 120 may enhance the contrast of the reference image in first memory 110 using any of a number of techniques now known or later developed. In some implementations, for example, the contrast enhancer 120 may determine an image characteristic of the reference image (eg, a luminance histogram of the reference image). Then, the contrast enhancer 120 may generate an enhanced image by applying an arbitrary contrast enhancement transfer function (which may or may not be based on the characteristics of the reference image) to the luminance value of the reference image.

  FIG. 2 is a plot 200 that includes an example of a contrast enhancement transfer function 230 that the contrast enhancer 120 can use. For reference purposes, the transfer function 230 is shown in relation to a single gain line 210 and a continuous contrast enhancement transfer curve 220 close to the function 230. Of course, a continuous curve 220 can be used in some embodiments, but in other embodiments, a piecewise linear curve such as the four-point transfer function 230 can be easily implemented.

  When attention is paid to the transfer function 230, different properties are shown for different luminance input values in the dark region, the intermediate region, and the bright region as illustrated. The boundaries of these regions may be set by the contrast enhancer 120 based on the luminance characteristics of the reference image. For example, in some embodiments, the boundary between the “dark” and “bright” portions of function 230 is set at 5% of full-scale luminance to prevent coring from occurring in the output due to noise. However, other thresholds are possible and are considered. As another example, the boundary between the “intermediate” and “bright” portions of function 230 may be set at 75% of full scale luminance, although other thresholds are possible and are considered.

  In the intermediate region, the function 230 may change the slope by, for example, the median luminance value or the average value of the reference image. Within the bright region, the transfer function 230 may change the slope with an average of the luminance value (eg, 75% of full scale) and the full scale (eg, maximum possible) value at the beginning of the bright region. In the specific example above, the inflection point or break of the function 230 in the bright region can occur at about 87.5% of full scale. In both the intermediate and bright areas, the function 230 may have slopes for different line segments that are slightly larger and / or smaller than each other. In some embodiments, for example, at least some segments of the transfer function 230 may be (1 ± δ) for some relatively small value δ.

  In some embodiments, the contrast enhancer 120 may use the transfer function 230 to enhance the contrast of the reference image in the first memory 110 and output the resulting enhanced image to the second memory 130. Contrast may be enhanced by “stretching” the luminance distribution of the image (eg, making bright pixels brighter and dark pixels darker) by transfer function 230. However, the contrast enhancer 120 may apply any contrast enhancement transfer function to the reference image to generate an enhanced image, and the claimed invention is given by way of example only, as shown in FIG. Note that the particular transfer function 230 is not limited.

  Returning to FIG. 1, the second memory 130 receives the enhanced image (such as luminance data) from the image enhancer 120 and stores it. After the enhancer 120 transfer function is applied, the enhanced image in the second memory 130 may match in size and other features with the reference image in the first memory 110. In some embodiments, the second memory 130 is a random access memory (RAM) or similarly addressable memory that allows individual pixels in the enhanced image in the second memory 130 to be read out as needed. May be included.

  The frequency extractor 140 may extract at least some high frequency components from the reference image in the first memory 110 and store the extracted reference frequency components in the third memory 150. The frequency extractor 140 may extract at least some high frequency components from the enhanced image in the second memory 120 and store the extracted enhanced frequency components in the fourth memory 160. In some implementations, the frequency extractor 140 may perform operations on the reference image and the enhanced image substantially simultaneously. However, in some implementations, the frequency extractor 140 first performs an operation on the reference image, during which time, for example, the contrast enhancer 120 may generate an enhanced image.

  In some implementations, the frequency extractor 140 may perform a two-dimensional (2D) transformation on the reference image and the enhanced image. Various transforms that can be implemented by the frequency extractor 140 to extract high frequency components can generate Fourier components, discrete cosine transforms (DCT), Hadamard transforms, wavelet transforms, and / or high frequency components from images. Any one-dimensional or two-dimensional transformation may be included, but is not limited to these. In some embodiments, the frequency extractor 140 generates a reference frequency component in the third memory 150 by performing a two-dimensional discrete wavelet transform (DWT) on the reference image, and also generates a two-dimensional discrete An enhanced frequency component may be generated in the fourth memory 160 by performing wavelet transform (DWT) on the enhanced image.

  FIG. 3A shows a typical image 310 before frequency extraction by the frequency extractor 140. For illustrative purposes, the image 310 may be a reference image or an enhanced image, and may have, for example, a luminance (Y) value.

  FIG. 3B shows typical components of image 310 after two-dimensional discrete wavelet transform (DWT) by extractor 140. The two-dimensional DWT divides the image 310 in the horizontal direction (ie, the horizontal component 330), the vertical direction (ie, the vertical component 340), and the diagonal direction (ie, the diagonal component 350), with one low-definition component 320 and three high-definition. It may be broken down into components (eg, containing high frequency information). For those skilled in the art of DWT, FIG. 3B shows a first level of decomposition, and the low-detail component 320 can be continuously decomposed in a second, third, etc. stepwise decomposition. You will understand. The claimed invention is implemented using components from a first level wavelet decomposition, but is not limited thereto, and may be implemented using higher level wavelet decomposition, and / or Alternatively, for example, high frequency components 330 to 350 or similar high frequency information may be extracted by other types of conversion.

  Referring to the exemplary components of FIG. 3B, the frequency extractor 140 stores a set of components 320 to 350 (eg, reference components) generated from the reference image in the first memory 110 in the third memory 150. It's okay. Similarly, the frequency extractor 140 may store another set of components 320 to 350 (for example, enhanced components) generated from the enhanced image in the second memory 130 in the fourth memory 160. Depending on the characteristics of the particular two-dimensional transformation, the number of horizontal, vertical and diagonal components 320 to 350 may coincide (or be correlated in a known manner) with the number of pixels in the reference image or enhanced image.

  Returning to FIG. 1, the third memory 150 may receive and store the reference component (for example, including the high-frequency component of the reference image) from the frequency extractor 140. In some implementations, the third memory 150 includes random access RAM (random access) such that individual components in the third memory 150 (eg, corresponding to pixels in the corresponding reference image) can be read as needed. Memory) or similar addressable memory.

  The four memories 160 may receive and store the enhanced components (for example, including the high frequency components of the enhanced image) from the frequency extractor 140. In some implementations, the fourth memory 160 is a random access RAM (random access) such that individual components in the fourth memory 160 (eg, corresponding to pixels in the corresponding enhanced image) can be read out as needed. Memory) or similar addressable memory.

  Based on the comparison of the corresponding reference component and enhancement component in the third and fourth memories 150 and 160, the comparison logic 170 corresponds the pixel of the reference image in the first memory 110 to the corresponding enhancement image in the third memory 130. It may be selectively replaced with a pixel. In general, the comparison logic 170 may replace the reference pixel with the enhancement pixel when the energy of the high frequency enhancement component is greater than the energy of the high frequency reference component. When the energy of the enhancement component is not greater for a given pixel, the comparison logic 170 may not replace the reference image pixel in the first memory 110.

ここで、「ｅｎｈ」および「ｒｅｆ」は、高周波強調および基準成分を示し、さまざまなｆａｃｔｏｒは、重み係数または尺度係数であり、１つの成分（例えば水平成分）を他より多く、少なく、または、同じに重み付けするのに用いられる。そして、比較ロジック１７０は、この重み付けされた合計をゼロと比較して絶対エネルギーの増加を識別するか、または、ゼロでない数と比較して特定の量より多いエネルギーの増加を識別してよい。 Comparison logic 170 may use one, some, or all of the reference and enhancement components in memories 150 and 160 to make a decision to increase or decrease high frequency energy for a given image pixel. Referring to the particular example in FIG. 3B, the comparison logic may use one, some, or all of the horizontal component 330, vertical component 340, and diagonal component 350 for comparison. For example, the comparison logic 170 may determine the following sum:

Here, “enh” and “ref” indicate high frequency enhancement and reference components, and various factors are weighting factors or scale factors, and one component (eg, horizontal component) is more or less than the other, or Used to weight the same. The comparison logic 170 may then compare this weighted sum with zero to identify an absolute energy increase, or may compare with a non-zero number to identify an energy increase greater than a particular amount.

  In some embodiments, the comparison logic 170 may compare only one or two of the frequency components (such as 330 to 350) with or without a weighting factor. For example, the comparison logic may compare the horizontal component 330 only, the vertical component 340 only, the diagonal component 350 only, or a combination thereof. Further, the comparison logic 170 may cause the first memory 110 to output the reference image after the end condition is satisfied. An example of the termination condition may be when the entire reference image is moved by a predetermined number of pixels (for example, 1, 2, 3, 4, etc.), or for an enhanced image including continuous contrast enhancement by the contrast enhancer 120 It may be when the energy in the high frequency component starts to decrease.

  FIG. 4 shows an exemplary process 400 for enhancing the contrast of the reference image. 4 may be described with respect to the contrast enhancement system 100 of FIG. 1 for ease of explanation, it should be understood that the process 400 may also be performed by other hardware and / or software implementations. .

  The process begins by enhancing the contrast of the reference image with the contrast enhancer 120 to generate an enhanced image [operation 410]. Any contrast enhancement transfer function such as function 230 of FIG. 2 may be used to generate the enhanced image. In some embodiments, operation 410 may be performed on an image having a luminance value. Simultaneously with operation 410, the reference image and the enhanced image may be stored in the memories 110 and 130.

  Processing continues with the frequency extractor 140, where the reference component and the enhanced component are generated by calculating the high frequency components of the reference image and the enhanced image [operation 420]. Either one-dimensional or two-dimensional transformations may be used to calculate the high-frequency component, but in some embodiments, a first level two-dimensional DWT is used to level the reference image and the enhanced image, Vertical and / or diagonal high frequency components may be determined. Concurrently with operation 420, these reference and enhancement components may be stored in memories 150 and 160.

  Comparison logic 170 may compare the reference and enhanced high frequency components pixel by pixel and, if the energy in the enhanced high frequency component is sufficiently large, may replace a pixel in the reference image with a corresponding pixel in the enhanced image [operation 430]. The enhanced image's horizontal, vertical, and / or diagonal high frequency components and the reference image's horizontal, vertical, and / or diagonal, with or without weighting and / or scale factors that weight one or more different components Any combination with any of the high frequency components may be compared.

  In operation 430, the comparison logic 170 determines that the problem in the reference image is only when the energy of the high frequency component of the enhanced image is greater than the energy of the high frequency component of the reference image (absolutely or only by a predetermined difference). May be replaced with corresponding pixels of the enhanced image. The comparison logic 170 may perform such comparison and selective replacement operations for all pixels of the reference image in the first memory 110.

  If the end condition is not reached after performing operation 430 on all pixels (or a set of subsampled pixels) in the reference image [operation 440], the modified reference image (some pixels are replaced) May be highlighted again at operation 410 and operations 420 and 430 may be repeated. Several termination conditions are also possible and considered. In some implementations, operations 410 through 430 may be repeated until there are no pixels in the enhanced image that have a higher frequency energy component than the higher frequency energy component in the reference image. In some implementations, the termination condition is that the number of pixel replacements in the reference image is sufficiently small (eg, 5, 10, 50, 100, etc.). That is, operations 410 through 430 may be repeated until fewer pixels are replaced in operation 430.

  In some implementations, the termination condition may be to move the reference image a predetermined number of times (eg, after operation 430 has been performed once, twice, five times, ten times, etc.). . However, the reference image is moved a predetermined number of times in the initial stage, then an emphasized image is generated (repeated five times), and a predetermined number of pixels are replaced in the reference image (for example, 50 or less pixels) Other examples such as (substitution) are also possible.

  When the comparison logic 170 determines that the end condition has been reached [operation 440], the comparison logic 170 may instruct the first memory 110 to output a reference image [operation 450]. After selective pixel replacement, the reference image may be enhanced in contrast while retaining high frequency energy (eg, sharpness is perceived). Thus, by paying attention to the frequency components of the reference image before and after contrast enhancement, it is possible to prevent the details of the image from disappearing due to contrast enhancement. With such frequency analysis, it is possible to avoid losing details of an image (or a video of which part is an image) while enhancing the contrast of the image.

  The illustrations and descriptions of the one or more embodiments are not intended to be exhaustive or to limit the scope of the invention to the precise forms disclosed. Modifications and changes in light of the above teachings are possible, or modifications and changes may be made by implementing various embodiments of the invention.

  For example, the above scheme includes pixel-by-pixel comparison and selective replacement, and in some embodiments, such comparison / replacement based on high frequency components may be performed on alternating pixels, groups of pixels, etc. in the image. May be executed. The schemes described herein and in the claims can include any contrast enhancement technique using any type of frequency analysis that preserves sharpness of details while enhancing contrast. Further, “high frequency” as used herein may include any frequency or frequency range that corresponds to “fine” spatial details throughout the context of the image of interest.

  Furthermore, the operations in FIG. 4 need not be performed in the order shown in the figure, and not all operations need to be performed. Also, these operations are independent of each other and may be performed simultaneously with other operations. Further, at least some of the operations in FIG. 4 may be performed as instructions or groups of instructions implemented on a machine-readable medium.

  Unless otherwise stated explicitly, any component, operation, or instruction used in the description of the present application is not essential or construed as essential to the invention. Also, the article “a” as used herein is intended to include one or more items. Changes and modifications may be made to the above-described embodiments of the claimed invention without departing substantially from the spirit and principles of the invention. Such modifications and variations are intended to be within the scope of this disclosure and protected by the following claims.

Claims (20)

Increasing the brightness contrast of the reference image to generate an enhanced image;
Comparing at least one frequency component of the reference image with at least one frequency component of the enhanced image;
Selectively replacing a portion of the reference image with a corresponding portion of the enhanced image based on the comparing step;
Equipped with a,
Said comparing comprises determining whether said at least one frequency component of said enhanced image has an energy greater than said at least one frequency component of said reference image;
The step of replacing comprises replacing a portion of the reference image with a corresponding portion of the enhancement image when the at least one frequency component of the enhancement image has greater energy than the at least one frequency component of the reference image. Replace with
Method. The method of claim 1, further comprising storing the reference image and the enhanced image in a memory. Generating the at least one frequency component of the reference image by performing a transformation on the reference image;
Generating the at least one frequency component of the enhanced image by performing the transformation on the enhanced image;
Further comprising the method of claim 1 or 2. The transform is a two-dimensional wavelet transform;
The method of claim 3.   Generating the enhanced image, comparing, and replacing are repeated two or more times;
The method according to any one of claims 1 to 4.   Generating the enhanced image, comparing, and replacing are repeated until the number of replaced pixels in the reference image is less than a predetermined number;
The method of claim 5. A contrast enhancer that applies a luminance contrast enhancement transfer function to the first image to generate a second image;
A frequency extractor for extracting a first frequency component from the first image and extracting a second frequency component from the second image;
Comparison logic that compares the first frequency component with the second frequency component and selectively replaces pixels of the first image with pixels of the second image based on the result of the comparison; ,
Equipped with a,
The comparison logic replaces pixels of the first image with pixels of the second image when the second frequency component has greater energy than the first frequency component;
system. A first memory coupled to the contrast enhancer for storing the first image;
A second memory coupled to the contrast enhancer and storing the second image;
The system of claim 7 , further comprising: A third memory coupled to the frequency extractor for storing the first frequency component;
A fourth memory coupled to the frequency extractor for storing the second frequency component;
The system according to claim 7 or 8 , further comprising: The frequency extractor performs a Fourier transform, a cosine transform, or a wavelet transform on the first image and the second image;
The system according to any one of claims 7 to 9 . The comparison logic determines whether the sum of the second frequency components has an energy greater than the sum of the first frequency components;
The system according to any one of claims 7 to 10 . Applying a luminance contrast enhancement transfer function to a reference image to generate an enhanced image;
Performing a conversion on the reference image to generate a reference high-frequency component, and executing the conversion on the enhanced image to generate an enhanced high-frequency component;
For a pixel in the reference image, determining whether an enhanced high frequency component has a higher energy than a corresponding reference high frequency component;
If the enhanced high frequency component has a higher energy than the corresponding reference high frequency component , replacing the pixel in the reference image with a corresponding pixel in the enhanced image;
A method comprising: The contrast enhancement transfer function is a piecewise linear function;
The method of claim 12 . The transform includes a wavelet transform,
14. A method according to claim 12 or 13 . The determining includes scaling the enhanced high-frequency component or the reference high-frequency component;
15. A method according to any one of claims 12 to 14 . If the enhanced high frequency component does not have higher energy than the corresponding reference high frequency component, further comprising not replacing the pixel in the reference image;
The method according to any one of claims 12 to 15 . Generating a second image with enhanced brightness contrast from the first image;
Analyzing the frequency components of the first image and the second image and increasing the contrast of the first image while retaining details of the first image;
Equipped with a,
The step of increasing the contrast corresponds to the pixel in the first image corresponding to the pixel in the second image when the frequency component of the second image has a larger energy than the frequency component of the first image. Replace with the pixel to
Method. Further comprising repeating the generating and analyzing steps until an end condition is reached;
The method of claim 17 . Step to increase the contrast, the sum of the frequency components of the second image, only if they have a sum greater than the energy of the frequency components of the first image, the said pixel in the first image first Replace with the corresponding pixel in the two images ,
The method according to claim 17 or 18 . The method further comprises performing wavelet transform on the first image and the second image to generate the frequency component of the first image and the frequency component of the second image. The method according to any one of 17 to 19 .

Images