JP4359662B2 - Color image exposure compensation method - Google Patents

Description

  The present invention relates to a method for correcting exposure of a color image, and in particular, improper exposure that is referred to as “overexposure” or “collapse” in a color image obtained in a shooting environment under adverse conditions such as a backlight scene with a digital camera. The present invention relates to a color image exposure correction method for correcting an image area to a visually natural image.

  Today, with the spread of personal computers at home, the demand for digital cameras is increasing. However, while digital cameras can be handled easily, images with inadequate exposure due to overexposure or loss of color may be acquired depending on the shooting environment such as backlight. Humans can recognize and recognize pupils even in such a scene, but with digital cameras, the range of brightness that can be reproduced is narrow, resulting in images that differ from human senses.

  In order to correct such “skipping” and “collapsed” parts and improve visibility, contrast is obtained by methods such as linear density conversion and histogram flattening, which are methods using the characteristics of the luminance distribution histogram. Has been proposed (Non-Patent Documents 1, 2, and 3). However, since these methods perform enhancement based on information only in the image, there is a limit to enhancement in a region where exposure is extremely inappropriate. In addition, in an image in which a proper exposure area and an inappropriate exposure area are mixed in one image, too much emphasis is placed on the inappropriate exposure area, and the proper exposure area is excessively emphasized. There is a drawback that it is converted into a visually unnatural image.

As another technique, Patent Document 1 discloses a signal processing method using a luminance histogram as a technique for correcting “backlight”, “whiteout”, and “blackout”. Further, Patent Document 2 and Patent Document 3 disclose an image processing program and method using the RAW mode. However, the frequency image obtained from the color histogram and the information of the L ^* a ^* b ^* color histogram are not used, and a technique for processing a natural image as seen by human eyes is not disclosed. . Therefore, these techniques have not yet led to an image processing technique that solves the above-described problems of overexposure and collapse and corrects the image to a visually natural image.
“Introduction to Computer Image Processing” (by Hideyuki Tamura), Research Institute Publishing, 1985 “Image processing in C language” (by Takeshi Yasui, Masayuki Nakajima, Hideko Kimijiri) Shosodo, 1990 Yukiichi Sakai: Introduction to Digital Image Processing with Visual Basic & Visual C ++, CQ Publisher, 2002 JP 2003-179809 A JP 2004-220438 A Japanese Patent Application Laid-Open No. 2004-222076

  An object of the present invention relates to correction of overexposure and collapse due to backlight or the like in a method for processing a color image based on data acquired from image information of a digital camera, and it cannot be processed into a visually natural image by the related art. An object of the present invention is to provide a color image exposure correction method capable of solving the problems.

Exposure correction method for a color image of the present invention for achieving the above object, by changing the exposure for the same scene to obtain a plurality of color images sequence, for the two color image definitive to the plurality of color images sequence Obtain the ratio of the color appearance frequency for each part of the image, and obtain the blend ratio such that the smaller frequency is blended more than the greater frequency in each part according to the color appearance frequency ratio. A blend ratio map is created for the entire image, and the two color images are blended for each of the portions in accordance with the blend ratio map to create one exposure-corrected image .

  In the above, the present invention obtains the ratio of the color appearance frequency for each part of the image for the two color images, and the frequency is smaller in each part according to the color appearance frequency ratio. Find a blend ratio that blends more than the larger one, create a blend ratio map for the entire image, and blend the two color images for each part according to this blend ratio map to get one exposure A corrected image is created.

  According to the present invention, in the above, the first frequency image in which the color value of the original color image is replaced with the appearance frequency of the color, and the weight of the color that humans do not feel a difference in the frequency value of the first frequency image. A blend ratio map is created using any one of the second frequency images obtained by multiplying the increased coefficient.

  In the above, the present invention provides two images: an overexposed image in which exposure is corrected so as to brighten dark portions of the original image, and an underexposed image in which the original image is corrected so as to be close to the number of colors of the overexposed image. The corrected image is created and used.

  One of the techniques of the present invention is that, in the above, the original image is divided into a foreground portion and a background portion, and an overexposed image is created by correcting exposure so that the foreground portion of the original image is brightened. Alternatively, an image obtained by correcting the exposure of the original image is used as an underexposed image.

  In the above, the present invention creates one or a plurality of intermediate brightness images between the underexposed image and the overexposed image, and each of the plurality of images including the intermediate brightness image and the underexposed image. A plurality of blend ratio maps are created using the ratio between the color appearance frequency and the color appearance frequency in the overexposed image, and a new blend ratio map is created from the plurality of blend ratio maps.

Another method of the present invention is to develop two images having different brightness using the RAW data of the image and use the two images.
In the above, the present invention develops one or a plurality of intermediate brightness images, in addition to a bright image and a dark image having different brightness, and a plurality of images composed of the intermediate brightness image and the dark image. Create multiple blend ratio maps using the ratio of the color appearance frequency in each of the images and the color appearance frequency in the bright image, and create a new blend ratio map from these multiple blend ratio maps. To do.

According to the present invention, the blend ratio map is smoothed by a Gaussian filter according to the intensity of the edge in the image of the edge, thereby reducing the smoothness near the edge and increasing the smoothness other than the edge. It is preferable to create a map.

  Further, according to the present invention, an image converted into a color system based on human perception such as a Ycc color system is created from each of two color images created in the RGB color system, and these conversions are made. It is preferable to create a single image by blending these two images according to the blend ratio map, and converting this image into the original RGB color system to create a color image.

  Further, according to the present invention, a reduced image having a reduced size is created from the original image, a reduced size blend ratio map is created from the created reduced image, and the original size is enlarged from the reduced size blend ratio map. It is preferable to create and use a blend ratio map.

  The present invention has the characteristics described above, so that even when an image having a poorly exposed region such as “whiteout” or “collapsed” is acquired by capturing a scene in which backlight and light / dark regions are mixed, Therefore, it is possible to easily obtain a visually natural image without an improperly exposed area according to the user's needs without requiring a special device.

  That is, a region with an inappropriate exposure generally has a small color difference between pixels, and has a feature that the frequency of the same color is higher in an image than a region with an appropriate exposure. Utilizing this fact, the present invention determines that an image with a low color appearance frequency is an image with an appropriate exposure at the image position.

  In the present invention, the above features are extracted using a new color histogram designed based on human perception, and visually natural correction is performed. At this time, in order to evaluate the appropriateness of exposure, a color image processing technique developed by the present inventors and having frequency information and position information on a color histogram space (Japanese Patent Laid-Open No. 2001-155173). A frequency image technique that visually expresses the difference in frequency required is used. Therefore, the present invention has a feature that the exposure of the local area can be corrected without requiring an area dividing process for spatially detecting an inappropriate area of exposure.

  In addition, the present invention is characterized in that a plurality of images with different brightnesses are generated from an original image or a single imaged image data, so that no special photographing means is required. Further, the present invention has a great feature in that it is not necessary to divide the region, and the blend ratio can be determined and handled in units of pixels.

  Furthermore, according to the present invention, when determining the blend ratio, a color histogram corresponding to human perceptual characteristics is created and determined based on color frequency information obtained from the histogram. The blend ratio can be determined on a pixel-by-pixel basis.

  That is, according to the present invention, it is possible to visually correct an inappropriate exposure area, and depending on the correction result, even in a scene with “overexposure” or “collapse”, both bright and dark areas are almost equal. An image that can be viewed with proper exposure is obtained. As a result, the exposure of the shadow area can be corrected visually and naturally, and the obtained image can be free from an inappropriate exposure area.

  In addition, according to the present invention, it is possible to correct the exposure of the local area without requiring an area dividing process for detecting an inappropriately exposed area. In addition, the present invention can be applied to any kind of digital camera without requiring a special device, and has an effect of excellent versatility.

[Exposure frequency of exposure and color]
A description will be given of the relationship between the visibility of an object of interest and the appearance frequency of colors when a plurality of images of the same scene are captured with different exposures. FIG. 1 is a group of images obtained by photographing the same scene of FIG. 2 in nine ways with different exposures, and numbers 1 to 9 are assigned to the respective images. As shown in FIG. 2, a plastic bowl is placed on the desk, and a plastic bird doll is placed under the desk. Matching the correct exposure to the bowl will cause the bird doll to collapse darkly. On the other hand, if you match the exposure to a bird doll, the bowl will turn white.

  No. 1 in FIG. No. 5 and No. 5 image. 3A and 3B are enlarged views of the bird doll portion in the nine images. No. In the image of No. 5, the bird doll part is crushed. In the image of 9, the bird doll is properly exposed. No. No. 5 and No. 5 image. When the area of only the bird doll part in the image of 9 is plotted in the RGB color histogram space, it is as shown in FIGS. From this, it can be seen that in the RGB space, the part of the appropriate exposure is plotted in a broad manner, and the part of the collapsed part is plotted in a concentrated manner in a low level area.

  Now, if the appearance frequency of the color is given to each position in the RGB space, the color of the appropriate part of the exposure is dispersed and thus the appearance frequency becomes low. On the other hand, since the same color is gathered in the portion that is crushed or jumped, the frequency of appearance of the color increases.

  Accordingly, when comparing the appearance frequency of colors for the same region in a plurality of images taken with different exposures, it can be said that an image with a lower frequency of appearance has a more appropriate exposure.

[Frequency image]
By examining the frequency in a color histogram space having color frequency information, it is possible to determine an appropriate exposure of an image taken with different exposures. This will be specifically described below.

  First, when a color image is expressed by a function f (x, y) of two independent variables x and y representing the coordinates, as shown in FIG. The frequency image g (x, y) is obtained by replacing “red”) with a frequency value (for example, “2”) obtained from the color histogram space. This frequency image g (x, y) visually represents the difference in frequency obtained from the color histogram.

  When displaying a frequency image with a finite gradation, the frequency threshold is set to d, and an image with a frequency smaller than d is displayed in shades according to the frequency value. An image displayed in grayscale according to this frequency value is called a frequency feature detection image. The details are as described in JP 2001-155173 A.

  FIGS. 6A and 6B show frequency feature detection images for FIGS. 3A and 3B when the threshold value d is the same. Here, the lower frequency portion is displayed darker, and the higher frequency portion is displayed brighter.

  Changes in the average frequency of the pot and bird doll region in FIG. 1 obtained using the frequency image are shown in FIGS. From these, it can be seen that the exposure with the lowest average frequency is the most visible.

[Create frequency image]
In an actual scene, the RGB value of a pixel is often affected by various noises, so a frequency image is created using a color image obtained by continuously acquiring a plurality of the same scenes. The specific procedure is shown below.
(1) A color image acquired at time t is denoted by f ^(t), and f ^(t) plotted on the color histogram space is denoted by h ^(t) . h ^(t) has the coordinate value of each pixel of f ^(t) and the frequency at that coordinate.
(2) When the density value of the acquired image f ^(t) is replaced with a frequency value obtained from h ^(t) , f ^(t) becomes a grayscale image representing the appearance frequency of the density value. This image is called a frequency image g ^(t) .
(3) A total S of n frequency images g ^(t) obtained from {f ^(t) ; t = 1, 2,.

さらに、頻度画像の平均をｇｍとすると、ｇｍは次のように表される。

Further, assuming that the average frequency image is gm, gm is expressed as follows.

ｇｍを平均頻度画像と呼び、ｇｍの各画素の値は実数値を取るものとする。
（４）画像として表現する頻度の閾値をｄとし、これを検出平均頻度と呼ぶ。ｉビットの画像メモリを用いて、検出平均頻度ｄまでを表した頻度画像は次式のようになる。

gm is called an average frequency image, and the value of each pixel of gm takes a real value.
(4) The threshold value of the frequency of representation as an image is d, which is called the detection average frequency. A frequency image representing the detection average frequency d using an i-bit image memory is represented by the following equation.

検出した濃淡画像ｇ_ｄを頻度特徴検出画像と呼ぶ。なお、式（３）でｄから１を減じているのは、最低の頻度値を０にするためである。ｇ_ｄにおいて暗い領域が低頻度領域を表し、明るい領域が高頻度領域を表す。以後、頻度画像ｇ^（ｔ）、頻度特徴検出画像ｇ_ｄをまとめて頻度画像と呼ぶ。

The detected grayscale image g _d is referred to as a frequency feature detection image. The reason why 1 is subtracted from d in equation (3) is to set the lowest frequency value to 0. g dark regions in _d represent low frequency region, the bright region represents a high frequency region. Hereinafter, the frequency image g ^(t) and the frequency feature detection image g _d are collectively referred to as a frequency image.

The above creation procedure is shown in FIG. FIG. 8 shows a method of creating a frequency image based on the color image f ^(t) . The color image f ^(t) is plotted on the color histogram, and at the same time, the frequency of appearance of density values is counted. Let the color histogram at this time be h. The frequency image g is an image obtained by replacing the density value of the original image f ^{(t) with} a value obtained by dividing the frequency value obtained from h by the number of added sheets. In the frequency image g, the frequency is expressed in gray scale.

  In the following, in consideration of actual use, a method of creating a frequency image from one acquired image is used without creating an average frequency image.

[Color histogram of RGB color system with uniform color space]
In the RGB color histogram space, it is known that even if the distance is the same color, a human feels a different distance depending on the position. For example, even if the color difference near green and the color difference near blue are the same in the RGB space, humans feel a difference near blue.

For this reason, if the frequency information obtained from the RGB color histogram space is used as it is, an unnatural result that does not match the human sense may be obtained. Therefore, the RGB space is given the property of this uniform color space with reference to the color space of the L ^* a ^* b ^* color system in which the distances between colors that appear to be the same color difference by humans are made equal.

As a technique for that, the frequency of each position in the RGB space having frequency information is multiplied by a weight different depending on the position, and the frequency of the position is obtained. These weights are set so that colors that humans do not feel a difference have a high frequency on the color histogram space, and this color histogram should have the same properties as the L ^* a ^* b ^* color system. Can do.

The L ^* a ^* b ^* color system is said to be a color space suitable for human perception, and in applications such as color reproduction, subtle color differences do not cause much problem, so performance is good and used in all fields. It has been.

L ^* a ^* b ^* color system is L ^* : lightness a ^* : the larger the value, the more red, the smaller the green b ^* : the larger the value, the yellow, and the smaller the blue, the color is represented by three values.

This will be described in detail below. The conversion formula of L ^* a ^* b ^* color system is as shown in the following formula (4).

なお、式（４）のＸ、Ｙ、Ｚは以下の式（５）で表され、Ｘｎ、Ｙｎ、Ｚｎは白色の三刺激値であり環境によって設定する。本発明では６５００Ｋの色温度照明における白基準値としてＸｎ＝９４．０４５、Ｙｎ＝１００．０、Ｚｎ＝１０８．８９２を用いることができる。

In addition, X, Y, and Z of Formula (4) are represented by the following Formula (5), and Xn, Yn, and Zn are white tristimulus values and are set according to the environment. In the present invention, Xn = 94.045, Yn = 100.0, and Zn = 108.892 can be used as white reference values in color temperature illumination of 6500K.

しかし、このＬ^＊ａ^＊ｂ^＊表色系をカラーヒストグラムとすると、以下のような問題点が生じる。（ｉ）負の値が存在し、プログラム上、空間の設定が難しい。（ｉｉ）画像の全ての画素を毎回ＲＧＢからＬ^＊ａ^＊ｂ^＊に変換する必要があり、処理時間がかかる。（ｉｉｉ）ＲＧＢからＬ^＊ａ^＊ｂ^＊への変換が１対１対応ではないため非可逆である。

However, if this L ^* a ^* b ^* color system is a color histogram, the following problems occur. (I) Negative values exist, and it is difficult to set a space on the program. (Ii) It is necessary to convert all the pixels of the image from RGB to L ^* a ^* b ^* each time, which takes processing time. (Iii) Since conversion from RGB to L ^* a ^* b ^* is not one-to-one correspondence, it is irreversible.

Therefore, a color histogram using the RGB color system has the properties of the L ^* a ^* b ^* color system. A specific procedure will be described next.
(1) An L ^* a ^* b ^* color histogram is prepared and the frequency of all points is set to zero.
(2) All colors represented by the RGB color system are plotted on the L ^* a ^* b ^* color histogram.
(3) An RGB color system color histogram is prepared, and the frequency of all points is set to zero.
(4) Convert the point by point in the RGB color system color histogram in ^{L * a * b *, L} * a * b * examine the frequency values of the color histogram, and stores the frequency values in the RGB color histogram. As a result, an RGB color histogram with non-uniform frequency values is created. At this time, it can be said that a color with high frequency is a color that humans do not feel a difference, and a color with low frequency is a color that humans feel a difference.
(5) Create a weight table corresponding to the frequency distribution of the created RGB color system color histogram. Since the lowest frequency is 1 and the highest frequency is 77 in the RGB color system color histogram, the weight can be linearly set to 0.1 to 7.7. When the RGB color histogram is created, the weight of the plotted points is multiplied by the frequency, and the frequency is added. This procedure is shown in FIG.

Once the weight table is created in this way, an RGB color histogram having the same properties as the L ^* a ^* b ^* color system can be created. it can. In the present invention, exposure is evaluated and corrected using a frequency image created by a color histogram having the properties of the L ^* a ^* b ^* color system.

  Once the weighting table for weighting is created, it is not necessary to recreate it every time. The frequency after the color image is plotted in the color histogram space is multiplied by a coefficient obtained from the weight table, and a frequency image is created based on the value, and this image is used for exposure evaluation. This frequency image creation process is performed by replacing each frequency value of the frequency image created by replacing the color value of the original color image with the appearance frequency of the color, and then adding the coefficient of the weight table for the color of the original color image. This is the same as creating a new frequency image by multiplying.

[Exposure Correction Method-1]
(basic way of thinking)
There are two basic concepts of exposure correction when a 24-bit full color image obtained from a digital camera or the like is used.

  (A) The original image that needs to be corrected for exposure can be divided into two areas, an overexposed part and an underexposed part. At this time, the underexposed part often comes to the front of the scene like shooting a person in the backlight, so this is called “foreground”, and the other overexposed part is called “background” To do.

(B) From the original image, an image f _o (x, y) having the optimum foreground exposure is created, and another image f _u (x, y) having the optimum background exposure is created. Then, the two images are blended according to the frequency ratio of each pixel with respect to the frequency image obtained from each image.

  Hereinafter, a specific exposure correction method based on the above two concepts will be described.

(Determination of foreground and background)
The original image is binarized with the L ^* value of the L ^* a ^* b ^* color system. The threshold at this time is automatically determined by a discriminant analysis method, and a portion having a lightness lower than the threshold is used as the foreground, and a portion having a high lightness is used as the background.

(Create f _o (x, y) and f _u (x, y))
It is assumed that the optimal exposure image f _o (x, y) of the foreground part is created by enhancing the brightness of the dark part of the full-color image by linear density conversion. At this time, the gradient of the density conversion straight line is changed little by little to find the one that maximizes the lightness entropy of the foreground portion.

  Assuming that the appearance of a pixel of lightness level I is P (I), the entropy E is obtained by the following equation.

背景部分の最適露出画像ｆ_ｕ（ｘ，ｙ）は、原画像を利用することができる。ただし、他の画像でもよい。なお、後述の理由により、後述のようにして、背景部分の最適露出画像ｆ_ｕ（ｘ，ｙ）の色数を、前景部分の最適露出画像ｆ_ｏ（ｘ，ｙ）の色数に合わせておく。

An original image can be used as the optimal exposure image f _u (x, y) of the background portion. However, other images may be used. For reasons described later, as described later, the number of colors of the optimal exposure image f _u (x, y) in the background portion is matched with the number of colors of the optimal exposure image f _o (x, y) in the foreground portion. deep.

(Create frequency images)
The obtained optimum exposure images f _o (x, y) and f _u (x, y) are respectively plotted in the RGB color histogram space having the above-mentioned uniform color space property, and the respective frequency images g _o (x, y, y) and g _u (x, y) are created.

(Create blend ratio)
Since the exposure is more appropriate in the region with lower frequency, the frequency of the same position is compared in two frequency images, and the blend ratio r _o (x, y) is calculated for each pixel by the following formula so that the lower frequency is blended more. ), R _u (x, y) is determined.

(Create blend ratio map)
Next, a blend image representing the blend ratios r _o (x, y) and r _u (x, y) of the optimum exposure images f _o (x, y) and f _u (x, y) obtained as described above as grayscale images. Create a ratio map image. This image becomes brighter as the blend ratio of the image f _o (x, y) in which the exposure of the foreground portion is appropriate is higher.

(Smoothing of blend ratio map)
Smooth the blend ratio map to avoid the effects of noise. If the smoothing is simply performed, the vicinity of the edge in the image is blurred. Therefore, a smoothing filter whose weight is dynamically changed depending on the location in the image is used. This filter is a Gaussian filter as shown in FIG. 10, but the variance value decreases near the edge as shown in FIG. 10A, and the variance value σ increases as the distance from the edge increases as shown in FIG. 10B. Set.

  The Gaussian filter is a filter in which weights are assigned according to a normal distribution obtained from Expression (8), and the degree of blur varies depending on σ to be given (FEST Project Editorial Board: New Practice Image Processing, Linx, 2002). FIG. 10 is a graph of the weight by σ. FIG. 10A shows a case where σ is 5, and FIG. 10B shows a case where σ is 21.

式（８）のｗは重みでありｘ，ｙは座標、σはフィルタの広がりを示す値である。図１０（ａ）（ｂ）から分かるように、σが小さいほど画像はぼけず、大きいほどぼける。この特性を利用して、エッジ付近ではσを小さく、エッジから離れるに従ってσが大きくなるようにスムージングを行う。この処理をたとえば２枚の画像の平均画像より検出したエッジを参考に行う。平均画像とは、単にブレンドする２枚の画像の同じ位置の画素のＲＧＢの値を加算し、２で除して得られるＲＧＢの値によって画像を構成したものである。また、エッジは公知の画像処理手法であるＰｒｅｗｉｔｔオペレータを用いて検出する。Ｐｒｅｗｉｔｔオペレータとは、画像中のエッジを検出するための手法である。

In Equation (8), w is a weight, x and y are coordinates, and σ is a value indicating the spread of the filter. As can be seen from FIGS. 10A and 10B, the smaller the σ, the less the image is blurred, and the larger the image, the more blurred. Using this characteristic, smoothing is performed so that σ is small near the edge and σ increases as the distance from the edge increases. This processing is performed with reference to an edge detected from an average image of two images, for example. An average image is an image formed by adding RGB values of pixels at the same position of two images to be blended and dividing the result by two. Further, the edge is detected by using a Prewitt operator which is a known image processing method. The Prewitt operator is a method for detecting an edge in an image.

(blend)
By using the RGB values of each pixel of the foreground optimum exposure image f _o (x, y) and the background optimum exposure image f _u (x, y), the blend ratios r _o (x, y) and r _u (x, y) When actually blending, the RGB values of the pixels are once converted into known Y / C separation signals using the following equation (9), and the Y component and the C component are blended separately, and then the equation (9 ) Is inversely converted into RGB values using the equation (10) solved for RGB to obtain an image.

Ｙ成分は輝度であり、Ｃ成分は色である。このようにするのは、ＲＧＢのバランスが崩れないようにするためである。この処理を全ての画素に対して行い、露出補正画像を作成する。

The Y component is luminance, and the C component is color. This is to prevent the RGB balance from being lost. This process is performed on all the pixels to create an exposure correction image.

(Concrete example)
A 24-bit full-color image was obtained from a JPEG image taken with a digital camera, and the experiment was performed using this as an original image.

FIG. 11 shows an original image. First, the foreground and background were determined from this original image by binarization. Here, two people are used as the foreground, and the others are used as the background. Next, the foreground entropy was calculated to obtain the optimal exposure image f _o (x, y) of the foreground portion. This is shown in FIG. The optimal exposure image f _u (x, y) of the background portion was created by reducing the original image to the same number of colors as f _o (x, y).

Next best exposure image _f o _{(x, y),} the frequency image _g o (x, _y) of f u _(x, y), created a g u (x, y). FIG. 13 shows the frequency image g _u (x, y) after color reduction of FIG. 11, and FIG. 14 shows the frequency image g _o (x, y) of FIG. The appropriate part of the exposure was infrequent, the density was high in the frequency image, and the dark part in FIGS. On the contrary, the portion where the exposure is not appropriate has a high frequency, the density is low in the frequency image, and the bright portion in FIGS.

Blend ratios r _o (x, y) and r _u (x, y) were determined, and an image of the blend ratio map of FIG. 15 was created. As described above, this image becomes brighter as the blend ratio of the image f _o (x, y) in which the exposure of the foreground portion is appropriate is higher.

  The blend ratio map was smoothed and blended. The result is shown in FIG. The image of FIG. 16 has proper exposure for both the foreground and the background, the image of FIG. 11 where the background is properly exposed but the foreground is underexposed, and conversely the foreground is properly exposed but the background is overexposed. Compared with the image of FIG. 12, exposure correction could be performed without a sense of incongruity.

(Summary)
As described above, it is possible to locally correct an inappropriately exposed portion using a 24-bit full color image. This is based on the knowledge that, in an image shot with different exposures in the same scene, the appropriate exposure part has a lower color appearance frequency. Unlike the method of blending and correcting based on the brightness information of a color image, this method uses information obtained from a color space based on human perception, so it creates a natural corrected image that is visually uncomfortable. be able to.

(Use of intermediate images)
Next, a case where an intermediate image is used according to the present invention will be described. In this process, for example, a background image obtained by reducing the color of the original image and a foreground image obtained by increasing the brightness of the original image are used. Create an image.

At this time, if a foreground image with a brightened original image is created, the original JPEG image is RGB 8-bit, so the foreground image has fewer colors than the original image.
In a full-color image, there are 256 gradations of RGB, and there are 256 × 256 × 256 colors.

  The foreground image is created by enlarging the original image to a certain brightness. If this brightness is 64 for each of RGB, a portion of 0 to 64 is assigned to 0 to 255 to create a bright image. However, since the original dark portion has only 64 × 64 × 64 colors, the number of colors remains small even when assigned to 0 to 255. Here, when the intermediate image is created, the number of colors is matched with the foreground image.

  The reason for matching the number of colors in this way is that the frequency balance is lost unless a frequency image is created from a color image of the same gradation. For example, when shooting the same scene with 256 gradations for each of RGB and shooting with 64 gradations for each of RGB, even if the exposure level is the same at the same place, The latter is less frequent and the latter is more frequent.

For example, when creating 3 intermediate images,
Frequency image created from background image: Frequency image created from intermediate image 1 Frequency image created from background image: Frequency image created from intermediate image 2 Frequency image created from background image: Frequency image created from intermediate image 3 Background image More frequent images: Like the frequency images created from the foreground image, create a blend ratio map for each of the background image, one or more intermediate images, and the foreground image, and use them to create a new blend ratio map. Get. Specifically, an average of each blend ratio map, a result obtained by multiplying each blend ratio map by a coefficient, and the like can be used as a new blend ratio map. When each blend ratio map is multiplied by a coefficient, for example, the coefficient value multiplied by the blend ratio map of the background image and the foreground image is larger than the coefficient value multiplied by the blend ratio map when the intermediate image is used. It can be.

By doing so, better results can be obtained.
FIG. 17 shows an example of a processing algorithm that uses an intermediate image in addition to the exposure correction method described above.

(Acceleration algorithm)
As shown in FIG. 18, a reduced image that is reduced from the original image is created for the speed-up algorithm. The reduction uses a general algorithm such as nearest neighbor method, linear interpolation method, and cubic interpolation method. Using this reduced image as the original image, a blend ratio map is created by the same processing as described above.

  Next, the blend ratio map obtained as described above is enlarged to the size of the original image by the nearest neighbor method, the linear interpolation method, and the cubic interpolation method. Using the enlarged blend ratio map, an exposure correction image is created from the original image and the foreground image created by brightening the original image.

  By creating a blend ratio map using the reduced image as the original image in this way, the processing speed can be increased.

[Exposure Correction Method-2]
Here, processing is performed using RAW data of the image.
(RAW data and development)
The RAW data is RAW mode data generated by the image sensor at the time of shooting in the digital camera, and is not an image. In general, an image output by a digital camera is generated from RAW data by setting parameters such as a gamma value, brightness, color, and white balance. This process is generally called “development”. The parameters listed above cannot be set arbitrarily with development software released so far. Therefore, for example, as shown in FIG. 19, no color change is observed in the shadow area or the sky area. So, here in developing software free license of "dcraw" that can process the RAW data directly ( "RAW Digital Photo Decoding in Linux ", http: //www.cybercom.net/ ~ dcoffin / dcraw /) using the . When dcraw is used, two images having different brightness can be developed as shown in FIGS.

  Subsequent generation of a frequency image, creation of a blend ratio, creation of a blend ratio map, and smoothing of the blend ratio map are the same as in the case of “Exposure correction method-1” described above. Hereinafter, a specific example of smoothing of the blend ratio map will be described. Here, as described above, smoothing is performed with reference to the edge detected from the average image of the two images. As described above, the average image is obtained by simply adding the RGB values of the pixels at the same position in the two images to be blended and dividing the result by 2 to form an image. An average image of the images of FIGS. 20 and 21 is shown in FIG. FIG. 23 shows the result of detecting edges by applying the Prewitt operator to FIG. FIG. 23 is a grayscale image, where the black part is a sharper edge. Therefore, the σ of the Gaussian filter corresponding to the sharpness of the edge may be determined.

  The result of blending the images shown in FIGS. 20 and 21 by the above processing and the blend ratio map at that time are shown in FIGS. 24 and 25, respectively. Further, FIG. 26 shows a blend ratio map when smoothing is not performed, and FIG. 27 shows a distribution chart in which the state of change of σ can be confirmed by color. There is no noise or unnatural boundary in the blend result, and it can be confirmed that σ is small near the edge in the distribution diagram of σ.

(Create a blend ratio map using multiple images with different brightness)
The blending result of FIG. 24 in which the exposure is corrected by blending two images does not feel unnatural at first glance, but the shadow on the left shoulder of the person is unnatural when viewed together with the original image. This is because there is a difference in the brightness of the two images to be blended, so that the area of the flying region in the overexposed image is increased, the frequency becomes extremely high, and the blending becomes impossible. Therefore, a method for creating an optimum blend ratio map will be described.

  For this purpose, as in the case of “use of intermediate image” described above, not only two images to be actually blended but also an image having an intermediate brightness is used. Specifically, an average image of several blend ratio maps obtained by blending an under-ish image and an intermediate brightness image and the blend ratio map shown in FIG. 25 is created and blended. A ratio map is used. This process is shown in FIG. At this time, the smoothing of the blend ratio map described above is applied only to the optimum blend ratio map to be created. This is to shorten the processing time. Note that the RGB values used for blending are data of two images that are the most distant from each other. In the case of FIG. 28, an image with brightness 1 and an image with brightness 2.5 on the left and lower right in the figure. In the images shown in FIGS. 20 and 21, the blend ratio map created from the three blend ratio maps and blended are shown in FIGS. 29 and 30, respectively. The shadow of the person's left shoulder disappears and better results are obtained.

  Instead of creating an average image of a plurality of blend ratio maps as described above, for example, a plurality of blend ratio maps multiplied by appropriate coefficients are added to form a new blend ratio map. It is also possible to use an appropriate method such as.

(Acceleration algorithm)
Similarly, when processing is performed using the RAW data, a reduced image reduced from the original image is created. Using this reduced image as the original image, a blend ratio map can be created, and the resulting blend ratio map can be enlarged to the size of the original image, and this enlarged blend ratio map can be used to create an exposure compensation image. Processing speed can be increased.

(Process flowchart)
FIG. 31 shows the above processing in a flowchart. This process includes the following steps.

(A) acquiring image information;
(B) An RGB color system using one image each of underexposed (I _u ) and overexposed (I _o ) and intermediately exposed N images (where N is a natural number of 0 or 1). The process to create in
(C) creating an RGB color system color histogram having L ^* a ^* b ^* color system properties using a weight table;
(D) creating a frequency image of I _u and I _o ;
(E) creating an average image I _m of I _u and I _o , detecting steepness of I _m and determining parameters of a position-variable Gaussian filter;
(F) determining a blend ratio for each pixel and creating a corrected blend ratio map using a Gaussian filter;
(G) a step of performing Y / C separation of RGB values in each image;
(H) creating a composite image using a blend ratio for each of the Y component and the C component;
(I) a step of converting the Y component and the C component into an RGB color system;
(J) obtaining an image display (final image);

(Concrete example)
In order to verify the effectiveness of the present invention, an image in which an improperly exposed area and an appropriate area are mixed was acquired and experimented. The image used in the experiment is a 1088 × 724 [pixel] RGB 8-bit full-color image captured in the RAW mode of the digital camera Canon EOS D30 and developed in half. The processing was performed using a personal computer equipped with an Athlon XP 2500+ CPU and 512 MB RAM memory. The processing time was about 2.5 (min) although there were differences depending on the images.

  Two original images to be processed in the experiment are shown in FIGS. 32 and 33 and FIGS. 36 and 37, respectively. FIGS. 34 and 38 show the results of development using commercially available development software, while FIGS. 35 and 39 show images obtained by correcting areas with inappropriate exposure by the method of the present invention.

  As a result of the experiment, in the underexposed image shown in FIG. 32, the room is crushed by backlight, and in the overexposed image shown in FIG. 33, the scenery outside the window is flying and cannot be confirmed. The image in FIG. 34 is the result of development by commercially available software, but the room is crushed and no color change is seen in the empty area.

  In the image shown in FIG. 36, which is another experimental result, the left half is shaded and dark. In FIG. 37, the shadow area is bright enough to be seen, but other areas are flying. The development results shown in FIG. 38 in which these are corrected by commercially available software are images that are generally dark and difficult to see.

  On the other hand, the image of FIG. 35 which shows the experimental result to which the color image exposure correction method of the present invention is applied is an image that can be confirmed well both indoors and outdoors. Further, it can be seen that the image of FIG. 39 showing another experimental result based on the present invention does not have an improperly exposed area, and the exposure of the shadow area can be corrected visually and naturally.

  As described above in detail, according to the color image exposure correction method of the present invention, from an image with a scene in which bright areas and dark areas such as backlight and shadow are mixed, so that it actually looks natural to the human eye, It is possible to obtain a visually natural image without an inappropriate area of exposure.

  It should be noted that the method described in “Exposure correction method-1” and the method described in “Exposure correction method-2” can be mutually used as much as possible in each correction method.

It is a figure which shows the example of the image group from which exposure differs in steps for demonstrating the principle of this invention. FIG. 2 is an enlarged view of a scene for the image group of FIG. 1. It is an enlarged view of the principal part in FIG. FIG. 4 is a diagram in which the region of FIG. 3 is plotted in an RGB color histogram space. It is a figure explaining the frequency image based on this invention. It is a figure which shows the frequency image of FIG. It is a figure which shows the change of the average frequency of the principal part of FIG. It is a figure explaining the creation procedure of the frequency image based on this invention. It is a figure explaining the specific procedure for giving the property of L ^* a ^* b ^* color system to the color histogram using RGB color system based on this invention. It is a figure which shows the form of the weight by (sigma) value of a Gaussian filter based on this invention. It is a figure which shows an example of the original image for enforcing the exposure correction method of this invention. It is a figure which shows the optimal exposure image of the foreground part in FIG. It is a figure which shows the frequency image about the image of FIG. It is a figure which shows the frequency image about the image of FIG. It is an image of the blend ratio map of a foreground part and a background part. It is a figure which shows a blend result. It is a figure explaining the example using an intermediate image. It is a figure explaining the speed-up algorithm by producing a reduced image. It is a figure which shows the image development result by the developing software of a commercial item about a certain real image for implementing the exposure correction method of this invention. It is a figure which shows the image of the brightness 1 by the development software about the same photographed image. It is a figure which shows the image of the brightness 2.5 by the development software about the same photographed image. It is a figure which shows the average image of the image of FIG. 20, and the image of FIG. It is a figure which shows the edge detection image of FIG. 22 by a Prewitt operator. It is a figure which shows the image showing the result of the blend and smoothing of the image of FIG. 20 and FIG. It is a figure which shows the blend ratio map used in order to obtain the image of FIG. It is a figure which shows the blend ratio map which does not smooth of the image of FIG. 20 and FIG. FIG. 6 is a distribution diagram of σ showing changes in σ in a Gaussian filter on an image. It is a conceptual diagram explaining the preparation method of the optimal blend ratio map based on this invention. It is a figure which shows the image showing the blend result using the several image from which brightness differs based on this invention. It is a figure which shows the blend ratio map used in order to obtain the image of FIG. It is a flowchart of an example of the exposure correction method of this invention. It is a figure which is another actual image and is an original image of the under-exposure. FIG. 33 is a photographed image of the same target as that in FIG. 32 and showing an overexposed original image. It is a figure which shows the image showing the image development result by the commercial software about the object of FIG. 32 and FIG. It is a figure which shows the image showing the image of the image development by the method of this invention about the object of FIG. It is a figure which is another actual image and is an original image of the underexposure. FIG. 37 is a photographed image of the same target as in FIG. 36 and showing an overexposed original image. It is a figure which shows the image showing the image development result by the commercial software about the object of FIG. It is a figure which shows the image showing the image development result by the method of this invention about the object of FIG. 36 and FIG.

Claims (10)

A plurality of color image series is obtained by changing the exposure for the same scene, and a ratio of color appearance frequency is obtained for each part of the image for two color images in the plurality of color image series . In accordance with the ratio of the appearance frequencies of the colors, a blend ratio is determined such that a smaller frequency is blended more than a larger frequency, and a blend ratio map is created for the entire image. An exposure correction method for a color image, wherein one exposure correction image is created by blending one color image for each portion .   Obtained by multiplying the frequency value of the first frequency image by replacing the color value of the original color image with the appearance frequency of the color and the frequency value of the first frequency image multiplied by a coefficient that increases the weight of the color that humans do not feel a difference. The color image exposure correction method according to claim 1, wherein a blend ratio map is created using any one of the second frequency images.   Two corrected images are created: an overexposed image in which exposure is corrected so that dark portions of the original image are brightened, and an underexposed image in which the original image is corrected to be close to the number of colors of the overexposed image. 3. The color image exposure correction method according to claim 1, wherein the color image exposure correction method is used.   The original image is divided into a foreground portion and a background portion, and an overexposure image is created by correcting exposure so that the foreground portion of the original image is brightened. The color image exposure correction method according to claim 3, wherein:   Create one or more images with intermediate brightness between underexposed and overexposed images, and the frequency of color appearance and exposure for each of the images with intermediate brightness and underexposed images. 5. The blend ratio map of a plurality of sheets is created using a ratio with the appearance frequency of the color in the over image, and a new blend ratio map is created from the blend ratio map of the plurality of sheets. Color image exposure compensation method.   3. The color image exposure correction method according to claim 1, wherein two images having different brightness are developed using RAW data of the image, and the two images are used.   In addition to a bright image and a dark image having different brightness, one or a plurality of intermediate brightness images are developed, and the color of each of the plurality of images including the intermediate brightness image and the dark image is changed. A plurality of blend ratio maps are created using a ratio between an appearance frequency and a color appearance frequency in the bright image, and a new blend ratio map is created from the plurality of blend ratio maps. Item 7. A color image exposure correction method according to Item 6. By smoothing the blend ratio map with a Gaussian filter according to the edge intensity in the edge image, the smoothness near the edge is reduced, and a new blend ratio map is created in which the smoothness other than the edge is increased. 8. The color image exposure correction method according to claim 1, wherein the color image exposure is corrected.   From each of the two color images created in the RGB color system, an image converted to a color system based on human perception such as the Ycc color system is created, and the two converted images are blended. 9. The method according to claim 1, wherein a single image is created by blending according to a ratio map, and a color image is created by converting the image into an original RGB color system. Color image exposure compensation method.   Create a reduced image with a reduced size from the original image, create a reduced size blend ratio map from the created reduced image, and create a blend ratio map enlarged to the original size from this reduced size blend ratio map 10. The color image exposure correction method according to claim 1, wherein the color image exposure correction method is used.

Images