JP2008052428A - Image processing method, image processor, image processing program and imaging device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing method appropriately adjusting a white balance even when there are a plurality of illumination light sources. <P>SOLUTION: In the image processing method, an image is acquired, the acquired image is divided into a plurality of regions, and a white balance processing is performed for every divided region. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing method, an image processing apparatus, an image processing program, and an imaging apparatus that appropriately perform white balance processing on an image.

  If the photographic light source is outdoors during the day, its color temperature changes depending on the altitude of the sun, and the color temperature also differs between the sun and the shade. There are various types of artificial lighting such as fluorescent lamps and incandescent lamps indoors. When shooting with flash, mixed light of flash lighting and external light is used as a shooting light source. Under such various photographic light sources, the human eye adapts to color and feels that the spectrally flat reflector is almost achromatic. In accordance with this, the camera as an input device also adjusts the RGB gain to achieve a so-called white balance. Usually, since the photographing light source is unknown, the type (color temperature) of photographing illumination is estimated from the image or designated by the user, and white balance is obtained according to the illumination type.

  On the other hand, in Patent Document 1, when flash photography is performed, both the image photographed using flash and the image photographed without flash are acquired, and the image is divided into blocks, and then the luminance ratio of the two images is obtained. A configuration is disclosed in which white balance adjustment is performed based on strobe light for blocks whose luminance ratio is higher than a reference level, and white balance adjustment is performed based on external light for blocks lower than the reference level.

JP-A-8-51632

  However, when two consecutive images are taken, there is a possibility that the image may be shaken due to the camera shake of the photographer himself or the movement of the subject. If there is a flashing light source in the external light, or if the external light is flickering at a certain frequency, such as a fluorescent lamp, the brightness of the fired images is compared. It is difficult to determine whether the difference in brightness is a change in brightness due to flash emission or other effects. If the brightness difference is higher than the reference level, the white balance is adjusted with the strobe light, but the white balance adjustment is not adjusted to the mixed light of the external light and the strobe light, so the colors are unnatural. It will be adjusted.

The invention of claim 1 is applied to an image processing method, acquires one image, divides the acquired image into a plurality of areas, sets information on the color of illumination for each of the divided areas, and divides the image. For each region, color correction is performed in accordance with information related to the color of the corresponding illumination.
The invention according to claim 2 is the image processing method according to claim 1, wherein the acquired image is divided into a plurality of regions by designation by the user on the image displayed on the monitor. .
According to a third aspect of the present invention, in the image processing method according to the first aspect, the acquired image is divided into a plurality of regions using distance information to the subject.
According to a fourth aspect of the present invention, in the image processing method according to the first aspect, a human face is recognized in the acquired image, and the acquired image is divided into a recognized human face region and other regions. It is characterized by this.
According to a fifth aspect of the present invention, in the image processing method according to the first aspect, the acquired image is divided into a main subject and a background.
According to a sixth aspect of the present invention, in the image processing method according to the first aspect, the acquired image is divided into a sunny area and a shaded area based on the luminance information.
The invention according to claim 7 is the image processing method according to claim 1, wherein the acquired image is divided into a skin color part and a part other than the skin color.
According to an eighth aspect of the present invention, in the image processing method according to any one of the first to seventh aspects, the information regarding the color of the illumination is set by estimating each illumination light source of the plurality of divided regions. It is a feature.
According to a ninth aspect of the present invention, in the image processing method according to any one of the first to seventh aspects, the information regarding the color of the illumination is set by designating each of the plurality of divided areas by a user. It is a feature.
According to a tenth aspect of the present invention, in the image processing method according to the eighth aspect, the illumination light source is estimated using color information and photographing information for each of the divided areas.
The invention according to claim 11 is the image processing method according to claim 3, wherein shooting information as to whether or not flash photography has been performed is acquired, and if flash photography is being performed, the subject is selected from a plurality of divided areas. The illumination light source in the area whose distance information is less than or equal to a predetermined value is estimated to be a flash, and the white balance processing suitable for the flash is performed on the area where the illumination light source is estimated to be a flash. is there.
According to a twelfth aspect of the present invention, in the image processing method according to the fourth or seventh aspect, in the case of acquiring shooting information as to whether or not flash shooting has been performed and performing flash shooting, The illumination light source of the partial area is estimated to be a flash, and white balance processing suitable for the flash is performed on the area where the illumination light source is estimated to be a flash.
The invention of claim 13 is applied to an image processing program, and is an image processing program for causing a computer to execute the image processing method according to any one of claims 1 to 12.
The invention of claim 14 is applied to an image processing apparatus, and is an image processing apparatus equipped with the image processing program of claim 13.
The invention of claim 15 is applied to an image pickup apparatus, and is an image pickup apparatus having the image processing program according to claim 13 mounted thereon.

  Since the present invention is configured as described above, color correction is appropriately adjusted in the entire image, and color reproduction is improved.

-First embodiment-
FIG. 1 is a diagram showing an image processing apparatus according to an embodiment of the present invention. The image processing apparatus is realized by the personal computer 1. The personal computer 1 is connected to a digital camera 2, a recording medium 3 such as a CD-ROM, another computer 4, etc., and receives various image data. The personal computer 1 performs image processing described below on the provided image data. The computer 4 is connected via the Internet and other telecommunication lines 5.

  A program executed by the personal computer 1 for image processing is provided from a recording medium such as a CD-ROM or another computer via the Internet or other electric communication line, as in the configuration of FIG. 1 is installed. The personal computer 1 includes a CPU (not shown) and its peripheral circuits (not shown), and executes a program in which the CPU is installed.

  The personal computer 1 according to the present embodiment performs image processing for appropriately white balance the captured image even under illumination with a plurality of color temperatures. Specifically, image processing is performed on image data of a captured image, but in the following description, the expression that image processing is performed on an image is also used.

  An example will be described in which white balance correction is performed on an image obtained by flashing a person against a sunset. The shooting illumination of the person who is hit by the flash is the flash used mainly, and the color temperature of the shooting illumination by the flash is about 5400K. However, if the white balance of the entire image is obtained at this color temperature, the white balance of the flashed area is corrected appropriately, the achromatic color is achromatic, and the skin color is reproduced as skin color, but the background sunset is reddish white. The red color of the sunset is not reproduced.

  On the other hand, if you want to adjust the white balance to match the impression of red when you see the sunset, you can intentionally set the color temperature to a high value (8000K), etc., to reproduce the scenery that is colored red and sunset. However, even in this case, the person is dyed unnaturally red.

  In the present embodiment, when such a plurality of objects under illumination are mixed, the image area is first divided. As for the division method, the user divides the area using an input device such as a mouse so as to divide the image into a person and a background while viewing the image displayed on the monitor of the personal computer 1. Thereafter, the user designates the color temperature and the type of photographing illumination for each of the divided areas, and sets information regarding the color of the illumination. Thereafter, when the user clicks a color correction execution button or the like with the mouse, the personal computer 1 performs a process of performing color correction such as white balance suitable for the designated color temperature and the type of photographing illumination for each area.

  In the above example, the image may be divided into a person area and a background area. If the color temperature of the flash is designated for the divided person area or the photographing illumination is designated as a flash, color correction such as white balance suitable for the flash illumination is performed. As for the background, if a color temperature higher than the sunset color temperature is specified so as to leave the redness of the sunset, a white balance in which the red of the sunset remains is obtained.

  FIG. 2 is a diagram illustrating a flowchart of an image processing program executed by the personal computer 1 according to the present embodiment.

  In step S1, an image (image data) taken (captured) with a digital camera or the like is acquired. In step S2, the acquired image is displayed on the monitor of the personal computer 1.

  In step S3, the image is divided according to the user's input operation. For example, the cursor is moved by the user's mouse operation so that the locus of movement of the cursor and the area closed on the outer periphery of the image can be designated. In the example of the person and background image described above, if the user traces the outline of the person with the cursor, the person can be designated as the person area. The background can be designated as a background area by collecting the areas not designated.

  Alternatively, the entire displayed image may be divided in advance into a plurality of grid areas, and the divided grid areas may be appropriately selected with a mouse or the like so as to be designated as a single area.

  In step S4, information about the color of the illumination for performing color correction such as white balance is set for each divided region. In this embodiment, the color temperature of illumination is set. The user designates a color temperature for each divided area on the monitor. The color temperature may be specified by specifying a specific numerical value of the color temperature or by specifying the type of illumination. The type of illumination is selected from, for example, sunlight (sunshine), sunlight (shade), sunlight (cloudy), flash, incandescent lamp, fluorescent lamp, and the like. Sunlight may be classified according to the altitude of the sun, etc., or it may be classified according to sunset, sunrise, daytime, etc., or may be classified according to time of day. Fluorescent lamps may also be further classified according to the type of fluorescent lamp. Moreover, you may add another illumination.

  In step S5, white balance processing is performed for each of the divided areas based on information on the set illumination color. If the color temperature set for each region is different, different white balance processing is performed for each region.

  In the above description, the user specifies information regarding the illumination color of each area. If the color temperature of the sunset is specified for the sunset background, white balance is taken so that the white appears appropriately white even with red light such as sunset. Therefore, when it is desired to retain the redness of the sunset, it is necessary to devise designating a color temperature higher than the color temperature of the sunset in that region.

  In such a case, a plurality of illumination types such as sunset 1 and sunset 2 may be designated as illumination types. If sunset 1 is specified, white balance is processed so that white appears appropriately white even if the light is red like sunset. If sunset 2 is specified, the redness of the sunset is set. It is also possible to perform white balance processing that leaves The same applies to incandescent lamps in the room.

  In addition to the method of setting the gain of the output RGB value like white balance, color correction may be performed using chromatic adaptation conversion. In the chromatic adaptation conversion, information regarding the color of the illumination is input for each region, and the white point of the output device is further input to perform color correction for each region. You may make it consider the chromatic adaptation degree D which shows how much it adapts to object illumination with this method.

This chromatic adaptation can be expressed by the following equation.
D = F [1- (1 / 3.6) exp ((-LA-42) / 92)]
Note that F is a parameter indicating ambient brightness, LA is adaptation luminance, and normally 20% of illumination luminance is used.

  In the image example in which a person is flash-photographed against the background of the sunset described above, the shooting illumination is set to flash for the area where the person who is the main subject is cut out, and D = 1.0, which is completely adapted. For the background area illuminated by the sunset, the shooting illumination is sunset and D <1.0 is not fully adapted. In this way, the chromatic adaptation conversion is appropriately performed for the person in the flash illumination, and the chromatic adaptation conversion for the background in which red of the sunset is left is performed.

(1) As described above, in this embodiment, color correction such as white balance can be appropriately performed for any subject in an image, and good color reproduction can be realized in the entire image.
(2) Since the user can arbitrarily specify the divided area and can also arbitrarily specify the color temperature and the type of illumination, color correction such as white balance can be performed according to the user's preference.

-Second Embodiment-
FIG. 3 is a diagram showing a configuration of a digital camera 100 that is an imaging apparatus according to an embodiment of the present invention. The digital camera 100 includes a photographing lens 102, an image pickup device 103 including a CCD, a control device 104 including a CPU and peripheral circuits, a memory 105, a lens driving unit 106, a flash unit 107, and the like.

  The image sensor 103 captures (captures) the subject 101 via the photographing lens 102 and outputs the captured image data to the control device 104. The control device 104 performs image processing described below on the image data captured by the image sensor 103 and stores the image data after the image processing in the memory 105 as appropriate. Further, the control device 104 drives the photographing lens 102 via the lens driving unit 106 and acquires distance measurement data up to the subject. Further, the control device 104 drives the flash unit 107 to perform flash photography. Note that the image processing program executed by the control device 104 is stored in a ROM (not shown).

  In the digital camera 100 of the present embodiment, image processing for appropriately white balance is performed on an image photographed (captured) by the image sensor 103 even under illumination with a plurality of color temperatures. Specifically, image processing is performed on image data of a captured image, but in the following description, the expression that image processing is performed on a captured image is also used.

  This embodiment will be described by taking an example of flash photography indoors. When the flash is shot indoors under incandescent lighting, if the white balance is adjusted predominantly according to the color temperature of the flash, the white balance of the subject (such as a person) hit with the flash is completely corrected. The image is white-balanced and has a cold tone.

  On the other hand, when shooting without using a flash, the white balance is corrected predominantly by the color temperature of the incandescent lamp in the room, but the photo is finished with warmth. I prefer the latter warm photos, and shoot without using the flash despite the disadvantage of being dark and blurry, or not using the flash at the expense of noise and increasing the ISO sensitivity Some users do.

  In the present embodiment, in a photo having such a composition, the background portion where the flash unit 107 does not reach the flash takes a white balance with respect to the color temperature estimated only from the background area. For a short-distance subject area where the flash reaches, a white balance is obtained at the color temperature of the flash. As a result, an appropriate white balance can be obtained even for an image captured by the flash unit 107 using the flash.

  FIG. 4 is a diagram showing a composition when flash photography is performed indoors. FIG. 5 is a diagram showing a captured image divided into 5 × 5.

  In the present embodiment, the distance to the subject is measured in each of the 5 × 5 areas shown in FIG. The distance measurement uses the distance measurement function of the digital camera. For example, a contrast method used in autofocus is used. Specifically, the control device 104 controls the lens driving unit 106 to drive the focusing lens of the photographing lens 102 from the closest side to the infinite edge side with a predetermined resolution. Then, while driving the focusing lens of the photographing lens 102 from the closest side to the infinite edge side, focus evaluation values are acquired in each region divided into 5 × 5 of the image.

  The point at which the focus evaluation value reaches the peak is that the subject is in focus, and the distance to the subject can be calculated by the driving position of the focusing lens of the photographing lens 102 at that time. Since the distance measuring method to the subject in the digital camera has a publicly known content, a detailed description is omitted. Various other ranging methods can also be used.

  As shown in Fig. 4, when a person at a short distance is flashed against a background illuminated by an incandescent lamp as shown in Fig. 4, if there is information on the use of the flash, the flash can reach each area. The area can be divided into other areas.

  It is assumed that the area where the person in FIG. 4 is located (areas indicated by reference numerals 14, 19, and 24 in FIG. 5) is as short as 1 m, for example, based on the distance measurement information described above. In this case, if there is information indicating that the flash is turned on and flash performance information, it can be determined that the flash is used as information regarding the color of the photographic illumination of the subject.

  On the other hand, in other areas, it is assumed that the distance to the subject is far enough that the flash does not reach from the distance measurement information. The photographing illumination in this case is estimated by analyzing the image of FIG. 4 corresponding to the area other than the reference numerals 14, 19, and 24 in FIG. 5, and set as an incandescent lamp. For the regions other than the reference numerals 14, 19, and 24, the white balance is adjusted according to the set photographing illumination, and the white balance is adjusted so that the warm shade of the incandescent lamp remains. In the areas indicated by reference numerals 14, 19, and 24, the white balance is adjusted using the photographing illumination as a flash.

  In this way, appropriate color correction is performed on the main subject while leaving the warm color of the incandescent lamp.

  FIG. 6 is a diagram illustrating a flowchart of an image processing program executed by the control device 104 of the digital camera 100 according to the present embodiment.

  In step S11, the control device 104 acquires distance measurement data of each 5 × 5 area as described above. The distance measurement data may be acquired in cooperation with the autofocus process of the digital camera 100. In step S12, an image (image data) captured (captured) by the image sensor 103 is acquired. In step S13, photographing information indicating that the flash unit 107 has used the flash is acquired.

  In step S14, the image acquired in step S12 is divided based on the distance measurement data acquired in step S11. In this embodiment, on the premise that flash photography has been performed, the image is divided into an area where the flash can reach and an area where the flash cannot reach. Specifically, the area is divided depending on whether the distance measurement value of each area is greater than or less than a reference value at which the flash reaches. In the example of FIGS. 4 and 5, the area is divided into areas 14, 19, and 24 where the distance to the subject is 1 m, and other areas.

  In step S15, information about the color of the illumination is acquired from an image in an area where the flash does not reach. In the above example, it is estimated and set by a known method that the color of the illumination is equivalent to that of an incandescent lamp by image processing. For the area where the flash reaches, the illumination color is set as flash. The color temperature of the flash is stored in advance in a memory in the flash unit 107, and is obtained by reading the data. In step S16, white balance processing based on the estimated color temperature is performed for each divided area.

  In this way, the image is divided into areas where the flash can reach and areas where the flash does not reach, and color correction is performed according to the information regarding the illumination color of each area. As a result, in the image having the composition as described above, appropriate color correction corresponding to the light source color of the flash is performed on the main subject while leaving the warm shade of the incandescent lamp in the background.

  For areas 14, 19, and 24 within the reach of the flash, color correction is performed according to the light source color of the flash, and then color reproduction is performed when illuminated with the light source color of the background area. You may make it perform conversion. For this conversion, an expression similar to the white point conversion such as the von Chris expression may be used. In this way, for the areas 14, 19, and 24, the shooting illumination is a flash, but since the illumination that the photographer was actually looking at has the same color temperature as the surroundings, the color illuminated by the flash It can be adjusted so that the color is reproduced when illuminated at the same color temperature as the surroundings.

    A modification of the second embodiment will be described with reference to FIG. This variation estimates the mixed illumination color temperature of the flash and external light, and if the area corresponds to the boundary between the flash illumination area and the background illumination area by external light illumination, how much the area is from the flash illumination area A method for adjusting the white balance according to whether the subject is a distant subject or how far away from the background illumination area is will be described.

  FIG. 9 is a diagram illustrating a flowchart of a modification of the image processing program executed by the control device 104 of the digital camera 100 according to the above embodiment. In step S41, it is detected whether the flash is used. If a flash is used, the image sensor is divided into N × M areas in step S42.

  Next, in step S43, distance measurement is performed on representative pixels in each region, the data is held, shooting is performed, and image data is generated. Next, in step S44, the generated image data is divided so as to correspond to the area division in step S42, and it is determined whether or not the flash reaches a range in each area. Therefore, if it is determined that the area does not reach the flash, the background illumination area is determined in step S45. On the other hand, if it is determined that the area reaches the flash, it is determined as a flash illumination area in step S46. This is determined over the entire area (step S47).

  When all the areas are classified into the background illumination area and the flash illumination area, the entire flash illumination area is analyzed and estimated as the flash mixed illumination color temperature Tf in step S48. Alternatively, it may be set based on the color temperature of the flash with reference to the color information of the flash as information about the color of the illumination, but based on the data of the background illumination color temperature Tb obtained in the next step S49, The flash color temperature may be set to a value close to the background illumination color temperature Tb. In step S49, the entire background illumination area is analyzed to estimate the background illumination color temperature Tb.

  Next, in step S50, it is determined whether each area is the flash illumination area set in step S44. In step S51, it is determined whether the area determined as the background illumination area is at the boundary with the flash illumination area. If it is not the boundary region, white balance adjustment is performed at the background illumination color temperature Tb in step S53. On the other hand, if it is a boundary area, in step S54, how far the subject in this area is from the subject in the flash illumination area and from the subject in the adjacent background illumination area. It is determined that the distance is approximately, and a color temperature intermediate between Tf and Tb is set. The white balance is adjusted with this color temperature.

  Similarly, when it is determined in step S50 that the area is the flash illumination area performed in step S44, it is determined in step S52 whether or not the area is a boundary area with the background illumination area. If it is determined not to be at the boundary, white balance is corrected at Tf in step S55. If it is in the boundary region, the process of step S54 is performed as described above, and the white balance is adjusted. It is determined in step S56 whether this has been done for all areas, and the process ends.

  On the other hand, if it is determined in step S41 that the flash is not used, the process proceeds to step S57. In step S57, the illumination color temperature is estimated for the entire image, and white balance adjustment is performed.

  Note that the region division is not divided into blocks, but may be divided into arbitrary shapes using the result of edge extraction. After dividing into an arbitrary shape, each divided area may be classified into an area at a distance where the flash reaches (area 1) or an area where the flash does not reach (area 2).

  In the above example, the area is classified into two areas: the area that reaches or does not reach, but the area that is quite close (= strong influence of the flash) and the area that does not reach at all and the middle (the flash and background light source are mixed) As described above, the white balance may be adjusted by classifying more finely than two and estimating the color temperature for each area.

  If only two types of white balance adjustment values are switched by blocks, block distortion occurs at the boundary of different white balance conditions. Therefore, the area where the white balance (14, 19, 24 in FIG. 4) is performed with the mixed lighting of the flash and the indoor lighting and the area where the white balance is performed with the background color (except for 14, 19, 24 in FIG. 4) (see FIG. 4). 4 in 8, 9, 10, 13, 14, 15, 18, 19, 20, 23, 24, 25) (for example, about 100 pixels each) as a boundary area, as shown in FIG. Set to a color temperature interpolated between the set color temperature and the color temperature set in area 2.

  In FIG. 10, the display is one-dimensional for the sake of simplicity, but since the region is actually two-dimensional, the color temperature is estimated by interpolation so that the color temperature is smoothly connected in the two-dimensional region.

  Also, instead of interpolating the color temperature only at the boundary, if the color temperature of the illumination is determined for each block, the color temperature is the color temperature at the representative pixel (for example, the central pixel) of each block, and the entire image is two-dimensional. The color temperature distribution function T (x, y) may be created, and the white balance of each pixel may be adjusted using the equation.

  In the above example, the color temperature is used as the feature value of the illumination. However, the feature amount represented by the xy chromaticity coordinates (x, y) of the white point of the illumination or the camera RGB value is used as the feature value of the plurality of illuminations. It may be used. The color temperature alone can represent the feature quantity of illumination only in one dimension. However, if the feature quantity is represented by xy chromaticity coordinates and camera RGB values, it can more accurately represent mixed illumination and fluorescent lamps that do not follow black body radiation. be able to. If the feature quantity indicating the type of illumination uses two or more parameters, the approximate expression such as the interpolation of the boundary region (Fig. 10) and the color temperature distribution function T (x, y) described above is used for each parameter. create. For example, when the white RGB ratio (G / R, G / B) is used as the feature value, an approximate expression such as the above-described interpolation of the boundary region (FIG. 10) or the color temperature distribution function T (x, y) is used. This is performed for each of / R and G / B and used for WB adjustment of each pixel.

(1) As described above, a warm image of an incandescent lamp can be left as an entire image while obtaining a bright image free of camera shake using a flash, and a good color reproduction is possible.
(2) If the white point conversion is performed as shown in the above-described modification, the warm color of the incandescent lamp can be further improved as the entire image.
(3) Since the distance measurement data up to the subject is acquired by the same method as that of autofocus, it is possible to automatically and appropriately perform image division for which white balance processing is to be made different.

-Third embodiment-
In the third embodiment, an example will be described in which color correction is performed by dividing a region (main subject) determined to be a face and other regions using face recognition technology. When an area determined to be a face is detected and it is detected that the flash is turned on, it can be estimated that the illumination of the area determined to be a face is a flash. In the present embodiment, based on such estimation, different white balance processing is performed for a region determined to be a face and other regions depending on the difference in illumination light source.

  The third embodiment is realized by the personal computer 1 as in the first embodiment. Therefore, the configuration of the image processing apparatus according to the third embodiment is referred to FIG. 1 of the first embodiment. FIG. 7 is a diagram illustrating a flowchart of an image processing program executed by the personal computer 1 according to the present embodiment.

  In step S21, an image (image data) taken (captured) with a digital camera or the like is acquired. In step S22, shooting information indicating that the flash has been used is acquired. It is assumed that the shooting information is attached to the image data. In step S23, a human face is recognized from the acquired image. A known face recognition technique is used for recognizing a person's face. For example, the outline of a person's face is detected by image processing, or the skin color is detected to recognize the person's face.

  In step S24, based on the recognized face area of the person, the face area is divided into other areas. In step S25, the white balance is adjusted for the area divided as the human face, using the flash color temperature as the illumination temperature of the area. In step S26, the white balance is adjusted so that the average luminance of the region other than the face region is 18% gray.

  In the present embodiment, it is normally assumed that when a person is put in the object scene and flash photography is performed, the flash light will be photographed to reach the person. When a person is detected by skin color, the face color of the person who is the main subject can be detected. In this case, when white balance processing is performed, the detected white color may be taken into consideration, and appropriate white balance adjustment may be performed according to the color.

(1) As described above, in the present embodiment, the area recognized as a face is set as the range where the flash reaches, and the white balance is accurately adjusted at the color temperature of the flash. White balance is adjusted by a simple method. Thereby, appropriate white balance processing is performed.
(2) Since the appropriate white balance is adjusted around the person, the subject with a high degree of attention is surely properly adjusted in the white balance.

  Next, a modification of the third embodiment will be described. The region dividing method may be estimated from the average luminance in each region. Taking a histogram of brightness, the average brightness is high overall in the area hit by the flash, and the average brightness is low in the background area not hit by the flash due to insufficient light. The difference in illumination light source is estimated from the average brightness in the area. Is possible. Even when the sun and shade are mixed in the image, it is possible to divide the area so that the area with high average luminance is the sun and the area with low average luminance is the shade. If only the luminance of each pixel is determined, it is difficult to determine when the reflectance of the subject is originally low or when the illumination luminance is low. Therefore, when determining by luminance, the area is divided into large areas, and the average luminance within that area is determined. It is preferable to make a judgment with

  FIG. 11 is a diagram illustrating a flowchart of a modification example of the image processing program executed by the personal computer 1 according to the third embodiment. In step S61, an image luminance distribution histogram is created from the image data. In step S62, the presence or absence of valleys is determined from the luminance distribution histogram. If there is no valley, in step S78, a single illumination color temperature is set for the entire image, white balance adjustment is performed, and the process ends.

  On the other hand, FIG. 12 is a diagram showing an example of a histogram. If there is a valley as shown in FIG. 12, the luminance value Y0 is stored in step S63. Next, in step S64, the area is divided into N × M areas, and the average luminance Ym of each area is calculated in S65. In step S65A, it is determined whether or not the average luminance Ym is larger than Y0. If the average luminance Ym is larger, in step S67, the area of illumination 1 is set as the flash illumination area or the sunny area. On the other hand, if it is smaller, the area of illumination 2 is set as the background or shade area in step S66. In step S68, it is determined whether or not the determination has been completed over the entire area. If the determination has not been completed, the process returns to step S65 to repeat the process.

  When all the areas are classified into the illumination 1 area and the illumination 2 area, the entire area determined to be the illumination 1 area in step S69 is analyzed, information on the color of the illumination 1 is obtained, and the illumination 1 is obtained. Is estimated as the color temperature T1. Next, in step S70, the entire area determined as the area of the illumination 2 is analyzed, and the color temperature T2 of the illumination 2 is estimated.

  Next, based on the determination as to whether or not each region is the illumination 1 region set in step S65, whether or not the region determined as the illumination 2 region is in a boundary portion with the illumination 1 region in step S72. Make a decision. If it is not the boundary region, white balance adjustment is performed at the color temperature T2 in step S74. On the other hand, in the case of the boundary region, in step S75, how far the subject shown in this region is from the subject in the illumination 2 region and from the subject shown in the adjacent illumination 1 region. Judge the distance is some distance, and set the color temperature between T1 and T2. The white balance is adjusted with this color temperature. The distance information is acquired in advance as in the second embodiment and stored information is used.

  Similarly, based on the determination of the illumination area performed in step S65A, it is determined whether or not the area determined as the illumination 1 area in step S71 is a boundary area in step S73. If it is determined that it is not at the boundary, white balance is corrected at T1 in step S76. If it is in the boundary region, the process of step S75 is performed as described above, and the white balance is adjusted. It is determined in step S77 whether this has been done for all areas, and the process ends.

-Fourth embodiment-
In the fourth embodiment, an example will be described in which color correction is performed with a high luminance portion in an image as a sunny region and a low luminance portion as a shade region. Since the fourth embodiment is premised on outdoor shooting, the shooting mode is acquired, and it is detected that the mode is related to the outdoors.

  The fourth embodiment is realized by the personal computer 1 as in the first embodiment. Therefore, the configuration of the image processing apparatus according to the fourth embodiment is referred to FIG. 1 of the first embodiment. FIG. 8 is a diagram illustrating a flowchart of an image processing program executed by the personal computer 1 according to the present embodiment.

  In step S31, an image (image data) taken (captured) with a digital camera or the like is acquired. In step S32, shooting information such as a shooting mode at the time of shooting is acquired. In step S33, luminance information is extracted from the acquired image. If the image data is an RGB color system, a value obtained by adding the R, G, and B values with a predetermined calculation formula or a G value is used as the luminance value.

  In step S34, if the luminance value is equal to or greater than the sunshine reference value, the region is estimated as a sunshine region, and if the luminance value is equal to or less than the shade reference value, the region is estimated as a shaded region. The hinata reference value and the shade reference value are determined in advance by calculation or experiment. Of course, the sunny reference value is set higher than the shade reference value. Based on such a reference value, if the luminance is high, it is estimated as the sun, and if the luminance is low, it is estimated as the shade. With this estimation, the image is divided into a sunny area and a shaded area. In this case, it is assumed that the shooting mode acquired in step S32 is confirmed to be the outdoor mode.

  In step S35, with respect to the divided areas, the white balance is adjusted based on the color temperature of sunlight in the sunny area, and the white balance is adjusted based on the color temperature of sunlight in the shaded area. Do.

(1) In this way, when there are a sunlit area and a shaded area in the image, white balance adjustment suitable for each color temperature of sunlight is performed. As a result, appropriate natural color reproduction is performed for the entire image.
(2) Since the sunny area and the shaded area are automatically determined, the white balance for the shade and the white balance for the shade are appropriately performed without bothering the user.

-Modification-
(1) In the first embodiment, the third embodiment, and the fourth embodiment, the example in which image processing is performed by the personal computer 1 has been described. However, as in the second embodiment, The digital camera 100 may perform the image processing. Further, the image processing performed by the digital camera 100 of the second embodiment is performed by the personal computer 1 as in the first embodiment, the third embodiment, and the fourth embodiment. You may make it perform.
(2) In the second embodiment, an example has been described in which image data corresponding to regions other than the reference numerals 14, 19, and 24 is analyzed and estimated as an incandescent lamp. However, it is also possible to estimate by the shooting mode. For example, if the shooting mode is indoor shooting, it may be estimated as an incandescent lamp. Even in indoor shooting, if the incandescent lamp or the fluorescent lamp can be specified, the specified illumination may be estimated as the illumination light source.
(3) In the first embodiment, the example in which the user divides the person and the sunset background area has been described. In the first embodiment, the user can arbitrarily designate division for any image. Therefore, the user may manually specify the sun and shade as in the fourth embodiment as in the first embodiment.
(4) In the first embodiment, the example in which the user designates the color temperature and the type of illumination for each divided region has been described. However, the color temperature of the illumination light source in each divided area may be estimated by image processing. Conversely, in another embodiment, when the color temperature of the illumination light source in the area is estimated by image processing, the user may be able to specify the color temperature and the type of illumination instead.

  Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other embodiments conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention.

It is a figure which shows the image processing apparatus which is 1st Embodiment. It is a figure which shows the flowchart of the image processing program which the personal computer 1 of 1st Embodiment performs. It is a figure which shows the structure of the digital camera 100 which is an imaging device which is 2nd Embodiment. It is a figure which shows a composition at the time of flash photography indoors. It is a figure which shows a mode that the picked-up image was divided | segmented into 5x5. It is a figure which shows the flowchart of the image processing program which the control apparatus 104 of the digital camera 100 of 2nd Embodiment performs. It is a figure which shows the flowchart of the image processing program which the personal computer 1 of 3rd Embodiment performs. It is a figure which shows the flowchart of the image processing program which the personal computer 1 of 4th Embodiment performs. It is a figure which shows the flowchart of the modification of the image processing program which the control apparatus 104 of the digital camera 100 of 2nd Embodiment performs. It is a figure explaining a mode that it sets to the color temperature interpolated between the color temperature set in the area | region 1 and the color temperature set in the area | region 2 as a boundary area. It is a figure which shows the flowchart of the modification of the image processing program which the personal computer 1 of 3rd Embodiment performs. It is a figure which shows an example of a histogram.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Personal computer 2 Digital camera 3 Recording medium 4 Computer 5 Electric communication line 100 Digital camera 101 Subject 102 Shooting lens 103 Image pick-up element 104 Control apparatus 105 Memory 106 Lens drive part 107 Flash unit

Claims (15)

An image processing method comprising:
Get one image,
The acquired one image is divided into a plurality of regions,
Set information on the color of the lighting for each of the divided areas,
An image processing method, wherein color correction is performed for each of the divided areas in accordance with information about the color of the corresponding illumination. The image processing method according to claim 1,
An image processing method, wherein the acquired image is divided into a plurality of areas according to designation by a user on an image displayed on a monitor. The image processing method according to claim 1,
An image processing method, wherein the acquired image is divided into a plurality of regions using distance information to a subject. The image processing method according to claim 1,
Recognizing a human face in the acquired image;
An image processing method, wherein the acquired image is divided into a region of the recognized person's face and other regions. The image processing method according to claim 1,
An image processing method, wherein the acquired image is divided into a main subject and a background. The image processing method according to claim 1,
An image processing method, wherein the acquired image is divided into a sunny area and a shaded area based on luminance information. The image processing method according to claim 1,
An image processing method, wherein the acquired image is divided into a skin color part and a part other than the skin color. In the image processing method in any one of Claim 1 to 7,
An image processing method comprising: setting information related to the color of the illumination by estimating each illumination light source of the plurality of divided areas. In the image processing method in any one of Claim 1 to 7,
An image processing method comprising: setting information related to the color of the illumination by designating each of the plurality of divided areas from a user. The image processing method according to claim 8.
The image processing method characterized in that the illumination light source is estimated using color information and photographing information for each of the divided areas. The image processing method according to claim 3.
Get shooting information on whether or not flash shooting was done,
When performing the flash photography, it is estimated that the illumination light source of the area in which the distance information to the subject is a predetermined value or less among the plurality of divided areas is a flash,
A white balance process suitable for a flash is performed on an area where the illumination light source is estimated to be a flash. The image processing method according to claim 4 or 7,
Get shooting information on whether or not flash shooting was done,
When performing the flash photography, it is estimated that the illumination light source of the face area of the person or the skin color part area is a flash,
A white balance process suitable for a flash is performed on an area where the illumination light source is estimated to be a flash.   An image processing program for causing a computer to execute the image processing method according to claim 1.   An image processing apparatus having the image processing program according to claim 13 mounted thereon.   An imaging apparatus equipped with the image processing program according to claim 13.

Images