JP5459178B2 - Imaging apparatus, white balance adjustment method, and white balance adjustment program - Google Patents

Description

  The present invention relates to an imaging apparatus, a white balance adjustment method, and a white balance adjustment program, and more particularly to a technique for improving the color of an entire image taken at the time of flash emission.

Conventionally, various white balance adjustment methods have been devised in order to correct an unnatural white balance caused by a difference in color temperature at the time of flash light emission so that it becomes more natural.
As such a white balance adjustment method, for example, an image when the flash is fired and an image when the flash is not fired are divided into a plurality of regions, and the image is divided based on the luminance difference between the corresponding regions. A method is known in which white balance is set for each region (see Patent Document 1).

JP-A-8-51632

  However, when the method described in Patent Document 1 described above is applied, since the white balance is set independently for each of the divided areas, the color of the entire image may be unnatural. .

In particular, in the above method, the white balance is uniformly set in the area determined not to be irradiated with the strobe. In other words, the white balance is set based on the idea that there is no need to consider any area other than the area determined not to be irradiated with the strobe in setting the white balance.
However, as a practical problem, even in a region where it is determined that the strobe is not irradiated at all, it is often affected by some strobe. Therefore, even if it is determined that the strobe is not irradiated, if the white balance is not set in consideration of the strobe irradiation, the color of the entire image becomes unnatural as a result.

  The present invention has been made in view of such a situation, and an object of the present invention is to improve the color of the entire image taken at the time of flash emission.

In order to achieve the above-mentioned object, the invention described in claim 1 includes an imaging unit, a light emitting unit, a first image that is brightened by light emission by the light emitting unit, and the first image. An imaging control unit that controls the imaging unit to capture a second image with low brightness, and a white balance that is set when the first image and the second image are captured. A first acquisition unit that acquires a gain value of each color component, a dividing unit that divides an image captured by the imaging unit into a plurality of regions, the first image, and the second image, second acquisition means for respectively acquiring the luminance values of the plurality of regions, for each of the plurality of regions, the luminance value of the region of the second said calculated by obtaining means of the first image and the first the second corresponding to a region of the image A calculating means for calculating a relative value of the luminance value of the region of the image, from among a plurality of relative values calculated by the calculation means, selection means for selecting a predetermined number of relative values, said first obtaining means on the basis of the predetermined number of relative values selected by the selecting means and the gain value of each color component of the acquired first image and the second image by, at least a partial area of the first image It is an imaging device provided with the correction | amendment means which correct | amends the gain value of each color component of this.
In order to achieve the above object, the invention according to claim 2 is characterized in that, in the first image, the correction means is a gain of each color component in a region where the relative value is greater than or equal to a predetermined value or less. It is characterized by correcting the value.
In order to achieve the above object, the invention according to claim 3 is characterized in that the selection unit includes the first image of which the relative value calculated by the calculation unit is greater than or equal to a predetermined value or less. A specifying means for specifying an area is provided, and a predetermined number of relative values greater than or less than a predetermined value specified by the specifying means are selected.
In order to achieve the above object, according to a fourth aspect of the present invention, the calculation means includes: the second image corresponding to a region of the first image having a luminance value of the region of the first image. The value divided by the luminance value of the area is calculated as the relative value, and the specifying unit calculates a predetermined number of relative values from the relative value calculated by the calculating unit in order of increasing value or decreasing value. It is characterized by specifying.

In order to achieve the above object, according to a fifth aspect of the present invention, each color component of the image acquired by the first acquisition unit is converted into a set of pixel parameters including at least luminance information in another color space. The image processing apparatus further includes conversion means for converting, and the correction means is an imaging apparatus that corrects the gain value of each color component based on a set of pixel parameters converted by the conversion means.

In order to achieve the above object, the invention according to claim 6 is a first image that is an image when brightened by light emission, and a second image that is lower in brightness than the first image. An imaging control step for controlling the imaging means to capture an image, and a gain value for each color component for adjusting a white balance set at the time of capturing the first image and the second image. and one of the acquisition step, a division step of dividing an image obtained by the processing of the first acquisition step into a plurality of regions, for the first image and the second image, the luminance of the plurality of regions a second acquisition step of acquiring each a value, for each of the plurality of regions, a region of the luminance value and the first image region of the second said acquired in the step of obtaining the first image corresponding second image , Selection steps and, the first acquisition step of selecting a predetermined number of relative value among the calculation step, a plurality of relative values calculated in the calculation step of calculating a relative value of the luminance values of the regions on the basis of the predetermined number of relative values selected by said selecting step between the gain value of each color component of the acquired first image and the second image by, at least a portion of said first image And a white balance adjustment method including a correction step of correcting the gain value of each color component in the region.

In order to achieve the above object, according to a seventh aspect of the present invention, there is provided a first image that is an image when the computer is brightened by light emission, and a second image that is lower in brightness than the first image . Imaging control means for causing the imaging means to image the first image, first acquisition means for acquiring gain values of the respective color components for adjusting white balance of the first image and the second image, and the imaging means dividing means for dividing an image captured in a plurality of regions, the above for the first image and the second image, the second acquisition unit configured to respectively obtain the luminance values of the plurality of regions, said plurality of areas for each of the relative value of the luminance value of the region of the second image corresponding to a second region of the luminance value and the first image region of the acquired by the acquisition means and the first image each calculation means for calculating, the calculating From among the plurality of relative values calculated by the step, selecting means for selecting a predetermined number of relative values, the gain of each color component of the first said acquired by the acquisition means in the first image and the second image based on the value and a predetermined number of relative values selected by the selecting means is the white balance adjusting program which functions as a correction means for correcting the gain value of each color component of at least a partial area of the first image It is characterized by that.

  ADVANTAGE OF THE INVENTION According to this invention, the color of the whole image can be improved about the image imaged at the time of flash emission.

It is an area figure showing the composition of the hardware in one embodiment of the imaging device concerning the present invention. 3 is a flowchart for explaining a flow of strobe imaging processing executed by an imaging apparatus having the hardware configuration of FIG. 1. It is a schematic diagram which shows the state from which the division area of 8x8 64 was obtained from the through image at the time of non-light emission or the captured image at the time of light emission. It is a figure which shows an example of the table which stores the luminance ratio of each division area of the through image at the time of light emission, and the captured image at the time of non-light emission. 2 is a flowchart illustrating a flow of white balance processing executed by an imaging apparatus having the hardware configuration of FIG. 1.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a hardware configuration in an embodiment of an imaging apparatus 1 according to the present invention.
The imaging apparatus 1 shown in FIG. 1 can be configured by a digital camera, for example.

  The imaging apparatus 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an image processing unit 14, an image dividing unit 15, and a white balance gain calculating unit 16. The luminance acquisition unit 17, the bus 18, the input / output interface 19, the imaging unit 20, the light emitting unit 21, the operation unit 22, the display unit 23, the storage unit 24, the communication unit 25, and the drive 26. It is equipped with.

  The CPU 11 executes various processes according to a program recorded in the ROM 12 or a program loaded from the storage unit 24 to the RAM 13.

  The RAM 13 appropriately stores data necessary for the CPU 11 to execute various processes.

The image processing unit 14 is configured by a DSP (Digital Signal Processor), a VRAM (Video Random Access Memory), and the like, and performs various image processing on image data in cooperation with the CPU 11.
For example, the image processing unit 14 performs image processing such as noise reduction, white balance adjustment, and camera shake correction on captured image data output from the imaging unit 20 described later.

  The image dividing unit 15 divides image data used for white balance adjustment in the image processing performed in the image processing unit 14 into data of several regions in the spatial direction. Further details of the image dividing unit 15 will be described later.

  The white balance gain calculation unit 16 calculates a white balance gain used for white balance adjustment in the image processing performed in the image processing unit 14. Further details of the white balance gain calculator 16 will be described later.

  The luminance acquisition unit 17 acquires a luminance value or the like from image data used for white balance adjustment in the image processing performed in the image processing unit 14. Further details of the luminance acquisition unit 17 will be described later including the luminance comparison unit 41.

  The CPU 11, ROM 12, RAM 13, image processing unit 14, image division unit 15, white balance gain calculation unit 16, and luminance acquisition unit 17 are connected to each other via a bus 18. An input / output interface 19 is also connected to the bus 18. An imaging unit 20, a light emitting unit 21, an operation unit 22, a display unit 23, a storage unit 24, a communication unit 25, and a drive 26 are connected to the input / output interface 19.

  Although not shown, the imaging unit 20 includes an optical lens unit and an image sensor.

The optical lens unit is configured by a lens that collects light, for example, a focus lens or a zoom lens, in order to photograph a subject.
The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. The zoom lens is a lens that freely changes the focal length within a certain range.
The optical lens unit is also provided with a peripheral circuit for adjusting setting parameters such as focus, exposure, and white balance as necessary.

The image sensor includes a photoelectric conversion element, AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element or the like. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element photoelectrically converts (captures) the subject image, accumulates the image signal for a predetermined time, and sequentially supplies the accumulated image signal as an analog signal to the AFE.
The AFE performs various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. By various signal processing, a digital signal is generated and output as an output signal of the imaging unit 20.
Hereinafter, the output signal of the imaging unit 20 is referred to as “captured image data”. Accordingly, captured image data is output from the imaging unit 20 and is appropriately supplied to the CPU 11, the image processing unit 14, the image dividing unit 15, the white balance gain calculating unit 16, and the like.

  The light emitting unit 21 has a strobe, and causes the strobe to emit light according to the control of the CPU 11. In the present embodiment, the strobe light is triggered when the user performs an operation of recording an image to be recorded on the operation unit 22, for example, pressing a release button (not shown) of the operation unit 22.

  The operation unit 22 includes, for example, various buttons such as a release button (not shown), and receives a user instruction operation.

  The display unit 23 includes a display that can display various images.

  The storage unit 24 is configured by a DRAM (Dynamic Random Access Memory) or the like, and stores data of various images such as a through image described later, an original image to be displayed, and an image that can be combined with the original image.

  The communication unit 25 controls communication performed with another device (not shown) via a network including the Internet.

  A removable medium 31 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 26. The program read from the removable medium 31 by the drive 26 is installed in the storage unit 24 as necessary. The removable medium 31 can also store various data such as image data stored in the storage unit 24 in the same manner as the storage unit 24.

The hardware configuration of the imaging apparatus 1 according to the present embodiment has been described above with reference to FIG.
Next, the flow of strobe imaging processing executed by the imaging apparatus 1 having such a hardware configuration will be described. The strobe imaging process is a series of processes from shooting the subject by emitting a strobe and adjusting the white balance to the captured image data obtained as a result, and then recording it on the removable media 31 or the like. Say.
FIG. 2 is a flowchart for explaining the flow of strobe imaging processing executed by the imaging apparatus 1 having the hardware configuration of FIG.

First, as an operation mode of the imaging apparatus 1, in the present embodiment, it is assumed that a normal mode in which a subject is imaged without emitting a strobe light and a strobe mode in which the strobe light is emitted and a subject is imaged are provided. The user can selectively instruct the normal mode or the strobe mode as the operation mode by performing a predetermined operation on the operation unit 22.
In this case, when the selection of the strobe mode is instructed, the strobe imaging process starts.

In step S1, the CPU 11 executes through imaging processing and through display processing.
That is, the CPU 11 controls the imaging unit 20 and the image processing unit 14 to continue the imaging operation by the imaging unit 20. Then, while the imaging operation by the imaging unit 20 is continued, the CPU 11 temporarily stores the captured image data sequentially output from the imaging unit 20 in the memory (the storage unit 24 in the present embodiment). Such a series of processes is the “through imaging process” referred to herein.
Further, the CPU 11 sequentially reads out each data temporarily recorded in the memory (the storage unit 24 in the present embodiment) during the through imaging process, and causes the display unit 23 to sequentially display the corresponding captured images. Such a series of processes is the “through display process” here. The captured image displayed on the display unit 23 by the through display process is hereinafter referred to as a “through image”.

In step S <b> 2, the CPU 11 determines whether there is an instruction to record a captured image.
In the present embodiment, as described above, the user can issue an instruction to record a captured image by pressing the release button of the operation unit 22.
Therefore, if the release button has not been pressed, it is determined NO in step S2, and the process returns to step S1. That is, until the release button is pressed, the loop processing in steps S1 and S2 is repeatedly executed, so that the through imaging process and the through display process are repeatedly executed, and the through image of the subject is displayed in real time on the display unit 23. Will continue to be displayed.
Although not shown, if the release button is not pressed even after a predetermined time has elapsed, the CPU 11 or the like may forcibly terminate the strobe imaging process.

  Thereafter, when the release button is pressed, it is determined as YES in Step S2, and the process proceeds to Step S3.

In step S <b> 3, the CPU 11 controls imaging of the subject together with light emission. Specifically, the CPU 11 controls the light emitting unit 21 to emit a strobe light, and controls the imaging unit 20 to image the subject.
In this embodiment, the captured image data output from the imaging unit 20 at this time is temporarily stored in the storage unit 24 as data to be recorded.

In step S4, the CPU 11 uses the data of the through image in which the subject is imaged when the flash is not emitted in the through imaging process of step S1 and the data of the captured image in which the subject is imaged when the flash is emitted in the process of step S3. Then, processing for adjusting the white balance of the captured image to be recorded is executed.
Such processing in step S4 is hereinafter referred to as “white balance processing” in accordance with the description in FIG. Further, the data of the through image in which the subject is imaged when the flash is not emitted in the through imaging process in step S1 is hereinafter referred to as “data of the through image when no light is emitted”. On the other hand, the data of the captured image in which the subject is imaged at the time of flash emission in the process of step S3 is hereinafter referred to as “data of the captured image at the time of light emission”.
Here, in the present embodiment, the data of the captured image to be recorded is adopted as the data of the captured image to be recorded. However, separately from the captured image data at the time of light emission, after the white balance is set, the captured image data obtained as a result of the image capturing unit 20 capturing the subject again by emitting the strobe light may be adopted as the recording target. . In this case, the data of the captured image to be recorded is adjusted with the set white balance.
Further details of the white balance process will be described later.

In step S <b> 5, the CPU 11 records the captured image data to be recorded, which has been subjected to the white balance process in step S <b> 4, on the removable medium 31.
Thereby, the strobe imaging process is completed.

The strobe imaging process has been described above.
Next, the white balance process executed in step S4 in the strobe imaging process will be described.
Here, first, a functional configuration for executing the white balance processing will be described, and then a flow of white balance processing executed based on the functional configuration will be described.

  When the white balance process is executed, the image dividing unit 15, the white balance gain calculating unit 16, and the luminance acquiring unit 17 of the imaging device 1 in FIG. 1 function.

The image dividing unit 15 divides each data of the non-light-emitting through image and the light-emitting captured image into a plurality of areas, in this embodiment, 8 × 8 64 areas as shown in FIG. To do.
In the present specification, the region divided by the image dividing unit 15 in this way is hereinafter referred to as “divided region”.
FIG. 3 is a schematic diagram showing a state in which 64 divided regions of 8 × 8 are obtained from the through image at the time of non-light emission or the captured image at the time of light emission.
As shown in the figure, in this embodiment, a unique number that uniquely identifies each divided region, specifically, from the divided region at the upper left corner, the horizontal direction is the right direction, and the vertical direction is the downward direction. The numbers 1, 2, 3, 4, 5, 6, 7, 8, 9, 10,. Hereinafter, the numbers assigned to the divided areas are referred to as “division numbers”.
Since the division numbering method is unified for the non-light-emitting through image and the light-emitting captured image, the divided areas of the same number of the non-light-emitting through image and the light-emitting captured image are the same in the entire image, The position, size, and range are the same.
Note that the data of each divided area is stored and managed in the storage unit in a table format in association with, for example, a number assigned thereto.

The white balance gain calculation unit 16 calculates a white balance gain for the non-light-emitting through image and a white balance gain for the light-emitting captured image for each of the divided regions divided by the image dividing unit 15.
Specifically, in the present embodiment, it is assumed that each data of the through image at the time of non-light emission and the captured image at the time of light emission is composed of RGB (R: Red, G: Green, B: Blue) components.
In this case, the white balance gain calculation unit 16 uses an R component gain (hereinafter referred to as “SRG”) and a G component gain (hereinafter referred to as “SGG”) as the white balance gain for each divided region of the non-light-emitting through image. And the gain of the B component (hereinafter referred to as “SBG”). Hereinafter, SRG, SGG, and SBG are collectively referred to as “gain values of RGB components of the through image during non-light emission”. That is, the gain value of the RGB component for each divided region is calculated for the non-light-emitting through image.
Further, the white balance gain calculation unit 16 uses a gain of an R component (hereinafter referred to as “LRG”) and a gain of a G component (hereinafter referred to as “LGG”) as the white balance gain for each divided region of the captured image during light emission. And the gain of the B component (hereinafter referred to as LBG). Hereinafter, SRG, SGG, and SBG are collectively referred to as “gain values of RGB components of the captured image during light emission”. That is, the gain value of the RGB component for each divided region is calculated for the captured image during light emission.

Next, the white balance gain calculation unit 16 corrects the RGB component gain values of the captured image during light emission for each of the plurality of divided regions.
Here, in the present embodiment, the plurality of divided areas are divided areas (hereinafter referred to as “irradiated areas”) that are presumed that the strobe light is firmly irradiated, or other divided areas (hereinafter referred to as “irradiated areas”). , Called “non-irradiated region”). In addition, the classification method of an irradiation area | region and a non-irradiation area | region is mentioned later.
For this reason, in this embodiment, the correction method of the gain value of the RGB component of the captured image at the time of light emission differs between the irradiation region and the non-irradiation region.

Therefore, a method for correcting the gain value of the RGB component of the non-irradiated area in the captured image during light emission will be described below.
Even in the non-irradiated region, it is rare that no strobe light is irradiated, and it is necessary to adjust the white balance in consideration of the degree of strobe light irradiation. In order to enable such adjustment, the gain value of the RGB component in the non-irradiated region in the captured image during light emission is corrected as follows.

The white balance gain calculation unit 16 calculates the gain values of the RGB components of the non-irradiated areas of the non-light-emitting through image and the light-emitting captured image as YUV (Y: luminance, U: color difference between luminance and blue components, V: luminance). Converts to the red component color difference component gain value.
Hereinafter, the YUV component gain values converted from the RGB component gain values of the non-light-emitting through image and the light-emitting captured image are referred to as “YUV converted values”.

なお、式（１）の左辺の左から掛けられている３行３列の行列は、即ち、ｉ行ｊ列（ｉ，ｊは、相互に独立した、１乃至３の範囲内の整数値）の要素がａｉｊとなっている行列は、ＲＧＢ成分をＹＵＶ成分に変換する変換行列である。 Here, the YUV conversion value of the non-irradiation region of the through image during non-light emission includes a Y component gain (hereinafter referred to as SY), a U component gain (hereinafter referred to as SU), and a V component gain (hereinafter referred to as SU). , Referred to as SV).
In this case, the YUV conversion value of the through image when not emitting light is calculated from the following equation (1).

Note that the matrix of 3 rows and 3 columns multiplied from the left of the left side of Equation (1) is i rows and j columns (i and j are integer values in the range of 1 to 3 that are independent of each other). The matrix whose element is aij is a conversion matrix for converting RGB components into YUV components.

On the other hand, the YUV conversion value of the non-irradiated region of the captured image at the time of light emission includes a Y component gain (hereinafter referred to as LY), a U component gain (hereinafter referred to as LU), and a V component gain (hereinafter referred to as LV). Called).
In this case, the YUV conversion value of the non-irradiated region of the live view through image is calculated from the following equation (2).

式（３）において、Ｃは、非発光時スルー画像と発光時撮像画像との各非照射領域の輝度比率を総合的に考慮した重み付けを行うための可変係数であって、後述する輝度比較部４１で演算される平均輝度比率である。 Next, the white balance gain calculation unit 16 comprehensively considers the luminance ratio of each non-irradiated region between the non-light-emitting through image and the light-emitting captured image (the luminance ratio will be described later). Of the YUV conversion values in the non-irradiated region, the Y component gain LY is corrected.
Here, among the YUV conversion values of the non-irradiated area of the captured image during light emission, the corrected Y component gain is hereinafter referred to as “LY ′”. In other words, LY ′ is a value obtained by weighting LY in consideration of the luminance ratio of each non-irradiated area between the non-light-emitting through image and the light-emitting captured image.
For example, LY ′ is obtained by the following equation (3).

In Equation (3), C is a variable coefficient for weighting that comprehensively considers the luminance ratio of each non-irradiated region between the non-light-emitting through image and the light-emitting captured image, and is a luminance comparison unit described later 41 is an average luminance ratio calculated by 41.

なお、式（４）の右辺の列ベクトルが、逆変換後の発光時撮像画像の非照射領域のＲＧＢ成分のゲイン値を示している。逆変換後の発光時撮像画像の非照射領域のＲＧＢ成分のゲイン値は、逆変換後のＲ成分のゲイン（以下、式（４）の記載に併せて「ＬＲα」と呼ぶ）と、逆変換後のＧ成分のゲイン（以下、式（４）の記載に併せて「ＬＧα」と呼ぶ）と、逆変換後のＢ成分のゲイン（以下、式（４）の記載に併せて「ＬＢα」と呼ぶ）と、から構成される。
また、式（４）の左辺において、左から掛けられている行列は、式（１）や式（２）で用いられている変換行列の逆行列である。 Next, the white balance gain calculation unit 16 inversely converts the YUV conversion value of the non-irradiation region of the captured image at the time of emission in which the Y component is corrected (weighted) by Expression (3) into the gain value of the RGB component. Specifically, the white balance gain calculation unit 16 obtains the gain value of the RGB component of the non-irradiation region of the captured image at the time of light emission after inverse conversion according to the following equation (4).

Note that the column vector on the right side of Equation (4) indicates the gain value of the RGB component of the non-irradiated region of the captured image during light emission after inverse transformation. The RGB component gain values of the non-irradiated region of the captured image at the time of light emission after inverse conversion are the inverse R component gain (hereinafter referred to as “LRα” in conjunction with the description of equation (4)) and inverse conversion. The gain of the later G component (hereinafter referred to as “LGα” in conjunction with the description of equation (4)) and the gain of the B component after inverse transformation (hereinafter referred to as “LBα” in conjunction with the description of equation (4)) Call).
On the left side of Equation (4), the matrix multiplied from the left is an inverse matrix of the transformation matrix used in Equation (1) or Equation (2).

  In this way, the gain value of the RGB component of the non-irradiation area in the captured image during light emission is corrected.

In the above, the correction method of the gain value of the RGB component of a non-irradiation area among the captured images during light emission has been described.
In addition, the correction method of the gain value of the RGB component of the irradiation region in the captured image at the time of emission is not particularly limited. For example, in the present embodiment, a correction method based on a luminance ratio described later is adopted. .

  Then, the white balance gain calculation unit 16 sets the white balance of the captured image to be recorded for each divided region based on the gain value after correction of the RGB components for each divided region of the captured image during light emission.

As described above, the processing of the white balance gain calculation unit 16 requires a luminance ratio for each divided region. In order to perform such a luminance ratio calculation, a luminance acquisition unit 17 is provided.
That is, the luminance acquisition unit 17 acquires a luminance value for each divided area from the data of the non-light-emitting through image and the light-emitting captured image.
Here, since the divided area is composed of a plurality of pixels, the luminance of the divided area is a value calculated based on the luminance of each pixel constituting the divided area, for example, an average value of the luminance of each pixel. Shall mean.

式（５）において、Ｙｉ’は、発光時撮像画像のｉ番目の分割領域の輝度を示している。Ｙｉは、非発光時スルー画像のｉ番目の分割領域の輝度を示している。
本実施形態では、このような式（５）による輝度比較部４１の演算結果は、例えば図４に示すテーブル形式で記憶部２４に記憶されて、管理される。 The luminance acquisition unit 17 is provided with a luminance comparison unit 41.
The luminance comparison unit 41 calculates the luminance ratio of each divided region of the through image during light emission and the captured image during non-light emission.
The luminance ratio Pi of the region number i (i is a positive integer value equal to or less than the total number of divisions, and in the present embodiment, a positive integer value equal to or less than 64) is obtained by the following equation (5).

In Expression (5), Yi ′ represents the luminance of the i-th divided area of the captured image during light emission. Yi represents the luminance of the i-th divided area of the through image when no light is emitted.
In this embodiment, the calculation result of the brightness | luminance comparison part 41 by such Formula (5) is memorize | stored in the memory | storage part 24, for example in the table format shown in FIG. 4, and is managed.

FIG. 4 shows an example of a table that stores the luminance ratio of each divided region of the through image during light emission and the captured image during non-light emission.
In the present embodiment, since the table of FIG. 4 has a matrix structure, the collection of items in the horizontal direction in FIG. 4 is hereinafter referred to as “row”, and the collection of items in the vertical direction in FIG. Referred to as a column. A predetermined area number is associated with the predetermined line. In other words, items of “area number”, “brightness at non-light emission”, “brightness at light emission”, and “brightness ratio” for the area number corresponding to the row are arranged in a predetermined line, respectively. Yes.
The i-th line from the top (however, the top line in which the item names in FIG. 4 are described is excluded. The same applies hereinafter). The “region number” stores the region number i.
In the “illuminance at the time of non-light emission” on the i-th row from the top, the luminance Yi of the i-th divided area of the through image at the time of non-light emission is stored.
In the “iluminance at the time of light emission” on the i-th row from the top, the luminance Yi ′ of the i-th divided region of the captured image at the time of light emission is stored.
In the “brightness ratio” in the i-th row from the top, the i-th brightness ratio Pi, that is, the calculation result of Expression (5) is stored.

  In the present embodiment, each divided area is classified into a non-irradiated area or an irradiated area based on the luminance ratio for each divided area thus obtained. Specifically, for example, in this embodiment, an appropriate value as a minimum value is preset as a threshold value among the luminance ratios when the strobe light is shining firmly. In this case, if the luminance ratio Pi is equal to or smaller than the threshold value, the i-th divided area is classified as a non-irradiated area, while if the luminance ratio Pi exceeds the threshold value, the i-th divided area is classified as an irradiated area. The

式（６）において、Ｃｔ２乃至Ｃｔ４の各々は、下位２乃至４番目の各々の輝度比率を示している。
例えば、全分割領域の輝度比率（Ｃ１＝Ｙ１’／Ｙ１，Ｃ２＝Ｙ２’／Ｙ２，Ｃ３＝Ｙ３’／Ｙ３，・・・，Ｃ８＝Ｙ８’／Ｙ８，・・・，Ｃ２２＝Ｙ２２’／Ｙ２２，・・・，Ｃ６４＝Ｙ６４’／Ｙ６４）のうち、下位４番目までの輝度比率が次の通りであったとする。即ち、６４番目（下から１番目）：Ｃｔ１＝Ｃ１＝Ｙ１’／Ｙ１、６３番目（下から２番目）：Ｃｔ２＝Ｃ２＝Ｙ２’／Ｙ２、６２番目（下から３番目）：Ｃｔ３＝Ｃ８＝Ｙ８’／Ｙ８、６１番目（下から４番目）：Ｃｔ４＝Ｃ２２＝Ｙ２２’／Ｙ２２であったものとする、この場合、平均輝度比率Ｃは、次の式（７）により算出される。

このようにして求められた平均輝度比率Ｃは、上述した式（３）に係数として代入されて用いられる。 Next, the luminance comparison unit 41 sorts the luminance ratios for all the divided regions in ascending order of ratio.
The luminance comparison unit 41 obtains the average value of the lower to second to fourth luminance ratios as the above-described average luminance ratio C among the luminance ratios of the divided areas sorted in the order of decreasing luminance.
Specifically, the luminance comparison unit 41 calculates the average luminance ratio C by calculating the following equation (6).

In Expression (6), Ct2 to Ct4 indicate the luminance ratios of the second to fourth lower levels.
For example, the luminance ratio (C1 = Y1 ′ / Y1, C2 = Y2 ′ / Y2, C3 = Y3 ′ / Y3,..., C8 = Y8 ′ / Y8,..., C22 = Y22 ′ / Y22,..., C64 = Y64 ′ / Y64), the luminance ratio up to the fourth lower order is as follows. That is, 64th (first from the bottom): Ct1 = C1 = Y1 ′ / Y1, 63rd (second from the bottom): Ct2 = C2 = Y2 ′ / Y2, 62nd (third from the bottom): Ct3 = C8 = Y8 ′ / Y8, 61st (fourth from the bottom): Assume that Ct4 = C22 = Y22 ′ / Y22. In this case, the average luminance ratio C is calculated by the following equation (7).

The average luminance ratio C obtained in this way is used by being substituted as a coefficient in the above-described equation (3).

  The functional configuration for executing the white balance process in step S4 in FIG. 2 among the functional configurations of the imaging apparatus 1 in FIG. 1 according to the present embodiment has been described above with reference to FIGS. 3 and 4.

Next, a detailed flow of the white balance process in step S4 executed by the imaging apparatus 1 having such a functional configuration will be described.
In the white balance process, under the control of the CPU 11, any one of the white balance gain calculation unit 16 to the luminance acquisition unit 17 executes the process of each step. However, in the following explanation, explanation about control of CPU11 is omitted.
FIG. 5 is a flowchart for explaining the details of the flow of the white balance process in step S4 in the strobe imaging process of FIG. 2 executed by the imaging apparatus 1 of FIG.

  In step S21, the image dividing unit 15 divides each data of the non-light-emitting through image and the light-emitting captured image into 8 × 8 divided regions, and the luminance acquisition unit 17 acquires the luminance value of each divided region. .

In step S <b> 22, the luminance comparison unit 41 calculates a luminance ratio between the luminance value of the through image when not emitting light and the luminance value of the captured image when emitting light for each divided region.
Specifically, the luminance comparison unit 41 calculates the luminance ratio Pi for the i-th divided region by calculating the above-described equation (5). Such an operation of Expression (5) is executed for each of Nos. 1 to 64, and the luminance ratios P1 to P64 are calculated.
In this embodiment, the calculation result of the brightness | luminance comparison part 41 by such Formula (5) is memorize | stored for every division area in the memory | storage part, for example in the table format shown in FIG. 4, and is managed.

  In step S23, the white balance gain calculation unit 16 sets the gain value of the RGB component for each divided region for each data of the non-light-emitting through image and the light-emitting captured image.

  In step S24, the white balance gain calculation unit 16 sets “1” as the region number i to be processed (i = 1). That is, in the present embodiment, the divided areas to be processed in steps S25 to S30 described later are sequentially set in the order of the divided numbers. Therefore, “1” is first set as the division number i to be processed.

  In step S25, the white balance gain calculation unit 16 determines whether or not the luminance ratio Pi exceeds the threshold (Pi> threshold).

The case where the luminance ratio Pi is higher than the threshold means a case where the divided area of the area number i to be processed is an irradiation area as described above. In such a case, it is determined as YES in the process of step S25, and the process proceeds to step S26.
In step S <b> 26, the white balance gain calculation unit 16 corrects the gain value of the RGB component of the divided area (irradiation area) with the area number i based on the luminance ratio Pi.
Thereby, a process progresses to step S31. However, the processing after step S31 will be described later.

On the other hand, the case where the luminance ratio Pi is equal to or smaller than the threshold means a case where the divided region of the region number i to be processed is a non-irradiation region as described above. In such a case, it is determined as NO in the process of step S25, and the process proceeds to step S27.
Then, the following processing of steps S27 to S30 is executed to correct the RGB component gain values of the divided region of region number i.

That is, in step S27, the luminance comparing unit 41 calculates the average luminance ratio C based on the second to fourth luminance ratios among the total luminance ratios P1 to P64 calculated in the process of step S22.
Specifically, the average luminance ratio C is calculated according to the above-described equation (6).
Note that the process of step S27 need only be executed once after it is determined NO in the process of step S25 for the first time, and may be omitted thereafter.

In step S28, the white balance gain calculation unit 16 converts the gain value of the RGB component of the divided area (irradiation area) of number i into the YUV conversion value among the gain values set in the process of step S23.
Specifically, the white balance gain calculation unit 16 obtains the YUV conversion value of the divided area (non-irradiation area) having the area number i according to the above-described equation (1).

In step S29, the white balance gain calculation unit 16 determines, based on the average luminance ratio C obtained in the process of step S27, among the YUV converted values of the divided area (non-irradiation area) of the area number i of the captured image during emission. A weighted Y component value is calculated by correcting the Y component value.
Specifically, the white balance gain calculation unit 16 uses the average luminance ratio C calculated in the process of step S24 as a coefficient, and the divided area (non-irradiation area) of the area number i of the captured image at the time of emission according to the above equation (3). ) Y component gain LY is corrected to gain LY ′.
In this way, among the divided regions of the non-light-emitting through image and the light-emitting captured image, the luminance ratio of the non-irradiated region is comprehensively considered and the divided region (i.e., non-irradiated) The Y component gain LY is corrected among the YUV conversion values in the region), and the corrected Y component gain LY ′ is obtained.

In step S <b> 30, the white balance gain calculation unit 16 inversely converts the weighted YUV conversion value for the division area (non-irradiation area) with the area number i into the RGB component gain values.
Specifically, the white balance gain calculation unit 16 obtains a gain value after inverse conversion of the RGB components in the divided area (non-irradiation area) area of the area number i of the captured image at the time of emission according to the above-described equation (4).
That is, the white balance gain calculation unit 16 calculates the YUV converted value (LY ′, LU, LV) of the divided area (non-irradiation area) of the number i of the captured image at the time of emission, to which weighting is added in the process of step S29. The R component gain (LRα), the G component gain (LGα), and the B component gain (LBα) are respectively inversely transformed.

In this way, the gain value of the RGB component of the divided area of area number i of the captured image at the time of light emission is a step when the divided area is an irradiation area (when it is determined YES in the process of step S25). If the divided area is a non-irradiated area by the process of S26 (if it is determined NO in the process of step S25), the process is corrected by the processes of steps S27 to S30.
Thereby, a process progresses to step S31.

  In step S31, the white balance gain calculation unit 16 increments the region number i by 1 (i = i + 1).

In step S <b> 32, the white balance gain calculation unit 16 determines whether or not the region number i exceeds 64.
If the area number i does not exceed 64, that is, if there is a divided area in which the gain value of the RGB component of the captured image during light emission is not corrected, it is determined NO in step S32, and the process proceeds to step S25. And the subsequent processing is repeated.
That is, the loop process of steps S25 to S32 is performed for each of the divided areas of area numbers 1 to 64 each time. In this case, the gain value of the RGB component of the captured image during light emission is corrected by the process of step S26 for the divided area classified as the irradiation area (the divided area determined as YES in the process of step S25). . On the other hand, for the divided regions classified as non-irradiated regions (divided regions determined to be NO in the process of step S25), the gains of the RGB components of the captured image during light emission are processed by the processes of steps S27 to S30. The value is corrected.
When the process of step S31 is executed for the last divided area, that is, the divided area having the area number 64, the area number i becomes 65 and exceeds 64. Therefore, it is determined as YES in the next step S32, and the process proceeds to step S33.

In step S33, the white balance gain calculation unit 16 performs the processing in steps S27 to S30 for the non-irradiation region based on the gain value of the RGB component corrected in step S26 for the irradiation region for each divided region. Based on the corrected RGB component gain values, the white balance of the captured image during light emission is set.
As a result, the white balance process ends. That is, the process of step S4 in FIG. 2 ends, and the process proceeds to step S5.

  As described above, the imaging device 1 includes the light emitting unit 21, the imaging unit 20, the CPU 11, the white balance gain calculating unit 16, the image dividing unit 15, the luminance calculating unit, and the luminance comparing unit 41. Prepare.

The CPU 11 captures, by a user's operation, a captured image at the time of light emission that is brightened by light emission by the light emitting unit 21 and a through image at the time of non-light emission that is an image when light emission by the light emitting unit 21 is not performed. The unit 20 is controlled so as to capture an image.
The white balance gain calculation unit 16 calculates the gain value of each color component for adjusting the white balance of the captured image during light emission and the through image during non-light emission set at the time of imaging.
The image dividing unit 15 divides the imaging area captured by the imaging unit 20 into a plurality of areas.
The luminance acquisition unit 17 calculates the luminance values of the plurality of regions divided by the image dividing unit 15 for the captured image at the time of light emission and the through image at the time of non-light emission of the regions divided by the image dividing unit 15, respectively.
The luminance comparison unit 41 calculates, as relative values, values obtained by dividing the luminance value of the captured image area during emission calculated by the luminance acquisition unit 17 by the luminance value of the through image area during non-emission.
The white balance gain calculation unit 16 identifies regions where the relative values calculated by the luminance comparison unit 41 are equal to or less than a predetermined value.
The luminance comparison unit 41 selects a plurality of relative values having priority from among the calculated relative values.
The white balance gain calculation unit 16 determines each color of the captured image during light emission based on the calculated values of the color components of the captured image during emission and the through image during non-emission and the plurality of relative values selected by the luminance comparison unit 41. Correct the gain value of the component.

  In the imaging apparatus 1 configured as described above, natural color reproducibility of an image can be improved in an image captured when the strobe emits light.

Further, the white balance gain calculation unit 16 converts the RGB components of the acquired captured image at the time of emission and the through image at the time of non-light emission into a set of pixel parameters (YUV conversion value) including at least luminance information which is another color space. To do.
Further, the white balance gain calculation unit 16 further corrects the RGB component gain values based on the converted pixel parameter set (YUV conversion value).

  In the imaging apparatus 1 configured as described above, the luminance for each color component of the image captured at the time of flash emission is individually set in order to adjust the luminance at the gray balance level by converting the YUV conversion value. Therefore, a specific color change that may occur in the case of adjustment is not generated, and the natural color reproducibility of the image can be further improved.

  In addition, this invention is not limited to the above-mentioned embodiment, The deformation | transformation in the range which can achieve the objective of this invention, improvement, etc. are included in this invention.

In the above-described embodiment, the luminance ratio Pi is obtained by the above-described equation (5), but is not limited thereto. For example, the luminance ratio Pi can be obtained by the following equation (8). In this case, as the average luminance ratio C, the second to fourth luminance ratios Pi are used.

  In the above-described embodiment, the average luminance ratio is the second to fourth luminance ratio, but the present invention is not limited to this. The luminance ratio used for the average luminance ratio only needs to be a relatively low value at least in the entire luminance ratio.

  In the above-described embodiment, the lowest luminance ratio is not used. However, the present invention is not limited to this. If the lowest luminance ratio is not an unstable value such as a low value compared with other luminance ratios, the lowest luminance ratio may be used.

  In the above-described embodiment, the image is divided into each of the non-light-emitting through image and the light-emitting captured image data in a plurality of areas, in this embodiment, 8 × 8 64 as shown in FIG. Although it divided | segmented into each data of the division area, it is not restricted to this. The image may be divided into a plurality of regions, and the number of divisions and the divided regions can be determined as appropriate.

  In the above-described embodiment, the white balance processing is performed using the captured image at the time of light emission and the through image at the time of non-light emission captured by the imaging unit 20, but the present invention is not limited to this. For the white balance processing, for example, a captured image and a through image acquired from the outside by the CPU 11 or the image processing unit 14 may be used. In the above-described embodiment, the white balance processing is executed using the captured image when emitting light and the through image when not emitting light. However, the present invention is not limited to this. For example, the captured image when emitting light and the captured image when not emitting light are used. White balance processing may be executed.

  In the above-described embodiment, the electronic apparatus to which the present invention is applied has been described by taking the imaging apparatus 1 such as a digital camera as an example, but is not particularly limited thereto. The present invention can be applied to general electronic devices capable of executing the above-described white balance processing. Specifically, for example, the present invention can be applied to a notebook personal computer, a video camera, a portable navigation device, a mobile phone device, a portable game machine, a WEB camera, and the like.

  The series of processes described above can be executed by hardware or can be executed by software. For example, the series of processes described above can be executed by hardware or can be executed by software. In other words, the hardware configuration in FIG. 1 is merely an example, and is not particularly limited. That is, it is sufficient that the imaging apparatus 1 has a function capable of executing the above-described series of processing as a whole, and what functional block is used to realize this function is not particularly limited to the example of FIG. In addition, one functional block may be constituted by hardware alone, software alone, or a combination thereof.

  When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium. The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is not only constituted by the removable medium 31 of FIG. 1 distributed separately from the apparatus main body in order to provide the program to the user, but also in a state of being incorporated in the apparatus main body in advance. It is comprised with the recording medium etc. which are provided in this. The removable medium 31 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being preliminarily incorporated in the apparatus main body includes, for example, the ROM 12 in FIG. 1 in which the program is recorded, the hard disk included in the storage unit 24 in FIG.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.

  DESCRIPTION OF SYMBOLS 1 ... Imaging device, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... Image processing part, 15 ... White balance gain calculation part, 16 ... Image division part , 17 ... Luminance acquisition unit, 18 ... Bus, 19 ... Input / output interface, 20 ... Imaging unit, 21 ... Light emitting unit, 22 ... Operation unit, 23 ... Display unit , 24 ... storage unit, 25 ... communication unit, 26 ... drive, 31 ... removable media, 41 ... luminance comparison unit

Claims (7)

Imaging means;
A light emitting means;
An imaging control unit that controls the imaging unit to capture a first image that is an image brightened by light emission from the light emitting unit and a second image that is lower in brightness than the first image ; ,
First acquisition means for acquiring gain values of the respective color components for adjusting white balance set at the time of capturing the first image and the second image;
A dividing unit that divides an image captured by the imaging unit into a plurality of regions;
Second acquisition means for acquiring the luminance values of the plurality of regions for the first image and the second image, respectively.
For each of the plurality of regions, and the luminance value of the region of the second image corresponding to the region of the luminance value and the first image of the region of the calculated first image by the second acquisition unit A calculating means for calculating a relative value of
Selecting means for selecting a predetermined number of relative values from among a plurality of relative values calculated by the calculating means;
On the basis of the predetermined number of relative values selected by said selecting means and gain value for each color component of the first said acquired by the acquisition means in the first image and the second image, the first image An image pickup apparatus comprising: a correction unit that corrects a gain value of each color component in at least a part of the image area. The imaging apparatus according to claim 1, wherein the correction unit corrects a gain value of each color component in a region where the relative value is greater than or equal to a predetermined value or less than the predetermined value in the first image. The selection means includes
A specifying unit that specifies a region of the first image in which the relative value calculated by the calculating unit is a predetermined value or more or a predetermined value or less;
The imaging apparatus according to claim 1, wherein a predetermined number of relative values greater than or less than a predetermined value specified by the specifying unit are selected. The calculation means calculates a value obtained by dividing the luminance value of the first image area by the luminance value of the second image area corresponding to the first image area as the relative value;
The imaging apparatus according to claim 3, wherein the specifying unit specifies a predetermined number of relative values in descending order of the value from the relative values calculated by the calculating unit. Conversion means for converting each color component of the image acquired by the first acquisition means into a set of pixel parameters including at least luminance information in another color space;
Wherein the correction means further on the basis of the set of pixels of the parameter converted by the converting means, of the according to any one of claims 1, characterized in that to correct the gain value of each color component 4 Imaging device. An imaging control step for controlling the imaging means to image a first image that is an image that is brightened by light emission and a second image that is lower in brightness than the first image ;
A first acquisition step of acquiring a gain value of each color component for adjusting a white balance set at the time of capturing the first image and the second image;
A dividing step of dividing the image acquired by the processing of the first acquiring step into a plurality of regions;
A second acquisition step of acquiring the luminance values of the plurality of regions for the first image and the second image, respectively.
For each of the plurality of regions, the luminance value of the first image region acquired in the second acquiring step and the luminance value of the second image region corresponding to the first image region. a calculation step of calculating a relative value with,
A selection step of selecting a predetermined number of relative values from a plurality of relative values calculated in the calculation step;
Based on said predetermined number of relative values selected by said selecting step between the gain value of each color component of the first said acquired by the acquisition step of the first image and the second image, the first And a correction step of correcting a gain value of each color component in at least a partial area of the image . Computer
An imaging control unit that causes the imaging unit to capture a first image that is an image that is brightened by light emission and a second image that is lower in brightness than the first image ;
The first image and the second first obtaining means for obtaining respective gain values of each color component for adjusting the white balance of the image,
Dividing means for dividing the image captured by the imaging unit into a plurality of regions,
Second acquisition means for acquiring the luminance values of the plurality of regions for the first image and the second image, respectively.
For each of the plurality of regions, and the luminance value of the region of the second image corresponding to the region of the luminance value and the first image region of the acquired first image by said second acquisition means Calculating means for calculating the relative values of
Selecting means for selecting a predetermined number of relative values from among a plurality of relative values calculated by the calculating means;
On the basis of the predetermined number of relative values selected by said selecting means and gain value for each color component of the first said acquired by the acquisition means in the first image and the second image, the first image A white balance adjustment program that functions as a correction unit that corrects the gain value of each color component in at least a part of the area.

Images