JP6900577B2 - Image processing equipment and programs - Google Patents

Description

The present invention relates to an image processing apparatus, an image processing method, and a program, and particularly to a technique for white balance correction.

Conventionally, white balance correction (multi-area white balance correction) that applies different white balance gains to each pixel or area constituting an image has been performed. Multi-area white balance correction is preferably used for captured images of scenes having a particularly large brightness difference. Since the captured image of a scene with a large brightness difference is placed in a different light environment for each region, the color temperature may differ between the bright region and the dark region, and the multi-area white balance correction is preferably used. .. Here, the photographed image having a large brightness difference is, for example, a photographed image in which a shaded area and a sunlit area coexist in one photographed image, or a photographed image in which an indoor lighting area and a sunlight area coexist. And so on.

So far, techniques related to multi-area white balance correction have been proposed.

For example, in Patent Document 1, when a semi-water surface photograph is taken, a white balance gain for water is set for a subject area on water, and white for underwater is set for an image area corresponding to a subject area underwater. It is described that the balance gain is set.

Further, a technique for adjusting the white balance gain among a plurality of white balance gains obtained from each of the plurality of images is known.

For example, Patent Document 2 describes a technique for adjusting a white balance correction value of a color signal obtained from a high-sensitivity pixel and a white balance correction value of a color signal obtained from a low-sensitivity pixel.

Japanese Unexamined Patent Publication No. 2014-138217 Japanese Unexamined Patent Publication No. 2005-80190

However, as in Patent Document 1, when multi-area white balance correction is performed with different white balance gains independently obtained for each region of the photographed image, an image created in the brain when a human sees a scene is obtained. It makes me feel uncomfortable. When a human looks at a certain scene, he or she strongly corrects the gaze point, and the correction of the non-gaze point is pulled by the influence of the correction of the gaze point.

In the technique described in Patent Document 1, the white balance gain of the subject region on water and the subject region in water is determined by detecting the contrast of each region, and each determined white balance gain is determined in each region. Has been applied to. Therefore, the white balance gains in each region are not adjusted to each other, and the image may have a white balance correction that gives a sense of discomfort.

Further, Patent Document 2 does not mention the adjustment of the white balance gain obtained for each region of the captured image. That is, there is no mention of adjusting the white balance when performing multi-area white balance correction.

In order to achieve the above object, the image processing apparatus according to one aspect of the present invention divides the captured image into a plurality of regions and the image input unit into which the captured image is input, and the divided plurality of regions are first. An area setting unit that sets an area and a second area, a white balance gain acquisition unit that acquires a first white balance gain and a second white balance gain from the first area and the second area, respectively, and a first white balance gain. A determination unit that determines the adjusted second white balance gain by adjusting the second white balance gain based on the second white balance gain, and the first white balance gain and the adjusted second region in the first region and the second region. It is provided with a white balance correction unit that corrects the white balance by the white balance gain.

According to this aspect, the captured image is divided into a plurality of regions, and the divided plurality of regions are set as a first region and a second region, and the first white balance gain of the first region and the second region of the second region are set. The second white balance gain is determined based on the white balance gain. Then, since the first region is white-balanced by the first white balance gain and the second region is white-balanced by the adjusted second white balance gain, natural multi-area white balance correction can be performed.

Preferably, the white balance gain acquisition unit is provided with a first control unit that controls the white balance gain acquisition unit to acquire the first white balance gain and the second white balance gain based on the difference in brightness of the captured image input to the image input unit.

According to this aspect, the white balance gain acquisition unit can acquire the first white balance gain and the second white balance gain as needed, and efficient multi-area white balance correction can be performed.

Preferably, the first control unit causes the white balance gain acquisition unit to acquire the first white balance gain and the second white balance gain when the brightness difference of the captured image is equal to or greater than the threshold value.

According to this aspect, the white balance gain acquisition unit can acquire the first white balance gain and the second white balance gain as needed, and efficient multi-area white balance correction can be performed.

Preferably, the first control unit causes the white balance gain acquisition unit to acquire the first white balance gain and the second white balance gain based on the command input from the operation unit.

According to this aspect, since the first white balance gain and the second white balance gain are acquired based on the command input from the operation unit, the multi-area white balance correction that reflects the photographer's intention is performed. Can be done.

Preferably, the area setting unit sets the area for adjusting the focus or exposure of the captured image as the first area.

According to this aspect, since the region for adjusting the focus or exposure of the captured image is the first region, the multi-area white balance correction that reflects the photographer's intention can be performed.

Preferably, the white balance gain acquisition unit obtains the acquired white balance gain according to the size of the region in which the white balance gain is obtained or the brightness of the region in which the white balance gain is obtained, as the first white balance gain and the second white. Balance gain.

According to this aspect, the first white balance gain is determined by the size of the region where the white balance gain is obtained or the brightness of the region where the white balance gain is obtained. Area white balance correction can be performed.

Preferably, the determination unit determines the adjusted second white balance gain based on the first white balance gain and the adjustment threshold value determined according to the first white balance gain and the second white balance gain.

According to this aspect, the adjusted second white balance gain is determined based on the first white balance gain and the determined adjustment threshold value. Therefore, the white balance correction is performed by determining the adjustment threshold value according to the scene. Can be done properly.

Preferably, the determination unit determines the adjusted second white balance gain based on the first white balance gain and the adjustment threshold value determined according to the brightness difference of the captured image.

According to this aspect, the adjusted second white balance gain is determined based on the first white balance gain and the adjustment threshold determined according to the brightness difference of the captured image. The second white balance gain is adjusted accordingly, and natural multi-area white balance correction is appropriately performed.

Preferably, the white balance gain acquisition unit acquires the first white balance gain based on the pixel value in the first region, and acquires the second white balance gain based on the pixel value in the second region.

According to this aspect, the first white balance gain and the second white balance gain are acquired based on the pixel value of the first region and the pixel value of the second region, respectively, so that the multi-area corresponding to the scene of the captured image is obtained. White balance correction is performed.

Preferably, a short-exposure photographed image and a long-exposure photographed image are input to the image input unit, and the area setting unit first sets one region of either the short-exposure photographed image or the long-exposure photographed image. It is set as an area, and the other area is set as a second area.

According to this aspect, even when a high dynamic range composite image is generated from a short-exposure photographed image and a long-exposure photographed image, natural multi-area white balance correction can be performed.

Preferably, the image processing apparatus selects a mode in which a short-exposure photographed image and a long-exposure photographed image are input and the short-exposure photographed image and the long-exposure photographed image are combined, and the white balance is selected. The gain acquisition unit includes a second control unit that controls the acquisition of the first white balance gain and the second white balance gain.

According to this aspect, the white balance gain acquisition unit can acquire the first white balance gain and the second white balance gain as needed, and efficient multi-area white balance correction can be performed.

Preferably, the second control unit applies the first white balance gain and the second white balance gain to the white balance gain acquisition unit when the exposure of the short-exposure photographed image and the long-exposure photographed image is equal to or greater than the threshold value. Control to acquire.

According to this aspect, the white balance gain acquisition unit can acquire the first white balance gain and the second white balance gain as needed, and efficient multi-area white balance correction can be performed.

Preferably, the determination unit is adjusted based on the first white balance gain and the adjustment threshold determined from the exposure difference between the short-exposure photographed image and the long-exposure photographed image input to the image input unit. 2 Determine the white balance gain.

According to this aspect, the determination unit is based on the first white balance gain and the adjustment threshold determined from the exposure difference between the short-exposure photographed image and the long-exposure photographed image input to the image input unit. Since the second white balance gain is determined after the adjustment, a natural multi-area white balance correction can be performed.

Preferably, the white balance gain acquisition unit acquires the first white balance gain based on the short-exposure photographed image having the first region or the long-exposure photographed image, and the short-exposure photographed image or the short-exposure photographed image having the second region. The second white balance gain is acquired based on the long-exposure captured image.

According to this aspect, since the first white balance gain and the second white balance gain are calculated based on the captured image, a more appropriate white balance gain is acquired and natural multi-area white balance correction is performed.

Preferably, in the short-exposure photographed image and the long-exposure photographed image, the first and second image determination units for determining the first image and the second image are provided, and the white balance gain acquisition unit is the first white of the first image. When acquiring the balance gain, the first white balance gain is acquired based on the first importance given to the first region and the second region, and the second white balance gain of the second image is acquired. In this case, the second white balance gain is acquired based on the second priority, which is the opposite of the first priority.

According to this aspect, the second white balance gain is calculated based on the second degree of importance, which is the opposite of the first degree of importance obtained by obtaining the first white balance gain, so that an appropriate second white balance gain can be obtained. It can be obtained with high accuracy.

The image processing method according to another aspect of the present invention includes a step in which a captured image is input and a step in which the captured image is divided into a plurality of regions and the divided plurality of regions are set as a first region and a second region. And, based on the step of acquiring the first white balance gain and the second white balance gain from the first region and the second region, respectively, and the first white balance gain and the second white balance gain, the second white balance gain is obtained. A step of determining the adjusted second white balance gain, and a step of correcting the white balance of the first region and the second region by the first white balance gain and the adjusted second white balance gain.
including.

The program according to another aspect of the present invention includes a step of inputting a captured image, a step of dividing the captured image into a plurality of regions, and a step of setting the divided plurality of regions into a first region and a second region. The second white balance gain was adjusted based on the step of acquiring the first white balance gain and the second white balance gain from the first region and the second region, respectively, and the first white balance gain and the second white balance gain. A computer performs an image processing method including a step of determining a second white balance gain after adjustment and a step of correcting the white balance of the first region and the second region by the first white balance gain and the adjusted second white balance gain. To execute.

According to the present invention, the first region and the second region are set in the captured image, and the adjusted second white balance is based on the first white balance gain of the first region and the second white balance gain of the second region. Since the gain is determined and the white balance is corrected by the first white balance gain in the first region and by the adjusted second white balance gain in the second region, natural multi-area white balance correction can be performed.

FIG. 1 is a block diagram showing a control processing system of an imaging device. FIG. 2 is a diagram showing an example of a functional configuration of the image processing unit. FIG. 3 is a diagram showing an example of a photographed image. FIG. 4 is a diagram showing an example of setting the main region and the sub region. FIG. 5 is a diagram illustrating an example of adjusting the sub white balance gain. FIG. 6 is a flow chart showing the operation of the image processing unit. FIG. 7 is a diagram showing a long-exposure photographed image and a short-exposure photographed image. FIG. 8 is a diagram for explaining that the area is set as a main area and a sub area by the area setting unit. FIG. 9 is a diagram of the chromaticity point distribution of the short-exposure photographed image and the long-exposure photographed image. FIG. 10 is a flow chart showing an example of the operation of the image processing unit. FIG. 11 is a flow chart showing an example of the operation of the image processing unit. FIG. 12 is a diagram showing an example of a functional configuration of the image processing unit. FIG. 13 is a flow chart showing the operation of the image processing unit. FIG. 14 is a diagram showing a functional configuration example of the image processing unit. FIG. 15 is a flow chart showing an example of control of the control unit. FIG. 16 is a flow chart showing an example of control of the control unit. FIG. 17 is a flow chart showing an example of control of the control unit. FIG. 18 is a diagram showing a display example of the display unit of the image pickup apparatus. FIG. 19 is a diagram showing a display example of the display unit of the image pickup apparatus. FIG. 20 is a diagram showing a display example of the display unit of the image pickup apparatus. FIG. 21 is a diagram showing a captured image. FIG. 22 is a flow chart showing control for determining the main white balance gain according to the luminance value of the region.

Hereinafter, preferred embodiments of the image processing apparatus, image processing method, and program according to the present invention will be described with reference to the accompanying drawings.

FIG. 1 is a block diagram showing a control processing system of an image pickup apparatus 1 equipped with an image processing apparatus (image processing unit 31) of the present invention.

The subject light passes through the lens 12, the aperture 14, and the mechanical shutter 20, and is received by the image sensor 21. The lens 12 is composed of an imaging optical system including an imaging lens and an aperture 14.

The image pickup device 21 is an element that receives a subject image and generates an image pickup signal (image data), and includes a color filter such as RGB (red, green, and blue), a CCD (Charge Coupled Device) that converts an optical image into an electric signal, and the like. It has an image sensor such as CMOS (Complementary Metal Oxide Semiconductor). The image data output from the image pickup element 21 is processed by the process processing unit 22 by an AGC (Automatic Gain Control) circuit or the like, and then the analog format image data is converted into a digital format by the AD (Analog Digital) conversion unit 23. It is converted to the image data of. The digitized image data is stored in the main memory 24.

The main memory 24 is an area for temporarily storing image data, and is configured by a DRAM (Dynamic Random Access Memory) or the like. The image data sent from the AD conversion unit 23 and stored in the main memory 24 is read out by the image processing unit 31 controlled by the system control unit 25.

The image processing unit 31 uses the image data generated by the image pickup element 21 as input image data, performs various image processing such as white balance correction, gamma correction processing, and demosaic processing, and remains the image data after the image processing. Save in memory 24. The white balance correction performed by the image processing unit 31 will be described in detail later.

The image data that has been subjected to image processing in the image processing unit 31 and stored in the main memory 24 is read out by the display control unit 35 and the compression / decompression unit 32. The display control unit 35 controls the display unit 8 and causes the display unit 8 to display the image data read from the main memory 24. In this way, the image data output from the image sensor 21 and subjected to image processing by the image processing unit 31 is displayed on the display unit 8 as an image pickup confirmation image (post-view image).

On the other hand, the compression / decompression unit 32 performs compression processing of the image data read from the main memory 24 to create image data in an arbitrary compression format such as PEG (Joint Photographic Experts Group) or TIFF (Tagged Image File Format). .. The image data after the compression process is stored in the external memory 10 by the storage control unit 33 that controls the data storage process in the external memory 10 and the data read process from the external memory 10. The imaging information is added to the image data in any format, and for example, an Exif (Exchangeable image file format) format can be adopted.

When the shutter button is pressed (half-pressed) in the first stage, the AF processing function of the image pickup apparatus 1 integrates the absolute values of the high-frequency components of the image data corresponding to the AF area captured when the shutter button is pressed halfway, and this integration is performed. The value (AF evaluation value) is output to the system control unit 25.

When the shutter button is pressed (half-pressed) in the first stage, the AE detection function integrates the digital signals corresponding to the entire screen, or integrates the image data with different weights in the center and periphery of the screen. Then, the integrated value is output to the system control unit 25.

The system control unit 25 controls the main memory 24, the image processing unit 31, and the storage control unit 33 as described above, but also controls other units (AF processing function, AE detection function) in the image pickup apparatus 1.

The system control unit 25 operates the AE detection function when the shutter button is half-pressed in the auto shooting mode. The subject brightness (imaging Ev value) is calculated from the integrated value input by the AE detection function, and the aperture value and shutter speed (mechanical shutter 20 and / or) of the aperture 14 are calculated via the aperture driving unit 28 based on the imaged Ev value. The charge accumulation time of the image sensor 21) is determined according to the program diagram.

When the shutter button is fully pressed, the aperture 14 is controlled based on the determined aperture value, and the mechanical shutter 20 is controlled via the shutter drive unit 27 based on the determined shutter speed. The charge accumulation time of the image sensor 21 is controlled by the sensor drive unit 26.

When the shutter button is half-pressed in the auto shooting mode, the system control unit 25 moves the focus lens of the lens 12 from close to infinity via the lens drive unit 29, and operates the AF processing function to operate each of them. The AF evaluation value at the lens position is acquired from the AF processing function. Then, the focus is adjusted to the subject by searching for the focusing position where the AF evaluation value is maximum and moving the focus lens to the focusing position. The system control unit 25 acquires information on the focusing position based on the moving position of the focus lens.

Further, the system control unit 25 acquires an operation signal from the user interface 36 including the shutter button, the power switch, and the operation unit, and performs various processes and device control according to the operation signal.

Programs and data required for various processes and device control performed by the system control unit 25 are stored in the main memory 24. The system control unit 25 can read out the programs and data stored in the control memory 30 as needed, and can save new programs and data in the control memory 30.

<First Embodiment>

FIG. 2 is a diagram showing a functional configuration example of the image processing unit (image processing device) 31 of the first embodiment.

The image processing unit 31 includes an image input unit 101, an area setting unit 103, a white balance gain acquisition unit 105, a determination unit 107, and a white balance correction unit 109.

A captured image is input to the image input unit 101. For example, a captured image of a scene having a large brightness difference is input to the image input unit 101.

FIG. 3 is a diagram showing an example of a captured image input to the image input unit 101. The captured image 150 is a captured image obtained by capturing a retrograde scene. There is a large difference in brightness between the person 151 and the background 152, and the light environments are different from each other. That is, the person 151 is a dark region, the background 152 is a bright region, and the regions have different color temperatures. Other scenes with a large difference in brightness include a scene in which a shaded area and a sunlit area, an indoor lighting area, and a sunlight area coexist. In this embodiment, it is possible to perform natural multi-area white balance correction on a captured image having such a large brightness difference.

The area setting unit 103 sets a main area and a sub-area which is an area other than the main area in the captured image 150. The main area is the area that the photographer is gazing at in the captured image 150. In this way, by performing multi-area white balance correction, which will be described later, with the area that the photographer is gazing at as the main area, the correction of the gaze area is strong, as in the case of a human being looking at a scene. The correction of the part of can be a natural correction influenced by the correction of the part to be watched. The main area is also referred to as a "first area". The sub-region is also referred to as a "second region".

FIG. 4 is a diagram showing an example of setting the main region and the sub region in the captured image 150 of FIG.

The area setting unit 103 first divides the captured image 150 into eight vertical divisions and eight horizontal divisions (8 × 8), and then averages the color, brightness, or position within the angle of view of the divided areas. Based on the information in, set to divide into multiple areas. For example, when the area setting unit 103 divides an area based on the brightness, in the captured image 150, the person 151 belongs to a dark area and the background 152 belongs to a bright area due to backlight. Then, for example, the area setting unit 103 divides the area of the captured image 150 into an inner area of the frame W and an outer area of the frame W based on the brightness. For example, when the area inside the frame W includes the gaze area of the photographer, the area setting unit 103 sets the area inside the frame W as the main area and the area outside the frame W as the sub area. In this example, the captured image 150 is divided into 8 × 8 and the main area and the sub area are set, but the present invention is not limited to this. The area setting unit 103 may set the main area and the sub area for each pixel, or may divide the area into other numbers and set the main area and the sub area.

The white balance gain acquisition unit 105 acquires the main white balance gain and the sub white balance gain for white balance correction of the main region and the sub region, respectively. The main white balance gain and the sub white balance gain can be obtained by various methods. For example, the white balance gain acquisition unit 105 acquires the main white balance gain based on the pixel value of the main region, and acquires the sub white balance gain based on the pixel value of the sub region. The main white balance gain is also referred to as "first white balance gain". The sub-white balance gain is also referred to as "second white balance gain".

The determination unit 107 determines the adjusted sub-white balance gain by adjusting the sub-white balance gain based on the main white balance gain and the sub-white balance gain. The determination unit 107 adjusts the sub-white balance gain so as to be affected by the main white balance gain, and determines the sub-white balance gain after adjustment.

FIG. 5 is a diagram illustrating an example of adjusting the sub-white balance gain performed by the determination unit 107. In FIG. 5, the vertical axis shows the gain of blue (Blue), and the horizontal axis shows the gain of red (Red). Further, the main white balance gain of the captured image 150 acquired by the white balance gain acquisition unit 105 is indicated by MG, and the sub white balance gain is indicated by SG. Further, the adjusted sub-white balance gain adjusted by the determination unit 107 is indicated by R-SG.

The determination unit 107 determines the adjustment threshold value R-TH, and adjusts the sub-white balance gain SG so as to be on the main white balance gain MG side within the adjustment threshold value R-TH. For example, the adjusted sub white balance gain R-SG is calculated by setting a with the formula shown below so that the adjustment threshold value is within R-TH. Here, in FIG. 5, the adjustment threshold value R-TH is illustrated by a circle.

In the above equation, R-SG_G _r indicates the red adjusted sub white balance gain, and R-SG_G _b indicates the blue adjusted sub white balance gain. MG_G _r indicates the red main white balance gain, and MG_G _b indicates the blue main white balance gain. SG_G _r indicates a red sub-white balance gain, and SG_G _b indicates a blue sub-white balance gain. a is a numerical value set so that R-SG_G _r and R-SG_G _b are within the adjustment threshold value.

For example, in a, the value of a becomes smaller as the distance between the main white balance gain MG and the sub white balance gain SG is farther.

Further, the determination unit 107 determines the adjustment threshold value R-TH from the relationship between the main white balance gain MG and the sub white balance gain SG. Further, the determination unit 107 determines the adjustment threshold value R-TH according to the difference in brightness of the captured image input to the image input unit 101. Further, when a plurality of captured images having different exposures (short-exposure captured image and long-exposure captured image) are input, the determination unit 107 exposes between the plurality of captured images, which will be described in the second embodiment. The adjustment threshold R-TH may be determined based on the difference.

FIG. 6 is a flow chart (corresponding to an image processing method) showing the operation of the image processing unit 31 of the present embodiment.

First, the captured image 150 is input to the image input unit 101 (step S10). After that, the area setting unit 103 divides the area based on the brightness, the color, and / and the pixel position (step S11), and the area setting unit 103 also acquires the gaze area (step S12). In this case, the region including the gaze region is the main region, and the other regions are the sub regions. Then, the white balance gain acquisition unit 105 acquires the main white balance gain and the sub white balance gain (step S13). After that, the determination unit 107 recalculates the sub-white balance gain from the main white balance gain and the sub-white balance gain, and determines the adjusted sub-white balance gain (step S14). After that, the white balance correction unit 109 performs multi-area white balance correction by the main white balance gain in the main region and the adjusted sub white balance gain in the sub region (step S15).

As described above, the adjusted sub white balance gain is determined based on the main white balance gain of the main region and the sub white balance gain of the sub region, the main region is the main white balance gain, and the sub region is the adjusted sub. Since the white balance is corrected by the white balance gain, it is possible to perform a natural multi-area white balance correction.

In the above embodiment, the hardware structure of the processing unit that executes various processes is various processors as shown below. For various processors, the circuit configuration can be changed after manufacturing the CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), etc., which are general-purpose processors that execute software (programs) and function as various processing units. Includes a dedicated electric circuit, which is a processor having a circuit configuration specially designed to execute a specific process such as a programmable logic device (PLD), an ASIC (Application Specific Integrated Circuit), etc. Is done.

One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types (for example, a plurality of FPGAs or a combination of a CPU and an FPGA). You may. Further, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with one processor, first, one processor is configured by a combination of one or more CPUs and software, as represented by a computer such as a client or a server. There is a form in which the processor functions as a plurality of processing units. Secondly, as typified by System On Chip (SoC), there is a form in which a processor that realizes the functions of the entire system including a plurality of processing units with one IC (Integrated Circuit) chip is used. is there. As described above, the various processing units are configured by using one or more of the above-mentioned various processors as a hardware-like structure.

Further, the hardware structure of these various processors is, more specifically, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.

Each of the above configurations and functions can be appropriately realized by any hardware, software, or a combination of both. For example, for a program that causes a computer to perform the above-mentioned processing steps, a computer-readable recording medium (non-temporary recording medium) that records such a program, or a computer on which such a program can be installed. However, it is possible to apply the present invention.

<Second embodiment>

Next, the second embodiment will be described. In the present embodiment, the short-exposure photographed image acquired by the short-time exposure and the long-exposure photographed image acquired by the long-time exposure are input to the image input unit 101. This embodiment is preferably used when a short-exposure photographed image and a long-exposure photographed image are combined to obtain a high dynamic range (HDR) composite image.

FIG. 7 is a diagram showing a long-exposure photographed image and a short-exposure photographed image input to the image input unit 101. The short-exposure photographed image 163 and the long-exposure photographed image 165 capture the same scene as in FIG.

FIG. 8 is a diagram for explaining that the main area and the sub area are set by the area setting unit 103. By the same method as the method described with reference to FIG. 4, the region is divided into an inner region of the frame W and an outer region of the frame W. Then, the area setting unit 103 sets either one of the short-exposure photographed image 163 and the long-exposure photographed image 165 as the main area, and sets the other area as the sub-area. For example, when the region setting unit 103 uses a region of the short-exposure photographed image 163 as the main region, the region setting unit 103 sets the region other than the region corresponding to the region of the long-exposure photographed image 165 as the sub-region.

When the gaze area is the area inside the frame W, the area setting unit 103 sets the main area according to the characteristics of the area inside the frame W of the short-exposure photographed image 163 and the long-exposure photographed image 165. For example, the area setting unit 103 includes a photographed image in which the brightness of the area inside the frame W is in a predetermined range, a photographed image in which the area inside the frame W has high saturation, and a photographed image in which the contrast of the area inside the frame W is high. The area inside the frame W in the captured image is set as the main area.

FIG. 9 is a diagram of the chromaticity point distribution of the short-exposure photographed image 163 and the long-exposure photographed image 165 of FIG. FIG. 9A shows the chromaticity point distribution of the short-exposure photographed image 163, and FIG. 9B shows the chromaticity point distribution of the long-exposure photographed image 165. In FIGS. 9A and 9B, the vertical axis shows the value of B (blue) / G (green), and the horizontal axis shows the value of R (red) / G (green). ing.

In the short-exposure photographed image 163, the integrated value 181 of the pixels in the frame W is blacked out, and the integrated value is too dark, so that the SN (signal-noise) ratio is poor and the values are scattered. On the other hand, the integrated value 183 of the pixels outside the frame W indicates an appropriate value because the integrated value 183 of the pixels outside the frame W can ignore the integrated value 181 of the pixels inside the frame W.

In the long-exposure photographed image 165, the integrated value 185 of the pixels in the frame W is gathered in a certain area and shows a correct value. On the other hand, since the integrated value 187 of the pixels outside the frame W is overexposed (saturated), the chromaticity points are abnormally concentrated in one place.

In such a case, the area inside the frame W of the long-exposure photographed image 165 is set as the main area, and the area outside the frame W of the short-exposure photographed image 163 is set as the sub-area, and the main white balance gain and the adjusted sub-white balance gain are gained. By obtaining and performing multi-area white balance correction, it is possible to obtain an HDR composite image without a sense of discomfort.

FIG. 10 is a flow chart showing an example of the operation of the image processing unit 31 of the present embodiment.

First, the captured image is input to the image input unit 101 (step S20). After that, the area setting unit 103 divides the areas of the short-exposure photographed image 163 and the long-exposure photographed image 165 based on the brightness, the color, and the pixel position (step S21), and the area setting unit 103 divides the gaze area. Acquired (step S22), the main area and the sub area are set. After that, the white balance gain acquisition unit 105 acquires the white balance gain for each region (step S23). Next, the white balance gain acquisition unit 105 calculates the average brightness of the gaze region from each of the short-exposure photographed image 163 and the long-exposure photographed image 165 (step S24). The white balance gain obtained from the gaze region of the image whose average brightness in the gaze region is closer to the middle is set as the main white balance gain (step S25). Further, the white balance gain acquisition unit 105 sets the white balance gain obtained from the region other than the gaze region of the image whose average brightness in the gaze region is far from the middle as the sub white balance gain (step S26). Then, the determination unit 107 determines the adjusted sub-white balance gain from the main white balance gain and the sub-white balance gain (step S27). After that, the white balance correction unit 109 corrects the multi-area white balance by the main white balance gain in the main region and the adjusted sub white balance gain in the sub region (step S28).

FIG. 11 is a flow chart showing another example of the operation of the image processing apparatus of the present embodiment.

First, the captured image is input to the image input unit 101 (step S30). After that, the white balance gain acquisition unit 105 obtains the white balance gain for each of the short-exposure photographed image 163 and the long-exposure photographed image 165 (step S31). Next, the area setting unit 103 divides the area in the short-exposure photographed image 163 and the long-exposure photographed image 165 based on the brightness, the color, and the pixel position (step S32). After that, the area setting unit 103 acquires the gaze area (step S33). The average brightness of the gaze area is calculated from each of the short-exposure photographed image 163 and the long-exposure photographed image 165 (step S34). The area setting unit 103 uses the white balance gain obtained from the image whose average brightness is closer to the middle as the main white balance gain (step S35). Then, the determination unit 107 determines the adjusted sub-white balance gain from the main white balance gain and the sub-white balance gain (step S36). After that, the white balance correction unit 109 performs multi-area white balance correction with the main area having the main white balance gain and the sub area having the adjusted sub white balance gain (step S37).

<Third embodiment>

Next, the third embodiment will be described. In the present embodiment, the sub white balance gain is obtained based on the degree of importance. As a result, an appropriate sub-white balance gain can be obtained with high accuracy.

FIG. 12 is a diagram showing a functional configuration example of the image processing unit 31 of the present embodiment. The parts already described in FIG. 2 are designated by the same reference numerals and the description thereof will be omitted.

The image processing unit 31 includes an image input unit 101, an area setting unit 103, a white balance gain acquisition unit 105, a determination unit 107, a white balance correction unit 109, and a main / sub image determination unit 111.

The main / sub-image determination unit 111 determines the main image and the sub-image in the short-exposure photographed image 163 and the long-exposure photographed image 165. The main image is also called the "first image". The sub-image is also called a "second image". The main / sub image determination unit is also referred to as an “image determination unit”.

When acquiring the main white balance gain of the main image, the white balance gain acquisition unit 105 acquires the main white balance gain based on the first degree of importance given to the main region and the sub region, and acquires the main white balance gain of the sub image. When acquiring the sub-white balance gain of, the sub-white balance gain is acquired based on the second emphasis, which is the opposite of the first emphasis.

FIG. 13 is a flow chart showing the operation of the image processing unit 31 of the present embodiment.

First, the short-exposure photographed image 163 and the long-exposure photographed image 165 are input to the image input unit 101 (step S40). After that, the gaze area is acquired by the area setting unit 103 (step S41). After that, the main / sub-image determination unit 111 calculates the average brightness of the gaze region from each of the short-exposure photographed image 163 and the long-exposure photographed image 165 (step S42). Then, the main / sub-image determination unit 111 sets the average brightness of the gaze region of the short-exposure photographed image 163 and the long-exposure photographed image 165 to the median value of the brightness of the short-exposure photographed image 163 and the long-exposure photographed image 165. The closer image is used as the main image (step S43). Then, the white balance gain acquisition unit 105 obtains the main white balance gain from the determined main image (step S44). At this time, the white balance gain acquisition unit 105 sets the degree of importance in each region and acquires the white balance gain based on the degree of importance. For example, the gaze area has an emphasis of 2, and the area other than the gaze area has an emphasis of 1. Based on this emphasis (first emphasis), the white balance gain acquisition unit 105 sets the main white balance from the main image. Get the gain. After that, the white balance gain acquisition unit 105 acquires the degree of importance for each region when the main white balance gain is obtained from the main image (step S45). Then, the white balance gain acquisition unit 105 sets the importance degree, which is the opposite of the importance degree of the main image, to the sub image, and obtains the sub white balance gain from the sub image (step S46). That is, in the sub-image, the importance of the gaze area is set to 1, the importance of the area other than the gaze area is set to 2, and the sub-white balance gain is obtained from the sub-image based on this importance (second importance). .. Next, the determination unit 107 recalculates and adjusts the sub-white balance gain acquired by the white balance gain acquisition unit 105 from the main white balance gain and the sub-white balance gain, and calculates the adjusted sub-white balance gain. (Step S47). After that, the white balance correction unit 109 corrects the multi-area white balance by the main white balance gain in the main region and the adjusted sub white balance gain in the sub region (step S48).

<Fourth Embodiment>

Next, a fourth embodiment will be described. In the present embodiment, the control unit 112 (FIG. 14) controls the acquisition of the main white balance gain and the sub white balance gain of the white balance gain acquisition unit 105. The control unit 112 has the functions of the first control unit and the second control unit.

FIG. 14 is a diagram showing a functional configuration example of the image processing unit 31 of the present embodiment. The parts already described in FIG. 2 are designated by the same reference numerals and the description thereof will be omitted.

The image processing unit 31 of the present embodiment includes an image input unit 101, an area setting unit 103, a white balance gain acquisition unit 105, a determination unit 107, a white balance correction unit 109, and a control unit 112.

The control unit 112 determines whether or not the white balance gain acquisition unit 105 acquires the main white balance gain and the sub white balance gain, and causes the white balance gain acquisition unit 105 to execute the process.

FIG. 15 is a flow chart showing an example of control of the white balance gain acquisition unit 105 of the control unit 112. In this example, the control unit (first control unit) 112 controls the white balance gain acquisition unit 105 based on the brightness difference of the input image.

First, the control unit 112 calculates the brightness difference of the input captured image (step S50). Then, the control unit 112 determines whether or not the calculated luminance difference is equal to or greater than the threshold value (step S51), and if the luminance difference is equal to or greater than the threshold value, the white balance gain acquisition unit 105 is contacted with the main white balance gain and the sub. A plurality of white balance gains of the white balance gain are obtained (step S52). On the other hand, when the brightness difference is less than the threshold value, the white balance gain acquisition unit 105 is not made to obtain a plurality of white balance gains of the main white balance gain and the sub white balance gain (step S53). The luminance difference referred to here means the luminance difference in one captured image or the luminance between a plurality of captured images.

FIG. 16 is a flow chart showing an example of control of the white balance gain acquisition unit 105 of the control unit 112. In this example, when the control unit (second control unit) 112 selects a mode in which the short-exposure photographed image and the long-exposure photographed image are combined, the white balance gain acquisition unit 105 receives the main white balance gain. And control to acquire the sub-white balance gain.

The control unit 112 determines whether or not the mode in which the short-exposure photographed image 163 and the long-exposure photographed image 165 are combined is selected (step S60). This mode selection is performed by, for example, the photographer via the user interface 36 of the image pickup apparatus 1. Then, when the mode to be combined is selected, the control unit 112 causes the white balance gain acquisition unit 105 to calculate the main white balance gain and the sub white balance gain (step S61). On the other hand, the control unit 112 does not cause the white balance gain acquisition unit 105 to calculate the main white balance gain and the sub white balance gain when the mode to be combined is not selected (step S62).

FIG. 17 is a flow chart showing an example of control of the white balance gain acquisition unit 105 of the control unit 112. In FIG. 17, a mode in which the control unit (second control unit) 112 combines the short-exposure captured image 163 and the long-exposure captured image 165 is selected, and the short-exposure captured image 163 and the long-exposure captured image 163 are captured. The white balance gain acquisition unit 105 is controlled on the basis that the exposure difference from the image 165 is equal to or greater than a certain threshold value.

The control unit 112 determines whether the mode in which the short-exposure photographed image and the long-exposure photographed image are combined is selected (step S70). When the mode to be combined is selected, the control unit 112 further determines whether or not the mode for shooting with an exposure difference of 2 Ev or more is selected (step S71). Then, when the mode for shooting with an exposure difference of 2 Ev or more is selected, the control unit 112 causes the white balance gain acquisition unit 105 to obtain the main white balance gain and the sub white balance gain (step S72). .. On the other hand, when the mode to be combined is not selected, or when shooting with an exposure difference of 2 Ev or more is not performed, the control unit 112 sends the white balance gain acquisition unit 105 to the main white balance gain and the sub white balance gain. And are not calculated (step S73).

<Example of determining the main or sub white balance gain and the main or sub region>

Next, the determination of the main white balance gain and the sub-white balance gain, and the determination of the main region and the sub region will be described. In forms other than those described above, the main white balance gain and the sub white balance gain are determined, and the main region and the sub region are determined.

<< Example 1 >>

In this example, the main white balance gain is selected based on a command from the photographer.

FIG. 18 is a diagram showing a display example of the display unit 8 of the image pickup apparatus 1. The display unit 8 displays the white balance gain acquired by the white balance gain acquisition unit 105 with respect to the captured image input to the image input unit 101. In the case shown in FIG. 18, “white balance 1 sunny” (indicated by reference numeral 173) and “white balance 2 shade” (indicated by reference numeral 174) acquired by the white balance gain acquisition unit 105 are displayed on the display unit 8. Has been done. Since the photographer is gazing at the sunny area in the photographed image, "white balance 1 sunny" is selected. As a result, "white balance 1 sunny" becomes the main white balance gain.

≪Example 2≫

This example is an example in which the main area is selected based on a command from the photographer.

FIG. 19 is a diagram showing a display example of the display unit 8 of the image pickup apparatus 1. The captured image input to the image input unit 101 is displayed on the display unit 8. Then, the photographer selects the area to be watched by the icon 191. As a result, the area to be watched by the photographer is selected as the main area, and the area setting unit 103 sets the selected area as the main area.

≪Example 3≫

This example is an example in which the area for adjusting the focus or exposure of the captured image is the main area.

FIG. 20 is a diagram showing a display example of the display unit 8 of the image pickup apparatus 1. A captured image (preview image) at the time of shooting is displayed on the display unit 8. At the time of shooting, the photographer moves a reference area 193, which is an area for focusing on the area of gaze or an area for adjusting exposure. In this case, the area setting unit 103 uses the reference area 193 or the area including the reference area 193 as the main area.

≪Example 4≫

This example is an example in which the area where the face is detected is the main area.

FIG. 21 is a diagram showing a captured image 150. In the captured image 150, the face detection region 195 is performed. The area setting unit 103 sets the face detection area 195 or the area including the face detection area 195 as the main area.

<< Example 5 >>

In this example, the main white balance gain is determined according to the brightness of the region where the white balance gain is obtained.

FIG. 22 is a flow chart showing control for determining the main white balance gain according to the luminance value of the region by the white balance gain acquisition unit 105. Here, the white balance gain acquisition unit 105 determines the main white balance gain for the captured image 150 shown in FIG.

First, the white balance gain acquisition unit 105 calculates the average of the brightness values of the region inside the frame W and the region outside the frame W, respectively (step S80). After that, the white balance gain acquisition unit 105 determines, among the region inside the frame W and the region outside the frame W, a region having an average luminance value at the median luminance value of the captured image 150 (step S81). ). When the average brightness value of the region inside the frame W is close to the median, the white balance gain calculated based on the region inside the frame W is set as the main white balance gain (step S82). On the other hand, when the average of the brightness values in the region outside the frame W is close to the median, the white balance gain calculated based on the region outside the frame W is set as the main white balance gain (step S83).

<< Example 6 >>

In this example, the main white balance gain is determined according to the size of the region where the white balance gain is obtained.

In FIG. 2 described above, when the white balance gain is calculated based on the region inside the frame W and the region outside the frame W, the region outside the frame W is larger, so that the white balance is larger. The gain acquisition unit 105 uses the white balance gain in the region outside the frame W as the main white balance gain.

Although the examples of the present invention have been described above, it goes without saying that the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present invention.

1: Imaging device

8: Display

10: External memory

12: Lens

14: Aperture

20: Mechanical shutter

21: Image sensor

22: Process processing unit

23: AD conversion unit

24: Main memory

25: System control unit

26: Sensor drive unit

27: Shutter drive unit

28: Aperture drive unit

29: Lens drive unit

30: Control memory

31: Image processing unit

32: Compression / expansion part

33: Memory control unit

35: Display control unit

36: User interface

101: Image input unit

103: Area setting unit

105: White balance gain acquisition unit

107: Decision section

109: White balance correction unit

111: Main / sub image determination unit

112: Control unit

Claims (21)

The image input section where the captured image is input and
An area setting unit that divides the captured image into a plurality of areas and sets the divided plurality of areas as a main area to be watched by the photographer and a sub-area to other areas.
A white balance gain acquisition unit that acquires a first white balance gain and a second white balance gain from the main region and the sub region, respectively.
A determination unit that determines the adjusted second white balance gain by adjusting the second white balance gain based on the first white balance gain and the second white balance gain.
A white balance correction unit that corrects the white balance of the main region and the sub region by the first white balance gain and the adjusted second white balance gain.
With
A short-exposure photographed image and a long-exposure photographed image are input to the image input unit.
The area setting unit sets either one of the short-exposure photographed image and the long-exposure photographed image as the main area, and sets the other area as the sub-area.
Image processing device. The image input section where the captured image is input and
The captured image is divided into a plurality of areas, and the divided plurality of areas are divided into an area selected by a command from the photographer, an area for focusing, an area for adjusting exposure, and a face detection area. An area setting unit that sets at least one of the included areas as the main area and the other areas as the sub area.
A white balance gain acquisition unit that acquires a first white balance gain and a second white balance gain from the main region and the sub region, respectively.
A determination unit that determines the adjusted second white balance gain by adjusting the second white balance gain based on the first white balance gain and the second white balance gain.
A white balance correction unit that corrects the white balance of the main region and the sub region by the first white balance gain and the adjusted second white balance gain.
With
A short-exposure photographed image and a long-exposure photographed image are input to the image input unit.
The area setting unit sets either one of the short-exposure photographed image and the long-exposure photographed image as the main area, and sets the other area as the sub-area.
Image processing device. When the short-exposure photographed image and the long-exposure photographed image are input and a mode in which the short-exposure photographed image and the long-exposure photographed image are combined is selected, the white balance gain acquisition unit. The image processing apparatus according to claim 1 or 2, further comprising a second control unit that controls the acquisition of the first white balance gain and the second white balance gain. When the exposure of the short-exposure photographed image and the long-exposure photographed image is equal to or greater than the threshold value, the second control unit supplies the white balance gain acquisition unit with the first white balance gain and the second white. The image processing apparatus according to claim 3, wherein the control for acquiring the balance gain is performed. The determination unit is based on the first white balance gain and an adjustment threshold determined from the exposure difference between the short-exposure photographed image and the long-exposure photographed image input to the image input unit. The image processing apparatus according to any one of claims 1 to 4, which determines the adjusted second white balance gain. The white balance gain acquisition unit acquires the first white balance gain based on the short-exposure captured image having the main region or the long-exposure captured image, and captures the short-exposure having the sub-region. The image processing apparatus according to any one of claims 1 to 5, which acquires the second white balance gain based on an image or a captured image of the long exposure. An image determination unit for determining a first image and a second image in the short-exposure photographed image and the long-exposure photographed image is provided.
When acquiring the first white balance gain of the first image, the white balance gain acquisition unit obtains the first white based on the first degree of importance given to the main region and the sub region. When the balance gain is acquired and the second white balance gain of the second image is acquired, the second white balance gain is acquired based on the second importance degree which is the opposite of the first importance degree. The image processing apparatus according to any one of claims 1 to 6. The determination unit determines the adjusted second white balance gain based on the first white balance gain and the adjustment threshold determined according to the first white balance gain and the second white balance gain. The image processing apparatus according to any one of claims 1 to 7. Any of claims 1 to 7, wherein the determination unit determines the adjusted second white balance gain based on the first white balance gain and the adjustment threshold value determined according to the brightness difference of the captured image. The image processing apparatus according to item 1. The image input section where the captured image is input and
An area setting unit that divides the captured image into a plurality of areas and sets the divided plurality of areas as a main area to be watched by the photographer and a sub-area to other areas.
A white balance gain acquisition unit that acquires a first white balance gain and a second white balance gain from the main region and the sub region, respectively.
A determination unit that determines the adjusted second white balance gain by adjusting the second white balance gain based on the first white balance gain and the second white balance gain.
A white balance correction unit that corrects the white balance of the main region and the sub region by the first white balance gain and the adjusted second white balance gain.
With
The determination unit determines the adjusted second white balance gain based on the first white balance gain and the adjustment threshold value determined according to the first white balance gain and the second white balance gain. ,
Image processing device. The image input section where the captured image is input and
The captured image is divided into a plurality of areas, and the divided plurality of areas are divided into an area selected by a command from the photographer, an area for focusing, an area for adjusting exposure, and a face detection area. An area setting unit that sets at least one of the included areas as the main area and the other areas as the sub area.
A white balance gain acquisition unit that acquires a first white balance gain and a second white balance gain from the main region and the sub region, respectively.
A determination unit that determines the adjusted second white balance gain by adjusting the second white balance gain based on the first white balance gain and the second white balance gain.
A white balance correction unit that corrects the white balance of the main region and the sub region by the first white balance gain and the adjusted second white balance gain.
With
The determination unit determines the adjusted second white balance gain based on the first white balance gain and the adjustment threshold value determined according to the first white balance gain and the second white balance gain. ,
Image processing device. A first control unit that controls the white balance gain acquisition unit to acquire the first white balance gain and the second white balance gain based on the difference in brightness of the captured image input to the image input unit. The image processing apparatus according to any one of claims 1 to 11. The 12th aspect of claim 12, wherein the first control unit causes the white balance gain acquisition unit to acquire the first white balance gain and the second white balance gain when the brightness difference of the captured image is equal to or greater than a threshold value. Image processing equipment. According to claim 12 or 13, the first control unit causes the white balance gain acquisition unit to acquire the first white balance gain and the second white balance gain based on a command input from the operation unit. The image processing apparatus described. The image processing apparatus according to any one of claims 1 to 14, wherein the area setting unit has an area for adjusting the focus or exposure of the captured image as the main area. The white balance gain acquisition unit uses the acquired white balance gain as the first white balance gain and the second white balance gain according to the size of the region for which the white balance gain has been obtained or the brightness of the region for which the white balance gain has been obtained. The image processing apparatus according to any one of claims 1 to 14, wherein the white balance gain is used. The white balance gain acquisition unit acquires the first white balance gain based on the pixel value of the main region, and acquires the second white balance gain based on the pixel value of the sub region. The image processing apparatus according to any one of the above items. The step of inputting the short-exposure shot image and the long-exposure shot image,
A step of setting either one of the short-exposure photographed image and the long-exposure photographed image as a main area to be watched by the photographer and setting the other area as a sub-area.
A step of acquiring a first white balance gain and a second white balance gain from the main region and the sub region, respectively.
A step of adjusting the second white balance gain based on the first white balance gain and the second white balance gain to determine the adjusted second white balance gain.
A step of performing white balance correction on the main region and the sub region by the first white balance gain and the adjusted second white balance gain.
A program that causes a computer to execute image processing methods including. Steps where the captured image is input and
A step of dividing the captured image into a plurality of regions and setting the divided plurality of regions as a main region to be watched by the photographer and a sub region for other regions.
A step of acquiring a first white balance gain and a second white balance gain from the main region and the sub region, respectively.
Adjusted second white balance gain by adjusting the second white balance gain based on the first white balance gain and the adjustment threshold determined according to the first white balance gain and the second white balance gain. Steps to determine and
A step of performing white balance correction on the main region and the sub region by the first white balance gain and the adjusted second white balance gain.
A program that causes a computer to execute image processing methods including. The step of inputting the short-exposure shot image and the long-exposure shot image,
An area selected by a command from the photographer, an area for focusing, an area for adjusting exposure, and an area of either one of the short-exposure photographed image and the long-exposure photographed image. A step of setting at least one of the areas including the face detection area as the main area and setting the other area as the sub area.
A step of acquiring a first white balance gain and a second white balance gain from the main region and the sub region, respectively.
A step of adjusting the second white balance gain based on the first white balance gain and the second white balance gain to determine the adjusted second white balance gain.
A step of performing white balance correction on the main region and the sub region by the first white balance gain and the adjusted second white balance gain.
A program that causes a computer to execute image processing methods including. Steps where the captured image is input and
The captured image is divided into a plurality of areas, and the divided plurality of areas are divided into an area selected by a command from the photographer, an area for focusing, an area for adjusting exposure, and a face detection area. A step of setting at least one of the included areas as the main area and the other areas as the sub-areas,
A step of acquiring a first white balance gain and a second white balance gain from the main region and the sub region, respectively.
Adjusted second white balance gain by adjusting the second white balance gain based on the first white balance gain and the adjustment threshold determined according to the first white balance gain and the second white balance gain. Steps to determine and
A step of performing white balance correction on the main region and the sub region by the first white balance gain and the adjusted second white balance gain.
A program that causes a computer to execute image processing methods including.

Images