JP5659574B2 - Image processing apparatus, imaging apparatus, and image processing program - Google Patents

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

  One of the camera white balance adjustment functions is a function for presetting (presetting) the white balance gain based on the color of the image specified by the user as an achromatic object image. If it has changed, the preset white balance gain becomes inadequate, and it becomes necessary to reset it. For this reason, the camera of Patent Document 1 automatically adjusts the preset white balance gain based on the result of auto white balance calculation.

Japanese Patent No. 4069690

  By the way, the reason why the preset white balance gain becomes inappropriate is not limited to the case where the illumination color changes. For example, when the camera shooting conditions change and the spectral transmittance of the shooting lens, the spectral reflectance of the subject, the setting value of the image sensor, the setting value of the signal processing circuit, etc., the color balance of the image is subtly Since it changes, the preset white balance gain can be inappropriate.

  However, since the conventional general auto white balance calculation is easily influenced by the color inherent in the object in the object field, there is a possibility that it cannot cope with the subtle changes that occur in the color balance of the image. .

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus, an imaging apparatus, and an image processing program that can cope with subtle changes in image color balance.

The image processing apparatus according to the present invention includes a storage unit that stores information on a designated color that is a color designated in advance, and a pixel group that belongs to a predetermined color range in the vicinity of the designated color from a color image that is a target of white balance adjustment. A selection means for selecting a color distribution; a distribution calculation means for calculating a frequency distribution on the color coordinates of the selected pixel group; and a white balance adjustment amount to be applied to the color image based on the calculated frequency distribution Adjustment amount calculating means, and prior to the calculation of the white balance adjustment amount, the adjustment amount calculating means gives a greater weight to the frequency of each color included in the frequency distribution as the color frequency is closer to the specified color. Weight.

  An imaging apparatus according to the present invention includes an imaging element that acquires a color image, and an image processing apparatus according to the present invention that calculates a white balance adjustment amount to be applied to the acquired color image.

An image processing program according to the present invention includes a storage procedure for storing information on a designated color, which is a color designated in advance, and a pixel group belonging to a predetermined color range in the vicinity of the designated color from a color image to be subjected to white balance adjustment. A selection procedure for selecting a color, a distribution calculation procedure for calculating a frequency distribution on the color coordinates of the selected pixel group, and a frequency of a color close to the specified color with respect to the frequency of each color included in the frequency distribution weighted by the weight, based on the frequency distribution issued calculated, to execute the adjustment amount calculation step of calculating the white balance adjusting amount to be applied to the color image on a computer.

  According to the present invention, an image processing apparatus, an imaging apparatus, and an image processing program that can cope with subtle changes in the color balance of an image are realized.

It is a block diagram of an electronic camera. It is an operation | movement flowchart of CPU21 in a white balance preset mode. It is an operation | movement flowchart of CPU21 in a semiautomatic white balance imaging | photography mode. 6 is an operation flowchart of semi-automatic white balance calculation processing (step S25 in FIG. 3) by the white balance calculation circuit 15; It is a figure explaining the reference color range Ar. It is a figure explaining a pixel. It is a conceptual diagram of frequency distribution. It is a figure explaining weight distribution.

[Embodiment]
Hereinafter, an electronic camera will be described as an embodiment of the present invention.

  FIG. 1 is a configuration diagram of an electronic camera. As shown in FIG. 1, the electronic camera 1 includes an image sensor 11, a signal processing circuit 12, a buffer memory 13, an image processing circuit 14, a white balance arithmetic circuit 15, a display control circuit 16, an internal monitor 17, a compression / expansion circuit 18, A card interface 19, a CPU 21, an imaging circuit 22, an input device 23, an external connection terminal 24, and the like are provided. Among these, the buffer memory 13, the image processing circuit 14, the white balance arithmetic circuit 15, the display control circuit 16, the compression / decompression circuit 18, the card interface 19, and the CPU 21 are connected to a common bus.

  The image pickup device 11 is a color image pickup device that picks up an object scene image formed by the taking lens 2 and is, for example, a single-plate color image pickup device having a Bayer array color filter.

  The signal processing circuit 12 sequentially performs analog signal processing and A / D conversion processing on the analog image signal acquired by the imaging device 11 through imaging, and converts the analog image signal into a digital image signal.

  The buffer memory 13 sequentially accumulates digital image signals output from the signal processing circuit 12. When this accumulation is performed over the charge reading period for one frame of the image sensor 11, digital image data for one frame is accumulated in the buffer memory 13. Hereinafter, digital image data for one frame is simply referred to as “image”.

  The image processing circuit 14 performs a series of image processing such as white balance adjustment processing, color interpolation processing (so-called Debayer processing), gradation conversion processing, and KNEE processing on the image designated by the CPU 21.

  The white balance calculation circuit 15B calculates a white balance gain to be applied to the image based on the image designated by the CPU 21, and sets the calculated white balance gain in the image processing circuit 14.

  The display control circuit 16 has a display memory, and displays an image on the internal monitor 17 by sending the image written in the display memory to the internal monitor 17. Note that during the period in which an external monitor is connected to the external connection terminal 24, the display destination of the image by the display control circuit 16 is not the internal monitor 17 but the external monitor. However, for simplicity, it is assumed that an external monitor is not connected to the external connection terminal 24.

  The internal monitor 17 is a monitor that is provided on the back surface of the electronic camera 1 and used when a user confirms an image acquired by the electronic camera 1. This monitor is composed of, for example, an LCD panel.

  The external connection terminal 24 is a terminal for connecting the display control circuit 16 to an external monitor. The external connection terminal 24 is configured by, for example, a USB port.

  The compression / decompression circuit 18 performs data compression processing on the image or the image group stored in the buffer memory 13. The compression / decompression circuit 18 can also perform data decompression processing on an image or image group (image file or moving image file) that has been subjected to data compression.

  The card interface 19 writes a data-compressed image (image file or moving image file) to the card memory 20 installed in the card slot of the electronic camera 1 or an image (image file) written in the card memory 20. Or a moving image file). The card memory 20 is a portable storage medium.

  The imaging circuit 22 adjusts the focal length (zoom position) of the photographic lens 2, the focusing distance of the photographic lens 2, the aperture value of the photographic lens 2, and the like by giving a drive signal to the photographic lens 2. Further, the imaging circuit 22 controls the drive timing of both by providing a drive signal to the image sensor 11 and the signal processing circuit 12. The imaging circuit 22 controls the charge accumulation time per frame of the imaging device 11. Note that the shutter speed of the electronic camera 1 can be set by a combination of the charge accumulation time of the image sensor 11 and the opening period of a mechanical shutter (not shown), but can also be set only by the charge accumulation time of the image sensor 11.

  The CPU 21 controls each of the above units in accordance with a user instruction input via the input device 23 and firmware written in advance in the internal memory (ROM) of the CPU 21. The firmware of the CPU 21 is appropriately updated via a communication circuit (not shown) (known network terminal or the like) mounted on the electronic camera 1.

  The input device 23 is an operation member such as a release button provided on the top of the electronic camera 1 or a multi-selector provided on the back of the electronic camera 1. The user can set the electronic camera 1 to the white balance preset mode via the input device 23. The white balance preset mode is a mode for the user to set the white balance gain prior to shooting.

  In addition, the user can set the electronic camera 1 to the semi-auto white balance shooting mode via the input device 23. The semi-auto white balance shooting mode is a mode in which the electronic camera 1 automatically adjusts the white balance gain based on the white balance gain (preset white balance gain) set in the white balance preset mode.

  FIG. 2 is an operation flowchart of the CPU 21 in the white balance preset mode. Hereafter, each step of FIG. 2 is demonstrated in order.

  Step S11: The CPU 21 determines whether or not the release button has been fully pressed. If the release button has been fully pressed, the process proceeds to step S12.

  Step S12: The CPU 21 drives the image pickup device 11 for one frame in the draft mode (or frame mode) via the image pickup circuit 22. As a result, a preset image is acquired. The preset image is stored in the buffer memory 13 via the signal processing circuit 12.

  Step S13: The CPU 21 calculates the average chromaticity of the preset image stored in the buffer memory 13, and calculates a white balance gain for making the average chromaticity an achromatic color.

  Step S14: The CPU 21 stores the average chromaticity calculated in step S13 as a preset illumination color, stores the white balance gain calculated in step S13 as a preset white balance gain, and ends the flow. If there are preset illumination colors and preset white balance gains already stored, they are updated in this step.

  Accordingly, when the user images a gray or white object (achromatic object) while the electronic camera 1 is set to the white balance preset mode, the illumination color of the scene where the achromatic object is placed is set as the preset illumination color. A white balance gain suitable for the illumination color can be designated to the electronic camera 1 as a preset white balance gain. These preset illumination colors and preset white balance gain are used in the semi-auto white balance shooting mode.

  FIG. 3 is an operation flowchart of the CPU 21 in the semi-auto white balance shooting mode. Hereafter, each step of FIG. 3 is demonstrated in order.

  Step S21: The CPU 21 captures shooting parameters of the electronic camera 1 (aperture value, shutter speed, focus distance, zoom position, etc.), image processing parameters of the electronic camera 1 (white balance gain of white balance adjustment processing, gradation conversion processing) A gradation conversion curve, a KNEE characteristic curve for KNEE processing, etc.) are set to a predetermined combination. Of these, the white balance gain is set to the same value as the preset white balance gain stored at the present time.

  Step S22: The CPU 21 starts to continuously drive the image pickup device 11 in the draft mode via the image pickup circuit 22. As a result, the image sensor 11 starts acquiring a through image. The CPU 21 extracts the AE evaluation value from the through image output from the signal processing circuit 12 at this time, and the shutter speed of the electronic camera 1 and the photographing lens 22 through the imaging circuit 22 so that the AE evaluation value becomes appropriate. The combination with the aperture value is controlled (AE control). In parallel with this, the CPU 21 extracts the AF evaluation value from the through image output from the signal processing circuit 12, and sets the in-focus distance of the photographic lens 22 via the imaging circuit 22 so that the AF evaluation value is appropriate. Control (AF control). Note that when a zoom adjustment instruction is input from the input device 23 during the through image acquisition period, the CPU 21 adjusts the zoom position of the photographing lens 2 in accordance with the instruction.

  Step S23: The CPU 21 determines whether or not the release button has been fully pressed. If the release button has been fully pressed, the process proceeds to step S24.

  Step S24: The CPU 21 drives the image pickup device 11 for one frame in the frame mode via the image pickup circuit 22. As a result, a detailed image for storage is acquired. This detailed image is stored in the buffer memory 13 via the signal processing circuit 12.

  Step S25: The CPU 21 gives the preset illumination color stored at the present time to the white balance calculation circuit 15 and also gives the white balance calculation circuit 15 a semi-automatic white balance calculation start instruction regarding the detailed image on the buffer memory 13. The white balance calculation circuit 15 executes a semi-auto white balance calculation process (described later) regarding a detailed image based on the preset illumination color. The white balance gain calculated by the semi-auto white balance calculation process is output to the CPU 21.

  Step S <b> 26: The CPU 21 sets the white balance gain output from the white balance calculation circuit 15 in the image processing circuit 14. As a result, the white balance gain of the electronic camera 1 is reset to an optimum value at the present time.

  Step S27: The CPU 21 issues an instruction to the image processing circuit 14 to execute image processing. The image processing circuit 14 performs the above-described series of image processing on the detailed images stored in the buffer memory 13. Note that the white balance gain of the white balance adjustment process executed in this step is the white balance gain reset in step S26.

  Step S28: The CPU 21 instructs the compression / decompression circuit 18 to perform data compression processing on the detailed image after image processing. The compression / decompression circuit 18 creates an image file such as a JPEG file by compressing the detailed image. The CPU 21 writes the image file into the card memory 20 via the card interface 19 and ends the flow. As a result, the detailed image is saved as an image file.

  FIG. 4 is an operation flowchart of the semi-auto white balance calculation process (step S25 in FIG. 3) by the white balance calculation circuit 15. Hereafter, each step of FIG. 4 is demonstrated in order.

Step S25-1: The white balance calculation circuit 15 sets a reference color range on the color coordinates based on the preset illumination color given from the CPU 21. As shown in FIG. 5, reference color range Ar is a color range in the vicinity of chromaticity coordinates P ₀ of the preset illumination color, their size and shape is predetermined. For example, the reference color range Ar is set in a square area of a predetermined size around the chromaticity coordinates P _0.

Step S25-2: The white balance calculation circuit 15 maps each pixel of the detailed image onto the chromaticity coordinates according to the chromaticity, and the mapping destination is the reference area Ar among all the pixels of the detailed image. A pixel group is extracted. When the color filter array described above is a Bayer array, each pixel of the detailed image corresponds to the unit block _BP shown in FIG. The unit block _BP corresponds to the minimum unit of the color filter, and has two G pixels, one R pixel, and one B pixel.

  Then, the white balance calculation circuit 15 calculates a frequency distribution on the chromaticity coordinates of the extracted pixel group. However, when calculating the frequency distribution, the white balance calculation circuit 15 equally divides the reference area Ar into a plurality of blocks (color blocks) as indicated by dotted lines in FIG. 5, for example, and counts the pixel frequency for each color block. To do. Further, at that time, the white balance calculation circuit 15 counts the frequency more frequently for the higher luminance pixels (that is, assigns a larger weight to the frequency of the higher luminance pixels). Thereby, for example, a frequency distribution as shown in FIG. 7 is obtained.

  Step S25-3: The white balance calculation circuit 15 performs a smoothing filter process on the frequency distribution (FIG. 7) calculated in Step S25-2. The smoothing filter process is a so-called blur filter process in which the frequency of each color block is replaced with the average value (or weighted average value) of the frequency of the color block and the frequency of the surrounding color blocks. . As a result, each frequency of the frequency distribution is brought close to the surrounding frequencies.

  Step S25-4: The white balance calculation circuit 15 performs a weighting process on the frequency distribution after the smoothing filter process. This weighting process is a process of assigning a weight for each color block to the frequency distribution after the smoothing filter process, and the weight assigned in advance for each color block is as shown in FIG. A color block closer to the center of the reference color range Ar is assigned a greater weight. Therefore, according to the weighting process in this step, a greater weight is given to the frequency of the color closer to the preset illumination color.

  Step S25-5: The white balance calculation circuit 15 refers to the frequency distribution after the weighting process and finds the color block (specific color block) having the maximum frequency. Then, the white balance calculation circuit 15 regards the chromaticity of the center coordinates of the specific color block as the degree of color cast occurring in the detailed image (hereinafter referred to as “corrected color”), and the corrected color. The white balance gain for making the color achromatic is calculated. When the white balance gain is output to the CPU 21 as a calculation result, the white balance calculation circuit 15 ends the flow of the semi-auto white balance calculation process. Therefore, this white balance gain is applied to the detailed image. However, when the maximum frequency of the frequency distribution after the weighting process is less than the threshold, the white balance calculation circuit 15 in this step does not calculate the white balance gain for making the correction target color achromatic, The value of the preset white balance gain is output as it is to the CPU 21 as the calculation result (the description of step S25-5 above).

As described above, in the semi-automatic white balance photographing (FIGS. 3 and 4) of the present embodiment, the reference color range Ar is set in the vicinity of the chromaticity coordinate P ₀ of the preset illumination color, and each pixel of the detailed image is changed to the reference color range Ar. A pixel group to which the pixel group belongs is selected, a frequency distribution on the chromaticity coordinates of the pixel group is calculated, and a white balance gain to be applied to the detailed image is reset based on the frequency distribution. Therefore, among the pixels of the detailed image, pixels having a color that is more than a certain distance from the preset illumination color do not affect the resetting result.

  Therefore, suppose that the shooting conditions of the electronic camera 1 (spectral transmittance of the photographing lens, spectral reflectance of the subject) during the period from the time of the white balance preset (FIG. 2) to the time of the semi-automatic white balance shooting (FIGS. 3 and 4) Even if the setting value of the image sensor, the setting value of the signal processing circuit, etc.) change, the white balance gain of the electronic camera 1 can be adapted to the change.

Further, in the semi-automatic white balance photography (FIGS. 3 and 4) of this embodiment, the frequency of each color in the calculated frequency distribution is weighted with a greater weight as the frequency of the color closer to the chromaticity coordinate P ₀ is reset. The preset white balance gain can be appropriately reflected with respect to the white balance gain.

  Further, in the semi-automatic white balance shooting (FIGS. 3 and 4) of the present embodiment, the color of each part of the detailed image is evaluated for each pixel when the white balance gain is reset, so that the problem of color mixture can be avoided. it can.

  The problem of color mixing is a problem in which the frequency of chromaticity that does not actually exist on the detailed image adversely affects the white balance gain. The chromaticity of each part of the detailed image is not evaluated for each pixel. This is a problem that may occur when evaluation is performed for each block (for each image block) on the image. This is because when evaluating for each pixel block, if there are a plurality of different chromaticities in one image block, the average chromaticity of those chromaticities is regarded as the chromaticity of the image block. This is because the average chromaticity is a chromaticity that does not actually exist in the image block.

[Supplement of embodiment]
In step S25-1 of the above embodiment has been set a reference color range Ar in the vicinity of the chromaticity coordinates P ₀ of the preset illumination color, corrects the chromaticity coordinates P ₀ before setting of the reference color range Ar May be. In the correction, it is preferable to use a correction coefficient corresponding to the current shooting conditions (spectral transmittance of the shooting lens, spectral reflectance of the subject, setting value of the image sensor, setting value of the signal processing circuit, etc.).

  In step S25-2 of the above-described embodiment, the calculation target of the white balance calculation circuit 15 is the detailed image itself. However, in order to reduce the calculation load, the thinned-out detailed image (that is, the size of the detailed image) (Reduced version). In this way, it is possible to reduce the calculation load while avoiding the above-mentioned color mixing problem.

  In the electronic camera 1 according to the above-described embodiment, the information stored in the white balance preset mode is the information on both the preset white balance gain and the preset illumination color, but only one of the information may be stored. This is because the white balance gain and the preset illumination color can be calculated from one to the other. Therefore, when the electronic camera 1 stores only one information stored in the white balance preset mode, the other may be calculated appropriately.

  In the white balance preset mode of the above-described embodiment, as a method for the user to specify the preset white balance gain or the preset illumination color to the electronic camera 1, a method in which the user photographs an achromatic object with the electronic camera 1 is adopted. However, other methods may be adopted. For example, a method in which the user designates one of a plurality of predetermined light sources, a method in which the user designates the color temperature of the illumination color of the object scene, or the like may be employed.

  Further, in the above-described embodiment, the case where the present invention is applied to still image shooting for recording has been described. However, for confirmation video shooting before recording still image shooting or for confirmation before shooting moving image recording. The present invention can be similarly applied to the moving image shooting.

  However, when applied to video shooting, the frequency distribution is more controlled than when the white balance gain is feedback controlled (that is, the image from which the frequency distribution is extracted differs from the image from which the white balance is to be adjusted). The use effect of the present invention is considered to be higher when the image that is the extraction source and the image that is the target of white balance adjustment are common.

  In the electronic camera 1 of the above-described embodiment, the white balance gain resetting process (the processes in steps S25 to S28) is executed at the time of shooting. However, a RAW image file (= image file before image processing) saved in the past is used. ) May be executed during the development processing. However, in this case, when the RAW image file is stored, the electronic camera 1 needs to add information on the preset white balance gain or the preset illumination color at that time to the RAW image file.

  Such development processing (development processing including the processing of steps S25 to S28) is an image processing device provided separately from the electronic camera 1, such as an electronic photo frame, a printer, a general-purpose computer (image processing software). It is also possible to execute the program on a computer equipped with When causing the image processing apparatus to execute the development process, the computer program for the development process is installed in the image processing apparatus via a communication network or a storage medium.

DESCRIPTION OF SYMBOLS 1 ... Electronic camera, 11 ... Imaging device, 12 ... Signal processing circuit, 13 ... Buffer memory, 14 ... Image processing circuit, 15 ... White balance arithmetic circuit, 16 ... Display control circuit, 17 ... Internal monitor, 18 ... Compression / decompression Circuit, 19 ... Card interface, 21 ... CPU, 22 ... Imaging circuit, 23 ... Input device, 24 ... External connection terminal

Claims (6)

Storage means for storing information on a designated color that is a color designated in advance;
Selecting means for selecting a pixel group belonging to a predetermined color range in the vicinity of the designated color from a color image to be subjected to white balance adjustment;
Distribution calculating means for calculating a frequency distribution on the color coordinates of the selected pixel group;
Adjustment amount calculation means for calculating a white balance adjustment amount to be applied to the color image based on the calculated frequency distribution;
Equipped with a,
The adjustment amount calculating means includes
Prior to the calculation of the white balance adjustment amount, the frequency of each color included in the frequency distribution is weighted with a greater weight as the frequency of the color closer to the specified color . The image processing apparatus according to claim 1 .
The adjustment amount calculating means includes
An image processing apparatus that performs smoothing filter processing on the frequency distribution prior to calculating the white balance adjustment amount. The image processing apparatus according to claim 1 or 2 ,
The selection means is:
An image processing apparatus that performs thinning processing on the color image prior to selection of the pixel group. In the image processing apparatus according to any one of claims 1 to 3 ,
The distribution calculating means includes
In calculating the frequency distribution, the frequency is counted for each block on the color coordinates. An image sensor for acquiring a color image;
The image processing apparatus according to any one of claims 1 to 4 , wherein a white balance adjustment amount to be applied to the acquired color image is calculated;
An imaging apparatus comprising: A storage procedure for storing information on a designated color that is a color designated in advance;
A selection procedure for selecting a pixel group belonging to a predetermined color range in the vicinity of the specified color from a color image to be subjected to white balance adjustment;
A distribution calculation procedure for calculating a frequency distribution on color coordinates of the selected pixel group;
Weighted with larger weight as the frequency of a color close to the designated color with respect to frequency of each color included in the frequency distribution, based on the frequency distribution issued calculated, the white balance adjusting amount to be applied to the color image An adjustment amount calculation procedure to be calculated;
An image processing program for causing a computer to execute.

Images