JP6302555B2 - Image processing apparatus, imaging apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an imaging apparatus, an image processing method, and a program, and more particularly, to image processing of image data captured and acquired under flash light emission.

  An image (hereinafter, also referred to as “flash non-flash image”) obtained by shooting performed without flash light (hereinafter also referred to as “flash non-flash shooting”) is basically only from ambient light. The image is affected by the light. On the other hand, an image (hereinafter also referred to as “flash emission image”) obtained by shooting performed under flash light emission (hereinafter also referred to as “flash emission shooting”) is affected by flash light in addition to ambient light. Received image.

  In this flash emission image, “a portion of a main subject or the like where the influence of the flash light is large” and “a portion of a background or the like where the influence of the flash light is small or substantially unaffected” are mixed. That is, the main subject or the like present near the flash light emitting unit is an image affected by the flash light in addition to the environmental light in the flash light emission image. On the other hand, a background or the like that does not exist near the flash light emitting portion is an image that is mainly affected by ambient light in the flash light emitting image. Thus, in one flash emission image, there are a plurality of image portions having substantially different types of light sources. For this reason, when uniformly adjusting the overall white balance of the flash emission image, it is difficult to simultaneously optimize the white balance of a plurality of image portions having substantially different types of light sources.

  Therefore, in the white balance processing of the flash emission image, the image portion such as the background and the image portion such as the main subject are regarded as separate light source regions, and the color balance of the image is improved by performing white balance processing on each image portion individually. Things have been done.

  For example, Patent Document 1 discloses an image processing apparatus that calculates a white balance map from captured image data and monitor image data. This white balance map is appropriate for each pixel area in the captured image data that is brightened by the flash (flash) and pixels that are dark without being exposed to the flash. This is data for adjusting the white balance.

  Patent Document 2 discloses an imaging apparatus that performs image processing for the purpose of realizing stable white balance correction in white balance processing of an image acquired by flash (flash) flash photography. In this imaging apparatus, the white balance correction value is calculated from the flash emission image data, the flash non-flash image data, and the past image data when the flash is not flashing.

JP 2012-19293 A JP 2010-193002 A

  As described above, for example, using flash emission image data and flash non-emission image data, based on a plurality of captured images with the same subject but with different image portions such as the main subject, A method of separating an “image portion affected by light” from an “image portion not affected by flash light” is considered.

  However, even when flash light is evenly applied to the subject during flash photography, the brightness of the actual captured image changes depending on the color of the subject. Therefore, it is difficult to accurately estimate the influence of the flash light on the brightness of the flash emission image only from the difference data between the flash emission image data and the flash non-emission image data.

  As an example, flash photography and non-flash photography were performed using each of the red color chart, green color chart, and blue color chart pasted on a flat plate as subjects. In these shootings, a digital camera (imaging device) equipped with an image sensor having an RGB (red, green, blue) color filter is used, and a red color chart, a green color chart, and a blue color are located at approximately equal distances from the digital camera. Each of the color charts was placed. Then, based on the luminance value calculated from the difference data between the flash emission image and the flash non-emission image, the relationship between the brightness (luminance value) between the color charts for the image component by the flash light was obtained. Here, the “luminance value (Y)” is calculated by obtaining difference data between the flash emission image and the flash non-emission image based on RGB data (RGB pixel value), and calculating the RGB data of the difference data as “Y = 0.3 * R + 0.6 * G + 0.1 * B ", and the result shown in FIG. 32 was obtained.

  FIG. 32 is a graph illustrating the luminance relationship of “difference data between flash emission image and flash non-emission image” regarding the red color chart, the green color chart, and the blue color chart. The vertical axis in FIG. 32 indicates the magnitude of the luminance value (Y) of “difference data between the flash emission image and the flash non-emission image” of each color chart.

  Assuming that the brightness of the captured image does not depend on the color of the subject, the “flash emission image and flash non-emission image” of the red color chart, green color chart, and blue color chart pasted on the flat plate It is considered that the luminance values of the “difference data between” are equal to each other. However, as is apparent from FIG. 32, the luminance values of the “difference data between the flash emission image and the flash non-emission image” of the red color chart, the green color chart, and the blue color chart are actually mutual. It can be seen that the brightness of the flash emission image depends on the color of the subject.

  Therefore, the difference data between the flash emission image and the flash non-emission image is data in which the influence depending on the color of the subject itself is reflected in addition to the influence of the flash light. The “data indicating the influence of the flash light in the flash emission image” cannot always be obtained accurately from the “data from which the influence depending on the color of the subject is not removed”. Therefore, from the viewpoint of performing various image processing such as white balance processing with high accuracy, it is preferable to compensate for the brightness difference caused by the color of the subject in the image data to be processed.

  However, conventionally, there has not been proposed a method for performing various image processing such as white balance processing, paying attention to the difference in brightness caused according to the color of the subject. For example, in the image processing apparatus disclosed in Patent Document 1, white balance processing is performed after classifying photographed image data into a bright pixel region and a dark pixel region. However, there is a difference in brightness brought about according to the color of the subject. Not much attention. In the white balance processing disclosed in Patent Document 2, the white balance correction value is calculated from the strobe light emission image data and the strobe non-light emission image data. However, the difference in brightness caused according to the color of the subject in the white balance processing is as follows. It is not considered at all.

  The present invention has been made in view of the above circumstances, and provides an image processing technique capable of compensating for a difference in brightness caused by the color of a subject in image data to be processed and a related technique. With the goal.

  One embodiment of the present invention is a flash image component acquisition that acquires flash image component data indicating an image component of a subject by flash light based on first image data acquired by photographing the subject under flash light emission. A flash correction data acquisition unit for acquiring flash correction data reflecting the brightness difference of the flash image component data brought about according to the color of the subject based on the flash image component data, and the flash image component data and the flash correction data And a flash brightness correction unit that acquires flash image correction component data in which a difference in brightness of flash image component data caused according to the color of the subject is compensated.

  According to this aspect, based on “flash image component data indicating the image component of the subject by flash light” and “flash correction data reflecting the difference in brightness of the flash image component data brought about according to the color of the subject” The flash image correction component data in which the difference in brightness of the flash image component data is compensated (corrected) is acquired. By performing various types of image processing on the first image data based on the flash image correction component data, it is possible to obtain a processing result that compensates for the difference in the brightness of the flash image component data caused according to the color of the subject. .

  The “flash image component data indicating the image component of the subject by flash light” as used herein is, for example, ambient light from flash emission image data (first image data) acquired under the influence of flash light and ambient light. It can be expressed by data that removes the influence of. The “flash image component data” can be expressed by data that directly or indirectly indicates the influence of the flash light. For example, the “flash image component data” is affected by light having a color component ratio equivalent to that of the flash light used for photographing. It may be represented by data.

  In addition, “environment light” here refers to a light component other than flash light that affects a subject at the time of shooting, and “environment light” may be constituted by light from a single light source. The light from these light sources may be mixed to constitute “ambient light”.

  Desirably, the flash image component acquisition unit, from the first difference data representing the difference between the first image data and the second image data acquired by photographing the subject under non-flash light emission, Get flash image component data.

  According to this aspect, “first image data acquired by shooting a subject under flash light emission” and “second image acquired by shooting a subject under non-flash light emission”. With the first difference data representing the difference from “data”, “flash image component data directly indicating the image component of the subject by the flash light” can be accurately represented.

  Desirably, the flash image component acquisition unit captures the first image data acquired by photographing the subject under the light emission of the first light emission amount and the second light emission different from the first light emission amount. Flash image component data is acquired from first difference data representing a difference from second image data acquired by photographing a subject under the amount of flash light emitted.

  According to this aspect, “the first image data acquired by photographing the subject under the light emission of the first light emission amount” and “the subject under the light emission of the second light emission amount of the flash light”. The “flash image component data indicating the image component of the subject by the flash light” can be accurately acquired by the first difference data representing the difference from the “second image data acquired by photographing”. In this aspect, “flash image component data indirectly indicating an image component of a subject by flash light” can be represented by “data affected by light having a color component ratio equivalent to that of flash light”.

  Preferably, the flash image component acquisition unit acquires flash image component data by applying a flash light white balance gain to the first difference data.

  According to this aspect, the “flash image component data indicating the image component of the subject by the flash light” is applied with the white balance gain for the flash light and the white balance is adjusted, so that the proper color tone inherent to the subject is reproduced. The obtained flash image component data can be acquired.

  Preferably, the flash correction data acquisition unit acquires color ratio data representing a color component ratio of the flash image component data in each of the plurality of processing blocks of the flash image component data divided into the plurality of processing blocks, and the flash image component Based on the color ratio data of the data, flash correction data reflecting the difference in brightness between the plurality of processing blocks of the flash image component data brought about according to the color of the subject is acquired.

  According to this aspect, the flash correction data can be acquired for each processing block.

  Desirably, the flash correction data is applied to the first brightness data indicating the brightness of the flash image component data to compensate for the difference in the brightness of the flash image component data caused according to the color of the subject. The flash correction data acquisition unit acquires flash correction data for each of the plurality of processing blocks of the flash image component data.

  According to this aspect, the flash correction can be applied to the first brightness data indicating the brightness of the flash image component data to compensate for the difference in the brightness of the flash image component data caused according to the color of the subject. Data can be acquired.

  Preferably, the flash lightness correction unit acquires first lightness data in each of the plurality of processing blocks based on the flash image component data, and adds flash correction data to the first lightness data for each of the plurality of processing blocks. By applying this, flash image correction component data is acquired.

  According to this aspect, the flash image correction is applied to the first lightness data indicating the lightness of the flash image component data, and the difference in the lightness of the flash image component data caused according to the color of the subject is compensated. Component data is acquired.

  Preferably, the flash correction data acquisition unit acquires flash correction data for each of the plurality of processing blocks based on achromatic color ratio data.

  According to this aspect, the flash correction data is acquired based on the achromatic color. Achromatic colors are “white”, “black” and “color obtained by mixing white and black”. In this specification, “gray” means “white”, “black” and “white”. It is a concept that can include “color obtained by mixing with black”. In the color space using the primary color, the achromatic color has the same primary color amount. For example, in the RGB color space, the achromatic color is “R data = G data = B data (that is,“ R data: G data: B data = 1: 1: 1 ")".

  Preferably, the first image data is acquired by an image sensor including a plurality of color filters, the flash image component data includes color component data for each of the plurality of colors, and the flash correction data acquisition unit includes the color component data. Based on this, the color ratio data of the flash image component data in each of the plurality of processing blocks is acquired.

  According to this aspect, the color ratio data of the flash image component data can be easily acquired based on the color of the color filter of the image sensor used for photographing.

  Preferably, the color ratio data of the flash image component data is a ratio of the color component data of the flash image component data for each of the plurality of colors to the sum of the color component data of the plurality of colors of the flash image component data in each of the plurality of processing blocks. Based on.

  According to this aspect, the color ratio data of the flash image component data can be acquired as data normalized based on the sum of the color component data of the plurality of colors of the flash image component data in each of the plurality of processing blocks.

  Desirably, the flash correction data acquisition unit is configured so that each of the plurality of processing blocks is based on arithmetic processing for deriving lightness index data indicating the lightness of the flash image component data corresponding to the color of the subject from the color ratio data of the flash image component data. The lightness index data of the flash image component data in each of the plurality of processing blocks is obtained from the color ratio data of the flash image component data in each of the plurality of colors with respect to the sum of the color component data of the plurality of colors in each of the plurality of processing blocks The flash image component in each of the plurality of processing blocks for the lightness index data for the achromatic color is obtained from the color ratio data of the achromatic color having the same ratio of the color component data for the achromatic color based on the arithmetic processing. Flash based on the ratio of data brightness index data It acquires flash correction data for each of the plurality of processing blocks of the image component data.

  According to this aspect, the flash correction data is acquired based on the ratio of the “brightness index data of the flash image component data” to the “brightness index data regarding the achromatic color”.

  Note that the “arithmetic processing” used for deriving the lightness index data may be any process capable of calculating data indicating the lightness directly or indirectly, and uses a predetermined arithmetic expression or reference table. Is possible. Therefore, for example, an arithmetic expression used for calculating the “luminance value” may be used in the “arithmetic processing” here, and the “luminance value” calculated by such an arithmetic expression may be used as “lightness index data”. Good.

  Preferably, each of the plurality of processing blocks is configured by a single pixel.

  According to this aspect, the flash correction data can be acquired in pixel units. The “pixel” here is a structural unit of flash image component data, and can be based on the pixels constituting the image sensor.

  Desirably, each of the plurality of processing blocks includes a plurality of pixels.

  According to this aspect, the flash correction data can be acquired in units of a plurality of pixels. For example, even if each pixel data constituting the flash image component data includes an unexpected noise component, the effect of such a noise component can be reduced by performing flash correction data acquisition processing in units of multiple pixels. It can be substantially reduced.

  Desirably, the image sensor used for photographing the first image data is a single-plate image sensor including a plurality of color filters, and a plurality of color filters have a basic arrangement of specific color patterns repeatedly arranged. The plurality of pixels constituting each of the plurality of processing blocks corresponds to the pixels constituting the basic array.

  According to this aspect, the flash correction data can be acquired based on the basic arrangement of the color filters of the image sensor, and the flash correction data acquisition process can be easily performed.

  Preferably, the image processing apparatus includes an environment light image component acquisition unit that acquires environment light image component data indicating an image component of the subject by the environment light, and an environment brought about according to the color of the subject based on the environment light image component data. An ambient light correction data acquisition unit that acquires ambient light correction data that reflects the difference in brightness of the light image component data, and an ambient light image that is generated according to the color of the subject based on the ambient light image component data and the ambient light correction data And an ambient light brightness correction unit that acquires ambient light image correction component data in which the difference in brightness of the component data is compensated.

  According to this aspect, “environmental light image component data indicating the image component of the subject by ambient light” and “environmental light correction data reflecting the difference in brightness of the ambient light image component data brought about according to the color of the subject” and Based on the above, ambient light image correction component data in which the difference in brightness of the ambient light image component data is compensated is acquired. By performing various types of image processing on the first image data based on the ambient light image correction component data, a processing result that compensates for the difference in brightness of the ambient light image component data caused according to the color of the subject is obtained. Can do.

  The “environment light image component data indicating the image component of the subject by ambient light” here is, for example, flash data from flash emission image data (first image data) acquired under the influence of flash light and ambient light. It can be represented by data excluding the influence of light. This “ambient light image component data” can be expressed by data influenced by light having a color component ratio equivalent to that of the ambient light used to capture and acquire the data of the flash emission image (first image data). it can.

  Preferably, the ambient light image component acquisition unit acquires the ambient light image component data using the second image data acquired by photographing the subject under non-flash light as basic data.

  According to this aspect, the ambient light image component data can be obtained with high accuracy based on the second image data.

  The ambient light image component data may be the second image data itself or other data derived from the second image data.

  Preferably, the ambient light image component acquisition unit represents a difference between the first image data, the first image data, and the second image data acquired by photographing the subject under non-flash light emission. The ambient light image component data is acquired using the second difference data representing the difference from the first difference data as basic data.

  According to this aspect, the ambient light image component data can be acquired by removing the flash image component data represented by the first difference data from the first image data that is the data of the flash emission image.

  The ambient light image component acquisition unit is different from the first image data acquired by photographing the subject under the light emission of the flash light having the first light emission amount, the first image data, and the first light emission amount. Basic data is derived based on the first difference data representing the difference from the second image data acquired by photographing the subject under the light emission of the second light emission amount, and the basic data is Based on this, ambient light image component data may be acquired.

  Preferably, the ambient light image component acquisition unit acquires color ratio data representing the color component ratio of the basic data in each of the plurality of processing blocks, and the color ratio data and basic information of the flash image component data in each of the plurality of processing blocks. The ambient light white balance gain is acquired based on the data color ratio data, and the ambient light image component data is acquired by applying the ambient light white balance gain to the basic data.

  According to this aspect, the “ambient light image component data indicating the image component of the subject by ambient light” is adjusted by applying the white balance gain for ambient light to adjust the white balance, so that the appropriate color tone of the subject is reproduced. The ambient light image component data can be acquired.

  Desirably, when the ambient light white balance gain is applied to the color ratio data of the basic data in each of the plurality of processing blocks, the color ratio data of the ambient light image component data in each of the plurality of processing blocks is derived.

  According to this aspect, the color ratio data of the ambient light image component data is derived using the ambient light white balance gain.

  Preferably, the ambient light correction data acquisition unit acquires color ratio data representing the color component ratio of the ambient light image component data in each of the plurality of processing blocks, and based on the color ratio data of the ambient light image component data, the subject Ambient light correction data reflecting the difference in brightness between the plurality of processing blocks of the ambient light image component data brought about according to the color of the ambient light is acquired.

  According to this aspect, ambient light correction data can be acquired in units of processing blocks.

  Preferably, the ambient light correction data is applied to the second brightness data indicating the brightness of the ambient light image component data to compensate for the difference in brightness of the ambient light image component data caused according to the color of the subject. The ambient light correction data acquisition unit acquires ambient light correction data for each of the plurality of processing blocks of the ambient light image component data.

  According to this aspect, by applying to the second brightness data indicating the brightness of the ambient light image component data, it is possible to compensate for the difference in brightness of the ambient light image component data caused according to the color of the subject. Ambient light correction data can be acquired.

  Desirably, the ambient light brightness correction unit acquires second brightness data in each of the plurality of processing blocks based on the ambient light image component data, and converts the ambient light to the second brightness data for each of the plurality of processing blocks. Ambient light image correction component data is acquired by applying the correction data.

  According to this aspect, the ambient light correction data is applied to the second brightness data indicating the brightness of the ambient light image component data, and the difference in the brightness of the ambient light image component data caused according to the color of the subject is compensated. Ambient light image correction component data is acquired.

  Preferably, the ambient light correction data acquisition unit acquires ambient light correction data for each of the plurality of processing blocks based on achromatic color ratio data.

  According to this aspect, the ambient light correction data is acquired based on the achromatic color.

  Preferably, the first image data is captured by an image sensor including a plurality of color filters, the ambient light image component data includes color component data for each of the plurality of colors, and the ambient light correction data acquisition unit includes the color component Based on the data, the color ratio data of the ambient light image component data in each of the plurality of processing blocks is acquired.

  According to this aspect, the color ratio data of the ambient light image component data can be easily obtained based on the color of the color filter of the image sensor used for photographing.

  Preferably, the color ratio data of the ambient light image component data is the color component of the ambient light image component data for each of the plurality of colors with respect to the sum of the multiple color component data of the ambient light image component data in each of the plurality of processing blocks. Based on data ratio.

  According to this aspect, the color ratio data of the ambient light image component data can be acquired as data normalized based on the sum of the color component data of the multiple colors of the ambient light image component data in each of the multiple processing blocks. is there.

  Preferably, the ambient light correction data acquisition unit is configured to perform a plurality of processes based on an arithmetic process for deriving brightness index data indicating the brightness of the ambient light image component data according to the color of the subject from the color ratio data of the ambient light image component data. The brightness index data of the ambient light image component data in each of the plurality of processing blocks is obtained from the color ratio data of the ambient light image component data in each of the blocks, and the sum of the color component data of the plurality of colors in each of the plurality of processing blocks The lightness index data related to the achromatic color is obtained from the color ratio data of the achromatic color having the same ratio of the color component data for each of the plurality of colors based on the arithmetic processing, and a plurality of processing blocks for the lightness index data related to the achromatic color are obtained. Based on the ratio of the brightness index data of the ambient light image component data in each, a plurality of processing blocks of the ambient light image component data are obtained. It acquires ambient light correction data for each of the click.

  According to this aspect, the ambient light correction data is acquired based on the ratio of the “brightness index data of the ambient light image component data” to the “brightness index data related to the achromatic color”.

  The “arithmetic processing” used for deriving the lightness index data may be any process that can calculate data indicating the lightness directly or indirectly. Therefore, the arithmetic expression used for calculating the “luminance value” may be used in the “arithmetic processing” here, and the “luminance value” calculated by such an arithmetic expression may be used as “lightness index data”. .

  Preferably, the image processing apparatus further includes a white balance processing unit that adjusts the white balance of the first image data based on the flash image correction component data.

  According to this aspect, the white balance of the first image data can be adjusted based on the flash image correction component data in which the difference in brightness of the flash image component data caused according to the color of the subject is compensated.

  Preferably, the white balance processing unit applies a flash light white balance gain to the first image data based on the flash image correction component data.

  According to this aspect, the flash light white balance gain is applied to the first image data based on the flash image correction component data in which the difference in the brightness of the flash image component data caused according to the color of the subject is compensated. The

  Preferably, the image processing apparatus further includes a white balance processing unit that adjusts the white balance of the first image data based on the flash image correction component data and the ambient light image correction component data.

  According to this aspect, “the flash image correction component data in which the difference in the brightness of the flash image component data brought about according to the subject color is compensated” and “the lightness of the ambient light image component data brought about according to the subject color” The white balance of the first image data can be adjusted based on the ambient light image correction component data compensated for the difference.

  Preferably, the white balance processing unit applies the flash light white balance gain and the ambient light white balance gain to the first image data based on the flash image correction component data and the ambient light image correction component data.

  According to this aspect, “the flash image correction component data in which the difference in the brightness of the flash image component data brought about according to the subject color is compensated” and “the lightness of the ambient light image component data brought about according to the subject color” The flash light white balance gain and the environmental light white balance gain are applied to the first image data based on the environmental light image correction component data in which the difference is compensated.

  Desirably, the brightness is luminance.

  According to this aspect, the flash image correction component data in which the difference in the brightness of the flash image component data caused according to the color of the subject is compensated is acquired.

  Another aspect of the present invention relates to an imaging device including an imaging device and the above-described image processing device.

  According to another aspect of the present invention, there is provided a step of acquiring flash image component data indicating an image component of a subject by flash light from first image data acquired by photographing the subject under flash light emission, and a flash Based on the image component data, obtaining flash correction data reflecting the difference in brightness of the flash image component data brought about according to the color of the subject, and depending on the color of the subject based on the flash image component data and the flash correction data Obtaining flash image correction component data in which a difference in brightness of the flash image component data caused is compensated.

  According to another aspect of the present invention, there is provided a procedure for acquiring flash image component data indicating an image component of a subject by flash light from first image data acquired by photographing the subject under flash light emission, a flash Based on the image component data, the procedure for obtaining flash correction data reflecting the difference in brightness of the flash image component data brought about according to the subject color, and the subject color based on the flash image component data and the flash correction data The present invention relates to a program for causing a computer to execute a procedure for acquiring flash image correction component data in which a difference in brightness of flash image component data caused by the correction is compensated.

  According to the present invention, it is possible to obtain flash image correction component data in which a difference in brightness of flash image component data caused according to the color of a subject is compensated. Using this “flash image correction component data in which the difference in brightness of the flash image component data brought about according to the color of the subject is compensated”, various image processing such as white balance processing is performed on the image data to be processed. Thus, it is possible to compensate for the difference in brightness caused according to the color of the subject, and it is possible to increase the processing accuracy.

It is a front perspective view of a digital camera. It is a rear perspective view of a digital camera. It is a block diagram which shows the control processing system of a digital camera. FIG. 4A is a diagram for explaining a difference in brightness (brightness) of image data brought about according to the color of a subject, and FIG. 4A shows a relationship between brightness values of image data relating to each of a gray subject and a blue subject. FIG. 4B is a diagram illustrating the relationship between the luminance values of the image data after compensating for the difference in brightness of the image data caused according to the color of the subject. It is a block diagram which shows the function structure of the image processing apparatus which concerns on 1st Embodiment. It is a flowchart of the white balance process which concerns on 1st Embodiment. It is a block diagram which shows the function structure of the flash image component acquisition part which concerns on 2nd Embodiment. It is a block diagram which shows the structural example of the 1st image data extraction part of a flash image component acquisition part. It is a block diagram which shows the structural example of the 1st extraction data adjustment part of a flash image component acquisition part. It is a conceptual diagram which shows the planar structure of the some pixel which comprises an image pick-up element. It is a top view which shows the basic array example of a color filter, (a) shows the basic array pattern of a Bayer array, (b) shows the basic array pattern of X-Trans (trademark) type. It is a block diagram which shows the function structure of the flash correction data acquisition part which concerns on 2nd Embodiment. It is a block diagram which shows the function structure of a 1st color ratio data acquisition part. It is a block diagram which shows the function structure of a 1st correction data calculation part. It is a flowchart of the white balance process which concerns on 2nd Embodiment. It is a block diagram which shows the function structure of the flash lightness correction | amendment part which concerns on 3rd Embodiment. It is a flowchart of the white balance process which concerns on 3rd Embodiment. It is a block diagram which shows the function structure of the image processing apparatus which concerns on 4th Embodiment. It is a block diagram which shows the function structure of the ambient light image component acquisition part which concerns on 4th Embodiment. It is a block diagram which shows the structural example of the 2nd image data extraction part of an ambient light image component acquisition part. It is a block diagram which shows the structural example of the 2nd extraction data adjustment part of an ambient light image component acquisition part. It is a block diagram which shows the function structure of an environmental light correction data acquisition part. It is a block diagram which shows the function structure of the 2nd color ratio data acquisition part. It is a block diagram which shows the function structure of a 2nd correction data calculation part. It is a block diagram which shows the function structure of an ambient light brightness correction | amendment part. It is a block diagram which shows the function structure of the white balance process part which concerns on 4th Embodiment. It is a flowchart of the white balance process which concerns on 4th Embodiment. It is a flowchart which shows the detail of the processing flow (step S42 of FIG. 27) for acquiring flash image correction component data in 4th Embodiment. It is a flowchart which shows the detail of the processing flow (step S43 of FIG. 27) for acquiring environmental light image correction component data in 4th Embodiment. It is a figure which shows the external appearance of a smart phone. It is a block diagram which shows the structure of the smart phone shown in FIG. It is a graph which illustrates the brightness | luminance relationship of "the difference data between a flash light emission image and a flash non-light emission image" regarding a red color chart, a green color chart, and a blue color chart.

  Embodiments of the present invention will be described with reference to the drawings. In the following embodiments, an example in which the present invention is applied to a digital camera (imaging device) will be described. However, the present invention can also be applied to image processing apparatuses other than digital cameras, imaging apparatuses, image processing methods, and programs.

  FIG. 1 is a front perspective view of the digital camera 2. FIG. 2 is a rear perspective view of the digital camera 2.

  The digital camera 2 includes a camera body 3 and a lens barrel 4 attached to the front surface of the camera body 3. The camera body 3 and the lens barrel 4 may be provided integrally, or may be provided detachably as a lens interchangeable camera.

  In addition to the lens barrel 4, a flash light emitting unit 5 is provided on the front surface of the camera body 3, and a shutter button 6 and a power switch 7 are provided on the upper surface of the camera body 3. The shutter button 6 is a shooting instruction unit that receives a shooting instruction from a user, and the power switch 7 is a power supply switching unit that receives a power ON / OFF switching instruction of the digital camera 2 from the user. The flash light emitting unit 5 may be a flash device built in the digital camera 2 (camera body 3) or an external flash device.

  On the back surface of the camera body 3, a display unit 8 constituted by a liquid crystal panel or the like and an operation unit 9 directly operated by a user are provided. The display unit 8 displays a live view image (through image) in a shooting standby state and functions as an electronic viewfinder, and functions as a playback image display unit during playback of a captured image and a memory storage image.

  The operation unit 9 includes an arbitrary operation device such as a mode change switch, a cross key, and an execution key. For example, the mode switch is operated by the user when the operation mode of the digital camera 2 is switched. The operation mode of the digital camera 2 includes a shooting mode for capturing an image of a subject and obtaining a captured image, a playback mode for reproducing and displaying an image, and the like. As other photographing modes, there are an AF (Auto Focus) mode for performing auto focus and an MF (Manual Focus) mode for performing manual focus operation. On the other hand, the cross key and the execution key display a menu screen or a setting screen on the display unit 8, move a cursor displayed in the menu screen or the setting screen, and confirm various settings of the digital camera 2. In some cases, it is operated by the user.

  On the bottom (not shown) of the camera body 3, a memory slot into which the main memory 10 is loaded and a loading lid for opening and closing the opening of the memory slot are provided. The main memory 10 is detachably provided on the camera body 3 and is electrically connected to a storage control unit 33 (see FIG. 3) provided on the camera body 3 when attached to the camera body 3. The main memory 10 can generally be constituted by a semiconductor memory such as a card-type flash memory, but is not particularly limited, and a storage medium of any storage system such as a magnetic medium can be used as the main memory 10. is there.

  FIG. 3 is a block diagram showing a control processing system of the digital camera 2.

  The subject light passes through the lens unit 12 provided in the lens barrel 4 and the mechanical shutter 20 provided in the camera body 3 and is received by the image sensor 21. The lens unit 12 includes a photographic optical system including a photographic lens (lens group) and a diaphragm. The image sensor 21 is an element that receives a subject image and generates image data. The image sensor 21 is a color filter such as RGB (Red Green Blue), a CCD (Charge Coupled Device) that converts an optical image into an electric signal, or a CMOS (Complementary Metal). And an image sensor (photodiode) such as an Oxide Semiconductor. The image data output from the image sensor 21 is processed by an AGC (Automatic Gain Control) circuit or the like in the process processing unit 22, and then the analog image data is converted into digital image data by the AD conversion unit 23. Is done. The digitized image data is stored in the buffer memory 24.

  The buffer memory 24 is an area for temporarily storing image data, and is configured by a DRAM (Dynamic Random Access Memory) or the like. Image data sent from the AD conversion unit 23 and stored in the buffer memory 24 is read by the image processing device 31 controlled by the system control unit 25.

  The image processing device 31 uses the image data read from the buffer memory 24 as a processing target, performs various image processing such as gamma correction processing and demosaicing processing in addition to the white balance processing described later, and performs image processing after image processing. The data is stored again in the buffer memory 24.

  Image data subjected to image processing in the image processing device 31 and stored in the buffer memory 24 is read by the display control unit 35 and the compression / decompression unit 32. The display control unit 35 controls the display unit 8 to display the image data read from the buffer memory 24 on the display unit 8. In this manner, the image data output from the image sensor 21 and subjected to image processing in the image processing device 31 is displayed on the display unit 8 as a shooting confirmation image (postview image).

  On the other hand, the compression / decompression unit 32 compresses the image data read from the buffer memory 24 to create image data in an arbitrary compression format such as JPEG (Joint Photographic Experts Group) or TIFF (Tagged Image File Format). . The compressed image data is stored in the main memory 10 by a storage control unit 33 that controls data storage processing to the main memory 10 and data reading processing from the main memory 10. When data such as image data is stored in the main memory 10, the storage control unit 33 adds editing date / time information to the data based on date / time information acquired from the clock device 34 via the system control unit 25. Shooting information such as (update date information) and other related information are added. The related information including the shooting information is added to the image data in an arbitrary format, and, for example, an Exif (Exchangeable image file format) format can be adopted.

  In the reproduction mode for reproducing the image data stored in the main memory 10, the image data stored in the main memory 10 is read by the storage control unit 33 controlled by the system control unit 25, and the compression / decompression unit 32. After being subjected to the decompression process, the data is stored in the buffer memory 24. Then, the image data is read from the buffer memory 24 by the display control unit 35 in the same procedure as the confirmation display of the photographed image, and the image data is reproduced and displayed on the display unit 8.

  The system control unit 25 controls the buffer memory 24, the image processing device 31, and the storage control unit 33 as described above, but also controls other units in the digital camera 2. For example, the system control unit 25 controls the lens driving unit 27 to control the driving of the lens unit 12, controls the shutter driving unit 26 to control the driving of the mechanical shutter 20, and controls the image sensor 21 to control the image. Control the output of data. Further, the system control unit 25 controls the flash light emitting unit 5 to control light emission and non-light emission of the flash, and controls the power supply control unit 28 to detect the presence or absence of the battery in the power source 29, the type of battery, and the remaining battery level. Etc. Further, the system control unit 25 acquires date and time information counted by the clock device 34 and uses it for various processes. The system control unit 25 controls various processing units that constitute the image processing apparatus 31.

  Further, the system control unit 25 acquires an operation signal from the user interface 36 including the shutter button 6, the power switch 7, and the operation unit 9, and performs various processes and device control according to the operation signal. For example, the system control unit 25 controls the shutter driving unit 26 according to the release signal received from the shutter button 6 to control the opening / closing of the mechanical shutter 20. Further, the system control unit 25 controls the power control unit 28 in accordance with the power on / off signal received from the power switch 7, and controls on / off of the power source 29.

  Programs and data necessary for various processes performed by the system control unit 25 and device control are stored in the control memory 30. The system control unit 25 can read out programs and data stored in the control memory 30 as necessary, and can store new programs and data in the control memory 30. For example, the system control unit 25 can write condition data such as the type of white balance mode and white balance gain that have been set in the control memory 30. The system control unit 25 can control the display control unit 35 to display various information acquired from each unit on the display unit 8. The system control unit 25 can change various information displayed on the display unit 8 in accordance with an operation signal input by the user via the user interface 36.

  Next, white balance processing in the image processing apparatus 31 will be described.

  In the following embodiments, white balance processing is described as an example of image processing. However, the image processing process described below can be applied to processing other than white balance processing.

  In each of the following embodiments, in the white balance processing of the flash emission image, correction processing for compensating (correcting) a difference in brightness caused according to the color of the subject is performed. In the following embodiments, “luminance” is used as “brightness” of image data.

  FIG. 4 is a diagram for explaining the difference in brightness (luminance) of image data brought about according to the color of the subject. FIG. 4A is a diagram illustrating the relationship between the luminance values of the image data for each of a gray (achromatic) subject and a blue (chromatic) subject, and FIG. 4B corresponds to the color of the subject. It is a figure which shows the relationship of the luminance value of the image data after compensating the difference in the brightness of the image data brought about. The vertical axes in FIGS. 4A and 4B indicate the luminance values of the image data for each of the gray subject and the blue subject. 4A and 4B show a case where a gray subject and a blue subject are arranged at an equal distance from the flash light emitting unit. 4 (a) and 4 (b), “Fg1” conceptually indicates a portion to which flash light contributes in the luminance of the image data of the gray subject, and “Lg1” indicates the image data of the gray subject. The part which environmental light contributes among luminance is shown notionally. “Fb1” and “Fb2” conceptually indicate a portion to which flash light contributes in the luminance of the image data of the blue subject, and “Lb1” and “Lb2” indicate the environment of the luminance of the image data of the blue subject. The part which light contributes is shown notionally.

  In general, according to the distance between the flash light emitting unit 5 and the subject, “the influence amount of flash light (see“ Fg1 ”and“ Fb1 ”in FIG. 4A)” and “the influence amount of environmental light” in the image data. (See “Lg1” and “Lb1” in FIG. 4A). Further, since the brightness of the image depends on the color of the subject, even if the distance between the flash light emitting unit 5 and the subject is the same, if the color of the subject is different, the influence amount of the flash light on the image and the ambient light The amount of influence varies. Therefore, in the example shown in FIG. 4A, the size of the luminance portion contributed by the flash light does not match between the image data of the gray subject and the image data of the blue subject (that is, “Fg1 ≠ Fb1”). In addition, the size of the luminance portion to which the ambient light contributes does not match (that is, “Lg1 ≠ Lb1”), and the overall luminance also differs (that is, “Fg1 + Lg1 ≠ Fb1 + Lb1”).

In each of the embodiments described below, the difference in brightness (brightness) of the image data brought about according to the color of the subject is compensated, and as shown in FIG. Processing for realizing at least one of the relationships is performed.
Formula 1 “Fg1 = Fb2 (where“ Fb2 = Fb1 × correction data for flash light (flash correction data) ”)”
Formula 2 “Lg1 = Lb2 (where“ Lb2 = Lb1 × ambient light correction data (ambient light correction data) ”)
Formula 3 “Fg1 + Lg1 = Fb2 + Lb2”
In this way, by removing the influence such as the difference in brightness depending on the color of the subject and realizing the relationship represented by at least one of the above formulas 1 to 3, a more accurate “effect of flash light” The image data indicating “can be acquired, and image processing is performed using the image data to compensate for a difference in brightness (brightness) of the image data.

  As described above, the image processing apparatus 31 performs image processing other than the white balance processing. However, the white balance processing will be mainly described below, and the description of image processing other than the white balance processing will be omitted.

<First Embodiment>
FIG. 5 is a block diagram illustrating a functional configuration of the image processing apparatus 31 according to the first embodiment. The image processing apparatus 31 of this embodiment includes a flash image component acquisition unit 40, a flash correction data acquisition unit 42, a flash brightness correction unit 44, and a white balance processing unit 46.

  The flash image component acquisition unit 40, based on the first image data (flash emission image data D1) acquired by photographing the subject under the emission of flash light, the flash image component indicating the image component of the subject by flash light. Data (RGB flash influence map) D3 is acquired.

  A specific method for acquiring the flash image component data D3 in the flash image component acquisition unit 40 is not particularly limited. For example, the flash image component acquisition unit 40 acquires “flash emission image data D1 (first image data)” and “flash non-emission image data D2 (second image) acquired by photographing a subject under non-emission of flash light”. The difference data from the image data) ”can be the flash image component data D3. The flash image component data D3 is “image data indicating the influence of flash light” in which the influence of ambient light is canceled from the flash emission image data D1. That is, the flash image component data D3 is data obtained by excluding an area affected by the ambient light without being affected by the flash light and separating the area affected by the flash light from the flash emission image. Become.

  The method for acquiring the flash emission image data D1 and the flash non-emission image data D2 by the flash image component acquisition unit 40 is not particularly limited. For example, the flash emission image data D1 and the flash non-emission image data D2 relating to the same subject that are continuously photographed and acquired are transferred from the image sensor 21 (see FIG. 3) to the process processing unit 22, the AD conversion unit 23, and the buffer memory 24. Via the flash image component acquisition unit 40 of the image processing apparatus 31. The flash image component acquisition unit 40 may acquire flash emission image data D1 and flash non-emission image data D2 related to the same subject stored in the main memory 10 as processing targets. In this case, the flash light emission image data D1 and the flash non-light emission image data D2 stored in the main memory 10 are read by the storage control unit 33 and decompressed by the compression / decompression unit 32, and then stored in the buffer memory 24. To the flash image component acquisition unit 40 of the image processing device 31.

  Further, various conditions such as shooting acquisition conditions, image types, and image sizes of the flash emission image data D1 and the flash non-emission image data D2 supplied to the image processing device 31 (flash image component acquisition unit 40) are not particularly limited. That is, in the case of “image data captured and acquired under flash emission” and “image data acquired and acquired under non-flash emission”, the flash emission supplied to the image processing device 31 (flash image component acquisition unit 40). The image data D1 and the flash non-light-emitting image data D2 may be image data captured and acquired under any shooting condition. For example, the flash light emission image data D1 and the flash non-light emission image data D2 may be still image data or moving image data. Further, as the flash non-light emitting image data D2, image data acquired during high-speed operation such as live view image data or flash light dimming image data may be used.

  The process of supplying the flash emission image data D1 and the flash non-emission image data D2 to the image processing device 31 (flash image component acquisition unit 40) is appropriately performed under the control of the system control unit 25 (see FIG. 3). .

  Details of a configuration example of the flash image component acquisition unit 40 that acquires the flash image component data D3 will be described in a second embodiment to be described later.

  The flash correction data acquisition unit 42 shown in FIG. 5 is based on the flash image component data D3, and flash correction data (flash impact map for each color) that reflects the difference in brightness of the flash image component data D3 brought about according to the color of the subject. D4 is acquired. The flash correction data D4 can be expressed by numerical data (for example, a gain value) for eliminating the influence (difference in brightness) caused according to the color of the subject.

  The flash brightness correction unit 44 is based on the flash image component data D3 and the flash correction data D4, and the flash image correction component data (the effect of the combined flash) is compensated for the difference in brightness of the flash image component data D3 brought about according to the color of the subject. Degree map) D5 is acquired. The flash image correction component data D5 indicates that the flash image component data D3 is adjusted based on the flash correction data D4 and the difference in luminance based on the spectral characteristics corresponding to the color of the subject is compensated. Image) data ".

  Details of configuration examples of the flash correction data acquisition unit 42 and flash brightness correction unit 44 and specific examples of the flash correction data D4 and flash image correction component data D5 will be described in second and third embodiments described later. .

  The white balance processing unit 46 adjusts the white balance of the flash emission image data (first image data) D1 based on the flash image correction component data D5. For example, from the flash image correction component data D5, it is possible to estimate an area where the influence of flash light is strong and weak in the flash emission image data D1. Therefore, the white balance processing unit 46 determines an area where the influence of the flash light is relatively strong in the flash emission image data (first image data) D1 based on the flash image correction component data D5. The white balance gain for flash light can be applied to the image data (flash emission image data (first image data) D1) of “relatively strong area”.

  The specific method of white balance processing in the white balance processing unit 46 is not particularly limited. For example, white balance processing for eliminating only the influence of flash light in the flash emission image data D1 may be performed, or white balance processing for eliminating the influence of each of flash light and ambient light in the flash emission image data D1 is performed. May be. More specifically, a white balance gain for flash light may be applied to an area where the influence of flash light is relatively strong in the flash emission image data D1. In this case, the white balance gain may not be applied to the region other than the “region where the influence of the flash light is relatively strong”, or the ambient light white balance gain may be applied. Further, with respect to each area of the flash emission image data D1, the ratio between the influence of the flash light and the influence of the environmental light is obtained, and both the white balance gain for the flash light and the white balance gain for the environmental light are determined according to the ratio. May be applied.

  FIG. 6 is a flowchart of white balance processing according to the first embodiment.

  First, the flash image component acquisition unit 40 acquires the flash emission image data D1 and the flash non-emission image data D2 (S11 in FIG. 6), and the flash image component data is obtained from the flash emission image data D1 and the flash non-emission image data D2. (RGB flash influence map) D3 is acquired (S12).

  Then, the flash correction data acquisition unit 42 acquires the flash correction data D4 from the flash image component data D3 (S13). Further, the flash brightness correction unit 44 obtains flash image correction component data D5 from the flash image component data D3 and the flash correction data D4 (S14).

  In the white balance processing unit 46, white balance processing corresponding to the flash image correction component data D5 is performed on the flash emission image data D1 (S15).

  As described above, according to the image processing apparatus 31 and the image processing method of the present embodiment, flash correction data for removing the influence such as the difference in brightness (brightness) caused according to the color of the subject is acquired, Flash image correction component data indicating the degree of influence of flash light is acquired in consideration of the flash correction data. By using such flash image correction component data, highly accurate white balance processing can be performed on the flash emission image data D1.

Second Embodiment
In the present embodiment, the same or similar components as those in the first embodiment described above are denoted by the same reference numerals, and detailed description thereof is omitted.

  In the present embodiment, the brightness of the image data provided according to the color of the subject is adjusted by adjusting the brightness (brightness) of the photographed image of the subject with reference to the spectral characteristics (spectral reflectance) of the achromatic (gray) subject. A process for compensating for the difference is performed.

  Similar to the image processing apparatus 31 of the first embodiment shown in FIG. 5, the image processing apparatus 31 according to the present embodiment has a flash image component acquisition unit 40, a flash correction data acquisition unit 42, a flash brightness correction unit 44, and a white balance. Although the processing unit 46 is provided, the flash image component acquisition unit 40 and the flash correction data acquisition unit 42 have the following specific configurations.

  First, an aspect of a specific configuration of the flash image component acquisition unit 40 will be described.

  The flash image component acquisition unit 40 of this embodiment captures a subject under flash emission image data (first image data) D1 acquired by photographing the subject under flash light emission and a subject under non-flash light emission. Flash image component data D3 is acquired from first difference data representing a difference from flash non-emission image data D2 (second image data) acquired by photographing. In particular, the flash image component acquisition unit 40 of the present embodiment acquires the flash image component data D3 by applying a flash light white balance gain to the first difference data.

  FIG. 7 is a block diagram illustrating a functional configuration of the flash image component acquisition unit 40 according to the second embodiment. The flash image component acquisition unit 40 according to the present embodiment includes a first image data extraction unit 47 and a first extraction data adjustment unit 49. The first image data extraction unit 47 acquires first difference data representing a difference between the flash emission image data D1 and the flash non-emission image data D2. The first extraction data adjustment unit 49 obtains flash image component data D3 by applying a flash light white balance gain to the first difference data.

  FIG. 8 is a block diagram illustrating a configuration example of the first image data extraction unit 47 of the flash image component acquisition unit 40. The first image data extraction unit 47 of this embodiment has a subtracter 48. The subtracter 48 receives the flash emission image data D1 and the flash non-emission image data D2, and receives the expression "first difference data (first RGB flash influence map) D6 = flash emission image data D1-flash non-emission image data". First difference data D6 represented by “D2” is calculated. The first difference data D6 calculated by the subtracter 48 is sent from the first image data extraction unit 47 to the first extraction data adjustment unit 49.

  FIG. 9 is a block diagram illustrating a configuration example of the first extraction data adjustment unit 49 of the flash image component acquisition unit 40. The first extraction data adjustment unit 49 of the present embodiment includes a first white balance adjustment unit 50 and a flash light WB gain storage unit 51. The first white balance adjustment unit 50 reads the flash light white balance gain WB1 stored in the flash light WB gain storage unit 51, and applies the flash light white balance gain WB1 to the first difference data D6. Thus, the flash image component data (second RGB flash influence map) D3 is acquired.

  As described above, the flash image component data D3 obtained through the first image data extraction unit 47 and the first extraction data adjustment unit 49 is “flash” subjected to white balance correction according to the spectral characteristics of the flash light. The image data indicates the influence of light. Therefore, the flash image component acquisition unit 40 of the present embodiment outputs the flash image component data D3 with the gray balance adjusted.

  The method of acquiring the flash light white balance gain WB1 and storing it in the flash light WB gain storage unit 51 is not particularly limited. For example, when the flash light emitting unit 5 (see FIGS. 1 and 3) is a flash device of the type incorporated in the digital camera 2 (camera body 3), the spectral characteristics of the flash light are known, so the digital camera 2 ( When manufacturing the camera body 3), the flash light white balance gain WB1 can be acquired in advance and stored in the flash light WB gain storage unit 51. Further, when the flash light emitting unit 5 is an external flash device, the user takes, for example, a gray card under flash light emission, and obtains an appropriate flash light white balance gain WB1 from the photographed image data in advance. The flash light white balance gain WB 1 can be stored in the flash light WB gain storage unit 51.

  The first image data extraction unit 47 (flash image component acquisition unit 40) may acquire the flash image component data D3 using the flash emission image data instead of the flash non-emission image data D2. That is, the flash image component acquisition unit 40 acquires the first image data (flash emission image data D1) acquired by photographing the subject under the emission of flash light having the first emission amount, the first emission amount, and the like. Is flash image component data from first difference data representing a difference from second image data (flash emission image data) acquired by photographing a subject under the emission of flash light having a different second emission amount. D3 may be acquired.

  Next, a processing unit (processing block) of the flash image component data D3 in the flash correction data acquisition unit 42 (see FIG. 5) will be described.

  FIG. 10 is a conceptual diagram showing a planar structure of a plurality of pixels 14 constituting the image sensor 21. The image sensor 21 is configured by two-dimensionally arranging a plurality of pixels 14, and each pixel 14 includes a color filter and an image sensor. Image data photographed and acquired by the image sensor 21 is constituted by a set of pixel data (pixel values) output from each of these pixels 14 (image sensors). Accordingly, each of the above-described flash emission image data D1, flash non-emission image data D2, first difference data D6, and flash image component data D3 is also configured by a set of pixel data.

  The flash correction data acquisition unit 42 of the present embodiment performs processing by dividing the flash image component data D3 into a plurality of processing blocks. Each of the plurality of processing blocks may be configured by a single pixel. It may be constituted by a plurality of pixels. When the constituent pixels of each processing block are single pixels, the following processing in the flash correction data acquisition unit 42 and the like is performed for each pixel. On the other hand, when the constituent pixels of each processing block are a plurality of pixels, the following processing in the flash correction data acquisition unit 42 or the like is performed for each “plural pixels constituting the processing block”.

  When each of the plurality of processing blocks is configured by a plurality of pixels, the plurality of pixels configuring each of the plurality of processing blocks may correspond to the pixels 14 configuring the basic array of the color filter 16 of the image sensor 21. preferable. That is, the image sensor 21 used for photographing the flash emission image data (first image data) D1 is a single-plate image sensor including a plurality of color filters such as RGB. When the basic arrangement of a specific color pattern is repeatedly arranged, each processing block is preferably determined so as to correspond to the basic arrangement of the color filter.

  FIG. 11 is a plan view showing an example of the basic arrangement of the color filter 16, where (a) shows the basic arrangement pattern of the Bayer arrangement, and (b) shows the X-Trans (registered trademark) type basic arrangement pattern. In FIG. 11A, “R” indicates the R (red) color filter 16, “B” indicates the B (blue) color filter 16, and “G1” and “G2” indicate G (green). The color filter 16 is shown. In FIG. 11B, “R1” to “R8” indicate the R (red) color filter 16, “B1” to “B8” indicate the B (blue) color filter 16, and “G1” to “G1”. “G20” indicates the G (green) color filter 16.

  For example, when the image sensor 21 has the color filter 16 with the Bayer arrangement, each processing block is preferably configured by 2 pixels × 2 pixels (a total of 4 pixels) as shown in FIG. On the other hand, when the image sensor 21 has the X-Trans type color filter 16, it is preferable that each processing block is composed of 6 pixels × 6 pixels (36 pixels in total) as shown in FIG.

  The unit of pixels constituting each processing block in the processing of the flash correction data acquisition unit 42 and the like and the unit of pixels constituting each processing block in the white balance processing in the white balance processing unit 46 described later are set to be the same. May be. That is, when each processing block in the white balance processing is configured by a single pixel, each processing block in the processing by the flash correction data acquisition unit 42 and the like can also be configured by a single pixel. Further, when each processing block in the white balance processing is configured by a plurality of pixels constituting the basic array of the color filter 16, each processing block in the flash correction data acquisition unit 42 or the like also includes a plurality of the basic array of the color filter 16. It can be composed of pixels.

  In the following description, an example in which each processing block is configured by a single pixel will be described, but basically the same processing is performed when each processing block is configured by a plurality of pixels. Therefore, in the case where each processing block is constituted by a plurality of pixels, there are cases where it is appropriate to refer to “pixels” as “regions constituted by a plurality of pixels”. Call it.

  Next, an aspect of a specific configuration of the flash correction data acquisition unit 42 will be described.

  The flash correction data acquisition unit 42 of the present embodiment reduces the color-dependent data (RGB) of the flash image component data acquired for each processing block in order to reduce the color-dependent influence based on the spectral characteristics (spectral reflectance) of the subject. Brightness index data (luminance data) is acquired from the ratio etc., and flash correction data (correction gain) for compensating for differences in brightness (luminance) between colors is acquired from the brightness index data.

  FIG. 12 is a block diagram illustrating a functional configuration of the flash correction data acquisition unit 42 according to the second embodiment.

  The flash correction data acquisition unit 42 according to the present embodiment includes a first color ratio data acquisition unit 53 and a first correction data calculation unit 54. The first color ratio data acquisition unit 53 acquires color ratio data D7 representing the color component ratio of the flash image component data D3 in each of the plurality of processing blocks from the flash image component data D3. Based on the color ratio data D7 of the flash image component data D3, the first correction data calculation unit 54 reflects the difference in lightness among the plurality of processing blocks of the flash image component data D3 that is generated according to the color of the subject. Flash correction data D4 is acquired.

  FIG. 13 is a block diagram illustrating a functional configuration of the first color ratio data acquisition unit 53. The first color ratio data acquisition unit 53 of this example includes a first demosaic processing unit 56 and a first color ratio calculation unit 57.

  The first demosaic processing unit 56 performs demosaic processing (synchronization processing) on the flash image component data D3. By this demosaic processing, color component data D8 is obtained in which pixel data relating to each color (RGB, etc.) constituting the color filter 16 of the image sensor 21 is assigned to each pixel constituting the flash image component data D3. .

  The first color ratio calculation unit 57 acquires the color ratio data D7 of the flash image component data D3 in each of the plurality of processing blocks based on the color component data D8. The color ratio data D7 indicates the color component ratio (RGB ratio, etc.) of the color component data D8 constituting the flash image component data D3, and is calculated for each processing block. The flash image component data D3 derived from the flash emission image data (first image data) D1 acquired by the imaging device 21 including the color filters 16 of a plurality of colors (RGB, etc.) RGB color component data D8. The first color ratio calculation unit 57 of the first color ratio data acquisition unit 53 (flash correction data acquisition unit 42), based on the color component data D8, stores the flash image component data D3 in each of the plurality of processing blocks. Color ratio data D7 is acquired.

  The color ratio data D7 of the flash image component data D3 is the color of the flash image component data D3 for each of the plurality of colors with respect to the sum of the color component data D8 of the plurality of colors of the flash image component data D3 in each of the plurality of processing blocks. Based on the ratio of component data D8. For example, the flash image component data D3 is configured by RGB color component data D8, each processing block is configured by a single pixel, and each pixel of the flash image component data D3 is RGB by demosaic processing in the first demosaic processing unit 56. Consider the case of having each color component data D8. In this case, in the case where the ratio of the RGB color component data D8 in the “certain pixel” of the flash image component data D3 is, for example, “R: G: B = 1: 2: 3”, the color ratio data regarding this pixel. D7 is “R: G: B = 1 / (1 + 2 + 3): 2 / (1 + 2 + 3): 3 / (1 + 2 + 3)”, and this is data directly or indirectly.

  In the above example, the demosaic process of the flash image component data D3 is performed by the first demosaic processing unit 56, but the flash image component acquisition unit 40 performs the “desiccated flash emission image data D1 and flash non-emission image data”. It is good also as a structure which acquires "D2". That is, the flash emission image data D1 and the flash non-emission image data D2 may be demosaiced before being input to the flash image component acquisition unit 40. In the above-described example, the demosaic process of the flash image component data D3 is performed by the first demosaic processing unit 56. However, when the processing block includes a plurality of pixels, the first demosaic processing unit 56 (demosaic processing) May be omitted. When the first demosaic processing unit 56 is omitted, the color ratio data D7 in each processing block can be acquired from the color component data D8 related to a plurality of pixels included in each processing block.

  For example, when each processing block is configured by 2 pixels × 2 pixels corresponding to the basic array pattern of the Bayer array, pixel data (pixel data) represented by “R” in FIG. Treated as R color component data D8, pixel data represented by “B” in FIG. 11A is treated as B color component data D8 in each processing block, and “G1” and “G1” in FIG. The G color component data D8 of each processing block may be acquired from the data of two pixels represented by “G2”. When the processing block is composed of 6 pixels × 6 pixels corresponding to the X-Trans type basic array pattern, each process is performed from the data of 8 pixels represented by “R1” to “R8” in FIG. The R color component data D8 of the block is acquired, and the B color component data D8 of each processing block is acquired from the 8 pixel data represented by “B1” to “B8” in FIG. The G color component data D8 of each processing block may be acquired from the 20-pixel data represented by “G1” to “G20” in (b). The method for obtaining the single color component data D8 in each processing block from the data of a plurality of pixels is not particularly limited. For example, the “average value” of the data of a plurality of pixels related to the single color is set as the single color component data D8. May be. Therefore, in the example shown in FIG. 11A, the average value of the data of two pixels represented by “G1” and “G2” may be set in the G color component data D8 of each processing block. Similarly, in the example shown in FIG. 11B, the average value of the data of 8 pixels represented by “R1” to “R8” is set in the R color component data D8 of each processing block, and “B1” to “B”. The average value of the 8-pixel data represented by “B8” is set in the B color component data D8 of each processing block, and the average value of the 20-pixel data represented by “G1” to “G20” is set for each processing block. The G color component data D8 may be set.

  FIG. 14 is a block diagram illustrating a functional configuration of the first correction data calculation unit 54. The first correction data calculation unit 54 of this example includes a first brightness index data acquisition unit 59 and a first correction data calculation processing unit 60.

  The first brightness index data acquisition unit 59 acquires brightness index data D9 from the color ratio data D7. That is, the first lightness index data acquisition unit 59 (flash correction data acquisition unit 42) “uses the lightness index data D9 indicating the lightness of the flash image component data D3 according to the color of the subject to the color ratio data of the flash image component data D3. Based on the “calculation process derived from D7”, the lightness index data D9 of the flash image component data D3 in each of the plurality of processing blocks is acquired from the color ratio data D7 of the flash image component data D3 in each of the plurality of processing blocks.

  The first correction data calculation processing unit 60 acquires the flash correction data D4 from the lightness index data D9. In particular, the first correction data calculation processing unit 60 (flash correction data acquisition unit 42) of the present embodiment acquires flash correction data D4 related to each of a plurality of processing blocks based on achromatic color ratio data. That is, in the first correction data calculation processing unit 60 (flash correction data acquisition unit 42), the ratio of the color component data for each of the plurality of colors to the sum of the color component data of the plurality of colors in each of the plurality of processing blocks is mutually equal. Lightness index data related to the achromatic color is acquired from the color ratio data of the achromatic color based on the arithmetic processing. Then, the first correction data calculation processing unit 60 (flash correction data acquisition unit 42) is based on the ratio of the lightness index data D9 of the flash image component data D3 in each of the plurality of processing blocks to the lightness index data regarding the achromatic color. Flash correction data D4 relating to each of the plurality of processing blocks of the flash image component data D3 is acquired.

  The “calculation process” used to acquire the brightness index data D9 is not particularly limited as long as it is a process for deriving the brightness index data D9 that directly or indirectly indicates the brightness. An arithmetic process used to calculate “value” may be used. Therefore, even if the lightness index data D9 is calculated based on an expression represented by, for example, “Y = 0.3 * R + 0.6 * G + 0.1 * B”, which is used for calculating “luminance value (Y)”. Good. When arithmetic processing is performed using this equation, for example, the color ratio data D7 in “a certain processing block” of the flash image component data D3 is “R: G: B = 1 / (1 + 2 + 3): 2 // (1 + 2 + 3): 3 / In the case of data indicating (1 + 2 + 3) (= 1: 2: 3), the brightness index data D9 of the processing block is, for example, “D9 = 0.3 * 1 / (1 + 2 + 3) + 0.6 * 2 / (1 + 2 + 3) +0 1 * 3 / (1 + 2 + 3) = 0.3 / 6 + 1.2 / 6 + 0.3 / 6 = 1.8 / 6 = 0.9 / 3 ”. In this case, the color ratio data of achromatic colors is always “R: G: B = 1/3: 1/3: 1/3” (= 1: 1: 1) because the color component data for each of RGB is equal to each other. 1) ". Therefore, the achromatic lightness index data D9 obtained using the above formula is “D9 = 0.3 * 1/3 + 0.6 * 1/3 + 0.1 * 1/3 = 0.3 / 3 + 0.6 / 3 + 0. 1/3 = 1.0 / 3 ”. In this case, the flash correction data D4 obtained by the first correction data calculation processing unit 60 is the lightness index data of the flash image component data D3 in the processing block for the achromatic color lightness index data D9 (= 1.0 / 3). Obtained based on the ratio of D9 (= 0.9 / 3) (= (0.9 / 3) / (1.0 / 3)), for example, the reciprocal of the ratio (= 1.0 / 0.9) Can be used as the flash correction data D4. Note that Y (luminance) in the CIE XYZ color system (color space) and L * (lightness) in the CIE L * a * b * color system (color space) can also be used as the lightness index data D9. .

  The flash brightness correction unit 44 (see FIG. 5) acquires “flash image component data D3 output from the flash image component acquisition unit 40 (first extraction data adjustment unit 49)” and “flash” acquired as described above. The flash image correction component data D5 is acquired based on the flash correction data D4 output from the correction data acquisition unit 42 (first correction data calculation processing unit 60). Note that details of a specific method for acquiring the flash image correction component data D5 will be described in a third embodiment to be described later.

  FIG. 15 is a flowchart of white balance processing according to the second embodiment.

  As described above, the white balance process of this embodiment is an aspect of the white balance process of the first embodiment. Therefore, step S21 shown in FIG. 15 corresponds to step S11 of FIG. 6 according to the first embodiment, step S22 and step S23 shown in FIG. 15 correspond to step S12 of FIG. 6, and steps S24 to S24 shown in FIG. Step S26 corresponds to step S13 in FIG. 6, step S27 shown in FIG. 15 corresponds to step S14 in FIG. 6, and step S28 shown in FIG. 15 corresponds to step S15 in FIG.

  In other words, in the present embodiment, the first image data extraction unit 47 of the flash image component acquisition unit 40 acquires the flash emission image data D1 and the flash non-emission image data D2 (S21 in FIG. 15), and these flash emission images. First difference data D6 is acquired based on data D1 and flash non-light-emitting image data D2 (S22).

  Then, the first extracted data adjustment unit 49 of the flash image component acquisition unit 40 applies the flash light white balance gain WB1 to the first difference data D6 to acquire the flash image component data D3 (S23).

  In the first color ratio data acquisition unit 53 of the flash correction data acquisition unit 42, the first demosaic processing unit 56 acquires the color component data D8 related to each processing block from the flash image component data D3, and the first color ratio. The calculation unit 57 acquires the color ratio data D7 relating to each processing block from the color component data D8 (S24).

  Then, in the first correction data calculation unit 54 of the flash correction data acquisition unit 42, the first lightness index data acquisition unit 59 acquires the lightness index data D9 related to each processing block from the color ratio data D7 (S25). The correction data calculation processing unit 60 acquires the flash correction data D4 for each processing block from the brightness index data D9 (S26).

  Then, the flash brightness correction unit 44 obtains flash image correction component data D5 from the flash image component data D3 and the flash correction data D4 (S27), and the white balance processing unit 46 performs white balance corresponding to the flash image correction component data D5. The process is performed on the flash emission image data D1 (S28).

  As described above, according to the image processing device 31 and the image processing method of the present embodiment, as with the image processing device 31 according to the first embodiment, the lightness of the flash image component data (corresponding to the color of the subject) ( A highly accurate white balance process can be realized based on the flash correction data for compensating for the difference in luminance.

<Third Embodiment>
In the present embodiment, the same or similar components as those in the second embodiment described above are denoted by the same reference numerals, and detailed description thereof is omitted.

  The image processing apparatus 31 according to the present embodiment is an aspect of the image processing apparatus 31 according to the second embodiment, and the flash correction data D4 is provided for each processing block with respect to data indicating the brightness (luminance) of the flash image component data D3. As a result, flash image correction component data D5 is acquired. That is, the image processing apparatus 31 according to the present embodiment is similar to the image processing apparatus 31 according to the first embodiment and the image processing apparatus 31 according to the second embodiment shown in FIG. Unit 42, flash brightness correction unit 44, and white balance processing unit 46. In particular, the flash brightness correction unit 44 has the following specific configuration.

  FIG. 16 is a block diagram illustrating a functional configuration of the flash brightness correction unit 44 according to the third embodiment.

  The flash brightness correction unit 44 of the present embodiment includes a first brightness data acquisition unit 62 and a first correction calculation unit 63.

  The first brightness data acquisition unit 62 of the flash brightness correction unit 44 acquires the first brightness data D10 in each of the plurality of processing blocks based on the flash image component data D3 sent from the flash image component acquisition unit 40. . The first lightness data D10 in the present embodiment is data representing luminance, and the first lightness data (luminance flash influence map) D10 may be obtained by any method. That is, the first lightness data acquisition unit 62 converts the color component data (RGB data or the like) of the flash image component data D3 into luminance and acquires the first lightness data D10. For example, a “calculation process (for example,“ ”) used to acquire the brightness index data D9 in the first brightness index data acquisition unit 59 (flash correction data acquisition unit 42, first correction data calculation unit 54) shown in FIG. Y = 0.3 * R + 0.6 * G + 0.1 * B ")"), the first brightness data D10 (luminance value (Y)) can be calculated from the flash image component data D3.

  The first correction calculation unit 63 of the flash brightness correction unit 44 acquires the flash image correction component data D5 by applying the flash correction data D4 to the first brightness data D10 for each of the plurality of processing blocks. More specifically, the flash correction data D4 acquired by the flash correction data acquisition unit 42 of the present embodiment is applied to the first lightness data D10 indicating the lightness of the flash image component data D3. This is data for compensating for the difference in lightness of the flash image component data D3 (first lightness data D10) brought about according to the color. The flash correction data acquisition unit 42 acquires such flash correction data D4 for each of the plurality of processing blocks of the flash image component data D3.

  For example, by multiplying the first brightness data D10 by the flash correction data D4 for each processing block, the flash image correction component data D5 in which the difference in brightness caused by the color of the subject is compensated is calculated. It may be. In this case, the flash correction data acquisition unit 42 (the first correction data calculation processing unit 60 (see FIG. 14)) “reciprocal of the ratio of the lightness index data D9 of the flash image component data D3 to the lightness index data D9 of achromatic color. Can be acquired as the flash correction data D4. For example, as described above, the lightness index data D9 of the achromatic color is expressed by “D9 = 1.0 / 3”, and the lightness index data D9 of the flash image component data D3 in a certain processing block is “D9 = 0.9 / 3”. "The ratio of the lightness index data D9 of the flash image component data D3 to the achromatic color lightness index data D9" is "(0.9 / 3) / (1.0 / 3) = 0.9." The flash correction data D4 can be set to “D4 = 1.0 / 0.9”. In this case, the flash image correction component data D5 can be represented by a value obtained by multiplying the first brightness data D10 by “1.0 / 0.9”.

  As described above, the flash image correction component data D5 obtained by applying the flash correction data D4 to the first lightness data D10 indicating the luminance (lightness) of the flash image component data D3 is provided according to the color of the subject. The difference in luminance (lightness) of the flash image component data D3 (first lightness data D10) is compensated.

  FIG. 17 is a flowchart of white balance processing according to the third embodiment.

  As described above, the white balance process of this embodiment is an aspect of the white balance process of the second embodiment. For example, steps S31 to S36 shown in FIG. 17 correspond to steps S21 to S26 of FIG. 15 according to the second embodiment, and steps S37 and S38 shown in FIG. 17 correspond to step S27 of FIG. Step S39 corresponds to step S28 in FIG.

  That is, in the present embodiment, the flash emission image data D1 and the flash non-emission image data D2 are acquired (S31 in FIG. 17) and the first difference data D6 is acquired in the same manner as in steps S21 to S26 of FIG. (S32) Flash image component data D3 is acquired (S33), color ratio data D7 related to each processing block is acquired (S34), brightness index data D9 related to each processing block is acquired (S35), and each processing block is related. Flash correction data D4 is acquired (S36).

  Then, in the first lightness data acquisition unit 62 of the flash lightness correction unit 44, the first lightness data D10 relating to each processing block is acquired from the flash image component data D3 acquired by the flash image component acquisition unit 40 (S37). . Then, in the first correction calculation unit 63 of the flash brightness correction unit 44, flash image correction component data D5 is acquired from the first brightness data D10 and the flash correction data D4 (S38).

  In the white balance processing unit 46, white balance processing corresponding to the flash image correction component data D5 is performed on the flash emission image data D1 (S39). For example, “an area where the flash image correction component data D5 shows a value of 50% or more with respect to the maximum gradation value (maximum image data value)” may be determined as the flash area, and the white balance processing unit 46 may White balance processing may be performed by applying the flash light white balance gain WB1 to such a flash area in the light emission image data D1.

  As described above, according to the image processing apparatus 31 and the image processing method of the present embodiment, based on the flash correction data for compensating for the difference in the brightness (luminance) of the flash image component data caused according to the color of the subject. Thus, flash image correction component data D5 is acquired. By using the flash image correction component data D5, a highly accurate white balance process can be realized.

<Fourth embodiment>
In the present embodiment, the same reference numerals are given to the same or similar components as those in the third embodiment, and the detailed description thereof is omitted.

  The first to third embodiments described above mainly relate to white balance adjustment of flash light, but this embodiment relates to white balance adjustment of ambient light in addition to flash light. That is, the image processing apparatus 31 according to the present embodiment performs not only the correction process for each color related to the flash light in the flash emission image data D1, but also the correction process for each color related to the ambient light.

  In general, if the shooting light source of image data is unknown, it is appropriate if the image does not contain a gray (achromatic) subject or if the region (position) of the achromatic subject contained in the image is unknown. It is difficult to get a good white balance gain.

  However, for example, according to the image processing in the image processing apparatus 31 of the second embodiment and the third embodiment described above, the flash image component data D3 subjected to the white balance processing for canceling the influence of the flash light used at the time of photographing is performed. (See FIG. 9) can be acquired. The flash image component data D3 is data indicating the original color component ratio of the subject irradiated with flash light. Therefore, by using the flash image component data D3 as a reference (target), for the image area affected by the flash light, an appropriate white light for the ambient light is used regardless of whether the original subject color is an achromatic color or not. It is possible to acquire a balance gain.

  If the white balance gain for ambient light can be acquired, the brightness difference caused by the color of the subject is compensated by the same processing as the processing in the image processing apparatus 31 of the second and third embodiments described above. It is also possible to obtain “data indicating the influence of ambient light (environmental light image correction component data described later)”. By using such data indicating the influence of ambient light (environmental light image correction component data) in combination with the flash image correction component data D5 acquired in the second to third embodiments described above, a flash can be obtained. The ratio of the influence of the ambient light and the flash light can be obtained for each processing block of the luminescent image data D1. By using this “ratio of the influence of the ambient light and the flash light for each processing block of the flash emission image data D1”, the white balance processing for the flash light and / or the white balance processing for the environmental light is performed, thereby achieving high accuracy. White balance processing can be performed.

  Hereinafter, a specific functional configuration example according to the fourth embodiment will be described.

  FIG. 18 is a block diagram illustrating a functional configuration of the image processing apparatus 31 according to the fourth embodiment. The image processing apparatus 31 according to the present embodiment includes an ambient light image component acquisition unit 66, an ambient light, in addition to the flash image component acquisition unit 40, the flash correction data acquisition unit 42, the flash brightness correction unit 44, and the white balance processing unit 46 described above. A correction data acquisition unit 68 and an ambient light brightness correction unit 70 are provided.

  The flash image component acquisition unit 40, the flash correction data acquisition unit 42, and the flash brightness correction unit 44 have the same functional configuration as that of the third embodiment.

  The ambient light image component acquisition unit 66 uses the second image data (flash non-emission image data D2) acquired by photographing the subject under non-flash light as basic data, and the image component of the subject due to ambient light. Ambient light image component data D11 is obtained. Here, the basic data is data used as a basis when acquiring the ambient light image component data D11, and is not particularly limited as long as the ambient light image component data D11 indicating the image component of the subject by the ambient light can be derived. For example, the ambient light image component acquisition unit 66 may use “second image data (flash non-light emission image data D2)” itself as basic data, or “first image data (flash light emission image data D1)”. And “first difference data representing the difference between the first image data and the second image data (flash non-light-emitting image data D2)” and the second difference data representing the difference between the first image data and the second image data (flash non-light-emitting image data D2). Alternatively, the ambient light image component data D11 can be derived based on such basic data (for example, “second image data” or “second difference data”).

  FIG. 19 is a block diagram illustrating a functional configuration of the ambient light image component acquisition unit 66 according to the fourth embodiment. The ambient light image component acquisition unit 66 according to the present embodiment includes a second image data extraction unit 72 and a second extraction data adjustment unit 73.

  The second image data extraction unit 72 of this example is a processing unit that acquires basic data D14 of ambient light image component data indicating an image component of a subject by ambient light. Specifically, for example, “flash non-emission image data D2 only affected by ambient light” acquired by shooting without flash light is used as “basic data D14 of ambient light image component data”. Although possible, the acquisition method of the basic data D14 of the ambient light image component data is not limited to this.

  For example, the second image data extraction unit 72 of the ambient light image component acquisition unit 66 executes “flash emission image data (first image data) D1” and “flash emission image data D1 and the subject under non-emission of flash light”. The first difference data D6 representing the difference from the flash non-emission image data D2 (second image data) acquired by photographing the image, and the second difference data representing the difference is referred to as “environment light image component” It can be used as basic data D14 ". FIG. 20 is a block diagram illustrating a configuration example of the second image data extraction unit 72 of the ambient light image component acquisition unit 66. The second image data extraction unit 72 of this example includes a first subtracter 74 and a second subtracter 75. The first subtracter 74 receives the flash light emission image data D1 and the flash non-light emission image data D2, and is represented by “first difference data D6 = flash light emission image data D1−flash non-light emission image data D2”. 1 difference data D6 is calculated. The second subtractor 75 is input with the flash emission image data D1 and the first difference data D6, and “basic data (first RGB background influence map) D14 = second difference data = flash emission image data D1. -Calculate basic data D14 represented by "first difference data D6". The basic data D14 calculated by the second subtractor 75 is sent from the second image data extraction unit 72 to the second extraction data adjustment unit 73.

  FIG. 21 is a block diagram illustrating a configuration example of the second extraction data adjustment unit 73 of the ambient light image component acquisition unit 66. The second extracted data adjustment unit 73 of this embodiment includes a basic data color ratio acquisition unit 76, a white balance gain acquisition unit 77, and a second white balance adjustment unit 78.

  The basic data color ratio acquisition unit 76 (ambient light image component acquisition unit 66) acquires color ratio data D15 representing the color component ratio of the basic data D14 in each of the plurality of processing blocks.

  The white balance gain acquisition unit 77 (environmental light image component acquisition unit 66) is based on the color ratio data D7 of the flash image component data D3 and the color ratio data D15 of the basic data D14 in each of the plurality of processing blocks. A white balance gain WB2 is acquired. In the example shown in FIG. 21, the white balance gain acquisition unit 77 acquires the flash image component data D3 from the flash image component acquisition unit 40, and is the same as the first color ratio data acquisition unit 53 (see FIG. 12). By performing the processing, the color ratio data D7 of the flash image component data D3 in each of the plurality of processing blocks is acquired. However, the white balance gain acquisition unit 77 may acquire the color ratio data D7 of the flash image component data D3 from the first color ratio data acquisition unit 53 of the flash correction data acquisition unit 42.

  The flash image component data D3 is data in which the original color tone of the subject is reproduced by applying the white balance gain WB1 for flash light, and the “original color tone of the subject” is obtained by irradiating the subject with flash light. The same applies to the case where the subject is photographed with only the ambient light irradiated. Accordingly, for the same processing block, the white balance gain acquisition unit 77 applies the ambient light white balance gain WB2 to the color ratio data D15 of the basic data D14 to obtain the color ratio data of the flash image component data D3. The ambient light white balance gain WB2 is acquired so as to be equal to D7.

  Of the plurality of processing blocks constituting the flash image component data D3, the data of “processing block that is not affected by flash light but affected only by ambient light” has “original color tone of subject”. It is not reproduced data. Therefore, when the white balance gain acquisition unit 77 acquires the ambient light white balance gain WB2, among the color ratio data D7 of the flash image component data D3, the “processing block affected by only ambient light” is selected. The data is not used, and the ambient light white balance gain WB2 is acquired using the data of the “processing block affected by the flash light”.

  As described above, the ambient light white balance gain WB2 obtained by setting the color ratio data D7 of the flash image component data D3 representing the “original color tone of the subject” as the target data is the basic data in each of the plurality of processing blocks. When applied to the color ratio data D15 of D14, the color ratio data of the ambient light image component data D11 in each of the plurality of processing blocks is derived. Therefore, the second white balance adjustment unit 78 (environmental light image component acquisition unit 66) applies the environmental light white balance gain WB2 to the basic data D14 to thereby obtain the environmental light image component data (second RGB background influence map). ) Get D11.

  Since the ambient light image component data D11 may vary somewhat depending on the subject, the total average value of the ambient light white balance gains WB2 obtained for a plurality of processing blocks is applied to the basic data D14 for each processing block. The ambient light white balance gain WB2 may be used. However, the ambient light is not composed of a single light source, but is derived for each processing block (for each pixel), such as when a plurality of light sources are mixed or when a moving object is included in the subject. If the ambient light white balance gain WB2 is applied to the basic data D14 of the corresponding processing block, the image quality may be improved. Therefore, it is not necessary to apply the same ambient light white balance gain WB2 uniformly to all the processing blocks of the basic data D14.

  The ambient light image component data D11 obtained in this way is “captured image data indicating the influence of ambient light” that is not affected by the flash light.

  The ambient light image component acquisition unit 66 (second image data extraction unit 72) acquires the first difference data D6 and the basic data D14 using the flash emission image data instead of the flash non-emission image data D2. Also good. In other words, the ambient light image component acquisition unit 66 “the first image data (flash emission image data D1) acquired by photographing the subject under the emission of the first emission amount of flash light” and “this first emission data”. The difference between the first image data and the second image data (flash emission image data) acquired by photographing the subject under the flash light emission of the second emission amount different from the first emission amount The basic data D14 may be derived based on the first difference data D6 "representing the ambient light image component data D11 based on the basic data D14.

  FIG. 22 is a block diagram illustrating a functional configuration of the ambient light correction data acquisition unit 68.

  The ambient light correction data acquisition unit 68 includes a second color ratio data acquisition unit 80 and a second correction data calculation unit 81. Based on the ambient light image component data D11, the ambient light provided according to the color of the subject. Environmental light correction data (color-specific environmental light influence degree map) D12 reflecting the brightness difference of the image component data D11 is acquired. That is, the second color ratio data acquisition unit 80 acquires color ratio data D16 representing the color component ratio of the ambient light image component data D11 in each of the plurality of processing blocks from the ambient light image component data D11. Based on the color ratio data D16 of the ambient light image component data D11, the second correction data calculation unit 81 calculates the difference in brightness between the plurality of processing blocks of the ambient light image component data D11 that is provided according to the color of the subject. The reflected ambient light correction data D12 is acquired.

  The ambient light correction data acquisition unit 68 of the present embodiment reduces the color-dependent influence based on the spectral characteristics (spectral reflectance) of the subject, and the color ratio of the ambient light image component data D11 acquired for each processing block. Brightness index data (luminance data) is acquired from the data (RGB ratio, etc.), and ambient light correction data D12 (correction gain) for compensating for the difference in brightness (luminance) between colors is acquired from the brightness index data.

  FIG. 23 is a block diagram illustrating a functional configuration of the second color ratio data acquisition unit 80.

  The second color ratio data acquisition unit 80 of this example includes a second demosaic processing unit 82 and a second color ratio calculation unit 83.

  The second demosaic processing unit 82 performs demosaic processing of the ambient light image component data D11. By this demosaic processing, color component data D17 is obtained in which pixel data relating to each color (RGB or the like) constituting the color filter 16 of the image sensor 21 is assigned to each pixel constituting the ambient light image component data D11. The The second color ratio calculation unit 83 acquires the color ratio data D16 of the ambient light image component data D11 in each of the plurality of processing blocks based on the color component data D17. The color ratio data D16 indicates a ratio (RGB ratio or the like) of the color component data D17 constituting the ambient light image component data D11, and is calculated for each processing block.

  The ambient light image component data D11 includes color component data relating to each of a plurality of colors (RGB, etc.) constituting the color filter 16 of the image sensor 21. The second color ratio data acquisition unit 80 (environment light correction data acquisition unit 68) acquires the color ratio data D16 of the environment light image component data D11 in each of the plurality of processing blocks based on the color component data.

  The color ratio data D16 of the ambient light image component data D11 is the ambient light image component data related to each of the plurality of colors with respect to the sum of the multiple color component data D17 of the ambient light image component data D11 in each of the plurality of processing blocks. This is based on the ratio of the color component data D17 of D11.

  In the above-described example, the demosaic processing of the ambient light image component data D11 is performed by the second demosaic processing unit 82. However, when the processing block includes a plurality of pixels, the second demosaic processing unit 82 (demosaic processing) May be omitted. When the second demosaic processing unit 82 is omitted, the color ratio data D16 in each processing block can be acquired from the color component data D17 related to a plurality of pixels included in each processing block. As described above, the method for obtaining the monochrome color component data D17 in each processing block from the data of a plurality of pixels is not particularly limited. For example, the “average value” of the data of a plurality of pixels related to the monochrome is used as the monochrome color component data D17. It may be set.

  FIG. 24 is a block diagram showing a functional configuration of the second correction data calculation unit 81. The second correction data calculation unit 81 of this example includes a second brightness index data acquisition unit 84 and a second correction data calculation processing unit 85.

  The second brightness index data acquisition unit 84 acquires brightness index data D18 from the color ratio data D16. That is, the second lightness index data acquisition unit 84 (ambient light correction data acquisition unit 68) reads “the lightness index data D18 indicating the lightness of the environmental light image component data D11 according to the color of the subject. Based on the “calculation process derived from the color ratio data D16”, the lightness index data of the ambient light image component data D11 in each of the plurality of processing blocks is obtained from the color ratio data D16 of the ambient light image component data D11 in each of the plurality of processing blocks. D18 is acquired.

  The second correction data calculation processing unit 85 acquires the ambient light correction data D12 from the brightness index data D18. In particular, the second correction data calculation processing unit 85 (environment light correction data acquisition unit 68) of the present embodiment acquires the environment light correction data D12 related to each of the plurality of processing blocks on the basis of the achromatic color ratio data. . That is, the second correction data calculation processing unit 85 (environment light correction data acquisition unit 68) has a ratio of the color component data for each of the plurality of colors to the sum of the color component data of the plurality of colors in each of the plurality of processing blocks. Lightness index data related to the achromatic color is acquired from the color ratio data of the same achromatic color based on the arithmetic processing. Then, the second correction data calculation processing unit 85 (environment light correction data acquisition unit 68) is based on the ratio of the lightness index data D18 of the environmental light image component data D11 in each of the plurality of processing blocks to the lightness index data regarding the achromatic color. Thus, the ambient light correction data D12 relating to each of the plurality of processing blocks of the ambient light image component data D11 is acquired.

  The “calculation process” used to acquire the brightness index data D18 is not particularly limited as long as it is a process for deriving the brightness index data D18 indicating the brightness directly or indirectly. An arithmetic process used to calculate “value” may be used. Therefore, even if the lightness index data D18 is calculated based on an expression represented by, for example, “Y = 0.3 * R + 0.6 * G + 0.1 * B”, which is used for calculating “luminance value (Y)”. Good. In this case, the ambient light correction data D12 obtained by the second correction data calculation processing unit 85 is based on the ratio of the brightness index data D18 of the ambient light image component data D11 in the processing block to the achromatic color brightness index data D18. To be acquired.

  The ambient light brightness correction unit 70 shown in FIG. 18 obtains “the ambient light image component data D11 output from the ambient light image component acquisition unit 66 (second image data extraction unit 72)” acquired as described above, and Based on the “ambient light correction data D12 output from the environmental light correction data acquisition unit 68 (second correction data calculation processing unit 85)”, the difference in brightness of the environmental light image component data D11 brought about according to the color of the subject. To obtain the ambient light image correction component data (synthesized ambient light influence map) D13.

  FIG. 25 is a block diagram illustrating a functional configuration of the ambient light brightness correction unit 70.

  The ambient light brightness correction unit 70 of the present embodiment includes a second brightness data acquisition unit 86 and a second correction calculation unit 87.

  The second brightness data acquisition unit 86 of the ambient light brightness correction unit 70 acquires second brightness data (luminance ambient light influence map) D19 in each of the plurality of processing blocks based on the ambient light image component data D11. To do. The second lightness data D19 in the present embodiment is data representing luminance, and the second lightness data D19 may be obtained by any method. That is, the second lightness data acquisition unit 86 converts the color component data (RGB data or the like) of the ambient light image component data D11 into luminance and acquires the second lightness data D19. For example, in the same manner as the “calculation process” used to acquire the brightness index data D18 in the second brightness index data acquisition unit 84 (environment light correction data acquisition unit 68, second correction data calculation unit 81), Second brightness data D19 (luminance value (Y)) can be calculated from the ambient light image component data D11.

  The second correction calculation unit 87 of the ambient light brightness correction unit 70 obtains the ambient light image correction component data D13 by applying the ambient light correction data D12 to the second brightness data D19 for each of the plurality of processing blocks. To do. More specifically, the ambient light correction data D12 acquired by the ambient light correction data acquisition unit 68 of the present embodiment is applied to the second brightness data D19 indicating the brightness of the ambient light image component data D11. This is data that compensates for the difference in brightness of the ambient light image component data D11 caused according to the color of the subject. The ambient light correction data acquisition unit 68 acquires such ambient light correction data D12 for each of the plurality of processing blocks of the ambient light image component data D11.

  Therefore, for example, by multiplying the second lightness data D19 by the ambient light correction data D12 for each processing block, the ambient light image correction component data D13 in which the difference in lightness caused according to the color of the subject is compensated is calculated. You may be made to do. In this case, the ambient light correction data acquisition unit 68 (second correction data calculation processing unit 85 (see FIG. 24)) reads “the ratio of the brightness index data D18 of the ambient light image component data D11 to the achromatic color brightness index data D18. Can be acquired as the ambient light correction data D12.

  As described above, the ambient light image correction component data D13 obtained by applying the ambient light correction data D12 to the second brightness data D19 indicating the luminance (lightness) of the ambient light image component data D11 is the color of the subject. Accordingly, the difference in luminance (brightness) of the ambient light image component data D11 is compensated.

  Then, the white balance processing unit 46 of the present embodiment shown in FIG. 18 performs the white balance of the flash emission image data (first image data) D1 based on the flash image correction component data D5 and the ambient light image correction component data D13. adjust. That is, the white balance processing unit 46 adds the flash light white balance gain WB1 and the ambient light to the flash emission image data (first image data) D1 based on the flash image correction component data D5 and the ambient light image correction component data D13. The white balance gain WB2 is applied.

  FIG. 26 is a block diagram illustrating a functional configuration of the white balance processing unit 46 according to the fourth embodiment.

  The white balance processing unit 46 of this example includes a light source ratio data acquisition unit 88 and a white balance calculation processing unit 89. The light source ratio data acquisition unit 88 uses the flash image correction component data D5 and the ambient light image correction component data D13 to provide light source ratio data (total flash influence map) D20 indicating the ratio of flash light to ambient light for each processing block. To get.

  The white balance calculation processing unit 89 switches the application ratio of the flash light white balance gain WB1 and the ambient light white balance gain WB2 for each processing block based on the light source ratio data D20, and performs white balance on the flash emission image data D1. Process.

  The method by which the white balance calculation processing unit 89 (white balance processing unit 46) acquires the flash emission image data D1, the flash light white balance gain WB1, and the ambient light white balance gain WB2 is not particularly limited. For example, the white balance calculation processing unit 89 may acquire the flash emission image data D1 from the flash image component acquisition unit 40 or the ambient light image component acquisition unit 66. Further, the white balance calculation processing unit 89 may acquire the flash light white balance gain WB1 from the flash light WB gain storage unit 51 (see FIG. 9). The white balance calculation processing unit 89 may acquire the ambient light white balance gain WB2 from the white balance gain acquisition unit 77 (see FIG. 21).

  The specific method of the white balance processing calculation performed by the white balance calculation processing unit 89 is not particularly limited, and the flash light white balance gain WB1 and the ambient light white balance gain WB2 are both flash emission images for each processing block. It may be applied to the data D1. For example, when the influence ratio of flash light and ambient light in “a certain processing block” of the flash emission image data D1 is “flash light: environment light = 2: 1”, the white balance gain for the flash emission image data D1 in that processing block May be set as “white balance gain for flash light WB1: white balance gain for ambient light WB2 = 2 / (2 + 1): 1 / (2 + 1)”. Alternatively, for example, when the influence ratio of the flash light is larger than the ambient light in the “certain processing block” of the flash emission image data D1, the flash light white balance gain WB1 is applied to the processing block. When the influence ratio of the ambient light is larger than that of the light, the ambient light white balance gain WB2 may be applied to the processing block.

  As described above, the white balance processing using the flash light as the light source and the white balance using the environmental light as the light source according to the ratio between the influence amount of the flash light and the influence amount of the environmental light in each processing block of the flash emission image data D1. By performing the processing, appropriate white balance processing is performed for each of “image portion relatively affected by flash light” and “image portion relatively affected by ambient light” in the flash emission image data D1. It can be performed.

  FIG. 27 is a flowchart of white balance processing according to the fourth embodiment.

  First, the flash image component acquisition unit 40 and the ambient light image component acquisition unit 66 acquire the flash emission image data D1 and the flash non-emission image data D2 (S41 in FIG. 27). Then, the flash image component acquisition unit 40, the flash correction data acquisition unit 42, and the flash brightness correction unit 44 acquire the flash image correction component data D5 based on the flash emission image data D1 and the flash non-emission image data D2 (S42). The ambient light image component acquisition unit 66, the ambient light correction data acquisition unit 68, and the ambient light brightness correction unit 70 acquire the ambient light image correction component data D13 based on the flash emission image data D1 and the flash non-emission image data D2. (S43).

  The process of step S42 for acquiring the flash image correction component data D5 will be described later with reference to FIG. The process of step S43 for acquiring the ambient light image correction component data D13 will be described later with reference to FIG.

  Then, the light source ratio data acquisition unit 88 of the white balance processing unit 46 acquires the light source ratio data D20 based on the flash image correction component data D5 and the ambient light image correction component data D13 (S44).

  Then, in the white balance calculation processing unit 89 of the white balance processing unit 46, white balance processing corresponding to the light source ratio data D20 is performed on the flash emission image data D1 (S45). For example, the flash light white balance gain WB1 and the ambient light white balance gain WB2 corresponding to the light source ratio data D20 are applied to the flash emission image data D1.

  FIG. 28 is a flowchart showing details of a processing flow (step S42 in FIG. 27) for acquiring the flash image correction component data D5 in the fourth embodiment. Note that steps S51 to S57 shown in FIG. 28 are the same processes as steps S32 to S38 shown in FIG. 17 according to the third embodiment.

  That is, the first image data extraction unit 47 (see FIG. 7) of the flash image component acquisition unit 40 acquires the first difference data D6 from the flash emission image data D1 and the flash non-emission image data D2 (S51 in FIG. 28). ), The first extracted data adjustment unit 49 obtains the flash image component data D3 (S52).

  Then, in the first color ratio data acquisition unit 53 (see FIG. 12) of the flash correction data acquisition unit 42, the color ratio data D7 regarding each processing block is acquired from the flash image component data D3 (S53). Then, in the first lightness index data acquisition unit 59 (see FIG. 14) of the first correction data calculation unit 54, the lightness index data D9 relating to each processing block is acquired from the color ratio data D7 (S54). The correction data calculation processing unit 60 acquires flash correction data (color-specific flash influence map) D4 related to each processing block from the lightness index data D9 (S55).

  Then, in the first brightness data acquisition unit 62 (see FIG. 16) of the flash brightness correction unit 44, the first brightness data D10 relating to each processing block is acquired from the flash image component data D3 acquired by the flash image component acquisition unit 40. (S56). Then, in the first correction calculation unit 63 of the flash brightness correction unit 44, flash image correction component data D5 is acquired from the first brightness data D10 and the flash correction data D4 (S57).

  FIG. 29 is a flowchart showing details of a processing flow (step S43 in FIG. 27) for acquiring the ambient light image correction component data D13 in the fourth embodiment.

  First, in the second image data extraction unit 72 (see FIG. 19) of the ambient light image component acquisition unit 66, basic data D14 is acquired based on the flash non-emission image data D2 (S61 in FIG. 29).

  The basic data color ratio acquisition unit 76 (see FIG. 21) acquires the color ratio data D15 from the basic data D14 (S62), and the white balance gain acquisition unit 77 (see FIG. 21) acquires the color of the flash image component data D3. The ambient light white balance gain WB2 is acquired from the ratio data D7 and the color ratio data D15 of the basic data D14 (S63).

  Then, in the second white balance adjustment unit 78 (see FIG. 21), the ambient light white balance gain WB2 is applied to the basic data D14 to obtain the ambient light image component data D11 (S64). Then, the second color ratio data acquisition unit 80 (see FIG. 22) of the ambient light correction data acquisition unit 68 acquires the color ratio data D16 of the ambient light image component data D11 for each processing block (S65). In the second brightness index data acquisition unit 84 (see FIG. 24) of the correction data calculation unit 81, brightness index data D18 is acquired for each processing block from the color ratio data D16 of the ambient light image component data D11 (S66). Then, in the second correction data calculation processing unit 85 (see FIG. 24), the ambient light correction data D12 is acquired from the lightness index data D18 (S67).

  Then, the second brightness data acquisition unit 86 (see FIG. 25) of the ambient light brightness correction unit 70 acquires the second brightness data D19 from the ambient light image component data D11 (S68), and the second correction calculation unit 87. In (see FIG. 25), ambient light image correction component data D13 is acquired from the second brightness data D19 and the ambient light correction data D12 (S69).

  As described above, according to the image processing apparatus 31 and the image processing method of the present embodiment, the influence degree of the flash light and the influence degree of the environmental light can be accurately obtained, and the accurate influence degree and environment of the flash light can be obtained. Based on the degree of influence of light, highly accurate white balance processing can be performed on the flash emission image data D1.

  When shooting under flash light, there may be cases where the flash light does not reach all or part of the subject. It may be difficult to adjust to. However, according to the image processing apparatus 31 and the image processing method of the present embodiment, since the white balance processing is performed in consideration of the influence degree of the flash light and the influence degree of the ambient light, the original color of the subject is appropriately recovered. It becomes possible.

  For example, “the image portion where the flash light reaches sufficiently and the flash light has a dominant influence” and “the image portion where the flash light does not reach sufficiently and the flash light does not have a dominant influence” are flash emission images Even when the data D1 is mixed, appropriate white balance processing can be performed for each image portion in a state where a difference in brightness caused by the color of the subject is compensated.

<Other variations>
Arbitrary forms may be combined among the above-described embodiments and modifications. The above-described embodiments are merely examples, and the present invention may be applied to other configurations.

  In each of the first to fourth embodiments described above, the example in which the acquisition processing of the flash image correction component data D5 and the image processing using the flash image correction component data D5 are performed in the camera body 3 has been described. Some or all of these processes may be performed by another device such as a computer or server other than the camera body 3 (digital camera 2). For example, when processing image data in a computer, data such as a flash light emission image D1 and a flash non-light emission image D2 are input to an image processing device provided in the computer, so that the flash image correction component data D5 is acquired. In addition, image processing using the flash image correction component data D5 may be performed. When the server includes an image processing device, for example, data such as the flash light emission image D1 and the flash non-light emission image D2 are transmitted from the digital camera 2 (camera body 3) or the computer to the server, and the server image processing device. The image data is subjected to the flash image correction component data D5 acquisition processing and the image processing using the flash image correction component data D5, and the image data after the image processing is transmitted and provided to the transmission source. Good.

  Each functional configuration described above can be realized by arbitrary hardware, software, or a combination of both. For example, a program for causing a computer to execute an image processing method (processing step (processing procedure)) in each of the above-described apparatuses and processing units (image processing apparatus 31 and the like), and a computer-readable storage medium (non-temporary) storing the program The present invention can also be applied to a storage medium) or a computer in which the program can be installed.

  In addition, the aspect to which the present invention can be applied is not limited to a digital camera and a computer (server). In addition to the cameras whose main function is imaging, functions other than imaging (call function, The present invention can be applied to mobile devices having a communication function and other computer functions. Other modes to which the present invention can be applied include, for example, a mobile phone having a camera function, a smartphone, a PDA (Personal Digital Assistants), and a portable game machine. Hereinafter, an example of a smartphone to which the present invention can be applied will be described.

<Configuration of smartphone>
FIG. 30 is a diagram illustrating an appearance of the smartphone 101. A smartphone 101 illustrated in FIG. 30 includes a flat housing 102, and a display input in which a display panel 121 as a display unit and an operation panel 122 as an input unit are integrated on one surface of the housing 102. The unit 120 is provided. The housing 102 includes a speaker 131, a microphone 132, an operation unit 140, and a camera unit 141. Note that the configuration of the housing 102 is not limited to this, and for example, a configuration in which the display unit and the input unit are independent may be employed, or a configuration having a folding structure and a slide mechanism may be employed.

  FIG. 31 is a block diagram showing the configuration of the smartphone 101 shown in FIG. As shown in FIG. 31, the main components of the smartphone include a wireless communication unit 110, a display input unit 120, a call unit 130, an operation unit 140, a camera unit 141, a storage unit 150, and an external input / output unit. 160, a GPS (Global Positioning System) receiving unit 170, a motion sensor unit 180, a power supply unit 190, and a main control unit 100. As a main function of the smartphone 101, a wireless communication function for performing mobile wireless communication via the base station apparatus BS and the mobile communication network NW is provided.

  The radio communication unit 110 performs radio communication with the base station apparatus BS accommodated in the mobile communication network NW according to an instruction from the main control unit 100. Using such wireless communication, transmission / reception of various file data such as audio data and image data, e-mail data, and reception of Web data, streaming data, and the like are performed.

  The display input unit 120 displays images (still images and moving images), character information, and the like visually under the control of the main control unit 100, visually transmits information to the user, and detects user operations on the displayed information. This is a so-called touch panel, and includes a display panel 121 and an operation panel 122.

  The display panel 121 uses an LCD (Liquid Crystal Display), an OELD (Organic Electro-Luminescence Display), or the like as a display device. The operation panel 122 is a device that is placed so that an image displayed on the display surface of the display panel 121 is visible and detects coordinates operated by a user's finger or stylus. When such a device is operated with a user's finger or a stylus, a detection signal generated due to the operation is output to the main control unit 100. Next, the main control unit 100 detects an operation position (coordinates) on the display panel 121 based on the received detection signal.

  As shown in FIG. 30, the display panel 121 and the operation panel 122 of the smartphone 101 exemplified as an embodiment of the imaging apparatus of the present invention integrally constitute the display input unit 120. The arrangement 122 covers the display panel 121 completely. When such an arrangement is adopted, the operation panel 122 may have a function of detecting a user operation even in an area outside the display panel 121. In other words, the operation panel 122 includes a detection area (hereinafter referred to as “display area”) for the overlapping portion overlapping the display panel 121 and a detection area (hereinafter referred to as “display area”) for the other outer edge portion that does not overlap the display panel 121. (Referred to as “non-display area”).

  Note that the size of the display area and the size of the display panel 121 may be completely matched, but it is not always necessary to match the two. In addition, the operation panel 122 may include two sensitive regions of the outer edge portion and the other inner portion. Furthermore, the width of the outer edge portion is appropriately designed according to the size of the housing 102 and the like. Furthermore, examples of the position detection method employed in the operation panel 122 include a matrix switch method, a resistance film method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, and a capacitance method. You can also

  The call unit 130 includes a speaker 131 and a microphone 132, converts user's voice input through the microphone 132 into voice data that can be processed by the main control unit 100, and outputs the voice data to the main control unit 100. Alternatively, the audio data received by the external input / output unit 160 is decoded and output from the speaker 131. As shown in FIG. 30, for example, the speaker 131 can be mounted on the same surface as the surface on which the display input unit 120 is provided, and the microphone 132 can be mounted on the side surface of the housing 102.

  The operation unit 140 is a hardware key using a key switch or the like, and receives an instruction from the user. For example, as illustrated in FIG. 30, the operation unit 140 is mounted on the side surface of the housing 102 of the smartphone 101 and is turned on when pressed with a finger or the like, and is turned off when a finger is released with a restoring force such as a spring. It is a push button type switch.

  The storage unit 150 includes a control program and control data of the main control unit 100, application software, address data in association with the name and telephone number of a communication partner, transmitted / received e-mail data, Web data downloaded by Web browsing, The downloaded content data is stored, and streaming data and the like are temporarily stored. The storage unit 150 includes an internal storage unit 151 with a built-in smartphone and an external storage unit 152 having a removable external memory slot. Each of the internal storage unit 151 and the external storage unit 152 included in the storage unit 150 includes a flash memory type, a hard disk type, a multimedia card micro type, and a multimedia card micro type. It is realized using a storage medium such as a card type memory (for example, MicroSD (registered trademark) memory), a RAM (Random Access Memory), a ROM (Read Only Memory) or the like.

  The external input / output unit 160 serves as an interface with all external devices connected to the smartphone 101, and communicates with other external devices (for example, universal serial bus (USB), IEEE 1394, etc.) or a network. (For example, the Internet, wireless LAN, Bluetooth (registered trademark), RFID (Radio Frequency Identification), Infrared Data Association (IrDA) (registered trademark), UWB (Ultra Wideband) (registered trademark) ZigBee) (registered trademark, etc.) for direct or indirect connection.

  As an external device connected to the smartphone 101, for example, a wired / wireless headset, a wired / wireless external charger, a wired / wireless data port, a memory card (Memory card) or SIM (Subscriber) connected via a card socket, for example. Identity Module Card (UIM) / User Identity Module Card (UIM) card, external audio / video equipment connected via audio / video I / O (Input / Output) terminal, external audio / video equipment connected wirelessly, existence / non-existence There are wirelessly connected smart phones, wired / wireless connected personal computers, wired / wireless connected PDAs, wired / wireless earphones, and the like. The external input / output unit may transmit data received from such an external device to each component inside the smartphone 101 or may transmit data inside the smartphone 101 to the external device. .

  The GPS receiving unit 170 receives GPS signals transmitted from the GPS satellites ST1 to STn in accordance with instructions from the main control unit 100, executes positioning calculation processing based on the received plurality of GPS signals, and calculates the latitude and longitude of the smartphone 101. , Detect the position consisting of altitude. If the GPS receiving unit 170 can acquire position information from the wireless communication unit 110 or the external input / output unit 160 (for example, a wireless LAN), the GPS receiving unit 170 can also detect the position using the position information.

  The motion sensor unit 180 includes, for example, a three-axis acceleration sensor, and detects the physical movement of the smartphone 101 in accordance with an instruction from the main control unit 100. By detecting the physical movement of the smartphone 101, the moving direction and acceleration of the smartphone 101 are detected. The detection result is output to the main control unit 100.

  The power supply unit 190 supplies power stored in a battery (not shown) to each unit of the smartphone 101 in accordance with an instruction from the main control unit 100.

  The main control unit 100 includes a microprocessor, operates according to a control program and control data stored in the storage unit 150, and controls each unit of the smartphone 101 in an integrated manner. In addition, the main control unit 100 includes a mobile communication control function for controlling each unit of the communication system and an application processing function in order to perform voice communication and data communication through the wireless communication unit 110.

  The application processing function is realized by the main control unit 100 operating according to application software stored in the storage unit 150. Application processing functions include, for example, an infrared communication function that controls the external input / output unit 160 to perform data communication with the opposite device, an e-mail function that transmits and receives e-mails, and a web browsing function that browses web pages. .

  The main control unit 100 also has an image processing function such as displaying video on the display input unit 120 based on image data (still image or moving image data) such as received data or downloaded streaming data. The image processing function is a function in which the main control unit 100 decodes the image data, performs image processing on the decoding result, and displays an image on the display input unit 120.

  Further, the main control unit 100 executes display control for the display panel 121 and operation detection control for detecting a user operation through the operation unit 140 and the operation panel 122.

  By executing the display control, the main control unit 100 displays an icon for starting application software, a software key such as a scroll bar, or a window for creating an e-mail. Note that the scroll bar refers to a software key for accepting an instruction to move the display portion of a large image that does not fit in the display area of the display panel 121.

  In addition, by executing the operation detection control, the main control unit 100 detects a user operation through the operation unit 140, or accepts an operation on the icon or an input of a character string in the input field of the window through the operation panel 122. Or a display image scroll request through a scroll bar.

  Further, by executing the operation detection control, the main control unit 100 causes the operation position with respect to the operation panel 122 to overlap with the display panel 121 (display area) or other outer edge part (non-display area) that does not overlap with the display panel 121. And a touch panel control function for controlling the sensitive area of the operation panel 122 and the display position of the software key.

  The main control unit 100 can also detect a gesture operation on the operation panel 122 and execute a preset function according to the detected gesture operation. Gesture operation is not a conventional simple touch operation, but an operation of drawing a trajectory with at least one position from a plurality of positions by drawing a trajectory with a finger or the like, or specifying a plurality of positions simultaneously. means.

  The camera unit 141 is a digital camera that performs electronic photography using an imaging device such as a CMOS. In addition, the camera unit 141 converts image data obtained by imaging into compressed image data such as JPEG under the control of the main control unit 100, stores the compressed image data in the storage unit 150, the external input / output unit 160, the wireless communication unit, and the like. 110 can be output. As shown in FIG. 30, in the smartphone 101, the camera unit 141 is mounted on the same surface as the display input unit 120, but the mounting position of the camera unit 141 is not limited to this, and is mounted on the back surface of the display input unit 120. Alternatively, a plurality of camera units 141 may be mounted. When a plurality of camera units 141 are installed, the camera unit 141 used for shooting may be switched to perform shooting alone, or a plurality of camera units 141 may be used for shooting simultaneously. Also good.

  The camera unit 141 can be used for various functions of the smartphone 101. For example, an image acquired by the camera unit 141 can be displayed on the display panel 121, or the image of the camera unit 141 can be used as one of operation inputs of the operation panel 122. Further, when the GPS receiving unit 170 detects the position, the position can also be detected with reference to an image from the camera unit 141. Furthermore, referring to the image from the camera unit 141, the optical axis direction of the camera unit 141 of the smartphone 101 is determined without using the triaxial acceleration sensor or in combination with the triaxial acceleration sensor. It is also possible to determine the current usage environment. Of course, the image from the camera unit 141 can also be used in the application software.

  In addition, the position information acquired by the GPS receiver 170 on the image data of the still image or the moving image, the voice information acquired by the microphone 132 (the voice text may be converted by the main control unit or the like to become text information), Posture information and the like acquired by the motion sensor unit 180 may be added and stored in the storage unit 150 and output through the external input / output unit 160 or the wireless communication unit 110.

  The above-described image processing apparatus 31 can be realized by the main control unit 100, for example.

  2 ... Digital camera, 3 ... Camera body, 4 ... Lens barrel, 5 ... Flash light emitting unit, 6 ... Shutter button, 7 ... Power switch, 8 ... Display unit, 9 ... Operation unit, 10 ... Main memory, 12 ... Lens Part, 14 pixels, 16 color filter, 20 mechanical shutter, 21 imaging device, 22 process processing unit, 23 AD conversion unit, 24 buffer memory, 25 system control unit, 26 shutter drive unit, DESCRIPTION OF SYMBOLS 27 ... Lens drive part, 28 ... Power supply control part, 29 ... Power supply, 30 ... Control memory, 31 ... Image processing apparatus, 32 ... Compression / decompression part, 33 ... Storage control part, 34 ... Clock device, 35 ... Display control part, 36 ... User interface, 40 ... Flash image component acquisition unit, 42 ... Flash correction data acquisition unit, 44 ... Flash brightness correction unit, 46 ... White balance processing , 47 ... first image data extraction unit, 48 ... subtractor, 49 ... first extraction data adjustment unit, 50 ... first white balance adjustment unit, 51 ... WB light storage unit for flash light, 53 ... first 1 color ratio data acquisition unit, 54 ... first correction data calculation unit, 56 ... first demosaic processing unit, 57 ... first color ratio calculation unit, 59 ... first lightness index data acquisition unit, 60 ... First correction data calculation processing unit, 62... First brightness data acquisition unit, 63... First correction calculation unit, 66 .. Ambient light image component acquisition unit, 68. Degree correction unit, 72 ... second image data extraction unit, 73 ... second extraction data adjustment unit, 74 ... first subtractor, 75 ... second subtractor, 76 ... basic data color ratio acquisition unit, 77 ... White balance gain acquisition unit, 78 ... Second white balance tone 80, second color ratio data acquisition unit, 81, second correction data calculation unit, 82, second demosaic processing unit, 83, second color ratio calculation unit, 84, second brightness index data. Acquisition unit, 85 ... second correction data calculation processing unit, 86 ... second brightness data acquisition unit, 87 ... second correction calculation unit, 88 ... light source ratio data acquisition unit, 89 ... white balance calculation processing unit, 100 DESCRIPTION OF SYMBOLS ... Main control part, 101 ... Smartphone, 102 ... Case, 110 ... Wireless communication part, 120 ... Display input part, 121 ... Display panel, 122 ... Operation panel, 130 ... Call part, 131 ... Speaker, 132 ... Microphone, 140 ... Operation unit 141 ... Camera unit 150 ... Storage unit 151 ... Internal storage unit 152 ... External storage unit 160 ... External input / output unit 170 ... GPS reception unit 180 ... Motion sensor unit 190 ... Power source Part

Claims (31)

A flash image component acquisition unit that acquires flash image component data indicating an image component of the subject by the flash light based on the first image data acquired by photographing the subject under flash light emission;
Based on the flash image component data, a flash correction data acquisition unit that acquires flash correction data that reflects a difference in brightness of the flash image component data brought about according to the color of the subject;
A flash brightness correction unit that acquires flash image correction component data in which a difference in brightness of the flash image component data caused according to the color of the subject is compensated based on the flash image component data and the flash correction data; An image processing apparatus.   The flash image component acquisition unit includes first difference data representing a difference between the first image data and second image data acquired by photographing the subject under non-flash light emission. The image processing apparatus according to claim 1, wherein the flash image component data is acquired.   The flash image component acquisition unit is different from the first image data acquired by photographing the subject under the emission of flash light having a first light emission amount and a second light emission amount different from the first light emission amount. 2. The flash image component data according to claim 1, wherein the flash image component data is acquired from first difference data representing a difference from second image data acquired by photographing the subject under the light emission amount of flash light. Image processing device.   The image processing apparatus according to claim 2, wherein the flash image component acquisition unit acquires the flash image component data by applying a flash light white balance gain to the first difference data. The flash correction data acquisition unit
Obtaining color ratio data representing a color component ratio of the flash image component data in each of the plurality of processing blocks of the flash image component data divided into a plurality of processing blocks;
The flash correction data that reflects a difference in lightness among the plurality of processing blocks of the flash image component data that is produced according to the color of the subject is acquired based on the color ratio data of the flash image component data. Item 5. The image processing device according to any one of Items 1 to 4. The flash correction data is applied to the first brightness data indicating the brightness of the flash image component data, thereby compensating for the difference in brightness of the flash image component data caused according to the color of the subject. Yes,
The image processing apparatus according to claim 5, wherein the flash correction data acquisition unit acquires the flash correction data for each of the plurality of processing blocks of the flash image component data.   The image processing apparatus according to claim 5, wherein the flash correction data acquisition unit acquires the flash correction data relating to each of the plurality of processing blocks based on the color ratio data of an achromatic color. The first image data is obtained by an image sensor including a plurality of color filters, and the flash image component data includes color component data for each of the plurality of colors,
The flash correction data acquisition unit according to any one of claims 5 to 7, wherein the flash correction data acquisition unit acquires the color ratio data of the flash image component data in each of the plurality of processing blocks based on the color component data. Image processing device.   The color ratio data of the flash image component data includes the flash image component data of each of the plurality of colors with respect to a sum of the color component data of the plurality of colors of the flash image component data in each of the plurality of processing blocks. The image processing apparatus according to claim 8, wherein the image processing apparatus is based on a ratio of the color component data. The flash correction data acquisition unit
The flash image component in each of the plurality of processing blocks based on a calculation process for deriving lightness index data indicating the lightness of the flash image component data according to the color of the subject from the color ratio data of the flash image component data Obtaining the lightness index data of the flash image component data in each of the plurality of processing blocks from the color ratio data of the data;
From the color ratio data of the achromatic color in which the ratio of the color component data for each of the plurality of colors to the sum of the color component data of the plurality of colors in each of the plurality of processing blocks is equal, the lightness for the achromatic color Index data is acquired based on the calculation process,
The flash correction for each of the plurality of processing blocks of the flash image component data based on a ratio of the lightness index data of the flash image component data in each of the plurality of processing blocks to the lightness index data for the achromatic color. The image processing apparatus according to claim 9, which acquires data.   The image processing apparatus according to claim 5, wherein each of the plurality of processing blocks is configured by a single pixel.   The image processing apparatus according to claim 5, wherein each of the plurality of processing blocks includes a plurality of pixels. The image sensor used for capturing the first image data is a single-plate image sensor including a plurality of color filters, and a basic arrangement of specific color patterns is repeatedly arranged in the plurality of color filters. Composed of
The image processing apparatus according to claim 12, wherein the plurality of pixels constituting each of the plurality of processing blocks correspond to pixels constituting the basic array. An ambient light image component acquisition unit for acquiring ambient light image component data indicating an image component of the subject by ambient light;
Based on the ambient light image component data, an ambient light correction data acquisition unit that acquires ambient light correction data that reflects a difference in brightness of the ambient light image component data brought about according to the color of the subject;
Based on the ambient light image component data and the ambient light correction data, ambient light brightness for obtaining ambient light image correction component data in which a difference in brightness of the ambient light image component data caused according to the color of the subject is compensated is obtained. The image processing apparatus according to claim 5, further comprising a correction unit.   The ambient light image component acquisition unit acquires the ambient light image component data based on second image data acquired by capturing the subject under non-flash light emission as basic data. Image processing device.   The ambient light image component acquisition unit calculates a difference between the first image data, the first image data, and second image data acquired by photographing the subject under non-flash light emission. The image processing device according to claim 14, wherein the ambient light image component data is acquired using second difference data representing a difference between the first difference data to be represented as basic data. The ambient light image component acquisition unit
Obtaining color ratio data representing a color component ratio of the basic data in each of the plurality of processing blocks;
Obtaining a white balance gain for ambient light based on the color ratio data of the flash image component data and the color ratio data of the basic data in each of the plurality of processing blocks;
The image processing apparatus according to claim 15, wherein the ambient light image component data is acquired by applying the ambient light white balance gain to the basic data.   When the ambient light white balance gain is applied to the color ratio data of the basic data in each of the plurality of processing blocks, the color ratio data of the ambient light image component data in each of the plurality of processing blocks The image processing apparatus according to claim 17, wherein: The ambient light correction data acquisition unit
Obtaining color ratio data representing a color component ratio of the ambient light image component data in each of the plurality of processing blocks;
Based on the color ratio data of the ambient light image component data, the ambient light correction data reflecting a difference in lightness among the plurality of processing blocks of the ambient light image component data caused according to the color of the subject. The image processing apparatus according to any one of claims 14 to 18, which is acquired. The ambient light correction data is applied to second brightness data indicating the brightness of the ambient light image component data, thereby compensating for a difference in brightness of the ambient light image component data caused according to the color of the subject. Data to be
The image processing apparatus according to claim 19, wherein the ambient light correction data acquisition unit acquires the ambient light correction data for each of the plurality of processing blocks of the ambient light image component data.   21. The image processing apparatus according to claim 19, wherein the ambient light correction data acquisition unit acquires the ambient light correction data relating to each of the plurality of processing blocks based on the color ratio data of an achromatic color. The first image data is captured by an image sensor including a plurality of color filters, and the ambient light image component data includes color component data for each of the plurality of colors,
The ambient light correction data acquisition unit acquires the color ratio data of the ambient light image component data in each of the plurality of processing blocks based on the color component data. The image processing apparatus described.   The color ratio data of the ambient light image component data is the ambient light image relating to each of the plurality of colors with respect to a sum of the color component data of the plurality of colors of the ambient light image component data in each of the plurality of processing blocks. The image processing apparatus according to claim 22, wherein the image processing apparatus is based on a ratio of the color component data of component data. The ambient light correction data acquisition unit
The environment in each of the plurality of processing blocks based on a calculation process for deriving lightness index data indicating the lightness of the ambient light image component data according to the color of the subject from the color ratio data of the ambient light image component data Obtaining the lightness index data of the ambient light image component data in each of the plurality of processing blocks from the color ratio data of the light image component data;
From the color ratio data of the achromatic color in which the ratio of the color component data for each of the plurality of colors to the sum of the color component data of the plurality of colors in each of the plurality of processing blocks is equal, the lightness for the achromatic color Index data is acquired based on the calculation process,
Based on a ratio of the lightness index data of the ambient light image component data in each of the plurality of processing blocks to the lightness index data relating to the achromatic color, the related to each of the plurality of processing blocks of the ambient light image component data The image processing apparatus according to claim 23, wherein ambient light correction data is acquired.   25. The image processing apparatus according to claim 1, further comprising a white balance processing unit that adjusts a white balance of the first image data based on the flash image correction component data.   25. The white balance processing unit according to claim 14, further comprising a white balance processing unit that adjusts a white balance of the first image data based on the flash image correction component data and the ambient light image correction component data. Image processing device.   27. The image processing apparatus according to claim 1, wherein the brightness is luminance. An image sensor;
An image processing apparatus comprising: the image processing apparatus according to claim 1. Obtaining flash image component data indicating an image component of the subject by the flash light from first image data obtained by photographing the subject under light emission of flash light;
Obtaining flash correction data reflecting the difference in brightness of the flash image component data brought about according to the color of the subject based on the flash image component data;
Obtaining flash image correction component data in which a difference in brightness of the flash image component data caused according to the color of the subject is compensated based on the flash image component data and the flash correction data. Method. A procedure for acquiring flash image component data indicating an image component of the subject by the flash light from first image data acquired by photographing the subject under flash light emission; and
A procedure for acquiring flash correction data reflecting a difference in brightness of the flash image component data brought about according to the color of the subject based on the flash image component data;
A step of acquiring, in a computer, flash image correction component data in which a difference in brightness of the flash image component data caused according to the color of the subject is compensated based on the flash image component data and the flash correction data. Program to let you.   A non-transitory computer-readable recording medium on which the program according to claim 30 is recorded.

Images