JP5699482B2 - Image processing apparatus, image processing method, and imaging apparatus - Google Patents

Description

  The present invention relates to a technique for adjusting white balance of synthesized image data obtained by linearly synthesizing two or more image data obtained by imaging with different exposure times.

  Since an electronic camera has a narrow dynamic range (hereinafter referred to as “D range”) of an image sensor, exposure control and white balance adjustment are performed. In exposure control, the amount of light input to the image sensor is adjusted by the electronic shutter speed and the aperture so as to fall within the D range of the image sensor (see, for example, Patent Document 1). It also has an auto white balance (AWB) function for automatically adjusting white balance.

On the other hand, even when exposure control is performed and an appropriate image signal is obtained, if the contrast ratio of the subject is high, a region with a high luminance value is a saturated pixel. In a color imaging device equipped with a color filter, if a pixel of any color of red (R), green (G), or blue (B) is saturated beyond the D range, the color balance is lost and the image is lost. The color tone will become unnatural.
The AWB function is a technique for preventing this. For example, as shown in FIG. 15, the data of an unsaturated and low-saturated pixel is extracted from a video signal, and light measurement is performed based on the extracted pixel data. Estimate the color temperature. Next, a gain for correcting the R and B levels is calculated based on the estimated color temperature, and the calculated gain is multiplied by the R and B data to adjust the color tone to be natural (for example, , Patent Document 2 and Patent Document 3).

As described above, the AWB function is a closed loop control for controlling the monitor image so that the monitor image becomes a natural color while monitoring the image captured by the camera (the evaluation value while changing the gains of R, G, and B). The gain value is converged so that is within a preset numerical value range).
Note that Patent Document 4 refers to the AWB function and the way people see. Specifically, it is said that the operation of AWB imitates the function (visual characteristics) of the human brain. It can be inferred that a human being estimates the color temperature of the light source in the entire image, particularly focusing on the high luminance region, and corrects the entire image based on the color temperature. It should be noted that a low-D range camera saturates a high-brightness area, so that the area becomes white or color information is lost, so the ability of AWB based on the visual characteristics of humans cannot be used. It is. In other words, the image captured by the camera is correct as a physical quantity, but when a person views the image from the camera, the color temperature of the light source cannot be estimated, so it can be interpreted that the color tone feels unnatural.

  Further, there is an HDR camera as a camera that can obtain a captured image with a wide D range. In order to compensate for the narrowness of the D range of the image sensor, for example, a plurality of images with different exposure times are captured and combined to generate an HDR image with an expanded dynamic range (see Patent Document 5). . Since the HDR camera has no saturated pixels, it can capture an optical physical quantity with some accuracy and output an image. If the output image is included in the output image with a wide-angle HDR camera, the AWB ability based on human visual characteristics can be functioned normally from the HDR image, and the color tone does not feel unnatural. Have. Even if there is no light source information, there are memory colors such as sky, clouds, green trees, etc. in the captured image of the wide-angle HDR camera, and if they are reproduced, correction based on visual characteristics will function.

  Temporarily, the exposure amount at the time of exposure for the longest time is set as the long-time exposure amount S3, the exposure amount at the time of exposure at the shortest exposure time is set as the short-time exposure amount S1, and Let the exposure amount be a medium time exposure amount S2. The HDR camera can increase its dynamic range as the ratio of S3 and S1 increases. Further, it is known that the image quality (S / N) of an output image from the HDR camera is improved as the number of images taken with an intermediate exposure time is increased.

JP 2003-244538 A JP-A-5-122719 Japanese Patent Laid-Open No. 7-143497 JP 2006-31830 A JP 2009-49547 A

As described above, the image from the wide-angle HDR camera can be recognized as a natural color image output because the visual characteristics (AWB) of the human eye function normally. However, for example, when an image taken with a light source with an extreme color temperature is viewed by a person other than the photographer, that is, when there is no memory color in the captured image, the visual environment does not work well and the shooting environment It is conceivable that the color temperature cannot be estimated. For example, a sodium lamp in a tunnel has an orange color, and a mercury lamp has a pale color.
Therefore, the present invention has been made paying attention to such unsolved problems of the conventional technology. According to some aspects of the present invention, an image processing apparatus, an image processing method, and an image processing apparatus suitable for adjusting the white balance of composite image data obtained by linearly combining two or more image data obtained by imaging with different exposure times. An imaging device can be provided.

[Mode 1] In order to achieve the above object, an image processing apparatus according to mode 1 includes:
Image data for synthesis that acquires two or more image data from a plurality of image data obtained by imaging a subject with a plurality of different exposure times in each frame period from the imaging device capable of imaging the subject. From the imaging device, an acquisition unit, a composite image data generation unit that generates a composite image data by linearly synthesizing the two or more image data acquired by the synthesis image data acquisition unit based on a ratio of exposure time, Color measurement image data acquisition means for acquiring short exposure image data which is image data corresponding to the shortest exposure time among the plurality of image data;
Color information extraction means for extracting color information from the short exposure image data acquired by the color measurement image data acquisition means, and information relating to the light source at the time of imaging the subject based on the color information extracted by the color information extraction means A light source information estimating means for estimating the gain, a gain setting means for setting a gain used for white balance adjustment based on an estimation result of the light source information estimating means, and a gain set by the gain setting means. White balance adjustment means for adjusting white balance of synthesized image data generated by synthesizing the two or more image data in the same frame period as the short-exposure image data related to the setting ,
The image sensor outputs a pixel signal corresponding to the accumulated charge while maintaining the accumulated charge from each pixel.
It is possible to read out pixel signals by non-destructive readout method, and in each frame period
If each of the pixels is exposed in the plurality of different exposure times in order from the shortest exposure time,
In addition, the pixel signal is read out from each exposed pixel by the nondestructive readout method, and the readout order is
The pixel signal data constituting the image data corresponding to each exposure time is output.
The image data for synthesizing means obtains two or more pieces of image data including image data corresponding to the longest exposure time in each frame period from the image sensor, and the synthetic image data generating means Composite image data is generated based on two or more pieces of image data including image data corresponding to the exposure time of the image, and the color measurement image data acquisition unit is configured to output the image data for synthesis from the image sensor. The short-exposure image data is acquired prior to acquiring image data corresponding to the longest exposure time in a period, and the color information extraction unit obtains the color information from the short-exposure image data acquired in advance. The light source information estimating means estimates information relating to the light source based on the color information extracted from the previously obtained short exposure image data, and A white balance adjustment means for setting a gain used for white balance adjustment based on information relating to a light source estimated based on color information extracted from the short-exposure image data acquired in advance; The composite image generated by the composite image data generation means based on the two or more image data in the same frame period as the short exposure image data related to the setting of the gain using the gain set by the gain setting means Adjust the white balance of the data.

  With such a configuration, two or more pieces of image data out of a plurality of image data obtained by imaging a subject with a plurality of different exposure times in each frame period from the image sensor by the image data acquisition unit for synthesis are obtained. Can be acquired. Furthermore, the composite image data generation means can generate composite image data by linearly combining the acquired two or more pieces of image data based on the ratio of exposure time (exposure amount).

Further, the color measurement image data acquisition means can acquire short exposure image data that is image data corresponding to the shortest exposure time among the plurality of image data from the image sensor, and the color information extraction means Color information can be extracted from the acquired short exposure image data.
Further, the light source information estimation means can estimate light source information based on the extracted color information, and the gain setting means can set a gain used for white balance adjustment based on the estimated light source information.

Furthermore, the white balance adjustment means can adjust the white balance of the composite image data using the set gain.
Thereby, the gain for adjusting the white balance can be set based on the short exposure image data acquired in the same frame as the acquisition frame of the two or more image data that generated the composite image data. Then, using this set gain, the white balance of the composite image data generated from the two or more image data of the same frame can be adjusted.

Conventional white balance adjustment uses the set gain for composite image data composed of image data acquired in a frame different from the frame of image data used for gain setting (determination). The white balance is adjusted. That is, the feedback type control is performed. On the other hand, in the configuration of this embodiment, white balance can be adjusted by feedforward control.
Therefore, it is possible to always obtain a stable output image even when, for example, the light source is blinking and it is difficult to predict the behavior of the light source as compared with the case of another frame.

Further, since information related to the light source can be predicted based on the color information extracted from the short-exposure image data, when the light source is included in the imaging range, the high-luminance input light called the light source is not saturated. In contrast, a signal with a good level can be obtained. In other words, a signal having a level larger than “0” can be obtained only for input light having the same luminance level as the light source.
Here, the “color information” corresponds to, for example, the number of pixels of each color, an integrated value of pixel values of each color, and the like.

The “gain used for white balance adjustment” is a gain of an amplifier circuit, a multiplier, etc. for adjusting white balance, and is multiplied by the pixel signal of each color of the composite image data. . However, in the pixel signals of each color of R (red), G (green), and B (blue), for example, when white balance is adjusted based on G, the gain is multiplied by the R and B pixel signals. .
The “light source information” corresponds to information such as the type of light source and the color temperature of the light source.

Furthermore, with such a configuration, it is possible to easily acquire image data captured with different exposure times in each frame period from the image sensor.

In addition, with such a configuration, it is possible to set a gain used for white balance adjustment using short exposure image data output prior to other image data from the image sensor. Further, the set gain is used for white balance adjustment of composite image data obtained by linearly synthesizing two or more image data output in the same frame as the short exposure image data and later than the short exposure image data. Can do.
Thus, for example, in an application that requires real-time performance, such as an in-vehicle camera or machine vision, it is effective to acquire a composite image (video) with white balance adjusted at a timing close to real time. .

[Embodiment 2 ] Furthermore, the image processing apparatus of Embodiment 2 is the image processing apparatus of Embodiment 1 ,
The light source information estimation means estimates the type of light source as information relating to the light source.
With such a configuration, a gain used for white balance adjustment can be set based on the estimated type of light source.
Thereby, since the type of the light source is known, it is possible to adjust the white balance appropriately according to the type of the light source.

[Mode 3 ] Furthermore, the image processing apparatus of mode 3 is the same as the image processing apparatus of mode 2 ,
The light source information estimation unit is extracted by the color information extraction unit based on a database in which information in which each index value indicated by the color information is associated with the type of light source corresponding to each index value is registered. The type of light source corresponding to color information is estimated, and the gain setting means is configured to estimate the light source information based on a database in which information in which the type of light source is associated with the gain corresponding to each type of light source is registered. A gain corresponding to the type of light source estimated by the means is set.

With such a configuration, when the type of light source is estimated, a gain corresponding to the estimated type of light source is obtained from a database in which gains are registered in association with the type of light source, The acquired gain can be set as a gain used for white balance adjustment.
As a result, it is not necessary to perform calculation processing or the like in setting the gain, and therefore, it can be easily incorporated into a system that requires speed such as real-time processing.

[Form 4 ] Furthermore, the image processing apparatus of form 4 is the image processing apparatus of form 2 or 3 ,
When the light source information estimation unit cannot estimate the type of light source, the white balance is not adjusted.
With such a configuration, for example, when there are a plurality of types of light sources in the imaging region, inappropriate white balance adjustment can be prevented.

[Embodiment 5 ] Furthermore, the image processing device of embodiment 7 is the image processing device of embodiment 1 ,
The light source information estimation means estimates a color temperature of the light source as information relating to the light source.
With such a configuration, since the color temperature of the light source is known, appropriate white balance adjustment according to the color temperature can be performed.

[Mode 6 ] Further, the image processing apparatus of mode 6 is the same as the image processing apparatus of mode 5 ,
The light source information estimation means is extracted by the color information extraction means based on a database in which information in which each index value indicated by the color information is associated with the color temperature of the light source corresponding to each index value is registered. The color setting of the light source corresponding to the color information is estimated, and the gain setting means is estimated by the light source information estimation means based on a database in which information in which the color temperature of the light source is associated with the gain is registered. Set the gain corresponding to the color temperature of the light source.

With such a configuration, when the color temperature of the light source is estimated, a gain corresponding to the estimated color temperature of the light source is acquired from a database in which gains are registered in association with the color temperature of the light source. Thus, the acquired gain can be set as a gain used for white balance adjustment.
As a result, it is not necessary to perform calculation processing or the like in setting the gain, and therefore, it can be easily incorporated into a system that requires speed such as real-time processing.

[Mode 7 ] Furthermore, the image processing apparatus according to mode 7 is the image processing apparatus according to any one of modes 1 to 6 ,
The exposure time of the short-exposure image data is set to the shortest time during which pixel signal data for the light source included in the subject or the specular reflection light of the light source can be output.
With such a configuration, it is possible to cause the image sensor to output a signal having a level higher than “0” only for input light having a luminance equal to or higher than the luminance of the light source or the specular reflection light of the light source.
Thereby, the estimation precision of light source information can be improved.

[Embodiment 8 ] Furthermore, the image processing apparatus of Embodiment 8 is the image processing apparatus of any one of Embodiments 1 to 7 ,
Color information storage means for storing the color information extracted by the color information extraction means is provided.
With such a configuration, the color information storage means can store the color information extracted by the color information extraction means, so that processing using the stored color information can be performed in an external PC or the like. it can.

[Embodiment 9 ] Furthermore, the image processing apparatus of Embodiment 9 is the image processing apparatus of any one of Embodiments 1 to 8 ,
Image conversion means for converting the brightness of the composite image data after the white balance is adjusted by the white balance adjustment means to generate composite image data having different gradation characteristics.
With such a configuration, the composite image data after adjusting the white balance is converted into an image having a desired gradation characteristic, for example, by compressing the luminance range according to the gradation range that can be displayed by the display device. can do.

[Mode 10 ] On the other hand, in order to achieve the above object, an image processing method according to mode 10 includes:
Image data for synthesis that acquires two or more image data from a plurality of image data obtained by imaging a subject with a plurality of different exposure times in each frame period from the imaging device capable of imaging the subject. From the acquisition step, a combined image data generation step of generating a combined image data by linearly combining the two or more image data acquired in the combining image data acquisition step based on an exposure time ratio, Color measurement image data acquisition step for acquiring short exposure image data that is image data corresponding to the shortest exposure time among a plurality of image data, and color from the short exposure image data acquired in the color measurement image data acquisition step A color information extraction step for extracting information, and based on the color information extracted in the color information extraction step; A light source information estimation step for estimating information related to a light source at the time of imaging a subject, a gain setting step for setting a gain used for white balance adjustment based on an estimation result in the light source information estimation step, and a setting in the gain setting step A white balance adjustment step for adjusting white balance of synthesized image data generated by synthesizing the two or more image data in the same frame period as the short-exposure image data related to the setting of the gain using the gain , only including,
In the color measurement image data acquisition step, the image sensor can read out pixel signals by a non-destructive readout method of reading out pixel signals corresponding to the accumulated charges while maintaining accumulated charges from the pixels. In the frame period, each pixel is exposed at the plurality of different exposure times in order from the shortest exposure time, and the pixel signal is read from each exposed pixel by the nondestructive readout method, and the exposure time corresponds to each exposure time. The pixel signal data constituting the image data to be output is output, and in the synthesis image data acquisition step, two or more image data including the image data corresponding to the longest exposure time in each frame period are received from the image sensor. In the composite image data generation step, image data corresponding to the longest exposure time is obtained. It generates a composite image data on the basis of free 2 or more image data,
In the color measurement image data acquisition step, the short-exposure image data is acquired from the image sensor prior to the synthesis image data acquisition step acquiring image data corresponding to the longest exposure time in each frame period. In the color information extraction step, the color information is extracted from the previously acquired short exposure image data, and in the light source information estimation step, the color information extraction step is extracted from the previously acquired short exposure image data. Information on a light source is estimated based on color information, and in the gain setting step, white balance adjustment is performed based on information on the light source estimated based on color information extracted from the short-exposure image data acquired in advance. In the white balance adjustment step, the gain set by the gain setting means is used. Adjusting the white balance of the combined image data generated in the synthetic image data generating unit on the basis of the two or more image data of the same frame period as the short-exposure image data according to the setting of the gain.
Thereby, the same operation and effect as those of the image processing apparatus of aspect 1 can be obtained.

[Mode 11 ] In order to achieve the above object, an imaging apparatus according to mode 11
An image sensor capable of imaging a subject, and the image processing device according to any one of Embodiments 1 to 9 .
With such a configuration, operations and effects equivalent to those of the image processing apparatus according to any one of Embodiments 1 to 11 can be obtained.

1 is a block diagram illustrating a configuration of an imaging apparatus 1 according to the present invention. 2 is a block diagram showing an internal configuration of an HDR image sensor 10. FIG. It is a figure which shows the structure of the color filter array 10c. It is a figure which shows an example of the exposure for every line of each pixel in the sensor cell array of HDR sensor 10d, and the reading operation of a pixel signal. It is a figure which shows an example of the reset timing of stored charge, and the read-out timing of the pixel signal of each exposure time. It is a figure which shows an example of the output format of the pixel data from HDR sensor 10d. 2 is a block diagram illustrating an example of an internal configuration of an image processing device 20. FIG. 3 is a block diagram illustrating an example of an internal configuration of an HDR main line processing unit 26. FIG. 3 is a block diagram illustrating an example of an internal configuration of a color photometry unit 25, an HDR linear processing unit 30, and a WB adjustment unit 31. FIG. It is a block diagram which shows an example of an internal structure of the synthetic | combination process part 30a. 3 is a block diagram illustrating an example of an internal configuration of a color information extraction unit 40. FIG. It is a block diagram which shows an example of an internal structure of the light source information estimation part 41. FIG. 3 is a block diagram illustrating an example of an internal configuration of a WB gain setting unit 42. FIG. It is a figure which shows an example of the output timing of the pixel data in each sub-frame. It is a block diagram which shows the example of 1 structure of the imaging device which has the conventional AWB function.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. 1 to 14 are diagrams showing an embodiment of an image processing apparatus, an image processing method, and an imaging apparatus according to the present invention.
(1) Configuration Example of Imaging Device First, the configuration of the imaging device according to the present invention will be described with reference to FIGS. FIG. 1 is a block diagram showing a configuration of an imaging apparatus 1 according to the present invention. FIG. 2 is a block diagram showing an internal configuration of the HDR image sensor 10.

As illustrated in FIG. 1, the imaging device 1 includes an HDR imaging device 10 and an image processing device 20.
As shown in FIG. 2, the imaging element 10 includes an imaging lens 10a, a microlens 10b, a color filter array 10c, an HDR sensor 10d, a drive circuit 10e, and a readout circuit 10f.
The imaging lens 10a collects reflected light from the subject and guides it to the microlens 10b. There are types such as a single focus lens, a zoom lens, and an auto iris lens depending on the imaging conditions.

The microlens 10b collects light transmitted through the imaging lens 10a on each sensor cell (pixel) of the sensor cell array included in the HDR sensor 10d.
The color filter array 10c separates light having a wavelength corresponding to one predetermined type of color element from light transmitted through the microlens 10b and makes the separated light incident on each corresponding pixel (hereinafter referred to as CF). Part) and at least the number of pixels.

The HDR sensor 10d has a sensor cell array in which each sensor cell (pixel) includes a photodiode and a CMOS element, and outputs a plurality of types of image data having different exposure times by controlling the exposure time by an electronic shutter method. It is.
The drive circuit 10e drives the HDR sensor 10d and the readout circuit 10f based on a drive mode set by a register (not shown) and a synchronization signal from the image processing device 20. In the present embodiment, driving is performed so that a subject is imaged at different exposure times, and a plurality of types of captured image data having different exposure times are output.

The reading circuit 10f reads a captured image signal from the HDR sensor 10d, performs various signal processing including A / D conversion on the read captured image signal (analog), and outputs the signal to the image processing device 20 as digital captured image data. To do. At this time, the readout circuit 10f outputs captured image data in synchronization with a synchronization signal from the image processing device 20 supplied via the drive circuit 10e.
The image processing device 20 performs noise removal processing, composition processing, color processing, luminance image data generation processing, luminance value → gain conversion processing based on a plurality of types of captured image data output from the HDR imaging element 10 and having different exposure times. , Level conversion (tone mapping) processing, γ conversion processing, and the like. A detailed configuration will be described later.

(2) Configuration Example of Color Filter Array Next, the configuration of the color filter array 10c will be described based on FIG.
Here, FIG. 3 is a diagram showing a configuration of the color filter array 10c.
In FIG. 3, R, G, and B correspond to CF portions that selectively transmit light in the wavelength region of any one of the three primary colors (red, green, and blue), and are attached to these. The attached numbers indicate row numbers and column numbers. For example, if R _00, a filter portion that transmits light in the wavelength region of R corresponding to the first column of the pixel line number 1. Note that in the present embodiment, the horizontal direction is the row direction, and the vertical direction is the column direction, and the imaging apparatus 2 performs processing in units of lines composed of a plurality of pixels arranged in the horizontal direction. .

  Specifically, the color filter array 10c (hereinafter referred to as the CF array 10c) converts the light incident through the microlens 10b into light in a wavelength region corresponding to red (hereinafter referred to as R light), green. Separating light in a wavelength region corresponding to a predetermined color from light in a corresponding wavelength region (hereinafter referred to as G light) and light in a wavelength region corresponding to blue (hereinafter referred to as B light); It is composed of a plurality of CF sections that respectively input separated lights to corresponding pixels.

  More specifically, as shown in FIG. 3, the CF array 10 c separates R light from incident light, and a plurality of R light transmission filter units (R in FIG. 3) that enters the separated R light into pixels. ) And a plurality of G light transmission filter portions (G in FIG. 3) that separate the G light from the incident light and enter the separated G light into the pixel R → G → R → G → R → G → A plurality of filter lines RGFL having a configuration that is continuous in the horizontal direction in this order are provided.

  Further, a plurality of G light transmission filter sections and a plurality of B light transmission filter sections (B in FIG. 3) for separating the B light from the incident light and entering the separated B light into the pixels are G → B. A plurality of filter lines GBFL having a configuration that is continuous in the horizontal direction in the order of G → B → G → B →. The plurality of filter lines RGFL and filter lines GBFL are continuously arranged in the vertical direction in the order of RGFL → GBFL → RGFL → GBFL → RGFL →... That is, a plurality of G light transmission filter sections, a plurality of B light transmission filter sections, and a plurality of R light transmission filter sections are arranged in a Bayer shape.

(3) About the drive control method of the HDR sensor 10d Next, based on FIGS. 4-6, the control method of the exposure time of the HDR sensor 10d of the image pick-up element 10 and the reading method of the pixel signal from a sensor cell array are demonstrated. Here, FIG. 4 is a diagram illustrating an example of exposure and pixel signal readout operations for each pixel line in the sensor cell array of the HDR sensor 10d. FIG. 5 is a diagram showing an example of the reset timing of the accumulated charge and the readout timing of the pixel signal for each exposure time. FIG. 6 is a diagram illustrating an example of an output format of pixel data from the HDR sensor 10d.

Here, the exposure time of the present invention is controlled by the non-destructive readout line L1 for performing non-destructive readout of the pixel signal of the ultrashort exposure time T_S1 and the short exposure time T_S2 with respect to the exposure region (scanning region) of the sensor cell array. A non-destructive readout line L2 for performing non-destructive readout of the pixel signal is set. Further, a read & reset line L3 for resetting the accumulated charge of each pixel line and reading the pixel signal of the standard exposure time T_S3 is set.
The relationship between T_S1 to T_S3 is “T_S1 <T_S2 <T_S3” as shown in FIG.
Here, the ultrashort exposure time T_S1 is preferably 1/100 to 1/1000 of the standard exposure time T_S3, and the short exposure time T_S2 is 1/10 to 1/100 of the standard exposure time T_S3. Is desirable.

The standard exposure time T_S3 is set so that an effective output can be obtained from the HDR sensor 10d at a reflectance of 0 to 100%. On the other hand, the ultrashort exposure time T_S1 is set so that the reflectance is 10,000% to 100,000% (the signal output is not saturated even in specular reflection).
In particular, it is desirable to set the ultrashort exposure time T_S1 to a time for outputting a pixel signal that is “0” or an extremely low signal level with respect to incident light other than the light source and the specularly reflected light of the light source. In other words, it is desirable to set the ultrashort exposure time T_S1 to a time for outputting a signal having a level larger than “0” only for the light input in the luminance range of the light source. The detailed reason will be described later.

When the accumulated charge is reset by setting the read & reset line L3, first, the nondestructive read line L1 is set when T_S1 has elapsed. Next, the non-destructive read line L2 is set when T_S2 elapses, and the read & reset line L3 is set when T_S3 elapses.
Specifically, the non-destructive read lines L1 and L2 and the read & reset line L3 are read and reset when charges corresponding to the standard exposure time T_S3 are sequentially accumulated in the pixel lines in the exposure region, as shown in FIG. The line L3 is set so as to sequentially read out pixel signals of the lines of the respective pixels and to sequentially reset the accumulated charges. On the other hand, in the line of each pixel after resetting the exposure area, the pixel signal of the line of each pixel is obtained during the ultrashort exposure time T_S1 and the short exposure time T_S2 during the period in which the charge for the standard exposure time T_S3 is accumulated. Non-destructive read lines L1 and L2 are set to sequentially read non-destructively.

  In the present embodiment, as shown in FIG. 4, the pixel signal (analog data) S1 corresponding to the ultrashort exposure time T_S1 immediately after the reset is read out to the first line memory and at the short exposure time T_S2. The corresponding pixel signal (analog data) S2 is read out to the second line memory. Further, the pixel signal (analog data) S3 corresponding to the standard exposure time T_S3 is read to the third line memory. Then, as shown in FIG. 3, the read pixel signals S1 to S3 are sequentially output to the ADC in the order of S1 to S3 through the selection circuits, and are converted into digital data (pixel data) there. The converted pixel data is output to the image processing device 20 in line units in the order of conversion (in order of S1 to S3).

  In addition, as shown in FIG. 4, the readout timing of the pixel signals on the non-destructive readout lines L1 and L2 and readout & reset line L3 is sequentially scanned for each pixel line (see FIG. 4). Scan direction in FIG. 4). In the read & reset line L3, the accumulated charge is reset, and the pixel signal of the pixel that has been exposed for the standard exposure time T_S3 immediately before the accumulated charge is reset is read. Further, after reset, the pixel signal of the ultrashort exposure time T_S1 is read.

  For example, it is assumed that the pixel signal S3 of the standard exposure time T_S3 is read and reset on the first line that is the first line of the exposure region. Thereafter, every time the pixel signal S3 is read from the third line memory to the outside, scanning of the read & reset line L3 is sequentially performed one line at a time in the scan direction in FIG. At this time, when the read & reset line L3 reaches the first line again, scanning is performed so that the standard exposure time T_S3 has just passed. In such a procedure, the pixel signal is read out and the stored charge is reset at the time of standard exposure for each pixel line for the pixel line in the exposure region of the sensor cell array.

  On the other hand, when the accumulated charge is reset, non-destructive readout of the pixel signal S1 of the pixel that has been exposed for the ultrashort exposure time T_S1 in the non-destructive readout line L1 is performed on the reset pixel line. Subsequently, non-destructive readout of the pixel signal S2 of the pixel that has been exposed for the short exposure time T_S2 in the non-destructive readout line L2 is performed. According to such a procedure, the pixel signals S1 and S2 at the time of exposure are sequentially read out for each line of each pixel of the sensor cell array with the ultrashort exposure time T_S1 and the short exposure time T_S2.

  The pixel signals S1 to S3 read out in this way are stored in the first line memory to the third line memory for each line and output to the selection circuit in line units. From the selection circuit, analog pixel data S1 to S3 are output to the ADC in the order of S1 to S3. The ADC converts the analog pixel data S1 to S3 into digital pixel data S1 to S3. As shown in FIG. 6, the ADC sequentially outputs pixel data to the image processing apparatus 20 in the order of S1 to S3 in units of lines in synchronization with a horizontal synchronization (HSYNC) signal. That is, as shown in FIG. 6, the pixel data S <b> 1 to S <b> 2 are input to the image processing device 20 in the order of S <b> 1 → S <b> 2 prior to the pixel data S <b> 3.

(4) Configuration Example of Image Processing Device Next, a detailed configuration of the image processing device 20 will be described based on FIGS. 7 to 13.
Here, FIG. 7 is a block diagram illustrating an example of an internal configuration of the image processing apparatus 20. FIG. 8 is a block diagram illustrating an example of an internal configuration of the HDR main line processing unit 26. FIG. 9 is a block diagram illustrating an example of the internal configuration of the color photometry unit 25, the HDR linear processing unit 30, and the WB adjustment unit 31. FIG. 10 is a block diagram illustrating an example of an internal configuration of the composition processing unit 30a. FIG. 11 is a block diagram illustrating an example of an internal configuration of the color information extraction unit 40. FIG. 12 is a block diagram illustrating an example of an internal configuration of the light source information estimation unit 41. FIG. 13 is a block diagram illustrating an example of an internal configuration of the WB gain setting unit 42.

As shown in FIG. 7, the image processing apparatus 20 includes a preprocessing unit 21, delay units 22 to 24, a color photometric unit 25, an HDR main line processing unit 26, and a color photometric result storage unit 27. Is done.
Note that the image processing apparatus 20 is configured as a DSP (Digital Signal Processor) for image processing, and although not shown in the drawings, a processor that controls the operation of each component described above and the operation necessary for each component described above are required. A memory for storing data such as programs and control parameters, and a timing generator for controlling the operation timing of each of the above components are provided.

The pre-processing unit 21 generates fixed pattern noise using the pixel data S0 of the ultra-short exposure time T0 (T0 ≦ T1) immediately after the reset for the pixel signal (pixel data S1 to S3) from the HDR sensor 10d. Perform removal processing, clamping processing, and so on.
Specifically, the fixed pattern noise removal process is a process of subtracting the pixel data S0 corresponding to the pixel signal immediately after the reset for each corresponding pixel from the pixel data S1 to S3 of each exposure time. In other words, the pixel data S0 has a small amount of charge accumulation and is dominated by the fixed pattern noise component. Therefore, by subtracting this data from the other exposure time data, only the fixed pattern noise component is removed. can do. Note that the pixel data S0 immediately after the reset may be the same as the pixel data S1 of the ultrashort exposure time. In this case, the subtraction process is performed only on the pixel data S2 to S3.

The clamp process receives the pixel data S1 to S3 from the HDR sensor 10d, detects whether the signal is a signal in the light shielding area, and if the signal is detected as the light shielding area, the signal level is the black (reference) level. Thus, the DC component of all input pixel data is clamped.
The pixel data S <b> 1 to S <b> 3 that have undergone the fixed pattern noise removal process and the clamp process are output to the HDR main line processing unit 26 in synchronism with the output timing via the delay units 22 to 24. Hereinafter, the captured image data (pixel data for one frame) respectively corresponding to the exposure times T_S1 to T_S3 are referred to as captured image data S1 to S3, similarly to the pixel data S1 to S3.

The delay units 22 to 24 have delay elements that delay the pixel data S1 to S3 output in order according to the length of the exposure time by the delay amounts d_S1 to d_S3, respectively. And it has the function to synchronize the output timing of pixel data S1-S3.
Specifically, the delay element of the delay unit 22 delays the pixel data S3 by the delay amount d_S3, the delay element of the delay unit 23 delays the pixel data S2 by the delay amount d_S2, and the delay element of the delay unit 24 is the pixel data S1. Is delayed by a delay amount d_S3 (d_S3 <d_S2 <d_S1).

The output delay times of the pixel data S2 and S3 with respect to the pixel data S1 are simply dT2 (T_S2-T_S1) and dT3 (T_S3-T_S1), and delay amounts for correcting other processing delays are Δt1, Δt2, and Δt3. In this case, the relationship “(d_S3 = Δt3 + dT3) = (d_S2 = Δt2 + dT2) = (d_S1 = Δt1)” is established.
The color photometry unit 25 performs color information extraction processing, light source information estimation processing, and WB gain setting processing based on the captured image data S1 output from the preprocessing unit 21.

The HDR main line processing unit 26, based on the captured image data S1 to S3 input in synchronization via the delay units 22 to 24 and the WB gain input from the color photometry unit 25, Performs white balance adjustment processing, level correction processing, and gamma conversion processing. Then, the image data signal after the gamma conversion processing is output as a video signal. The video signal is an image signal compressed in an 8-bit range, for example, according to an industry standard.
The color photometry result storage unit 27 includes a memory that stores color information extracted from the pixel data S1 of each frame in the color photometry unit 25, and a memory interface (IF). The color photometry unit 25 passes through the memory interface. The input color information is stored in memory.

(4-1) Configuration Example of Color Photometry Unit 25 First, a detailed configuration of the color photometry unit 25 will be described with reference to FIGS. 9 to 12.
As shown in FIG. 9, the color photometry unit 25 extracts a color information from the pixel data S1 for one frame, and a light source information estimation unit 41 that estimates light source information based on the extracted color information. And a WB gain setting unit 42 that sets a WB gain based on the estimated light source information.

(4-1-1) Configuration Example of Color Information Extracting Unit 40 As shown in FIG. 10, the color information extracting unit 40 includes a color separator 60, adders 61a to 61c, an R output register 62a, and a G An output register 62b and a B output register 62c are included.
The color separator 60 adds pixel data S1 input in the reading order based on the reading order corresponding to the arrangement rule (Bayer arrangement) of each color of the color filter array 10c shown in FIG. Output to the instrument.

  Specifically, when the input pixel data S1 is pixel data S1 of a pixel that has received R light transmitted through the R light transmission filter unit, the pixel signal (hereinafter referred to as R signal) is sent to the adder 61a. Output. Further, when the pixel data S1 is a pixel data S1 of the pixel that has received the G light transmitted through the G light transmission filter unit, the pixel signal (hereinafter referred to as a G signal) is output to the adder 61b. Further, when the pixel data S1 is the pixel data S1 of the pixel that has received the B light transmitted through the B light transmission filter unit, the pixel signal (hereinafter referred to as the B signal) is output to the adder 61c.

The adder 61a adds the pixel value r indicated by the R signal input from the color separator 60 and the previous integrated value of r (initial value is 0) input from the R output register 62a. Then, an RA signal indicating the current accumulated value of r (hereinafter referred to as r accumulated value), which is the addition result, is output to the R output register 62a.
The adder 61b adds the pixel value g indicated by the G signal input from the color separator 60 and the previous integrated value of g (initial value is 0) input from the G output register 62b. The GA signal indicating the current integrated value of g (hereinafter referred to as “g integrated value”), which is the addition result, is output to the G output register 62b.

The adder 61c adds the pixel value b indicated by the B signal input from the color separator 60 and the previous b integrated value (initial value is 0) input from the B output register 62c. The BA signal indicating the current integrated value of b (hereinafter referred to as “g integrated value”), which is the result of the addition, is output to the B output register 62c.
The R output register 62a stores the r integrated value indicated by the RA signal input from the adder 61a, and indicates the stored r integrated value in synchronization with the input timing of the R signal to the adder 61a. The signal is output to the adder 61a.

Further, the R output register 62a resets the stored r accumulated value (sets to an initial value) in response to an input of a frame reset signal from a reset circuit (not shown).
That is, the adder 61a and the R output register 62a add the pixel value r of the latest R signal sequentially input and the r integrated value which is the previous addition result, and the addition result is used for the R output. The process of storing (overwriting) in the register 62a is repeated.
The G output register 62b stores the g integrated value indicated by the GA signal input from the adder 61b, and indicates the stored g integrated value in synchronization with the input timing of the G signal to the adder 61b. The signal is output to the adder 61b.

The G output register 62b resets the stored g integrated value (sets to an initial value) in response to an input of a frame reset signal from a reset circuit (not shown).
That is, the adder 61b and the G output register 62b add the pixel value g of the latest G signal sequentially input and the g accumulated value which is the previous addition result, and the addition result is used for the G output. The process of storing (overwriting) in the register 62b is repeated.
The B output register 62c stores the b integrated value indicated by the BA signal input from the adder 61c, and indicates the stored b integrated value in synchronization with the input timing of the B signal to the adder 61c. The signal is output to the adder 61c.

Further, the B output register 62c resets the stored b accumulated value (sets to an initial value) in response to an input of a frame reset signal from a reset circuit (not shown).
That is, the adder 61c and the B output register 62c add the pixel value b of the latest B signal sequentially input and the b integrated value which is the previous addition result, and the addition result is used for the B output. The process of storing (overwriting) in the register 62c is repeated.

Further, in this embodiment, when the r, g, b integrated values for the pixel data S1 for one frame are stored in the corresponding registers among the R, G, B output registers 62a, 62b, 62c, the stored data is stored. The obtained r, g, b integrated values are acquired by the light source information estimation unit 41 and the color photometry result storage unit 27. Specifically, the integrated values of r, g, and b stored in each register are output to the light source information estimation unit 41 and the color photometry result storage unit 27 in accordance with the acquired signal from the light source information estimation unit 41. Thereafter, before the pixel data S1 of the next frame is input to the adders 61a to 61c, the stored values of the R, G, and B output registers 62a, 62b, and 62c are reset by the frame reset signal.
That is, the r integrated value, the g integrated value, and the b integrated value for the pixel data S1 for one frame are output to the light source information estimation unit 41 and the color photometric result storage unit 27 as color information.
Thus, when the color photometric result storage unit 27 receives the integrated value of r, g, b from the color information extraction unit 40 via the memory IF, the received color, r, g, b integrated value is stored in the memory.

  Here, one of the reasons for performing color photometry (calculation processing of the above-mentioned r, g, b integrated value) using the pixel data S1 for one frame is that the pixel data S1 having an extremely short exposure time has a saturation level pixel. There is no data. Another reason is that the exposure time is so short that the light source itself reaches the imaging lens 10a to form an image, or the light source is specularly reflected and reaches the imaging lens 10a to form an image. A signal having a level higher than “0” is output from the HDR sensor 10d for incident light with a high luminance. On the other hand, when the reflected light from the subject that is absorbed and diffusely reflected reaches the imaging lens and forms an image, the signal level from the HDR sensor 10d is lower than the light source or the specular reflected light of the light source. Is almost “0”. For example, when a Macbeth chart is photographed, a captured image of the ultrashort exposure time T_S1 is not formed (the signal level generated by irregularly reflected light is almost 0). In other words, the ultrashort exposure time T_S1 is set to be a short time in which an image is not formed with reflected light that is irregularly reflected. Therefore, if the subject is covered with a red cloth, it is ignored in color photometry. By doing in this way, the pixel value of each color signal with respect to only the light of the light source or the specular reflection light of the light source can be integrated. Thereby, the estimation accuracy of light source information can be improved in the light source information estimation unit 41 in the subsequent stage.

(4-1-2) Configuration Example of Light Source Information Estimation Unit 41 The light source information estimation unit 41 includes a color information → light source information conversion unit 41a and a light source information storage unit 41b as shown in FIG. The
The color information → light source information conversion unit 41 a outputs an acquisition signal to the color information extraction unit 40 at the timing when one frame of color photometry processing ends, and thereby acquires color information from the color information extraction unit 40. Furthermore, light source information corresponding to the acquired color information is estimated based on the acquired color information and the light source information database stored in the light source information storage unit 41b.
The light source information storage unit 41b is a memory in which a light source information database in which information in which color information and light source information are associated is registered is stored.

Specifically, the light source information estimation unit 41 sets the ratio of integrated values “Rrg = ir / ig” based on ig, where “r integrated value is ir, g integrated value is ig, b integrated value is ib”, and “Rbg = Ib / ig ". Note that “Rgg = ig / ig = 1”.
The ratios Rrg and Rbg are calculated from the integrated values of the pixel values corresponding to the colors of the ultra-short exposure image that is largely dominated by the signal component with respect to the incident light such as the light of the light source or the specular reflection light of the light source. It is suitable for judging the color characteristics of the light source.

Examples of the light source include sunlight (morning, noon, evening), an incandescent bulb, a discharge lamp, a light emitting diode, and a laser.
Incandescent lamps include incandescent lamps and halogen lamps. Discharge lamps are broadly classified into arc discharge lamps and glow discharge lamps, and those belonging to arc discharge lamps include mercury lamps, sodium lamps, xenon lamps and metal halide lamps. . In addition, there are fluorescent lamps, high-pressure mercury lamps, ultra-high pressure mercury lamps, and the like that belong to mercury lamps. Moreover, there are a neon tube lamp, a neon lamp, a xenon flash, etc., belonging to the glow discharge lamp.

Here, the light source information is information relating to the light source such as the type of the light source and the color temperature of the light source.
In the present embodiment, the light source information database is a database in which information that associates the ratios Rrg and Rbg of the integrated values with the type of light source or the color temperature of the light source corresponding to the ratio is registered.
Note that the light source information database is based on the measurement results obtained by measuring Rrg and Rbg for various types of light sources or measuring Rrg and Rbg under various color temperature environments in advance through experiments. Are generated.

The color information → light source information conversion unit 41a then stores the information on the type of light source corresponding to the calculated ratios Rrg and Rbg or the information on the color temperature of the light source in the light source information database 41b. To obtain light source information corresponding to the acquired color information. Further, the light source information estimation unit 41 outputs a signal indicating the estimated light source information to the WB gain setting unit 42.
If there is no light source information corresponding to the acquired color information in the light source information database, a signal indicating that is output to the WB gain setting unit 42.

(4-1-3) Configuration Example of WB Gain Setting Unit 42 As shown in FIG. 12, the WB gain setting unit 42 includes a light source information → WB gain conversion unit 42a and a gain information storage unit 42b. The
The light source information → WB gain conversion unit 42a sets the WB gain corresponding to the acquired light source information based on the light source information acquired from the light source information estimation unit 41 and the gain information database stored in the gain information storage unit 42b. .
The light source information storage unit 41b is a memory in which a gain information database in which information in which light source information and WB gain are associated is registered is stored.
Here, the WB gain is a gain for correcting the signal level of the other two colors using one of R, G, and B as a reference. In the present embodiment, G is a reference (gain Gg = 1), and a gain Gr for correcting the level of the R_hdr signal and a gain Gb for correcting the level of the B_hdr signal in the HDR color image data signal described later are WB gains. Become.

In the present embodiment, the gain information database is a database in which information in which the type or color temperature of the light source is associated with the WB gain corresponding to the type or color temperature of the light source is registered.
Note that the gain information database obtains an appropriate WB gain for various types of light sources or obtains an appropriate WB gain for various color temperatures by experiments in advance, and based on the obtained results. Are generated.

  Specifically, white balance is adjusted while varying the WB gain for captured images taken under various types of light sources and under various color temperature environments, and the type and color temperature of each light source are adjusted. On the other hand, a WB gain that is in a normal state where no color cast is generated is obtained. In addition, since the determination of the WB gain (determination of appropriate image quality, etc.) is a subjective determination of the experimenter, for example, the adjusted captured image is shown to a large number of subjects and fine adjustment is performed based on the determination. It is also possible to decide by performing.

Then, the light source information → WB gain conversion unit 42a acquires the WB gain corresponding to the light source information acquired from the light source information estimation unit 41 from the gain information database stored in the gain information storage unit 42b. The WB gain is set by outputting the corresponding WB gain to the WB adjustment unit 31.
When a signal indicating that there is no light source information is acquired from the light source information estimation unit 41, WB gains “Gr = 1” and “Gb = 1” are output to the WB adjustment unit 31.

(4-2) Configuration Example of HDR Main Line Processing Unit 26 Next, a detailed configuration of the HDR main line processing unit 26 will be described based on FIGS. 8, 9, and 13.
As shown in FIG. 8, the HDR main line processing unit 26 includes an HDR linear processing unit 30, a white balance (WB) adjustment unit 31, and a level correction processing unit 32.
The HDR linear processing unit 30 generates HDR color image data based on the pixel data S1 to S3.
The WB adjustment unit 31 adjusts the white balance of the HDR color image data using the WB gain set in the color photometry unit 25.
The level correction processing unit 32 compresses the HDR color image data after adjusting the WB to an industry standard (standard) range (gradation range) and outputs an HDR video signal.

(4-2-1) Configuration Example of HDR Linear Processing Unit 30 As shown in FIG. 9, the HDR linear processing unit 30 includes a synthesis processing unit 30a and a color processing unit 30b.
The combining processing unit 30a combines the signals of the pixel data S1, S2, and S3 corresponding to the three types of exposure times T_S1, T_S2, and T_S3 based on the ratio of the exposure times to generate HDR_RAW pixel data.
Specifically, as illustrated in FIG. 13, the synthesis processing unit 30a includes normalization gain calculation units 70 and 71 that calculate gains for normalization, and normalization that normalizes pixel data using the normalization gains. Parts 72 and 73.

The normalization gain calculation unit 70 calculates a normalization gain RS3 corresponding to the ratio between the exposure times T_S3 and T_S1. For example, when normalization is performed based on T_S1, “RS3 = T_S1 / T_S3” is calculated.
The normalization gain calculation unit 71 calculates a normalization gain RS2 corresponding to the ratio between the exposure times T_S2 and T_S1. For example, when normalization is performed based on T_S1, “RS2 = T_S1 / T_S2” is calculated.

The normalizing unit 72 multiplies the pixel data S3 by the normalization gain RS3, normalizes the pixel data S3, and outputs normalized pixel data NS3.
The normalizing unit 73 multiplies the pixel data S2 by the normalization gain RS2, normalizes the pixel data S2, and outputs normalized pixel data NS2.
The composition processing unit 30a further weights the normalized pixel data NS1, NS2, NS3 using the composition weights W1, W2, W3 and the composition weights W1, W2, W3. Weighting units 75, 76, 77.

The combination weight calculation unit 74 calculates combination weights W1, W2, and W3 based on the pixel data S1, S2, and S3. Specifically, the composite weight is calculated using a weight function F (x) represented by the following formula (1), where the pixel data S1 to S3 are elements x.

F (x) = (x / MAX) ⁿ (1)

In the above equation (1), MAX is the maximum value of the gradation range of the pixel data S1, S2, S3, for example, “MAX = 255” if the gradation range is 8 bits, and the gradation range of 10 bits. For example, “MAX = 1023”.

Using the above equation (1), W1 to W3 can be calculated by, for example, the following equations (2) to (4).

W1 = F (S2) (2)
W2 = F (S3) -W1 = F (S3) -F (S2) (3)
W3 = 1-W2-W1 = 1-F (S3) (4)

The weighting unit 75 multiplies the normalized pixel data NS3 by the composite weight W3 and outputs the weighted pixel data WS3.
The weighting unit 76 multiplies the normalized pixel data NS2 by the composite weight W2 and outputs the weighted pixel data WS2.
The weighting unit 77 multiplies the pixel data S1 by the composite weight W1 and outputs weighted pixel data WS1.

The combination processing unit 30a further includes a combining unit 78 that combines the weighted pixel data WS1, WS2, and WS3 to generate HDR_RAW pixel data and outputs the generated HDR_RAW pixel data to the color processing unit 30b.
For example, the combining unit 78 generates 20-bit HDR_RAW pixel data by combining the weighted pixel data WS1, WS2, and WS3, assuming that the pixel data S1 to S3 are each 10-bit data. Note that HDR_RAW pixel data for one frame becomes HDR image data (HDR_RAW image data).

  The color processing unit 30b has a line memory (not shown), HDR_RAW pixel data (pixel data to be processed), HDR_RAW pixel data of pixels around the pixel to be processed, stored in the line memory, and Is used to perform color interpolation processing. That is, processing (color signal processing) for generating color signals (data) defined in the RGB color space is performed for each point of the image using the HDR_RAW pixel data delayed by the line memory.

The color processing unit 30b also performs sharpness processing that corrects the brightness difference and the hue difference of the contour portion of the captured image to clarify the contour portion of the captured image.
Through the color interpolation process, the HDR_RAW pixel data is converted into HDR color pixel data corresponding to each color element of RGB for each pixel.
Accordingly, pixel data R_hdr corresponding to the R (red) color element, pixel data G_hdr corresponding to the G (green) color element, and pixel data B_hdr corresponding to the B (blue) color element are provided for each pixel. HDR color pixel data is generated. In addition, HDR color image data is comprised from HDR color pixel data R_hdr, G_hdr, and B_hdr corresponding to the pixel data S1 to S3 for one frame.

(4-2-2) Configuration Example of WB Adjustment Unit 31 As shown in FIG. 9, the WB adjustment unit 31 includes a multiplier 31a and a multiplier 31b.
The multiplier 31 a multiplies the HDR color pixel data R_hdr input from the HDR linear processing unit 30 by the gain Gr input from the color photometry unit 25 and outputs the multiplication result to the level correction processing unit 32.
The multiplier 31b multiplies the HDR color pixel data B_hdr input from the HDR linear processing unit 30 by the gain Gb input from the color photometry unit 25, and outputs the multiplication result to the level correction processing unit 32.
The input timing of the HDR color pixel data R_hdr and B_hdr from the HDR linear processing unit 30 and the input timing of the gains Gr and Gb from the color photometry unit are configured to be synchronized by a delay unit (not shown). Yes.

The HDR color pixel data G_hdr from the HDR linear processing unit 30 is output through to the level correction processing unit 32 via the WB adjustment unit 31.
The output timing of the HDR color pixel data R_hdr, G_hdr, B_hdr from the WB adjustment unit 31 to the level correction processing unit 32 is configured to be synchronized by a delay unit (not shown).

(4-2-3) Processing of Level Correction Processing Unit 32 The level correction processing unit 32 multiplies each pixel data of the HDR color image data after adjusting the WB and the luminance level correction gain to perform level conversion (tone mapping). ), And level-converted HDR color image data is quantized to 8 bits (range compression) according to a γ curve (tone curve). Here, the level conversion is an operation for expanding (broadening) the histogram uniformly by expanding the dark area image.
Specifically, the HDR color image data whose WB has been adjusted by the WB adjustment unit 31 is converted into luminance image data Y, and the luminance image data Y is subjected to blurring processing using a low-pass filter (hereinafter referred to as LPF). Do. The purpose of performing this blurring process is to separate the illumination light component from the HDR color image data and obtain a luminance image of the separated illumination light component.

Further, the level correction processing unit 32 includes a first LUT (Look Up Table) in which a luminance level correction gain corresponding to input luminance (illumination light component) is registered, and a gradation range (for example, displayable by an external display device). And a second LUT in which information on the tone curve corresponding to the industry standard gradation range) is registered.
Then, a luminance level correction gain corresponding to the separated illumination light component is acquired from the first LUT, and level conversion is performed by multiplying the corresponding HDR color pixel data by the acquired luminance level correction gain.
Furthermore, the conversion value corresponding to the value of each pixel data of the HDR color image data after level conversion is acquired from the second LUT, and the gradation value (luminance value) indicated by each pixel data is corresponding to the tone curve. Γ conversion is performed by replacing the converted value.

  For example, when the gradation range of HDR color image data is 20 bits and the displayable gradation range of the external display device is 8 bits, the luminance of the 20-bit gradation range stored in the second LUT A conversion value corresponding to each input luminance value is acquired from tone curve information for converting the value into a luminance value in an 8-bit gradation range. At this time, information indicating the displayable gradation range may be acquired from the external display device.

(5) Operation Example of Imaging Device 1 Next, a specific operation of the imaging device 1 of the present embodiment will be described based on FIG.
Here, FIG. 14 is a diagram illustrating an example of output timing of pixel data in each subframe.
When imaging of the subject is started, in addition to the subject within the shooting range, when there is a light source or an object that specularly reflects the light source, light from the light source (including reflected light) and light reflected from the subject are The light is condensed by the imaging lens 10a and is incident on the microlens 10b. Incident light from the imaging lens 10a is collimated by the microlens 10b and is incident on each pixel of the sensor cell array via each CF portion of the CF array 10c. Since the CF array 10c has a configuration in which CF portions corresponding to the three primary colors of RGB are arranged in a Bayer arrangement, only light of color elements corresponding to each CF portion among R light, G light, and B light is applied to each pixel. It will be incident.

  On the other hand, when the imaging is started, the readout and reset line L3 is set one line at a time from the start line, and the pixel signal S3 is read from each pixel of the scanned line. Reset. Note that the pixel signal S3 that is read out first is processed so as to be ignored in each component in the subsequent stage because the pixel signal S3 that has been read out is not exposed with the standard exposure time T_S3. Subsequently, for each scanning line, after resetting each pixel, the nondestructive readout line L1 is set at the elapse timing of the ultrashort exposure time T_S1, and the pixel signal S1 is read out. Subsequently, for each scanning line, after resetting each pixel, the non-destructive readout line L2 is set at the elapse timing of the short exposure time T_S2, and the pixel signal S2 is read out. Then, the read & reset line L3 is set again in order from the start line, the pixel signal S3 is read from each pixel of the scanned line, and then the accumulated charge of each pixel is reset. The pixel signal S3 read at this time and the pixel signals S1 and S2 read before the pixel signal S3 are to be processed in each component in the subsequent stage.

Thereafter, while imaging is being performed, the above-described setting of L1 to L3, readout of the pixel signals S1 to S3, and reset processing are repeatedly performed.
Here, the ultrashort exposure time T_S1 is set such that a signal output larger than “0” is obtained with respect to incident light in the luminance range of the light source or the specular reflection light of the light source. Specifically, it is set to a time for outputting a signal having a level higher than “0” only for an input having a predetermined luminance or more set in consideration of the type and luminance of the target light source.

  The pixel signals S1 to S3 read out in this way are stored in the first to third line memories of the reading circuit 10f for each line and output to the selection circuit in units of lines. From the selection circuit, analog pixel data S1 to S3 are output to the ADC in the order of S1 to S3. The ADC converts the analog pixel data S1 to S3 into digital pixel data S1 to S3. Then, the pixel data is sequentially output from the ADC to the image processing apparatus 20 in the order of S1 to S3 in units of lines.

On the other hand, when the image processing apparatus 20 receives the pixel data S1 to S3 from the HDR sensor 10d in order from S1, the preprocessing unit 21 performs a fixed pattern noise removal process and a clamp on the pixel data in the order received. Apply processing. Of the pixel data S1 to S3 subjected to these processes, S1 is output to the color photometric unit 25 as it is, and the processed pixel data S1 to S3 is output to the delay units 22 to 24.
The delay units 22 to 24 absorb the time difference of reception of the pixel data S1 to S3 by the preprocessing unit 21, synchronize the output timing of the pixel data S1 to S3, and output them to the HDR main line processing unit 26.

On the other hand, when the pixel data S1 is input from the preprocessing unit 21, the color photometry unit 25 receives the input pixel data S1 from the adders 61a to 61c in the color separator 60 of the color information extraction unit 40. The data is output to an adder corresponding to the color of data S1.
Specifically, when the color of the pixel data S1 is red, the R signal is output to the adder 61a, when the color is green, the G signal is output to the adder 61b, and when the color is blue, the B signal is output to the adder 61c. Output.

When the R signal is input, the adder 61a adds the pixel value r and the previous r integrated value ir (initial value is 0) from the R output register 62a, and indicates the addition result. The RA signal is output to the R output register 62a.
Similarly, when the G signal is input, the adder 61b adds the pixel value g and the previous g integrated value ig (initial value is 0) from the G output register 62b, and adds the added value. A GA signal indicating the result is output to the G output register 62b.
Similarly, when the B signal is input, the adder 61c adds the pixel value b and the previous b accumulated value ib (initial value is 0) from the B output register 62c, and adds the added value. The BA signal indicating the result is output to the B output register 62c.

In this way, each time the pixel data S1 is input from the preprocessing unit 21, the distribution of each color signal in the color separator 60, the addition of the pixel value of each color and the integrated value in the adders 61a to 61c, and the addition result Are repeatedly stored in the R, G, B output registers 62a, 62b, 62c.
When such integration processing is repeatedly performed on the pixel data S1 for one frame, the r integration value ir for the pixel data S1 for one frame is stored in the R output register 62a, and the g integration value ig is stored in the G output register. The b integrated value ib is stored in the B output register 62c in 62b.

Then, in accordance with the acquired signal from the light source information estimation unit 41, the light source information estimation unit 41 and the color photometry result storage unit 27 store the R, G, B output registers 62a, 62b, and 62c in the respective registers. The r, g, b integrated values ir, ig, ib are output.
Thereafter, a frame reset signal from a reset circuit (not shown) becomes active, and the r, g, b integrated values ir, ig, ib stored in the R, G, B output registers 62a, 62b, 62c are reset.

In the present embodiment, when the light source is not included in the imaging range, the output signal level from the HDR sensor 10d becomes almost “0” or extremely small in the imaging with the ultrashort exposure time T_S1, and the r, g, b integration is performed. The values ir, ig, and ib are all “0” or extremely small values.
When the light source information estimation unit 41 acquires the r, g, b integrated values ir, ig, ib from the color information extraction unit 40, the color information → light source information conversion unit 41a sets “Rrg = ir / ig "and" Rbg = ib / ig ". Then, the light source information corresponding to the calculated Rrg and Rbg is acquired from the light source information database stored in the light source information storage unit 41b. Here, information on the type of light source is acquired as the light source information.

The color temperature of the light source is dark orange when the temperature is low due to the relationship between the temperature of the black body and the wavelength of the light emitted by the black body, and changes to yellow, white, and bluish white as the temperature rises. It will change.
Therefore, for example, when Rrg is 1 or more and larger than Rbg, it can be determined that there are a relatively large number of red pixels, and in this case, it can be estimated that the light source has a relatively low color temperature. For example, when Rbg is 1 or more and larger than Rrg, it can be determined that there are a relatively large number of blue pixels. In this case, it can be estimated that the light source has a relatively high color temperature.

The color information → light source information conversion unit 41 a outputs the acquired information on the type of light source to the WB gain setting unit 42 when the information on the type of light source corresponding to Rrg and Rbg can be acquired. On the other hand, when the information on the type of the light source corresponding to Rrg and Rbg cannot be acquired, information indicating that the type of the light source cannot be estimated is output to the WB gain setting unit 42.
When acquiring the light source type information from the light source information estimation unit 41, the WB gain setting unit 42 obtains the WB gain corresponding to the light source type information from the gain information database stored in the gain information storage unit 42b. get. For example, when it is estimated that the type of light source is sunlight (daylight), WB gains Gr and Gb corresponding to daylight are acquired. Then, the acquired WB gains Gr and Gb are output to the WB adjustment unit 31.

Further, when the WB gain setting unit 42 acquires information indicating that the type of the light source could not be estimated from the light source information estimation unit 41, “Gr = Gb = 1” is set as WB gain Gr, Gb. Output to the adjustment unit 31.
Further, when the color photometric result storage unit 27 acquires the r, g, b integrated values ir, ig, ib as the color photometric result from the color information extraction unit 40, it stores them in the memory via the memory IF. The color photometric result may be stored by overwriting for each frame, a plurality of frames may be stored, or after a plurality of frames are stored, the oldest one may be sequentially overwritten and stored. .

On the other hand, when the HDR linear processing unit 30 receives the pixel data S1 to S3 delayed in the delay units 22 to 24 and synchronized in output timing in parallel with the processing of the color photometry unit 25, first, the synthesis processing unit In 30a, HDR_RAW pixel data is generated.
Specifically, the synthesis processing unit 30a calculates the normalized gains RS3 and RS2 in the normalized gain calculation units 70 and 71 from the information of the exposure times T_S1 to T_S3 held in the memory or acquired from the HDR imaging device 10.
In parallel, the composite weight calculator 74 calculates composite weights W1 to W3 from the pixel data S1 to S3 using the above equations (1) to (4).

Next, the normalizing units 72 and 73 multiply the pixel data S3 and S2 by the normalization gains RS3 and RS2, respectively, thereby calculating the normalized pixel data NS3 and NS2.
Next, in the weighting units 75 to 77, the normalized pixel data NS3 is multiplied by the composite weight W3, the normalized pixel data NS2 is multiplied by the composite weight W2, and the pixel data S1 is multiplied by the composite weight W1 to obtain weighted pixels. Data WS1, WS2 and WS3 are generated.

Next, the combining unit 78 combines (adds) the weighted pixel data WS1, WS2, and WS3 to generate HDR_RAW pixel data, and outputs the generated HDR_RAW pixel data to the color processing unit 30b.
When the HDR_RAW pixel data is input from the composition processing unit 30a, the color processing unit 30b accumulates the input HDR_RAW pixel data in the line memory. When a certain amount of HDR_RAW pixel data is accumulated in the line memory, color correction processing and sharpness processing are performed to generate HDR color pixel data. Then, the generated HDR color pixel data is output to a line memory (not shown).

  When the line memory receives the HDR color pixel data from the color processing unit 30b, the line memory accumulates the data and delays the output timing. Specifically, the output timing Gr, Gb of the WB gain from the color photometry unit 25 and the output timing of the HDR color pixel data R_hdr, B_hdr corresponding to the gain are delayed so as to be synchronized. Further, the output timing of the HDR color pixel data G_hdr from the WB adjustment unit 31 and the output timing of the HDR color pixel data R_hdr and B_hdr after the WB adjustment from the WB adjustment unit 31 are synchronized with each other. The input timing to 31 is delayed.

When the HDR color pixel data R_hdr, G_hdr, B_hdr from the line memory and the WB gains Gr, Gb from the WB gain setting unit 42 of the color photometry unit 25 are input to the WB adjustment unit 31, white balance adjustment is performed. Is done.
Specifically, the HDR color pixel data R_hdr from the line memory is input to the multiplier 31a of the WB adjustment unit 31, the WB gain Gr from the color photometry unit 25 is input, and R_hdr is multiplied by Gr.

Also, the HDR color pixel data B_hdr from the line memory is input to the multiplier 31b of the WB adjustment unit 31, the WB gain Gb from the color photometry unit 25 is input, and B_hdr is multiplied by Gb.
Then, G_hdr from the line memory, R_hdr after multiplication by Gr, and B_hdr after multiplication by Gb are output to the level correction processing unit 32 in synchronization with the output timing.

On the other hand, when receiving the HDR color pixel data R_hdr, G_hdr, B_hdr after adjusting the WB, the level correction processing unit 32 converts these into the luminance pixel data P (x, y), and converts the converted luminance pixel data P (x x, y) is stored in the LP line memory (not shown).
When the luminance pixel data P (x, y) required for the blurring process is accumulated in the LP line memory, the blurring process using a Gaussian filter (low-pass filter) is executed, and the luminance pixel data P ( x, y) is converted into illumination light component data L (x, y).

Then, the brightness level correction gain corresponding to the illumination light component data L (x, y) is acquired from the first LUT. Further, in an internal multiplier (not shown), the HDR color pixel data input from the WB adjustment unit 31 is multiplied by a luminance level correction gain to perform level conversion (tone mapping) of the HDR color pixel data.
Further, a γ conversion value corresponding to the HDR color pixel data after level conversion is acquired from the second LUT, and the acquired conversion value is output as an RGB video signal.
As described above, the pixel signals S1 to S3 of each frame are input to the preprocessing unit 21 in the order of pixel data S1 → S2 → S3.

In FIG. 14, S1 frames 0 to 2, S2 frames 0 to 2, and S3 frames 0 to 2 are subframes, and the period from the start of S1 frame 0 to the end of S3 frame 0 is a period of one frame. Become.
In the HDR main line processing unit 26, as described above, processing (main line system processing) for generating HDR color image data using all the pixel data S1 to S3 is performed.
Therefore, in this embodiment, the start timing of the main line processing is set to the timing at which the first pixel data S3 (line unit) of each subframe is input, as shown in S3 frames 0 to 2 in FIG. In each subframe, every time new pixel data S3 is input, main line processing is sequentially performed as in pipeline processing.

Further, as described above, the color photometry unit 25 performs processing for extracting color information from the pixel data S1, estimating light source information based on the extracted color information, and setting a WB gain based on the estimated light source information. Is called.
In this embodiment, as shown in S1 frames 0 to 2 in FIG. 14, the start timing of the color photometry processing of the color photometry unit 25 is the first pixel data S1 (line unit) of each subframe from the preprocess unit 21. Input timing is assumed. In each subframe, each time new pixel data S1 is input, color photometry processing is sequentially performed as in pipeline processing.

As a result, the delay amount shown in FIG. 14 occurs between the end timing of the S1 frame and the start timing of the S3 frame in each subframe. Further, a time difference due to scanning shown in FIG. 14 occurs between the start timing of the main line system processing and the start timing of the color photometric system processing in each subframe. In this time difference, it is desirable to complete the color photometry processing (extraction of color information, estimation of light source information, setting of WB gain). However, extraction of color information requires one subframe of pixel data S1, which is insufficient during the scanning time difference period. In addition to this, time for light source information estimation processing and WB gain setting processing is also required. Therefore, in this embodiment, the main line system processing is delayed by these shortage times, so that the color photometric results in each frame are converted into HDR color image data generated from the pixel data S1 to S3 of the same frame. It can be reflected. That is, feedforward type control can be performed.
In the conventional feedback type control, the color photometric result is reflected in the next frame, and therefore a delay of one frame or more occurs. On the other hand, in the present embodiment, only the delay in the color photometry unit 25 is required.

(6) Action / Effect As described above, the imaging device 1 according to the present embodiment uses the color photometry unit 25 to obtain pixel data of the ultrashort exposure time T_S1 that is acquired prior to the pixel data S3 of the standard exposure time T_S3. Color information extraction processing can be performed using S1.
Therefore, color photometric processing can be performed with an image having no saturated pixels, and color information of the light source with high accuracy can be extracted.

Further, the ultrashort exposure time T_S1 is set to a time such that the HDR sensor 10d outputs a signal having a level higher than “0” only for light from the light source or light obtained by specularly reflecting light from the light source. I made it. Specifically, it is set to a time for outputting a signal having a level higher than “0” for an input having a predetermined luminance or higher.
Therefore, the color information can be extracted from the pixel data S1 containing a large amount of color information of the light source, so that the color information of the light source can be extracted with higher accuracy.

Furthermore, it is possible to estimate light source information such as the color temperature and type of the light source based on the extracted color information, and to set a WB gain for adjusting the white balance based on the estimation result.
Accordingly, the light source information can be estimated using highly accurate color information, and WB gain control suitable for the light source information thus estimated can be automatically set.
Further, the HDR linear processing unit 30 can generate HDR color pixel data by linearly synthesizing the pixel data S1 to S3 whose output timings are synchronized by the delay units 22 to 24 and performing color correction processing.

Further, in the multipliers 31a and 31b, the HDR color pixel data R_hdr and B_hdr output after being delayed in accordance with the output timing of the WB gains Gr and Gb are multiplied by the WB gains Gr and Gb, thereby obtaining the HDR color pixel data. The white balance can be adjusted.
Therefore, the color photometric result in each frame can be reflected on the HDR color image data generated from the pixel data S1 to S3 of the same frame, and feedforward control can be performed. In other words, since the image (frame) subject to color photometry and the image (frame) subject to luminance control can be made the same, it is possible to suppress the occurrence of oscillation (flicker) phenomenon according to the luminance variation of the subject. . Therefore, a stable output image can always be obtained. Further, it has an advantage that it can be easily incorporated into a system that requires real-time performance.

Further, the level correction processing unit 32 can generate luminance pixel data P (x, y) from the HDR color pixel data, and can generate illumination light component data L (x, y) from P (x, y). . Furthermore, the brightness level correction gain corresponding to the illumination light component data L (x, y) can be acquired from the first LUT.
Therefore, since the HDR color image data is subjected to level conversion (tone map) processing based on the illumination light component data L, the gradation is maintained and the color reproducibility is good in the entire dynamic range. Can be output.

  In addition, even when the color photometric image (frame) and the target image (frame) for WB adjustment are not the same, that is, when the result of the color photometric image (frame) is reflected in the next frame However, since the color photometry process is performed prior to the main line system process, a sufficient calculation time for the color photometry can be secured. Thereby, complicated control such as dynamic control based on the color photometric result is possible.

(7) Correspondence with the Present Invention In the above embodiment, the HDR imaging device 10 corresponds to the imaging device, and the preprocessing unit 21 corresponds to the image data acquisition unit for synthesis and the image data acquisition unit for color measurement, The HDR linear processing unit 30 corresponds to a composite image data generation unit.
In the above embodiment, the color information extraction unit 40 corresponds to the color information extraction unit, the light source information estimation unit 41 corresponds to the light source information estimation unit, the WB gain setting unit 42 corresponds to the gain setting unit, The color photometric result storage unit 27 corresponds to a color information storage unit.
In the above embodiment, the WB adjustment unit 31 corresponds to a white balance adjustment unit, and the level correction processing unit 32 corresponds to an image conversion unit.

In the above embodiment, the process of acquiring the pixel data S1 to S3 in the pre-processing unit 21 corresponds to the image data acquisition step for synthesis and the image data acquisition step for color measurement, and the HDR color image in the HDR linear processing unit 30. The data generation process corresponds to the composite image data generation step.
In the above embodiment, the color information extraction process from the pixel data S1 in the color information extraction unit 40 corresponds to the color information extraction step, and the light source information estimation process in the light source information estimation unit 41 is the light source information estimation. Corresponds to the step.
In addition, the setting process of the WB gains Gr and Gb in the WB gain setting unit 42 corresponds to the gain setting step, and the white balance adjustment process in the HDR color image data in the WB adjustment unit 31 corresponds to the white balance adjustment step.

(8) Modifications Note that the imaging device 1 in the above embodiment can be combined with other devices (not shown) such as a display device and a memory device to constitute an electronic device such as a digital camera or a digital video camera. It is.
In the above embodiment, the color information of the pixel data S1 that is equal to or higher than the input luminance of the light source or the specular reflected light of the light source is extracted by controlling the ultrashort exposure time T_S1 as the color information. Not limited to.
For example, other configurations such as a configuration in which pixel data S1 including a luminance component other than the light source is imaged, and only those having a predetermined luminance or higher are extracted when extracting color information.

Moreover, in the said embodiment, although it was set as the structure which calculates the integrated value of the pixel value (for example, luminance value) of each color as color information, it determines not only this structure but the luminance level of the pixel data of each color, for example, Other color information may be extracted, such as calculating the number of pixels of each luminance level of each color.
In each of the above embodiments, the HDR sensor 10d of the image sensor 10 has a sensor cell array configured using CMOS technology, but the configuration is not limited thereto. For example, other configurations such as a configuration having a sensor cell array composed of CCDs may be used.

The above embodiments are preferable specific examples of the present invention, and various technically preferable limitations are given. However, the scope of the present invention is described in particular in the above description to limit the present invention. As long as there is no, it is not restricted to these forms. In the drawings used in the above description, for convenience of illustration, the vertical and horizontal scales of members or parts are schematic views different from actual ones.
Further, the present invention is not limited to the above-described embodiment, and modifications, improvements, and the like within the scope that can achieve the object of the present invention are included in the present invention.

DESCRIPTION OF SYMBOLS 1 ... Imaging device, 10 ... HDR imaging element, 10a ... Imaging lens, 10b ... Micro lens, 10c ... Color filter array, 10d ... HDR sensor, 10d ... Drive circuit, 10f ... Reading circuit, 20 ... Image processing device, 21 ... Pre-processing unit, 22-24 ... delay unit, 25 ... color photometry unit, 26 ... HDR main line processing unit, 30 ... HDR linear processing unit, 30a ... synthesis processing unit, 30b ... color processing unit, 31 ... WB adjustment unit, 31a , 31b ... multiplier, 32 ... level correction processing unit, 40 ... color information extraction unit, 41 ... light source information estimation unit, 41a ... color information → light source information conversion unit, 41b ... light source information storage unit, 42 ... WB gain setting unit , 42a ... light source information → WB gain conversion unit, 42b ... gain information storage unit, 60 ... color separator, 61a to 61c ... adder, 62a ... R output register, 62b ... G output Use registers, 62c ... B output registers, 70, 71 ... normalizing gain calculation unit, 72, 73 ... normalization unit, 74 ... combining weight calculator, 75-77 ... weighting unit, 78 ... combining portion

Claims (11)

Image data for synthesis that acquires two or more image data from a plurality of image data obtained by imaging a subject with a plurality of different exposure times in each frame period from the imaging device capable of imaging the subject. Acquisition means;
Combined image data generating means for generating a combined image data by linearly combining the two or more image data acquired by the combining image data acquiring means based on an exposure time ratio;
Color measurement image data acquisition means for acquiring short exposure image data which is image data corresponding to the shortest exposure time among the plurality of image data from the imaging device;
Color information extraction means for extracting color information which is information relating to the color of the short exposure image from the short exposure image data acquired by the color measurement image data acquisition means;
Based on color information extracted by the color information extraction means, light source information estimation means for estimating information relating to a light source at the time of imaging the subject;
Based on the estimation result of the light source information estimation means, a gain setting means for setting a gain used for white balance adjustment;
Using the gain set by the gain setting means, white balance adjustment of composite image data generated by combining the two or more image data in the same frame period as the short-exposure image data related to the gain setting is performed. White balance adjustment means for performing,
The image sensor can read out a pixel signal by a nondestructive readout method that reads out a pixel signal corresponding to the accumulated charge while maintaining the accumulated charge from each pixel, and in order from the shorter exposure time in each frame period. The pixel signals are exposed at the different exposure times, the pixel signals are read out from the exposed pixels by the nondestructive readout method, and the pixel signal data constituting the image data corresponding to the exposure times in the read order. Is output,
The composition image data acquisition means acquires two or more image data including image data corresponding to the longest exposure time in each frame period from the image sensor,
The composite image data generating means generates composite image data based on two or more image data including image data corresponding to the longest exposure time;
The color measurement image data acquisition means precedes the short-exposure image data from the image sensor before the composition image data acquisition means acquires image data corresponding to the longest exposure time in each frame period. Get
The color information extraction means extracts the color information from the short exposure image data acquired in advance,
The light source information estimation means estimates information related to a light source based on color information extracted from the short-exposure image data acquired in advance.
The gain setting means sets a gain used for white balance adjustment based on information relating to a light source estimated based on color information extracted from the short-exposure image data acquired in advance.
The white balance adjusting means uses the gain set by the gain setting means,
Image processing characterized by adjusting white balance of the composite image data generated by the composite image data generation means based on the two or more image data in the same frame period as the short exposure image data relating to the gain setting apparatus. The image processing apparatus according to claim 1, wherein the light source information estimation unit estimates a type of a light source as information relating to the light source. The light source information estimation unit is extracted by the color information extraction unit based on a database in which information in which each index value indicated by the color information is associated with the type of light source corresponding to each index value is registered. Estimate the type of light source corresponding to the color information,
The gain setting means is a gain corresponding to the type of light source estimated by the light source information estimation means, based on a database in which information relating the type of light source and the gain corresponding to each type of light source is registered. The image processing apparatus according to claim 2, wherein: 4. The image processing apparatus according to claim 2, wherein the white balance adjustment is not performed when the type of the light source cannot be estimated by the light source information estimation unit. 5. The image processing apparatus according to claim 1, wherein the light source information estimation unit estimates a color temperature of the light source as information relating to the light source. The light source information estimation means is extracted by the color information extraction means based on a database in which information in which each index value indicated by the color information is associated with the color temperature of the light source corresponding to each index value is registered. The color temperature of the light source corresponding to the color information
The gain setting unit sets a gain corresponding to the color temperature of the light source estimated by the light source information estimation unit based on a database in which information in which the color temperature of the light source is associated with the gain is registered. The image processing apparatus according to claim 5. 7. The exposure time of the short-exposure image data is set to the shortest time during which pixel signal data can be output with respect to a light source included as the subject or specular reflection light of the light source. The image processing apparatus according to claim 1. The image processing apparatus according to claim 1, further comprising a color information storage unit that stores the color information extracted by the color information extraction unit. 9. The apparatus according to claim 1, further comprising image conversion means for converting the brightness of the composite image data after the white balance is adjusted by the white balance adjustment means to generate composite image data having different gradation characteristics. The image processing apparatus according to any one of the above. Image data for synthesis that acquires two or more image data from a plurality of image data obtained by imaging a subject with a plurality of different exposure times in each frame period from the imaging device capable of imaging the subject. An acquisition step;
A composite image data generation step of generating a composite image data by linearly combining the two or more image data acquired in the composite image data acquisition step based on an exposure time ratio;
Color measurement image data acquisition step for acquiring short exposure image data that is image data corresponding to the shortest exposure time among the plurality of image data from the imaging device;
A color information extraction step of extracting color information from the short exposure image data acquired in the color measurement image data acquisition step;
Based on the color information extracted in the color information extraction step, a light source information estimation step for estimating information related to a light source at the time of imaging the subject;
Based on the estimation result in the light source information estimation step, a gain setting step for setting a gain used for white balance adjustment;
Using the gain set in the gain setting step, white balance adjustment of composite image data generated by combining the two or more image data in the same frame period as the short-exposure image data related to the gain setting is performed. Performing a white balance adjustment step,
In the color measurement image data acquisition step, the image sensor can read out pixel signals by a non-destructive readout method of reading out pixel signals corresponding to the accumulated charges while maintaining accumulated charges from the pixels. In the frame period, each pixel is exposed at the plurality of different exposure times in order from the shortest exposure time, and the pixel signal is read from each exposed pixel by the nondestructive readout method, and the exposure time corresponds to each exposure time. Output the pixel signal data constituting the image data to be
In the composite image data acquisition step, two or more pieces of image data including image data corresponding to the longest exposure time in each frame period are acquired from the image sensor,
In the composite image data generation step, composite image data is generated based on two or more image data including image data corresponding to the longest exposure time,
In the color measurement image data acquisition step, the short-exposure image data is acquired from the image sensor prior to the synthesis image data acquisition step acquiring image data corresponding to the longest exposure time in each frame period. Get
In the color information extraction step, the color information is extracted from the short exposure image data acquired in advance,
In the light source information estimation step, information relating to the light source is estimated based on color information extracted from the short-exposure image data acquired in advance.
In the gain setting step, a gain used for white balance adjustment is set based on information on a light source estimated based on color information extracted from the short-exposure image data acquired in advance.
In the white balance adjustment step, using the gain set in the gain setting step , the composite image data generation step is performed based on the two or more image data in the same frame period as the short exposure image data related to the gain setting. And adjusting the white balance of the composite image data generated in this way. An image sensor capable of imaging a subject;
An image processing apparatus comprising: the image processing apparatus according to claim 1.

Images