JP2003173438A - Image processing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing device capable of producing a composite image correctly reflecting the color information of an image before composition. <P>SOLUTION: A first image 231 is acquired in a state that a part of comparatively high brightness of an object is properly exposed, and a second image 232 is acquired in a state that a part of comparatively low brightness is properly exposed in a digital camera. Both images are composed by an image composing part 204 on the basis of a weighted coefficient determined on each image element, after each image element value of the both images are converted into an apparatus-independent XYZ value by an XYZ conversion part 202, to produce the composite image. As the image processing is performed in a state that the image element values are represented by the apparatus-independent XYZ values, the composite image on which the color information of the first image and the second image before the composition is correctly reflected, can be formed. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for synthesizing images to obtain an image whose gradation is adjusted.

[0002]

2. Description of the Related Art In an image pickup device such as a digital camera, a CCD (Charge Coupled) having a two-dimensional pixel array is used.
When you shoot a subject using an image sensor such as Device),
While the pixel value saturation (so-called whiteout) phenomenon occurs in the area corresponding to the relatively high brightness part of the subject,
There may occur a phenomenon in which a region corresponding to a relatively low-brightness portion of the subject is blackened (so-called black shadow). This phenomenon is caused by the narrow dynamic range of a general image pickup element, and is a factor of image quality deterioration.

In order to eliminate such a phenomenon, conventionally, the exposure condition is changed for the same subject, two images are photographed, and the obtained two images are combined at an appropriate ratio. There is known a technique of performing gradation adjustment processing by means of and generating a composite image with a substantially expanded dynamic range.

[0004]

Generally, in an image acquired by an image pickup device such as a digital camera, the pixel value (color component value) indicating the color information of each pixel is RGB.
It is represented by a color system value (RGB value). Therefore, in the process of generating a gradation-adjusted composite image as described above, various image processes are performed using pixel values represented by RGB values.

However, since such RGB values depend on the characteristics of the acquired image pickup device (device dependent values), when a composite image is generated with the RGB values as it is, various generation processes are required. The color information is distorted in the image processing (arithmetic processing), and the color information of the two images before the combining cannot be accurately reflected in the final combined image.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing apparatus capable of generating a composite image that accurately reflects color information of an image before composition.

[0007]

In order to solve the above-mentioned problems, the invention of claim 1 is an image processing apparatus, wherein a first image and a second image acquired by changing the exposure condition for the same subject. A color system conversion unit that converts the color component value of each pixel of each image into a color component value in a device-independent color system, and the color component value of each pixel is converted by the color system conversion unit. An image synthesizing unit for synthesizing the first image and the second image to generate a grayscale-adjusted synthetic image.

According to a second aspect of the present invention, there is provided an image processing device, wherein a spectrum of an illumination environment for the object is spectrally divided from an image group acquired under an exposure condition in which a relatively high brightness portion of the object is properly exposed. A means for obtaining first object color component data corresponding to the image data from which the influence of the characteristic is removed, and an image group acquired under an exposure condition in which a relatively low luminance portion of the subject is properly exposed, to the subject. Means for obtaining second object color component data corresponding to image data from which the influence of the spectral characteristics of the illumination environment is removed, the first and second object color component data, and the influence of the spectral characteristics of the light source on the image. Means for generating a first image and a second image by synthesizing with the illumination component data indicating the color component value of each pixel of the first image and the second image, and a device-independent color system Color in A color system conversion means for converting the frequency value,
An image synthesizing unit for synthesizing the first image and the second image in which the color component values of the respective pixels are converted by the color system conversion unit to generate a gradation-adjusted combined image. ing.

According to a third aspect of the present invention, in the image processing apparatus according to the first or second aspect, for each pixel of the first image and the second image, the validity of the color component value of the pixel is determined. Further comprising means for setting a compositing ratio according to the above, wherein the image composing means composes the color component values after color system conversion of each pixel of the first image and the second image at the composition ratio, It is characterized in that the composite image is generated.

The invention according to claim 4 is the image processing apparatus according to any one of claims 1 to 3, wherein the first
Brightness adjusting means for adjusting the brightness of each pixel of the composite image based on the brightness of each of the image and the second image and the composite ratio of each pixel of the first image and the second image. Is further equipped.

According to a fifth aspect of the present invention, in the image processing apparatus according to the fourth aspect, the acceptance for accepting designation of an adjustment degree when the brightness adjusting means adjusts the brightness of each pixel of the composite image Means are further provided.

[0012]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

<1. First Embodiment><1-1. Structure of Digital Camera> FIG. 1 shows a first embodiment of the present invention.
2 is a perspective view showing the entire digital camera 1 which is the image processing apparatus according to the embodiment of FIG. The digital camera 1 includes a lens unit 11 that performs shooting, and a main body 12 that processes an image acquired as digital data by the lens unit 11.

The lens unit 11 has a lens system 111 having a plurality of lenses, and a CCD 112 for acquiring an image of a subject through the lens system 111. And
The image signal output from the CCD 112 is sent to the main body 12. Further, the lens unit 11 includes a finder 113 and a distance measuring sensor 11 for an operator to capture a subject.
4th grade is also placed.

The main body 12 is provided with a flash 121 and a shutter button 122. An operator captures an object through the finder 113 and operates the shutter button 122 to electrically capture an image with the CCD 112. It At this time, if necessary, the flash 121
Emits light. It should be noted that the CCD 112 is a three-band image pickup device that acquires values for each color of R, G, and B as the value of each pixel.

The image signal from the CCD 112 is the main body 12
The processing described below is performed inside the main body 1 as necessary.
It is stored in the external memory 123 (so-called memory card) mounted on the No. 2 memory. The external memory 123 is the main body 1
2 By opening the lid on the lower surface and operating the take-out button 124, the take-out button 124 is taken out. The data stored in the external memory 123, which is a recording medium, can be transferred to another device by using a computer or the like provided separately. Conversely, the digital camera 1 can read the data stored in the external memory 123 by another device.

FIG. 2 is a diagram showing a state when the digital camera 1 is viewed from behind. A liquid crystal display 125 is provided in the center of the back surface of the main body 12 for displaying a photographed image and a menu for the operator, and the side of the display 125 is used for inputting according to the menu displayed on the display 125. Operation buttons 12 for operating
6 is arranged. Thereby, the operation of the digital camera 1, the maintenance of the external memory 123, the reproduction of the image, the setting of the photographing mode and the like can be performed.

The digital camera 1 has a gradation adjusting mode as its photographing mode as well as a normal photographing mode for performing normal photographing, and one of the photographing modes can be selected by the operation button 126. ing. The gradation adjustment mode is to adjust the gradation by changing the exposure condition for the same subject, taking two images continuously, and combining these two images at an appropriate combining ratio. This mode is intended to obtain a composite image (hereinafter, also referred to as a “tone adjustment image”) with a wide dynamic range, and details of this processing will be described later.

FIG. 3 is a block diagram showing the main configuration of the digital camera 1.

In the configuration shown in FIG. 3, the lens system 111,
CCD 112, A / D converter 115, shutter button 1
22, the CPU 21, the ROM 22, and the RAM 23 realize the function of acquiring an image. That is, the lens system 111
When the image of the subject is formed on the CCD 112 by the, and the shutter button 122 is pressed, the image signal from the CCD 112 is, for example, 8 bits (0 to 0) by the A / D converter 115.
255) digital image signal. The digital image signal converted by the A / D converter 115 is the main body 12
Is stored in the RAM 23 as image data. The control of these processes is performed by the CPU 21 operating according to the program 221 stored in the ROM 22.

Further, the CPU 2 provided in the main body 12
1, ROM22 and RAM23 implement | achieve the function which processes an image. Specifically, according to the program 221 stored in the ROM 22, the CPU 21 performs image processing on the acquired image while using the RAM 23 as a work area.

The external memory 123 is connected to the RAM 23 and exchanges various data based on the input operation from the operation button 126. Also, the display 125
Also displays an image or displays information to the operator based on a signal from the CPU 21.

The flash 121 is a light emission control circuit 121a.
When the flash 21 is instructed to be emitted from the CPU 21, the light emission control circuit 121a controls the start / stop of light emission of the flash 121.

The exposure control in the digital camera 1 is performed by adjusting the shutter speed (the integration time of the CCD 112) and the aperture value (the aperture diameter of the aperture included in the lens system 111). In FIG. 3, the exposure control unit 201 is C
This is a function realized by the PU 21, the ROM 22, the RAM 23, and the like, and sets and controls such a shutter speed and aperture value (that is, exposure condition).

When setting the exposure conditions, the CCD 112
A signal indicating the brightness of the subject is input to the exposure control unit 201 from the exposure control unit 201 and the CCD 1 and the brightness of the subject.
The exposure value is determined based on 12 sensitivity characteristics. Then, based on the determined exposure value, the shutter speed and the aperture value are set with reference to a program diagram showing the relationship between the aperture value and the shutter speed prepared in advance in the digital camera 1.

As a photometric mode for measuring the brightness of the object, the digital camera 1 uses the normal photometric mode in which signals from predetermined pixels in several areas of the CCD 112 are used, and the pixel corresponding to the central portion of the screen of the finder 113. A spot metering mode that uses signals from the neighboring pixels is provided. Such a photometric mode is appropriately selected manually or automatically according to shooting.

<1-2. Digital camera operation> Figure 4
Mainly includes the CPU 21, the ROM 22, and the RAM 23.
FIG. 7 is a block diagram showing the configuration related to the gradation adjustment mode of the functions realized by, together with other configurations, and FIGS. 5 and 6 are diagrams showing the flow of shooting and image processing in the gradation adjustment mode. In the configuration shown in FIG. 4, the exposure control unit 201, the XYZ conversion unit 202, and the brightness level adjustment unit 20.
3, image composition unit 204, local contrast correction unit 20
5, the RGB conversion unit 206 and the weight setting unit 207 are C
The functions realized by the PU 21, the ROM 22, the RAM 23, etc. are shown. The operation of the digital camera 1 in the gradation adjustment mode will be described below with reference to these drawings.

In the tone adjustment mode, two images are obtained by exposing the high-luminance portion and the low-luminance portion of the subject respectively, and these two images are combined to obtain a tone adjusted image. It will be. Therefore, first, the user determines the composition through the viewfinder 113, and the exposure values of a relatively high-luminance portion and a relatively low-luminance portion of the subject included in the determined composition are calculated.
To get.

Specifically, the metering mode of the digital camera 1 is set to the spot metering mode, and the user aligns the central portion of the screen of the finder 113 with the high-luminance portion of the subject and performs a predetermined operation with the operation button 126. In response to this operation, a signal indicating the brightness of the subject is input from the CCD 112 to the exposure control unit 201, and the exposure control unit 201 acquires an exposure value based on this signal. The acquired exposure value is stored in the RAM 23 as the first exposure value Ev1 that matches the high-luminance portion of the subject (step ST1).
1).

Next, in the same manner, the user aligns the central portion of the screen of the finder 113 with the low-luminance portion of the subject and performs a predetermined operation with the operation button 126. In response to this operation, a signal indicating the brightness of the subject is input from the CCD 112 to the exposure control unit 201, and the exposure control unit 201 acquires an exposure value based on this signal. The acquired exposure value is the second exposure value Ev that matches the low-luminance portion of the subject.
2 is stored in the RAM 23 (step ST1
2).

FIG. 7 is a diagram showing a specific example of a subject having a high-luminance portion and a low-luminance portion. In FIG. 7, a portion 71 outside the window receives sunlight, and a portion 72 inside the room receives illumination such as a fluorescent lamp inside the room. Therefore, the part 71 outside the window has a relatively high brightness, and the part 72 inside the room has a relatively low brightness. When the subject shown in this example is photographed, the exposure value matched to the brightness of the portion 71 outside the window is acquired as the first exposure value Ev1, and the exposure value matched to the brightness of the portion 72 inside the room is the second exposure value. It will be acquired as Ev2.

When the first and second exposure values have been acquired, the user then operates the shutter button 122 while aligning the screen of the finder 113 with the determined composition. In response to this operation, the digital camera 1 shoots under the exposure condition based on the first exposure value Ev1, and an image in which a region corresponding to a high-brightness portion of the subject has a proper exposure (hereinafter, referred to as “first image”). .) That is, the CCD is set under the exposure condition that the aperture is relatively narrowed down and the shutter speed is relatively high.
An image is obtained at 112, and the obtained image (to be exact, an image signal) is sent from the A / D conversion unit 115 to the RAM 23 and stored as a first image 231 (step ST1
3).

Next, an image is taken under the exposure condition based on the second exposure value Ev2, and an image in which a region corresponding to a low-luminance portion of the subject has a proper exposure (hereinafter referred to as "second image").
To get That is, the CCD 1 is exposed under the exposure condition that the aperture diameter of the diaphragm is relatively open and the shutter speed is relatively low.
An image is obtained at 12, and the obtained image is the A / D conversion unit 1
It is sent from 15 to the RAM 23 and stored as the second image 232 (step ST14).

FIG. 8 is a diagram showing an example of the first image 231 when the subject shown in FIG. 7 is photographed, and FIG. 9 is an example of the second image 232 when the subject shown in FIG. 7 is photographed. FIG. As shown in FIG. 8, in the first image 231, a region corresponding to a high-intensity part of the subject (portion 71 outside the window: trees and the like are included in the figure) has proper exposure and is appropriately photographed. An area corresponding to the low-luminance portion of the subject (indoor portion 72: a vase or the like is included in the drawing) is underexposed. On the contrary, as shown in FIG. 9, in the second image 232, the area corresponding to the low-brightness portion of the subject is properly exposed and appropriately photographed,
The area corresponding to the high brightness portion of the subject is overexposed.

These two photographings are carried out quickly like continuous photographing. Therefore, the first image 231 and the second image 2
The shooting range with 32 is the same.

When the first image 231 and the second image 232 are stored in the RAM 23 by two times of photographing, the pixel values (color component values) represented by the RGB values of both images are changed by the XYZ conversion section 202. XYZ by predetermined matrix operation
Converted to XYZ values in the color system. The matrix used for this calculation is a matrix considering the spectral transmission characteristics of the color filter of the CCD 112. Therefore,
By this calculation, each pixel value is represented by an XYZ value that does not depend on the characteristics of the CCD 112, and the color information of each pixel is accurately represented in the device-independent XYZ color system.
The matrix used for the calculation is obtained in advance by measurement or the like and stored in the ROM 22 or the RAM 23 (step ST15).

Next, as a process before the first image 231 and the second image 232 are combined, their brightness levels are matched (step ST16). Since the first image 231 is a relatively underexposed image in which the region corresponding to the high-brightness portion of the subject is a proper exposure, the pixel value is relatively low, and conversely, the second image 232 is a low-brightness subject. The pixel value is relatively high because the image corresponding to a portion is a relatively overexposed image in which the proper exposure is obtained. Therefore, if the two images are simply combined at the same combination ratio, the influence of the pixel value of the second image 232 (the influence of the color information) becomes strong in the generated combined image. Therefore, the brightness level of the second image 232 is lowered so as to match the brightness level of the first image 231.

Specifically, first, a difference exposure value ΔEv which is a difference value between the first exposure value Ev1 and the second exposure value Ev2 is exposed as a value indicating a difference in brightness (brightness difference) between the two images. It is input from the control unit 201 to the brightness level adjusting unit 203.
The differential exposure value ΔEv is calculated by
It is calculated by 1.

[0039]

[Equation 1]

Then, the brightness level adjusting unit 203 uses the differential exposure value ΔEv to perform the calculation shown in the following expression 2, so that the brightness level of the overexposed second image 232 is the underexposed first image. 231 of the brightness level. In the equation 2, X ₂ , Y ₂ , Z ₂
Are XYZ values of one pixel of the second image 232 before brightness level adjustment, and X ₂ ′, Y ₂ ′, and Z ₂ ′ are XYZ of the pixel of the second image 232 after brightness level adjustment.
It is a value.

[0041]

[Equation 2]

When the exposure value is Ev, the value indicating the brightness of the subject is Bv, and the value indicating the sensitivity of the CCD 112 is Sv, the exposure value Ev is

[0043]

[Equation 3]

It is expressed as Therefore, since the value Sv indicating the sensitivity of the CCD 112 is constant, if the subject has a relatively high brightness (the value Bv is relatively large), the exposure value Ev.
Is determined as a relatively large value, and conversely, when the subject has relatively low brightness (value Bv is relatively small), the exposure value Ev
Is determined as a relatively small value. Therefore, the first
The exposure value Ev1 and the second exposure value Ev2 are different from each other (specifically, the first exposure value Ev1> the second exposure value Ev2).
The calculated differential exposure value ΔEv is a non-zero value (positive value).

In general, when the exposure value Ev is increased by one step to acquire an image, the brightness (brightness) of the acquired image is halved. Therefore, the operation of Equation 2 is performed by the second image 2
32 for each pixel so that the second image 232
Is adjusted so as to match the brightness level of 231 of the first image.

Even when the brightness level of the image is changed in this way, the pixel values are represented by real XYZ values rather than natural RGB values, and therefore the color information indicated by each pixel is accurate. It is held in and never lost.

Next, the weight setting unit 207 refers to the second image 232 before color system conversion, and the weighting coefficient WT (0
≦ WT ≦ 1) is determined for each pixel. That is, one pixel in the second image 232 is determined as the target pixel (FIG. 6: step ST17), and the weighting coefficient of the target pixel is determined based on the effectiveness of the pixel value of the target pixel (step ST18). . When the weighting coefficient WT for one target pixel is determined, step ST18 is repeated while sequentially changing the target pixel, and the weighting coefficient WT is determined for all pixels and stored in the RAM 23.

Next, using the determined weighting coefficient WT, the image combining unit 204 performs the calculation shown in the following formula 4 for each pixel to generate a combined image 241 and R
It is stored in the AM 23 (step ST20). In Expression 4, X ₁ , Y ₁ , and Z ₁ are the XYZ values of one pixel of the first image 231 (pixel to be calculated),
X ₂ ′, Y ₂ ′, and Z ₂ ′ are XYZ values of the pixels of the second image 232 after the brightness level adjustment (pixels at the same position as the target pixel of the first image 231), and Xc, Yc, and Zc are These are XYZ values of pixels of the composite image 241 (pixels at the same position as the target pixel of the first image 231).

[0049]

[Equation 4]

The calculation of equation 4 is a weighted average using the weighting coefficient WT, and the weighting coefficient WT is substantially the first image 2
This corresponds to the composition ratio of 31 and the second image 232.

In step ST18, the weighting coefficient W
As the validity of the pixel value used when determining T, the pixel value of the second image 232 before color system conversion (for example, R,
The average value of each of the G and B color components is referred to.

Since the second image 232 is an image in which the area corresponding to the low-brightness portion of the subject is properly exposed, when the pixel value is relatively low, the pixel value makes the color information of the subject valid. On the contrary, when the pixel value is relatively high, the pixel value does not effectively indicate the color information of the subject. That is, when the pixel value in the second image 232 is relatively low, the effectiveness of the color information of the pixel is relatively high. Conversely, when the pixel value is relatively high, the pixel value of the pixel is relatively high. It can be said that the effectiveness of the color information is relatively low.

For this reason, the weighting coefficient WT is applied to the pixel having the pixel value of high effectiveness in the second image 232.
To increase the combination ratio of the second image 232, and conversely, for the pixels having a pixel value with low effectiveness in the second image 232, decrease the weighting coefficient WT to reduce the combination ratio of the first image 231. I try to raise it.

Specifically, the weighting coefficient WT is as shown in FIG.
Is determined based on the relational expression indicating the relation between the pixel value of the second image 232 and the weighting coefficient WT. That is,
The pixel value of the pixel of interest of the second image 232 has a predetermined threshold value V
When it is less than 1, the weighting coefficient WT of the target pixel is determined to be 1. When the pixel value of the target pixel of the second image 232 exceeds a predetermined threshold value V2 (V2> V1), the weighting coefficient of the target pixel is set. WT is determined to be 0. Also, the second image 2
When the pixel value of the target pixel of 32 is not less than the threshold value V1 and not more than the threshold value V2, the weighting coefficient WT of the target pixel is determined by the following formula 5 with the pixel value of P.

[0055]

[Equation 5]

When the pixel value is in the range of 0 to 255, the threshold value V1 is, for example, 100 to 2
A value within the range of 00 is preferable, and the threshold value V2
Is preferably set to a value within the range of 150 to 250, for example. However, threshold value V1 <threshold value V2.

As described above, since the first image 231 and the second image 232 are combined to generate the combined image 241 at the combining ratio according to the effectiveness of the pixel value of each pixel, the first image before the combination is generated.
Effective color information in the image 231 and the second image 232 can be appropriately reflected in the composite image 241.

Next, the brightness of each pixel of the generated composite image 241 is adjusted by the local contrast correction section 205 (step ST21). As mentioned above, the second
Since the image 232 is used for composition after the brightness level is lowered, the area of the composite image 241 including the pixel value of the second image 232 becomes too dark.
The composite image 241 is an image having a brightness that is unnatural to human senses. For this reason, the brightness of such an area is increased, and a synthetic image 241 having a natural brightness is obtained.

Specifically, the local contrast correction unit 20
5, the first image 231 and the second image from the exposure control unit 201.
The differential exposure value ΔEv is input as a value indicating the difference in brightness of each image 232, and the weighting coefficient WT is input from the weight setting unit 207 as a value indicating the composition ratio of each pixel. The local contrast correction unit 205 uses the input differential exposure value ΔEv, the weighting coefficient WT, and the preset light-dark adaptation coefficient α (0 ≦ α ≦ 1) to perform the calculation shown in Formula 6 below for all pixels. By doing so, the brightness of the composite image 241 is adjusted. In addition, in Formula 6, X
c, Yc, and Zc are composite images 24 before brightness adjustment, respectively.
1 is the XYZ value of one pixel, and Xa, Ya, and Za are the XY of that pixel in the composite image 241 after the brightness adjustment.
Z value.

[0060]

[Equation 6]

Here, since the weighting coefficient WT is 0 for the areas of the composite image 241 in which the pixel value of the second image 232 is not included, the brightness of these areas is not adjusted. Therefore, in the composite image 241, the second
The brightness of only the area of the image 232 including the pixel value is increased according to the combination ratio. That is, since this process corresponds to locally correcting the contrast of the composite image 241, this process is also referred to as “local contrast correction” below.

In the local contrast correction, the light-dark adaptation coefficient α corresponds to the degree of adjustment when adjusting the brightness of each pixel, and its value is required to obtain the composite image 241 as a human-natural image. It is preferable to set the value within the range of 0.2 to 0.8. As the value of the light-dark adaptation coefficient α, a predetermined value may be stored in advance in the ROM 22 or the RAM 23, but in the present embodiment, the light-dark adaptation coefficient α is used as a coefficient for reflecting the user's preference. That is, before the image pickup in the gradation adjustment mode, the light-dark adaptation coefficient α
The value of is previously set by the user via the operation button 126 and is stored in the RAM 23, and the local contrast correction is performed using the value of the light-dark adaptation coefficient α. This makes it possible to generate a desired composite image 241 that reflects the user's preference.

When the local contrast correction is applied to the composite image 241, the RGB conversion unit 206 then
The pixel value represented by the XYZ value of each pixel of the composite image 241 is converted into the RGB value again by a predetermined matrix calculation (step ST22). As a matrix used when converting the pixel values of the composite image 241 into RGB values,
The inverse matrix of the matrix used in step ST15 may be used, or a matrix that converts RGB values of a predetermined RGB color system (for example, an RGB color system conforming to sRGB) may be used. Good.

Thereafter, the composite image 241 obtained as described above is stored in the external memory 123 (step ST23). The obtained composite image 241 becomes an image in which both the region corresponding to the high-intensity part and the region corresponding to the low-intensity part of the subject have appropriate brightness, and the dynamic range is substantially expanded. First of
In the image 231 and the second image 232, the image accurately reflects the color information of effective pixels.

The first embodiment of the present invention has been described above. However, when the digital camera 1 generates a composite image (gradation adjusted image) whose gradation is adjusted, the pixel value of each pixel is Is converted into an XYZ value in a device-independent XYZ color system, so that the color information of each pixel is expressed as an accurate value that does not depend on the device, and the color information is not distorted in the process of generating a composite image. . Further, since the pixel value is expressed by a real number XYZ value and is calculated as a real number, the fraction generated in the calculation is not rounded and is accurately calculated. For these reasons, there is no loss of color information in the process of generating a composite image, and it is possible to obtain a composite image that accurately reflects the color information of the first and second images before composition.

In the digital camera 1, the weighting coefficient WT is determined by judging the validity of the pixel value for each pixel, and the weighting coefficient WT is used to generate the composite image. And, the color information that is effective in the second image can be appropriately reflected in the combined image.

Further, the brightness of each pixel of the composite image is calculated based on the differential exposure value ΔEv which is the difference in brightness between the first image and the second image before the composition and the weighting coefficient WT indicating the composition ratio of each pixel. Therefore, it is possible to adjust the brightness appropriately for each pixel, and it is possible to obtain a natural image as a composite image.

<2. Second Embodiment> Next, a second embodiment of the present invention will be described.

In general, in an image obtained by a digital camera or the like, a phenomenon (so-called color cast) in which the entire image is biased to a color corresponding to the illumination light due to the influence of the spectral characteristic of the illumination light that illuminates the subject. Occur. Therefore, in a general digital camera, such influence is corrected by performing white balance correction or the like. In such white balance correction, processing is performed assuming that all the subjects in the image are illuminated by the same illumination light. Therefore, when a subject such as the subject shown in FIG. 7 that receives a plurality of illumination lights (in the figure, sunlight and indoor light) is photographed,
Appropriate white balance correction cannot be applied.

Therefore, in the digital camera 1 which is the image processing apparatus of the present embodiment, a composite image is generated in consideration of the influence of such spectral characteristics of the plurality of illumination lights. The configuration of the digital camera 1 according to the present embodiment is shown in FIGS.
Since it is similar to that shown in FIG. 1, detailed description thereof will be omitted, and the operation of the digital camera 1 will be mainly described.

FIG. 11 mainly shows the CPU 21, ROM 22 and RAM 23 of the digital camera 1 according to the present embodiment.
FIG. 10 is a block diagram showing a configuration related to a gradation adjustment mode among the functions realized by, together with other configurations. In the configuration shown in FIG. 11, the object color component data generation unit 208 and the reproduction image generation unit 209 include a CPU 21, a ROM 22, and a RAM.
23 shows functions realized by 23 and the like. Although omitted in FIG. 11, the digital camera 1 includes the XYZ conversion unit 202, the brightness level adjustment unit 203, the image composition unit 204, the local contrast correction unit 205, and the RGB shown in FIG.
The conversion unit 206 and the weight setting unit 207 are similarly provided. 12 and 13 are diagrams showing the flow of shooting and image processing in the gradation adjustment mode. The operation of the digital camera 1 in the gradation adjustment mode will be described below with reference to these drawings.

First, similarly to steps ST11 and ST12, the exposure value adjusted to the high-luminance portion of the subject is the first exposure value Ev1, and the exposure value adjusted to the low-luminance portion of the subject is the second exposure value Ev1. The value Ev2 is stored in the RAM 23 (steps ST31 and ST).
32).

Next, when the shutter button 122 is operated, the digital camera 1 responds to this by operating the first exposure value E.
An image of the subject (hereinafter, referred to as “third image”) is obtained by taking an image under the exposure condition based on v1 and the flash is on (step ST33). That is, the image of the subject exposed to the flash light is obtained as the third image 233 and stored in the RAM 23 under the exposure condition in which the high-luminance portion of the subject is properly exposed.

Next, photographing is performed under the exposure condition based on the first exposure value Ev1 and the flash is OFF to obtain an image of the subject (hereinafter referred to as "fourth image") (step ST34). That is, an image of the subject in an illumination environment that does not have flash light is obtained as the fourth image 234 under the exposure condition in which the high-luminance portion of the subject is properly exposed.
It is stored in AM23.

Then, an image of the subject exposed to the flash light (hereinafter referred to as "fifth image") is obtained by photographing under the exposure condition based on the second exposure value Ev2 and the flash is on (step ST35). ). That is, an image of the subject exposed to the flash light is obtained as the fifth image 235 and stored in the RAM 23 under the exposure condition in which the low-luminance portion of the subject is properly exposed.

Further, an image of the subject (hereinafter referred to as "sixth image") is obtained by photographing under the exposure condition based on the second exposure value Ev2 and the flash is OFF (step ST36). That is, an image of the subject in an illumination environment that does not have flash light is obtained as the sixth image 236 under an exposure condition in which the low-luminance portion of the subject is properly exposed,
It is stored in the RAM 23.

These four times of photographing are performed quickly like continuous shooting. Therefore, the third image 233 and the fourth image 2
34, the fifth image 235, and the sixth image 236 have the same shooting range.

Here, in the light emission of the flash 121 in the steps ST33 and ST35, the light emission control circuit 121 is controlled so that the voltage and the light emission time are constant.
The light emission characteristics of the flash 121 are controlled by a and do not vary from one shooting to another. The spectral distribution of the flash 121 is also kept constant by this light emission control, this spectral distribution is measured in advance, and the flash spectral data 24 is stored in the RAM 23.
It is stored as 3. To be precise, the relative spectral distribution of the flash light (referred to as a spectral distribution normalized with the maximum spectral intensity being 1, hereinafter referred to as “relative spectral distribution”) is used as the flash spectral data 243.

When four images are stored in the RAM 23 by four times of photographing, the object color component data generation unit 208 removes the influence of the spectral characteristic of the illumination environment from the third image 233 and the fourth image 234. Object color component data (hereinafter, referred to as “first object color component data”) 237 corresponding to the data in FIG. 13 (FIG. 13: step ST3).
7). Further, the object color component data generation unit 208 extracts object color component data (hereinafter, referred to as “second object color component data”) 238 from the fifth image 235 and the sixth image 236.
Is required (step ST38).

FIG. 14 shows the object color component data generator 208.
FIG. 7 is a diagram showing a flow of processing (steps ST37, ST38) when obtaining the first and second object color component data 237, 238. Hereinafter, referring to FIG. 14, the third image 23
The first object color component data 23 from the third and fourth images 234
7 will be described. The flow of processing when determining the second object color component data 238 from the fifth image 235 and the sixth image 236 will be described below with reference to the third image 233 and the fifth image 233. The image 235 has a fourth image 234.
Are almost equivalent to the sixth image 236, respectively, so only the different points will be described as appropriate.

First, the third image 233 to the fourth image 234.
Is subtracted to obtain a difference image. As a result, the third image 2
The R, G, B values of the corresponding pixel of the fourth image 234 are subtracted from the R, G, B values of the 33 pixel,
A difference image between the third image 233 and the fourth image 234 is obtained (step ST101).

Next, one pixel in the image (that is, the third pixel
The position of one pixel in the image 233, the fourth image 234, and the difference image is determined as the target pixel (the position thereof) (step ST102), and whether or not the pixel value of the target pixel in each image satisfies the predetermined condition. It is determined (step ST103). If the condition is satisfied, the spectral reflectance of the minute area on the subject corresponding to the pixel of interest is obtained as temporary object color component data for one pixel (step ST1).
04).

Further, the position of the pixel of interest for which the temporary object color component data is obtained is stored in a predetermined storage location as the position (for example, xy coordinate) of the pixel to be calculated (step ST).
105).

Step ST103 for one pixel of interest
When ST105 is executed, steps ST102 to ST105 are repeated while changing (the position of) the target pixel, and the calculation of the temporary object color component data and the storage of the position of the calculation target pixel are performed for all the pixels according to the conditions. (Step ST106).

When the temporary object color component data for each pixel to be calculated is obtained, the fourth image 234 for each pixel to be calculated is obtained.
The illumination component data regarding the calculation target pixel is obtained from the pixel value of and the temporary object color component data. Then, the average value of the illumination component data for each calculation target pixel is obtained as the illumination component data for all the calculation target pixels (step ST107). The illumination component data is data that substantially corresponds to the spectral distribution of the illumination light on the subject.

Next, the specification of the calculation target pixel (correctly, the position of the calculation target pixel) in step ST103 will be described. The specification of the calculation target pixel corresponds to a process of removing inappropriate pixels and noise components in the image when obtaining the illumination component data. That is, based on the third image 233, the fourth image 234, and the difference image, the pixels having an inappropriate value for calculating the illumination component data are removed from the calculation target.

Specifically, when any of the color components of the pixel value of the target pixel is almost saturated, it is excluded from the calculation target pixels. For example, the values of R, G, and B color components are 0 to 255.
If it is within the range of, the value of any color component is 2
If it is 50 or more, it is determined that the pixel is not treated as a calculation target pixel.

If any of the color components of the pixel value of the target pixel is crushed in black, it is excluded from the calculation target pixels. For example, if the R, G, and B color component values are in the range of 0 to 255, and any one of the color component values is 4 or less, the pixel is not treated as a calculation target pixel.

Since the difference image substantially corresponds to the image when the image is taken only with the flash light, a dark portion such as the background or a pixel which is too bright by reflecting the flash light is out of the calculation target pixel. Removed. That is, even in the difference image, if each color component of the pixel value is too low or too high, it is not treated as a calculation target pixel.

By performing the above three determinations in step ST103, the third image 233 and the fourth image 23
In both 4 and the difference image, the position of the pixel having an appropriate pixel value is obtained as the position of the calculation target pixel.

Since the third image 233 and the fourth image 234 are images acquired under the exposure condition in which the high-intensity part of the subject is properly exposed, the pixels in the region corresponding to the low-intensity part of the subject are blacked out. There is a possibility that According to the above determination, the pixels in such an area are not treated as the calculation target pixels, and therefore the pixels in the area corresponding to the comparatively high brightness portion of the subject are mainly the calculation target pixels.

The fifth image 235 and the sixth image 23
When determining the second object color component data 238 from No. 6, since the fifth image 235 and the sixth image 236 are images acquired under the exposure condition in which the low-luminance portion of the subject is properly exposed, It is possible that any of the color components of the pixels in the area corresponding to the luminance portion is saturated. Pixels in such a region are not treated as calculation target pixels by the above determination, and thus pixels in a region corresponding to a relatively low brightness portion of the subject are mainly used as calculation target pixels.

Next, in the calculation target pixel, the fourth image 23
4 is obtained by removing the effect of the spectral characteristics of the illumination environment (substantially equivalent to the spectral reflectance of the subject) as temporary object color component data, and then the illumination component data of the calculation target pixel is obtained. Will be described.

First, the spectral distribution of the illumination light that illuminates the subject (the illumination light in the illumination environment that includes direct light and indirect light from the light source) is E (λ), and this spectral distribution E ( λ) to three basis functions E ₁ (λ), E ₂ (λ), E ₃ (λ)
And the weighting factors ε ₁ , ε ₂ , ε ₃ ,

[0095]

[Equation 7]

Similarly, the spectral reflectance S (λ) at the position on the subject corresponding to one calculation target pixel (hereinafter referred to as “target pixel”) is represented by three basis functions S ₁ (λ), S
_{Using 2} (λ), S ₃ (λ) and weighting factors σ ₁ , σ ₂ , σ ₃ ,

[0097]

[Equation 8]

When expressed, the light I (λ) incident on the target pixel on the CCD 112 (incident light when the filter in the lens unit 11 is ignored) is

[0099]

[Equation 9]

It is expressed as follows. In addition, R, G, and
Assuming that the value relating to one of the colors of B (hereinafter referred to as “target color”) is ρ _c and the spectral sensitivity of the target color of the CCD 112 is R _c (λ), the value ρ _c is

[0101]

[Equation 10]

It is derived by

Here, the third image 233 of flash ON
Value of the target color of the target pixel is ρ _c1 , and the flash OF
If the corresponding value of the fourth image 234 of F is ρ _c2 , the corresponding value of the difference image ρ _s is

[0104]

[Equation 11]

It becomes: I ₁ (λ) is the light incident on the target pixel when the flash is ON, and ε ₁₁ , ε ₁₂ , and ε ₁₃ are weighting coefficients of the basis function for the illumination light including the flash light. Similarly, I ₂ (λ) is the light that is incident on the target pixel when the flash is off, and ε ₂₁ , ε ₂₂ , and ε ₂₃ are the basis function weighting coefficients for the illumination light that does not include the flash light. Furthermore, ε _si (i = 1, 2, 3) is (ε _1i −ε _2i ).
Is.

In equation 11, the basis functions E _i (λ), S
_j (λ) is a predetermined function, and the spectral sensitivity R _c (λ) is a function that can be obtained in advance by measurement. These pieces of information are stored in the ROM 22 and the RAM 23 in advance. On the other hand, the exposure conditions are controlled to be the same in the shooting of the third image 233 and the fourth image 234, and the third image 233 to the fourth image
Since the difference image obtained by subtracting the image 234 corresponds to the image in which only the flash light is used as the illumination light source, the weighting coefficient ε _si can be derived from the relative spectral distribution of the flash light by the method described later.

Therefore, in the equation shown in the equation 11, only _three weighting coefficients σ ₁ , σ ₂ and σ ₃ are unknowns. Further, the equation shown in the equation 11 is R, G, B in the target pixel.
Can be obtained for each of the three colors of, and by solving these three equations, the three weighting factors σ ₁ ,
σ ₂ and σ ₃ can be obtained. That is, the spectral reflectance at the position on the subject corresponding to the target pixel is obtained.

Next, a method for _obtaining the weighting coefficient ε _si will be described. As described above, the difference image corresponds to an image in which only the flash light is used as the illumination light, and the relative spectral distribution of the illumination light in the difference image is known. On the other hand, the area on the subject far from the flash receives the flash light less than the area near the flash 121. Therefore, in the difference image, the position farther from the flash 121 appears darker.

Therefore, the three weighting factors ε_s1, Ε_s2, Ε_s3
The target image in the difference image while keeping the relative relationship of
The value of these weighting factors is increased or decreased in proportion to the brightness of the element.
That is, when the brightness of the target pixel in the difference image is small
Is the weighting coefficient ε_s1, Ε_s2, Ε_s3Value is determined as a small value
If the brightness is large, the weighting coefficient ε_s1, Ε_s2, Ε _s3
The value of is determined as a large value. 3 weighting factors
ε_s1, Ε_s2, Ε_s3Is based on three basis functions E
₁(λ), E₂(λ), E₃The weighted sum of (λ) is the flash light
Pre-determined to be proportional to the light distribution, it
Weighting coefficient ε_siThe proportional relationship with and is previously obtained by measurement.

The weighting coefficient ε _si is a value indicating the spectral distribution of the flash light emitted to the position on the subject corresponding to the target pixel, and is the illumination light of the flash 121 between the third image 233 and the fourth image 234. Is a value indicating the spectral distribution of the change amount of. Therefore, the process of _{obtaining the} weighting coefficient ε _si from the flash spectral data 243 corresponds to the process of obtaining the spectral change amount of the illumination environment (illumination light) by the flash 121 from the relative spectral distribution of the flash light.

Based on the above principle, the object color component data generation unit 208 of the digital camera 1 refers to the pixel value of the calculation target pixel of the difference image and the flash spectral data 243 and refers to the object corresponding to each calculation target pixel. Determine the spectral reflectance of the position. The spectral reflectance of the subject corresponds to the image data from which the influence of the spectral characteristics of the illumination environment has been removed, and is stored in the RAM 23 as temporary object color component data (step ST104).

When the temporary object color component data is obtained,
And number 10 (where substitutes epsilon _2i number 9 of epsilon _i.) From the calculation target pixels of the fourth image 234 R, G, B
Based on the value of, the three equations for the weighting factors ε ₂₁ , ε ₂₂ , ε ₂₃ can be obtained. By solving these equations, the weighting coefficient ε _2i for each calculation target pixel in the image is obtained. The obtained weighting coefficient ε _2i becomes a component indicating the influence of the spectral characteristic of the illumination environment that does not include the flash light in the calculation target pixel.

Here, when the illumination environment of the illumination light is approximately uniform, there is little variation in the weighting coefficient ε _2i among the calculation target pixels. Therefore, for each of the weighting factors ε ₂₁ , ε ₂₂ , and ε ₂₃ , the average values ε _21a , ε _22a , and ε of all the calculation target pixels are set.
_23a is obtained, and the obtained three weighting factors are used as illumination component data for all the calculation target pixels (step ST1).
07).

Since the calculation target pixel is mainly a pixel in a region corresponding to a relatively high-brightness portion of the subject, the obtained illumination component data is the spectrum of the illumination light that illuminates the relatively high-brightness portion of the subject. The data substantially corresponds to the distribution.

The fifth image 235 and the sixth image 23
When calculating the second object color component data 238 from No. 6, since the calculation target pixel is mainly the pixel of the region corresponding to the relatively low luminance part of the subject, the obtained illumination component data is compared with the subject. The data is substantially equivalent to the spectral distribution of the illumination light that illuminates the portion of low luminance.

When the illumination component data is obtained, the object color component data of each pixel is obtained using the illumination component data and the fourth image 234 (step ST108). Specifically, regarding each of the R, G, and B color components,
The unknowns σ _j are obtained as object color component data.

[0117]

[Equation 12]

Thereafter, the obtained object color component data is stored in the RAM 23 as the first object color component data 237. The first object color component data 237 is data that substantially corresponds to the spectral reflectance of the subject, but is based on illumination component data that corresponds to the spectral distribution of the illumination light that illuminates a relatively high-luminance portion of the subject. Since it is obtained by the above, it is data showing the spectral reflectance of a relatively high-luminance portion of the subject with relatively high accuracy.

The fifth image 235 and the sixth image 23
When obtaining the second object color component data 238 from 6, the obtained object color component data is stored in the RAM 23 as the second object color component data 238. Since the second object color component data 238 is obtained based on the illumination component data corresponding to the spectral distribution of the illumination light that illuminates the relatively low-luminance portion of the subject, the relatively low-luminance of the subject is obtained. This is data that indicates the spectral reflectance of the portion with relatively high accuracy.

As described above, the first object color component data 2
37 and the second object color component data 238 are obtained, next, the obtained first object color component data 237 and
Light source (eg "CIE D65", "CIE D50"
A standard light source such as The reproduction image generation unit 209 performs the calculations shown in Expressions 9 and 10 using the illumination component data (data substantially corresponding to the spectral distribution) 244 indicating the influence of the spectral characteristics of (1) on the image. As a result, the first object color component data 237 and the illumination component data 2
44 and are combined to obtain R, G, B values ρ _r , ρ _{g of} each pixel,
ρ _b is calculated, and an image of the subject (hereinafter, referred to as “first reproduction image”) 239 is generated. The generated first reproduction image 239 is stored in the RAM 23. The illumination component data 244 is prepared in advance and stored in the ROM 22 and RA.
It is stored in M23.

The first reproduced image 239 is an image from which the color cast due to the illumination light for illuminating the relatively high-intensity part of the subject is removed, and the color of the relatively high-intensity part of the subject is accurately reproduced. (Fig. 13: Step ST3
9).

Next, the second object color component data 238 and the illumination component data 244 are combined by the reproduction image generation unit 209 to obtain the R, G, B values ρ _r , ρ _g , ρ _{b of} each pixel. The image of the subject (hereinafter, referred to as “second reproduction image”) 240 is obtained. Generated second reproduction image 2
40 is stored in the RAM 23.

The second reproduced image 240 is an image from which the color cast due to the illumination light that illuminates the relatively low luminance part of the subject has been removed, and the color of the relatively low luminance part of the subject is accurately reproduced. (Step ST40).

The first reproduced image 23 thus obtained
9 and the second reproduced image 240 are images corresponding to the first image 231 and the second image 232 of the first embodiment, respectively, and thereafter, the same processing as the processing after step ST15 in FIG. 5 is performed. . As a result, the gradation is adjusted from the first reproduction image 239 and the second reproduction image 240, and a composite image having a substantially expanded dynamic range is generated.

This combined image is the same as the first embodiment, that is, the first reproduced image 239 and the second image 240 before combination.
In this case, the image accurately reflects the effective pixel color information. Here, the effective pixels in the first reproduction image 239 are those in a region corresponding to a relatively high-luminance portion of the subject, and the influence of the spectral characteristics of the illumination environment is appropriately removed in this region. The color of the subject is accurately reproduced. On the other hand, the effective pixels in the second reproduction image 240 are in the area corresponding to the relatively low brightness portion of the subject, and the effect of the spectral characteristics of the illumination environment is appropriately removed in this area. Is accurately reproduced.
Therefore, the generated composite image appropriately removes the influence of the spectral characteristics of both the illumination environment that illuminates the relatively high-intensity part of the subject and the illumination environment that illuminates the relatively low-intensity part of the subject. It can be

<3. Third Embodiment> In the first and second embodiments described above, the image processing is performed inside the digital camera, but it is of course possible to perform the image processing by a computer. . Figure 15
FIG. 3 is a diagram showing a configuration of the image data acquisition system 3 in such a case.

The image data acquisition system 3 processes the image acquired by the CCD as it is in the external memory and the digital camera 31 and the image saved in the external memory to generate a gradation-adjusted composite image. Computer 32
Composed of and.

As the digital camera 31, one having the same configuration as that of the above-described embodiment shown in FIGS. 1 to 3 can be used. Further, the computer 32 is a CPU, R
It is composed of a general-purpose computer including an OM, a RAM, a fixed disk, a display, and the like.

Such a configuration can be used for any operation of the first and second embodiments.

For example, when the image data acquisition system 3 shown in FIG. 15 is used for the processing of the first embodiment, the digital camera 31 obtains the first exposure condition under which the high-intensity part of the subject is properly exposed. One image and a second image obtained under an exposure condition in which the low-luminance portion of the subject is properly exposed,
The differential exposure value is stored in the external memory 123, and these data are transferred to the computer 32 via the external memory 123.

The CPU, ROM, and the like inside the computer 32
The RAM or the like includes an XYZ conversion unit 202, a brightness level adjustment unit 203, an image composition unit 204, a local contrast correction unit 205, an RGB conversion unit 206, and a weight setting unit 20 shown in FIG.
7 and generates a composite image in which the gradation is adjusted from the first image, the second image, and the differential exposure value.

When the image data acquisition system 3 is used for the processing of the second embodiment, the digital camera 31 has four images (third image) obtained by changing the exposure condition and flash on / off. , Fourth image, fifth image, and sixth image), the differential exposure value, and the flash spectral data are stored in the external memory 123, and these data are transferred to the computer 32.

CPU, ROM in the computer 32,
The RAM or the like includes an XYZ conversion unit 202, a brightness level adjustment unit 203, an image composition unit 204, a local contrast correction unit 205, an RGB conversion unit 206, and a weight setting unit 20 shown in FIG.
7, and also functions as the object color component data generation unit 208 and the reproduction image generation unit 209 shown in FIG. Then, four images (third image, fourth image, fifth image
Image and sixth image), the differential exposure value, and the flash spectral data to generate a first reproduced image and a second reproduced image, and further, a gradation is obtained from the first reproduced image, the second reproduced image, and the differential exposure value. Generate an adjusted composite image.

The computer 32 has an XYZ conversion section 202, a brightness level adjustment section 203, and an image composition section 20 shown in FIG.
4, local contrast correction unit 205, RGB conversion unit 20
6 and the weight setting unit 207, or further, the object color component data generation unit 2 shown in FIG.
The program is installed in the computer 32 in advance via the recording medium 9 such as a magnetic disk, an optical disk, or a magneto-optical disk in order to function as the 08 and the reproduction image generation unit 209. This allows the general-purpose computer 32 to be used as a computer that performs the image processing according to the present invention.

As described above, the digital camera according to the first or second embodiment can be constructed as the image data acquisition system 3 including the digital camera 31 and the computer 32. In this case, the digital camera can be used. The amount of processing that the camera 31 should perform can be reduced.

<4. Modifications> Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various modifications can be made.

For example, in the above-described embodiment, the exposure condition is changed based on the first exposure value and the second exposure value that match the high-luminance portion and the low-luminance portion of the subject, respectively. Although the second image is photographed, the proper exposure value for the entire subject is calculated in the normal photometry mode, and the exposure condition is set to the over side or the under side by a predetermined number of steps to set the first image and The second image may be captured.

Further, in the above-described embodiment, the spot metering mode is set and the exposure value is obtained by adjusting the central portion of the screen of the finder 113 to the subject to be metered. 1 may be fixed, and a subject to be metered in the screen of the finder 113 may be designated by a cursor or the like.

Further, in the above-mentioned embodiment, the description has been made assuming that the synthetic image is generated as it is after the image of the subject is photographed, but the synthetic image is generated by the user at an arbitrary timing. May be.

Further, in the third embodiment,
The digital camera 31 only acquires an image, and the image processing related to generation of a composite image is described as being performed by the computer 32.
The digital camera 31 may perform a part of the processing performed by the digital camera 31, and only the remaining processing may be performed by the computer 32.

Further, in the above embodiment, the weighting coefficient of each pixel is determined based on the effectiveness of the pixel value of the second image. However, based on the effectiveness of the pixel value of the first image, The weighting coefficient of each pixel may be determined, or the weighting coefficient of each pixel may be determined by comprehensively determining the effectiveness of pixel values of both images to be combined.

In the above embodiment, each pixel value is set to X.
Although the description has been made assuming that the values are converted into the YZ color system values,
For example, any color system that is device-independent, such as the Lab color system, may be used.

Further, the synthesizing method for generating the synthesized image of which the gradation is adjusted is not limited to the one described above, and another method may be applied.

Further, in the above-described embodiment, various functions are realized by the CPU performing calculations according to programs, but all or part of the calculation processing may be realized by a dedicated electric circuit. Good. In particular, a high-speed operation can be realized by constructing a place where a repeated operation is performed with a logic circuit.

[0145]

As described above, according to the invention of claim 1, since the color component value of each pixel is converted into the color component value in the device-independent color system, the color of each pixel is changed. Since the information is represented as an accurate value that does not depend on the device, a combined image that accurately reflects the color information of the first and second images before combining can be obtained.

According to the second aspect of the present invention, an image of a subject that has received a plurality of illumination lights is combined so that the influence of the spectral characteristics of the illumination environment is appropriately removed and the gradation is appropriately adjusted. It can be acquired as an image. In addition, since the color component value of each pixel is converted to the color component value in the device-independent color system, the color information of each pixel is represented as an accurate value that does not depend on the device. A composite image that accurately reflects the color information of the first and second images can be obtained.

According to the third aspect of the present invention, since the first image and the second image are combined at a combination ratio according to the validity of the color component value of each pixel to generate a combined image, the pre-combination Color information effective in the first image and the second image can be appropriately reflected in the combined image.

According to the invention of claim 4, the brightness of each pixel of the composite image is adjusted based on the brightness of each image before composition and the composition ratio of each pixel.
Appropriate brightness adjustment can be performed for each pixel.
This makes it possible to obtain an image with natural brightness as the entire composite image.

According to the fifth aspect of the present invention, since the degree of adjustment when adjusting the brightness of each pixel of the composite image can be specified, a composite image having a desired brightness can be generated. .

[Brief description of drawings]

FIG. 1 is a perspective view showing an entire digital camera according to a first embodiment.

FIG. 2 is a rear view of the digital camera.

FIG. 3 is a block diagram showing a main configuration of a digital camera.

FIG. 4 is a block diagram showing a functional configuration related to a gradation adjustment mode of the digital camera.

FIG. 5 is a diagram showing a flow of operations in a gradation adjustment mode of the digital camera of the first embodiment.

FIG. 6 is a diagram showing a flow of operations in a gradation adjustment mode of the digital camera of the first embodiment.

FIG. 7 is a diagram showing a specific example of a subject having a high-luminance portion and a low-luminance portion.

FIG. 8 is a diagram showing an example of a first image.

FIG. 9 is a diagram showing an example of a second image.

FIG. 10 is a diagram showing a relationship between a pixel value of a second image and a weighting coefficient.

FIG. 11 is a block diagram showing a part of a functional configuration relating to a gradation adjustment mode of the digital camera of the second embodiment.

FIG. 12 is a diagram showing an operation flow in a gradation adjustment mode of the digital camera of the second embodiment.

FIG. 13 is a diagram showing a flow of operations in a gradation adjustment mode of the digital camera of the second embodiment.

FIG. 14 is a diagram showing a flow of processing when obtaining object color component data.

FIG. 15 is a diagram showing a configuration of an image data acquisition system according to a third embodiment.

[Explanation of symbols]

1 digital camera 3 Image data acquisition system 9 recording media 201 Exposure control unit 202 XYZ converter 203 Brightness level adjustment unit 204 image composition section 205 Local Contrast Correction Unit 208 Object Color Component Data Generation Unit 221 program

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) H04N 1/60 H04N 5/235 5/232 9/04 B 5/235 101: 00 9/04 1/40 D // H04N 101: 00 1/46 Z (72) Inventor Shigeki Nakauchi 2-33 Azuchi-cho, Chuo-ku, Osaka-shi, Osaka Osaka International Building Minolta Co., Ltd. F-term (reference) 5B057 BA02 CA01 CA08 CA12 CA16 CB01 CB08 CB12 CB16 CC01 CE08 CE17 CH07 CH08 CH11 CH12 DB02 DB06 DB09 5C022 AA13 AB15 AB17 AB68 AC52 AC69 5C065 AA03 BB48 DD19 EE18 FF02 GG26 5C077 LL19 MM19P23B01Q22 P06 PP23 PQ PPQ PP23 PP02 PP06 PP23 PQ LA40 MA01 MA04 MA11 NA05 PA00

Claims (5)

[Claims] 1. An image processing apparatus, comprising: a first object obtained by changing exposure conditions for the same subject.
A color system conversion unit that converts the color component value of each pixel of the image and the second image into a color component value in a device-independent color system, and the color component value of each pixel by the color system conversion unit. An image processing apparatus, comprising: an image synthesizing unit that synthesizes the converted first image and the second image to generate a synthesized image whose gradation is adjusted. 2. An image processing apparatus, wherein an image obtained by removing an influence of a spectral characteristic of an illumination environment on an object from an image group acquired under an exposure condition in which a relatively high-intensity part of the object is properly exposed. Corresponding to the first data
A means for obtaining object color component data, and image data obtained by removing the influence of the spectral characteristics of the illumination environment on the subject from the image group acquired under the exposure condition in which the relatively low-brightness portion of the subject is properly exposed. Means for obtaining the second object color component data corresponding to the above, each of the first and second object color component data, and the illumination component data indicating the influence of the spectral characteristics of the light source on the image, And a unit for generating a second image, and a color system conversion unit for converting the color component value of each pixel of the first image and the second image into a color component value in a device-independent color system. An image synthesizing unit for synthesizing the first image and the second image in which the color component values of the respective pixels are converted by the color system conversion unit to generate a composite image in which gradation is adjusted. Characterized by Image processing device. 3. The image processing apparatus according to claim 1, wherein for each pixel of the first image and the second image, a combination ratio is set according to the validity of the color component value of the pixel. The image synthesizing unit synthesizes the color component values of the respective pixels of the first image and the second image after the color system conversion at the synthesizing ratio to generate the synthetic image. An image processing device characterized by: 4. The image processing apparatus according to claim 1, wherein the brightness of each of the first image and the second image and each pixel of the first image and the second image are combined. An image processing apparatus, further comprising brightness adjusting means for adjusting the brightness of each pixel of the composite image based on the ratio. 5. The image processing apparatus according to claim 4, further comprising: a receiving unit that receives a designation of an adjustment degree when the brightness adjusting unit adjusts the brightness of each pixel of the composite image. A characteristic image processing device.