JP2000050160A - Picture compositing device and its method, and storage medium storing picture composition method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To eliminate the sence of incongruity in scenery and a general picture after synthesizing plural photographing pictures different in an exposure amt. SOLUTION: A picture composition processing part 16 uses a reference picture as it is in a range where the luminance of the reference picture is up to the certain luminance of low luminance and adds the reference picture and a background picture by adding weight in a range where the luminance exceeds the certain luminance of low luminance and composites them so that an output picture signal can obtain a characteristic decided by a fixed rule at the time of compositing the reference picture picked up by exposing and matching it with a main object and the background picture which is picked up by the exposure amt. shorten than the reference picture and which is obtained when the exposure is matched with a bright part so as to cause void in the reference picture. At the time of compositing the reference picture and the background picture, the saturation of a picture element whose luminance is not less than a luminance threshold which is decided by a photographing condition at the time of photographing, is corrected and the drop of the saturation is prevented.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing apparatus for expanding a dynamic range, such as a digital video camera and a digital still camera, an image synthesizing method, and a storage medium storing the image synthesizing method.

[0002]

2. Description of the Related Art A dynamic range of a solid-state image pickup device such as a CCD image pickup device such as a digital video camera and a digital still camera is very narrow as compared with a dynamic range of an image photographed by a silver halide photographic system, and a severe backlight state or a window. Even if the exposure condition such as inside and outside is not an extreme special condition, even when taking a general landscape photograph or a portrait photograph, the dark part of the subject will be overexposed, and the bright part of the subject will be overexposed. Image quality is reduced.
As a method of expanding the dynamic range by using such a solid-state imaging device, Japanese Patent Application Laid-Open No. Sho 60-52171 discloses an optical filter that logarithmically changes the intensity of incident light and outputs a plurality of different light intensities. A plurality of images of the same subject are taken with different exposure amounts, and the image signals of the plurality of images are added to perform image synthesis so as to expand the dynamic range. For example, as shown in FIG. 7A, when the same subject as an image obtained by photographing a person 22 against the sky 20 and clouds 21 by a silver halide photography system is imaged with a CCD image sensor such as a digital camera while changing the exposure amount. On the other hand, in the reference image obtained when the person 22 is photographed with proper exposure, the sky 20 and the cloud 21 have a narrow dynamic range as shown in FIG. In the case of a background image in which a portion of the sky 20 is exposed, the person portion 22 becomes dark as shown in FIG. 7C, and a dark portion such as the ground 24 becomes black. So person 22
Is synthesized with a reference image obtained when the image is photographed with proper exposure and a background image in which a part of the sky 20 is exposed, and as shown in FIG. An image with an expanded dynamic range including an image and a high-luminance image is obtained.

[0003]

However, an image is taken with a CCD image pickup device such as a digital camera while changing the exposure amount.
Even if a plurality of captured images with different exposures are combined and corrected by adding, etc., the part close to the proper exposure state such as the main subject is not so much, but the brightness of the background part, especially the bright part such as the sky, after combining is corrected. As shown in FIG. 7 (d), the saturation decreases because the image is raised and combined, and as shown in FIG. 7D, the sky 20 becomes whitish and the boundary with the cloud 21 becomes unclear. Occasionally, something uncomfortable occurred, and the effect of the synthesis was not obtained much.

The present invention has been made to solve the above-mentioned disadvantages, and to maintain the saturation of a high-brightness photographed image before combining as much as possible with respect to a high-brightness portion in which the saturation decreases after combining a plurality of photographed images having different exposure amounts. It is an object of the present invention to provide an image synthesizing apparatus that can correct a saturation and eliminate a sense of discomfort in a landscape or a general image.

[0005]

An image synthesizing apparatus according to the present invention has an imaging means and an image synthesizing processing means, wherein the imaging means images the same subject with different exposure amounts, and the image synthesizing processing means This is to combine a plurality of images of the same subject imaged with different exposure amounts, and when combining multiple images taken with different exposure amounts, the The image processing apparatus is characterized in that saturation correction is performed on an image having a predetermined brightness or higher determined by an imaging condition and then synthesized.

According to an image synthesizing method according to the present invention, when synthesizing a plurality of images of the same subject with different exposure amounts, a predetermined luminance determined by an imaging condition of the image obtained by reducing the exposure amount is set. It is characterized in that the above images are subjected to saturation correction and synthesized.

[0007] When synthesizing a plurality of images, it is preferable to perform the saturation correction by changing the correction amount of the saturation correction depending on the luminance after the synthesis.

In addition, it is desirable to perform saturation correction only on pixels that fall within a specific specific hue range when combining the plurality of images.

In a recording medium storing the image synthesizing method of the present invention, when synthesizing a plurality of images obtained by imaging the same subject with different exposure amounts, the recording medium is set under an imaging condition of an image obtained by reducing the exposure amount. It is characterized in that a control program for performing saturation correction on an image having a brightness equal to or higher than a predetermined luminance to be synthesized is stored, and the control program is read by a computer.

[0010]

DESCRIPTION OF THE PREFERRED EMBODIMENTS A digital camera according to the present invention forms an image on a CCD by adjusting the amount of light from a subject incident on a photographic lens with a diaphragm. The CCD converts each pixel of the formed image into an image signal according to the amount of light, and sends the image signal to the signal processing unit at regular intervals. The signal processing unit removes noise components and the like of the transmitted image signal, performs gain control, converts the analog signal into a digital signal, and converts the signal into an RGB signal and a luminance signal. The image synthesizing unit synthesizes a plurality of images of the same subject when the same subject is imaged by the CCD with different exposure amounts. The image data processing unit performs image processing such as gamma correction and compression of the combined image signal combined by the image combining unit.

When the image synthesis processing unit shoots a reference image captured by exposure to the main subject and with an exposure shorter than the reference image, and adjusts the exposure to a bright portion that causes overexposure in the reference image. When composing the background image of
In the range where the luminance of the reference image reaches a certain level of low luminance, the reference image is used as it is, and in the range exceeding the certain luminance of low luminance, the reference image and the background image are weighted and added, and the output image signal has a certain fixed value. The composition is performed so that the characteristics defined by the law can be obtained. When the reference image and the background image are combined, saturation correction is performed on pixels whose brightness is equal to or higher than a brightness threshold value determined by shooting conditions at the time of shooting to prevent a decrease in saturation.

[0012]

FIG. 1 is a block diagram showing the configuration of an embodiment of the present invention. As shown in the figure, light from a subject incident on a photographic lens 1 of a digital camera is formed on a CCD 3 by adjusting the amount of light by an aperture 2. The image formed on the CCD 3 is converted into an image signal in accordance with the amount of light for each pixel, and is sequentially sent to a CDS (correlated double sampling circuit) 4 at regular time intervals. After the gain control is performed, the A / D converter 6 converts the analog signal into a digital signal. The image signal converted into a digital signal is sent to the digital signal processing unit 7 and converted into an RGB signal and a luminance signal. The photometric processing unit 8 inputs the image data converted into the luminance signal by the digital signal processing unit 7, calculates the exposure amount of the CCD 3, and sends it to the system control unit 9. The system control unit 9 controls the aperture diameter of the aperture 2 by operating the aperture control unit 10 based on the transmitted exposure amount, and changes the electronic shutter speed of the CCD 3 by the CCD driving unit 11.

The image signal processed by the digital signal processing section 7 is output via a switch 12. The switch 12 switches the output destination to the system control unit 9, and changes the exposure conditions to perform image synthesis, and the image signals for one image of the subject that is continuously shot are changed to the first memory 13 and the second memory 14, respectively.
Are sequentially stored. An image signal not subjected to image synthesis is sent from the switch 12 to the image data processing unit 15. The image signals stored in the first memory 13 and the second memory 14 of the same subject with different exposure conditions are read by the image synthesizing processing unit 16 in response to a control signal from the system driving unit 9 and image synthesizing processing is performed. The conversion of the luminance signal or the RGB signal performed by the digital signal processing unit 7 may be performed before the image synthesis processing unit 16 synthesizes the images. The synthesized image signal synthesized by the image synthesis processing unit 16 and the image signal sent directly from the switch 12 are subjected to image processing such as gamma correction and compression by the image data processing unit 15 and stored in the external memory 17.

As shown in the block diagram of FIG. 2, the image synthesizing processing unit 16 includes a luminance judging unit 161, a hue judging unit 162, a saturation correcting unit 163, a low luminance synthesizing conversion unit 164, a high luminance synthesizing conversion unit 165, It has a synthesis processing unit 166. The brightness determination unit 161 obtains the brightness value of the image data of the high-brightness image captured by reducing the exposure amount, and determines whether the obtained brightness value is greater than a brightness threshold value defined by the shooting conditions at the time of shooting. . The hue determination unit 162 calculates the hue of the image data whose luminance value is larger than the luminance threshold value, and determines whether or not the hue is within a predetermined specific hue range. The saturation correction unit 163 performs saturation correction processing on image data whose hue is within a specific hue range, and generates image data for saturation correction synthesis. The low-luminance synthesis conversion unit 164 performs low-luminance image conversion processing on the image data of the low-luminance image to generate low-luminance synthesis image data. The high-luminance combining / converting unit 165 outputs image data of a high-luminance image determined by the luminance determining unit 161 to have a luminance value smaller than the luminance threshold determined by the imaging conditions at the time of imaging, and a specific hue range by the hue determining unit 162. The high-brightness image conversion processing is performed on the image data of the high-brightness image determined not to be within the image data to generate high-brightness synthesis image data. The synthesis processing unit 166 performs a synthesis process on the image data for saturation correction synthesis or the image data for high luminance synthesis with the image data for low luminance synthesis, and sends the synthesized image data to the image data processing unit 15.

An example of processing when the image synthesizing unit 16 synthesizes image data of picked-up images of the same subject with different exposure amounts will be described with reference to a characteristic diagram showing luminance of an input image signal and an output image signal in FIG. Will be explained. Here, for the sake of simplicity, a case is described in which synthesis is performed using two captured images having different exposure amounts. However, the same applies to a case where synthesis is performed on a plurality of images.

In FIG. 3, a reference image A shows an output image signal corresponding to an input image signal when exposed to a main subject, and a background image B is shot with a shorter exposure than the reference image A, An output image signal corresponding to the input image signal when the exposure is adjusted to a bright portion that causes a jump is shown. A composite image C indicates the characteristics of the output image signal with respect to the input image signal of the image synthesized by the image synthesis processing unit 16. As shown in FIG. 3, for the same luminance b of the input image signal, the output signal of the reference image A is much larger than the background image B with a reduced exposure amount. When synthesizing the reference image A and the background image B, the reference image A is used as it is in a range where the input image signal has a low luminance up to the luminance a, and the reference image A is used in a range where the input image signal exceeds the luminance a. The background image B is weighted and added, so that the output image signal can obtain a characteristic defined by a certain rule. ,
k1) and point (b, k2) on a straight line,
When the input image signal is equal to or higher than the luminance b, the output image signal is converted to be on a straight line connecting the point (b, k2) and the point (c, k3) to obtain a composite image C. In this manner, the reference image A and the background image B are weighted and added to generate a composite image C. As shown in FIG. 3, the composite image C and the background image B output difference (k3
−k2), so that the saturation decreases. Therefore, saturation correction is performed on pixels having a luminance threshold s or more determined by the photographing conditions when the luminance is photographed in the input image signal to prevent a decrease in the saturation.

A method for correcting the saturation will be described. When the red, blue, and green components of a pixel are L, M, and S from the brighter one, the saturation Sa can be expressed by equation (1), and the luminance Y is the three primary colors red, red, and blue.
It can be expressed by equation (2) using green, blue components R, G, and B.

[0018]

(Equation 1)

Here, t in the equation (1) is a conversion constant for representing the saturation Sa, and becomes 255 when represented by 8 bits.
In addition, a, b, and c in Expression (2) indicate weights, and generally, a = 0.30, b = 0.59, and c = 0.11,
Here, it is assumed that a = b = c = 1/3.

To correct the luminance without changing the luminance value (brightness) before and after the correction of the saturation, the saturation may be corrected according to the equation (3).

[0021]

(Equation 2)

Here, Rc, Gc, and Bc represent red, green, and blue components after saturation correction, k represents a saturation correction amount, and k =
0 indicates a monochrome image, k = 1 indicates no saturation correction, k> 1 enhances saturation, and k <1 decreases saturation. Then, the saturation Sa0 of the background image at the same location is obtained for the pixels of the input image signal having a certain luminance or higher in the reference image A, and the background image B
Saturation Sa after synthesis when is synthesized without saturation correction
1 is calculated, and the saturation correction amount k = (Sa0 / Sa1) * P is calculated, and the saturation correction is performed by equation (3). Here, P is a value for controlling the saturation correction amount, and can be changed according to the luminance value after conversion, the saturation before and after conversion, or an external instruction.
In this way, the luminance value (lightness) before and after the saturation correction
Can be corrected without changing.

The operation of the image synthesizing processing section 16 when synthesizing image data of captured images of the same subject with different exposure amounts by this saturation processing will be described with reference to the flowchart of FIG.

For example, as shown in FIG. 3, a reference image A whose input image is a low luminance image and a background image B whose input image is a high luminance image are picked up, and image data (R0, G
0, B0) is stored in the first memory 13 and the image data (R1, G1, B1) of the high-brightness image is stored in the second memory 14, and the system control unit 9
When the image synthesis processing is instructed from the first memory 13 and the second memory 14, the luminance determining unit 161 outputs the image data (R0, G0, B0) of the low luminance image and the image data (R1, G1, B1) of the high luminance image. B1) (Step S)
1). When the input image data is image data (R1, G1, B1) of a high brightness image, the image data (R
(1, G1, B1) to calculate the luminance value Y1 by the above equation (2) (steps S2, S3), and determine whether the calculated luminance value Y1 is larger than the luminance threshold value determined by the photographing conditions at the time of photographing. Is determined (step S4). As a result of this determination, when the luminance value Y1 is larger than the luminance threshold, the hue determination unit 162 calculates the hue H1 from the image data (R1, G1, B1) (step S5), and the calculated hue H1 is determined in advance. It is determined whether or not the color is within the specified hue range (step S6). As a result of this determination, when the calculated hue H1 is within a predetermined specific hue range, the saturation correction unit 163 firstly calculates the saturation Sa1 of the image data (R1, G1, B1) of the high-brightness image. The calculation is performed (step S7), and the image data (R1, G1, B1) is subjected to a high-luminance image conversion process for synthesis to obtain the image data for synthesis (R11, G11, B11) (step S7).
8). Thereafter, the image data for synthesis (R11, G11, B
The saturation Sa11 of 1l) is obtained (step S9). Then, the saturation Sa1 of the image data (R1, G1, B1) and the saturation Sa1 of the image data for synthesis (R11, G11, B11).
A saturation correction amount k is obtained from the saturation correction parameter P set to 1 and the like, and the image data for synthesis (R11, G11, B11) is subjected to the saturation correction by the equation (3) using the obtained saturation correction amount k. Image data (Rc11, Gc1
1, Bc1l) (step S10).

When the luminance judging section 161 judges that the luminance value of the image data (R1, G1, B1) of the high luminance image is smaller than the luminance threshold (step S4), Image data (R1, G1, B
When it is determined that 1) is not within the specific hue range (step S6), the high-luminance synthesis conversion unit 16 does not perform the saturation correction.
5, the image data (R1, G1, B
A luminance value is obtained from 1) and a high-luminance image conversion process is performed to obtain high-luminance synthesis image data (R12, G12, B12) (step S11). When the input image data is low-luminance image data (R0, G0, B0) (step S2), the low-luminance synthesis conversion unit 164 converts the low-luminance image image data (R0, G0, B0) into a luminance value. Y0 is obtained and low-luminance image conversion processing is performed to obtain low-luminance synthesis image data (R01, G01, B01) (step S1).
2). The synthesis processing unit 166 outputs the image data for saturation correction synthesis (Rc11, Gc11, Bc1) obtained by the saturation correction unit 163.
1) and the image data for low luminance synthesis (R01, G01, B01) obtained by the low luminance synthesis conversion unit 164, or the image data for high luminance synthesis (R1) obtained by the high luminance synthesis conversion unit 165.
, G12, B12) and the image data for low-luminance synthesis (R01, G01, B01) obtained by the low-luminance synthesis conversion unit 164.
Is performed, and the combined image data is sent to the image data processing unit 15 (step S13). This process is repeated for each pixel.

As described above, since the correction is performed without changing the luminance value (lightness) before and after the saturation correction, a decrease in the saturation can be suppressed in a high luminance portion such as a background portion. In addition, when performing saturation correction, by acquiring hue information of a captured image, particularly a background image, a color such as blue in the sky or orange at sunset is designated, and only colors close to that color are designated from the hue. By making a judgment and selectively performing saturation correction, it is also possible to emphasize the sky color and the like. Furthermore, by using the luminance information and the hue information at the same time, by designating the sky part as, for example, a blue part having a high luminance,
Saturation correction can be performed only on the sky portion that is not so clear in a general landscape image, and the reproducibility of the sky color can be improved. For example, as shown in FIG. 5A, the same subject as an image obtained by photographing a person 22 with a sky 20 and a cloud 21 as a background by a silver halide photography system is subjected to CCD.
3, the reference image A obtained when the person 22 is photographed with proper exposure is overexposed because the sky 20 and the cloud 21 have a narrow dynamic range as shown in FIG. 5B. Cause the image not to appear. Further, in the case of the background image B in which the sky 20 is exposed, the person portion 22 becomes dark as shown in FIG. 5C, and the dark portion such as the ground 24 becomes black. . Then, when the saturation correction is performed when combining the reference image A when the person 22 is photographed with the proper exposure and the background image B where the sky 20 is exposed, as shown in FIG. An image equivalent to a silver halide photograph in which the blue of the sky 20 remains and the boundary with the cloud 21 is clear can be obtained.

In the above-described embodiment, the case where the reference image A and the background image B captured by changing the exposure amount by the digital camera are synthesized by the image synthesis processing unit 16 provided in the digital camera and subjected to saturation correction, and are synthesized. Saturation correction may be performed when a plurality of images captured by changing the exposure amount with a generally used digital camera are synthesized by a personal computer or the like.

FIG. 6 is a block diagram showing a configuration of a personal computer 31 for displaying an image picked up by a digital camera 30 generally used. As shown in FIG.
CPU 32 and memory 33 for controlling the operation of the entire apparatus
, Input device 34, display device 35, and hard disk device 3
6 and a CD-ROM driver 37. Input device 3
4 inputs image data captured from the digital camera 30. The display device 35 displays the input image data and the CPU 32
The image based on the image data processed in step is displayed. CD-R
The OM driver 37 is a CD-ROM as an external storage medium
The control program and the like stored in 38 are read and input. The CD-ROM 38 stores a control program indicating a process equivalent to the process of performing saturation correction and combining by the image combining processing unit 16 of the embodiment.

When the image data of a plurality of images stored in the memory of the digital camera 30 is input from the input device 34 with the exposure amount changed by the digital camera 30, the CPU 32 stores the input image data in the memory 33, When the image data stored in the image data is processed by the control program stored in the CD-ROM 38 to synthesize an image captured by changing the exposure amount, the color information is obtained from the luminance information, the hue information, and the saturation information of the image data. Correct the degree. Image processing such as gamma correction and compression is performed on the image data synthesized by performing the saturation correction and displayed on the display device 35, and the image data is stored in the external memory by the hard disk device 36. In this manner, a plurality of images captured by changing the exposure amount can be quickly combined to obtain a high quality captured image.

In the above embodiment, the case where image data of a plurality of images stored in the memory of the digital camera 30 is input to the personal computer 31 is explained.
In an image photographing apparatus having a monitoring function for displaying the image, the saturation information, the luminance information, and the hue information of the reference image are input to the personal computer 31 in advance at the time of the periodic photographing condition measurement performed at the time of monitoring. If the background image is photographed and input to the personal computer 31 during the measurement of the periodic photographing conditions, the process of correcting the saturation and synthesizing can be performed more quickly.

[0031]

As described above, according to the present invention, when synthesizing a plurality of images obtained by imaging the same subject with different exposure amounts, the predetermined amount determined by the imaging conditions of the image obtained by reducing the exposure amount. Since the image with higher luminance than the luminance is corrected and synthesized, it is possible to prevent the saturation from dropping and whitish when synthesizing the image of the high-luminance part such as the background image. Obtainable.

Further, when synthesizing a plurality of images, by changing the correction amount of the saturation correction according to the luminance after the synthesis and performing the saturation correction, it is possible to prevent a change in hue when performing the saturation correction. This can prevent abnormally high saturation of the background portion.

Further, when a plurality of images are combined, saturation correction is applied only to pixels that fall within a specific specific hue range, thereby preventing a reduction in color information after combination for a specific color to be emphasized. And a desired high quality image can be obtained.

When synthesizing a plurality of images of the same subject with different exposure amounts, the images having a predetermined brightness or higher determined by the imaging conditions of the images obtained by reducing the exposure amount are used. By storing the control program for performing the saturation correction and combining in the recording medium, the processing of performing the saturation correction and combining can be performed appropriately and more quickly.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of an embodiment of the present invention.

FIG. 2 is a block diagram illustrating a configuration of an image synthesis processing unit according to the embodiment.

FIG. 3 is a characteristic diagram of an input image signal and an output image signal showing image synthesis processing.

FIG. 4 is a flowchart showing the operation of the embodiment.

FIG. 5 is an explanatory diagram showing an image synthesized in the embodiment.

FIG. 6 is a block diagram showing a configuration of another embodiment.

FIG. 7 is an explanatory diagram showing an image synthesized in a conventional example.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 photographing lens 2 aperture 3 CCD 7 digital signal processing section 9 system control section 10 aperture control section 11 CCD drive section 12 switch 13 first memory 14 second memory 15 image data processing section 16 image synthesis processing section 161 brightness determination section 162 Hue determination section 163 Saturation correction section 164 Low-luminance synthesis conversion section 165 High-luminance synthesis conversion section 165 Synthesis processing section

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5C023 AA11 AA21 AA37 BA01 BA03 BA08 BA11 BA15 CA01 CA08 DA04 DA08 EA02 5C024 AA01 CA21 CA26 DA01 DA07 FA01 HA10 HA11 HA23 HA24 5C052 AA12 AA17 DD02 GA02 GA07 GB01 GC05 GD07 GD09 FA FA09 FA27 GB21 KA01 KA03 KA08 KA24 LA02 5C055 AA06 BA03 EA02 EA05 HA14 HA17 HA18 HA31 HA37

Claims (7)

[Claims] An image pickup means includes an image pickup means and an image synthesizing processing means. The image pickup means picks up an image of the same subject with different exposure amounts, and the image synthesis processing means picks up a plurality of images of the same subject picked up with different exposure amounts. This is for combining images, and when combining a plurality of images taken with different exposure amounts, an image having a predetermined brightness or more determined by the imaging condition of the image captured with a smaller exposure amount is used. An image synthesizing device, wherein the image synthesizing is performed by performing saturation correction. 2. The image synthesizing apparatus according to claim 1, wherein said image synthesizing processing means performs a color saturation correction by changing a correction amount of the color saturation correction according to the luminance after the synthesis when synthesizing a plurality of images. 3. The image synthesizing apparatus according to claim 1, wherein said image synthesizing processing means performs a saturation correction only on a pixel falling within a specific specific hue range when synthesizing a plurality of images. 4. When synthesizing a plurality of images obtained by imaging the same subject with different exposure amounts, an image having a predetermined brightness or higher determined by the imaging conditions of the image obtained by reducing the exposure amount is used. An image synthesizing method characterized by performing degree correction and synthesizing. 5. The image synthesizing method according to claim 4, wherein when synthesizing the plurality of images, the saturation correction is performed by changing a correction amount of the saturation correction according to the luminance after the synthesis. 6. The image synthesizing method according to claim 4, wherein, when synthesizing the plurality of images, saturation correction is performed only on pixels that fall within a specific specific hue range. 7. When synthesizing a plurality of images obtained by imaging the same subject with different exposure amounts, an image having a predetermined brightness or higher determined by the imaging conditions of the image obtained by reducing the exposure amount is used. A recording medium storing a control program for performing a degree correction and combining the images, wherein the control program is read by a computer.