JPH10262182A - Image synthesizer - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide the image synthesizer where an image signal of a major subject in a synthesis image is reproduced with fidelity and an image signal output is obtain while expanding a dynamic range even when image quality is not deteriorated even in a high luminance part, when a plurality of images having different exposure are synthesized. SOLUTION: In the image synthesizer where a plurality of images with different exposure which are picked up by image pickup means (1-3, 7-11) and entered by image input means (4-9, 12, 13) are synthesized to form one image an image with a short exposure time is divided into three, based on the luminance, a low luminance image in a short exposure time is replaced with an image signal with along exposure time, a level of a medium part resulting from 1/3 division of an image in a short exposure time is comformed with a level of a luminance of an image signal in a long exposure time replaced into a low luminance image in a short exposure time and a luminance level of an image signal with a high luminance resulting from 1/3 division of the image in a short exposure time so as to obtain one image through the provision of synthesis means (14-19).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention is provided in a photographing apparatus such as a digital video camera and a digital still camera,
The present invention relates to an image synthesizing apparatus for synthesizing a plurality of images photographed by an imaging means into one image, and more particularly to an image synthesizing apparatus capable of expanding a dynamic range of the imaging apparatus.

[0002]

2. Description of the Related Art The dynamic range of a solid-state imaging device such as a CCD (Charge Coupled Device) used in a digital video camera, a digital still camera, or the like is higher than the dynamic range of an image captured by a silver halide photography system using a normal photographic film. There is a problem that the image quality is degraded because the image is very narrow and dark areas of the subject suffer from underexposure of black, and bright areas of the subject suffer from overexposure. Therefore, as a method of expanding the dynamic range by such a solid-state imaging device, a plurality of images of the same subject are taken with different exposure amounts, and the image signals of the plurality of images are added to synthesize an image having an expanded dynamic range. The method is described in JP-A-60-52172 and JP-A-61-21.
9270 publication.

FIGS. 5, 6 and 7 are diagrams showing an input image signal and an output image signal by conventional image synthesis. FIG. 5 is a graph showing input / output image signal characteristics of captured images having different exposure amounts, and FIG. FIG. 7 is an input / output image signal characteristic diagram obtained by adding input image signals of captured images having different exposure amounts, and FIG. 7 is an input / output image signal characteristic diagram obtained by compressing the added output characteristics as shown in FIG. As described above, the photographed images having different exposure amounts shown in FIG. 5 are added as shown in FIG. 6 and then compressed to include a low-luminance image and a high-luminance image that cannot be obtained by one photographed image. FIG.
An image having an extended dynamic range as described above can be obtained.

[0004]

However, when the images captured with different exposure amounts are simply added as described above, the short exposure time is maintained while the high-luminance portion of the image captured with the long exposure time is overexposed due to saturation. Therefore, there is a problem that an image state different from the image signal characteristic of the subject is obtained because the image state is added to the image photographed by the camera. As a method of avoiding such a problem, Japanese Patent Application Laid-Open No. 62-108678 discloses a method of reducing the influence of saturation by performing processing such as subtraction of a high-brightness part of an image captured with a long exposure time, thereby reducing the effect of saturation. There has been proposed a method of adding an image captured with an exposure time to an image. However, even with this method, the problem of approaching the proper image signal of the subject but having a different image state has not been completely solved. In addition, it is very difficult to add an image signal photographed with a long exposure time and an image signal photographed with a short exposure time, and then compress and correct the image signal changed by the addition so as to match an appropriate image signal of the main subject. There is a problem that it is difficult. In addition, since the image signal value is also added to the high-brightness part of the synthesized image, the data value is converted to a state of saturation or a state close to saturation, and the high-brightness part becomes a flying image state with no further difference in luminance. There is a problem.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problem of image composition in a conventional photographing apparatus, and when composing a plurality of images having different exposure amounts, an image signal of a main subject in the composite image is obtained. It is an object of the present invention to provide an image synthesizing apparatus which faithfully reproduces the image signal and which can provide a good-looking image signal output in which the dynamic range is expanded without deteriorating the image quality even in a high-luminance portion.

[0006]

In order to achieve the above object, the present invention combines a plurality of images having different exposure amounts photographed by a photographing means and inputted by an image input means into one image. In the image synthesizing apparatus, the image of the short exposure time is divided into three parts by luminance, the image signal of the long exposure time is replaced on the low luminance part side of the image of the short exposure time, and the image of the short exposure time is divided into three parts. The unit adjusts the image signal luminance level of the long exposure time replaced with the image low luminance part of the short exposure time and the high luminance side image signal luminance level obtained by dividing the image of the short exposure time into three. The image processing apparatus further comprises a synthesizing means for forming an image.

Further, in the image synthesizing apparatus of the present invention, in addition to the above configuration, the image signal having the longest exposure time is subjected to exposure shooting in accordance with the main subject, and the image signal having the short exposure time is adjusted in accordance with the main subject. And an image input means for obtaining an exposure time shorter than the exposure time. Further, the division position between the middle luminance portion and the high luminance portion in the image with the short exposure time is higher in luminance than the division position between the middle luminance portion and the high luminance portion in the image with the longer exposure time. 3). Further, the calculation of the number of images to be combined is set based on the luminance difference in the subject image.

[0008]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail based on illustrated embodiments. FIG. 1 is a view showing one embodiment of the present invention, and is a block diagram showing a schematic configuration of an image synthesizing apparatus. In FIG. 1, a subject image passing through a photographing lens 1 and a diaphragm 2 of a photographing means is formed on a CCD 3 which is a known image pickup device. Then, an image signal is output by the CCD 3 that converts the light amount of the formed subject image into an electric signal. The output image signal is converted into a signal by a CDS (correlated double sampling) circuit 4 of the image input means.
After noise and the like of the CD 3 are removed and the gain is controlled by the process processing circuit 5, the analog signal is converted into a digital signal by the A / D converter 6. An aperture driving unit 7 is connected to the aperture 2, and a CCD driving unit 8 is connected to the CCD 3. The aperture driving unit 7 and the CCD driving unit 8 are connected to a system control unit 9. System control unit 9
Is connected to a photometric circuit 11 for receiving a photometric light beam incident from a photometric lens 10. The diaphragm drive unit 7 is controlled by a system control unit 9 by photometry of a subject image. The aperture of the stop 2 is opened and closed so as to change the diameter of the light beam, and the exposure amount of the photographed light image is adjusted. The CCD driving unit 8 is controlled by the system control unit 9 and controls the operation of the CCD 3 so that the electric signal converted by the photographed light image can be stored and taken out. Accordingly, the CCD 3 performs a shutter operation under the control of the CCD driving unit 8, and image information for one image on the CCD 3 is sent to the CDS 4, passes through the process processing circuit 5, and is sent to the A / D converter 6.

The A / D converter 6 includes a digital signal processing unit 1
2 is connected, and the image signal converted into the digital image signal by the A / D converter 6 is
Is converted into a luminance signal or an RGB signal. The switch 13 is connected to the digital signal processing unit 12, and image information of one image continuously shot by changing the exposure condition of the subject image is switched by the system control unit 9 to the connection destination of the switch 13. , The first memory 14, the second memory 15, and the third memory 19. The switch 13 is also connected to the image data processing unit 17, and an image signal that is not subjected to image synthesis is sent from the switch 13 to the image data processing unit 17. The first memory 14, the second memory 15, and the third memory 19 are connected to a synthesizing processing unit 16 as an image synthesizing unit, and the first memory 14, the second memory 15, and the third memory 15 stored for image synthesizing. The image signal of the memory 19 is subjected to image composition processing by the composition processing section 16 based on image information from the system drive section 9. The image signal sent from the A / D converter 6 is extracted by the digital signal processing unit 12 only from the first memory 14 and the second memory 14.
Needless to say, by storing in the memory 15 and the third memory 19, the capacity of the memory can be reduced. The conversion of the luminance signal and the RGB signal performed by the digital signal processing unit 12 may be performed before the image processing is performed by the synthesis processing unit 16. The synthesis processing unit 16 is connected to the image data processing unit 17, and the synthesized image signal synthesized by the synthesis processing unit 16 and the image signal directly sent from the switch 13 are subjected to gamma correction and image processing by the image data processing unit 17. After being subjected to image processing such as compression, the image data is output to an external memory 18 such as a memory card or a magneto-optical disk (MO) and stored in the external memory 18.

Next, FIG. 2 is an input / output image signal characteristic diagram showing an example of an input image signal and an output image signal of photographed images having different exposure amounts which are photographed and image-processed by the apparatus of FIG.
FIG. 3 is an input / output image signal characteristic diagram showing an input image signal and an output image signal after synthesis in an image synthesis processing unit when image synthesis of a plurality of captured images shown in FIG. 2 is performed. In FIG. 2, a solid line H is an image signal obtained by photographing a main subject with an appropriate exposure, a solid line M is an image signal photographed with an exposure time shorter than the H image, and a solid line S is an image signal photographed with an exposure time shorter than the M image. . 2 and 3 are diagrams based on the input image signal (for example, a value obtained by converting the image data (R, G, B) of the image data) of the solid line S into an image. Saturated signal (brightness: 0) from image signal 0 (brightness: 0)
MAX).

In FIG. 2, the H image is saturated at the position a of the S image (output image signal: MAX), and the M image is saturated at the position b of the S image (output image signal: MAX). Therefore, the M image is divided into three at the positions of c and d according to the luminance of the output image signal, and the S image is divided into three at the position of e, which is higher in luminance than d. In other words, the division position between the middle luminance part and the high luminance part in the image with the short exposure time is higher than the division position between the middle luminance part and the high luminance part in the image with the longer exposure time. As a result, no luminance inversion occurs in the composite image. Note that the position of c is set in the high-luminance portion of the output image that has little effect even if the image signal slightly changes. Further, in the shooting of the M image, the exposure amount is set to H in order to avoid an unstable image signal area such as a blackened area at the position of c to be combined with the H image.
It is set within 10 times of the image. Therefore, when the exposure of 10 times or more of the H image is required to eliminate the overexposure by the preliminary photometry of the subject image, the M image and the S image are photographed. Only images are taken. Naturally, if there is no overexposed portion in the H image, there is no need to perform the combining process, so the H image is sent from the digital signal processing unit 12 to the image data processing unit 17 through.

FIG. 3 is a diagram obtained by synthesizing the respective photographed images H, M, and S of FIG. 2 as described above. In FIG. 3, a solid line from 0 to c of the input image signal is an H image without conversion of the input image signal. , C to d are M images obtained by converting the input image signal,
A solid line from d to e is an S image obtained by converting the input image signal, and a solid line from e to MAX is an S image obtained without conversion of the input image signal.

By the way, as described above, in the shooting of the M image, the exposure amount is set within 10 times of the H image in order to avoid an unstable image signal area where the position c to be combined with the H image is underexposed. However, the reason will be described below.

Although the sensitivity varies depending on the performance of the CCD 3, which is an image pickup device, and the noise region changes, noise is generated and the color difference is small when the luminance of the photographed image data is low. Therefore, when the position of the connection portion (c in FIG. 2) of the M image connected to the H image becomes an area where noise occurs, the image data of the connection portion (c) of the M image is replaced with the image data of the connection portion of the H image. When the luminance is changed in accordance with the luminance, the ratio of the H image to the image data R, G, and B at the connection portion becomes a value far apart. On the other hand, the image data is saturated when the luminance increases, and in the case of the H image in FIG. 2, the image data is saturated at the position a of the input image signal (output image signal: MAX). B is a saturated position, and R, G, B
Some of them are already saturated. R, G, B
Is saturated, the color balance is lost, and the ratio of the image data R, G, and B at the connection portion of the two images becomes a different value regardless of the same image. As described above, the connection position of the image to be synthesized needs to be at a luminance intermediate point where the image data is stable without being affected by noise and not saturated.

As described above, the noise generation area varies depending on the performance of the CCD. However, in the evaluated CCD, the image data was not stable at an luminance of 2 ⁴ (16) or less in an 8-bit (MAX: 255) image. Therefore, it is preferable to set the image stable area at low luminance to 2 ⁵ (32) in view of the margin. On the high-brightness side, when an image with a small amount of exposure is synthesized at a high-brightness position, the image data is subjected to brightness conversion near saturation and the effect of the synthesis is diminished. Also, depending on the exposure difference between the captured images to be synthesized, when the image is synthesized at a high luminance, the luminance of the output image signal changes abruptly at the connection position c between the H image and the M image whose luminance has been converted. In consideration of the above content and saturation, the high luminance side is stable at a luminance of 160 or less, the luminance of the connection position c of the M image is set to 32, and the luminance of the connection position c of the H image is set to 1
If it is set to 60, the saturation position b of the M image is about five times the saturation position a of the H image, that is, five times the exposure amount difference (luminance difference). Note that the output image signal at the connection position c is formed as a straight line or a curved curve so that the output image signal at the connection position c between the H image and the luminance-converted M image does not suddenly change (to improve the appearance of the output image). It is also possible to change it gently.

In the above description, the image stable area at low luminance is set to 2 ⁵ (32) in view of the margin. However, as described above, the image data R and G of the H image are located near the connection position of the M image to be subjected to luminance conversion. , B, the data value is corrected in accordance with the color data of the first photographed image (H image) even with the luminance 2 ⁴ (16) having no margin in stability. Therefore, when the brightness at the connection position c of the M image is set to 16 and the brightness at the connection position c of the H image is set to 160, the saturation position b of the M image is 10 times the saturation position a of the H image, that is, 10 times the exposure. This results in an amount difference (luminance difference). As described above, if the exposure amount of the M image is within 10 times (more preferably, within 5 times) of the H image, the unstable image signal area can be avoided.

Next, a specific image synthesis processing operation by the synthesis processing unit 16 shown in FIG. 1 will be described. In FIG. 1, a first memory 14 stores a first captured image obtained by capturing a main subject with an appropriate exposure, and a second memory 15 stores a second captured image captured with a shorter exposure time than the first captured image. You. FIG. 2 is an input / output image signal characteristic diagram showing an input image signal and an output image signal of captured images having different exposure amounts captured by the apparatus of FIG. 1 as described above, and the horizontal axis (input based on the S image) Image signal) and a vertical axis (output image signal) are graphs in which respective image data (R, G, B) are values converted into luminance. The first captured image in the first memory 14 is an H image, The second captured image in the second memory 15 corresponds to an M image, and the third captured image in the third memory 19 corresponds to an S image. The saturation of the H image (image output signal: MAX) starts from the position a of the input image signal (in the S image),
Image saturation begins at position b. The first captured image or the second
The presence or absence of saturation of the captured image is determined by dividing the image signal sent from the digital signal processing unit 12 into screens by the system control unit 9 and determining the luminance (saturation etc.) state of each division unit as an AE exposure amount selection criterion. Has been detected), the second
When it is determined that the captured image also includes the saturation state, the third captured image is captured with a shorter exposure time than the second captured image and stored in the third memory 19. If it is determined that the third photographed image also includes a saturated state, the memory is further increased to increase the number of photographed images to eliminate the saturated state, thereby obtaining the minimum exposure (first) photographing of the image to be synthesized. Other than the image, the luminance is converted to the same high level, but a wider dynamic range can be achieved. As described above, in the present invention, the calculation of the number of images to be combined is set based on the luminance difference in the subject image.

The AE photographing state (direct light, back light, over-direct light, etc.) of the first photographed image and the EV value difference are input from the system control unit 9 to the synthesizing processing unit 16. The connection luminance position between the (H image in FIG. 2) and the second captured image (M image in FIG. 2) and the connection luminance position between the second captured image and the third captured image (S image in FIG. 2) are determined. When the connection luminance position is determined, the second position is adjusted according to the connection position as shown in FIG.
The captured image and the third captured image are subjected to luminance conversion, but the R, G, B image data of the first captured image and the R, G, B image data of the second captured image at the connection position are compared, and R, G, If there is a difference between B and B, the R, G, and B image data of the second captured image is corrected so that the luminance does not change, and the R, G, and B image data of the second captured image are replaced with the first captured image. R, G, B
Approximate to image data. Similarly, the R, G, and B image data at the connection position of the second captured image and the third captured image are compared, and if there is a difference, the R, G, and B image data of the third captured image are corrected.

FIG. 3 shows a composite of the input image signals in this manner. The luminance of the input image signal of the H image, which is equal to or higher than c, is converted into the M image and the S image, and the luminance of the H image is shifted to the saturated image data position of the H image. (In this case, if the replacement is performed without increasing the luminance of the M image and the S image, the image that should be at the high luminance of the H image is output at a low luminance, so that a discomfort occurs in the composite image.)

The synthesis processing section 16 sequentially extracts image data corresponding to the coordinate position of the output screen from each memory. The first photographed image (H image in FIG. 2) stored in the first memory 14 is obtained. 0 to c image data and luminance image data of e or more of the third captured image (S image in FIG. 2) stored in the third memory 19 are raw captured image data not subjected to luminance conversion, and other image data Is sent to the image data processing unit 17 as an output image signal while performing the image data value conversion as described above. As a result, an image signal output in which the dynamic range is expanded and the image quality is not deteriorated even in the high luminance portion can be obtained. An operation example of the above-described image synthesizing process is shown in the image synthesizing operation flowchart of FIG.

Another method is as follows.
The digital signal processing unit 12 transfers captured image data to the synthesis processing unit 16. The transferred H image data is stored in the memory of the synthesis processing unit 16 as it is,
The image data of the image is subjected to luminance conversion of the input image data cd based on information from the system control unit 9 and replaced with the image data of the H image stored in the memory corresponding to the coordinate position of the image data cd. I do. Similarly, the image data of the S image is input image data de to e subjected to luminance conversion, and the image data of e or more is replaced with the image data of the M image stored in the memory of the synthesis processing unit 16 to create a synthesized image. . By performing such image processing, it is not necessary to store all image data of the captured image to be synthesized in a plurality of memories,
The memory capacity can be reduced.

[0022]

As described above, in the image synthesizing apparatus according to the first aspect, a composite image connecting portion of an image signal photographed with a long exposure time and an image signal photographed with a short exposure time has a blackened area or a white area. Since a luminance intermediate image signal with a short exposure time without skipping is used, an image signal of a main subject can be synthesized with an image signal that is faithfully reproduced. Further, since the processing of the photographed image is a part of the image signal photographed with a short exposure time, the synthesis algorithm is simple and the processing time can be shortened. In addition, since the image signal photographed with the short exposure time is combined with the high luminance image signal part of the image signal photographed with the long exposure time without correction, even in a photographing scene with a wide luminance range, the high luminance part is clear. Can be output in the state.

In the image synthesizing apparatus according to the second aspect, the image signal of the main subject photographed with the proper exposure is synthesized as it is without correcting the photographed image signal. Can maintain an image state that does not impair the image quality of the image.

In the image synthesizing apparatus according to the third aspect, since the synthesized image is sequentially synthesized and converted in luminance from a low luminance, the synthesized image is in an image state in which there is no luminance reversal with respect to the photographed subject and there is no uncomfortable feeling. can do.

In the image synthesizing apparatus according to the fourth aspect, since the number of images is set and synthesized according to the brightness of the subject, an image state with good gradation can be obtained even for a subject having a wide luminance range.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating an embodiment of the present invention, and is a block diagram illustrating a schematic configuration of an image combining device.

FIG. 2 is an input / output image signal characteristic diagram showing an example of an input image signal and an output image signal of photographed images having different exposure amounts which are photographed and image-processed by the apparatus of FIG.

FIG. 3 is an input / output image signal characteristic diagram showing an input image signal and an output image signal after synthesis in an image synthesis processing unit when image synthesis of a plurality of captured images shown in FIG. 2 is performed.

4 is an operation flowchart illustrating an example of an image synthesis processing operation performed by a synthesis processing unit of the apparatus illustrated in FIG. 1;

FIG. 5 is a diagram showing an example of the prior art, and is an input / output image signal characteristic diagram of a high-exposure photographed image with a large amount of exposure and a low-exposure photographed image with a small amount of exposure.

6 is an input / output image signal characteristic diagram of a combined image signal obtained by adding input image signals of captured images having different exposure amounts shown in FIG. 5;

7 is an input / output image signal characteristic diagram of a synthesized image signal obtained by compressing the image signal obtained by addition and synthesis shown in FIG. 6 into an output width.

[Explanation of symbols]

 Reference Signs List 1 shooting lens 2 aperture 3 CCD 4 CDS circuit 5 process processing circuit 6 A / D converter 7 aperture drive unit 8 CCD drive unit 9 system control unit 10 photometric lens 11 photometric circuit 12 digital signal processing unit 13 switch 14 first memory 15 second memory 16 synthesis processing unit 17 image data processing unit 18 external memory 19 third memory

Claims (4)

[Claims] An image synthesizing apparatus for synthesizing a plurality of images having different exposure amounts photographed by a photographing means and input by an image input means into one image, wherein an image having a short exposure time is divided into three parts by luminance. Then, the image signal of the long exposure time was replaced with the image low-brightness part of the image of the short exposure time, and the middle part obtained by dividing the image of the short exposure time into three was replaced with the image low-brightness part of the short exposure time. Image synthesizing means for synthesizing a single image by matching an image signal luminance level of a long exposure time with a high-luminance image signal luminance level obtained by dividing the image of the short exposure time into three. apparatus. 2. An image synthesizing apparatus according to claim 1, wherein an image signal having the longest exposure time is subjected to exposure photographing according to the main subject, and an image signal having a short exposure time is shorter than the exposure time conforming to the main subject. An image synthesizing apparatus comprising an image input unit for obtaining an exposure time. 3. The image synthesizing device according to claim 1, wherein the dividing position of the intermediate luminance portion and the high luminance portion in the image of the short exposure time is divided into the dividing position of the intermediate luminance portion and the high luminance portion in the image of the longer exposure time. An image synthesizing device characterized by higher brightness. 4. The image synthesizing apparatus according to claim 1, wherein the calculation of the number of images to be synthesized is set based on a luminance difference in a subject image.