JP2002223350A - Image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor capable of processing which relieves discontinuity at boundary parts of a composite image with a small amount of operations and configuring a processing system having maintenance work facilitated. SOLUTION: In this image processor for processing an image group composed of at least two or more images obtained by picking up the same object under different exposure conditions and generating a composite image of a wide dynamic range, a proper exposure extracting part 113 being an extracting means extracts a proper exposure area in each image of the image group, a conversion characteristics calculating part 114 being a calculating means calculates a conversion curve for performing gradation conversion about the proper exposure area in each image, and a smoothing interpolation processing part 121 being a smoothing means smoothes the conversion curve. A compositing means consisting of a luminance correcting part 115, a color- difference correcting part 116, a Y/C compositing part 117 and an image compositing part 118 performs gradation conversion of each image on the basis of the conversion curve smoothed by the compositing means and composites one wide dynamic range image. It is possible to relieve discontinuity simply by smoothing a one-dimensional conversion curve.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus for obtaining a wide dynamic range image by synthesizing a plurality of images having different exposure amounts, and more particularly, to a discontinuity generated at a boundary portion when synthesizing a plurality of images. The present invention relates to an image processing apparatus that generates a high-quality composite image by alleviating an image.

[0002]

2. Description of the Related Art CC used in general electronic cameras
In the imaging device composed of D, there is a problem that the dynamic range width relating to gradation is narrower than that of a silver halide film. With respect to such a problem, a method of generating one wide dynamic range image by combining a plurality of images having different exposure amounts has been proposed. However, the wide dynamic range image obtained in this manner cannot be handled by a display output system such as a normal monitor or a printer, so that the dynamic range needs to be compressed again.

In the case of performing the above synthesis, it is necessary to match the gradation width of each image to be synthesized based on the exposure ratio at the time of photographing. However, it is difficult to accurately perform the exposure at the set exposure amount ratio in actual photographing, and some errors occur. In other words, a slight error occurs between the set exposure ratio and the exposure ratio at the time of actually photographing the exposure ratio between the images to be combined. For this reason, in the composite image, there is a problem that discontinuity occurs at a boundary portion where the image is switched. To cope with such a problem, Japanese Patent No. 2755366 discloses a method of providing an overlapping area between images to be combined and performing weighting addition in this area to reduce discontinuity. Also, Japanese Patent Application Laid-Open No. Hei 7-131708 discloses a method of reducing the discontinuity by correcting the luminance level of each image to be synthesized.

[0004]

As described above, in the prior art, weighting addition and correction of the luminance level in the overlapping area have been performed as processing for alleviating the discontinuity at the boundary. The weighted addition of the overlapping area increases the effect of reducing the mitigation of the discontinuity as the overlapping area is wider, but has a problem that the calculation time becomes longer because the addition is performed on the two-dimensional image area.

The brightness level is corrected by calculating an average brightness level of the entire screen or a partial area, and calculating a correction coefficient from the difference. Since this processing is also performed on a two-dimensional image area, there is a problem that the calculation time becomes long.

Further, in the conventional processing, the synthesis processing for generating a wide dynamic range image, the processing for alleviating the discontinuity at the boundary, and the compression processing for adjusting the wide dynamic range image to the display output system are independent. There is a problem that the processing system becomes complicated.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and a first object of the present invention is to provide an image processing apparatus capable of reducing a discontinuity of a boundary portion of a synthesized image with a small amount of calculation. A second object is to integrate a synthesis process for generating a wide dynamic range image, a process for alleviating discontinuity at a boundary portion, and a compression process for matching a wide dynamic range image with a display output system. An object of the present invention is to provide an image processing apparatus which can form an image processing system having excellent maintainability.

[0008]

According to the first aspect of the present invention,
In an image processing apparatus for processing an image group consisting of at least two or more images captured under different exposure conditions for the same subject to generate a wide dynamic range composite image,
Extracting means for extracting a proper exposure area for each image in the image group, calculating means for calculating a conversion curve for performing gradation conversion for the proper exposure area for each image, and smoothing the conversion curve And a synthesizing unit for performing tone conversion of each image based on the smoothed conversion curve and synthesizing one wide dynamic range image.

According to the first aspect of the present invention, the conversion curves obtained from the appropriate exposure areas of the respective images are integrated, and after smoothing the conversion curves, the gradation conversion of each image is performed. The discontinuity at the boundary can be reduced only by smoothing the one-dimensional conversion curve instead of the image. As a result, the image processing system is simplified, and the discontinuity at the boundary can be reduced with a small amount of calculation.

According to a second aspect of the present invention, in the image processing apparatus of the first aspect, there is further provided changing means for changing an output range of the conversion curve.

According to the second aspect of the present invention, the output range of the conversion curve can be set, and the dynamic range width of the composite image is controlled. When a wide dynamic range image is generated, compression processing can be performed at the same time, and the image processing system is simplified.

According to a third aspect of the present invention, in the image processing apparatus according to the first or second aspect, the smoothing means smoothes the entirety of the conversion curve, and smoothes a boundary of the conversion curve. And at least one of an interpolating means for interpolating a boundary of the conversion curve.

According to the third aspect of the present invention, the conversion curves obtained from the appropriate exposure areas of the respective images are integrated, and after smoothing or interpolating the conversion curves, the gradation conversion of each image is performed. The discontinuity at the boundary can be reduced only by smoothing or interpolating a one-dimensional conversion curve instead of an image. As a result, the image processing system is simplified, and the discontinuity at the boundary can be reduced with a small amount of calculation.

[0014]

Embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention. In the present embodiment, an image processing device in an electronic camera system for recording a still image will be described. 2 is a block diagram showing the configuration of the conversion characteristic calculation unit in FIG. 1, FIG. 3 is an explanatory diagram for explaining calculation of a conversion curve and generation of a discontinuous region in FIG. 1, and FIG. 4 is a spline as a smoothing means in FIG. FIG. 4 is an explanatory diagram illustrating interpolation.

[Configuration] In FIG. 1, a lens system 100, an aperture 1
01, a low-pass filter 102, and an image captured via a single-plate CCD 103 in which color filters are arranged on the light receiving surface are A
The digital signal is converted by the / D converter 104.

The signal from the A / D converter 104 is transferred to the interpolation unit 110 and the working buffer memory 111 via the image buffer memory 105 or the image buffer memory 106.
The signal from the image buffer memory 105 is also transferred to the photometry evaluation unit 107 and the focus detection unit 108. Photometry evaluation unit 10
Reference numeral 7 is connected to the aperture 101, the CCD 103, and the control unit 120, and the focus detection unit 108 is connected to the AF motor 109 and the control unit 120. The working buffer memory 111 sends the Y / C
The separation unit 112 is connected to an appropriate exposure extraction unit 113 and a color difference correction unit 116, which are extraction units. The appropriate exposure extraction unit 113 is connected to a conversion characteristic calculation unit 114, a luminance correction unit 115, a color difference correction unit 116, and an image synthesis unit 118, which are calculation units. The conversion characteristic calculation unit 114 is a smoothing unit that is a smoothing unit. It is connected to the interpolation processing unit 121. The smoothing interpolation processing unit 121 is connected to an output range setting unit 122 that is an output range changing unit, and the output range setting unit 122 is connected to a luminance correction unit 115. The luminance correction unit 115 is connected to the Y / C synthesis unit 117 and the color difference correction unit 116. Further, the color difference correction section 116 is connected to the Y / C synthesis section 117. Y / C synthesis unit 1
17 is connected to an output unit 119 as a recording unit (recording medium) such as a memory card or a disk via an image synthesizing unit 118. The control unit 120 such as a microcomputer
Are an interpolation unit 110, a Y / C separation unit 112, an appropriate exposure extraction unit 113, a conversion characteristic calculation unit 114, a smoothing interpolation processing unit 121, an output range setting unit 122, a luminance correction unit 115, an image synthesis unit 118, and an output unit Connected to 119.

In the image processing apparatus for an electronic camera shown in FIG. 1, an appropriate exposure area for each image is determined by an appropriate exposure extracting unit 113 as extracting means, and each appropriate exposure area is calculated by a conversion characteristic calculating unit 114 as calculating means. A conversion curve is calculated for each image from the area, the conversion curve is smoothed by a smoothing interpolation processing unit 121 as a smoothing unit, an output range setting unit 122 sets an output range of the conversion curve, and A certain luminance correction unit 115, color difference correction unit 116, Y / C synthesis unit 117, and image synthesis unit 118
Each image is subjected to gradation conversion based on the smoothed conversion curve to synthesize one wide dynamic range image.

[Operation] The signal flow will be described with reference to FIG. The pre-imaging mode is entered by half-pressing a shutter button (not shown). A video signal captured via the lens system 100, the aperture 101, the low-pass filter 102, and the CCD 103 is converted into a digital signal by the A / D converter 104 and transferred to the image buffer memory 105. The gradation width of the digitized signal is 10 bits in this embodiment.
The video signal in the image buffer memory 105 is
07 and transferred to the focus detection unit 108. The photometric evaluation unit 107 obtains a luminance level in the image and generates a control signal so that appropriate exposure (first exposure condition) is obtained.
Control the electronic shutter speed of 03, etc. Focus detection unit
108 detects the edge strength of the video signal, controls the autofocus (AF) motor 109 so that the edge strength becomes maximum, adjusts the position of the lens system 100, and obtains a focused image.

Next, the main photographing is performed by fully pressing a shutter button (not shown). Photometry evaluation unit 107
The first image (short-time exposure image) is photographed under the second exposure condition such that the exposure ratio becomes 1/8 with respect to the first exposure condition obtained in the above. , Are converted into digital signals by the A / D converter 104 and transferred to the image buffer memory 106. Subsequently, a second image (long-time exposure image) is captured under the above-described first exposure condition obtained by the photometry evaluation unit 107, and is converted into a digital signal by the A / D converter 104. The data is transferred to the image buffer memory 105. The interpolating unit 110 sequentially reads the single-plate video signals on the image buffer memory 105 and the image buffer memory 106 based on the control of the control unit 120, and performs known interpolation processing, white balance processing, enhancement processing, and the like. The three-plate state signal is generated and transferred to the working buffer memory 111. Y /
The C separation unit 112 reads the three-plate signals in the working buffer memory 111 in the order of the long-time exposure image and the short-time exposure image based on the control of the control unit 120, and separates them into a luminance signal and a color difference signal. As for the luminance signal of the long-time exposure image, the overexposure area is extracted by the appropriate exposure extraction section 113, and the signals of the other areas are converted into the conversion characteristic calculation section 114 and the luminance correction section 115.
Transferred to In the case of a short-time exposure luminance signal, the appropriate exposure extraction unit 113 converts a luminance signal corresponding to an area where the long-time exposure image luminance signal is overexposed into a conversion characteristic calculation unit 11.
4. Transfer to the brightness correction unit 115. The conversion characteristic calculation unit 114
A conversion curve for performing gradation conversion is generated based on the cumulative histogram of the luminance signal transmitted from the appropriate exposure extracting unit 113. The conversion curve is transferred to the smoothing interpolation processing unit 121. The smoothing interpolation processing unit 121 smoothes the conversion curve based on the control of the control unit 120. The smoothed conversion curve is transferred to output range setting section 122. The output range setting unit 122 can set the output range of the conversion curve based on the control of the control unit 120. Then, the conversion curve is transferred to the brightness correction unit 115. The luminance correction unit 115 performs gradation conversion of the luminance signal of the long / short exposure image based on the conversion curve and the control of the control unit 120. The luminance signal after gradation conversion is supplied to a color difference correction unit 116 and a Y / C synthesis unit.
Transferred to 117. The color difference correction unit 116 receives a luminance signal before gradation conversion from the Y / C separation unit 112 and a luminance signal after gradation conversion from the luminance correction unit 115. The color difference correction unit 116 calculates a correction coefficient for correcting the color difference signal from the luminance signal before and after the gradation conversion and a theoretical limit model in which a color can exist, and multiplies the correction coefficient by the color difference signal from the Y / C separation unit 112. The corrected color difference signal is transferred to the Y / C synthesizing unit 117, and the Y / C synthesizing unit 117
The signal is combined with the luminance signal after the gradation conversion from 115 and transferred to the image combining unit 118 as a normal signal. The image synthesizing unit 118 generates a wide dynamic range image by synthesizing the long / short-time exposure image subjected to the gradation conversion processing based on the information from the appropriate exposure extracting unit 113 under the control of the control unit 120. Is transferred to the output unit 119.

FIG. 2 shows the configuration of the conversion characteristic calculation unit 114, which comprises an edge extraction unit 200, a histogram creation unit 201, a conversion curve calculation unit 202, and a conversion curve buffer memory 203. The luminance signal of the appropriate exposure area extracted by the appropriate exposure extraction unit 113 is connected to the smoothing interpolation processing unit 121 via the edge extraction unit 200, the bitogram creation unit 201, the conversion curve calculation unit 202, and the conversion curve buffer memory 203. Have been. Further, the control unit 120 is connected to the edge extraction unit 200. The edge extraction unit 200, based on the control of the control unit 120, the appropriate exposure extraction unit 11
The luminance signal of the long-time exposure image from 3 is read, and a known edge extraction is performed. The histogram creation unit 201 selects pixels having an edge strength equal to or greater than a predetermined threshold value, and creates a histogram. The conversion curve calculation unit 202 generates a conversion curve by accumulating the histogram. The generated conversion curve is stored in the conversion curve buffer memory 203.
Next, under the control of the control unit 120, the edge extraction unit 200 reads the luminance signal of the short-time exposure image from the appropriate exposure extraction unit 113, and performs known edge extraction. Histogram creation unit 20
In step 1, a pixel having an edge strength equal to or higher than a predetermined threshold value is selected, and a histogram is created. The conversion curve calculation unit 202
A conversion curve is generated by accumulating the histogram. The conversion curve calculated by the same process as the long exposure image is stored in the conversion curve buffer memory 203.

Under the control of the control unit 120, the smoothing interpolation processing unit 204 combines the two conversion curves into one conversion curve based on the exposure ratio at the time of photographing, and then performs a moving average method, a median filter, and a morphology process. A known smoothing is performed on the entire curve or near the boundary. The smoothed conversion curve is sent to output range setting section 205. The output range setting unit 205 assigns an output range of 0 to 1023 for a predetermined range, for example, 10-bit gradation, to the gradation conversion curve based on the control of the control unit 120 and transfers the curve to the luminance correction unit 115. As described above, the luminance correction unit 115 performs gradation conversion of the luminance signal of the long / short-time exposure image based on the gradation conversion curve and the control of the control unit 120.

FIG. 3 is an explanatory diagram showing a process of calculating a conversion curve and generating a discontinuous region. The long-exposure image shown in FIG.
The luminance signal is calculated by the / C separation unit 112, and the appropriate exposure extraction unit 113
, An appropriate exposure area is extracted. Thereafter, the conversion characteristic calculation unit
Transferred to the conversion characteristic calculation unit 114, and the edge extraction unit 20 in the conversion characteristic calculation unit 114.
0, the histogram of the edge portion is obtained by the bistogram creating unit 201. FIG. 3 (b) shows the obtained bistogram. The horizontal axis indicates the pixel value which is the luminance level, and the vertical axis indicates the ratio of the appearance ratio to the total number of pixels. In the present embodiment, since a 10-bit gradation is assumed as the luminance level (pixel value), the pixel value on the horizontal axis is 0 to 1023. A gradation conversion curve (hereinafter simply referred to as a conversion curve) is calculated by cumulatively adding the histogram in the conversion curve calculation unit 202. FIG. 3 (c) shows the obtained conversion curve. The horizontal axis is Fig. 3
The pixel value is shown in the same manner as in (b), and the vertical axis represents the ratio (percent) to the total number of pixels. In addition, the proper exposure extraction unit 113
The area actually used is shown in black, with the luminance value determined to be appropriate as the threshold value. Similarly, in the short-time exposure image of FIG. 3D, the histogram of FIG. 3E and the conversion curve of FIG. 3F are obtained. The short-time exposure image has a 10-bit gradation as a luminance level (pixel value) like the long-time exposure image. However, since the exposure amount is different, it is necessary to match the gradation width at the time of synthesis. In this embodiment, since the short-time exposure assumes an exposure amount ratio of 1/8 with respect to the long-time exposure, the gradation width of the short-time exposure image is 13 bits, 0 to 8192. Fig. 3 (e), (f)
The abscissa indicates both of the 13-bit gradation matched with the original 10-bit gradation. Further, the brightness value determined to be appropriate by the appropriate exposure extraction unit 113 is set as a threshold, and the area actually used is shown in black. This threshold value is the threshold value of FIG.
1/8.

When the gradation conversion curves shown in FIGS. 3C and 3F are integrated as a threshold value with the luminance value determined to be appropriate by the appropriate exposure extracting unit 113, the entire conversion curve shown in FIG. can get. FIG.
In (g), the horizontal axis indicates the input pixel value of the 13-bit expression, and the vertical axis indicates the input pixel value of the 10-bit expression. In FIG. 3G, the scale on the horizontal axis is expressed in logarithmic scale. Here, the two conversion curves are synthesized with the threshold value as a boundary. However, since the slopes of the two curves are not uniform, the gradation change corresponding to the two curves in the image after the gradation conversion is performed. Will be different. This causes a discontinuity at the boundary between the two conversion curves. On the other hand, the smoothing interpolation processing unit 121
After combining the two conversion curves, the difference between the slopes can be reduced by smoothing the entire conversion curve or the boundary between the two conversion curves.

Since the discontinuity at the boundary portion is caused by the inconsistency of the slopes of the two curves, it is also possible to apply not the simple smoothing but the interpolation for equalizing the slopes.

FIG. 4 is an explanatory diagram of an example using spline interpolation. FIG. 4A shows an area where interpolation is performed. Here, the long-time exposure side is left as it is, and the short-time exposure side is interpolated. FIG. 4B shows a section when the spline interpolation is performed, and the luminance value determined to be appropriate by the appropriate exposure extracting unit 113 is set as a threshold Th. In the present embodiment, the threshold value (luminance value) Th in the long-time exposure is corrected to Th / 8 in the short-time exposure by the exposure amount ratio. In other words, it is Th / 8 when converted to 13-bit representation. As an example, the section Th-
α to Th, and for short-time exposure, the sections Th / 8 to (Th + α) / 8, (T
h + α) / 8 to (Th + 2α) / 8. α means the section width. Interpolation is actually performed in the section from Th / 8 to (Th + α) / 8, but two sections are provided on both sides to increase the accuracy of spline interpolation. FIG. 4C shows a cubic expression used in cubic spline interpolation. Interpolation becomes possible by obtaining the coefficient of the cubic expression by a known method. Since the third-order spline interpolation is the same up to the second-order differentiation, it is possible to eliminate a difference in inclination.

Next, the output setting section 122 as an output range changing means will be described. The vertical axes of the conversion curves shown in FIGS. 3C and 3F are relative values and can be freely set. FIG.
In the example of (g), the same input 10-bit gradation is assigned.
In the present embodiment, since the exposure amount ratio of the short-time exposure image is assumed to be 1/8, the composite image has a 13-bit gradation, but the obtained composite image has a gradation width of 10 bits. That is, the compression processing can be shared. By changing the output range on the vertical axis in FIG. 3G other than the 10-bit gradation, an output image with an arbitrary gradation can be obtained. The output range setting section 122 sets this range.

With the above configuration, the discontinuity that occurs at the boundary when a wide dynamic range image is synthesized can be reduced by smoothing or interpolating one-dimensional data called a conversion curve. Therefore, the processing time can be reduced. Further, the three processes of synthesis, discontinuity countermeasure, and compression can be integrated, so that the maintainability of image processing can be improved and the process can be simplified.

Although the composition of the embodiment described above shows the synthesis of two images of the long / short exposure image, the present invention is not limited to this, and the present invention is not limited to this. It is also possible to adopt a configuration of combining from a plurality of images of the condition.

[0029]

As described above, according to the image processing apparatus of the present invention, it is possible to reduce the amount of computation and reduce the discontinuity at the boundary of the composite image. In addition, the synthesis process that generates a wide dynamic range image, the process to reduce the discontinuity at the boundary, and the compression process to match the wide dynamic range image to the display output system are integrated, making the image simpler and easier to maintain. A processing system can be configured.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration of a conversion characteristic calculator in FIG.

FIG. 3 is an explanatory diagram illustrating calculation of a conversion curve and generation of a discontinuous region in FIG. 1;

FIG. 4 is an explanatory diagram illustrating spline interpolation as a smoothing unit in FIG. 1;

[Explanation of symbols]

 113: suitable exposure extraction unit (extraction means) 114: conversion characteristic calculation unit (calculation means) 115: luminance correction unit 116: color difference correction unit 117: Y / C synthesis unit 118: image synthesis unit 121: smoothing interpolation processing unit ( Smoothing means) 122: Output range setting unit (change means)

 ──────────────────────────────────────────────────続 き Continued on the front page F-term (reference)

Claims (3)

[Claims] An image processing apparatus for processing a group of at least two or more images of the same subject captured under different exposure conditions to generate a composite image having a wide dynamic range. Extracting means for extracting a proper exposure area for each image; calculating means for calculating a conversion curve for performing gradation conversion on the proper exposure area for each image; smoothing means for smoothing the conversion curve; An image processing apparatus comprising: a synthesizing unit that performs tone conversion of each image based on a smoothed conversion curve and synthesizes one wide dynamic range image. 2. The image processing apparatus according to claim 1, further comprising changing means for changing an output range of said conversion curve. 3. The smoothing means includes: a total smoothing means for smoothing the entire conversion curve; a boundary smoothing means for smoothing a boundary of the conversion curve; an interpolation for interpolating a boundary of the conversion curve. 3. The image processing apparatus according to claim 1, further comprising at least one of the means.