KR102247597B1 - Image processing apparatus and image processing method - Google Patents

Abstract

A use-image selection unit 40 for generating use-image selection information to select one of a short exposure image and a long exposure image detected by the image sensor 10 for each pixel; A WDR synthesis unit 60 for synthesizing a short exposure image and a long exposure image to obtain a synthesized image according to the use-image selection information; And a mixing unit 90 that calculates a mixing ratio of the long exposure image and the synthesized image according to the signal amplification degree of the image sensor 10, and mixes the synthesized image and the long exposure image according to the mixing ratio. 1) is provided.

Description

Image processing apparatus and image processing method TECHNICAL FIELD

The present invention relates to an image processing apparatus and an image processing method.

Recently, by continuously photographing and synthesizing an image of short exposure (hereinafter, simply referred to as "short exposure image") and an image of long exposure (hereinafter, simply referred to as "long exposure image"), the dynamic range that the image sensor can take A shooting function called Wide Dynamic Range (WDR) or High Dynamic Range (HDR) that captures an image that exceeds the dynamic range is increasing. This shooting function is particularly effective in scenes with very high contrast ratios such as backlit compositions.

As a method of obtaining the WDR signal, it is common to adopt a method of continuously photographing and synthesizing multiple images while changing the exposure time. The dynamic range obtained here may reach 16 bits to 20 bits or more. The WDR signal is output after being compressed in a range of 8 to 12 bits according to the display capability of a display or printer.

This WDR synthesis process produces good results in situations where there is a lot of light, such as during the day, but in a dark situation, the exposure amount of the short exposure image becomes insufficient, resulting in noticeable noise around the area of use of the short exposure image in the composite image. . In particular, when the signal is amplified in the image sensor, the noise of the short-exposed image becomes very significant, so that the noise interference problem becomes larger than the effect of the WDR synthesis process. In this case, it is preferable not to use a short exposure image by turning off the WDR synthesis process. In addition, a method of applying noise reduction processing to short-exposure images can be conceived. For short-exposure images by amplification of the image sensor in dark scenes, the signal-to-noise ratio (SNR) is very low. Noise cannot be sufficiently reduced by noise reduction.

As a method of turning off the WDR synthesis process depending on the situation, the following techniques have been disclosed.

For example, a technique has been disclosed in which pre-photographing is performed to determine whether a subject contains motion, and when the subject contains motion, the WDR synthesis process is turned off (Japan as Patent Document 1). See Patent Publication No. 2000-50151). In addition, a technique for determining the number of pre-photographed images of each exposure image by the sensitivity of an image sensor has been disclosed using a technique for reducing noise by overlapping multiple shot images (Japanese Patent Publication 2012- as Patent Document 2). 239077). Accordingly, it is possible to obtain a result of good WDR synthesis processing in which noise of a short exposure image is reduced.

Japanese Patent Laid-Open No. 2000-50151 Japanese Patent Laid-Open No. 2012-239077

However, in the technique described in Patent Document 1 (Japanese Patent Laid-Open No. 2000-50151), it determines whether or not the subject contains motion, but does not determine the amount of noise, so this technique is included in the result of the WDR synthesis process. It is different from noise reduction technology.

In addition, since the technique described in Patent Document 1 is based on pre-shooting, it is a technique for still images, and even if it is applied to a moving picture, the frame rate is greatly reduced. Since the technique described in Patent Document 2 (Japanese Patent Application Laid-Open No. 2012-239077) is also based on pre-shooting, it is a technique for still images, and even if applied to moving pictures, the frame rate is greatly reduced.

In addition, there is a prior art in which the WDR synthesis processing is turned off depending on the situation, but a technology for gradually switching the processing results of on and off of the WDR synthesis processing in time has not been disclosed. In a system in which the result of the WDR synthesis processing is obtained as a video, when the WDR synthesis processing is suddenly switched on and off, a rapid change occurs in the result of the WDR synthesis processing obtained by the WDR synthesis processing. Considering that the situation where the WDR synthesis process is necessary and the unnecessary situation should be gradually changed, it is preferable that the result of the WDR synthesis process also changes gradually.

Accordingly, an object of the present invention is to provide a technique capable of gradually reducing noise included in a WDR synthesized image. In addition, it is intended to provide a technology that can gradually switch on and off the WDR synthesis process in time according to changes in the lighting environment of the subject. In addition, by setting the WDR synthesis-result ratio, it is intended to provide a technology capable of outputting the same image as the long exposure image from the blending unit when WDR is off.

An image processing apparatus according to an aspect of the present invention includes a use-image selection unit, a synthesis unit, and a blending unit.

The use-image selection unit generates use-image selection information to select one of a short exposure image and a long exposure image detected by an image sensor for each pixel.

The synthesis unit synthesizes the short exposure image and the long exposure image according to the use-image selection information to obtain a synthesized image.

The mixing unit calculates a blending ratio of the long exposure image and the composite image according to a signal amplification degree of the image sensor, and mixes the composite image and the long exposure image according to the blending ratio.

According to this configuration, the WDR synthesis-result ratio can be controlled according to the signal amplification degree of the image sensor. For example, as the signal amplification degree of the image sensor increases, noise included in the WDR synthesized image becomes noticeable. Accordingly, by changing the WDR synthesis-result ratio according to the signal amplification level of the image sensor, the mixing unit may gradually reduce noise included in the WDR synthesized image. In addition, it is possible to gradually switch on and off the WDR synthesis process in time according to changes in the lighting environment of the subject. In addition, by setting the WDR synthesis-result ratio, it is possible to output the same image as the long exposure image from the mixing unit when WDR is off.

The mixing unit may reduce a mixing ratio of the composite image to the long exposure image according to an increase in the signal amplification degree in a predetermined section of the signal amplification degree. According to this configuration, as the signal amplification degree of the image sensor increases in a predetermined section, the selection ratio of the long exposure image increases. Accordingly, it is possible to gradually reduce noise included in the WDR synthesized image in a predetermined section.

According to the image processing method of another aspect of the present invention,

Generating use-image selection information to select one of a short exposure image and a long exposure image detected by the image sensor for each pixel;

Obtaining a synthesized image by synthesizing the short exposure image and the long exposure image according to the use-image selection information; And

And calculating a blending ratio of the long exposure image and the composite image according to a signal amplification degree of the image sensor, and mixing the composite image and the long exposure image according to the blending ratio.

According to this method, the WDR synthesis-result ratio can be controlled according to the signal amplification degree of the image sensor. For example, as the signal amplification degree of the image sensor increases, noise included in the WDR synthesized image becomes noticeable. Accordingly, by changing the WDR synthesis-result ratio according to the signal amplification level of the image sensor, the mixing unit may gradually reduce noise included in the WDR synthesized image. In addition, it is possible to gradually switch on and off the WDR synthesis process in time according to changes in the lighting environment of the subject. In addition, by setting the WDR synthesis-result ratio, it is possible to output the same image as the long exposure image from the mixing unit when WDR is off.

According to the present invention, noise included in the WDR synthesized image can be gradually reduced. In addition, according to the present invention, on and off of the WDR synthesis process can be gradually switched over time. In addition, by setting the WDR synthesis-result ratio, the same image as the long exposure image may be output from the mixer when WDR is off.

1 and 2 are diagrams for explaining a general WDR synthesis technique.
3 is a diagram illustrating a function and configuration of an image processing apparatus according to an exemplary embodiment of the present invention.
4 is a diagram for explaining the calculation of the WDR synthesis-result ratio.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In addition, in the present specification and drawings, elements having substantially the same functional configuration are denoted by the same reference numerals, so that redundant descriptions are omitted.

In addition, in the present specification and drawings, a plurality of constituent elements having substantially the same functional configuration may be distinguished by adding different alphabets after the same reference numerals. However, when it is not necessary to specifically distinguish each of a plurality of constituent elements having substantially the same functional configuration, only the same reference numerals are attached.

(Common WDR synthesis technology)

First, a general WDR synthesis technique will be described.

1 and 2 are diagrams for explaining a general WDR synthesis technique. Referring to FIG. 1, images taken during the day (short exposure image (Img-a), long exposure image (Img-b), and composite image (Img-c)) are disclosed.

Further, referring to FIG. 2, images captured at night (short exposure image (Img-d), long exposure image (Img-e), and composite image (Img-f)) are disclosed.

As shown in Fig. 1, in a scene in which a clear sky outdoors is photographed indoors during the day, a long exposure image (Img-b) is used indoors, and a short exposure image (Img-a) is used outdoors. , Synthetic images with high visibility (Img-c) can be obtained both indoors and outdoors.

However, in the evening, as both outdoors and indoors become dark and the brightness of the subject decreases, the image sensor tries to maintain the brightness by electronically amplifying and outputting a signal. From the block that performs WDR synthesis, the input signal in the daytime and the amplified input signal in the evening time cannot be distinguished because the brightness is the same or similar. However, since the input signal in the evening time is amplified by the image sensor, it contains a lot of noise, and noise is particularly conspicuous in a region in which a short exposure image is used.

In addition, as shown in Fig. 2, in the time zone at night, both outdoors and indoors become darker, and the image sensor also amplifies and outputs a signal with a large width. In particular, short-exposure images are photographed as images with a large amount of noise because the exposure amount is small and the amount of light is amplified. However, looking at the long exposure image at night (Img-e), the outdoor electric signboard or the moon is saturated, and if the WDR is in operation, the short exposure image is used for the electric signboard or the area including the moon. As a result, there is a problem in that noise becomes a very remarkable synthesis result in an area including an electric signboard or the moon.

In this situation, interference due to noise may increase compared to the effect of the WDR synthesis process. When the quality of the short exposure image is too low, it is preferable to stop the WDR synthesis process and use only the long exposure image.

Therefore, in the present specification, on the premise of a WDR system that obtains the WDR processing result as a video, the signal amplification degree of the image sensor is referred to, and when the signal amplification degree of the image sensor is a predetermined section (for example, when the signal amplification degree of the image sensor is high) ), increase the selection ratio of long exposure images. According to this configuration, a result of WDR processing with less noise can be obtained, and the result of the WDR processing can be gradually switched over time.

(Example)

First, a functional configuration of the image processing apparatus 1 according to an embodiment of the present invention will be described. 3 is a diagram showing a functional configuration of an image processing apparatus 1 according to an embodiment of the present invention.

As shown in FIG. 3, the image processing apparatus 1 includes an image sensor 10, a frame memory 20, a use-image selection unit 40, a motion detection unit 50, a WDR synthesis unit 60, and a gray scale. A compression unit 80 and a mixing unit 90 are provided. Hereinafter, functions of each functional block included in the image processing apparatus 1 will be described in detail in order.

The image processing apparatus 1 continuously photographs two images while changing the exposure setting of the image sensor 10. In the case of this embodiment, short exposure photography is performed first, followed by long exposure photography. Of course, long exposure photography may be performed first, followed by short exposure photography.

The short-exposure image and the long-exposure image photographed in this way are written in the frame memory 20 as a pair. The long-exposure image and the short-exposure image are photographed, and the photographed long-exposure image and the short-exposure image are written in the frame memory 20 in succession.

In addition, in the example shown in FIG. 3, the image processing device 1 has one common system for outputting a long exposure image and a short exposure image, and the image sensor 10 generates a long exposure image and a short exposure image. Output in time division. However, the image processing apparatus 1 may simultaneously output a long exposure image and a short exposure image. In this case, the image processing apparatus 1 may have a system for outputting a long exposure image from the image sensor 10 and a system for outputting a short exposure image. Each shutter time is determined by, for example, a dynamic range of an object to be photographed or an image sensor specification.

In addition, in the exemplary embodiment of the present invention, the terms short exposure image and long exposure image are used, but these terms do not limit the absolute exposure time of each of the two captured images. Accordingly, when two images having different exposure times are captured, an image having a relatively short exposure time among the two images corresponds to a short exposure image, and an image having a relatively long exposure time corresponds to a long exposure image.

The image sensor 10 is constituted by an image image sensor that forms an image of light from outside on a light-receiving plane of an image pickup device, photoelectrically converts the imaged light into an amount of charge, and converts the amount of charge into an electric signal. The type of the image image sensor is not particularly limited, and may be, for example, a CCD (Charge 　 Coupled 　 De `) or a CMOS (Complementary 　 Metal 　 Oxide 　 Semiconductor).

For example, the image sensor 10 takes an exposure ratio of a predetermined magnification (number to tens of times) and detects (photographs) a short exposure image and a long exposure image. When the subject becomes dark, the image sensor 10 electronically amplifies and outputs an image signal, and provides information on the amplification degree (hereinafter, also referred to as “signal amplification degree”) to the mixing unit 90.

The use-image selection unit 40 detects the saturation state or movement of each of the long exposure image and the short exposure image by referring to the short exposure image and the long exposure image read out from the frame memory 20, and the short exposure image and the Use-image selection information for selecting any one of the long exposure images as a use-image is generated. There are various algorithms as an algorithm for selecting either a short exposure image or a long exposure image.

For example, the use-image selection unit 40 may perform a short exposure image and a long exposure image according to the relationship between the pixel value of the long exposure image or the short exposure image and the saturation detection threshold controlled by the threshold value control unit 30. In order to select any one of the exposed images for each pixel, use-image selection information is generated.

In more detail, the use-image selector 40 may select a short exposure image as a use-image of the pixel in the case of a pixel having a pixel value exceeding the saturation detection threshold in the long exposure image. In addition, the use-image selector 40 may select a long exposure image as a use-image of the pixel in the case of a pixel having a pixel value lower than the saturation detection threshold in the short exposure image.

The motion detection unit 50 detects motion and generates motion detection information. Although the motion detection method is not limited, in the case of detecting motion from a short exposure image and a long exposure image, it is preferable to calculate a difference after normalizing by multiplying a gain according to an exposure ratio for any one image. In addition, the method of generating the motion detection information is not particularly limited. For example, the motion detection unit 50 may generate motion detection information according to a relationship between a motion detected according to a short exposure image and a long exposure image and a predetermined amount of motion.

Specifically, in the short exposure image and the long exposure image, the motion detection unit 50 may detect a difference between a pixel value or a slope of a corresponding region, and detect a region having a difference greater than a predetermined amount of motion as a motion region. . Meanwhile, the motion detection unit 50 may detect an area in which the difference is smaller than a predetermined amount of motion as a non-motion area. An area in which the difference is equal to the predetermined amount of motion may be detected as any one area. The motion detection unit 50 may generate such a detection result as motion detection information.

The WDR synthesizer 60 generates a WDR synthesized image by synthesizing a short exposure image and a long exposure image according to the use-image selection information generated by the use-image selection unit 40. Specifically, the WDR synthesizing unit 60 generates a composite image by using a short exposure image in a short exposure image use area and a long exposure image in a long exposure image use area with reference to the use-image selection information. When synthesizing, it is preferable that any one image is normalized by multiplying the gain according to the exposure ratio before synthesizing.

Here, the WDR synthesizing unit 60 may synthesize a short exposure image and a long exposure image according to the use-image selection information generated by the use-image selection unit 40. However, with such processing alone, artifacts such as a double contour may occur in a region with large motion. Therefore, the WDR synthesizing unit 60 may perform processing to reduce the phenomenon that the contour becomes double due to movement. An algorithm for selecting one of a short exposure image and a long exposure image including such processing is not particularly limited. In addition, when the resolution of the image sensor 10 is 12 bits, each pixel of the WDR synthesized image may be extended to about 16 bits.

The gradation compression unit 80 performs compression processing for obtaining a bit range of a video signal having a wide dynamic range within a predetermined bit range on the WDR synthesized image generated by the WDR synthesis unit 60. In this compression process, tone mapping according to a look-up table (LUT) may be used, but is not particularly limited thereto, and any method may be used.

The rear end of the gradation compression unit 80 is connected to an image processing engine including, for example, a demosaic unit that generates an RGB frame from Bayer data, an outline enhancement unit, and color management. Accordingly, it is preferable that the amount of data of the output signal from the grayscale compression unit 80 is adjusted (eg, about 12 bits) to suit the size of the input data to the image processing engine, for example. Since it is only converted into a dark image by simply reducing the data size, it is preferable that the high luminance side is strongly compressed so as to approximate the visual characteristics of humans.

The blending unit 90 calculates a blending ratio of the grayscale compression processing result output from the grayscale compression unit 80 and the long-exposure image retrieved from the frame memory 20 according to the signal amplification degree of the image sensor 10, and calculates The result of the grayscale compression processing and the long exposure image are mixed according to one mixing ratio.

The calculation of the blending ratio of the WDR synthesis result for the long exposure image (hereinafter referred to as "WDR synthesis-result ratio") will be described in detail with reference to FIG. 4.

When the subject is bright, the image sensor 10 does not do any signal amplification. Therefore, the mixing unit 90 uses 100% (%) of the WDR synthesis result generated by the WDR synthesis unit 60 according to information indicating that there is no signal amplification input from the image sensor 10. The result ratio is calculated, and the WDR synthesis result generated by the WDR synthesis unit 60 is output as the final result of the WDR.

On the other hand, when the subject becomes dark, the image sensor 10 performs signal amplification. Therefore, when the signal amplification degree of the image sensor 10 falls within the range of a predetermined section, the mixing unit 90 reduces the WDR synthesis-result ratio according to an increase in the signal amplification degree of the image sensor 10, and Control to increase the mixing ratio. In addition, when the signal amplification degree of the image sensor 10 reaches the maximum value, the blending unit 90 sets the blending ratio of the long-exposed image to 100 percent (%) to eliminate the effect of noise, which is particularly pronounced in the short-exposure image.

For example, in a situation in which the brightness of the subject gradually decreases, the signal amplification degree of the image sensor 10 gradually increases, and accordingly, the WDR synthesis-result ratio gradually decreases (since the selection ratio of the long exposure image gradually increases). In addition, the effect of the WDR synthesis treatment can be gradually changed over time. In addition, according to this configuration, it is possible to output the same image as the long exposure image from the mixing unit 90 when WDR is off.

In the example shown in FIG. 3, the signal amplification degree of the image sensor from the image sensor 10 is directly output to the mixing unit 90. For example, when the entire mixing unit 90 is configured by hardware, or when implemented by executing software, such a configuration may be provided. However, a configuration for controlling the WDR synthesis-result ratio among the mixing unit 90 may be configured by the firmware. In this case, the signal amplification degree by the image sensor 10 is uploaded to the firmware from the image sensor 10, and the WDR synthesis-result ratio used by the mixing unit 90 may be controlled by the firmware. .

In addition, when the signal amplification degree by the image sensor 10 reaches the maximum value, an example in which the mixing ratio of the long exposure image is also set to 100 percent (%) has been described above. However, an object of an embodiment of the present invention is to make the noise of the WDR synthesis result inconspicuous. Therefore, when the noise included in the WDR composite image is not noticeable, it may be set to a value lower than 100 percent (%) as the blending ratio of the long exposure image.

Further, the timing of starting signal amplification in the image sensor 10 and the timing of starting to decrease the WDR synthesis-result ratio may not be simultaneously. As described above, the object of the embodiment of the present invention is to make the noise of the WDR synthesis result less pronounced. Therefore, even if the signal amplification of the image sensor 10 is started, the WDR synthesis-result ratio may not be changed when noise included in the WDR synthesized image is not prominent.

As described above, according to the exemplary embodiment of the present invention, when the exposure condition deteriorates and the amount of noise included in the short exposure image increases significantly, the effect of noise is reduced by increasing the selection ratio of the long exposure image during WDR synthesis. One WDR composite image can be obtained. In addition, the effect of the WDR synthesis process can be gradually switched over time according to the lighting environment or exposure state of the subject. In addition, according to this configuration, it is possible to output the same image as the long exposure image from the mixing unit 90 when WDR is off.

(theorem)

In an embodiment of the present invention, in accordance with the accident that WDR processing using a short exposure image is turned off in a situation in which the exposure state in which the signal is amplified by the image sensor is poor, the long exposure image according to the signal amplification degree of the image sensor Control the ratio of. For example, when the signal amplification degree of the image sensor is a predetermined period (for example, when the signal amplification degree of the image sensor is high), the ratio of the long exposure image is increased. Accordingly, noise included in the WDR-processed image can be gradually reduced. In addition, it is possible to gradually switch on and off the WDR processing in time according to changes in the lighting environment of the subject.

WDR (Wide Dynamic Range) technology is becoming important. In the prior art, when a short-exposure image is synthesized in a dark scene, it is a big problem that noise becomes very prominent. In addition, when WDR processing is attempted to be turned off in such a situation, a sudden change occurs in the processed image, and there has been a request to suppress such a change as much as possible.

According to an exemplary embodiment of the present invention, noise, which has been a problem in the related art, can be effectively reduced by increasing the use ratio of a long exposure image by referring to a signal amplification degree of an image sensor that is highly related to the lighting environment of a subject. In addition, it is also possible to gradually change the on (On) and off (Off) of the WDR process in time. A more natural WDR-processed image can be obtained in a configuration in which the mixing ratio of the WDR processing result and the long exposure image is changed. In addition, even when combined with two-dimensional gray scale processing, a configuration that is realized with a small amount of used memory has been proposed.

Not only can the conventional problem be effectively solved, but also the fact that it can be realized without performance degradation such as a reduction in frame rate, and that the configuration is simple and can be realized in a small circuit scale are also excellent points.

Hereinafter, preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, but the present invention is not limited to these examples. It is clear that a person of ordinary skill in the technical field to which the present invention pertains can reach various examples of changes or modifications within the scope of the technical idea described in the claims, and for these, of course, the present invention It is understood to be within the technical scope of.

It is likely to be used in all fields related to image processing.

1: image processing device, 10: image sensor,
20: frame memory, 40: use-video selector,
50: motion detection unit, 60: WDR synthesis unit,
80: gradation compression unit, 90: mixing unit.

Claims (3)

A use-image selector for generating use-image selection information to select one of a short exposure image and a long exposure image detected by the image sensor for each pixel;
A synthesis unit for synthesizing the short exposure image and the long exposure image according to the use-image selection information to obtain a synthesized image; And
According to the signal amplification degree of the image sensor, a mixing ratio of the composite image and the long exposure image obtained by combining the short exposure image and the long exposure image is calculated, and the short exposure image and the long exposure image are calculated according to the mixing ratio. Containing, a mixing unit for mixing the synthesized composite image and the long exposure image. The method of claim 1, wherein the mixing unit,
The image processing apparatus, wherein a mixing ratio of the composite image to the long exposure image is reduced according to an increase in the signal amplification degree in a predetermined section of the signal amplification degree. Generating use-image selection information to select one of a short exposure image and a long exposure image detected by the image sensor for each pixel;
Obtaining a synthesized image by synthesizing the short exposure image and the long exposure image according to the use-image selection information; And
According to the signal amplification degree of the image sensor, a mixing ratio of the composite image and the long exposure image obtained by combining the short exposure image and the long exposure image is calculated, and the short exposure image and the long exposure image are calculated according to the mixing ratio. Containing, mixing the synthesized composite image and the long exposure image.

Images