KR102282464B1 - Image processing apparatus and image processing method - Google Patents

Abstract

Amplifying units 61 and 62 for amplifying at least one short exposure image other than the long exposure image among the plurality of images according to respective amplification levels; a synthesizing unit 70 for synthesizing the long exposure image and the amplified short exposure image; and an amplification degree setting unit 30 configured to set a value less than the exposure ratio of the long exposure image to the short exposure image as the amplification degree.

Description

Image processing apparatus and image processing method

The present invention relates to an image processing apparatus and an image processing method.

The dynamic range that the image sensor can record is expanded compared to the conventional one, and the shooting function called WDR (Wide Dynamic Range) or HDR (High Dynamic Range), which can express a dynamic range close to or higher than that of the human eye, is increasingly used. there is. Such a shooting function can be particularly effective in a scene with a fairly large contrast ratio, such as a composition in backlight.

In recent years, the dynamic range that can be synthesized is increasing, and it is desirable that visibility is improved in more scenes. However, since the scale of the signal after synthesizing is dramatically increased, an increase in the circuit scale according to an increase in the number of bits of the signal, an increase in cost and an increase in power consumption are problematic. As a method for rationalizing the circuit scale of WDR synthesis processing, the following technique is disclosed.

For example, a technique capable of suppressing the number of bits of a synthesized signal by changing the amplification level (Gain) according to the brightness of each captured image and synthesizing it (for example, Japanese Patent Laid-Open No. 2004-266347, Patent Documents hereinafter) 1) is disclosed. In this technique, as the pixel value increases, a gradually smaller amplification is applied, so compared to the conventional example in which the exposure ratio is uniformly multiplied over the entire screen, the maximum value of the pixel value after synthesizing can be significantly suppressed while synthesizing the boundary of the image smoothly. can In addition, in this technology, even with respect to a bare signal, the amplification degree is determined by extracting the representative value of RGB in the 2×2 pixel, and the amplification degree is applied to the area from which the representative value is extracted, so that the process can be performed while maintaining the color balance. can

In addition, a technique capable of holding data of a wide range with a small number of bits by expressing a multi-bit signal with mantissa and exponent (for example, Patent Document of Japanese Patent Laid-Open No. 2008-236726, hereinafter referred to as Patent Document 2) This is disclosed. Once the signal is synthesized using the exposure ratio, by using this technique, the number of bits of the signal can be reduced and the load of processing after the synthesis is reduced.

In addition, a technique of determining which image pixel value to be synthesized by referring to one image from a group of images of different exposures and selecting a pixel value according to the pixel value (e.g., Patent Document of Japanese Patent Application Laid-Open No. 2012-151732, the following patents) Reference 3) is also disclosed.

Japanese Patent Laid-Open No. 2004-266347 Japanese Patent Laid-Open No. 2008-236726 Japanese Patent Laid-Open No. 2012-151732

However, in the technique described in Patent Document 1, when noise is included in a bare signal, it is a problem that noise is emphasized in units of a 2×2 pixel area. In addition, since processing is performed in units of 2x2 pixels, when implemented in hardware, one line of memory is required for calculating the amplification degree.

Here, a line of the memory requires an exposure amount equal to the number of different signals. For example, if three types of signals are synthesized, three lines are needed. In addition, in order to reduce the above-mentioned noise enhancement problem, a range wider than 2×2 may be referred to, which further increases the amount of memory. This is problematic when the constraints on circuit scale are difficult.

Further, according to the technique described in Patent Document 2, when the pixel value exceeds the range of the mantissa, there is a problem in that the data is not clear and the smoothness of the image is impaired. For example, if the mantissa is 8 bits, 10 is the radix, and the exponent is 4 bits, when the pixel value is 256 or more, it changes every 10, when the pixel value is 2,560 or more, it changes every 100, and when the pixel value is 25,600 or more Since it changes every 1,000, the precision decreases.

Further, according to the technique described in Patent Document 3, when the dynamic range is quite large, there is a problem in that the quality of the synthesized image is highly likely to be deteriorated. This is because it is difficult to determine the exposure state of all images from one captured image.

Accordingly, an object of the present invention is to provide a technology capable of generating a composite image while suppressing a decrease in smoothness of the composite image and suppressing an increase in circuit scale.

The image processing apparatus of the first aspect of the present invention includes an amplification unit, a synthesis unit, and an amplification degree setting unit.

The amplification unit amplifies the at least one short exposure image from among a plurality of images including a long exposure image and at least one short exposure image according to an amplification degree.

The synthesizing unit synthesizes the long exposure image and the amplified short exposure image.

The amplification degree setting unit sets a value less than an exposure ratio of the long exposure image to the short exposure image as the amplification degree.

In the image processing apparatus of the first aspect, a value smaller than an exposure ratio of the long exposure image to the short exposure image is applied as the amplification degree. Accordingly, by suppressing the number of bits of the synthesized image, it is possible to generate the synthesized image while suppressing an increase in circuit scale.

The image processing apparatus of the second aspect of the present invention includes an amplifying unit, a synthesizing unit, and an amplification degree setting unit.

The amplification unit amplifies the at least one short exposure image from among a plurality of images including a long exposure image and at least one short exposure image according to an amplification degree.

The synthesizing unit synthesizes the long exposure image and the amplified short exposure image.

The amplification degree setting unit may be configured to set the exposure ratio as an amplification degree of the short exposure image when the exposure ratio of the long exposure image to the short exposure image is not greater than a threshold value, and the exposure ratio is greater than the threshold value. In this case, a value less than the exposure ratio is set as the amplification degree of the short exposure image.

According to the image processing apparatus of the second aspect, when the exposure ratio is greater than the threshold value, a value less than the exposure ratio is applied as an amplification degree of the short exposure image. Accordingly, by suppressing the number of bits of the synthesized image, it is possible to generate the synthesized image while suppressing an increase in circuit scale.

Furthermore, when the exposure ratio of the long exposure image to the short exposure image is not greater than a threshold value, the exposure ratio is applied as an amplification degree of the short exposure image. Accordingly, deterioration of the smoothness of the composite image can be prevented.

An image processing method of a third aspect of the present invention includes amplifying the at least one short exposure image from among a plurality of images including a long exposure image and at least one short exposure image according to an amplification degree. The long exposure image and the amplified short exposure image are synthesized. A value smaller than the exposure ratio of the long exposure image to the short exposure image is set as the amplification degree.

According to the image processing method of the third aspect, a value smaller than an exposure ratio of the long exposure image to the short exposure image is applied as the amplification degree. Accordingly, by suppressing the number of bits of the synthesized image, it is possible to generate the synthesized image while suppressing an increase in circuit scale.

As described above, according to the embodiment of the present invention, when the exposure ratio is greater than the threshold value, a value less than the exposure ratio is applied as the amplification degree of the short exposure image. Accordingly, by suppressing the number of bits of the synthesized image, it is possible to generate the synthesized image while suppressing an increase in circuit scale.

Furthermore, when the exposure ratio of the long exposure image to the short exposure image is not greater than a threshold value, the exposure ratio is applied as an amplification degree of the short exposure image. Accordingly, deterioration of the smoothness of the composite image can be prevented.

1 is a diagram for explaining a general WDR synthesis technique.
FIG. 2 is a diagram for explaining a case in which values smaller than an exposure ratio are set as an amplification degree.
3 is a diagram illustrating a functional configuration of an image processing apparatus according to an embodiment of the present invention.
4 is a diagram for explaining a case where the exposure ratio is small.
5 is a diagram illustrating control characteristics of an amplification degree according to an embodiment of the present invention.
6 is a diagram for explaining an example of synthesizing two frames.
7 is a diagram illustrating examples of each synthesized image when an exposure ratio is set as an amplification degree and when 1/2 of an exposure ratio is set as an amplification degree.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In addition, in the present specification and drawings, components having substantially the same functional configuration are given the same reference numerals, and thus redundant descriptions are omitted.

In addition, in this specification and drawings, a plurality of components having substantially the same functional configuration may be distinguished by assigning different alphabets after the same reference numerals. However, when there is no need to specifically distinguish each of a plurality of components having substantially the same functional configuration, only the same reference numerals are assigned.

(Common WDR synthesis technique)

An embodiment of the present invention relates to a technique for obtaining a wide dynamic range (WDR) signal by synthesizing a plurality of image signals photographed by changing exposure. A general WDR synthesis process will be described with reference to FIG. 1 .

1 shows three image signals (3 frames) of Long (long exposure image), Short2 (short exposure image), and Short1 (ultra-short exposure image) from the long exposure time, the exposure ratio of Long to Short2; An example of synthesizing the image signals by changing the exposure so that the exposure ratio of Short2 to Short1 becomes the exposure ratio Exp is shown. The WDR signal is a 12-bit signal having a value of 0 to 4095.

In synthesizing a plurality of image signals, in order to match the brightness with the long exposure image, Short2' is obtained by multiplying the short exposure image by the exposure ratio (Exp) of the long exposure image to the short exposure image, and Short2' is obtained. use to synthesize Similarly, for the ultra-short exposure image, Short1' is obtained by multiplying the exposure ratio (Exp) of the long exposure image to the ultra-short exposure image to the power of the square, and is synthesized using Short1'. Referring to this example, it can be seen that the maximum value of the WDR signal after synthesis is proportional to the square of the exposure ratio.

In order to synthesize a wider dynamic range, it is possible to increase the number of images to be synthesized, increase the exposure ratio, or both. Whichever method is adopted, the maximum value of the pixel value after video signal synthesis rises exponentially. That is, when synthesizing a wider dynamic range, the number of bits of a circuit required for synthesizing increases, resulting in an increase in circuit scale.

Accordingly, an embodiment of the present invention proposes a technique capable of generating a synthesized image while suppressing a decrease in smoothness of the synthesized image and suppressing an increase in circuit scale. More specifically, a scene with a very wide dynamic range is linearly synthesized up to a constant dynamic range, as shown in FIG. 1, paying attention to the fact that the occurrence frequency is low. On the other hand, for a scene having a dynamic range greater than or equal to a certain level, as in the example shown in FIG. 2 , an amplification degree smaller than the exposure ratio is applied to an image other than the long exposure image.

(Example)

First, a functional configuration of the image processing apparatus 1 according to an embodiment of the present invention will be described. 3 shows a functional configuration of an image processing apparatus 1 according to an embodiment of the present invention.

Referring to FIG. 3 , the image processing apparatus 1 includes an image sensor 10 , a frame memory 20 , an amplification level setting unit 30 , a clip 31 , a clip 32 , and a use-image selection unit 40 , a motion detecting unit 50 , an amplifying unit 61 , an amplifying unit 62 , a synthesizing unit 70 , and a gradation compression unit 80 . Hereinafter, the function of each functional block included in the image processing apparatus 1 will be described in detail in turn.

The image processing apparatus 1 changes the exposure setting of the image sensor 10 to continuously photograph three images. In the present embodiment, ultra-short exposure imaging is performed first, followed by short exposure imaging, and then long exposure imaging is performed. However, long exposure imaging may be performed first, then short exposure imaging may be performed, and then ultra short exposure imaging may be performed.

The ultra-short exposure image, the short exposure image, and the long exposure image photographed in this way are stored in the frame memory 20 as a combination. The shooting of the ultra-short exposure image, the short exposure image and the long exposure image, and the writing of the captured ultra-short exposure image, the short exposure image and the long exposure image to the frame memory 20 are successively performed.

In addition, in the example shown in FIG. 3 , the image processing apparatus 1 has one common system for outputting an ultra-short exposure image, a short exposure image, and a long exposure image, and the image sensor 10 is configured to generate an ultra-short exposure image. , a short exposure image, and a long exposure image are output in time division. However, the ultra-short exposure image, the short exposure image, and the long exposure image may be simultaneously output. In this case, the image processing apparatus 1 may have a system for outputting an ultra-short exposure image from the image sensor 10 , a system for outputting a short exposure image, and a system for outputting a long exposure image. Each shutter time is determined by, for example, a dynamic range of a photographing target, an image sensor specification, or the like.

In addition, in the embodiment of the present invention, two types of short-exposure images having different exposure times as images other than the long-exposure image are captured by the image sensor 10 . However, an image other than the long exposure image captured by the image sensor 10 is not limited to two types of short exposure images having different exposure times. For example, as described below, the image other than the long exposure image captured by the image sensor 10 may be only one type of short exposure image. Alternatively, the image other than the long exposure image may be three or more types of short exposure images having different exposure times.

In addition, in the embodiment of the present invention, the terms ultra-short exposure image, short exposure image, and long exposure image are used, but these terms do not limit the absolute exposure time of each of the three captured images. Accordingly, when three images with different exposure times are captured, the image with the shortest exposure time among the three images corresponds to the ultra-short exposure image, and the image with the next short exposure time corresponds to the short exposure image, and the exposure time corresponds to the short exposure image. The image with the longest time corresponds to the long exposure image.

The image sensor 10 forms an image of light from the outside on a light receiving plane of the imaging device, photoelectrically converts the imaged light into an electric charge, and converts the electric charge into an electric signal. The type of the image sensor 10 is not particularly limited, and may be, for example, a CCD (Charge   Coupled   Dec. For example, the image sensor 10 detects (photographs) the ultra-short exposure image, the short exposure image, and the long exposure image by taking the exposure ratio Exp of a predetermined magnification (eg, several to several tens of times).

The use-image selection unit 40 refers to the ultra-short-exposure image, the short-exposure image, and the long-exposure image read out from the frame memory 20 to determine the saturation state or movement of each of the long-exposure image, the short-exposure image and the ultra-short-exposure image. etc., and generate use-image selection information for selecting any one of an ultra-short exposure image, a short exposure image, and a long exposure image as a use image for each pixel. Algorithms for selecting any one of the ultra-short exposure image, the short exposure image, and the long exposure image are various.

For example, the use-image selection unit 40 selects any one of the ultra-short exposure image, the short exposure image, and the long exposure image, the relationship between the pixel values of the ultra-short exposure image, the long exposure image, or the short exposure image and the prescribed value. By selecting for each pixel according to , use-image selection information is generated. For example, the use-image selection unit 40 may select a short exposure image as a use image of a pixel having a pixel value greater than an upper limit value in the long exposure image. Alternatively, for example, the use-image selection unit 40 may select a long exposure image as a use image of a pixel having a pixel value less than a lower limit value in the short exposure image.

The motion detection unit 50 detects a motion. Here, the motion detection method is not limited.

For example, the motion detector 50 may detect a difference in pixel values or inclinations of corresponding regions in the short exposure image and the long exposure image, and detect a region in which the difference is greater than a predetermined amount of motion as the motion region. . Meanwhile, the motion detector 50 may detect an area in which the difference is less than a predetermined amount of motion as a non-motion area. A region in which the difference is equal to a predetermined amount of movement may be detected as any one region. The motion detection unit 50 may obtain such a motion region and a non-motion region as a result of motion detection.

Also, the motion detector 50 may detect a difference in pixel values or inclinations of corresponding regions in the ultra-short exposure image and the short exposure image, and detect a region in which the difference is greater than a predetermined motion amount as the motion region. Meanwhile, the motion detector 50 may detect an area in which the difference is less than a predetermined amount of motion as a non-motion area. A region in which the difference is equal to a predetermined amount of movement may be detected as any one region. The motion detection unit 50 may obtain such a motion region and a non-motion region as a result of motion detection. Alternatively, the motion detection unit 50 may integrate the two motion detection results.

The amplification unit 61 amplifies the ultra-short exposure image according to the amplification degree. Specifically, the amplification unit 61 amplifies the ultra-short exposure image by multiplying the amplification degree by the ultra-short exposure image. In general, the exposure ratio of the long exposure image to the ultrashort exposure image is set as an amplification degree, and the ultrashort exposure image is amplified by multiplying the amplification degree by the ultrashort exposure image. In the embodiment of the present invention, the amplification level setting unit 30 can appropriately control the amplification level.

Similarly, the amplification unit 62 amplifies the short exposure image according to the amplification degree. Specifically, the amplification unit 62 amplifies the short exposure image by multiplying the amplification degree with respect to the short exposure image. In general, the exposure ratio of the long exposure image to the short exposure image is set as an amplification degree, and the short exposure image is amplified by multiplying the amplification degree by the short exposure image. In the embodiment of the present invention, the amplification level setting unit 30 may appropriately control the amplification level.

The synthesizing unit 70 generates a WDR synthesized image by synthesizing the ultra-short exposure image, the short exposure image, and the long exposure image for each pixel according to the use-image selection information generated by the use-image selection unit 40 .

Specifically, the synthesizing unit 70 uses the ultra-short exposure image for the ultra-short exposure-image use region, uses the short-exposure image for the short exposure-image use region, and the long exposure with reference to the use-image selection information. - Create a composite image using the long exposure image in the image use area. When synthesizing, it is preferable to synthesize one image after normalizing it by multiplying the amplification level according to the exposure ratio.

In this way, the synthesizing unit 70 synthesizes the ultra-short exposure image, the short exposure image, and the long exposure image according to the use-image selection information generated by the use-image selection unit 40 . However, this processing alone may cause artifacts such as double contours in the motion region.

Accordingly, the synthesizing unit 70 may perform processing for reducing the phenomenon in which the outline becomes double in the motion region. For example, the synthesizing unit 70 may select an ultra-short exposure image or a short exposure image as an image used for the motion region. Also, when the resolution of the image sensor 10 is 12 bits, each pixel of the WDR composite image may be extended to about 16 bits.

The gradation compression unit 80 performs compression processing on the WDR synthesized image generated by the synthesizing unit 70 in order to convert the bit range of the video signal having a wide dynamic range into a predetermined bit range. As such compression processing, tone mapping according to a look-up table (LUT) may be used, but is not particularly limited thereto, and any method may be used.

The rear end of the gradation compression unit 80 is connected to an image processing engine including, for example, a demosaicing unit for generating RGB frames from bare data, an outline enhancement unit, and color management. Accordingly, it is preferable that the output data from the grayscale compression unit 80 be adjusted (eg, about 12 bits) to fit, for example, the size of input data to the image processing engine. Since a dark image is converted to a dark image by simply lowering the data size, it is preferable that the high luminance side be strongly compressed so as to approximate human visual characteristics.

A specific example of amplification degree control will be described with reference to FIG. 2 .

As shown in FIG. 2 , the amplification level setting unit 30 sets the amplification level multiplied by Short2 (short exposure image) not the original amplification level (exposure ratio Exp), for example, a value of 1/2 thereof. can also be set to When the amplification level is set in this way, a step occurs at the boundary between Short2' (short exposure image multiplied by the amplification level) and Long (long exposure image), and this step means that the composite boundary in the composite image is not reproduced smoothly. Accordingly, the periphery of a bright object appears somewhat dark, or streaks appear in gradation.

However, if this step is too large (for example, as shown in Fig. 2, when the amplification level multiplied for Short2 (short exposure image) is about 1/2 of the original amplification level (exposure ratio (Exp))) , it can be seen that it is not a big problem because the sense of incongruity is reduced in the final image. 7 shows examples of each synthesized image when the amplification level is set to the exposure ratio (Exp) and when the amplification level is set to 1/2 of the exposure ratio (Exp). The amplification degree is not limited to 1/2 of the exposure ratio Exp, and any value smaller than the exposure ratio Exp may be sufficient.

In addition, the amplification level setting unit 30 may set the amplification level multiplied by Short1 (ultra-short exposure image) to a value of, for example, 1/8 of the original amplification level (the square of the exposure ratio Exp). there is. When the amplification level is set in this way, a step is created at the boundary between Short1' (short exposure image multiplied by the amplification level) and Short2' (short exposure image multiplied by the amplification level). Since it is strongly compressed at high luminance, the step difference is reduced and becomes insignificant.

Therefore, even if the step between Short1' and Short2' is made larger than the step between Short2' and Long (for example, as in the example shown in FIG. 2, the amplification degree multiplied with respect to Short1 (ultra-short exposure image) is amplified by the original (when it is set to about 1/8 of the square of the exposure ratio (Exp)), the sense of incongruity in the final image is not conspicuous. In addition, since there are few scenes of shooting an ultra-high luminance object that is not exposed properly unless it is Short1, the frequency of occurrence of visual problems is also low. In addition, the amplification degree is not limited to 1/8 of the square of the exposure ratio Exp, and may be any value smaller than the exposure ratio Exp.

In this way, in the example shown in FIG. 2 , the maximum value of the synthesized image can be suppressed to 1/8 of the original by applying an amplification degree smaller than the original exposure ratio and synthesizing. That is, in the example shown in FIG. 2, the size of the synthesis circuit can be saved by 3 bits. In addition, since it is only necessary to control the degree of amplification to be applied, there is no need for complicated calculations or calculations requiring the use of line memory. In addition, by setting the amplification level to such an extent that the step generated at the boundary of the other images is not conspicuous, deterioration of the smoothness of the synthesized image can be suppressed.

The description will be continued with reference to FIG. 4 . 4 is a diagram showing the relationship between the horizontal axis (subject luminance) and the vertical axis (composite pixel value), respectively, "when exposure ratio = Exp" and "when exposure ratio = Exp'.

As shown in FIG. 4 , Exp' is less than Exp, and the subject luminance is lower in "when the exposure ratio = Exp' than "when the exposure ratio = Exp". In the example shown in FIG. 4 , when the exposure ratio = Exp', the maximum value of the synthesized image is the same as "when the exposure ratio = Exp" even if the exposure ratio Exp' is used as the amplification in a general method. Therefore, there is no need to intentionally use an amplification level smaller than Exp', and accordingly, a step difference is not generated at the boundary of another image.

That is, based on "when the exposure ratio = Exp' in FIG. 4 , the amplification level setting unit 30 sets the level of amplification when the exposure ratio is not greater than the threshold value (or the exposure ratio is equal to the threshold value). You can set the exposure ratio.

In addition, based on "when the exposure ratio = Exp" in FIG. 4 , the amplification degree setting unit 30 clips the amplification degree to a value smaller than the exposure ratio when the exposure ratio is greater than the threshold value.

The control characteristic of the degree of amplification with respect to the exposure ratio is shown in FIG. 5 .

For example, when the exposure ratio can be changed from 4 times to 16 times, the amplification level setting unit 30 sets the threshold value TH to "8 times". Accordingly, the same degree of amplification as the exposure ratio is applied in the range of 4 to 8 times the exposure ratio, but the amplification degree is clipped to 8 times the exposure ratio in the range where the exposure ratio is greater than 8 times and equal to or less than 16 times. can Then, when the exposure ratio is 16 times, the amplification degree becomes 8 times the exposure ratio, and the ratio of the amplification degree to the exposure ratio becomes 1/2.

In addition, as described above, in order to reduce artifacts such as double outlines in the motion region, a process for applying a short exposure image to the motion region (hereinafter referred to as “motion-adaptive synthesis process”) is sometimes carried out. When the motion-adaptive synthesis process is executed in this way, it is preferable that the amplification degree control of FIG. 5 is not applied to the short exposure image of the motion region.

More specifically, when the motion-adaptive synthesis processing is executed, the exposure ratio of the long-exposure image to the short-exposure image is set as the amplification degree of the short-exposure image in the motion region. This is because, if a value less than the exposure ratio of the long exposure image to the short exposure image is set as the amplification degree of the short exposure image in the motion region, the object to be photographed becomes dark at the moment the object to be photographed moves. Accordingly, the amplification degree setting unit 30 can switch whether to invalidate the control shown in Fig. 5 depending on whether the motion-adaptive synthesis processing is executed or not. That is, when the motion-adaptive synthesis process is not executed, the amplification degree control shown in FIG. 5 may be switched to be applied to the short-exposure image of the motion region.

Whether to execute the motion-adaptive synthesis processing may be variously set. For example, whether or not the motion-adaptive synthesis processing is to be executed is predetermined and may become non-switchable. Alternatively, whether or not the motion-adaptive synthesis processing is executed may be made switchable. In the case where it is possible to switch whether or not the motion-adaptive synthesis processing is executed, the switching may be performed in association with an operation by the user.

On the other hand, even in the case of shooting with the same exposure ratio, if the number of composites is different, the control of the amplification degree may be different. This will be described while comparing FIGS. 2 and 6 .

2 and 6 are examples of synthesizing a plurality of images taken at the same exposure ratio (Exp), but FIG. 2 shows an example of synthesizing three frames (long exposure image, short exposure image, and ultra short exposure image). , FIG. 6 shows an example of synthesizing two frames (long exposure image and short exposure image).

Here, as shown in FIG. 2 , in the case of 3-frame synthesis, an amplification degree smaller than the exposure ratio is applied to the short exposure image and the ultra short exposure image, and the upper bit limit of the system is not exceeded.

As shown in FIG. 6 , in the case of two-frame synthesis, even if the same amplification as the exposure ratio is applied to the short exposure image, the upper bit limit of the system is not reached, so there is no need to apply an amplification less than the exposure ratio. Therefore, even with the same exposure ratio, whether or not the amplification control of FIG. 5 is applied may be switched according to the number of synthesis.

The synthesized image is not limited to 2 frames and 3 frames. Accordingly, when the number of synthesis is less than the predetermined number, the amplification degree setting unit 30 may set the exposure ratio of the long exposure image to the used image as the amplification degree of the used image. In addition, when the number of composites is equal to or greater than a predetermined number and the exposure ratio of the long exposure image to the used image is not greater than the threshold value TH, the amplification degree setting unit 30 sets the amplification degree of the long exposure image for the used image as the amplification of the used image. You can set the exposure ratio of the image. And, when the number of composites is equal to or greater than a predetermined number and the exposure ratio of the long exposure image to the used image is greater than the threshold value TH, the amplification level setting unit 30 clips the amplification level of the used image to a value less than the exposure ratio. can be clipped.

As described above, according to the exemplary embodiment of the present invention, it is possible to obtain a smooth synthesized image with a linear characteristic with respect to a dynamic range that has a high frequency of occurrence or is assumed as a typical scene. In addition, for a scene having a fairly large dynamic range with a low occurrence frequency or importance, it is possible to cope with synthesis by suppressing an increase in circuit scale even if there is some deterioration in image quality.

(organize)

The image processing apparatus 1 of this embodiment includes amplification units 61 and 62 , a synthesis unit 70 , and an amplification degree setting unit 30 .

The amplification units 61 and 62 amplify at least one short-exposure image that is an image other than the long-exposure image among the plurality of images according to each amplification degree.

The synthesis unit 70 synthesizes the long exposure image and the amplified short exposure image.

The amplification degree setting unit 30 sets a value less than the exposure ratio of the long exposure image to the short exposure image as the amplification degree.

According to this configuration, it is possible to generate a synthesized image while suppressing a decrease in the smoothness of the synthesized image and suppressing an increase in circuit scale.

WDR (Wide Dynamic Range) technology is gaining importance. In the prior art, it is a big problem that the circuit scale increases as the dynamic range that can be synthesized increases. In the embodiment of the present invention, paying attention to the fact that the occurrence frequency of a scene with a fairly wide dynamic range is low, linear synthesis is performed up to a constant dynamic range, and for a scene having a dynamic range above a certain level, even if there is some deterioration in image quality, there is little By applying the amplification degree, it is possible to suppress an increase in the system scale.

According to the embodiment of the present invention, two conflicting problems of performance improvement and circuit size increase can be solved with a reasonable and simple configuration.

In the above, preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, but the present invention is not limited to these examples. It is clear that those of ordinary skill in the art to which the present invention pertains can arrive at various changes or modifications within the scope of the technical idea described in the claims, and of course the present invention It is understood to be within the technical scope of

It has the potential to be used in all fields related to image processing.

1: image processing device, 10: image sensor,
20: frame memory, 30: amplification setting unit,
31: clip, 32: clip,
40: use-image selection unit, 50: motion detection unit,
61: amplification unit, 62: amplification unit,
70: a synthesis part, 80: a gradation compression part.

Claims (3)

delete an amplifying unit amplifying the at least one short exposure image from among a plurality of images including a long exposure image and at least one short exposure image according to an amplification degree;
a synthesizing unit for synthesizing the long exposure image and the amplified short exposure image; and
When the exposure ratio of the long exposure image to the short exposure image is not greater than a threshold value, the exposure ratio is set as the amplification degree of the short exposure image, and when the exposure ratio is greater than the threshold value, the exposure ratio is higher than the exposure ratio and an amplification level setting unit configured to set a small value as an amplification level of the short-exposure image. amplifying the at least one short exposure image from among a plurality of images including a long exposure image and at least one short exposure image according to an amplification degree;
synthesizing the long exposure image and the amplified short exposure image; and
When the exposure ratio of the long exposure image to the short exposure image is not greater than a threshold value, the exposure ratio is set as the amplification degree of the short exposure image, and when the exposure ratio is greater than the threshold value, the exposure ratio is higher than the exposure ratio and setting a small value as the amplification degree of the short-exposure image.

Images