JP6563646B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method.

  The WDR (Wide Dynamic Range) or HDR (High Dynamic Range) imaging function is used to expand the dynamic range that the sensor can shoot, and to express a dynamic range that is close to or greater than that of the human eye. Opportunities are being increased. Such a photographing function can be particularly effective in a scene having a very large contrast ratio such as a composition of backlight.

  In recent years, the dynamic range that can be synthesized has been increasing, and it is a pleasure to have improved the visibility in more scenes. However, on the other hand, since the scale of the signal after synthesis is drastically expanded, an increase in the circuit scale due to an increase in the number of bits of the signal, a cost increase and an increase in power consumption associated therewith, are problems. The following technique is disclosed as a method for rationalizing the circuit scale of WDR synthesis processing.

  For example, a technique is disclosed that makes it possible to reduce the number of bits of a composite signal by combining gains with different gains according to the brightness of each captured image (see, for example, Patent Document 1). In such a technique, since a gradually smaller gain is applied as the pixel value becomes larger, compared with the conventional example in which the exposure ratio is uniformly applied to the entire screen, the combined pixel while smoothly synthesizing the boundary portion of the image. It is possible to greatly suppress the maximum value. Also, in such a technique, a gain is determined by extracting a representative value of RGB in 2 × 2 pixels for the Bayer signal, and the color balance is obtained by applying the gain to the region from which the representative value is extracted. Processing can be performed while maintaining.

  In addition, a technique is disclosed that allows a wide range of data to be held with a small number of bits by expressing a multi-bit signal with a mantissa and an exponent (see, for example, Patent Document 2). Once the signal is synthesized using the exposure ratio, if such a technique is used, the number of bits of the signal can be reduced, and the processing load after the synthesis is reduced.

  In addition, a technique is also disclosed in which one image is referred to from different exposure image groups, and the pixel value of which image is selected and combined according to the pixel value (see, for example, Patent Document 3). .

JP 2004-266347 A JP 2008-236726 A JP 2012-151732 A

  However, in the technique described in Patent Document 1, when noise is included in the Bayer signal, the problem is that the noise is emphasized in units of 2 × 2 pixels. In addition, since processing is performed in units of 2 × 2 pixels, one line of memory is required for gain calculation when realized by hardware. That requires an amount corresponding to the number of signals with different exposures. For example, if three types of signals are combined, three lines are required. Furthermore, if the above-described noise enhancement problem is to be reduced, a range wider than 2 × 2 may be referred to, and the amount of memory will further increase. These are problems when restrictions on the circuit scale are severe.

  In the technique described in Patent Document 2, when the pixel value exceeds the mantissa range, the data becomes coarse and the smoothness of the image is impaired. For example, when the mantissa is 8 bits, the base is 10 and the exponent is 4 bits, the pixel value changes to 10 when the pixel value is 256 or more, changes to 100 when the pixel value is 2560 or more, and when the pixel value is 25600 or more Since it changes in increments of 1000, the accuracy becomes coarse.

  Furthermore, in the technique described in Patent Document 3, when the dynamic range is very large, it is difficult to determine the exposure state of all images from one captured image, so that the quality of the composite image may be reduced. It is a difficult point that the property is high.

  Therefore, the present invention provides a technique capable of generating a composite image while suppressing a decrease in smoothness of the composite image and suppressing an increase in circuit scale.

  According to an embodiment of the present invention, an amplification unit that amplifies each of one or more short exposure images that are images other than the long exposure image among a plurality of images based on each gain value, and the long exposure image is amplified. An image processing apparatus comprising: a combining unit that combines the short exposure image; and a gain setting unit that sets a predetermined value smaller than an exposure ratio of the long exposure image to the short exposure image as the gain value. Provided. According to such a configuration, it is possible to generate a composite image while suppressing an increase in circuit scale by suppressing the number of bits of the composite image.

  The gain setting unit sets the exposure ratio as a gain value of the short exposure image when the short exposure image in which the exposure ratio falls below a threshold value is present in the one or more short exposure images. When there is a short exposure image in which the exposure ratio exceeds the threshold value in the short exposure image, the gain value of the short exposure image may be clipped to the threshold value. According to such a configuration, it is possible to suppress a decrease in smoothness of the composite image by not clipping the gain value when the composite image does not reach the system bit limit.

  The image processing apparatus includes a use image selection unit that selects one of the long exposure image and the one or more short exposure images as a use image for each pixel, and the combination unit combines the use image for each pixel. May be. According to such a configuration, the use image is selected by the use image selection unit.

  The image processing apparatus includes a motion detection unit that detects a motion region, and the gain setting unit performs a process of selecting one of the one or more short-exposure images as the motion region in the use image. In this case, an exposure ratio of the long exposure image to the use image may be set as the gain value of the use image. According to such a configuration, since a gain value smaller than the exposure ratio of the long exposure image to the short exposure image is not set as the gain value of the short exposure image in the motion region, the object may become dark at the moment the object moves. Can be prevented.

  When the process for selecting any one of the one or more short-exposure images as the motion region in the use image is not executed, the motion detection unit sets an exposure ratio of the long exposure image to the use image to the threshold value. If the exposure ratio is lower, the exposure ratio is set as the gain value of the use image, and if the exposure ratio of the long exposure image to the use image exceeds the threshold value, the gain value of the use image is clipped to the threshold value. May be. According to this configuration, it is possible to generate a composite image while suppressing an increase in circuit scale.

  The gain setting unit may set an exposure ratio of the long exposure image to the use image as a gain value of the use image when the plurality of images are less than a predetermined number. According to such a configuration, when the number of bits of the composite image is suppressed to be smaller than a certain level, it is possible to suppress a decrease in smoothness of the composite image.

  The gain setting unit, when the plurality of images is equal to or greater than the predetermined number, when the exposure ratio of the long exposure image to the use image is less than the threshold, the exposure ratio as the gain value of the use image. If the exposure ratio of the long exposure image to the use image exceeds the threshold value, the gain value of the use image may be clipped to the threshold value. According to such a configuration, when the number of bits of the composite image becomes larger than a certain level, it is possible to generate a composite image while suppressing an increase in circuit scale.

  According to an embodiment of the present invention, one or more short-exposure images that are images other than the long-exposure image among a plurality of images are amplified based on each gain value, and amplified with the long-exposure image There is provided an image processing method including combining the short exposure image and setting a predetermined value smaller than an exposure ratio of the long exposure image to the short exposure image as the gain value. According to such a method, it is possible to generate a composite image while suppressing an increase in circuit scale by suppressing the number of bits of the composite image.

  As described above, according to the present invention, it is possible to generate a composite image while suppressing a decrease in smoothness of the composite image and suppressing an increase in circuit scale.

It is a figure for demonstrating a general WDR synthetic | combination technique. It is a figure for demonstrating the case where the predetermined value smaller than exposure ratio is set as a gain value. It is a figure which shows the function structure of the image processing apparatus which concerns on embodiment of this invention. It is a figure for demonstrating the case where an exposure ratio is small. It is a figure which shows the control characteristic of the gain value which concerns on embodiment of this invention. It is a figure for demonstrating the example which synthesize | combines 2 frames. It is a figure which shows the example of a synthesized image in the case where exposure ratio is set as a gain value, and when 1/2 of exposure ratio is set as a gain value.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

  In the present specification and drawings, a plurality of constituent elements having substantially the same functional configuration may be distinguished by attaching different alphabets after the same reference numeral. However, when it is not necessary to particularly distinguish each of a plurality of constituent elements having substantially the same functional configuration, only the same reference numerals are given.

(General WDR synthesis technology)
Embodiments described herein relate generally to a technique for obtaining a wide dynamic range signal (WDR signal) by synthesizing a plurality of image signals shot with different exposures. A general WDR synthesis process will be described with reference to FIG. FIG. 1 shows three image signals (3 frames) of Long (long exposure image), Short 2 (short exposure image), and Short 1 (ultra short exposure image) from the longest exposure time, the exposure ratio of Long to Short 2, Short 1 In this example, the exposure is changed so that each of the exposure ratios of Short2 to the exposure ratio becomes the exposure ratio Exp, and these image signals are combined. The WDR signal is a 12-bit signal having a value of 0 to 4095.

  When synthesizing a plurality of image signals, Short2 ′ is obtained by multiplying the short exposure image by the exposure ratio Exp of the long exposure image to the short exposure image in order to align the brightness with the long exposure image. Use 'to compose. Similarly, Short1 'is obtained by multiplying the ultrashort exposure image by the square of the exposure ratio Exp of the long exposure image to the ultrashort exposure image, and is synthesized using Short1'. Referring to this example, it can be seen that the maximum value of the combined WDR signal is proportional to the square of the exposure ratio.

  In order to be able to synthesize a wider dynamic range, the number of synthesized image signals may be increased, the exposure ratio may be increased, or both may be employed. In any case, the maximum pixel value after the image signal is combined increases exponentially. That is, if it is possible to synthesize a wider dynamic range, the number of bits of the circuit required for synthesis increases, and the circuit scale becomes a big problem.

  Therefore, in the embodiment of the present invention, a technique capable of generating a composite image while suppressing a decrease in smoothness of the composite image and suppressing an increase in circuit scale is proposed. More specifically, attention is paid to the fact that a scene with a very wide dynamic range has a low frequency of occurrence, and linear synthesis is performed up to a certain dynamic range as in the example shown in FIG. On the other hand, a gain value smaller than the exposure ratio is applied to an image other than the long-exposure image, as in the example shown in FIG.

(Embodiment)
First, a functional configuration of the image processing apparatus 1 according to the embodiment of the present invention will be described. FIG. 3 is a diagram showing a functional configuration of the image processing apparatus 1 according to the embodiment of the present invention. As shown in FIG. 3, the image processing apparatus 1 includes a sensor 10, a frame memory 20, a gain setting unit 30, a clip 31, a clip 32, a use image selection unit 40, a motion detection unit 50, an amplification unit 61, an amplification unit 62, A synthesis unit 70 and a gradation compression unit 80 are provided. Hereinafter, the function of each functional block included in the image processing apparatus 1 will be described in detail sequentially.

  The image processing apparatus 1 changes the exposure setting of the sensor 10 and continuously shoots three images. Here, the ultra-short exposure shooting is performed first, followed by the short exposure shooting, and then the long exposure shooting. Shall be performed. However, long exposure photography may be performed first, followed by short exposure photography, followed by ultrashort exposure photography. The ultra-short exposure image, the short exposure image, and the long exposure image shot in this way are written in the frame memory 20 as one combination. The photographing of the ultrashort exposure image, the short exposure image, and the long exposure image and the writing of the photographed ultrashort exposure image, the short exposure image, and the long exposure image to the frame memory 20 are continuously performed.

  In the example shown in FIG. 3, the image processing apparatus 1 has one common system for outputting an ultrashort exposure image, a short exposure image, and a long exposure image, and the sensor 10 has an ultrashort exposure image. Although the short exposure image and the long exposure image are output in a time division manner, the ultra short exposure image, the short exposure image, and the long exposure image may be output simultaneously. In such a case, the image processing apparatus 1 may have a system for outputting an ultrashort exposure image from the sensor 10, a system for outputting a short exposure image, and a system for outputting a long exposure image. Each shutter time is determined by, for example, a dynamic range of a subject to be imaged, a sensor specification, and the like.

  In the embodiment of the present invention, an example in which two types of short exposure images having different exposure times as images other than the long exposure image are captured by the sensor 10 will be mainly described. However, images other than the long exposure image photographed by the sensor 10 are not limited to two types of short exposure images with different exposure times. For example, as described below, the image other than the long exposure image captured by the sensor 10 may be only one type of short exposure image. Alternatively, the images other than the long exposure image may be three or more types of short exposure images having different exposure times.

  In the embodiments of the present invention, the terms ultra-short exposure image, short exposure image, and long exposure image are used, but these terms limit the absolute exposure time of each of the three captured images. It is not a thing. Therefore, when three images with different exposure times are taken, among the three images, the image with the shortest exposure time corresponds to the ultrashort exposure image, and the image with the next shortest exposure time becomes the short exposure image. The image with the longest exposure time corresponds to the long exposure image.

  The sensor 10 is configured by an image sensor that forms an image of light from the outside on the light receiving plane of the image sensor, photoelectrically converts the imaged light into a charge amount, and converts the charge amount into an electric signal. The type of the image sensor is not particularly limited, and may be, for example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). For example, the sensor 10 detects (photographs) an ultrashort exposure image, a short exposure image, and a long exposure image with an exposure ratio Exp of a predetermined magnification (for example, several times to several tens of times).

  The use image selection unit 40 refers to the ultra-short exposure image, the short exposure image, and the long exposure image read from the frame memory 20, and determines the saturation state and movement of each of the long exposure image, the short exposure image, and the ultra short exposure image. Detection is performed, and use image selection information for selecting any one of the ultra-short exposure image, the short exposure image, and the long exposure image as the use image for each pixel is generated. Various algorithms are assumed as an algorithm for selecting one of the ultra-short exposure image, the short exposure image, and the long exposure image.

  For example, the use image selection unit 40 selects one of the ultrashort exposure image, the short exposure image, and the long exposure image based on the relationship between the pixel value of the ultrashort exposure image, the long exposure image, or the short exposure image and the specified value. Use image selection information is generated by selecting each pixel. For example, the use image selection unit 40 may select a short exposure image as a use image of a pixel having a pixel value exceeding the upper limit value in the long exposure image. Alternatively, for example, the use image selection unit 40 may select a long exposure image as a use image of a pixel having a pixel value lower than the lower limit value in the short exposure image.

  The motion detector 50 detects motion. The motion detection method is not limited. For example, the motion detection unit 50 may detect a difference between pixel values or gradients of corresponding regions in the short exposure image and the long exposure image, and detect a region where the difference is larger than a predetermined motion amount as a motion region. On the other hand, the motion detection unit 50 may detect a region where the difference is smaller than a predetermined motion amount as a non-motion region. An area where the difference is the same as the predetermined amount of motion may be detected as any area. The motion detection unit 50 may obtain such a motion region and a non-motion region as a motion detection result.

  In addition, the motion detection unit 50 detects a pixel value or gradient difference in a corresponding region in the ultrashort exposure image and the short exposure image, and detects a region where the difference is larger than a predetermined motion amount as a motion region. Good. On the other hand, the motion detection unit 50 may detect a region where the difference is smaller than a predetermined motion amount as a non-motion region. An area where the difference is the same as the predetermined amount of motion may be detected as any area. The motion detection unit 50 may obtain such a motion region and a non-motion region as a motion detection result. Alternatively, the motion detection unit 50 may integrate two motion detection results.

  The amplifying unit 61 amplifies the ultrashort exposure image based on the gain value. Specifically, the amplifying unit 61 amplifies the ultrashort exposure image by multiplying the ultrashort exposure image by a gain value. In general, the ultrashort exposure image is amplified by multiplying the ultrashort exposure image by using the exposure ratio of the long exposure image to the ultrashort exposure image as a gain value. However, in the embodiment of the present invention, the gain setting unit 30 can appropriately control this gain value.

  Similarly, the amplifying unit 62 amplifies the short exposure image based on the gain value. Specifically, the amplifying unit 62 amplifies the short exposure image by multiplying the short exposure image by a gain value. In general, the short exposure image is amplified by multiplying the short exposure image by using the exposure ratio of the long exposure image to the short exposure image as a gain value. In the embodiment of the present invention, the gain setting unit 30 It is possible to appropriately control the gain value.

  The combining unit 70 generates a WDR combined image by combining the ultrashort exposure image, the short exposure image, and the long exposure image for each pixel based on the use image selection information generated by the use image selection unit 40. Specifically, the composition unit 70 refers to the use image selection information, uses the ultrashort exposure image in the ultrashort exposure image use region, uses the short exposure image in the short exposure image use region, In the exposure image use area, a composite image is generated using a long exposure image. At the time of composition, it is preferable to normalize one of the images by multiplying by a gain corresponding to the exposure ratio.

  At this time, the combining unit 70 may combine the ultra-short exposure image, the short exposure image, and the long exposure image according to the use image selection information generated by the use image selection unit 40. Artifacts such as double contours in the region can occur. Therefore, the synthesis unit 70 may perform processing for reducing a phenomenon in which the contour becomes double in the motion region. Specifically, the synthesis unit 70 may select an ultrashort exposure image or a short exposure image as the use image of the motion region. When the resolution of the sensor 10 is 12 bits, each pixel of the WDR composite image may be expanded to about 16 bits.

  The gradation compressing unit 80 performs compression processing on the WDR synthesized image generated by the synthesizing unit 70 so that the bit range of the image signal having a wide dynamic range falls within a predetermined bit range. As such compression processing, tone mapping according to a look-up table (LUT) may be used, but any method may be used.

  The subsequent stage of the gradation compression unit 80 is connected to an image processing engine including, for example, a demosaic unit that generates an RGB plane from Bayer data, a contour enhancement unit, and color management. For this reason, it is preferable that the data amount of the output signal from the gradation compression unit 80 is adjusted so as to match the size of the input data to the image processing engine (for example, about 12 bits). If the data size is simply reduced, the image is converted into a dark image. Therefore, it is preferable that the high luminance side is strongly compressed so as to approximate human visual characteristics.

  A specific example of gain value control will be described with reference to FIG. As shown in FIG. 2, the gain setting unit 30 sets the gain value to be multiplied to Short2 (short exposure image) to a value that is ½ of the gain value (exposure ratio Exp) instead of the original gain value (exposure ratio Exp). It's okay. When the gain value is set in this way, a step is formed at the boundary between Short 2 ′ (short exposure image multiplied by the gain value) and Long (long exposure image), but this step smoothes the composite boundary in the composite image. It means that it is not reproduced. Therefore, the periphery of a bright object looks somewhat dark or a stripe pattern appears in the gradation.

  However, if this step is not too large (for example, as shown in FIG. 2, the gain value multiplied by Short2 (short exposure image) is about ½ of the original gain value (exposure ratio Exp). ), It is known that the final image is less uncomfortable and is not a big problem. FIG. 7 shows an example of a composite image when the gain value is set to the exposure ratio Exp and when the gain value is set to ½ of the exposure ratio Exp. The gain value is not limited to 1/2 of the exposure ratio Exp, and may be a predetermined value smaller than the exposure ratio Exp.

  Also, the gain setting unit 30 sets the gain value to be multiplied to Short1 (ultra-short exposure image) to a value that is 1/8 of the value instead of the original gain value (exposure ratio Exp squared). Good. When the gain value is set in this manner, a step is formed at the boundary between Short 1 ′ (an ultra-short exposure image multiplied by the gain value) and Short 2 ′ (a short exposure image multiplied by the gain value), but the gradation of the latter stage Since the composite image is strongly compressed by the compression unit 80 as the luminance increases, the level difference is reduced and it becomes difficult to see.

  Therefore, even if the level difference between Short 1 ′ and Short 2 ′ is larger than the level difference between Short 2 ′ and Long (for example, as in the example shown in FIG. 2, short 1 (ultra short exposure image)) When the gain value to be multiplied is set to about 1/8 of the original gain value (the square of the exposure ratio Exp), a sense of incongruity is hardly noticeable in the final image. In addition, since there are few scenes that capture an ultra-bright object that does not achieve proper exposure unless it is Short1, the frequency of adverse effects is low. The gain value is not limited to 1/8 of the square of the exposure ratio Exp, and may be a predetermined value smaller than the exposure ratio Exp.

  In this way, by combining by applying a gain value smaller than the original exposure ratio, in the example shown in FIG. 2, the maximum value of the combined image can be suppressed to the original 1/8. That is, in the example shown in FIG. 2, the size of the synthesis circuit can be saved by 3 bits. In addition, since it is only necessary to control the gain value to be applied, no complicated calculation or calculation using a line memory is required. Moreover, it is possible to set a gain value to such an extent that a step formed at a boundary portion between different images is not conspicuous, thereby suppressing a decrease in smoothness of the composite image.

  The description will be continued with reference to FIG. FIG. 4 is a diagram illustrating a relationship between the horizontal axis (subject luminance) and the vertical axis (composite pixel value) in “when exposure ratio = Exp” and “when exposure ratio = Exp ′”. As shown in FIG. 4, Exp ′ is smaller than Exp, and the subject brightness is smaller in “when exposure ratio = Exp ′” than in “when exposure ratio = Exp”. In the example shown in FIG. 4, when the exposure ratio = Exp ′, even if the exposure ratio Exp ′ is used as the gain value according to a general method, the maximum value of the composite image is “when the exposure ratio = Exp”. It is the same. Therefore, it is not necessary to use a gain value smaller than Exp 'on purpose, so that no step is produced at the boundary between different images.

  From “when exposure ratio = Exp” and “when exposure ratio = Exp ′”, the gain setting unit 30 determines the gain value when the exposure ratio is lower than the threshold (or when the exposure ratio is equal to the threshold). It is understood that the exposure ratio may be set as follows, and if the exposure ratio exceeds the threshold value, the gain value may be clipped to the threshold value. That is, the control characteristic of the gain value with respect to the exposure ratio is as shown in FIG.

  For example, when the exposure ratio can be changed between 4 times and 16 times, the gain setting unit 30 sets the threshold value (TH) to “8 times”. At this time, when the exposure ratio is 4 to 8 times, the gain setting unit 30 uses the same gain value as the exposure ratio, but when the exposure ratio is 8 to 16 times, the gain value Can be clipped at an exposure ratio of 8 times. Then, when the exposure ratio is 16 times, the gain value becomes 8 times the exposure ratio, and the ratio of the gain value to the exposure ratio becomes 1/2.

  As described above, in order to reduce artifacts such as double contours in the motion area, a process of applying a short exposure image to the motion area (hereinafter also referred to as “motion adaptive synthesis process”). May do. Here, in the embodiment of the present invention, when the motion adaptive synthesis process is executed, the control shown in FIG. 5 should not be applied to the short exposure image of the motion region.

  That is, when the motion adaptive synthesis process is executed, the gain setting unit 30 may set the exposure ratio of the long exposure image to the short exposure image as the gain value of the short exposure image in the motion region. This is because if the gain value smaller than the exposure ratio of the long exposure image to the short exposure image is set as the gain value of the short exposure image in the moving area, the object becomes dark at the moment when the object moves. Therefore, the gain setting unit 30 may switch whether to disable the control shown in FIG. 5 depending on whether the motion adaptive synthesis process is executed. On the other hand, when the motion adaptive synthesis process is not executed, the control shown in FIG. 5 may be validated for the short-exposure image in the motion region.

  Note that whether or not to perform the motion adaptive synthesis process may be determined in any way. For example, whether or not to execute the motion adaptive synthesis process is determined in advance and may not be switched. Alternatively, whether to perform the motion adaptive synthesis process may be switchable. When it is possible to switch whether or not to execute the motion adaptive synthesis processing, the switching may be performed in conjunction with the operation by the user.

  Furthermore, even when shooting with the same exposure ratio, the gain value may be controlled differently if the number of combined images is different. 2 and FIG. 6 will be described for comparison. 2 and 6 are examples in which a plurality of images taken with the same exposure ratio Exp are combined. FIG. 2 shows an example of combining three frames (long exposure image, short exposure image, and ultrashort exposure image). FIG. 6 shows an example of synthesis of two frames (long exposure image and short exposure image).

  Here, as shown in FIG. 2, in the case of 3 frame synthesis, a gain value smaller than the exposure ratio is applied to the short exposure image and the ultra short exposure image so as not to exceed the bit upper limit of the system. I have to. On the other hand, as shown in FIG. 6, in the case of two-frame synthesis, even if the same gain value as the exposure ratio is applied to the short exposure image, the system does not reach the upper bit limit of the system, so a gain value smaller than the exposure ratio is applied. There is no need to do. This means that even if the exposure ratio is the same, it is preferable to switch whether or not the control shown in FIG.

  The synthesized image is not limited to 2 frames and 3 frames. Therefore, the gain setting unit 30 may set the exposure ratio of the long exposure image to the use image as the gain value of the use image when the composite number is less than the predetermined number. On the other hand, when the composite number is equal to or greater than the predetermined number, the gain setting unit 30 sets the exposure ratio of the long exposure image to the use image as the gain value of the use image when the exposure ratio of the long exposure image to the use image falls below the threshold value. When the exposure ratio of the long exposure image to the use image exceeds the threshold value, the gain value of the use image may be clipped to the threshold value.

  As described above, according to the embodiment of the present invention, it is possible to obtain a smooth composite image with a linear characteristic for a dynamic range that is frequently generated or assumed as a typical scene. For a scene having a very large dynamic range with low frequency of occurrence or low importance, it is possible to cope with composition while suppressing an increase in circuit scale in exchange for some deterioration in image quality.

(Summary)
In the embodiment of the present invention, amplification units 61 and 62 that amplify each of one or more short exposure images, which are images other than the long exposure image among a plurality of images, based on each gain value, and the long exposure image are amplified. The image processing apparatus 1 includes a combining unit 70 that combines the short exposure image and a gain setting unit 30 that sets a predetermined value smaller than the exposure ratio of the long exposure image to the short exposure image as a gain value. Is done. According to such a configuration, it is possible to generate a composite image while suppressing a decrease in smoothness of the composite image and suppressing an increase in circuit scale.

  WDR (wide dynamic range) technology is regarded as important. In the prior art, the circuit scale has been increased as the dynamic range that can be synthesized increases. The embodiment of the present invention pays attention to the fact that a scene with a very wide dynamic range has a low occurrence frequency, and linear synthesis is limited to a certain dynamic range, and for a scene having a certain dynamic range or more. By applying a small gain value in exchange for some image quality degradation, an increase in system scale can be suppressed.

  According to the embodiment of the present invention, it is possible to solve two conflicting problems of performance improvement and circuit scale increase with a rational and simple configuration.

  The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 10 Sensor 20 Frame memory 30 Gain setting part 31 Clip 32 Clip 40 Use image selection part 50 Motion detection part 61 Amplification part 62 Amplification part 70 Synthesis | combination part 80 Tone compression part

Claims (7)

An amplifying unit for amplifying one or more short-exposure images that are images other than the long-exposure image among the plurality of images based on each gain value;
A combining unit that combines the long exposure image and the amplified short exposure image;
A gain setting unit that sets a predetermined value smaller than an exposure ratio of the long exposure image to the short exposure image as the gain value;
Bei to give a,
The gain setting unit sets the exposure ratio as a gain value of the short exposure image when the short exposure image in which the exposure ratio falls below a threshold value is present in the one or more short exposure images. When there is a short exposure image in which the exposure ratio exceeds the threshold value in the short exposure image, the gain value of the short exposure image is clipped to the threshold value.
Image processing device. The image processing apparatus includes:
A use image selection unit that selects one of the long exposure image and the one or more short exposure images as a use image for each pixel;
The synthesizing unit synthesizes the use image for each pixel;
The image processing apparatus according to claim 1 . The image processing apparatus includes:
A motion detection unit for detecting a motion region;
When the process of selecting any one of the one or more short-exposure images as the movement region in the use image is performed, the gain setting unit performs the long-exposure image with respect to the use image as the gain value of the use image. Set the exposure ratio of
The image processing apparatus according to claim 2 . When the process of selecting any one of the one or more short-exposure images as the movement region in the use image is not performed, the gain setting unit determines that the exposure ratio of the long-exposure image to the use image has the threshold value. If the exposure ratio is lower, the exposure ratio is set as the gain value of the use image, and if the exposure ratio of the long exposure image to the use image exceeds the threshold value, the gain value of the use image is clipped to the threshold value. To
The image processing apparatus according to claim 3 . The gain setting unit sets an exposure ratio of the long exposure image to the use image as a gain value of the use image when the plurality of images is less than a predetermined number.
The image processing apparatus according to claim 2 . The gain setting unit, when the plurality of images is equal to or greater than the predetermined number, when the exposure ratio of the long exposure image to the use image is less than the threshold, the exposure ratio as the gain value of the use image. And when the exposure ratio of the long exposure image to the use image exceeds the threshold value, the gain value of the use image is clipped to the threshold value.
The image processing apparatus according to claim 5 . Amplifying each of one or more short-exposure images that are images other than the long-exposure image among the plurality of images based on each gain value;
Combining the long exposure image and the amplified short exposure image;
Setting a predetermined value smaller than an exposure ratio of the long exposure image to the short exposure image as the gain value;
Only including,
When there is a short exposure image in which the exposure ratio falls below a threshold value in the one or more short exposure images, the exposure ratio is set as a gain value of the short exposure image, and the one or more short exposure images Clipping the gain value of the short exposure image to the threshold when there is a short exposure image in which the exposure ratio exceeds the threshold;
Image processing method.