JP6184290B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method.

  In recent years, a short-exposure image (hereinafter also simply referred to as “short-exposure image”) and a long-exposure image (hereinafter also simply referred to as “long-exposure image”) are continuously photographed and combined. An imaging function called WDR (wide dynamic range) or HDR (high dynamic range) that obtains an image that captures a dynamic range that exceeds the dynamic range that the sensor can shoot is increasing. Such a photographing function is particularly effective in a scene with a very large contrast ratio such as a composition of backlight.

  Further, in recent years, there is an example in which a wide dynamic range is captured by increasing the exposure setting to three types or more and photographing / combining. However, due to the mechanism of combining sequentially photographed images with different exposures, there is a problem in that if the subject moves, a shift occurs during the combining and artifacts occur. The following technique is disclosed as a WDR synthesis method aiming at reducing this artifact.

  For example, a technique for detecting the presence or absence of motion for each pixel and using a short exposure image for pixels with motion and performing a WDR synthesis using a short exposure image or a long exposure image for pixels without motion is disclosed. (For example, refer to Patent Document 1). According to such a technique, it is possible to generate a composite image in which blur caused by combining a moving object (hereinafter also referred to as “animal body”) is reduced.

JP 2012-231273 A

  However, according to the technique described in Patent Literature 1, a moving object can be noisy because a short-exposure image is brightened by applying a large gain to the motion region and then used for the composite image.

  Patent Document 1 also mentions a case where three or more sheets are combined, and similarly, only the shortest exposure image in the motion region is used. According to such a configuration, as in the embodiment in Patent Document 1, if the difference in exposure amount is as small as 2 to 4 times, the noise of the shortest exposure image is slightly increased, but no significant problem occurs. If the difference in exposure amount is increased in an attempt to expand the dynamic range, the noise of the shortest exposure image increases significantly, which may cause a serious problem.

  Therefore, the present invention provides a technique capable of obtaining a WDR composite image in which noise generation is suppressed and artifacts at the time of WDR composition are suppressed in WDR in which shooting and composition are performed with three or more types of exposure. To do.

  According to an embodiment of the present invention, based on the motion detection result and the use image selection result, the determination unit that determines whether or not the motion region and the region that uses the medium exposure image or more overlap, If it is determined that the motion area and the area that uses the middle exposure image or more overlap, the middle exposure image is used for the motion area, and the middle exposure image and the length are used for the non-motion area. An image processing apparatus is provided that includes a combining unit that generates a first combined image using at least one of the exposure images.

  According to this configuration, when the condition that the motion region and the region using the medium exposure image overlap is satisfied, the medium exposure image is used for the motion region and the medium exposure image is used for the non-motion region. WDR synthesis using a long exposure image is performed. Therefore, according to such a configuration, it is possible to obtain a WDR composite image in which generation of noise is suppressed and artifacts at the time of WDR composition are suppressed in WDR in which shooting and composition are performed with three or more types of exposure. .

  The determination unit measures the number of overlapping pixels between the motion region and a region using the middle-exposure image or more, and determines whether the number of overlapping pixels exceeds a threshold, thereby determining the motion region and the middle region. You may determine whether the area | region which uses more than an exposure image overlaps. For example, the threshold value may be set according to the accuracy of motion detection.

  When it is determined that the motion area and the area that uses the medium-exposure image or more do not overlap, the synthesizing unit, among the short-exposure image, the medium-exposure image, and the long-exposure image, according to the use image selection result You may produce | generate a 2nd synthesized image using at least any one.

  The combining unit may gradually switch a method of generating a composite image between the generation of the first composite image and the generation of the second composite image when the determination result by the determination unit changes. Good. In particular, when the determination result changes so as to indicate that at least the motion region and the region using the medium exposure image or more do not overlap, the combining unit starts generating the first combined image. The method of generating the composite image may be gradually switched to the generation of the second composite image.

  In addition, when the determination result by the determination unit changes, the combining unit is predetermined before switching the method of generating a composite image between the generation of the first composite image and the generation of the second composite image. You may wait for the time. In particular, when the determination result changes so as to indicate that at least the motion region and the region using the medium exposure image or more do not overlap, the combining unit starts generating the first combined image. You may wait for a predetermined time before switching the method of generating the composite image to the generation of the second composite image.

  According to such a configuration, even if a situation occurs in which a moving object temporarily stops and moves again within an area that uses a medium-exposure image or more, or repeats intrusion / escape at the boundary of the area, An abrupt change does not occur and an easy-to-view moving image is obtained.

  According to another embodiment of the present invention, based on the motion detection result and the use image selection result, the step of determining whether or not the motion region and the region using the middle exposure image or more overlap. When it is determined that the motion region and the region using the middle exposure image or more overlap, the middle exposure image is used for the motion region, and the middle exposure image is used for the non-motion region. And generating a first composite image using at least one of the long exposure image and a long exposure image.

  According to this method, when the condition that the motion region and the region using the medium exposure image overlap is satisfied, the medium exposure image is used for the motion region and the medium exposure image is used for the non-motion region. WDR synthesis using a long exposure image is performed. Therefore, according to such a configuration, it is possible to obtain a WDR composite image in which generation of noise is suppressed and artifacts at the time of WDR composition are suppressed in WDR in which shooting and composition are performed with three or more types of exposure. .

  As described above, according to the present invention, in a WDR in which shooting and composition are performed with three or more types of exposure, a WDR composite image in which artifacts at the time of WDR composition are suppressed and artifacts generated in a motion region are suppressed is obtained. It is possible.

It is a figure for demonstrating a general WDR synthetic | combination technique. It is a figure which shows the function structure of the image processing apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows the detailed structural example of the determination part which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the example of the flow of operation | movement of the determination part which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the example of the flow of operation | movement of the WDR synthetic | combination part which concerns on the 1st Embodiment of this invention. It is a figure for demonstrating in detail the example of the process result by the image processing apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows the example of the time change of the use image by the image processing apparatus which concerns on the 2nd Embodiment of this invention.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

  In the present specification and drawings, a plurality of constituent elements having substantially the same functional configuration may be distinguished by attaching different alphabets after the same reference numeral. However, when it is not necessary to particularly distinguish each of a plurality of constituent elements having substantially the same functional configuration, only the same reference numerals are given.

(General WDR synthesis technology)
First, a general WDR synthesis technique (for example, the WDR synthesis technique described in Patent Document 1 above) will be described. FIG. 1 is a diagram for explaining a general WDR synthesis technique. Referring to FIG. 1, examples of a short exposure image, a long exposure image, a motion detection result, a use image selection result, a composite image (2 images), and a composite image (3 images) in a general WDR composition technique are shown. ing. Each of the “short exposure image” and the “long exposure image” shown in FIG. 1 is an image obtained by photographing the indoors against a background of a bright outdoor daylight.

  In the “short exposure image”, the scenery outside the window is shot with appropriate exposure, but the indoor and indoor moving objects are shot dark with insufficient exposure. On the other hand, in the “long exposure image”, the room and the moving object in the room are photographed with appropriate exposure, but the scenery outside the window is saturated and cannot be seen. The person in the foreground is moving toward the left.

  Further, it is assumed that a “motion detection result” as shown in FIG. 1 is obtained. Referring to the “motion detection result”, the motion area detected so as to cover the moving object is represented as a white area, and the non-motion area is represented as a black area. Here, if the short exposure image is selected as the use image for the motion region, the “use image selection result” as shown in FIG. 1 is obtained. When the “short exposure image” and the “long exposure image” are synthesized based on the “use image selection result” shown in FIG. 1, a double image like “composite image (two images)” shown in FIG. No composite image is obtained.

  However, as shown in “Composite image (2 images)”, according to the technique described in Patent Document 1, a short-exposure image is brightened by applying a large gain to the motion region and then used for the composite image. Therefore, the moving object can be noisy.

  Patent Document 1 also mentions a case where three or more sheets are combined, and similarly, only the shortest exposure image in the motion region is used. According to such a configuration, as in the embodiment in Patent Document 1, if the difference in exposure amount is as small as 2 to 4 times, the noise of the shortest exposure image is slightly increased, but no significant problem occurs. If the difference in exposure amount is increased in an attempt to expand the dynamic range, the noise of the shortest exposure image increases significantly, which may cause a serious problem.

  For example, when the exposure time is 1/60 seconds, a long exposure image, an exposure time of 1/480 seconds is taken as a medium exposure image, and an exposure time of 1/3840 seconds is taken as a short exposure image, the exposure amount is changed by 8 times. A WDR that covers a vast dynamic range by taking and combining images is assumed. In such WDR, since a short exposure image with a very short shutter time of 1/3840 seconds is used, it is possible to photograph an ultra-high brightness region that would be saturated even in a middle exposure image.

  However, assuming that such WDR is applied to the technique described in Patent Document 1, the motion region in the short exposure image is taken very dark with a very short shutter time of 1/3840 seconds. For this reason, this motion region is amplified with a very large gain and used in a composite image, and a composite image that is covered with noise is generated so that details cannot be recognized. Images are not practical. The “composite image (three images)” shown in FIG. 1 is a long-exposure image when it is assumed that the window area has such a high brightness that it can finally be photographed with a short exposure of 1/3840 seconds. The example of the result obtained by synthesize | combining a medium exposure image and a short exposure image is shown.

  Therefore, in the present specification, in WDR in which shooting and synthesis are performed with three or more types of exposure, it is possible to obtain a WDR synthesized image in which noise generation is suppressed and artifacts during WDR synthesis are suppressed. Propose. In the present specification, as an example of N (N is a natural number of 3 or more) images photographed with different exposure amounts, three images of a short exposure image, a middle exposure image, and a long exposure image are arranged in order of increasing exposure amount. An example of taking an image will be mainly described. However, as will be described later, the image processing apparatus 1 according to the first embodiment of the present invention can also be applied when four or more images are captured.

  First, the functional configuration of the image processing apparatus 1 according to the first embodiment of the present invention will be described. FIG. 2 is a diagram showing a functional configuration of the image processing apparatus 1 according to the first embodiment of the present invention. As shown in FIG. 2, the image processing apparatus 1 includes a sensor 10, a short exposure image memory 21, a medium exposure image memory 22, a long exposure image memory 23, a use image selection unit 30, a motion detection unit 40, a determination unit 50, a WDR. A synthesis unit 60 and a gradation compression unit 80 are provided. Hereinafter, the function of each functional block included in the image processing apparatus 1 will be described in detail sequentially.

  The image processing apparatus 1 changes the exposure setting of the sensor 10 and continuously shoots three images. Here, the short exposure shooting is performed first, followed by the medium exposure shooting, and then the long exposure shooting. Assumed to be performed. The short exposure image taken by the short exposure is written in the short exposure image memory 21. When the short exposure shooting is completed, the image processing apparatus 1 changes the exposure setting and performs medium exposure shooting. The middle exposure image taken by the middle exposure is written in the middle exposure image memory 22. The image processing apparatus 1 changes the exposure setting when the medium exposure shooting is completed, and performs the long exposure shooting. The long exposure image taken by the long exposure is written in the long exposure image memory 23.

  In the example illustrated in FIG. 2, the image processing apparatus 1 has one common system for outputting a long exposure image, a medium exposure image, and a short exposure image, and the sensor 10 includes a long exposure image and a medium exposure image. Although the short exposure image is output in a time division manner, the long exposure image, the medium exposure image, and the short exposure image may be output simultaneously. In such a case, if the image processing apparatus 1 has three systems, a system for outputting a long exposure image from the sensor 10, a system for outputting a medium exposure image, and a system for outputting a short exposure image. Good. Each shutter time is determined by, for example, a dynamic range of a subject to be imaged, a sensor specification, and the like.

  Note that the readout operation for each of the short-exposure image, the medium-exposure image, and the long-exposure image can vary depending on how the system is assembled in the case of hardware implementation. For example, each of the short-exposure image, the medium-exposure image, and the long-exposure image may be stored once in a memory such as a DRAM (Dynamic Random Access Memory) and then read again at the time of synthesis.

  Here, in the first embodiment of the present invention, the terms short-exposure image, medium-exposure image, and long-exposure image are used, and these terms indicate the absolute exposure time of each of the three captured images. It is not intended to limit. Therefore, when three images with different exposure times are taken, the three images are named as a short exposure image, a medium exposure image, and a long exposure image in order of increasing exposure time.

  The sensor 10 is configured by an image sensor that forms an image of light from the outside on the light receiving plane of the image sensor, photoelectrically converts the imaged light into a charge amount, and converts the charge amount into an electric signal. The type of the image sensor is not particularly limited, and may be, for example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

  The use image selection unit 30 refers to the short exposure image read from the short exposure image memory 21, the medium exposure image read from the medium exposure image memory 22, and the long exposure image read from the long exposure image memory 23. Then, the use image selection unit 30 detects the saturation state or movement of each of the short exposure image, the medium exposure image, and the long exposure image, and uses any of the short exposure image, the medium exposure image, and the long exposure image. The use image selection result for selecting as is generated. As an algorithm for selecting one of the short exposure image, the medium exposure image, and the long exposure image, various algorithms are assumed.

  For example, since a region that has been saturated in the long exposure image is highly likely not to be saturated in the medium exposure image, the medium exposure image may be selected as the use image of the region. However, when the area is saturated even in the medium exposure image, the short exposure image may be selected as the use image of the area. However, with this process alone, artifacts such as a double outline may occur in an area where there is a large movement. Therefore, a process of detecting a motion and reducing a phenomenon that the contour becomes double may be performed.

  The use image selection result may be a set of data for selecting one of the short exposure image, the medium exposure image, and the long exposure image for each pixel. In addition, the use image selection result may be a set of mixing ratios for each pixel indicating at what ratio each short exposure image, medium exposure image, and long exposure image are mixed. For example, the use image selection result is a set of combinations of data for selecting two adjacent images of a short exposure image, a medium exposure image, and a long exposure image and a mixing ratio indicating how much they are mixed. It may be. The algorithm for calculating the mixing ratio is not particularly limited.

  The motion detector 40 detects motion. The method for detecting motion is not particularly limited, but may be a method for detecting motion from the difference between a plurality of long-exposure images, or a method for detecting motion from the difference between a plurality of medium-exposure images. Alternatively, a technique of detecting motion from the difference between a plurality of short-exposure images may be used. Alternatively, the motion detection method may be a method of detecting motion from the difference between the long exposure image and the medium exposure image, or a method of detecting motion from the difference between the medium exposure image and the short exposure image. Also good. When detecting motion from a plurality of images having different exposure amounts, it is preferable to calculate the difference after normalizing one of the images by a gain corresponding to the exposure ratio.

  Alternatively, the motion detection method integrates the first motion detected from the difference between the long exposure image and the medium exposure image and the second motion detected from the difference between the medium exposure image and the short exposure image. It may be a technique for detecting. The integration of the first movement and the second movement may be performed in any way, for example, by adopting a larger movement among the first movement and the second movement. An integration of the movement and the second movement may be made.

  Furthermore, the motion detection unit 40 detects a motion region and a non-motion region based on the detected motion to obtain motion detection information. For example, the motion region is a region where the motion is larger than the threshold, and the non-motion region is a region where the motion is smaller than the threshold. An area where the motion is the same as the threshold may be detected as any area.

  Based on the motion detection result from the motion detection unit 40 and the use image selection result from the use image selection unit 30, the determination unit 50 performs the middle exposure image in the motion region detected by the motion detection unit 40 and the use image selection result. It is determined whether or not the area where the above is used overlaps. The medium exposure image or higher can correspond to an image that can capture a brighter area than the middle exposure image (an image captured with an exposure amount equal to or lower than the middle exposure). For example, the determination unit 50 measures the number of overlapping pixels between the moving region and the region using the middle exposure image or more, and determines whether the number of overlapping pixels exceeds the threshold, thereby determining whether the moving region and the middle exposure image or more. What is necessary is just to determine whether the area | region which uses is overlapped. In such a case, the threshold value may be set according to the accuracy of motion detection.

  Here, the function of the determination unit 50 will be described in more detail with reference to FIGS. 3 and 4. FIG. 3 is a diagram illustrating a detailed configuration example of the determination unit 50 according to the first embodiment of the present invention. As illustrated in FIG. 3, the determination unit 50 includes a comparison determination unit 51, a counter 52, and an overlap determination unit 53. FIG. 4 is a flowchart showing an example of the operation flow of the determination unit 50 according to the first embodiment of the present invention.

  First, as shown in FIG. 4, the comparison / determination unit 51 determines whether or not the target pixel is a pixel in a motion area and a pixel in an area using a medium exposure image or more ( Step S11). When the comparison determination unit 51 determines that the target pixel is a pixel in the motion region and a pixel in the region using the middle exposure image or more, the counter 52 is added (step S12). . On the other hand, the comparison / determination unit 51 does not add the counter 52 when the target pixel is a pixel in the non-motion region or a pixel that is not in the region using the medium exposure image or more.

  If the determination for all the pixels has not been completed (“No” in step S13), the comparison determination unit 51 returns to step S11. On the other hand, when the determination for all the pixels is completed (“Yes” in step S13), since the measurement of the number of overlapping pixels between the motion region and the short exposure image using region ends, the overlapping determination unit 53 It is determined whether or not the number exceeds a threshold value (step S14).

  If the number of overlapping pixels exceeds the threshold (“Yes” in step S14), the overlap determining unit 53 determines that the motion region and the region using the middle exposure image or more overlap (step S15). ). On the other hand, when the number of overlapping pixels does not exceed the threshold value (“No” in step S14), the overlap determination unit 53 determines that the motion region and the region using the middle exposure image or more do not overlap ( Step S16).

  The WDR synthesis unit 60 receives the motion detection result from the motion detection unit 40, the use image selection result from the use image selection unit 30, and the determination result from the determination unit 50, and the motion detection result and the use image selection result A WDR image is generated by combining the short exposure image, the medium exposure image, and the long exposure image based on the determination result. Specifically, when the determination unit 50 determines that the motion region and the region using the medium exposure image or more overlap, the WDR synthesis unit 60 uses the medium exposure image for the motion region. For the non-motion region, the first composite image is generated using the medium exposure image and the long exposure image according to the use image selection result.

  According to this configuration, when the condition that the motion region and the region using the medium exposure image overlap is satisfied, the medium exposure image is used for the motion region and the medium exposure image is used for the non-motion region. WDR synthesis using a long exposure image is performed. Therefore, according to such a configuration, in WDR in which shooting and composition are performed with three or more types of exposure, artifacts during WDR composition are suppressed, and a short-exposure image is used unconditionally with respect to a motion region In comparison, it is possible to obtain a WDR composite image in which noise is suppressed.

  Note that the WDR synthesis unit 60 may perform synthesis when the determination unit 50 determines that the motion region and the region using the middle exposure image or more do not overlap. According to the use image selection result, the second composite image may be generated using the short exposure image, the medium exposure image, and the long exposure image.

  The combining method by the WDR combining unit 60 is not particularly limited. For example, if the use image selection result is a set of data for selecting any one of the short exposure image, the medium exposure image, and the long exposure image for each pixel, the WDR synthesis unit 60 adds the data to the data. Based on this, the second composite image may be generated using any one of the short exposure image, the medium exposure image, and the long exposure image.

  Alternatively, the WDR synthesis unit 60, when the use image selection result is a set of mixing ratios for each pixel indicating how much the short exposure image, the medium exposure image, and the long exposure image are mixed, Based on each mixing ratio, the second exposure image may be generated by mixing the short exposure image, the medium exposure image, and the long exposure image.

  In addition, the WDR synthesis unit 60 uses the result of selecting the use image to indicate data for selecting two adjacent images of the short exposure image, the medium exposure image, and the long exposure image and the ratio of mixing the data. In the case of a set of combinations with ratios, a second composite image may be generated by mixing two images based on the two images selected according to the data and the respective mixing ratios.

  The functions of the WDR synthesis unit 60 will be described in more detail with reference to FIG. FIG. 5 is a flowchart showing an example of an operation flow of the WDR synthesis unit 60 according to the first embodiment of the present invention. First, as illustrated in FIG. 5, when the determination unit 50 determines that the motion region and the region that uses the middle exposure image do not overlap (see “No” in step S21), the WDR synthesis unit 60 In accordance with the use image selection result for the pixels, at least one of the short exposure image, the medium exposure image, and the long exposure image is used (step S22), and WDR composition is performed (step S26).

  On the other hand, when it is determined by the determination unit 50 that the motion region and the region using the medium exposure image are overlapped (“Yes” in step S21), the WDR synthesis unit 60 determines that the target pixel is within the motion region. It is determined whether or not it is a pixel. When the WDR synthesis unit 60 determines that the target pixel is not a pixel in the motion region (“No” in step S23), at least one of the medium exposure image and the long exposure image for the target pixel. Is used (step S24) to perform WDR synthesis (step S26).

  More specifically, when the WDR synthesis unit 60 determines that the target pixel is not a pixel in the motion region, the upper limit of the used image may be limited to the middle exposure image for the target pixel. As a result, for areas that use medium-exposure images or more, medium-exposure images are used for WDR composition, and for areas that use less than medium-exposure images, use images according to the use-image selection result are used for WDR composition. The

  On the other hand, when the WDR composition unit 60 determines that the target pixel is a pixel in the motion region (“Yes” in step S23), the WDR composition is performed using the medium exposure image (step S25). (Step S26). If the synthesis for all the pixels has not been completed ("No" in step S27), the WDR synthesis unit 60 returns to step S21. On the other hand, when the synthesis for all the pixels is completed (“Yes” in step S27), the WDR synthesis unit 60 finishes the WDR synthesis.

  The gradation compression unit 80 uses the WDR synthesis unit 60 to perform compression processing for bringing the bit range of an image signal having a wide dynamic range into a predetermined bit range and gradation correction that approximates a scene viewed by human eyes. This is performed on the generated WDR image. The compression process and the gradation correction may be performed simultaneously or at different timings.

  The subsequent stage of the gradation compression unit 80 is connected to an image processing engine including, for example, a demosaic unit that generates an RGB plane from Bayer data, a contour enhancement unit, and color management. For this reason, it is preferable that the data amount of the output signal from the gradation compression unit 80 is adjusted so as to match the size of the input data to the image processing engine (for example, about 12 bits). If the data size is simply reduced, the image is converted into a dark image. Therefore, it is preferable that the high luminance side is strongly compressed so as to approximate human visual characteristics.

  Subsequently, an example of a processing result by the image processing apparatus 1 according to the first embodiment of the present invention will be described in detail. FIG. 6 is a diagram for explaining in detail an example of a processing result by the image processing apparatus 1 according to the first embodiment of the present invention. In the first embodiment of the present invention, the use image selection result (3 Sheet), composite image (3 sheets), use image selection result (2 sheets / intrusion), composite image (2 sheets / intrusion), use image selection result (2 sheets / after movement) and composite image (2 sheets / It is a figure which shows each example after a movement.

  In “Used image selection result (3 images)”, “Used image selection result (2 images / during entry)” and “Used image selection result (2 images / after movement)” in FIG. The region used is shown, the gray region is shown as the region using the medium exposure image, and the black region is shown as the region using the long exposure image. First, referring to “use image selection result (3 images)”, a short exposure image is selected as a use image of an area where a window exists, and a medium exposure image is selected as a use image of the peripheral portion. A long exposure image is selected as a use image of a region where a wall exists.

  In a state before the moving object enters the screen, a “composite image (3 images)” using the short exposure image, the medium exposure image, and the long exposure image is displayed according to the “use image selection result (3 images)”. Generated. Referring to “synthesized image (3 images)”, the area where the window exists is not saturated before the moving object (in this example, the area where the person exists) enters the screen. .

  An example of the use image selection result when the moving object enters the screen is shown in “Use image selection result (two images / when entering)”. Referring to “Used image selection result (2 images / intrusion)”, since the moving object overlaps the area that uses the medium exposure image or more, the medium exposure image is used for the movement area, and the entire screen is used. The image is also limited to a middle exposure image or less (in this example, a middle exposure image and a long exposure image). When the composition is performed in this way, a “composite image (two images / when entering)” is generated.

  Referring to “Composite image (2 images / intrusion)”, for a region where a window exists, a short exposure image should be used in the use image selection result. Because it is used, saturation occurs and the background disappears. Instead, the moving object can be sufficiently visually recognized with a slight increase in noise, and can have the advantage that no moving artifacts occur. This situation is maintained while the moving object overlaps at least one of the medium exposure image use area and the short exposure image use area.

  An example of the use image selection result when the moving object moves in the screen is shown in “Use image selection result (two images after movement)”. Referring to “Used image selection result (2 images / after movement)”, the moving object is moved, but it still overlaps the area that uses more than the middle exposure image. The used image of the entire screen is also limited to the middle exposure image or less (in this example, the middle exposure image and the long exposure image). When the composition is performed in this way, a “composite image (two images after movement)” is generated. When the moving object escapes from the area where the medium exposure image or more is used, the state before the moving object enters the screen is restored.

  The WDR synthesis method according to the embodiment of the present invention reduces noise generated in a region where a moving object exists, instead of temporarily allowing saturation in a short-exposure image usage region, as compared with a general WDR synthesis method. In addition, the occurrence of motion artifacts can be suppressed. First, according to the WDR composition method according to the embodiment of the present invention, it is an advantage that the saturation in the short-exposure image use area can be limited only when the moving object passes through the area. Also, since moving objects are generally more noticeable than short-exposure image usage areas that do not move, even if the background without movement temporarily saturates, the actual disadvantages are small, and areas where moving objects exist It is considered reasonable not to generate large noise in For these reasons, it can be said that the WDR combining method according to the embodiment of the present invention is an effective method as a WDR control method for combining three or more frames.

(Second Embodiment)
Subsequently, a second embodiment of the present invention will be described. When the WDR synthesis technique according to the first embodiment that switches the synthesis method depending on whether or not the moving object overlaps the area that uses the medium exposure image or more, one problem may occur. For example, when a moving object stops and moves again in an area that uses a medium-exposure image or more, or when it repeatedly enters and exits at the boundary of the area, the background is visible or saturated frequently. There may be a phenomenon that a moving image is difficult to see.

  Therefore, in the second embodiment of the present invention, this phenomenon is mitigated by creating a mechanism that gradually changes in the time direction. FIG. 7 is a diagram illustrating an example of temporal change of a used image by the image processing apparatus 1 according to the second embodiment of the present invention. For example, as illustrated in FIG. 7, the WDR synthesis unit 60 can alleviate the above phenomenon by controlling the upper limit of the used image and the lower limit of the used image assigned to the motion region in the time direction. Is possible.

  First, in a state before the moving object enters the screen (see “Composite image (3 images)” in FIG. 6), either the upper limit of the used image or the lower limit of the used image assigned to the motion area is selected. There are no restrictions. When the moving object enters an area using the medium exposure image or more, the lower limit of the use image allocated to the movement area is set to the medium exposure image. As a result, the middle exposure image is assigned to the motion region. Further, the upper limit of the used image of the entire screen is set to the middle exposure image. As shown in FIG. 7, such a setting may be made by using the result of the overlap determination in the frame in which intrusion is detected at the time of synthesis in the next frame.

  Here, when the determination result changes between “overlapping” and “non-overlapping”, the WDR synthesizing unit 60 performs the interval between the generation of the first synthesized image and the generation of the second synthesized image. The method of generating the composite image is switched at, but it is preferable to wait for a predetermined time before switching. In particular, the WDR synthesis unit 60 may wait for a predetermined time before switching when the determination result changes at least from “overlap” to “non-overlap”.

  Specifically, as shown in FIG. 7, even if the moving object escapes from the area where the middle exposure image or more is used, the limited state may be maintained for the “WAIT” time. For example, the number of frames from the time of exit may be counted and the restricted state may be maintained until it reaches the number of frames corresponding to the “WAIT” time.

  In addition, when the determination result changes between “overlapping” and “non-overlapping”, the WDR combining unit 60 generates between the generation of the first combined image and the generation of the second combined image. It is preferable to gradually switch the method of generating the composite image. In particular, when the determination result changes at least from “overlapping” to “non-overlapping”, the WDR combining unit 60 generates a combined image from the generation of the first combined image to the generation of the second combined image. It is good to switch the method gradually.

  Specifically, as shown in FIG. 7, the upper limit of the used image and the lower limit of the used image assigned to the motion region may be gradually returned. For example, as shown in “SLOPE” in FIG. 7, the return speed can be determined by the amount of increase / decrease per frame of the α blend ratio used for mixing the images.

  The gradual processing may be terminated when the upper limit of the use image becomes equal to or greater than the short exposure image. The lower limit of the used image assigned to the motion area may be controlled in the same manner as the upper limit of the used image. If an intrusion occurs during the operation period of “WAIT” or “SLOPE”, the state may be reset to return to the state immediately after the intrusion. Even if a situation where the moving object stops once in the area using the medium exposure image or more and moves again or repeats intrusion / escape at the boundary of the area by the method as described above, An abrupt change in the composition method does not occur, and an easy-to-view moving image is obtained.

  Heretofore, the first embodiment and the second embodiment of the present invention have been described. According to the first embodiment of the present invention, based on the motion detection result and the use image selection result, the determination unit 50 that determines whether or not the motion region and the region using the medium exposure image or more overlap. When it is determined that the motion region and the region using the medium exposure image or more overlap, the medium exposure image is used for the motion region, and the medium exposure image and the long exposure are used for the non-motion region. There is provided an image processing apparatus 1 including a WDR synthesis unit 60 that generates a first synthesized image using at least one of images.

  According to this configuration, when the condition that the motion region and the region using the medium exposure image overlap is satisfied, the medium exposure image is used for the motion region and the medium exposure image is used for the non-motion region. WDR synthesis using a long exposure image is performed. Therefore, according to such a configuration, it is possible to obtain a WDR composite image in which generation of noise is suppressed and artifacts at the time of WDR composition are suppressed in WDR in which shooting and composition are performed with three or more types of exposure. .

  On the other hand, if it is determined that the motion region and the region that uses the medium exposure image or more do not overlap, the WDR synthesis unit 60 selects the short exposure image, the medium exposure image, and the long exposure image according to the use image selection result. What is necessary is just to produce | generate a 2nd synthesized image using at least any one.

  In the second embodiment of the present invention, when the determination result by the determination unit 50 is changed, the WDR synthesis unit 60 generates a composite image between the generation of the first composite image and the generation of the second composite image. The generation method is gradually switched. In particular, when the determination result changes so as to indicate that at least the motion region and the region using the medium exposure image or more do not overlap, the combining unit starts generating the first combined image. The method of generating the composite image may be gradually switched to the generation of the second composite image.

  According to such a configuration, even if a situation occurs in which a moving object temporarily stops and moves again within an area that uses a medium-exposure image or more, or repeats intrusion / escape at the boundary of the area, An abrupt change does not occur and an easy-to-view moving image is obtained.

  In addition, when the determination result by the determination unit 50 changes, the WDR synthesis unit 60 performs a predetermined process before switching the method of generating the composite image between the generation of the first composite image and the generation of the second composite image. You may wait for hours. In particular, the WDR composition unit 60 generates the first composite image when the determination result changes so as to indicate that at least the motion region and the region using the medium exposure image or more do not overlap. From this time on, a predetermined time may be waited before switching the method of generating the composite image to the generation of the second composite image.

  According to such a configuration, even if a situation occurs in which a moving object temporarily stops and moves again within an area that uses a medium-exposure image or more, or repeats intrusion / escape at the boundary of the area, An abrupt change does not occur and an easy-to-view moving image is obtained.

  The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

  For example, in this specification, as an example of N (N is a natural number of 3 or more) images photographed with different exposure amounts, three images of a short exposure image, a middle exposure image, and a long exposure image are arranged in order of increasing exposure amount. An example of taking an image has been mainly described. However, as already described, the image processing apparatus 1 according to the embodiment of the present invention can be applied even when four or more images are captured. For example, when shooting four images, there are two types of medium exposure images. Which of the two types of medium-exposure images is used in the motion region to be the upper limit of the used image may be selected in advance. For example, the selection may be made based on the amount of noise generated in the moving object, the degree of saturation generated when the short exposure image is not used, or the like.

  Next, explanation will be given on the method described in Patent Document 2 to Patent Document 4 below and its problems.

  In Patent Document 2 (Japanese Patent Application Laid-Open No. 2011-205546), while repeating underexposure and overexposure, the exposure ratio is changed between a plurality of types and images are taken. A technique for selecting and synthesizing two images suitable for a dynamic range is disclosed. In Patent Document 2, the exposure ratio is changed between six types. According to such a technique, since two images that are temporally adjacent to each other are always used for composition, it is possible to suppress motion blur at the time of composition, and even if the subject includes an extremely bright object, an appropriate exposure ratio setting is possible. If two appropriate images are selected, they can be combined.

  However, in the technique described in Patent Document 2, since two temporally adjacent sheets are used for composition, motion blur at the time of composition can be suppressed, but composition is possible even if two images are used for composition. The movement of time will occur. In the technique described in Patent Document 2, since many images with different exposure ratios must be prepared, there is a problem that the frame rate is significantly reduced when a moving image is obtained as a synthesis result.

  Japanese Patent Laid-Open No. 2008-160881 discloses a technique for continuously capturing images with a short unit exposure time and adding the obtained unit images to generate a long exposure image. According to such a technique, if the short-exposure image is a unit image photographed at the center time of long-exposure image generation, a composite image with almost no displacement can be generated.

  However, in the technique described in Patent Document 3, it is considered that if the unit exposure time is too short, the transfer speed of the photographic data will not be in time, and the power consumption and heat generation during transfer will become very large. Therefore, the unit exposure time can be made very small, so that the performance of photographing high luminance cannot be improved so much.

  In addition, in the technique described in Patent Document 3, it is an advantage that there is no positional deviation, but when the moving object is photographed in a situation where the luminance difference between the background and the object is large, motion blur is present in the long-exposure image. As a result, the object is photographed large, while the short-exposure image has no motion blur, so the object is photographed small. If composition is performed in this situation, a shadow such as “haze” will appear around the object, which may result in an unnatural image.

  In Patent Document 4 (Japanese Patent Application No. 2013-173699), as a technique for solving the problem that a moving object can be noisy in the technique described in Patent Document 1, noise reduction effective only for a moving object is disclosed. The method of applying was proposed by the inventors of the present application. However, since the noise of an ultrashort exposure image such as 1/3840 second cannot be sufficiently eliminated, the problem is not solved even if such a method is used.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 10 Sensor 21 Short exposure image memory 22 Medium exposure image memory 23 Long exposure image memory 30 Use image selection part 40 Detection part 50 Determination part 51 Comparison determination part 52 Counter 53 Overlapping determination part 60 WDR composition part (composition part)
80 gradation compression unit

Claims (7)

A determination unit that determines whether or not the motion region and the region using the medium exposure image or more overlap based on the motion detection result and the use image selection result;
When it is determined that the motion region and the region using the middle exposure image or more overlap, the middle exposure image is used for the motion region, and the middle exposure image and the non-motion region are used. A combining unit that generates a first combined image using at least one of the long exposure images;
Bei to give a,
When it is determined that the motion area and the area that uses the medium-exposure image or more do not overlap, the synthesizing unit, among the short-exposure image, the medium-exposure image, and the long-exposure image, according to the use image selection result Generating a second composite image using at least one of the following:
Image processing device. The determination unit measures the number of overlapping pixels between the motion region and a region using the middle-exposure image or more, and determines whether the number of overlapping pixels exceeds a threshold, thereby determining the motion region and the middle region. Determine whether the area that uses more than the exposure image overlaps,
The image processing apparatus according to claim 1. The combining unit gradually switches a method of generating a composite image between the generation of the first composite image and the generation of the second composite image when the determination result by the determination unit changes;
The image processing apparatus according to claim 1 . When the determination result changes so as to indicate that at least the moving region and the region using the middle-exposure image or more do not overlap, the synthesizing unit starts from the generation of the first synthesized image to the second Gradually switch the method of generating the composite image to the generation of the composite image of
The image processing apparatus according to claim 3 . When the determination result by the determination unit changes, the combining unit performs a predetermined time before switching the method of generating the combined image between the generation of the first combined image and the generation of the second combined image. stand by,
The image processing apparatus according to claim 1 . When the determination result changes so as to indicate that at least the moving region and the region using the middle-exposure image or more do not overlap, the synthesizing unit starts from the generation of the first synthesized image to the second Waiting for a predetermined time before switching the method of generating a composite image to generating a composite image of
The image processing apparatus according to claim 5 . Determining whether the motion region and the region using the medium exposure image or more overlap based on the motion detection result and the use image selection result;
When it is determined that the motion region and the region using the middle exposure image or more overlap, the middle exposure image is used for the motion region, and the middle exposure image and the non-motion region are used. Generating a first composite image using at least one of the long exposure images;
Only including,
When it is determined that the motion area and the area that uses the medium exposure image or more do not overlap, at least one of a short exposure image, a medium exposure image, and a long exposure image is determined according to the use image selection result. Generating a second composite image using
Image processing method.