JP6173027B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method.

  In recent years, a short-exposure image (hereinafter also simply referred to as “short-exposure image”) and a long-exposure image (hereinafter also simply referred to as “long-exposure image”) are continuously photographed and combined. An imaging function called WDR (wide dynamic range) or HDR (high dynamic range) that obtains an image that captures a dynamic range that exceeds the dynamic range that the sensor can shoot is increasing. Such a photographing function is particularly effective in a scene with a very large contrast ratio such as a composition of backlight.

  However, due to the mechanism of synthesizing the short-exposure image and the long-exposure image, there is a problem that if the subject moves, a shift occurs during the synthesis and artifacts such as doubled contours occur. As a technique for performing WDR synthesis in response to movement, there is a technique described in Patent Document 1. In this technique, a common area is found from two images with different exposures, and the position is corrected, and then motion compensation is performed, and then synthesis is performed.

JP 2011-004353 A

  However, such a technique is a suitable technique when there is a global movement with respect to the entire image, but it cannot cope with a local movement with respect to a region constituting the image, and generates an unnatural artifact. Therefore, the present invention provides a technique capable of obtaining a WDR composite image in which artifacts are not noticeable even when local motion exists.

  According to an embodiment of the present invention, the first LPF that removes high-frequency components from the selection information indicating the mixing ratio for each corresponding pixel in the long exposure image and the short exposure image, and the motion detection information by detecting the motion. Based on the obtained motion detection unit, the second LPF that removes the high frequency component from the motion detection information, and the motion detection information after the high frequency component is removed, the selection information and the high frequency component before the high frequency component is removed are removed. A selection unit that selects one of the selected selection information for each pixel and obtains a selection result, and a synthesis unit that combines the long exposure image and the short exposure image based on the selection result. An image processing apparatus is provided.

  With such a configuration, it is possible to smoothly connect the short-exposure image and the long-exposure image in the motion region, and it is possible to obtain a WDR composite image in which artifacts are not noticeable even when local motion exists. In addition, in an image where the subject does not move or in a non-moving area in a moving image, the boundary between the short exposure image and the long exposure image can be switched sharply, and the short exposure image and the long exposure image can be switched. It is also possible to bring about an effect that adverse effects such as blurring of the boundary are less likely to occur.

  For example, the selection unit selects the selection information after the high-frequency component is removed for the motion region detected from the motion detection information after the high-frequency component is removed, and the motion detection after the high-frequency component is removed For the non-motion region detected from the information, selection information before the high-frequency component is removed may be selected.

  Further, according to another embodiment of the present invention, the step of removing high frequency components from the selection information indicating the mixing ratio for each corresponding pixel in the long exposure image and the short exposure image, and detecting the motion to detect the motion detection information. A step of removing a high frequency component from the motion detection information, and a selection information before removing the high frequency component and a selection after removing the high frequency component based on the motion detection information after the high frequency component is removed. An image processing method is provided, comprising: selecting any of the information for each pixel to obtain a selection result; and synthesizing the long exposure image and the short exposure image based on the selection result. The

  According to this method, it is possible to smoothly connect the short-exposure image and the long-exposure image in the motion region, and it is possible to obtain a WDR composite image in which artifacts are not noticeable even if local motion exists. In addition, in an image where the subject does not move or in a non-moving area in a moving image, the boundary between the short exposure image and the long exposure image can be switched sharply, and the short exposure image and the long exposure image can be switched. It is also possible to bring about an effect that adverse effects such as blurring of the boundary are less likely to occur.

  As described above, according to the present invention, it is possible to obtain a WDR composite image in which artifacts are not noticeable even when a local motion exists.

It is a figure which shows the function structural example of a general image processing apparatus. It is a figure which shows the example of the various information in a general WDR synthetic | combination technique. It is a figure which shows the function structural example of the image processing apparatus which concerns on embodiment of this invention. It is a figure which shows the example of the various information in the WDR synthetic | combination technique which concerns on embodiment of this invention. It is a figure which shows the example of the flow of operation | movement of the image processing apparatus which concerns on embodiment of this invention.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

  In the present specification and drawings, a plurality of constituent elements having substantially the same functional configuration may be distinguished by attaching different alphabets after the same reference numeral. However, when it is not necessary to particularly distinguish each of a plurality of constituent elements having substantially the same functional configuration, only the same reference numerals are given.

  First, FIG. 1 shows a functional configuration example of an image processing apparatus 9 for realizing a general WDR synthesis technique. The image processing device 9 changes the exposure setting of the sensor 10 and continuously shoots two images. Here, it is assumed that short exposure shooting is performed first, and then long exposure shooting is performed. The short exposure image taken by the short exposure is written in the memory 20. When the short exposure shooting is completed, the image processing apparatus 9 changes the exposure setting and performs the long exposure shooting.

  In the example shown in FIG. 1, the image processing apparatus 9 has one common system for outputting the long exposure image and the short exposure image, and the sensor 10 sometimes outputs the long exposure image and the short exposure image. Although the output is divided, the long exposure image and the short exposure image may be taken simultaneously. In such a case, the image processing device 9 may have two systems, that is, a system for outputting a long exposure image from the sensor 10 and a system for outputting a short exposure image.

  The use image selection unit 30 refers to the long exposure image detected by the sensor 10 and the short exposure image read out from the memory 20 to detect the saturation state and movement of each of the long exposure image and the short exposure image, thereby reducing the short exposure image. Selection information for selecting either the exposure image or the long exposure image as the use image is generated. The synthesizing unit 60 receives the selection information from the use image selecting unit 30, and generates a WDR image by synthesizing the short exposure image and the long exposure image based on the selection information.

  The gradation conversion unit 70 generates a compression process for bringing the bit range of an image signal having a wide dynamic range into a predetermined bit range and a gradation correction that approximates a scene viewed by human eyes. To the WDR image. The compression process and the gradation correction may be performed simultaneously or at different timings.

  FIG. 2 is a diagram showing examples of a short exposure image, a long exposure image, selection information, and a composite image in a general WDR composition technique. Each of the short-exposure image and the long-exposure image shown in FIG. 2 is an image obtained by photographing indoors against a background of a bright outdoor daylight. The person in the foreground is moving toward the left.

  For example, when the selection information generated by the use image selection unit 30 is a set of binary data indicating which of the long exposure image and the short exposure image is used, like the selection information shown in FIG. A region using the exposure image can be represented as a black region, and a region using the short exposure image as a white region.

  Subsequently, the combining unit 60 combines the short exposure image and the long exposure image based on the selection information generated in this way. When the composition is performed in this way, as shown in the composite image shown in FIG. 2, a noticeable artifact whose position is shifted at the time of composition at the boundary between the bright window and the moving person and the contour is doubled is generated. The problem of occurring can occur. Such a problem is due to the fact that local motion is not taken into consideration when generating the selection information, and images in which the positions of the objects are shifted are combined in the motion region.

  Therefore, the present specification proposes a technique capable of obtaining a WDR composite image in which artifacts are not noticeable even when local motion exists.

  Subsequently, a functional configuration of the image processing apparatus 1 according to the embodiment of the present invention will be described. FIG. 3 is a diagram showing a functional configuration of the image processing apparatus 1 according to the embodiment of the present invention. As shown in FIG. 3, the image processing apparatus 1 includes a sensor 10, a memory 20, a first use image selection unit 31, a second use image selection unit 32, a motion detection unit 40, and an LPF (Low Pass Filter) 1 (51). , LPF2 (52), a synthesis unit 60, and a compression unit 80. Hereinafter, the function of each functional block included in the image processing apparatus 1 will be described in detail sequentially.

  The image processing apparatus 1 changes the exposure setting of the sensor 10 and continuously shoots two images. Here, it is assumed that short exposure shooting is performed first and then long exposure shooting is performed. The short exposure image taken by the short exposure is written in the memory 20. The image processing apparatus 1 changes the exposure setting when short exposure shooting is completed, and performs long exposure shooting.

  In the example shown in FIG. 3, the image processing apparatus 1 has one common system for outputting the long exposure image and the short exposure image, and the sensor 10 sometimes outputs the long exposure image and the short exposure image. Although the output is divided, the long exposure image and the short exposure image may be taken simultaneously. In such a case, the image processing apparatus 1 may have two systems, that is, a system for outputting a long exposure image from the sensor 10 and a system for outputting a short exposure image. Each shutter time is determined by, for example, a dynamic range of a subject to be imaged, a sensor specification, and the like.

  Here, in the embodiment of the present invention, the terms short exposure image and long exposure image are used, but these terms do not limit the absolute exposure time of each of the two captured images. Therefore, when two images with different exposure times are taken, an image with a relatively short exposure time corresponds to a short exposure image and an image with a relatively long exposure time is a long exposure. Corresponds to an image.

  The sensor 10 is configured by an image sensor that forms an image of light from the outside on the light receiving plane of the image sensor, photoelectrically converts the imaged light into a charge amount, and converts the charge amount into an electric signal. The type of the image sensor is not particularly limited, and may be, for example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

  The first use image selection unit 31 refers to the long exposure image detected by the sensor 10 and the short exposure image read from the memory 20, and detects the saturation state and movement of each of the long exposure image and the short exposure image. Selection information for selecting either the short exposure image or the long exposure image as the use image is generated. As an algorithm for selecting either the short exposure image or the long exposure image, various algorithms are assumed.

  For example, since a region that has been saturated in the long exposure image is highly likely not to be saturated in the short exposure image, the short exposure image may be selected as the use image of the region. However, with this process alone, artifacts such as a double outline may occur in an area where there is a large movement. Therefore, a process of detecting a motion and reducing a phenomenon that the contour becomes double may be performed. An algorithm for selecting either a short exposure image or a long exposure image including such processing is not particularly limited.

  As described above, the selection information may be a set of binary data indicating which of the short exposure image and the long exposure image is to be selected, but at what ratio each of the long exposure image and the short exposure image is selected. It may be a set of mixing ratios indicating whether to mix. For example, the first use image selection unit 31 may increase the mixing ratio of the short exposure images as the saturation degree of the long exposure images is stronger. The first use image selection unit 31 may increase the mixing ratio of the short exposure images as the movement of the short exposure images or the long exposure images increases. The algorithm for calculating the mixing ratio of the short exposure image and the long exposure image is not particularly limited.

  Below, the 1st use image selection part 31 produces | generates the selection information which shows the mixing ratio for every pixel which respond | corresponds in a long exposure image and a short exposure image, and the said selection information is used for LPF1 (51) and a 2nd use image selection part. The case of outputting to 32 will be described as an example.

  The LPF 1 (51) performs processing for removing high frequency components from the selection information. With this function, when the selection information is greatly blurred and the composition unit 60 composes the short exposure image and the long exposure image, the boundary between the short exposure image and the long exposure image can be smoothly connected. “Selection information + LPF1” shown in FIG. 4 shows an example of the selection information after the high frequency component is removed.

  The bit accuracy of the output signal from the LPF 1 (51) should be sufficiently ensured so that the boundaries between the short exposure image and the long exposure image are smoothly connected. Various methods are assumed for the method of blurring the selection information, and it is not particularly limited. However, it is desirable that the blurring method is determined in consideration of the blurring effect, the calculation amount, the circuit scale, and the like.

  The motion detector 40 detects motion. The motion detection method is not particularly limited. For example, the motion detection method may be a method for detecting motion from the difference between the long exposure image and the short exposure image, or may be a method for detecting motion from the difference between a plurality of long exposure images. There may be a method of detecting motion from the difference between a plurality of short-exposure images, or another method. When the method of detecting motion from the difference between the long exposure image and the short exposure image is adopted, since the brightness differs between the long exposure image and the short exposure image as it is, the motion detection unit 40 performs the short exposure. It is preferable to calculate the difference after multiplying the image by a gain corresponding to the exposure amount.

  Furthermore, the motion detection unit 40 detects a motion region and a non-motion region based on the detected motion to obtain motion detection information. For example, the motion region is a region where the motion is larger than the threshold, and the non-motion region is a region where the motion is smaller than the threshold. An area where the motion is the same as the threshold may be detected as any area. Referring to the “motion detection information” shown in FIG. 4, the motion area is represented as a white area and the non-motion area is represented as a black area.

  The LPF 2 (52) performs processing for removing high frequency components from the motion detection information. With this function, it is possible to greatly blur motion detection information and enlarge a significant area. “Motion detection information + LPF2” shown in FIG. 4 shows an example of motion detection information after the high-frequency component is removed. Referring to this example, it can be understood that the motion region is expanded by the process of removing the high frequency component from the motion detection information.

  In addition, the LPF 2 (52) may not only perform the process of removing the high frequency component from the motion detection information, but may multiply the result of the process of removing the high frequency component from the motion detection information by a predetermined gain. Then, it is possible to further expand a significant area. The bit accuracy of the output signal from the LPF 2 (52) is also preferably sufficiently ensured that the boundaries between the motion region and the non-motion region are smoothly connected.

  Based on the motion detection information after the high-frequency component is removed, the second use image selection unit 32 selects either the selection information before the high-frequency component is removed or the selection information after the high-frequency component is removed as a pixel. Select every time to get a selection result. For example, the second use image selection unit 32 selects the selection information after the high frequency component is removed for the motion region detected from the motion detection information after the high frequency component is removed, and the high frequency component is removed. For the non-motion region detected from the subsequent motion detection information, selection information before the high-frequency component is removed may be selected.

  Referring to the “selection result” shown in FIG. 4, for the region corresponding to the motion region (white region) indicated by “motion detection information + LPF2”, “selection information + LPF1” after the high-frequency component is removed is It is grasped that it is selected. In addition, regarding the region corresponding to the non-motion region (black region) indicated by “motion detection information + LPF2”, it is understood that the selection information before the high-frequency component is removed is selected.

  The synthesizing unit 60 receives the selection result from the second use image selecting unit 32, and generates a WDR image by synthesizing the short exposure image and the long exposure image based on the selection result. For example, it is assumed that a value indicating that a long exposure image is selected is “0” and a value indicating that a short exposure image is selected is “1”. In such a case, the synthesizing unit 60 sets α as the mixing ratio constituting the selection result, and α × (pixel value of the short exposure image) + (1−α) for the corresponding pixels in the long exposure image and the short exposure image. ) × (pixel value of the long-exposure image) is calculated, and the calculation result can be used as an image after synthesis (WDR image). The combining method by the combining unit 60 is not particularly limited.

  The compression unit 80 performs compression processing on the WDR image generated by the synthesizing unit 60 so that the bit range of the image signal having a wide dynamic range falls within a predetermined bit range. The subsequent stage of the compression unit 80 is connected to an image processing engine including, for example, a demosaic unit that generates an RGB plane from Bayer data, a contour enhancement unit, and color management. For this reason, it is preferable that the data amount of the output signal from the compression unit 80 be adjusted so as to match the size of the input data to the image processing engine (for example, about 12 bits). If the data size is simply reduced, the image is converted into a dark image. Therefore, it is preferable that the high luminance side is strongly compressed so as to approximate human visual characteristics.

  The “composite image” illustrated in FIG. 4 is an example of a composite image output by the image processing apparatus 1 according to the embodiment of the present invention. Referring to this example, it can be understood that a short-exposure image and a long-exposure image are smoothly connected at the boundary of a moving object (in this example, a person), and a composite image with reduced artifacts is obtained. . In addition, the boundary between the short-exposure image and the long-exposure image is smoothly connected in the motion region, but the adverse effect that the boundary between the short-exposure image and the long-exposure image is blurred in the non-motion region is unlikely to occur.

  FIG. 5 is a diagram illustrating an example of an operation flow of the image processing apparatus 1 according to the embodiment of the present invention. With reference to FIG. 5, a case where the selection information generated by the first use image selection unit 31 is output to the LPF 1 (51) and the second use image selection unit 32 will be described as an example. As shown in FIG. 5, first, the LPF 1 (51) removes high frequency components from the selection information (step S1). Subsequently, the motion detection unit 40 detects motion and obtains motion detection information (step S2). The LPF 2 (52) removes the high frequency component from the motion detection information (step S3).

  Further, the second use image selection unit 32 selects either the selection information before the high frequency component is removed or the selection information after the high frequency component is removed based on the motion detection information after the high frequency component is removed. Is selected for each pixel to obtain a selection result (step S4). Subsequently, the synthesizing unit 60 synthesizes the long exposure image and the short exposure image based on the selection result (step S5). As described above, the composite image is preferably compressed by the compression unit 80.

  According to the embodiment of the present invention, LPF1 (51) that removes high-frequency components from selection information indicating the mixing ratio for each corresponding pixel in the long exposure image and the short exposure image, and motion detection information is obtained by detecting the motion. Based on the motion detection unit 40, the LPF 2 (52) that removes the high frequency component from the motion detection information, and the motion detection information after the high frequency component is removed, the selection information and the high frequency component before the high frequency component is removed are removed. A second use image selection unit 32 that selects one of the selection information after each pixel and obtains a selection result; a synthesis unit 60 that combines the long exposure image and the short exposure image based on the selection result; An image processing apparatus 1 is provided.

  With such a configuration, it is possible to smoothly connect the short-exposure image and the long-exposure image in the motion region, and it is possible to reduce significant artifacts in which the contour of the moving object appears double. In addition, in an image where the subject does not move or in a non-moving area in a moving image, the boundary between the short exposure image and the long exposure image can be switched sharply, and the short exposure image and the long exposure image can be switched. It is difficult for adverse effects such as blurring of boundaries to occur.

  The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 9 Image processing apparatus 10 Sensor 20 Memory 30 Use image selection part 31 1st use image selection part 32 2nd use image selection part (selection part)
40 detector 51 LPF1 (first LPF)
52 LPF2 (second LPF)
60 Composition Unit 70 Gradation Conversion Unit 80 Compression Unit

Claims (3)

A first LPF that removes high frequency components from the selection information indicating the mixing ratio of each pixel corresponding in said length exposure image before the synthesis of the long exposure image and the short-exposure image and the short-exposure image,
A motion detector that detects motion and obtains motion detection information;
A second LPF that removes high frequency components from the motion detection information;
Based on the motion detection information after the high-frequency component is removed, either the selection information before the high-frequency component is removed or the selection information after the high-frequency component is removed is selected for each pixel to obtain a selection result. A selection section;
A combining unit that combines the long exposure image and the short exposure image based on the selection result;
An image processing apparatus comprising: The selection unit selects the selection information after the high frequency component is removed for the motion region detected from the motion detection information after the high frequency component is removed, and the motion detection information after the high frequency component is removed. For the detected non-motion region, select the selection information before the high frequency component is removed,
The image processing apparatus according to claim 1. And removing high frequency components from the selection information indicating the mixing ratio of each corresponding pixel in said length exposure image and the short-exposure image before the synthesis of the long exposure image and the short-exposure image,
Detecting motion to obtain motion detection information;
Removing high frequency components from the motion detection information;
Based on the motion detection information after the high-frequency component is removed, either the selection information before the high-frequency component is removed or the selection information after the high-frequency component is removed is selected for each pixel to obtain a selection result. Steps,
Combining the long exposure image and the short exposure image based on the selection result;
Including an image processing method.