JP2014110621A - Evaluation value calculation device and evaluation value calculation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique capable of calculating an evaluation value for dynamically determining whether a flicker is present or absent per pixel in WDR processing which synthesizes a long exposure image and a short exposure image.SOLUTION: There is provided a WDR system 10 comprising: a first calculation unit 161 which calculates a first difference value between a pixel value of a long exposure image of a current frame and a pixel value of a long exposure image of a previous frame; an average value calculation unit 163 which calculates an average value between the pixel value of the long exposure image of the current frame and the pixel value of the long exposure image of the previous frame; a second calculation unit which calculates a second difference value between the average value and a value corresponding to a pixel value of a short exposure image of the current frame; and an evaluation value calculation unit 164 which calculates an evaluation value representing the intensity of the flicker on the basis of the first difference value and the second difference value.

Description

  The present invention relates to an evaluation value calculation device and an evaluation value calculation method.

  In recent years, a short-exposure image (hereinafter also simply referred to as “short-exposure image”) and a long-exposure image (hereinafter also simply referred to as “long-exposure image”) are continuously photographed and combined. An imaging function called WDR (wide dynamic range) or HDR (high dynamic range) that obtains an image that captures a dynamic range that exceeds the dynamic range that the sensor can shoot is increasing. Such a photographing function is particularly effective in a scene with a very large contrast ratio such as a composition of backlight.

  However, there are two problems that arise from the mechanism of combining the short exposure image and the long exposure image. One problem is that if there is a movement of the subject, a shift occurs at the time of composition, and artifacts such as doubled contours occur. Another problem is that flicker may be captured in a short-exposure image, and when this short-exposure image is used for composition, a composite image including flicker is output, resulting in an image that is difficult to see. is there. In this specification, an attempt is mainly made to solve the problem of the latter flicker.

  Fluorescent lamps without inverters are well known as those that cause flicker, but some of the LED (Light Emitting Diode) lightings that have begun to spread in recent years have low flickering frequencies. May be a source of In addition, television apparatuses equipped with an LCD (Liquid Crystal Display) and displays for PCs (Personal Computers) are increasing in number to blink the backlight for the purpose of improving the video display performance. Flickering can also be a source of flicker.

  As the WDR flicker reduction method, there are the following methods. For example, there is a technique in which a flicker detection sequence over several frames is photographed before the photographing is started, and the presence / absence of flicker and the frequency are determined based on the photographing result (see, for example, Patent Document 1). In such a technique, ISO (International Organization for Standardization) sensitivity and EV (Exposure Value) sensitivity are controlled based on the determination result, and the shutter speed is set so that the influence of flicker is small.

  In addition, there is a method for detecting flicker by comparing images taken at shutter times of 1/240 seconds and 1/60 seconds (see Patent Document 2). In such a method, priorities are set for each of a plurality of WDR composite images and a single gradation-corrected image of a short exposure image based on the flicker detection result, and a plurality of WDR composite images and short images are based on the priorities. Display control or recording control of the gradation correction image of one exposure image is performed.

  Also, a technique mainly intended to align the difference between WB (White Balance) when combining images shot in an environment with a flash and WB when combining images shot in an environment without a flash. (See Patent Document 3). In this method, flicker is reduced by averaging a plurality of images. Furthermore, according to this method, the WDR effect can also be obtained by averaging a plurality of images.

JP2012-129972A JP 2012-119761 A JP 2011-35894 A

  However, the technique disclosed in Patent Document 1 determines the presence / absence of flicker and the frequency before shooting, and requires several frames for the determination, so the flicker light source starts or ends lighting during moving image shooting. It is not possible to cope with it. When there is a high-brightness flicker light source, there is a possibility that information on the high-brightness side cannot be captured sufficiently.

  Further, according to the technique disclosed in Patent Document 2, when flicker is detected, the priority order of the WDR composite image is lowered. If the priority order of the WDR composite image is lowered, for example, even if flicker exists only in a part of the screen, the WDR effect on the entire screen is lost.

  Further, according to the technique disclosed in Patent Document 3, flicker is reduced by adding and averaging a plurality of photographed images. Therefore, it is necessary to repeatedly perform photographing and addition until the flicker disappears. The WDR effect is also obtained by adding a plurality of frames, and the range expansion effect is small.

  Therefore, the present invention intends to provide a technique capable of calculating an evaluation value for dynamically determining the presence / absence of flicker for each pixel in WDR processing for combining a long exposure image and a short exposure image. Is. Further, for example, the present invention provides a technique that enables flicker to be avoided by selecting a long exposure image when flicker is detected. In addition, for example, the present invention provides a technique that makes it possible to accurately detect a flicker area and obtain a WDR-processed moving image without flicker.

  According to an embodiment of the present invention, the first calculation unit that calculates the first difference value between the pixel value of the long exposure image of the current frame and the pixel value of the long exposure image of the previous frame, and the long exposure of the current frame An average value calculation unit for calculating an average value of the pixel value of the image and the pixel value of the long exposure image of the previous frame, and a second difference value between the average value and a value corresponding to the pixel value of the short exposure image of the current frame An evaluation value calculation unit, and an evaluation value calculation unit that calculates an evaluation value representing flicker intensity based on the first difference value and the second difference value. An apparatus is provided.

  According to this configuration, it is possible to calculate an evaluation value for dynamically determining the presence or absence of flicker for each pixel. For example, according to this configuration, when flicker is detected, flicker can be avoided by selecting a long exposure image. Further, for example, according to such a configuration, it is possible to accurately detect a flicker area and obtain a WDR-processed moving image without flicker.

  The evaluation value calculation apparatus may include a mixing ratio control unit that controls a mixing ratio between the long exposure image of the current frame and the short exposure image of the current frame based on the evaluation value. According to this configuration, it is possible to control the mixing ratio of the long exposure image of the current frame and the short exposure image of the current frame based on the evaluation value calculated by the evaluation value calculation unit.

  For example, when the evaluation value is lower than the first threshold, the mixing ratio control unit may set the mixing ratio of the long exposure image in the pixel to be “0”, assuming that the pixel to be evaluated is not flicker. On the other hand, when the evaluation value exceeds the threshold value, the mixture ratio control unit may assume that the pixel to be evaluated is flicker and set the mixture ratio of the long exposure image at the pixel as the value of Clip.

  The evaluation value calculation apparatus may include a combining unit that combines the long exposure image of the current frame and the short exposure image of the current frame based on the mixing ratio controlled by the mixing ratio control unit. According to this configuration, the long exposure image of the current frame and the short exposure image of the current frame can be synthesized based on the mixture ratio as a result of the control.

  The evaluation value calculation unit may calculate the evaluation value larger as the first difference value is smaller. According to this configuration, the smaller the first difference value is, the larger the evaluation value that represents the flicker intensity can be calculated. The small first difference value means that the pixel value of the long exposure image is stable, which is one of the properties of flicker.

  The evaluation value calculation unit may calculate the evaluation value larger as the second difference value is larger. According to such a configuration, the evaluation value representing the flicker intensity can be calculated larger as the second difference value is larger. The large second difference value means that the long exposure image is different from the short exposure image, which is one of the properties of flicker.

  The mixing ratio control unit may control to increase the mixing ratio of the long exposure image of the current frame as the evaluation value increases. According to this configuration, the larger the evaluation value representing the flicker intensity, the larger the mixing ratio of the long exposure image in the current frame, the smaller the mixing ratio of the short exposure image in the current frame, and the flicker is included in the WDR composite image. The possibility of being lost can be reduced.

  According to another embodiment of the present invention, the step of calculating the first difference value between the pixel value of the long exposure image of the current frame and the pixel value of the long exposure image of the previous frame, and the long exposure image of the current frame And calculating a second difference value between the average value and a value corresponding to the pixel value of the short exposure image of the current frame. And an evaluation value representing flicker intensity based on the first difference value and the second difference value, and an evaluation value calculation method is provided.

  According to this method, it is possible to calculate an evaluation value for dynamically determining the presence or absence of flicker for each pixel. For example, according to this method, when flicker is detected, it is possible to avoid flicker by selecting a long exposure image. Further, for example, according to such a method, it is possible to accurately detect a flicker area and obtain a WDR-processed moving image without flicker.

  As described above, according to the present invention, there is provided a technique for calculating an evaluation value for dynamically determining the presence or absence of flicker for each pixel in WDR processing for combining a long exposure image and a short exposure image. Is possible. For example, according to the present invention, when flicker is detected, flicker can be avoided by selecting a long exposure image. For example, according to the present invention, it is possible to accurately detect a flicker area and obtain a WDR-processed moving image without flicker.

It is a figure which shows the function structural example of a general WDR system. It is a figure which shows the example of a short exposure image. It is a figure which shows the example of a long exposure image. It is a figure which shows the example of the WDR synthetic | combination image by a general WDR system. It is a figure which shows the function structure of the WDR system which concerns on embodiment of this invention. It is a figure which shows the detailed functional structure of a flicker detection part. It is a figure which shows the detailed functional structure of a use image selection part. It is a figure which shows the example of the time change of the pixel value of a still area. It is a figure which shows the example of the time change of the pixel value of a motion area. It is a figure which shows the example of the time change of the pixel value of a flicker area | region. It is a figure which shows the example of the relationship between evaluation value and control of a mixture ratio. It is a figure which shows the example of the flicker detection result by the WDR system which concerns on embodiment of this invention. It is a figure which shows the example of the WDR synthetic | combination image by the WDR system which concerns on embodiment of this invention. It is a figure which shows the example of the flow of operation | movement of the WDR system which concerns on embodiment of this invention.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

  In the present specification and drawings, a plurality of constituent elements having substantially the same functional configuration may be distinguished by attaching different alphabets after the same reference numeral. However, when it is not necessary to particularly distinguish each of a plurality of constituent elements having substantially the same functional configuration, only the same reference numerals are given.

  First, a functional configuration example of a general WDR system 90 will be described. FIG. 1 is a diagram illustrating a functional configuration example of a general WDR system 90. The WDR system 90 continuously shoots two images while changing the exposure setting of the sensor 910. Here, the short exposure shooting is performed first, and then the long exposure shooting is performed. The short exposure image taken by the short exposure is written in the memory 920. The WDR system 90 changes the exposure setting when short exposure shooting is completed, and performs long exposure shooting.

  Here, in the embodiment of the present invention, the terms short exposure image and long exposure image are used, but these terms do not limit the absolute exposure time of each of the two captured images. Therefore, when two images with different exposure times are taken, an image with a relatively short exposure time corresponds to a short exposure image and an image with a relatively long exposure time is a long exposure. Corresponds to an image.

  The use image selection unit 930 refers to the long exposure image detected by the sensor 910 and the short exposure image read from the memory 920, detects the saturation state and movement of each of the long exposure image and the short exposure image, and detects the short exposure image. Selection information for selecting either the exposure image or the long exposure image as the use image is generated. As an algorithm for selecting either the short exposure image or the long exposure image, various algorithms are assumed.

  For example, since a region that has been saturated in the long exposure image is highly likely not to be saturated in the short exposure image, the short exposure image may be selected as the use image of the region. However, with this process alone, artifacts such as a double outline may occur in an area where there is a large movement. Therefore, a process of detecting a motion and reducing a phenomenon that the contour becomes double may be performed. An algorithm for selecting either a short exposure image or a long exposure image including such processing is not particularly limited.

  The synthesizing unit 940 receives the selection information from the use image selecting unit 930 and generates a WDR image by synthesizing the short exposure image and the long exposure image based on the selection information. The gradation conversion unit 950 generates a compression process for bringing the bit range of an image signal having a wide dynamic range into a predetermined bit range and a gradation correction that approximates a scene viewed by human eyes by the synthesis unit 940. To the WDR image. The compression process and the gradation correction may be performed simultaneously or at different timings.

  2A is a diagram illustrating an example of a short exposure image, FIG. 2B is a diagram illustrating an example of a long exposure image, and FIG. 2C is a diagram illustrating an example of a WDR composite image by a general WDR system 90. is there.

  The composition shown in FIGS. 2A, 2B, and 2C is a backlight composition in which a person U stands in front of the window W indoors in the daytime on a sunny day. There is a display D. In the short exposure image Im-s shown in FIG. 2A, the clouds outside the window W can be clearly seen, but the person U and the room in the foreground will sink darkly. Further, since the shutter time is short, stripe flicker is visible on the display D in the vertical direction.

  The long-exposure image Im-l shown in FIG. 2B shows that the person U and the room in the foreground can be viewed with appropriate brightness, and that the shutter time is long so that there is no flicker on the display D, but the outside of the window W is too bright and saturated. The clouds are completely invisible. When these are WDR synthesized, a WDR synthesized image Im-g shown in FIG. 2C is obtained. By using the short exposure image Im-s for a region saturated with the long exposure image Im-1, the outside of the window W is obtained. The clouds become visible.

  However, depending on the algorithm, when a region where flicker exists is detected as a motion region, the short exposure image Im-s is selected by the use image selection unit 930 as a use image of the region where the flicker exists. . Therefore, the display D included in the short exposure image Im-s is combined by the combining unit 940, and the flicker of the display D can be seen.

  Subsequently, a functional configuration of the WDR system 10 according to the embodiment of the present invention will be described. FIG. 3 is a diagram showing a functional configuration of the WDR system 10 according to the embodiment of the present invention. As shown in FIG. 3, the WDR system 10 includes a sensor 110, a first memory 121, a second memory 122, a use image selection unit 130, a synthesis unit 140, a gradation conversion unit 150, and a flicker detection unit 160. Hereinafter, the functions of the respective functional blocks provided in the WDR system 10 will be sequentially described in detail. The WDR system 10 functions as an example of an evaluation value calculation apparatus according to the embodiment of the present invention.

  The sensor 110 is configured by an image sensor that forms an image of light from the outside on a light receiving plane of the image sensor, photoelectrically converts the imaged light into a charge amount, and converts the charge amount into an electric signal. The type of the image sensor is not particularly limited, and may be, for example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

  Specifically, the sensor 110 continuously shoots long exposure images and short exposure images alternately by changing exposure settings. The long-exposure image of the previous frame taken with the long exposure is written in the first memory 121. On the other hand, the short exposure image of the current frame that has been shot with short exposure is written in the second memory 122. Subsequently, the WDR system 10 changes the exposure setting when the short exposure shooting of the current frame is completed, and acquires the long exposure image of the current frame by performing the long exposure shooting. The previous frame corresponds to a frame one frame before the current frame.

  The flicker detection unit 160 includes the long exposure image of the previous frame output from the first memory 121, the short exposure image of the current frame output from the second memory 122, and the long exposure image of the current frame output from the sensor 110. Flicker detection is performed for each pixel using three images. The flicker detection result by the flicker detection unit 160 is used by the use image selection unit 130.

  FIG. 4 is a diagram illustrating a detailed functional configuration of the flicker detection unit 160. FIG. 5 is a diagram illustrating a detailed functional configuration of the use image selection unit 130. Here, the phenomenon that becomes the background of flicker detection according to the embodiment of the present invention will be described. As a precondition in the embodiment of the present invention, among the two images of the short exposure image and the long exposure image, the long exposure image contains almost no flicker and is always stable.

  In order to obtain a situation in which flicker is hardly included in the long exposure image, for example, the shutter time of the long exposure image must be 1/60 second (or 1/50 second) or less. The shooting condition for the short exposure image may be determined freely according to the specification of the dynamic range expansion performance, and is set in the range of a fraction to a tenth of the shutter time for long exposure photography.

  Based on such a premise, the difference between the stationary region, the motion region, and the flicker region will be described. 6A is a diagram illustrating an example of a temporal change in the pixel value in the still region, FIG. 6B is a diagram illustrating an example of the temporal change in the pixel value in the motion region, and FIG. 6C is a diagram illustrating the pixel value in the flicker region. It is a figure which shows the example of a time change.

  6A, 6B, and 6C, S represents a value corresponding to the pixel value of the short exposure image, and L represents the pixel value of the long exposure image. The value corresponding to the pixel value of the short exposure image is, for example, the result of normalization by multiplying the pixel value of the short exposure image by the ratio of the exposure time of the long exposure image to the short exposure image. Also good. For example, when a short exposure image is captured with an exposure time of “1/10” with respect to a long exposure image, the pixel value of the short exposure image is multiplied by “10” to thereby respond to the pixel value of the short exposure image. Values can be calculated.

  As shown in FIG. 6A, if the shooting area is a static area and the illumination environment is constant, the state where the pixel value hardly fluctuates in both the short exposure image and the long exposure image continues. On the other hand, as shown in FIG. 6B, when the shooting area is a motion area and the shooting area changes from a dark texture to a bright texture, the pixel value of the short exposure image and the pixel value of the long exposure image From the steady state where the pixel values are substantially the same, the pixel value rises through the transient state, and becomes the steady state again.

  As shown in FIG. 6C, when the shooting area is a flicker area, the flicker is included in the short-exposure image and the pixel value fluctuates greatly, whereas the long-exposure image contains almost no flicker. Therefore, there is a feature that the pixel value hardly fluctuates. The short exposure image may capture when the flicker light source blinks or may turn off, while the long exposure image captures the average brightness of them.

  Therefore, the pixel value of the short exposure image may be larger or smaller than the pixel value of the long exposure image. Referring to FIG. 6B and FIG. 6C, it is not possible to distinguish whether the shooting area is a motion area or a flicker area only by comparing the pixel values of a pair of short exposure images and long exposure images. The flicker can be accurately determined by further adding the long exposure image and comparing the three pixel values. Based on such flicker characteristics, the flicker evaluation function is defined as follows.

  The first term in the equation (1) evaluates the stability of the long exposure image, and the first calculation unit 161 calculates the pixel value of the long exposure image of the previous frame and the pixel value of the long exposure image of the current frame. Difference value (first difference value) is calculated. The smaller the difference value, the more stable the pixel value of the long exposure image. In the formula (1), as an example, the first calculation unit 161 subtracts this difference value from a constant, and the value calculated by the flicker evaluation function (hereinafter also referred to as “evaluation value”) is more stable. It is possible to evaluate that it is. When the first term takes a negative value, the value of the first term may be clipped to “0” by the first calculation unit 161.

  The second term in equation (1) evaluates the degree of divergence of the short exposure image from the long exposure image, and the average value calculation unit 163 calculates the pixel value of the long exposure image of the previous frame and the long exposure image of the current frame. An average value with the pixel value is calculated, and the second calculation unit 162 calculates a difference value (second difference value) between the average value and a value corresponding to the pixel value of the short exposure image of the current frame. A larger difference value indicates that the short exposure image is different from the long exposure image. Examples of values corresponding to the pixel values of the short exposure image of the current frame are as described above.

  Subsequently, the evaluation value calculation unit 164 calculates an evaluation value based on the difference value calculated by the first calculation unit 161 and the difference value calculated by the second calculation unit 162. The evaluation value represents the strength of flicker. As described above, the smaller the difference value calculated by the first calculation unit 161 is, the more stable the pixel value of the long-exposure image is. Therefore, for example, the evaluation value calculation unit 164 includes the difference value. The smaller the value is, the larger the evaluation value may be calculated. In addition, the larger the difference value calculated by the second calculation unit 162 is, the more the short exposure image is different from the long exposure image. Therefore, for example, the evaluation value calculation unit 164 increases the difference value. What is necessary is just to calculate an evaluation value largely.

  As an example, the evaluation value is calculated using a flicker evaluation function as shown in Expression (1). As shown in the equation (1), the flicker evaluation function is obtained by multiplying the above two terms, for example, “the long exposure image is stable” and “the short exposure image is different from the long exposure image. When the condition “Yes” is satisfied, the pixel to be evaluated is evaluated as flicker. The larger the evaluation value, the higher the possibility that the pixel to be evaluated is flicker and the stronger flicker, and the smaller the evaluation value, the lower the possibility that the pixel to be evaluated is flicker and the weak flicker. Represents that.

  Subsequently, the mixing ratio calculation unit 131 calculates the mixing ratio of the short exposure image and the long exposure image. For example, the mixing ratio calculation unit 131 may calculate the mixing ratio between the short-exposure image and the long-exposure image based on the saturation state and movement of the captured image. For example, the mixing ratio calculation unit 131 may increase the mixing ratio of the short exposure image as the saturation degree of the long exposure image is higher. Further, the mixing ratio calculation unit 131 may increase the mixing ratio of the short exposure image as the movement of the short exposure image or the long exposure image increases.

  The mixing ratio control unit 132 controls the mixing ratio between the long exposure image of the current frame and the short exposure image of the current frame based on the evaluation value calculated by the evaluation value calculation unit 164. In the embodiment of the present invention, it is assumed that the mixing ratio to be controlled by the mixing ratio control unit 132 is the mixing ratio calculated by the mixing ratio calculation unit 131. The mixing ratio may not be the mixing ratio calculated by the mixing ratio calculation unit 131.

  As the mixing ratio control method by the mixing ratio control unit 132, various methods are assumed. For example, the mixing ratio control unit 132 may perform control so that the mixing ratio of the long exposure image of the current frame increases as the evaluation value calculated by the evaluation value calculation unit 164 increases. FIG. 7 is a diagram illustrating an example of the relationship between the evaluation value and the control of the mixing ratio. For example, the mixing ratio control unit 132 may adjust the mixing ratio according to the input / output characteristics as shown in FIG.

  More specifically, as illustrated in FIG. 7, when the evaluation value is lower than the first threshold value TH1, the mixture ratio control unit 132 detects that the pixel to be evaluated is not flicker, and determines the length of the pixel. The mixing ratio of the exposure image may be “0”. On the other hand, as shown in FIG. 7, when the evaluation value exceeds the second threshold value TH2, the mixture ratio control unit 132 detects that the pixel to be evaluated is flicker, and the long exposure image of the pixel is detected. The mixing ratio may be the value of Clip. The value of Clip may be “1” or another value.

  A region between the first threshold TH1 and the second threshold TH2 corresponds to a transition region. As illustrated in FIG. 7, for example, for a pixel corresponding to the transition region, the mixture ratio control unit 132 increases the mixture ratio of the long exposure image of the current frame as the evaluation value calculated by the evaluation value calculation unit 164 increases. Control may be performed to increase the value.

  Flicker is a variation in luminance in the time direction and / or spatial direction, and many flickers can be detected by comparison using the Y signal. However, when strong flicker is included in any of the RGB components, the flicker intensity may be weakened in the Y signal. Even in such a case, the evaluation value calculation unit 164 can calculate an evaluation value for each of the RGB signals and use the calculation result to perform highly accurate flicker detection.

  More specifically, for example, when the evaluation value is calculated for each of the RGB signals, the evaluation value calculation unit 164 may use the maximum evaluation value among the evaluation values of each of the RGB signals. This is because it is considered that there is a higher possibility that flicker can be reduced if the maximum evaluation value is used.

  The synthesizing unit 140 synthesizes the long exposure image of the current frame and the short exposure image of the current frame based on the mixing ratio controlled by the mixing ratio control unit 132. For example, if the mixing ratio of the long exposure image is α, the synthesizing unit 140 calculates α × (the long exposure image of the current frame) for each corresponding pixel in the long exposure image of the current frame and the short exposure image of the current frame. (Pixel value) + (1−α) × (pixel value of the short exposure image of the current frame) can be calculated, and the calculation result can be used as a combined image (WDR image).

  The functions of the gradation conversion unit 150 are the same as the functions of the gradation conversion unit 950 described above.

  FIG. 8A is a diagram showing an example of the flicker detection result by the WDR system 10 according to the embodiment of the present invention. FIG. 8B is a diagram showing an example of a WDR composite image by the WDR system 10 according to the embodiment of the present invention. As shown in FIG. 8A, the flicker detection result Im-f includes the display D as a pixel group determined to be flicker by the mixture ratio control unit 132. Thus, according to the WDR system 10 according to the embodiment of the present invention, it is possible to accurately detect the flicker area.

  Further, as shown in FIG. 8B, referring to the WDR composite image Im-g ′, as with the WDR composite image Im-g shown in FIG. 2C, a short exposure image is output as a WDR composite image for the window W. Is grasped. Further, referring to the WDR composite image Im-g ′, it is understood that a long exposure image is output as a WDR composite image for the display D corresponding to the pixel group determined to be flicker. On the other hand, for other pixel groups (for example, the person U), it is understood that a long exposure image is output as a WDR composite image. As described above, according to the WDR system 10 according to the embodiment of the present invention, flickers appearing on the display D are removed while maintaining the performance of the WDR processing without affecting the area other than the display D. It becomes possible.

  FIG. 9 is a diagram illustrating an example of an operation flow of the WDR system 10 according to the embodiment of the present invention. As shown in FIG. 9, the first calculation unit 161 calculates a first difference value between the pixel value of the long exposure image of the current frame and the pixel value of the long exposure image of the previous frame (step S1). Subsequently, the average value calculation unit 163 calculates an average value of the pixel value of the long exposure image of the current frame and the pixel value of the long exposure image of the previous frame (step S2). The second calculation unit 162 calculates a second difference value between the average value and a value corresponding to the pixel value of the short exposure image of the current frame (step S3).

  Subsequently, the evaluation value calculation unit 164 calculates an evaluation value based on the first difference value calculated by the first calculation unit 161 and the second difference value calculated by the second calculation unit 162. (Step S4). When the mixing ratio control unit 132 controls the mixing ratio based on the evaluation value (step S5), the synthesizing unit 140 determines the long-exposure image of the current frame and the current frame based on the mixing ratio controlled by the mixing ratio control unit 132. Are combined with the short-exposure image (step S6).

  If there is a pixel for which the operation of step S1 to step S6 has not been performed (“No” in step S7), the process returns to step S1, and the operation of step S1 to step S6 is repeated for that pixel, but for all pixels When the operation from S1 to Step S6 is completed (“Yes” in Step S7), the operation from Step S1 to Step S6 is terminated.

  According to the method according to the embodiment of the present invention, it is possible to calculate an evaluation value for dynamically determining the presence / absence of flicker for each pixel in the WDR processing for combining a long exposure image and a short exposure image. . As described above, in the method according to the embodiment of the present invention, for example, when flicker is detected, it is avoided that an image including flicker is synthesized by selecting a long exposure image as a synthesized image. It is possible.

  Further, in the method according to the embodiment of the present invention, when flicker is detected, it is possible to avoid combining an image including flicker by selecting a long exposure image as a composite image. In the method according to the embodiment of the present invention, a flicker area can be accurately detected, and a WDR-processed moving image without flicker can be obtained.

  Hereinafter, effects obtained by the technique according to the embodiment of the present invention will be described in more detail. According to the embodiment of the present invention, in a WDR process in which a short-exposure image and a long-exposure image are alternately captured and combined, one short-exposure image including flicker and two long-exposure images before and after flicker are used. According to the flicker detection configuration and the evaluation function, the effect “(1) flicker can be accurately determined” is achieved.

  Further, according to the embodiment of the present invention, since flicker determination is performed for each pixel, “(2) even if a region including flicker and a region not including flicker are mixed in the captured image, only the flicker region is included. There is an effect that processing can be performed in a limited manner, and it is not necessary to turn off the WDR effect on the entire screen, and the WDR synthesis processing can be maintained. Further, according to the embodiment of the present invention, there is an effect that “(3) It is possible to respond even if the flicker light source is turned on or off during moving image shooting”.

  In the embodiment of the present invention, the condition that the long exposure image does not include flicker is necessary, and the shutter time of the long exposure image needs to be set to 1/60 seconds or 1/50 seconds or less. There is no limitation on the short exposure image side. Therefore, there is an effect that “(4) there is little deterioration in WDR performance in realizing flicker reduction”. According to the embodiment of the present invention, it is possible to perform WDR synthesis processing having the characteristics as described above.

  The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

  Note that a flicker determination according to the embodiment of the present invention adds a memory for holding a long exposure image one frame before, and this may cause a cost increase. However, since the long-exposure image one frame before is used only for flicker determination, the accuracy can be lowered. For example, even if the output data from the sensor has 12 bits, even if the data amount is reduced to about 8 bits and written in the memory, the accuracy of flicker determination is hardly affected. Further, it is difficult to think of a flicker of one pixel, and the flicker area usually has a certain area. Therefore, the resolution may be reduced and written to the memory, and the flicker determination may be performed on the low resolution image. By these means, the size of the additional memory can be greatly reduced.

  As the signal to be processed by the method according to the embodiment of the present invention, it is most preferable to use Bayer data. However, the signal used as the signal to be processed is not limited to Bayer data, and may be RGB data. Alternatively, YUV data may be used.

  In the embodiment of the present invention, as an example of WDR shooting, a short exposure image is shot first and a long exposure image is shot later. However, the shooting order of the short exposure image and the long exposure image is particularly Since it is not limited, a long exposure image may be first taken and a short exposure image may be taken later.

  In the embodiment of the present invention, the long exposure image is used for the area where flicker is detected. However, when the long exposure image is saturated, the WDR effect cannot be obtained. Therefore, there may be a case where it is desired to use a short exposure image even in an area including flicker. In that case, the condition for determining the flicker area may be weakened, or the short exposure image may be used unconditionally without the condition. These may be set according to selection by the user.

  Furthermore, a comparison result between the method described in Patent Documents 1 to 3 and the method according to the embodiment of the present invention will be described in more detail.

  The technique described in Patent Document 1 performs flicker detection before photographing. In addition, since the technique described in Patent Document 1 requires several frames for flicker detection, when the technique is applied to moving image shooting, it cannot cope with the flicker light source starting or ending during shooting. In addition, the shutter speed that is less affected by flicker is either a shutter speed of 1/50 seconds or less or synchronized with the frequency of the flicker source. However, when taking a short-exposure image, even if the ISO is lowered while reducing the shutter speed to 1/50 sec or less, the upper limit is usually about ISO 100, and the high-brightness portion is sufficiently captured because it is saturated immediately. There are many cases that cannot be done.

  Unlike the method disclosed in Patent Document 1 described above, the method according to the embodiment of the present invention can be processed in real time, and therefore can be applied to the case of moving images. In addition, the method according to the embodiment of the present invention is excellent in that there is no deterioration in WDR performance because the shooting conditions are not affected at all.

  The technique described in Patent Literature 2 adopts an image in which the priority of a plurality of WDR composite images is lowered and tone correction is performed on one short-exposure image when flicker or a large movement is detected from a captured image. The WDR effect cannot be obtained. Even if the flicker is a part of the screen, it is not reasonable that the WDR effect on the entire screen is lost.

  The technique according to the embodiment of the present invention is different from the technique disclosed in Patent Document 2 described above. The output image changes between when the flicker is present in the photographed image and when the flicker is absent in the photographed image. It is excellent that it is only the detected area and the WDR effect is not lost in other areas.

  In the method described in Patent Document 3, it is necessary to repeatedly perform photographing / addition until no flicker is detected. In the method described in Patent Document 3, the WDR effect is also obtained by adding a plurality of frames, and the range expansion effect is small.

  The method according to the embodiment of the present invention has a larger dynamic range expansion effect than the method disclosed in Patent Document 3 described above, and it is not necessary to take many images to eliminate flicker. Etc. are excellent.

10 WDR system (Evaluation value calculation device)
DESCRIPTION OF SYMBOLS 110 Sensor 121 1st memory 122 2nd memory 130 Use image selection part 131 Mixing ratio calculation part 132 Mixing ratio control part 140 Composition part 150 Gradation conversion part 160 Flicker detection part 161 1st calculation part 163 Average value calculation part 162 1st 2 calculation unit 164 evaluation value calculation unit

Claims (7)

A first calculation unit that calculates a first difference value between the pixel value of the long exposure image of the current frame and the pixel value of the long exposure image of the previous frame;
An average value calculating unit that calculates an average value of the pixel value of the long exposure image of the current frame and the pixel value of the long exposure image of the previous frame;
A second calculation unit that calculates a second difference value between the average value and a value corresponding to the pixel value of the short-exposure image of the current frame;
Based on the first difference value and the second difference value, an evaluation value calculation unit that calculates an evaluation value representing flicker intensity;
An evaluation value calculation device comprising: The evaluation value calculation device includes:
A mixing ratio control unit for controlling a mixing ratio between the long exposure image of the current frame and the short exposure image of the current frame based on the evaluation value;
The evaluation value calculation apparatus according to claim 1. The evaluation value calculation device includes:
Based on the mixing ratio controlled by the mixing ratio control unit, a combining unit that combines the long exposure image of the current frame and the short exposure image of the current frame is provided.
The evaluation value calculation apparatus according to claim 2. The evaluation value calculation unit calculates the evaluation value to be larger as the first difference value is smaller.
The evaluation value calculation apparatus according to claim 1. The evaluation value calculation unit calculates the evaluation value to be larger as the second difference value is larger.
The evaluation value calculation apparatus according to claim 1. The mixing ratio control unit controls to increase the mixing ratio of the long exposure image of the current frame as the evaluation value increases.
The evaluation value calculation apparatus according to claim 2. Calculating a first difference value between a pixel value of the long exposure image of the current frame and a pixel value of the long exposure image of the previous frame;
Calculating an average value of the pixel value of the long exposure image of the current frame and the pixel value of the long exposure image of the previous frame;
Calculating a second difference value between the average value and a value corresponding to the pixel value of the short exposure image of the current frame;
Calculating an evaluation value representing flicker intensity based on the first difference value and the second difference value;
An evaluation value calculation method including

Images