JP2023162912A - Image processing device, image processing method, and program - Google Patents

Abstract

To prevent motion blur caused by moving objects while preventing images from being captured in a state where an illuminator is turned off in an HDR image acquired by using a plurality of captured images with different exposure conditions.SOLUTION: An image processing device 100 acquires data of a first captured image and data of a second captured image captured with a shorter exposure time than the first captured image. Also, the image processing device 100 determines a synthesis ratio to be used when synthesizing pixel values of the first captured image and pixel values of the second captured image on the basis of spatial frequency components of the second captured image.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a technique for acquiring an HDR (High Dynamic Range) image from an image captured by an imaging device.

Acquire HDR image data (hereinafter also simply referred to as "HDR image data") using data of a plurality of captured images (hereinafter also simply referred to as "captured image data") captured under mutually different exposure conditions. There are known techniques to do this. When an illuminator such as a signboard or traffic light consisting of a light source that repeatedly turns on and off at high speed (hereinafter referred to as a "flicker light source") such as an LED (Light Emitting Diode) or a fluorescent lamp appears in the captured image, the following Something like this can happen. In this case, even if the illuminator appears to be lit to a natural person, the illuminator will appear unlit in a captured image captured by exposure only during the period when the flicker light source is off. Even when HDR image data is acquired using such captured image data, there is an LFM (LED Flicker Mitigation) technique for suppressing the appearance of an illuminator in an unlit state in an HDR image. On the other hand, when a moving object such as a car (hereinafter referred to as "moving object") is captured in the captured image, motion blur occurs in the image area where the moving object is captured (hereinafter referred to as "moving object area") in the long exposure image. Sometimes. Even when HDR image data is acquired using such captured image data, there is a technique for suppressing the appearance of a moving object area in an HDR image with motion blur.

Patent Document 1 discloses an image processing method for acquiring HDR image data using long exposure image data and short exposure image data. Here, a long exposure image is an image obtained by imaging with an exposure time longer than the blinking cycle of the flickering light source, and a short exposure image is an image obtained by capturing an exposure time shorter than the blinking cycle of the flickering light source. This is a captured image. Specifically, in the method disclosed in Patent Document 1 (hereinafter referred to as the "conventional method"), gain adjustment is first performed on a long exposure image and a short exposure image according to the ratio of exposure times. Next, a pixel whose absolute value of the difference in pixel values between the long exposure image and the short exposure image after gain adjustment (hereinafter referred to as "pixel difference value") is larger than a predetermined threshold value is identified as a flicker pixel. Next, for the identified flicker pixels, HDR image data is acquired by increasing the ratio of pixel values of the long-exposure image or short-exposure image after gain adjustment, depending on the imaging conditions of the captured image. Specifically, in the conventional method, under imaging conditions where a flickering light source appears in the captured image, the ratio of compositing the pixel values of the long exposure image after gain adjustment is increased for the identified flickering pixels, and the data of the HDR image is get. On the other hand, under imaging conditions in which a moving object is captured in the captured image, HDR image data is acquired by increasing the ratio of pixel values of the short exposure image after gain adjustment for the identified flicker pixels.

In this way, in the conventional method, even if flicker pixels in a short exposure image are captured with the flicker light source turned off, they are suppressed from appearing in the HDR image with the flicker light source turned off. Further, even if a long exposure image has motion blur caused by a moving object, the moving object region in the HDR image is suppressed from being captured with motion blur.

Special table number 2019-517215

However, in the conventional method, if the ratio of compositing the pixel values of the long exposure image after gain adjustment is increased in a captured image obtained under imaging conditions such that both a flickering light source and a moving object are captured in the captured image, the HDR image Motion blur caused by moving objects. Conversely, increasing the ratio of compositing pixel values of short-exposure images after gain adjustment can suppress motion blur caused by moving objects in HDR images, but it can also suppress motion blur caused by moving objects in HDR images. Sometimes it happens.

An image processing device according to the present disclosure includes an image acquisition unit that acquires data of a first captured image and data of a second captured image captured with a shorter exposure time than the first captured image; ratio determining means for determining a combination ratio to be used when combining pixel values of the first captured image and pixel values of the second captured image, based on spatial frequency components of the second captured image. .

According to the present disclosure, in an HDR image acquired using a plurality of captured images captured under different exposure conditions, it is possible to suppress motion blur caused by a moving object while suppressing an image in which the illuminator is turned off. can.

1 is a block diagram illustrating an example of a functional configuration of an image processing apparatus according to a first embodiment; FIG. 1 is a block diagram illustrating an example of a hardware configuration of an image processing apparatus according to a first embodiment. FIG. FIG. 2 is an explanatory diagram for explaining an example of a method of capturing a long exposure image and a short exposure image according to the first embodiment. 3 is a graph showing an example of input/output characteristics of the image sensor according to the first embodiment. FIG. 3 is an explanatory diagram for explaining an example of a method for determining a first combination ratio according to the first embodiment. FIG. 3 is a block diagram showing an example of a functional configuration of a correction term determining unit according to the first embodiment. 7 is a graph showing an example of the relationship between a correction term and a flicker evaluation value. 3 is a flowchart illustrating an example of a processing flow of the image processing apparatus according to the first embodiment. FIG. 7 is an explanatory diagram for explaining an example of a method for determining a first provisional correction term and a second provisional correction term according to the second embodiment. 7 is a flowchart illustrating an example of a processing flow of an image processing apparatus according to a second embodiment.

Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that the configurations shown in the embodiments below are merely examples, and the scope of the present disclosure is not limited to only those configurations.

[Embodiment 1]
An image processing apparatus 100 according to a first embodiment will be described with reference to FIGS. 1 to 8. First, the configuration of the image processing apparatus 100 will be described with reference to FIGS. 1 and 2. FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus 100 according to the first embodiment. The image processing device 100 is applied to the image processing system 1, for example. The image processing system 1 includes an image processing device 100 and an imaging section 110. The image processing device 100 and the imaging unit 110 are communicably connected to each other via a dedicated line or a network such as a LAN (Local Area Network).

The imaging unit 110 is an imaging device such as a digital still camera or a digital video camera. Specifically, for example, the imaging unit 110 includes an optical system including a lens and the like, an image sensor and an image processing unit including a CMOS (Complementary Metal-Oxide-Semiconductor), and the like. The imaging unit 110 outputs data of a captured image obtained by imaging (hereinafter also referred to as "captured image data"). Specifically, the imaging unit 110 outputs data of a long exposure image 107 and data of a short exposure image 108, which have different exposure times, as captured image data.

Here, the long exposure image 107 is one of the blinking cycles of a light source (hereinafter referred to as a "flicker light source") that repeats turning on and off at high speed, such as an LED (Light Emitting Diode) or a fluorescent lamp, which is the object to be imaged. This is a captured image obtained when the image is captured with an exposure time longer than the period. Further, the short exposure image 108 is a captured image captured with an exposure time shorter than that of the long exposure image 107, and is an image obtained when the image is captured with an exposure time of less than one cycle of the flickering cycle of the flickering light source of the imaging target. It is an image.

A method for capturing the long exposure image 107 and the short exposure image 108 will be described with reference to FIG. 3. FIG. 3 is an explanatory diagram for explaining an example of a method for capturing the long exposure image 107 and the short exposure image 108 in the imaging unit 110 according to the first embodiment. The long exposure image 107 and the short exposure image 108 are captured, for example, by sequentially exposing each pixel of the image sensor of the imaging unit 110 twice. Specifically, for example, first, a long exposure image 107 is captured by the first exposure, and after the exposure is completed, output signal values (hereinafter also referred to as "pixel values") are read out from the image sensor. Next, a short exposure image 108 is captured by a second exposure, and after the exposure is completed, output signal values (pixel values) are read from the image sensor. For example, as shown in FIG. 3 as an example, a long exposure image 107 and a short exposure image 108 are obtained by performing such exposure while changing the image position to be imaged and the exposure timing.

Referring to FIG. 4, images of the exposure when capturing the long exposure image 107 (hereinafter referred to as "long exposure") and the exposure when capturing the short exposure image 108 (hereinafter referred to as "short exposure") The input/output characteristics of the photoelectric element of the sensor will be explained. FIG. 4 is a graph showing an example of the input/output characteristics of the photoelectric element of the image sensor according to Embodiment 1, and shows input illuminance and output signal value (pixel value) of the photoelectric element of the image sensor during long exposure and short exposure. ) is a graph showing an example of the relationship (input/output characteristics). In FIG. 4, the horizontal axis represents the magnitude of the illuminance of light input to the photoelectric element of the image sensor (input illuminance), and the vertical axis represents the magnitude of the output signal value (output pixel value) read from the photoelectric element. Show that. Hereinafter, the input illuminance will be explained as being constant regardless of the passage of time.

In FIG. 4, a solid line 400 represents an example of the input/output characteristics of the photoelectric element of the image sensor during long exposure. Furthermore, a broken line 401 represents an example of the input/output characteristics of the photoelectric element of the image sensor during short exposure. During long exposure, the output signal value increases linearly as the input illuminance increases until the photoelectric element of the image sensor is saturated, and becomes constant at the saturated output signal value when the input illuminance exceeds a certain value. As with long exposure, during short exposure, the output signal value increases linearly as the input illuminance increases until the photoelectric element of the image sensor is saturated, and when the input illuminance exceeds a certain value, the output signal value reaches the saturated output signal value. becomes constant. However, the rate of increase in the output signal value during short exposure with respect to input illuminance is slower than that during long exposure. Image sensors such as CMOS generally accumulate electric charge in photodiodes corresponding to each pixel due to incident light, and output a digital signal value proportional to the amount of accumulated electric charge as an output signal value. Therefore, the amount of charge accumulated in the photodiode is shorter during short exposure than during long exposure, so the amount of charge accumulated at the same input illuminance is smaller during short exposure. As a result, the input/output characteristics as shown in FIG. 4 are obtained.

The image processing device 100 includes an image acquisition section 101 , a first ratio determination section 102 , a correction term determination section 103 , a second ratio determination section 104 , and an image composition section 105 . Note that in the first embodiment, the image processing apparatus 100 is described as having the image composition section 105 as an example, but the image composition section 105 may be provided outside the image processing apparatus 100. In this case, for example, the image synthesis unit 105 is provided inside an information processing device configured by a computer or the like (not shown in FIG. 1) included in the image processing system 1.

Processing of each part included in the image processing apparatus 100 is performed by hardware such as an ASIC (Application Specific Integrated Circuit) built into the image processing apparatus 100. The processing may be performed by hardware such as an FPGA (Field Programmable Gate Array). The imaging unit 110 and the image capturing device 100 may be integrated. In this case, for example, the processing of each part of the image processing device 100 may be performed by an integrated circuit integrated with the CMOS constituting the imaging unit 110, or by another integrated circuit connected to the CMOS. good. Further, the processing may be performed by software using a memory such as a RAM (Random Access Memory) and a processor such as a CPU (Central Processor Unit).

With reference to FIG. 2, the hardware configuration of the image processing apparatus 100 in the case where each unit included in the image processing apparatus 100 according to the first embodiment operates as software will be described. FIG. 2 is a block diagram showing an example of the hardware configuration of the image processing apparatus 100 according to the first embodiment. The image processing apparatus 100 is configured by a computer, and as shown in FIG. It has 208.

The CPU 201 is a processor that controls the computer using programs or data stored in the ROM 202 or the RAM 203, thereby causing the computer to function as each unit included in the image processing apparatus 100 shown in FIG. Note that the image processing apparatus 100 may include one or more dedicated hardware different from the CPU 201, and the dedicated hardware may execute at least part of the processing by the CPU 201. Examples of dedicated hardware include ASICs, FPGAs, and DSPs (digital signal processors). The ROM 202 is a memory that stores programs that do not require modification. The RAM 203 is a memory that temporarily stores programs or data supplied from the auxiliary storage device 204, or data supplied from the outside via the communication unit 207. The auxiliary storage device 204 is configured by, for example, a hard disk drive, and stores programs or various data such as image data or audio data.

The display unit 205 is configured with, for example, a liquid crystal display or an LED, and displays a GUI (Graphical User Interface) or the like for the user to operate the image processing apparatus 100 or view the processing status in the image processing apparatus 100. The operation unit 206 includes, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and inputs various instructions to the CPU 201 in response to user operations. The CPU 201 also operates as a display control unit that controls the display unit 205 and an operation control unit that controls the operation unit 206.

The communication unit 207 is used for communication such as sending and receiving data, etc., to and from devices external to the image processing apparatus 100. For example, when the image processing apparatus 100 is connected to an external device by wire, a communication cable is connected to the communication unit 207. When the image processing apparatus 100 has a function of wirelessly communicating with an external device, the communication unit 207 includes an antenna. A bus 208 connects each unit of the image processing apparatus 100 and transmits information. In the first embodiment, the display section 205 and the operation section 206 will be described as being present inside the image processing apparatus 100. However, at least one of the display section 205 and the operation section 206 is located outside the image processing apparatus 100. It may exist as a device.

The processing of each unit shown in FIG. 1 included in the image processing apparatus 100 will be described. The image acquisition unit 101 acquires data of a long exposure image 107 and data of a short exposure image 108. Specifically, for example, the image acquisition unit 101 acquires the data of the long exposure image 107 and the data of the short exposure image 108 output by the imaging unit 110 via the network. The image acquisition unit 101 retrieves the data of the long exposure image 107 and the data of the short exposure image 108 from a storage device (not shown in FIG. 1) in which the data of the long exposure image 107 and the data of the short exposure image 108 are stored in advance. The data may be acquired by reading it. In this case, for example, the imaging unit 110 outputs the data of the long exposure image 107 and the data of the short exposure image 108 to the storage device, and stores the data in the storage device in advance.

The first ratio determination unit 102 determines the pixel value L and the pixel value of the short exposure image 108 (hereinafter referred to as "pixel value S") based on the pixel value of the long exposure image 107 (hereinafter referred to as "pixel value L"). A provisional synthesis ratio (hereinafter referred to as "first synthesis ratio") to be used when synthesizing the ``first synthesis ratio'' is determined. Specifically, the first ratio determining unit 102 determines, for each pixel of the long exposure image 107, the value of the pixel of the long exposure image 107 (pixel value L) and the value of the pixel of the short exposure image 108 corresponding to the pixel ( A first combination ratio to be used when combining the pixel value S) is determined.

A method for determining the first combination ratio will be described with reference to FIG. 5. FIG. 5 is an explanatory diagram for explaining an example of a method for determining the first combination ratio (α) according to the first embodiment. In FIG. 5, the horizontal axis indicates the magnitude of the pixel value L, and the vertical axis indicates the magnitude of the first synthesis ratio (α). Further, in FIG. 5, a polygonal line 500 indicated by a solid line represents an example of the relationship between the pixel value L and the first combination ratio (α) determined by the first ratio determination unit 102. As shown in FIG. 5, the first synthesis ratio (α) is 0 or 0 when the pixel value L is less than Th1_L, which is a predetermined first threshold (hereinafter referred to as "first pixel threshold"). (hereinafter referred to as "approximately 0"). On the other hand, if the pixel value L is greater than or equal to Th1_H, which is a predetermined second threshold (hereinafter referred to as "second pixel threshold") that is greater than or equal to the first pixel threshold, then the pixel value L is 1 or a value close to 1 (hereinafter referred to as (referred to as "approximately 1"). Hereinafter, description will be made assuming that 1 includes approximately 1 and 0 includes approximately 0. Further, when the pixel value L is greater than or equal to Th1_L and less than Th1_H, the pixel value L is determined to be any value from 0 to 1 that increases linearly with respect to the pixel value L, for example. Specifically, the first ratio determination unit 102 determines the first composite ratio (α) using the following equation 1, for example.

Here, α is a first synthesis ratio determined by the first ratio determination unit 102, and in the second synthesis ratio actually used when combining the pixel value L and the pixel value S, This is the basic composite ratio excluding the above. Further, L is a pixel value (pixel value L) of the long exposure image 107. Note that α indicates the rate at which the pixel values S are combined, and the value (1−α) obtained by subtracting α from 1 indicates the rate at which the pixel values L are combined. In this way, the first ratio determination unit 102 determines the first combination ratio (α) so that as the pixel value L increases from Th1_L to Th1_H, the ratio of combining the pixel values S gradually increases. Although the above description assumes that the minimum value of the first synthesis ratio (α) is 0, the minimum value of the first synthesis ratio (α) is not limited to 0, and may be a positive value near 0. It may be a real number. Further, in the above description, the maximum value of the first synthesis ratio (α) is assumed to be 1, but the maximum value of the first synthesis ratio (α) is not limited to 1, but is close to 1 and less than 1. may be a positive real number.

With this configuration, in the HDR (High Dynamic Range) image obtained by combining the pixel value L and the pixel value S, there is a difference in gradation at the joint between the long exposure image 107 and the short exposure image 108. This can be prevented from occurring. Furthermore, it is possible to prevent the obtained HDR image from appearing as an artificially created image (artifact). Note that the difference in gradation is caused by a difference in noise value between the long exposure image 107 and the short exposure image 108, and the like.

The correction term determining unit 103 determines a correction term for correcting the first synthesis ratio (α) based on the spatial frequency component of the short exposure image 108. Specifically, first, based on the pixel value L and the pixel value S, the correction term determination unit 103 sets the ratio of combining the pixel value L when combining the pixel value L and the pixel value S so that the ratio of combining the pixel value L is increased. A temporary correction term (hereinafter referred to as "temporary correction term") for correcting the first synthesis ratio (α) is obtained. Next, the correction term determination unit 103 determines a correction term for correcting the first synthesis ratio (α) by changing the acquired provisional correction term based on the spatial frequency component of the short exposure image 108. The internal configuration of correction term determining section 103 will be described with reference to FIG. 6. FIG. 6 is a block diagram illustrating an example of the functional configuration of the correction term determination unit 103 according to the first embodiment. The correction term determination unit 103 includes a temporary correction term acquisition unit 601 , a clip unit 602 , a first filter unit 603 , a second filter unit 604 , and a change unit 605 .

The temporary correction term acquisition unit 601 obtains, for each pixel of the long exposure image 107, the value of the pixel of the long exposure image 107 (pixel value L) and the value of the pixel of the short exposure image 108 corresponding to the pixel (pixel value S). A temporary correction term (temporary correction term) to be used when synthesizing is determined and obtained. More specifically, the temporary correction term acquisition unit 601 determines a temporary correction term for each pixel based on the output ratio between the long exposure image 107 and the short exposure image 108, and the pixel value L and the pixel value S. . The temporary correction term acquisition unit 601 first calculates a flicker evaluation value in order to determine a temporary correction term. The flicker evaluation value is determined by combining the pixel value of the long exposure image 107 (pixel value L), the pixel value of the short exposure image 108 corresponding to the pixel (pixel value S), and the long exposure image 107 and the short exposure image 108. This is the difference value from the pixel value of the short exposure image 108 after gain adjustment multiplied by the output ratio. Hereinafter, the short exposure image 108 after gain adjustment will be referred to as an "adjusted short exposure image". Specifically, the provisional correction term acquisition unit 601 calculates the flicker evaluation value (f) using the following equation 2, for example.

Here, f is the flicker evaluation value calculated by the temporary correction term acquisition unit 601, L is the pixel value of the long exposure image 107 (pixel value L), and S is the pixel value of the short exposure image 108 (pixel value value S), R is the power ratio. The output ratio (R) is the exposure time when capturing the long exposure image 107 (hereinafter referred to as "long exposure time") and the exposure time when capturing the short exposure image 108 (hereinafter referred to as "short exposure time"). ), and is calculated using the following equation 3, for example.

Here, E_L is the long exposure time and E_S is the short exposure time. The output ratio (R) is a value determined only by the long exposure time (E_L) and the short exposure time (E_S). Therefore, it corresponds to the value of the ratio between the pixel value L and the pixel value S of the pixel corresponding to the photoelectric element of the image sensor where the input illuminance does not change over time, that is, the light other than the light from the flicker light source is incident. However, this is a case where the input illuminance incident on the pixel is smaller than the value at which the pixel value L and the pixel value S reach the saturated output signal value. Therefore, on the premise that the pixel value L and the pixel value S are smaller than the saturated output signal value, the flicker evaluation value (f ) is 0 or a value close to 0. On the other hand, the flicker evaluation value (f) of a pixel corresponding to a photoelectric element of an image sensor into which light from a flicker light source is incident is a correspondingly large value.

Referring to FIG. 7, the relationship between the temporary correction term determined by the temporary correction term acquisition unit 601 and the flicker evaluation value (f) calculated by the temporary correction term acquisition unit 601 will be described. FIG. 7 is a graph showing an example of the relationship between the temporary correction term (b) and the flicker evaluation value (f). In FIG. 7, the horizontal axis indicates the magnitude of the flicker evaluation value (f), and the vertical axis indicates the magnitude of the temporary correction term (b). In FIG. 7, a solid line 700 indicates an example of the output characteristic of the temporary correction term (b) when the flicker evaluation value (f) is input. Further, in FIG. 7, Th2_L indicates the first threshold of the flicker evaluation value (hereinafter referred to as "first evaluation threshold"), and Th2_H indicates the second threshold of the flicker evaluation value (hereinafter referred to as "second evaluation threshold"). ). The second evaluation threshold is a value greater than or equal to the first evaluation threshold. If the flicker evaluation value (f) is less than the first evaluation threshold (Th2_L), the temporary correction term acquisition unit 601 determines that the target pixel is a pixel on which light from a flicker light source enters (hereinafter referred to as a "flicker pixel"). ), the provisional correction term (b) is determined to be 0. Furthermore, if the flicker evaluation value (f) is greater than or equal to the first evaluation threshold (Th2_L) and less than the second evaluation threshold (Th2_H), the target pixel is determined to be a flicker pixel. Then, the temporary correction term (b) is determined to be any value between 0 and 1 that increases linearly with respect to the flicker evaluation value (f). Furthermore, if the flicker evaluation value (f) is equal to or greater than the second evaluation threshold (Th2_H), the target pixel is assumed to be a flicker pixel (f), and the temporary correction term (b) is determined to be 1. Here, the flicker pixel means a pixel corresponding to a photoelectric element of an image sensor into which light from a flicker light source is incident. Specifically, the provisional correction term acquisition unit 601 determines the provisional correction term (b) using the following equation 4, for example.

Here, b is the value of the temporary correction term calculated by the temporary correction term acquisition unit 601. In addition, although the above explanation assumes that the minimum value of the provisional correction term (b) is 0, the minimum value of the provisional correction term (b) is not limited to 0, but may be a positive real number near 0. There may be. In addition, although the above description assumes that the maximum value of the provisional correction term (b) is 1, the maximum value of the provisional correction term (b) is not limited to 1; may be a real number. The pixel value L and the pixel value S have fluctuations in pixel value even if the pixel is not affected by flicker due to the influence of noise, so the flicker evaluation value (f) may not be completely zero. Therefore, as described above, it is preferable to provide an interval in which the temporary correction term (b) is 0 or a value close to 0.

The clipping unit 602 performs, for example, low-luminance clipping processing expressed by the following Equation 5 on each pixel value of the input adjusted short exposure image.

Here, M is a pixel value of the adjusted short exposure image, and M' is a pixel value after low brightness clipping processing is performed on the pixel value (M) of the adjusted short exposure image. Further, th_clip is a brightness threshold value in low brightness clip processing, and is an arbitrary value within the gradation range of each pixel value (M) in the adjusted short exposure image. By setting th_clip in consideration of the characteristics of the image sensor or the exposure time during image capture, the adjusted short exposure image after low-brightness clipping is an image in which the low-brightness gradation where noise is likely to occur is clipped by th_clip. becomes. Hereinafter, the adjusted short-exposure image after the low-luminance clipping process will be referred to as a clipped short-exposure image.

The first filter section 603 is a filter that allows only a predetermined first spatial frequency component of the clip short exposure image to pass through. Specifically, for example, the first filter section 603 reduces high frequency components higher than the first spatial frequency and passes only low frequency components lower than the first spatial frequency. In this case, for example, the first filter section 603 is configured with a low-pass filter such as an averaging filter or a Gaussian filter. Further, the second filter section 604 is a filter that passes only a predetermined second spatial frequency component. Specifically, for example, the second filter unit 604 reduces high frequency components higher than a predetermined second spatial frequency and lowers high frequency components lower than the second spatial frequency in the image after passing through the first filter unit 603. Passes only low frequency components. In this case, for example, the second filter section 604 is configured with a low-pass filter such as an averaging filter or a Gaussian filter.

Note that the configurations of the first filter section 603 and the second filter section 604 are not limited to low-pass filters, but are configurations that can pass predetermined spatial frequency components of the clip short exposure image. Good to have. Therefore, the first filter section 603 and the second filter section 604 may be configured by, for example, a high-pass filter or a band-pass filter. Further, it may be configured by combining a low-pass filter, a high-pass filter, a band-pass filter, or the like. In addition, in this embodiment, a mode is shown in which the first filter section 603 and the second filter section 604 are configured as two filter sections, but the first filter section 603 and the second filter section 604 are integrated to form one filter section. It may also be configured by a filter section. Further, the first filter section 603 and the second filter section 604 may be configured by three or more filter sections including the first filter section 603 and the second filter section 604.

The changing unit 605 uses the images that have passed through the first filter unit 603 and the second filter unit 604 to change the temporary correction term (b) acquired by the temporary correction term acquisition unit 601, and performs the first synthesis. A correction term for correcting the ratio (α) is determined. For example, the changing unit 605 first uses the images that have passed through the first filter unit 603 and the second filter unit 604 to change only the high frequency component, medium frequency component, or low frequency component of the clipped short exposure image. Get the image containing. Specifically, for example, the changing unit 605 includes a subtracting unit not shown in FIG. 6, and the subtracting unit subtracts the image after passing through the second filter unit 604 from the image after passing through the first filter unit 603. to obtain the image after subtraction. Note that the subtraction section is configured by, for example, a well-known subtraction circuit. By changing the settings of the first filter section 603, the second filter section 604, and the subtraction section, it is possible to obtain an image containing only high frequency components, medium frequency components, or low frequency components of the clipped short exposure image. can. Hereinafter, the process of using the first filter section 603, the second filter section 604, and the subtraction section to obtain an image containing only predetermined frequency components from the input image will be referred to as filtering processing.

For example, when it is desired to obtain an image containing only high frequency components, the first filter section 603 does not perform the processing, and the second filter section 604 sets an appropriate filter size and performs the processing. Further, a subtraction unit subtracts the image passed through the second filter unit 604 from the image passed through the first filter unit 603. In this way, the bandwidth of the image to be acquired can be changed by changing the filter size settings. It is possible to obtain only contour information etc.

Further, for example, if it is desired to obtain an image containing only medium frequency components, the filter sizes of the first filter section 603 and the second filter section 604 are set as follows. In this case, the first filter section 603 is set as a 3x3 tap averaging filter with filter coefficients of 111/111/111. The second filter section 604 is set as a 5x5 tap averaging filter having filter coefficients of 11111/11111/11111/11111/11111. The first filter section 603 cuts high frequency components, and the second filter section 604 cuts high frequency components in a frequency band lower than the high frequency components cut by the first filter section 603. By subtracting the image that has passed through the second filter section 604 from the image that has passed through the first filter section 603 in the subtraction section, it is possible to obtain an image that includes only medium frequency components. In this way, the bandwidth of the image to be acquired can be changed by changing the filter size settings, and by setting the filter size, for example, it is possible to obtain contour information, etc. in an image in which high frequency bands containing noise are cut in clip short exposure. I can do it.

For example, if you want to obtain an image containing only low frequency components, for example, the first filter section 603 sets an appropriate filter size and performs the processing, and the second filter section 604 and the subtraction section both perform the processing. Avoid implementing it. With this configuration, the image that has passed through the first filter section 603 configured with a low-pass filter includes only low frequency components. The bandwidth of the image to be acquired can be changed by changing the filter size settings.For example, by setting the filter size, you can obtain images with gradual changes in brightness for areas where there is a sudden change in brightness during clip short exposure. be able to.

In the changing unit 605, for example, an averaging filter or a Gaussian filter is used as the acquisition low-pass filters 311 and 312 in the temporary correction term acquisition unit 601. The low-pass filter 311 passes low frequency components below a predetermined frequency in the short-time exposure image. The low-pass filter 311 passes low frequency components below a predetermined frequency in the short-time exposure image. The correction term is determined by changing the value of the provisional correction term (b). Specifically, the changing unit 605 changes the value of the temporary correction term (b) so that the ratio of combining the pixel values S increases. Specifically, the changing unit 605 determines the correction term (b') using Equation 6 below, for example.

Here, N is the pixel value (tone value). Further, Val_mod is an arbitrary real number specified in advance between 0 and -1. Th_mod is a threshold value of the temporary correction term, and is an arbitrary real number specified in advance within the range of the temporary correction term (b), that is, between 0 and 1.

A pixel for which the value of the temporary correction term (b) is close to 1 is highly likely to be a flicker pixel. Further, as the value of the correction term (b') approaches 1, the ratio of combining pixel values L increases, and as the value of the correction term (b') approaches 0, the ratio of combining pixel values S increases. Therefore, for example, if Th_mod is set to a value close to 1, the temporary correction term (b) will be changed to increase the ratio of compositing pixel values S for pixels that are likely to be flicker pixels. become. As a result, in this case, there is a possibility that the HDR image generated by synthesis may appear with the flicker light source turned off. On the other hand, by setting Th_mod to 0 or a value close to 0, the temporary correction term (b) is changed so that the ratio of compositing the pixel value L for pixels that are likely to be flicker pixels is increased. The correction term (b') is determined. As a result, in this case, it is possible to suppress the flicker light source from appearing in a turned-off state in the HDR image generated by synthesis.

Th_fil is a spatial frequency threshold, and is an arbitrary pixel value (gradation value) specified in advance within the gradation range of the image after filtering processing. For example, if the image after the filtering process is a clipped short exposure image in which the high frequency band has been cut and includes contour information, the correction term (b') is determined as follows. In this case, the temporary correction term (b) of the pixel corresponding to the contour portion in the clipped short exposure image is changed so that the ratio of combining the pixel values S is increased, and the correction term (b') is determined.

The second ratio determination unit 104 determines the pixel value L based on the first synthesis ratio (α) determined by the first ratio determination unit 102 and the correction term (b′) determined by the correction term determination unit 103. A second combination ratio to be used when actually combining the pixel value S and the pixel value S is determined for each pixel. For example, the second ratio determination unit 104 determines a value obtained by subtracting the value of the correction term (b') from the first combination ratio (α) as the second combination ratio. The second combination ratio (d) is expressed using the following equation 7, for example.

Here, max(x0, x1) is a function that returns the maximum value of x0 and x1. The second combination ratio (d) represents the final combination ratio used when combining the pixel value L and the pixel value S, and is determined for each pixel. When pixel values L and pixel values S are combined to generate an HDR image, the ratio of combining pixel values S is the second combination ratio (d), and the ratio of combining pixel values L is from 1 to the second combination ratio The value obtained by subtracting (d) is (1-d). In the case of a pixel for which the temporary correction term (b) has not been changed by the changing unit 605, the value of the correction term (b') is the same as the value of the temporary correction term (b), so the first synthesis ratio ( The second combination ratio (d) is calculated using α) and the value of the temporary correction term (b). On the other hand, in the case of a pixel for which the temporary correction term (b) has been changed by the changing unit 605, the value of the correction term (b') is a negative value from -1 to 0, and the second synthesis ratio (d ) is larger than the value of the first synthesis ratio (α). Therefore, in combining pixel values L and pixel values S, the ratio of combining pixel values S increases.

The effects of different filtering processes will be explained. First, a case will be described in which the image after filtering processing is an image obtained by extracting only high frequency components from a clipped short exposure image. In this case, the temporary correction term (b) of the pixel corresponding to the outline portion of the clipped short exposure image is changed so that the ratio of combining the pixel values S is increased. Therefore, in the outline portion of the HDR image generated by combining the pixel values L and the pixel values S, motion blur caused by imaging a moving object such as a car (hereinafter referred to as a "moving object") is reduced. Note that the high frequency components of the image may include noise or excessively fine texture. Therefore, when changing the temporary correction term (b) using an image that simply extracts only the high frequency components of the clipped short exposure image, the temporary correction term (b) for each pixel corresponding to almost the entire surface of the image is There are cases where the ratio of combining the values S is changed so as to increase.

Next, a case will be described in which the image after the filtering process is an image obtained by extracting only the medium frequency components of the clipped short exposure image. In this case, the image after the filtering process is an image in which noise or excessively fine texture is removed from the clipped short exposure image. Therefore, the temporary correction term (b) of the pixel corresponding to the outline portion of the clipped short exposure image is changed so that the ratio of combining the pixel values S is increased. Therefore, in the contour portion of the HDR image generated by combining the pixel value L and the pixel value S, motion blur caused by imaging the moving object is reduced.

Next, a case will be described in which the image after the filtering process is an image obtained by extracting only the low frequency components of the clipped short exposure image. For example, by setting a large number of taps in the first filter unit 603 when extracting low frequency components of a clipped short exposure image, an image in which image areas with large brightness values and image areas with small brightness values are separated can be filtered. It can be extracted as an image. Therefore, the temporary correction term (b) for pixels corresponding to bright image areas with large luminance values in the clipped short exposure image is changed so that the ratio of combining pixel values S is increased. Therefore, motion blur is reduced in the HDR image generated by combining the pixel value L and the pixel value S, in which the luminance value is large, that is, in the bright portion.

The image synthesis unit 105 synthesizes the pixel value L and the pixel value S based on the second synthesis ratio (d). Specifically, the image synthesis unit 105 synthesizes the pixel value (pixel value L) of the long exposure image 107 and the pixel value (pixel value S) of the short exposure image 108 corresponding to the pixel for each pixel. Then, an HDR image is generated. The pixel value (p) of the HDR image is expressed using the following equation 8, for example.

As shown in Equation 8, the pixel value L is the ratio of the value (1-d) obtained by subtracting the second combination ratio (d) from 1, and the pixel value (R・S) of the adjusted short exposure image is the second combination ratio (1-d). d) are added up at the rate.

The operation of the image processing device 100 will be described with reference to FIG. 8. FIG. 8 is a flowchart illustrating an example of a processing flow of the image processing apparatus 100 according to the first embodiment. Note that in the explanation of FIG. 8, the symbol "S" means a step. First, in S801, the image acquisition unit 101 acquires data of the long exposure image 107 and the short exposure image 108. Next, in S802, the first ratio determination unit 102 determines a first synthesis ratio (α) for each pixel. Next, in S803, the correction term determining unit 103 obtains a temporary correction term (b) for each pixel. Next, in S804, the correction term determining unit 103 determines a correction term (b') for each pixel. Next, in S805, the second ratio determination unit 104 determines a second combination ratio (d) for each pixel. Next, in S806, the image synthesis unit 105 synthesizes the pixel value L and the pixel value S for each pixel to generate an HDR image. After S806, the image processing apparatus 100 ends the process of the flowchart shown in FIG.

According to the image processing apparatus 100 configured as described above, in an HDR image acquired using a plurality of captured images captured under different exposure conditions, moving object It is possible to suppress motion blur due to

[Embodiment 2]
An image processing apparatus 100 according to a second embodiment will be described with reference to FIGS. 1, 9, and 10. The image processing device 100 according to the first embodiment calculates a difference value (flicker evaluation value (f )), the provisional correction term (b) for each pixel was obtained. In contrast, the image processing apparatus 100 according to the second embodiment acquires a temporary correction term based on the similarity between the long exposure image 107 and the short exposure image 108. Note that the image processing apparatus 100 according to the first embodiment performs processing on the premise that the closer the value of the temporary correction term (b) is to 1, the higher the possibility that the pixel is a flicker pixel. However, the image processing apparatus 100 according to the first embodiment simply identifies whether a pixel is a flicker pixel based on the flicker evaluation value (f). Therefore, even if the value of the temporary correction term (b) is close to 1, the pixel may be identified as a flickering pixel even though it is not actually a flickering pixel. Furthermore, there have been cases where a pixel specified as a flicker pixel is a pixel in which motion blur occurs (hereinafter referred to as a "motion blur pixel") even though it is not actually a flicker pixel.

The image processing apparatus 100 according to the second embodiment (hereinafter simply referred to as "image processing apparatus 100"), like the image processing apparatus 100 according to the first embodiment, includes an image acquisition unit 101 and a first It includes a ratio determining section 102, a correction term determining section 103, a second ratio determining section 104, and an image combining section 105. Similar to the image processing apparatus 100 according to the first embodiment, the processing of each part included in the image processing apparatus 100 is performed by hardware such as ASIC or FPGA built into the image processing apparatus 100. The processing may be performed by software using a memory such as a RAM and a processor such as a CPU. Further, the image processing apparatus 100 is applied to the image processing system 1 shown as an example in FIG. 1, similarly to the image processing apparatus 100 according to the first embodiment. Note that in the second embodiment, the image processing apparatus 100 will be described as including the image composition section 105 as an example, but the image composition section 105 may be provided outside the image processing apparatus 100. In this case, for example, the image synthesis unit 105 is provided inside an information processing device configured by a computer or the like (not shown in FIG. 1) included in the image processing system 1.

With reference to FIG. 1, the processing of each unit included in the image processing apparatus 100 will be described. Note that the image acquisition unit 101, the first ratio determination unit 102, and the image composition unit 105 according to the second embodiment are the same as the corresponding units according to the first embodiment, so a description thereof will be omitted. Hereinafter, the image acquisition section 101, first ratio determination section 102, and image composition section 105 according to the second embodiment will be simply referred to as the image acquisition section 101, first ratio determination section 102, and image composition section 105. . In the image processing apparatus 100, the correction term determining unit 103 and the second ratio determining unit 104 according to the second embodiment are different in processing from the corresponding units according to the first embodiment. Hereinafter, the correction term determination section 103 and the second ratio determination section 104 according to the second embodiment will be simply referred to as the correction term determination section 103 and the second ratio determination section 104.

The correction term determining unit 103 obtains the similarity between the long exposure image 107 and the adjusted short exposure image, and obtains a temporary correction term based on the obtained similarity. In addition, the correction term determining unit 103 determines a correction term for correcting the first synthesis ratio (α) for each pixel by changing the acquired provisional correction term based on the spatial frequency component of the short exposure image 108. do. Similar to the correction term determining unit 103 according to the first embodiment, the correction term determining unit 103 includes a temporary correction term obtaining unit 601, a clip unit 602, a first filter unit 603, a second filter unit 604, and a changing unit 605. . The processing of the correction term determining unit 103 will be described with reference to FIG. 6. The clip section 602, the first filter section 603, and the second filter section 604 (hereinafter simply referred to as "the clip section 602, the first filter section 603, and the second filter section 604") according to the second embodiment are as follows. Since each part is the same as the corresponding part according to the first embodiment, a description thereof will be omitted. The correction term determining unit 103 allows the provisional correction term acquisition unit 601 and the change unit 605 (hereinafter simply referred to as "the provisional correction term acquisition unit 601 and the change unit 605") according to the second embodiment to obtain the provisional correction term according to the first embodiment. The acquisition unit 601 and the change unit 605 have different processes.

The temporary correction term acquisition unit 601 first divides the image areas of the long exposure image 107 and the short exposure image 108 into a plurality of mutually corresponding blocks. Next, the degree of similarity between the blocks of the long exposure image 107 and the blocks of the adjusted short exposure image, which correspond to each other in the plurality of divided blocks, is calculated for each block. Specifically, for example, the temporary correction term acquisition unit 601 first calculates a correlation value between the pixel value L within the block of the long exposure pixel 107 and the pixel value within the block in the adjusted short exposure image. Next, a temporary correction term is calculated based on the calculated correlation value. Methods for calculating correlation values within a block include SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), and NCC (Normalized Cross-Correlation). SAD is a calculation method that calculates the sum of absolute differences of each pixel value in a block, SSD is the sum of squares of differences of each pixel value in a block, and NCC is a calculation method that calculates the normalized sum of correlations in a block as a correlation value. It is. Hereinafter, as an example, it will be explained that the temporary correction term acquisition unit 601 uses NCC to calculate the correlation value of mutually corresponding blocks as the degree of similarity. Specifically, the temporary correction term acquisition unit 601 calculates the degree of similarity (P) using the following equation 9, for example.

Here, i is, for example, a positive integer indicating the pixel position in the horizontal direction, and j is, for example, a positive integer indicating the pixel position in the vertical direction. L(i,j) is the value of a pixel (pixel value L) in the long exposure image 107 whose horizontal position is i and whose vertical position is j. Further, S (i, j) is the value of the pixel (pixel value S) at the horizontal position i and the vertical position j in the short exposure image 108, and R is the output It is a ratio and is a value calculated using Formula 3.

On the premise that both the pixel value L and the pixel value S are smaller than the saturated output signal value, when the following two conditions are simultaneously satisfied, the degree of similarity (P) approaches 1. One of the two conditions is a case where no light from a flicker light source is incident on the area of the image sensor corresponding to the block (hereinafter referred to as "no flicker occurs within the block"). The other case is a case where no motion blur occurs within the block in the captured image, particularly the long exposure image 107, due to the imaging of the moving object.

On the other hand, when light from a flickering light source is incident on the area of the image sensor corresponding to the block (hereinafter referred to as "flicker occurs within the block"), the block similarity (P) is This value is smaller than when flicker or motion blur does not occur. Similarly, even when motion blur occurs within a block in the captured image due to imaging of a moving object, the block similarity (P) is compared to when no flicker or motion blur occurs within the block. It becomes a small value.

Here, unless extremely large motion blur occurs, when flicker occurs within a block, the block similarity (P) is smaller than when motion blur occurs. Utilizing this characteristic, the correction term determining unit 103 evaluates the degree of similarity (P) to determine whether flicker, motion blur, or both flicker and motion blur occur within the block. It can also be determined whether or not this has occurred.

The temporary correction term acquisition unit 601 acquires a temporary correction term for each block based on the calculated similarity (P) corresponding to each block. Specifically, the provisional correction term acquisition unit 601 uses a provisional correction term (hereinafter referred to as “first provisional correction term)) for each block. The provisional correction term acquisition unit 601 also provides a provisional correction term (hereinafter referred to as "second provisional correction term") that is corrected to increase the ratio of combining pixel values S based on the similarity (P) calculated for each block. ) for each block. Specifically, for example, the temporary correction term acquisition unit 601 determines whether the similarity (P) of the blocks is a determination threshold value (hereinafter referred to as "flicker detection") for detecting whether or not flicker occurs (hereinafter referred to as "flicker detection"). (referred to as "flicker threshold value"). When the similarity (P) of blocks is less than or equal to the flicker threshold, the first provisional correction term and the second provisional correction term are set so that the first provisional correction term acts more favorably than the second provisional correction term. Decide and get. Further, the provisional correction term acquisition unit 601 detects whether the similarity (P) of the blocks is greater than the flicker threshold and the similarity (P) indicates whether motion blur has occurred (hereinafter referred to as "motion blur detection"). ) (hereinafter referred to as "motion blur threshold"). When the block similarity (P) is greater than the flicker threshold and less than or equal to the motion blur threshold, the first provisional correction term is set so that the second provisional correction term acts more dominantly than the first provisional correction term. and determine the second provisional correction term.

With reference to FIG. 9, a method for determining a temporary correction term for a block based on the similarity (P) of the block will be described. FIG. 9 is an explanatory diagram for explaining an example of a method for determining the first provisional correction term and the second provisional correction term in the provisional correction term acquisition unit 601 according to the second embodiment. First, with reference to FIG. 9(a), a method for determining the first provisional correction term for a block based on the similarity (P) of the block will be described. FIG. 9A is an explanatory diagram for explaining a method for determining the first provisional correction term. In FIG. 9A, the horizontal axis indicates the magnitude of block similarity (P), and the vertical axis indicates the magnitude of the first provisional correction term (β). In addition, a polygonal line 900 indicated by a solid line indicates the block similarity (P) and the The correspondence relationship with the first provisional correction term (β) is shown.

The temporary correction term acquisition unit 601 determines the first temporary correction term (β) to be 1 when the block similarity (P) is less than the lower flicker threshold Th2_L. On the other hand, if the block similarity (P) is greater than or equal to the upper flicker threshold Th2_H, the first provisional correction term (β) is determined to be zero. Furthermore, when the block similarity (P) is greater than or equal to Th2_L and less than Th2_H, the first provisional correction term (β) is decreased linearly, for example, as the block similarity (P) increases. Decide on a value between 1 and 0. That is, when the similarity (P) of the block is less than or equal to TH2_H, the temporary correction term acquisition unit 601 determines that flicker has occurred within the block. The relationship between the block similarity (P) and the first provisional correction term (β) is expressed, for example, by the following equation 10.

FIG. 9(b) is an explanatory diagram for explaining the method for determining the second provisional correction term. In FIG. 9(b), the horizontal axis indicates the magnitude of block similarity (P), and the vertical axis indicates the magnitude of the second temporary correction term (γ). In addition, a polygonal line 901 indicated by a solid line indicates the block similarity (P) and the The correspondence relationship with the second provisional correction term (γ) is shown.

The temporary correction term acquisition unit 601 determines whether the block similarity (P) is less than Th3_L, which is the first motion blur threshold from the bottom, or greater than Th4_H, which is the first motion blur threshold from the top. , the second provisional correction term (γ) is determined to be zero. On the other hand, if the block similarity (P) is greater than or equal to Th3_H, which is the second from the bottom, and less than Th4_L, which is the second motion blur threshold from the top, the second temporary correction term (γ) is determined to be 1. Furthermore, when the block similarity (P) is greater than or equal to Th3_L and less than Th3_H, the second provisional correction term (γ) is increased linearly, for example, as the block similarity (P) increases. Decide on a value between 0 and 1. Furthermore, when the block similarity (P) is greater than or equal to Th4_L and less than Th4_H, the second provisional correction term (γ) is decreased linearly, for example, as the block similarity (P) increases. Decide on a value between 1 and 0. That is, when the similarity (P) of the block is greater than or equal to TH3_L and less than or equal to Th4_H, the temporary correction term acquisition unit 601 determines that motion blur has occurred within the block. The relationship between the block similarity (P) and the second correction term (γ) is expressed, for example, by the following Equation 11.

However, it is assumed that the relationship between the threshold values shown in FIG. 9 satisfies the relationship Th2_L<Th2_H<Th3_L<Th3_H<Th4_L<Th4_H. That is, in the block to be processed, either flicker or motion blur is detected, and a state where flicker and motion blur occur simultaneously is not detected. By determining the first provisional correction term (β) and the second provisional correction term (γ) using Equations 4 and 5, if flicker occurs within the block, the second provisional correction term ( The first provisional correction term (β) comes to have a predominant effect compared to γ). Furthermore, when motion blur occurs within a block, the second provisional correction term (γ) acts more dominantly than the first provisional correction term (β). Furthermore, if neither flicker nor motion blur occurs in the block, the first provisional correction term (β) and the second provisional correction term (γ) are both 0, so the first synthesis ratio (α) No corrections will be made.

The changing unit 605 changes the second temporary correction term (γ) acquired by the temporary correction term acquisition unit 601 using the filtered images that have passed through the first filter unit 603 and the second filter unit 604. do. With this change, the changing unit 605 obtains a correction term for correcting the first synthesis ratio (α) for each pixel. The second provisional correction term (γ') after modification by the modification unit 605 is expressed, for example, by the following Equation 12.

Here, N is a pixel value (gradation value) of the image after filtering processing. Val_mod is the value of the second provisional correction term after the change, and is an arbitrary set value between 0 and 1. Th_fil is a spatial frequency threshold, and is an arbitrary pixel value (gradation value) specified in advance within the gradation range of the image (N) after the filtering process. A pixel for which the first provisional correction term (β) is 0 may be not a flicker pixel but a motion blur pixel, or a pixel that is neither a flicker pixel nor a motion blur pixel (hereinafter referred to as a "still pixel"). It is a pixel with high quality. Therefore, when the first temporary correction term (β) is 0, the changing unit 605 changes the second temporary correction term (γ). Note that since the first temporary correction term (β) is not changed by the changing unit 605, the correction term acquired by the changing unit 605, that is, the correction term determined by the correction term determining unit 103 is the first temporary correction term. The value (β-γ') is obtained by subtracting the changed second provisional correction term (γ') from (β).

As a result, if the image after the filtering process is an image that includes contour information of the clip short exposure image, the second temporary correction term (γ) of the pixel corresponding to the contour part of the clip short exposure image where motion blur occurs is changed so that the ratio of combining pixel values S is increased. Here, an image in which the image after the filtering process includes outline information of the clipped short exposure image means, for example, as explained in the first embodiment, the high frequency component or the medium frequency component, etc. in the clipped short exposure image is removed by the filtering process. This is the extracted image.

The second ratio determination unit 104 determines whether the pixel A second combination ratio to be used when actually combining the value L and the pixel value S is determined for each pixel. For example, the second ratio determination unit 104 determines a value obtained by subtracting the correction term (β-γ') from the first combination ratio (α) as the second combination ratio. The second combination ratio (d) is expressed using the following equation 13, for example.

Here, min (x0, x1) is a function that returns the minimum value of x0 and x1, and max (x0, x1) is a function that returns the maximum value of x0 and x1. The second combination ratio (d) represents the final combination ratio used when combining the pixel value L and the pixel value S, and is determined for each pixel. When pixel values L and pixel values S are combined to generate an HDR image, the ratio at which pixel values S are combined is the second combination ratio (d), and the ratio at which pixel values L are combined is from 1 to the second combination. The value obtained by subtracting the ratio (d) is (1-d). If the values of the first provisional correction term (β) and the changed second provisional correction term (γ') are both 0, the value of the second combination ratio (d) is equal to the first combination ratio (α). The first synthesis ratio (α) becomes the second synthesis ratio (d) as it is. The larger the value of the first provisional correction term (β), the smaller the value of the second synthesis ratio (d). The ratio of On the other hand, the larger the second provisional correction term (γ') after the change, the larger the value of the second synthesis ratio (d) becomes. The rate at which the values S are combined increases.

The formula for calculating the second composite ratio (d) shown in Formula 13 is just an example. The formula for calculating the second combination ratio (d) may be as follows. The second synthesis ratio (d) is set such that as the magnitude of the value obtained by subtracting the changed second temporary correction term (γ') from the first temporary correction term (β) increases, the proportion at which pixel values L are synthesized increases. Determine. In addition, the second synthesis ratio ( d). Furthermore, the second ratio determination unit 104 determines that the value of the first composite ratio (α) is The second combination ratio (d) is determined so as to have the value of the second combination ratio (d).

The operation of the image processing apparatus 100 will be described with reference to FIG. 10. FIG. 10 is a flowchart illustrating an example of a processing flow of the image processing apparatus 100 according to the second embodiment. Note that in the explanation of FIG. 10, the symbol "S" means a step. First, in S1001, the image acquisition unit 101 acquires data of the long exposure image 107 and the short exposure image 108. Next, in S1002, the first ratio determination unit 102 determines a first combination ratio (α) for each pixel of the long exposure image 107. Next, in S1003, the correction term determining unit 103 calculates the degree of similarity (P) for each block. Specifically, the correction term determining unit 103 first divides the image areas of the long exposure image 107 and the short exposure image 108 into a plurality of mutually corresponding blocks. Next, the degree of similarity (P) between the blocks of the long exposure image 107 and the blocks of the adjusted short exposure image that correspond to each other in the plurality of divided blocks is calculated for each block.

Next, in S1004, the correction term determining unit 103 obtains a first provisional correction term (β) and a second provisional correction term (γ) for each block based on the similarity (P) calculated for each block. do. Next, in S1005, the correction term determining unit 103 changes the second provisional correction term (γ) for each pixel and determines the correction term (β−γ') for each pixel. Next, in S1006, the second ratio determination unit 104 determines a second synthesis ratio (d) for each pixel. Next, in S1007, the image synthesis unit 105 synthesizes the pixel value L and the pixel value S for each pixel to generate an HDR image. After S1007, the image processing apparatus 100 ends the process of the flowchart shown in FIG.

According to the image processing apparatus 100 configured as described above, in an HDR image acquired using a plurality of captured images captured under different exposure conditions, moving object It is possible to suppress motion blur due to Furthermore, according to the image processing apparatus 100 configured as described above, the image processing apparatus 100 according to the first embodiment has problems with regard to pixels that are identified as flicker pixels even though they are actually motion blur pixels. , can be identified as a motion blur pixel. As a result, the image processing apparatus 100 configured as described above can obtain a more accurate HDR image than the image processing apparatus 100 according to the first embodiment.

[Other embodiments]
The present disclosure provides a system or device with a program that implements one or more functions of the embodiments described above via a network or a storage medium, and one or more processors in a computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit such as an ASIC that realizes one or more functions.

Note that within the scope of the present disclosure, the embodiments of the present disclosure can be freely combined, any constituent elements of the embodiments may be modified, or any constituent elements of the embodiments may be omitted.

[Configuration of this disclosure]
<Configuration 1>
image acquisition means for acquiring data of a first captured image and data of a second captured image captured with a shorter exposure time than the first captured image;
Ratio determining means for determining a synthesis ratio to be used when synthesizing pixel values of the first captured image and pixel values of the second captured image, based on a spatial frequency component of the second captured image;
An image processing device comprising:

<Configuration 2>
The ratio determining means determines a temporary combination ratio based on pixel values of the first captured image, and includes a correction term for correcting the temporary combination ratio based on a spatial frequency component of the second captured image. The image processing device according to configuration 1, further comprising determining the combining ratio based on the temporary combining ratio and the correction term.

<Configuration 3>
The ratio determining means changes, based on the spatial frequency component, a provisional correction term for correcting the provisional combination ratio so that the ratio of combining pixel values of the first captured image increases. The image processing device according to configuration 2, further comprising determining a correction term.

<Configuration 4>
The ratio determining means determines a ratio for synthesizing pixel values of images of the spatial frequency component in a predetermined range in the second captured image when changing the temporary correction term based on the spatial frequency component. The image processing apparatus according to configuration 3, characterized in that the temporary correction term is changed so that .

<Configuration 5>
Configuration 3, characterized in that, when changing the temporary correction term based on the spatial frequency component, the ratio determining means changes the temporary correction term based on a high frequency component in the spatial frequency component. 4. The image processing device according to 4.

<Configuration 6>
Configuration 3 characterized in that, when changing the temporary correction term based on the spatial frequency component, the ratio determining means changes the temporary correction term based on a low frequency component in the spatial frequency component. 6. The image processing device according to any one of 5 to 5.

<Configuration 7>
When changing the temporary correction term based on the spatial frequency component, the ratio determining means changes the temporary correction term based on a medium frequency component excluding a high frequency component and a low frequency component in the spatial frequency component. The image processing device according to any one of configurations 3 to 6, characterized in that: .

<Configuration 8>
The ratio determining means determines whether or not to change the temporary correction term based on the spatial frequency component, and when it is determined to change the temporary correction term, changes the temporary correction term based on the spatial frequency component. The image processing device according to any one of configurations 3 to 7, characterized in that the correction term is changed.

<Configuration 9>
The ratio determining means determines whether or not to change the temporary correction term based on the pixel value of the second captured image, and when it is determined that the temporary correction term is to be changed, the ratio determination unit The image processing device according to any one of configurations 3 to 8, wherein the temporary correction term is changed based on.

<Configuration 10>
The ratio determining means is configured to determine the pixel value of the first captured image and the first captured image and the second captured image in a plurality of blocks obtained by dividing the image area of the first captured image and the second captured image. The image processing device according to any one of configurations 3 to 9, wherein the temporary correction term is acquired for each block based on the degree of similarity of the blocks corresponding to each other in the captured image of No. 2. .

<Configuration 11>
The ratio determining means adjusts a pixel value of the first captured image and a pixel value of the second captured image based on an output ratio between the first captured image and the second captured image. The image processing device according to configuration 10, wherein the degree of similarity of the mutually corresponding blocks in the first captured image and the second captured image is calculated using the following.

<Configuration 12>
The ratio determining means, when determining the provisional correction term for each block based on the similarity of the blocks, corrects the provisional correction term such that the combination ratio of the first captured image is increased. obtaining a certain first provisional correction term and a second provisional correction term that is the provisional correction term that is corrected so that the combination ratio of the second captured image is increased;
When the degree of similarity of the blocks is less than or equal to a flicker threshold that is a determination threshold for flicker detection, the first provisional correction term acts more favorably than the second provisional correction term; The second provisional correction term is obtained, and if the similarity of the block is greater than the flicker threshold and the similarity is less than or equal to a motion blur threshold that is a determination threshold for motion blur detection, the second obtaining the first provisional correction term and the second provisional correction term in which the provisional correction term acts more dominantly than the first provisional correction term;
The image processing device according to configuration 10 or 11, wherein the temporary correction term is changed by changing the second temporary correction term based on the spatial frequency component.

<Configuration 13>
image synthesis means for synthesizing pixel values of the first captured image and pixel values of the second captured image based on the synthesis ratio;
The image processing device according to any one of configurations 1 to 12, characterized in that the image processing device has:

<Configuration 14>
an image acquisition step of acquiring data of a first captured image and data of a second captured image captured with a shorter exposure time than the first captured image;
a ratio determining step of determining a synthesis ratio to be used when combining pixel values of the first captured image and pixel values of the second captured image, based on a spatial frequency component of the second captured image;
An image processing method characterized by having the following.

<Configuration 15>
A program for causing a computer to operate as the image processing device according to any one of Configurations 1 to 13.

100 Image processing device 101 Image acquisition unit 102 First ratio determination unit 103 Correction term determination unit 104 Second ratio determination unit

Claims (15)

image acquisition means for acquiring data of a first captured image and data of a second captured image captured with a shorter exposure time than the first captured image;
Ratio determining means for determining a synthesis ratio to be used when synthesizing pixel values of the first captured image and pixel values of the second captured image, based on a spatial frequency component of the second captured image;
An image processing device comprising: The ratio determining means determines a temporary combination ratio based on pixel values of the first captured image, and includes a correction term for correcting the temporary combination ratio based on a spatial frequency component of the second captured image. The image processing apparatus according to claim 1, further comprising determining the combining ratio based on the temporary combining ratio and the correction term. The ratio determining means changes, based on the spatial frequency component, a provisional correction term for correcting the provisional combination ratio so that the ratio of combining pixel values of the first captured image increases. The image processing device according to claim 2, further comprising determining a correction term. The ratio determining means determines a ratio for synthesizing pixel values of images of the spatial frequency component in a predetermined range in the second captured image when changing the temporary correction term based on the spatial frequency component. The image processing apparatus according to claim 3, characterized in that the temporary correction term is changed so that . 3. The ratio determining means, when changing the temporary correction term based on the spatial frequency component, changes the temporary correction term based on a high frequency component in the spatial frequency component. The image processing device described in . Claim characterized in that, when changing the temporary correction term based on the spatial frequency component, the ratio determining means changes the temporary correction term based on a low frequency component in the spatial frequency component. 3. The image processing device according to 3. When changing the temporary correction term based on the spatial frequency component, the ratio determining means changes the temporary correction term based on a medium frequency component excluding a high frequency component and a low frequency component in the spatial frequency component. The image processing apparatus according to claim 3, characterized in that the image processing apparatus changes: . The ratio determining means determines whether or not to change the temporary correction term based on the spatial frequency component, and when it is determined to change the temporary correction term, changes the temporary correction term based on the spatial frequency component. The image processing device according to claim 3, characterized in that the correction term is changed. The ratio determining means determines whether or not to change the temporary correction term based on the pixel value of the second captured image, and when it is determined that the temporary correction term is to be changed, the ratio determination unit The image processing apparatus according to claim 3, wherein the temporary correction term is changed based on. The ratio determining means is configured to determine the pixel value of the first captured image and the first captured image and the second captured image in a plurality of blocks obtained by dividing the image area of the first captured image and the second captured image. The image processing apparatus according to claim 3, wherein the temporary correction term is acquired for each block based on the degree of similarity between the blocks corresponding to each other in the second captured image. The ratio determining means adjusts a pixel value of the first captured image and a pixel value of the second captured image based on an output ratio between the first captured image and the second captured image. The image processing apparatus according to claim 10, wherein the degree of similarity of the mutually corresponding blocks in the first captured image and the second captured image is calculated using the following. The ratio determining means, when determining the provisional correction term for each block based on the similarity of the blocks, corrects the provisional correction term such that the combination ratio of the first captured image is increased. obtaining a certain first provisional correction term and a second provisional correction term that is the provisional correction term that is corrected so that the combination ratio of the second captured image is increased;
When the degree of similarity of the blocks is less than or equal to a flicker threshold that is a determination threshold for flicker detection, the first provisional correction term acts more favorably than the second provisional correction term; The second provisional correction term is obtained, and if the similarity of the block is greater than the flicker threshold and the similarity is less than or equal to a motion blur threshold that is a determination threshold for motion blur detection, the second obtaining the first provisional correction term and the second provisional correction term in which the provisional correction term acts more dominantly than the first provisional correction term;
The image processing apparatus according to claim 10, wherein the temporary correction term is changed by changing the second temporary correction term based on the spatial frequency component. image synthesis means for synthesizing pixel values of the first captured image and pixel values of the second captured image based on the synthesis ratio;
The image processing device according to claim 1, characterized in that it has: an image acquisition step of acquiring data of a first captured image and data of a second captured image captured with a shorter exposure time than the first captured image;
a ratio determining step of determining a synthesis ratio to be used when combining pixel values of the first captured image and pixel values of the second captured image, based on a spatial frequency component of the second captured image;
An image processing method characterized by having the following. A program for causing a computer to operate as the image processing apparatus according to any one of claims 1 to 13.

Images