WO2023107166A1 - Acquisition d'image à plage dynamique élevée au moyen d'un filtrage temporel - Google Patents

Acquisition d'image à plage dynamique élevée au moyen d'un filtrage temporel Download PDF

Info

Publication number
WO2023107166A1
WO2023107166A1 PCT/US2022/042582 US2022042582W WO2023107166A1 WO 2023107166 A1 WO2023107166 A1 WO 2023107166A1 US 2022042582 W US2022042582 W US 2022042582W WO 2023107166 A1 WO2023107166 A1 WO 2023107166A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
image frame
frame
hdr
weight map
Prior art date
Application number
PCT/US2022/042582
Other languages
English (en)
Inventor
Christian Markus MAEKELAE
Michael BLEYER
Original Assignee
Microsoft Technology Licensing, Llc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US17/590,641 external-priority patent/US11831996B2/en
Application filed by Microsoft Technology Licensing, Llc. filed Critical Microsoft Technology Licensing, Llc.
Priority to CN202280078344.7A priority Critical patent/CN118318454A/zh
Publication of WO2023107166A1 publication Critical patent/WO2023107166A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/741—Circuitry for compensating brightness variation in the scene by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G06T5/92—Dynamic range modification of images or parts thereof based on global image properties
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/50—Control of the SSIS exposure
    • H04N25/57—Control of the dynamic range
    • H04N25/58—Control of the dynamic range involving two or more exposures
    • H04N25/587—Control of the dynamic range involving two or more exposures acquired sequentially, e.g. using the combination of odd and even image fields
    • H04N25/589—Control of the dynamic range involving two or more exposures acquired sequentially, e.g. using the combination of odd and even image fields with different integration times, e.g. short and long exposures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10141—Special mode during image acquisition
    • G06T2207/10144—Varying exposure
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20208—High dynamic range [HDR] image processing

Definitions

  • High dynamic range (HDR) imaging was developed in an effort to improve images of captured environments that include both brightly illuminated areas/objects and relatively darker areas/objects.
  • HDR imaging involves capturing at least two separate images. One image is generated using a prolonged exposure time to detect enough photons to capture the dark, shadowy areas/objects within a captured environment. The other image uses a very short exposure time to detect photons for capturing the brightly lit areas/objects while avoiding over saturation.
  • these two images are then blended/stitched together via signal/image processing to generate a composite image that shows objects in the bright areas (as opposed to simply a washed-out white blur) as well as objects in the dark areas (as opposed to simply a dark blur), effectively increasing the dynamic range of the combined images.
  • the long-exposure image and the short-exposure image used to form an HDR image are typically captured at different times (i.e., one after the other).
  • the objects captured in the long-exposure image can become spatially misaligned with the same objects captured in the short-exposure image. This misalignment can result in blurring artifacts that reduce image quality and/or intelligibility.
  • the lengthy exposure time used to capture the long-exposure image can further contribute to the motion blur present in the final image.
  • Disclosed embodiments are directed at least to systems, methods, and devices for facilitating high dynamic range image acquisition using temporal filtering.
  • Some embodiments include a system for generating high dynamic range (HDR) imagery.
  • the system includes one or more processors and one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to perform various acts.
  • the system is configurable to obtain a history frame.
  • the history frame is associated with a first timepoint and with a first set of image capture parameters.
  • the system is also configurable to obtain an image frame associated with a particular set of image capture parameters.
  • the image frame is captured at a second timepoint that is subsequent to the first timepoint, and the particular set of image capture parameters comprises the first set of image capture parameters or a second set of image capture parameters.
  • the system is also configurable to generate a normalized image frame by applying a normalization operation to the image frame.
  • the normalization operation may be determined based upon the particular set of image capture parameters.
  • the system is further configurable to generate a weight map based upon at least the normalized image frame and generate an HDR image frame based upon the normalized image frame, the history frame, and the weight map.
  • Some embodiments include a system for generating a weight map.
  • the system includes one or more processors and one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to perform various acts.
  • the system is configurable to generate a global weight map based on motion data associated with the system and generate a local weight map based upon a difference image.
  • the difference image may be based upon a motion compensated history frame associated with a first timepoint and (i) an image frame associated with a particular set of image capture parameters and captured at a second timepoint that is subsequent to the first timepoint or (ii) a normalized image frame generated by applying a normalization operation to the image frame.
  • the system is also configurable to generate a local high dynamic range (HDR) weight map based upon the image frame, the normalized image frame, a denoised image frame, or a denoised normalized image frame.
  • the system is further configurable to generate a final weight map by combining the global weight map, the local weight map, and the local HDR weight map.
  • HDR high dynamic range
  • Figure 1 illustrates example components of an example system that may include or be used to implement one or more disclosed embodiments
  • FIGS. 2A through 2C illustrate example image processing flow diagrams depicting image processing associated with HDR image acquisition using temporal filtering
  • Figure 3 illustrates an example flow diagram depicting acts associated with generating high dynamic range imagery
  • Figure 4 illustrates an example flow diagram depicting acts associated with generating a weight map.
  • Disclosed embodiments are generally directed to systems, methods, and devices that facilitate high dynamic range (HDR) image acquisition using temporal filtering.
  • HDR high dynamic range
  • HDR image acquisition utilizing temporal filtering may comprise obtaining a history frame associated with a first timepoint.
  • the history frame may comprise a previously generated HDR image frame that was generated in accordance with the steps that follow (or an intermediate result used to generate the previously generated HDR image frame).
  • an image frame associated with a second, subsequent timepoint may be obtained (e.g., captured utilizing an image sensor, which may be the same image sensor used to capture a previous image frame used to generate the history frame).
  • the image frame is associated with a particular set of image capture parameters (e.g., a particular exposure time and/or gain parameters), which may be the same as or different than image capture parameters used to capture a previous image used to generate the history frame.
  • a normalization operation may then be applied to the image frame to generate a normalized image frame.
  • the normalization operation may be selected based on the particular image capture parameters for capturing the image frame.
  • the normalization operation may cause pixel values of the normalized image frame to be within the same range as pixel values of the history frame.
  • a weight map may be generated using (at least) the normalized image frame.
  • a weight map may include a local HDR weight component, where per-pixel weights may be determined based on the intensity values present in the input image (e.g., the normalized image frame, or, in some instances, a different image frame, as will be described in more detail hereinafter).
  • the per-pixel weights may be determined based on predefined rules, such as threshold values and/or threshold ranges of values, and the predefined rules applied for generating the per-pixel weights may be selected based on the image capture parameters used to capture the image frame. For instance, when a short exposure time is used to capture the image frame, the predefined rules may cause greater weight to be assigned to higher-intensity pixels, which may include texture information for objects that would appear saturated if a long exposure time was used to capture the image frame. Correspondingly, if a long exposure is used to capture the image frame, the predefined rules may cause greater weight to be assigned to lower-intensity pixels, which may include texture information for objects that would appear underexposed if a short exposure time was used to capture the image frame.
  • predefined rules such as threshold values and/or threshold ranges of values
  • An HDR image frame may be generated based upon the normalized image frame, the history frame, and the weight map.
  • the HDR image frame may then be defined as a history frame that can be used in combination with a subsequently captured image frame to generate a subsequent HDR image frame.
  • the techniques discussed above and elsewhere herein for generating an HDR image frame may be performed repeatedly to advantageously cause an image capture rate to match an HDR image acquisition/generation rate, thereby providing a desirable approach for generating/capturing high- framerate HDR video.
  • the history frame used to generate the HDR image is a motion compensated history frame.
  • the weight map further includes an intensity similarity component (e.g., based on a comparison between intensities of temporally consecutive frames, where dissimilarity between pixel intensities leads to lower weight) and/or a global weight component (e.g., based on camera motion across timepoints, where history frame weighting is reduced if high camera motion is detected).
  • intensity similarity component e.g., based on a comparison between intensities of temporally consecutive frames, where dissimilarity between pixel intensities leads to lower weight
  • a global weight component e.g., based on camera motion across timepoints, where history frame weighting is reduced if high camera motion is detected.
  • the disclosed techniques for HDR image acquisition naturally implement temporal averaging (e.g., by utilizing a history frame in combination with a current image frame to generate a current HDR image frame), which may improve the signal-to-noise ratio for HDR imagery.
  • the disclosed techniques may be implemented utilizing any suitable type of image sensor (e.g., CMOS, CCD, SPAD, etc ). Having just described some of the various high-level features and benefits of the disclosed embodiments, attention will now be directed to Figures 1 through 4. These Figures illustrate various conceptual representations, architectures, methods, and supporting illustrations related to the disclosed embodiments.
  • Figure 1 illustrates various example components of a system 100 that may be used to implement one or more disclosed embodiments.
  • a system 100 may include processor(s) 102, storage 104, sensor(s) 110, image sensor(s) 112, input/output system(s) 114 (I/O system(s) 114), and communication system(s) 116.
  • Figure 1 illustrates a system 100 as including particular components, one will appreciate, in view of the present disclosure, that a system 100 may comprise any number of additional or alternative components.
  • the processor(s) 102 may comprise one or more sets of electronic circuitry that include any number of logic units, registers, and/or control units to facilitate the execution of computer- readable instructions (e.g., instructions that form a computer program). Such computer-readable instructions may be stored within storage 104.
  • the storage 104 may comprise physical system memory and may be volatile, non-volatile, or some combination thereof.
  • storage 104 may comprise local storage, remote storage (e.g., accessible via communication system(s) 116 or otherwise), or some combination thereof. Additional details related to processors (e.g., processor(s) 102) and computer storage media (e.g., storage 104) will be provided hereinafter.
  • processor(s) 102 may comprise or be configurable to execute any combination of software and/or hardware components that are operable to facilitate processing using machine learning models or other artificial intelligence-based structures/architectures.
  • processor(s) 102 may comprise and/or utilize hardware components or computerexecutable instructions operable to carry out function blocks and/or processing layers configured in the form of, by way of non-limiting example, single-layer neural networks, feed forward neural networks, radial basis function networks, deep feed-forward networks, recurrent neural networks, long-short term memory (LSTM) networks, gated recurrent units, autoencoder neural networks, variational autoencoders, denoising autoencoders, sparse autoencoders, Markov chains, Hopfield neural networks, Boltzmann machine networks, restricted Boltzmann machine networks, deep belief networks, deep convolutional networks (or convolutional neural networks), deconvolutional neural networks, deep convolutional inverse graphics networks, generative adversarial networks, liquid state machines, extreme learning machines, echo
  • the processor(s) 102 may be configured to execute instructions 106 stored within storage 104 to perform certain actions. The actions may rely at least in part on data 108 stored on storage 104 in a volatile or non-volatile manner.
  • the actions may rely at least in part on communication system(s) 116 for receiving data from remote system(s) 118, which may include, for example, separate systems or computing devices, sensors, and/or others.
  • the communications system(s) 118 may comprise any combination of software or hardware components that are operable to facilitate communication between on-system components/devices and/or with off-system components/devices.
  • the communications system(s) 118 may comprise ports, buses, or other physical connection apparatuses for communicating with other devices/components.
  • the communications system(s) 118 may comprise systems/components operable to communicate wirelessly with external systems and/or devices through any suitable communication channel(s), such as, by way of non-limiting example, Bluetooth, ultra-wideband, WLAN, infrared communication, and/or others.
  • Figure 1 illustrates that a system 100 may comprise or be in communication with sensor(s) 110.
  • Sensor(s) 110 may comprise any device for capturing or measuring data representative of perceivable or detectable phenomenon.
  • the sensor(s) 110 may comprise one or more image sensor(s) 112 (e.g., CMOS, CCD, SPAD, and/or others), microphones, thermometers, barometers, magnetometers, accelerometers, gyroscopes, inertial measurement units (IMUs) and/or others.
  • image sensor(s) 112 e.g., CMOS, CCD, SPAD, and/or others
  • microphones e.g., CMOS, CCD, SPAD, and/or others
  • thermometers e.g., CMOS, CCD, SPAD, and/or others
  • barometers e.g., magnetometers
  • accelerometers e.g., accelerometers, gyroscopes, inertial measurement units
  • FIG. 1 illustrates that a system 100 may comprise or be in communication with I/O system(s) 114.
  • I/O system(s) 114 may include any type of input or output device such as, by way of non-limiting example, a touch screen, a mouse, a keyboard, a controller, and/or others, without limitation.
  • the I/O system(s) 114 may include a display system that may comprise any number of display panels, optics, laser scanning display assemblies, and/or other components.
  • Figure 1 conceptually represents that the components of the system 100 may comprise or utilize various types of devices, such as mobile electronic device 100A (e.g., a smartphone), personal computing device 100B (e.g., a laptop), a mixed-reality head-mounted display 100C (HMD 100C), an aerial vehicle 100D (e.g., a drone), and/or other devices.
  • mobile electronic device 100A e.g., a smartphone
  • personal computing device 100B e.g., a laptop
  • HMD 100C mixed-reality head-mounted display 100C
  • an aerial vehicle 100D e.g., a drone
  • FIG. 1 conceptually represents that the components of the system 100 may comprise or utilize various types of devices, such as mobile electronic device 100A (e.g., a smartphone), personal computing device 100B (e.g., a laptop), a mixed-reality head-mounted display 100C (HMD 100C), an aerial vehicle 100D (e.g., a drone), and/or other devices.
  • FIGS 2A through 2C illustrate example image processing flow diagrams depicting image processing associated with HDR image acquisition using temporal filtering. Aspects of the processing depicted in Figures 2A, 2B, and 2C may be performed utilizing one or more components of a system (e.g., system 100 of Figure 1).
  • Figure 2A illustrates a captured image frame 202.
  • the image frame 202 is captured utilizing an image sensor of a system (e.g., sensor(s) 112 of a system 100, which may be implemented as an HMD 100C).
  • the captured frame 202 may be one of a plurality of consecutively captured image frames captured by the system, each image frame being associated with a respective timepoint.
  • the processing described herein with reference to Figures 2A through 2C may be applied, without loss of generality, to any number of consecutively captured image frames to generate HDR imagery (e.g., each consecutively captured image frame, providing output HDR imagery at a same rate as the image capture rate).
  • the image frame 202 may be captured with a particular set of image capture parameters, which may include a particular exposure time, a particular analog gain, a particular digital gain, etc.
  • a particular set of image capture parameters may include a particular exposure time, a particular analog gain, a particular digital gain, etc.
  • different sets of image capture parameters are used by a system to capture a plurality of image frames, which may cause consecutively captured image frames to be associated with different sets of image capture parameters (e.g., different exposure times, different gains, etc.).
  • Figure 2A also illustrates a history frame 212.
  • the history frame 212 may comprise a previously generated HDR image frame that was generated in accordance with the techniques discussed herein (see, for example, Figure 2C and attendant description).
  • the history frame 212 may be generated based upon a previously captured image frame (e.g., an image frame that was captured at a timepoint that precedes the capture timepoint of the image frame 202 shown in Figure 2A).
  • the previously captured image frame is captured using a different set of image capture parameters (e.g., a different exposure time and/or gain) than the particular set of image capture parameters used to capture the image frame 202 of Figure 2A.
  • the image frame 202 may have been captured using a long exposure time, whereas the previously captured image frame may have been captured using a short exposure time, or vice-versa.
  • the history frame 212 (generated based on the previously captured image frame) and the image frame 202 may be associated with different sets of image capture parameters, and the history frame may be associated with a capture timepoint that precedes the capture timepoint of the image frame 202.
  • Figure 2A illustrates that various processing acts may be performed on the image frame 202 and the history frame 212 to generate an HDR image frame 216.
  • Figure 2A allows the processing acts in a particular order, no ordering is required unless expressly stated or unless performance of one act relies upon completion of another act.
  • FIG. 2A illustrates that a system may generate a normalized image frame 204 based on the captured image frame 202.
  • a normalization operation may be applied to the image frame 202 to generate the normalized image frame 204, and the normalization operation may include scaling pixel values of the image frame 202.
  • the normalization operation may be performed to cause the image pixels of the normalized image frame 204 to be within the same range as image pixels of the history frame 212 (or motion compensated history frame 214, as will be discussed in more detail hereinafter).
  • the normalization operation applied to generate the normalized image frame 204 is selected based on the set of image capture parameters associated with the image frame 202. For instance, where the history frame 212 is associated with a long exposure time and the image frame 202 is associated with a short exposure time, a normalization operation may be selected to cause the image pixel values of the normalized image frame 204 to be within the same range as the image pixel values of the history frame 212.
  • the amount of scaling accomplished by the normalization operation may depend upon a ratio between the long exposure time and the short exposure time. For example, where the long exposure time is 32 times longer than the short exposure time, the image pixel values of the image frame 202 may be scaled by a factor of 32 to generate the normalized image frame 204.
  • a different normalization operation may be performed (e.g., scaling by a factor of one, or even refraining from performing normalization, as discussed further with reference to figure 2B).
  • short exposure image frames and long exposure image frames are both normalized to an intermediate range of values (e.g., where the long exposure is 32 times longer than the short exposure time, a scale factor of 16 may be used for short exposures, and a scale factor of 1/2 may be used for long exposures) or another range of values.
  • the image frame 202 captured by an image sensor is a gamma-corrected image.
  • a system may remove gamma correction from the image frame 202 prior to performing the normalization operation to generate the normalized image frame 204.
  • gamma correction is re-applied to the normalized image (however, the performance of such a step may depend upon whether the history frame comprises a gamma-corrected history frame; a history frame and captured image frame may be used to generate an HDR image frame in radiance space without gamma correction).
  • a system generates a denoised normalized image frame 206 based upon the normalized image frame 204.
  • a denoised normalized image frame 206 may be generated, in some implementations, by downscaling the normalized image frame 204 to generate a downscaled normalized image frame and then subsequently upscaling the downscaled normalized image.
  • downscaling the normalized image frame 204 reduces the high-frequency noise present in the imagery.
  • downscaling may include using average values of neighborhoods of pixels in the input image as pixel values in the downscaled image, which may operate to smooth out noise. Upscaling may then be performed to preserve the original image size.
  • Various rescaling algorithms are within the scope of the present disclosure, such as nearest-neighbor interpolation, bilinear and/or bicubic techniques, and/or others.
  • Figure 2A illustrates that a system may utilize the denoised normalized image frame 206 to generate a local HDR weight map 208 (as will be discussed with reference to Figure 2B, a normalized image frame 204, or even an image frame 202 may be utilized to generate the local HDR weight map).
  • the local HDR weight map may comprise per-pixel weight values, which may indicate how pixels based on the image frame 202 and pixels based on the history frame 212 should be combined to represent scene information with high dynamic range (e.g., for an HDR image frame 216).
  • the local HDR weight map 208 is generated by applying one or more intensity thresholds to pixels of the denoised normalized image frame 206 and selecting per-pixel weights for the local HDR weight map 208 based upon whether the corresponding input pixels of the denoised normalized image frame 206 satisfy the one or more intensity thresholds.
  • the intensity threshold(s) applied to the pixels of the denoised normalized image frame 206 are selected based upon the set of image capture parameters associated with the capture of the image frame 202. For example, one set of thresholds may be applied for image frames captured with a short exposure time, whereas a different set of thresholds may be applied for image frames captured with a long exposure time.
  • a threshold value that is at or near the maximum value may be utilized to generate weights of the local HDR weight map 208 for input images associated with a long exposure time.
  • the threshold value is satisfied by an input pixel, the output pixel in the local HDR weight map 208 may be set to zero. Additional thresholds or ranges of thresholds may be used to provide intermediate weight values.
  • a threshold range may be defined as pixel values from 220 to 245 such that input pixels within the threshold range receive an interpolated weight value (e.g., a weight of 0.3 for pixel values of 220 and a weight of 0 for pixel values of 245, with interpolated weight values used for pixel values between 220 and 245).
  • an interpolated weight value e.g., a weight of 0.3 for pixel values of 220 and a weight of 0 for pixel values of 245, with interpolated weight values used for pixel values between 220 and 245
  • Such operations may cause the local HDR weight map 208 to give less or zero weight to pixels of the input long exposure image that provide saturated or nearly saturated representations of objects in the captured scene.
  • a threshold value of 255 may be utilized for input images associated with a short exposure time. When the threshold value is satisfied by an input pixel, the output pixel in the local HDR weight map 208 may be set to a maximum weight value (e.g., 1). Additional thresholds or ranges of thresholds may be used to provide intermediate weight values.
  • a threshold range may be defined as pixel values from 128 to 255 such that input pixels within the threshold range receive an interpolated weight value (e.g., a weight of 0 for pixel values of 128 and a weight of 1 for pixel values of 255, with interpolated weight values used for pixels between 128 and 255).
  • an interpolated weight value e.g., a weight of 0 for pixel values of 128 and a weight of 1 for pixel values of 255, with interpolated weight values used for pixels between 128 and 255).
  • Such operations may cause the local HDR weight map 208 to give high or full weight to pixels of the input short exposure image that capture objects that would appear saturated or nearly saturated if a long exposure time were used instead to capture the input image.
  • the local HDR weight map 208 may optionally be combined with other weight maps to form a final weight map 210, as shown in Figure 2A. Additional details related to other components that may used to form the final weight map 210 will be described in more detail hereinafter.
  • the HDR image frame 216 may be generated based on the HDR weight map 208 (or the final weight map 210), the motion compensated history frame 214, and the normalized image frame 204.
  • the motion compensated history frame 214 may be obtained, for example, using motion data associated with the system (e.g., obtained utilizing an IMU) at different timepoints.
  • an IMU may obtain first pose data for the system at a first timepoint associated with the history frame 212 (or associated with a previously captured image frame used to generate the history frame) and may obtain second pose data for the system at a second timepoint.
  • the system may use the first and second pose data to reproject or otherwise transform the history frame 212 to generate the motion compensated history frame 214, which may depict the objects represented in the history frame 212 as though they were captured from the second pose (e.g., the same pose from which the image frame 202 was captured).
  • the motion compensated history frame 214 and the normalized image frame 204 may be substantially spatially aligned. This alignment may lead to reduced motion artifacts when combining the motion compensated history frame 214 and the normalized image frame 204 to generate the HDR image frame 216.
  • the local HDR weight map 208 enables the motion compensated history frame 214 to be used in combination with the image frame 202 (or an image based thereon, such as the normalized image frame 204) to represent scene information with high dynamic range.
  • HDR image frames 216 may be generated using only a single newly captured image frame 202, thereby enabling HDR image acquisition at a rate that can match the image capture rate.
  • a final weight map 210 may be used to generate the HDR image frame 216, and the final weight map 210 may comprise additional weight components (e.g., aside from the local HDR weight map 208 discussed above).
  • Figure 2A shows that the final weight map 210 may be further based upon an intensity similarity weight map 2224and/or a global weight map 226.
  • the intensity similarity may be generated based upon a difference image 220.
  • the difference image is computed as a difference between a denoised motion compensated history frame 218 and the denoised normalized image frame 206.
  • the denoised motion compensated history frame 218 may be generated using rescaling techniques discussed hereinabove with reference to the generation of the denoised normalized image frame 206 (instead using the motion compensated history frame 214 as the input image).
  • the difference image 220 may capture per-pixel differences between the motion compensated history frame 214 and the normalized image frame 204. Such difference may arise, for example, when a high amount of system motion occurs between the capturing of the history frame 212 (or the previously captured image frame upon which the history frame 212 is based) and the capturing of the image frame 202, and/or when there are changes in the captured scene (e.g., moving objects in the scene, changes in state of objects in the scene, etc.).
  • pixels in the difference image 220 that indicate a large difference value may be assigned a weight value for the intensity similarity weight map 222 that causes additional weight to be given to the normalized image frame 204 when generating the HDR image frame 216 (e.g., giving preference to the more temporally recent representation of the scene).
  • the global weight map 226 may be generated based upon motion data 224.
  • the motion data 224 may correspond to the motion data discussed above for generating the motion compensated history frame 214.
  • pose data for the system e.g., obtained via an IMU
  • the global weight map 226 may be defined as a global weight value that causes additional weight to be given to the normalized image frame 204 when generating the HDR image frame 216 (e.g., giving preference to the more temporally recent representation of the scene).
  • the final weight map 210 may be generated as a combination of the intensity similarity weight map 222, the local HDR weight map 208, and the global weight map 226.
  • the final weight map 210 may be computed as a per-pixel product of the global weight map 226, the intensity similarity weight map 222, and the local HDR weight map 208.
  • the HDR image frame 216 may be generated as a per-pixel sum of two terms.
  • the first term may comprise a per-pixel product of the motion compensated history frame 214 and the final weight map 210.
  • the second term may comprise a per-pixel product of the normalized image frame 204 and the complement of the final weight map 210 (e.g., where each pixel is computed as 1 minus the corresponding pixel value of the final weight map 210).
  • the processing shown with reference to Figure 2A may result in an HDR image frame 216 that includes pixel values that are outside of an acceptable range (e.g., for 8-bit data, within a range of 0 to 255).
  • a tone mapping operation e.g., applying a logarithmic mapping function
  • an acceptable range e.g., 0 to 255 for 8-bit data.
  • Figure 2B illustrates various dashed lines, which depict alternative processing acts that may be performed to generate an HDR image frame 216.
  • normalization may optionally be skipped (or a normalization operation including a scale factor of 1 may be applied) for image frames captured with a long exposure time (e.g., in implementations where short exposure image frames are normalized to correspond to long exposure image frames).
  • Long exposure image frames may be regarded in some instances as already normalized.
  • denoising need not be performed on image frames captured with a long exposure time (e.g., because high-frequency noise may be less prevalent in long exposure image frames than for short exposure image frames).
  • Figure 2B illustrates dashed lines extending from the image frame 202 to the denoised normalized image frame 206, the local HDR weight map 208, the difference image 220, and the HDR image frame 216, indicating that the image frame 202 may alternatively be utilized as an input for generating any of the foregoing.
  • Figure 2B illustrates a dashed line extending from the motion compensated history frame 214 to the difference image 220, indicating that, in some implementations, the motion compensated history frame 214 (e.g., rather than a denoised motion compensated history frame 218) may be used as an input to generate the difference image 220.
  • Figure 2B illustrates a dashed line extending from the normalized image frame 204 to the difference image, indicating that, in some implementations, the normalized image frame 204 (e.g., rather than the denoised normalized image frame 206) may be used as an input to generate the difference image 220.
  • Figure 2B also illustrates a dashed line extending from the normalized image frame 204 to the local HDR weight map 208, indicating that, in some implementations, the normalized image frame (e.g., rather than the denoised normalized image frame 206) may be used as an input to generate the local HDR weight map 208.
  • Figure 2C depicts the HDR image frame 216 discussed above with reference to Figures 2A and 2B.
  • Figure 2C illustrates the HDR image frame 216 being defined as a history frame 228 to be used in combination with a subsequent image frame 230 to generate a subsequent HDR image frame 232.
  • the subsequent image frame 230 may be captured at a timepoint that is subsequent to the capture timepoint for the image frame 202 of Figures 2A and 2B.
  • the processing discussed hereinabove for using a history frame and a captured image frame to generate an HDR image frame is represented in Figure 2C by the ellipsis.
  • the rate of HDR image frame acquisition may match the rate of image capture with an image sensor.
  • an intermediate result e.g., a radiance image
  • a history frame for generating a subsequent HDR image frame.
  • any number of sets of image capture parameters may be utilized in accordance with the present disclosure.
  • Figure 3 illustrates an example flow diagram 300 depicting acts associated with generating high dynamic range imagery.
  • the discussion of the various acts represented in the flow diagrams include references to various hardware components described in more detail with reference to Figure 1.
  • Act 302 of flow diagram 300 of Figure 3 includes obtaining a history frame.
  • Act 302 is performed, in some instances, by a system 100 utilizing processor(s) 102, storage 104, sensor(s) 110, I/O system(s) 114, communication system(s) 116, and/or other components.
  • the history frame is associated with a first timepoint and with a first set of image capture parameters.
  • the history frame may comprise a previously generated HDR image associated with the first timepoint or an intermediate result used in generating the previously generated HDR image (e.g., a radiance image).
  • Act 304 of flow diagram 300 includes obtaining an image frame associated with a particular set of image capture parameters.
  • Act 304 is performed, in some instances, by a system 100 utilizing processor(s) 102, storage 104, sensor(s) 110, I/O system(s) 114, communication system(s) 116, and/or other components.
  • the image frame is captured at a second timepoint that is subsequent to the first timepoint.
  • the particular set of image capture parameters of the image frame may comprise the first set of image capture parameters (associated with the history frame) or a second set of image capture parameters (different than the first set of image capture parameters.
  • Act 306 of flow diagram 300 includes generating a normalized image frame by applying a normalization operation to the image frame.
  • Act 306 is performed, in some instances, by a system 100 utilizing processor(s) 102, storage 104, sensor(s) 110, I/O system(s) 114, communication system(s) 116, and/or other components.
  • the normalization operation may be determined based upon the particular set of image capture parameters.
  • the normalization operation includes scaling pixel values of the image frame based on a ratio of a first exposure time (e.g., of the first image capture parameters) to a second exposure time (e.g., of the second image capture parameters).
  • the normalization operation may include removing gamma correction prior to the scaling of the pixel values of the image.
  • Act 308 of flow diagram 300 includes generating a weight map based upon at least the normalized image frame. Act 308 is performed, in some instances, by a system 100 utilizing processor(s) 102, storage 104, sensor(s) 110, I/O system(s) 114, communication system(s) 116, and/or other components. In some instances, the weight map is generated based upon a local HDR weight map.
  • the local HDR weight map may be generated by applying one or more intensity thresholds to pixels of the image frame, the normalized image frame, a denoised image frame, or a denoised normalized image frame. The intensity threshold(s) may be selected based on the particular set of image capture parameters associated with the image frame.
  • Output pixels of the local HDR weight map may be selected based on whether corresponding pixels of the image frame, the normalized image frame, the denoised image frame, or the denoised normalized image frame satisfy the intensity threshold(s).
  • the intensity threshold(s) comprise one or more intensity ranges
  • output pixels of the local HDR weight map may comprise interpolated weight values when corresponding pixels of the image frame, the normalized image frame, the denoised image frame, or the denoised normalized image frame fall within the one or more intensity ranges.
  • the denoised image frame is generated by downscaling the image frame to generate a downscaled image frame and upscaling the downscaled image.
  • the denoised normalized image frame may be generated by downscaling the normalized image to generate a downscaled normalized image and upscaling the downscaled normalized image.
  • the weight map is generated further based upon a global weight map.
  • the global weight map may be generated using motion data associated with the system (e.g., assigning a lower global pixel weight when a large amount of motion is detected).
  • the weight map is generated further based upon an intensity similarity weight map.
  • the intensity similarity weight map may be generated based upon a difference image, which may be computed based upon the motion compensated history frame and (i) the image frame or (ii) the normalized image frame.
  • Act 310 of flow diagram 300 includes generating an HDR image frame based upon the normalized image frame, the history frame, and the weight map. Act 310 is performed, in some instances, by a system 100 utilizing processor(s) 102, storage 104, sensor(s) 110, I/O system(s) 114, communication system(s) 116, and/or other components. In some implementations, a motion compensated history frame is obtained, and the HDR image frame is generated using the motion compensated history frame.
  • Act 312 of flow diagram 300 includes defining the HDR image frame as a subsequent history frame for generating a subsequent HDR image frame. Act 312 is performed, in some instances, by a system 100 utilizing processor(s) 102, storage 104, sensor(s) 110, I/O system(s) 114, communication system(s) 116, and/or other components.
  • the acts of flow diagram 300 may be performed iteratively to capture image frames and generate HDR image frames.
  • Consecutive sets of one or more image frames may be captured using at least a first set of image capture parameters and at least a second set of image capture parameters.
  • the different sets of image capture parameters may include different exposure times and/or gains.
  • the rate of generating HDR image frames matches a rate of capturing image frames.
  • Figure 4 illustrates an example flow diagram 400 depicting acts associated with generating a weight map.
  • the discussion of the various acts represented in the flow diagrams include references to various hardware components described in more detail with reference to Figure 1.
  • Act 402 of flow diagram 400 of Figure 4 includes generating a global weight map based on motion data associated with a system. Act 402 is performed, in some instances, by a system 100 utilizing processor(s) 102, storage 104, sensor(s) 110, VO system(s) 114, communication system(s) 116, and/or other components.
  • Act 404 of flow diagram 400 includes generating a local weight map based upon a difference image.
  • Act 404 is performed, in some instances, by a system 100 utilizing processor(s) 102, storage 104, sensor(s) 110, I/O system(s) 114, communication system(s) 116, and/or other components.
  • the difference image is based upon a motion compensated history frame associated with a first timepoint and (i) an image frame associated with a particular set of image capture parameters and captured at a second timepoint that is subsequent to the first timepoint or (ii) a normalized image frame generated by applying a normalization operation to the image frame.
  • the normalization operation may be determined based upon the particular set of image capture parameters.
  • Act 406 of flow diagram 400 includes generating a local high dynamic range (HDR) weight map.
  • Act 406 is performed, in some instances, by a system 100 utilizing processor(s) 102, storage 104, sensor(s) 110, I/O system(s) 114, communication system(s) 116, and/or other components.
  • the HDR weight map may be generated based upon the image frame, the normalized image frame, a denoised image frame, or a denoised normalized image frame.
  • the local HDR weight map may be generated by applying one or more intensity thresholds to pixels of the image frame, the normalized image frame, the denoised image frame, or the denoised normalized image frame.
  • the one or more intensity thresholds may be selected based on the particular set of image capture parameters associated with the image frame.
  • Output pixels of the local HDR weight map may be selected based on whether corresponding pixels of the image frame, the normalized image frame, the denoised image frame, or the denoised normalized image frame satisfy the one or more intensity thresholds.
  • the one or more intensity thresholds include one or more intensity ranges.
  • Output pixels of the local HDR weight map may comprise interpolated weight values when corresponding pixels of the image frame, the normalized image frame, the denoised image frame, or the denoised normalized image frame fall within the one or more intensity ranges.
  • Act 408 of flow diagram 400 includes generating a final weight map by combining the global weight map, the local weight map, and the local HDR weight map. Act 408 is performed, in some instances, by a system 100 utilizing processor(s) 102, storage 104, sensor(s) 110, I/O system(s) 114, communication system(s) 116, and/or other components.
  • the principles disclosed herein may be implemented in various formats.
  • the various techniques discussed herein may be performed as a method that includes various acts for achieving particular results or benefits.
  • the techniques discussed herein are represented in computer-executable instructions that may be stored on one or more hardware storage devices.
  • the computer-executable instructions may be executable by one or more processors to carry out (or to configure a system to carry out) the disclosed techniques.
  • a system may be configured to send the computer-executable instructions to a remote device to configure the remote device for carrying out the disclosed techniques.
  • Disclosed embodiments may comprise or utilize a special purpose or general-purpose computer including computer hardware, as discussed in greater detail below.
  • Disclosed embodiments also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures.
  • Such computer-readable media can be any available media that can be accessed by a general-purpose or special-purpose computer system.
  • Computer-readable media that store computer-executable instructions in the form of data are one or more “physical computer storage media” or “hardware storage device(s).”
  • Computer-readable media that merely carry computer-executable instructions without storing the computer-executable instructions are “transmission media.”
  • the current embodiments can comprise at least two distinctly different kinds of computer-readable media: computer storage media and transmission media.
  • Computer storage media are computer-readable hardware storage devices, such as RAM, ROM, EEPROM, CD-ROM, solid state drives (“SSD”) that are based on RAM, Flash memory, phase-change memory (“PCM”), or other types of memory, or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code means in hardware in the form of computer-executable instructions, data, or data structures and that can be accessed by a general-purpose or specialpurpose computer.
  • RAM random access memory
  • ROM read-only memory
  • EEPROM electrically erasable programmable read-only memory
  • CD-ROM Compact Disk Read Only Memory
  • SSD solid state drives
  • PCM phase-change memory
  • a “network” is defined as one or more data links that enable the transport of electronic data between computer systems and/or modules and/or other electronic devices.
  • a network or another communications connection can include a network and/or data links which can be used to carry program code in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Combinations of the above are also included within the scope of computer-readable media.
  • program code means in the form of computer-executable instructions or data structures can be transferred automatically from transmission computer-readable media to physical computer-readable storage media (or vice versa).
  • program code means in the form of computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., a “NIC”), and then eventually transferred to computer system RAM and/or to less volatile computer-readable physical storage media at a computer system.
  • NIC network interface module
  • computer-readable physical storage media can be included in computer system components that also (or even primarily) utilize transmission media.
  • Computer-executable instructions comprise, for example, instructions and data which cause a general-purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions.
  • the computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, or even source code.
  • a cloud model can be composed of various characteristics (e.g., on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, etc.), service models (e.g., Software as a Service (“SaaS”), Platform as a Service (“PaaS”), Infrastructure as a Service (“laaS”), and deployment models (e.g., private cloud, community cloud, public cloud, hybrid cloud, etc.).
  • service models e.g., Software as a Service (“SaaS”), Platform as a Service (“PaaS”), Infrastructure as a Service (“laaS”)
  • deployment models e.g., private cloud, community cloud, public cloud, hybrid cloud, etc.
  • the invention may be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, pagers, routers, switches, wearable devices, and the like.
  • the invention may also be practiced in distributed system environments where multiple computer systems (e.g., local and remote systems), which are linked through a network (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links), perform tasks.
  • program modules may be located in local and/or remote memory storage devices.
  • the functionality described herein can be performed, at least in part, by one or more hardware logic components.
  • illustrative types of hardware logic components include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), central processing units (CPUs), graphics processing units (GPUs), and/or others.
  • executable module can refer to hardware processing units or to software objects, routines, or methods that may be executed on one or more computer systems.
  • the different components, modules, engines, and services described herein may be implemented as objects or processors that execute on one or more computer systems (e.g., as separate threads).

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Image Processing (AREA)

Abstract

L'invention concerne un système de génération d'imagerie à plage dynamique élevée (HDR) qui obtient une trame d'historique associée à un premier instant et à un premier ensemble de paramètres de capture d'image. Le système obtient une trame d'image associée à un ensemble particulier de paramètres de capture d'image et capturée à un second instant qui est ultérieur au premier instant, et l'ensemble particulier de paramètres de capture d'image comprend le premier ensemble de paramètres de capture d'image ou un second ensemble de paramètres de capture d'image. Le système génère une trame d'image normalisée par application d'une opération de normalisation à la trame d'image. L'opération de normalisation peut être déterminée sur la base de l'ensemble particulier de paramètres de capture d'image. Le système génère une carte de poids sur la base d'au moins la trame d'image normalisée et génère une trame d'image HDR sur la base de la trame d'image normalisée, de la trame d'historique, et de la carte de poids.
PCT/US2022/042582 2021-12-10 2022-09-05 Acquisition d'image à plage dynamique élevée au moyen d'un filtrage temporel WO2023107166A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202280078344.7A CN118318454A (zh) 2021-12-10 2022-09-05 使用时间滤波的高动态范围图像采集

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US202163288416P 2021-12-10 2021-12-10
US63/288,416 2021-12-10
US17/590,641 2022-02-01
US17/590,641 US11831996B2 (en) 2021-12-10 2022-02-01 High dynamic range image acquisition using temporal filtering

Publications (1)

Publication Number Publication Date
WO2023107166A1 true WO2023107166A1 (fr) 2023-06-15

Family

ID=83508571

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2022/042582 WO2023107166A1 (fr) 2021-12-10 2022-09-05 Acquisition d'image à plage dynamique élevée au moyen d'un filtrage temporel

Country Status (1)

Country Link
WO (1) WO2023107166A1 (fr)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110254976A1 (en) * 2009-04-23 2011-10-20 Haim Garten Multiple exposure high dynamic range image capture
EP3076364A1 (fr) * 2015-03-30 2016-10-05 Imagination Technologies Limited Filtrage d'image sur la base de gradients d'image
US20200267300A1 (en) * 2019-02-15 2020-08-20 Samsung Electronics Co., Ltd. System and method for compositing high dynamic range images
US11189017B1 (en) * 2018-09-11 2021-11-30 Apple Inc. Generalized fusion techniques based on minimizing variance and asymmetric distance measures

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110254976A1 (en) * 2009-04-23 2011-10-20 Haim Garten Multiple exposure high dynamic range image capture
EP3076364A1 (fr) * 2015-03-30 2016-10-05 Imagination Technologies Limited Filtrage d'image sur la base de gradients d'image
US11189017B1 (en) * 2018-09-11 2021-11-30 Apple Inc. Generalized fusion techniques based on minimizing variance and asymmetric distance measures
US20200267300A1 (en) * 2019-02-15 2020-08-20 Samsung Electronics Co., Ltd. System and method for compositing high dynamic range images

Similar Documents

Publication Publication Date Title
US11107205B2 (en) Techniques for convolutional neural network-based multi-exposure fusion of multiple image frames and for deblurring multiple image frames
US11244432B2 (en) Image filtering based on image gradients
US20220014684A1 (en) Image display method and device
US10708525B2 (en) Systems and methods for processing low light images
CN112602088B (zh) 提高弱光图像的质量的方法、系统和计算机可读介质
JP2014086956A (ja) 画像処理装置及び画像処理方法
US20240073523A1 (en) Systems and methods for generating depth information from low-resolution images
US20240112605A1 (en) Systems and methods for adding persistence to single photon avalanche diode imagery
US11683593B2 (en) High dynamic range image capture using single photon avalanche diodes
US11831996B2 (en) High dynamic range image acquisition using temporal filtering
KR20190064882A (ko) 딥러닝 기반 차량용 영상 처리 방법
US20240135496A1 (en) System and method for burst image restoration and enhancement
JP7025237B2 (ja) 画像処理装置およびその制御方法ならびにプログラム
WO2023107166A1 (fr) Acquisition d'image à plage dynamique élevée au moyen d'un filtrage temporel
US11765475B2 (en) Systems and methods for obtaining dark current images
US11758297B2 (en) Systems, methods, and media for high dynamic range imaging using single-photon and conventional image sensor data
US11563908B2 (en) Image acquisition techniques with reduced noise using single photon avalanche diodes
KR20240031246A (ko) 다수의 관심 영역들을 갖는 장면들에서 선택적으로 증가하는 피사계 심도
US11988835B2 (en) Systems and methods for power efficient image acquisition using single photon avalanche diodes (SPADs)
WO2022226963A1 (fr) Procédé et appareil de traitement d'images, dispositif électronique et support de stockage
CN114331893A (zh) 一种获取图像噪声的方法、介质和电子设备
CN118318454A (zh) 使用时间滤波的高动态范围图像采集
US20220353489A1 (en) Systems and methods for efficient generation of single photon avalanche diode imagery with persistence
Matsuoka et al. Weight optimization for multiple image integration
KR20220013290A (ko) 오토포커스를 보상하는 방법 및 오토포커스를 보상하는 전자 장치