WO2021025375A1 - Appareil et procédé d'alignement d'image régularisé efficace pour fusion multi-trame - Google Patents

Appareil et procédé d'alignement d'image régularisé efficace pour fusion multi-trame Download PDF

Info

Publication number
WO2021025375A1
WO2021025375A1 PCT/KR2020/010083 KR2020010083W WO2021025375A1 WO 2021025375 A1 WO2021025375 A1 WO 2021025375A1 KR 2020010083 W KR2020010083 W KR 2020010083W WO 2021025375 A1 WO2021025375 A1 WO 2021025375A1
Authority
WO
WIPO (PCT)
Prior art keywords
reference image
motion vector
tiles
image
tile
Prior art date
Application number
PCT/KR2020/010083
Other languages
English (en)
Inventor
Ruiwen ZHEN
John W. Glotzbach
Hamid R. Sheikh
Original Assignee
Samsung Electronics Co., Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US16/727,751 external-priority patent/US11151731B2/en
Application filed by Samsung Electronics Co., Ltd. filed Critical Samsung Electronics Co., Ltd.
Priority to EP20849055.7A priority Critical patent/EP3956863A4/fr
Priority to CN202080054064.3A priority patent/CN114503541A/zh
Publication of WO2021025375A1 publication Critical patent/WO2021025375A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/223—Analysis of motion using block-matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/32—Determination of transform parameters for the alignment of images, i.e. image registration using correlation-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/64—Circuits for processing colour signals
    • H04N9/646—Circuits for processing colour signals for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/64—Circuits for processing colour signals
    • H04N9/73—Colour balance circuits, e.g. white balance circuits or colour temperature control
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10141—Special mode during image acquisition
    • G06T2207/10144—Varying exposure
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20201—Motion blur correction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20208—High dynamic range [HDR] image processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • This disclosure relates generally to image capturing systems. More specifically, this disclosure relates to an apparatus and method for regularized image alignment for multi-frame fusion.
  • aligning each non-reference frame with a selected reference frame is a crucial step. If this step has low quality, it directly affects the following image blending step and may lead to insufficient blending level or even ghost artifacts.
  • Global image registration algorithms using a global transformation matrix is a common and efficient way to achieve alignment. But using a global transformation matrix only reduces misalignment due to camera motion and sometimes even fails to find a reliable solution in the absence of matched features. This situation frequently happens in high dynamic range (HDR) applications as input frames are under-exposed or over-exposed.
  • HDR high dynamic range
  • An alternative is to use methods, such as optical flow, to find a dense correspondence between frames. Though these methods produce high-quality alignment, they require a significant computational cost posing a great challenge for mobile platforms.
  • a method includes receiving a reference image and a non-reference image; dividing the reference image into a plurality of tiles; determining, using an electronic device, a motion vector map using coarse-to-fine based motion vector estimation; and generating an output frame using the motion vector map with the reference image and the non-reference image.
  • FIGURE 1 illustrates an example network configuration including an electronic device in accordance with this disclosure
  • FIGURES 2A and 2B illustrate an example process for efficient regularized image alignment using a multi-frame fusion algorithm in accordance with this disclosure
  • FIGURE 3 illustrates an example coarse-to-fine tile-based motion vector estimation in accordance with this disclosure
  • FIGURES 4A, 4B, 4C, and 4D illustrate an example outlier removal in accordance with this disclosure
  • FIGURES 5A and 5B illustrate an example structure-preserving refinement in accordance with this disclosure
  • FIGURE 6 illustrates an example improvement to ghost artifacts for HDR applications in accordance with this disclosure
  • FIGURE 7 illustrates an example blending issue reduction for MBR applications in accordance with this disclosure.
  • FIGURE 8 illustrates an example method for efficient regularized image alignment for multi-frame fusion in accordance with this disclosure.
  • This disclosure provides an apparatus and method for efficient regularized image alignment for multi-frame fusion.
  • a method includes: receiving a reference image and a non-reference image; dividing the reference image into a plurality of tiles; determining, using an electronic device, a motion vector map using coarse-to-fine based motion vector estimation; and generating an output frame using the motion vector map with the reference image and the non-reference image.
  • an electronic device in a second embodiment, includes at least one sensor and at least one processing device.
  • the at least one processing device is configured to receive a reference image and a non-reference image; divide the reference image into a plurality of tiles; determine a motion vector map using coarse-to-fine based motion vector estimation; and generate an output frame using the motion vector map with the reference image and the non-reference image.
  • a non-transitory machine-readable medium contains instructions that when executed cause at least one processor of an electronic device to: receive a reference image and a non-reference image; divide the reference image into a plurality of tiles; determine a motion vector map using coarse-to-fine based motion vector estimation; and generate an output frame using the motion vector map with the reference image and the non-reference image.
  • various functions described below can be implemented or supported by one or more computer programs, each of which is formed from computer readable program code and embodied in a computer readable medium.
  • application and “program” refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, related data, or a portion thereof adapted for implementation in a suitable computer readable program code.
  • computer readable program code includes any type of computer code, including source code, object code, and executable code.
  • computer readable medium includes any type of medium capable of being accessed by a computer, such as read only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory.
  • ROM read only memory
  • RAM random access memory
  • CD compact disc
  • DVD digital video disc
  • a non-transitory computer readable medium excludes wired, wireless, optical, or other communication links that transport transitory electrical or other signals.
  • a non-transitory computer readable medium includes media where data can be permanently stored and media where data can be stored and later overwritten, such as a rewritable optical disc or an erasable memory device.
  • phrases such as “have,” “may have,” “include,” or “may include” a feature indicate the existence of the feature and do not exclude the existence of other features.
  • the phrases “A or B,” “at least one of A and/or B,” or “one or more of A and/or B” may include all possible combinations of A and B.
  • “A or B,” “at least one of A and B,” and “at least one of A or B” may indicate all of (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B.
  • first and second may modify various components regardless of importance and do not limit the components. These terms are only used to distinguish one component from another.
  • a first user device and a second user device may indicate different user devices from each other, regardless of the order or importance of the devices.
  • a first component may be denoted a second component and vice versa without departing from the scope of this disclosure.
  • the phrase “configured (or set) to” may be interchangeably used with the phrases “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of” depending on the circumstances.
  • the phrase “configured (or set) to” does not essentially mean “specifically designed in hardware to.” Rather, the phrase “configured to” may mean that a device can perform an operation together with another device or parts.
  • the phrase “processor configured (or set) to perform A, B, and C” may mean a generic-purpose processor (such as a CPU or application processor) that may perform the operations by executing one or more software programs stored in a memory device or a dedicated processor (such as an embedded processor) for performing the operations.
  • Examples of an “electronic device” may include at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device (such as smart glasses, a head-mounted device (HMD), electronic clothes, an electronic bracelet, an electronic necklace, an electronic appcessory, an electronic tattoo, a smart mirror, or a smart watch).
  • PDA personal digital assistant
  • PMP portable multimedia player
  • MP3 player MP3 player
  • a mobile medical device such as smart glasses, a head-mounted device (HMD), electronic clothes, an electronic bracelet, an electronic necklace, an electronic appcessory, an electronic tattoo, a smart mirror, or a smart watch.
  • the same or similar reference denotations may be used to refer to the same or similar elements throughout the specification and the drawings.
  • a coarse-to-fine alignment on four-level Gaussian pyramids of the input frames is first performed to find similarities between image tiles. Then an outlier rejection step followed by a quadratic structure-preserving constraint is adopted to reduce image content distortion from the previous step.
  • aligning each non-reference frame with a selected reference frame is a crucial step. If this step has a low quality, it directly affects the following image blending step and may lead to insufficient blending level or even ghost artifacts.
  • Global image registration algorithms that achieve alignment using a global transformation matrix are a common and efficient way, but these algorithms only reduce misalignment due to camera motion, and sometimes even fail to find a reliable solution in the absence of matched features (This situation frequently happens in HDR applications as input frames are under-exposed or over-exposed).
  • An alternative is to find a dense correspondence between frames, such as optical flow. Though it produces high-quality alignment, its significant computational cost poses a great challenge for mobile platforms.
  • One or more embodiments of the present disclosure based on the requirement of speed/memory, provide a simple algorithm that strikes a balance between computational cost and correspondence quality.
  • a coarse-to-fine alignment on four-level Gaussian pyramids of the input frames is first performed to find correspondences between image tiles. Then an outlier rejection step followed by a quadratic structure-preserving constraint is adopted to reduce image content distortion from the previous step. Its effectiveness and efficiency has been demonstrated via large number of input frames used for HDR and MBR applications.
  • One or more embodiments of the present disclosure provide an algorithm capable of aligning multiple images/frames in the presence of camera motion or small object motion without introducing significant image distortion. It is a necessary component in the pipeline of any multi-frame blending algorithms, such as high dynamic range imaging and motion blur reduction technique, both of which fuse several images captured with different exposure/ISO settings.
  • FIGURE 1 illustrates an example network configuration 100 including an electronic device in accordance with this disclosure.
  • the embodiment of the network configuration 100 shown in FIGURE 1 is for illustration only. Other embodiments of the network configuration 100 could be used without departing from the scope of this disclosure.
  • an electronic device 101 is included in the network configuration 100.
  • the electronic device 101 can include at least one of a bus 110, a processor 120, a memory 130, an input/output (I/O) interface 150, a display 160, a communication interface 170, or a sensor 180.
  • the electronic device 101 may exclude at least one of these components or may add at least one other component.
  • the bus 110 includes a circuit for connecting the components 120-180 with one another and for transferring communications (such as control messages and/or data) between the components.
  • the processor 120 includes one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP).
  • the processor 120 is able to control on at least one of the other components of the electronic device 101 and/or perform an operation or data processing relating to communication.
  • the processor 120 is a graphics processor unit (GPU).
  • the processor 120 can receive image data captured by at least one camera during a capture event.
  • the processor 120 can process the image data (as discussed in more detail below) to generate HDR images of dynamic scenes using graph cut-based labeling.
  • the memory 130 can include a volatile and/or non-volatile memory.
  • the memory 130 can store commands or data related to at least one other component of the electronic device 101.
  • the memory 130 can store software and/or a program 140.
  • the program 140 includes, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and/or an application program (or “application”) 147.
  • At least a portion of the kernel 141, middleware 143, or API 145 may be denoted an operating system (OS).
  • OS operating system
  • the kernel 141 can control or manage system resources (such as the bus 110, processor 120, or memory 130) used to perform operations or functions implemented in other programs (such as the middleware 143, API 145, or application 147).
  • the kernel 141 provides an interface that allows the middleware 143, the API 145, or the application 147 to access the individual components of the electronic device 101 to control or manage the system resources.
  • the application 147 includes one or more applications for image capture as discussed below. These functions can be performed by a single application or by multiple applications that each carries out one or more of these functions.
  • the middleware 143 can function as a relay to allow the API 145 or the application 147 to communicate data with the kernel 141, for instance.
  • a plurality of applications 147 can be provided.
  • the middleware 143 is able to control work requests received from the applications 147, such as by allocating the priority of using the system resources of the electronic device 101 (like the bus 110, the processor 120, or the memory 130) to at least one of the plurality of applications 147.
  • the API 145 is an interface allowing the application 147 to control functions provided from the kernel 141 or the middleware 143.
  • the API 145 includes at least one interface or function (such as a command) for filing control, window control, image processing, or text control.
  • the I/O interface 150 serves as an interface that can, for example, transfer commands or data input from a user or other external devices to other component(s) of the electronic device 101.
  • the I/O interface 150 can also output commands or data received from other component(s) of the electronic device 101 to the user or the other external device.
  • the display 160 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum-dot light emitting diode (QLED) display, a microelectromechanical systems (MEMS) display, or an electronic paper display.
  • the display 160 can also be a depth-aware display, such as a multi-focal display.
  • the display 160 is able to display, for example, various contents (such as text, images, videos, icons, or symbols) to the user.
  • the display 160 can include a touchscreen and may receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a body portion of the user.
  • the communication interface 170 is able to set up communication between the electronic device 101 and an external electronic device (such as a first electronic device 102, a second electronic device 104, or a server 106).
  • the communication interface 170 can be connected with a network 162 or 164 through wireless or wired communication to communicate with the external electronic device.
  • the communication interface 170 can be a wired or wireless transceiver or any other component for transmitting and receiving signals, such as images.
  • the electronic device 101 further includes one or more sensors 180 that can meter a physical quantity or detect an activation state of the electronic device 101 and convert metered or detected information into an electrical signal.
  • one or more sensors 180 can include one or more buttons for touch input, one or more cameras, a gesture sensor, a gyroscope or gyro sensor, an air pressure sensor, a magnetic sensor or magnetometer, an acceleration sensor or accelerometer, a grip sensor, a proximity sensor, a color sensor (such as a red green blue (RGB) sensor), a bio-physical sensor, a temperature sensor, a humidity sensor, an illumination sensor, an ultraviolet (UV) sensor, an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an infrared (IR) sensor, an ultrasound sensor, an iris sensor, or a fingerprint sensor.
  • UV ultraviolet
  • EMG electromyography
  • EEG electroencephalogram
  • ECG electrocardiogram
  • IR infrared
  • the sensor(s) 180 can also include an inertial measurement unit, which can include one or more accelerometers, gyroscopes, and other components.
  • the sensor(s) 180 can further include a control circuit for controlling at least one of the sensors included here. Any of these sensor(s) 180 can be located within the electronic device 101.
  • the first external electronic device 102 or the second external electronic device 104 can be a wearable device or an electronic device-mountable wearable device (such as an HMD).
  • the electronic device 101 can communicate with the electronic device 102 through the communication interface 170.
  • the electronic device 101 can be directly connected with the electronic device 102 to communicate with the electronic device 102 without involving with a separate network.
  • the electronic device 101 can also be an augmented reality wearable device, such as eyeglasses, that include one or more cameras.
  • the wireless communication is able to use at least one of, for example, long term evolution (LTE), long term evolution-advanced (LTE-A), 5th generation wireless system (5G), millimeter-wave or 60 GHz wireless communication, Wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunication system (UMTS), wireless broadband (WiBro), or global system for mobile communication (GSM), as a cellular communication protocol.
  • the wired connection can include, for example, at least one of a universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232), or plain old telephone service (POTS).
  • the network 162 includes at least one communication network, such as a computer network (like a local area network (LAN) or wide area network (WAN)), Internet, or a telephone network.
  • the first and second external electronic devices 102 and 104 and server 106 each can be a device of the same or a different type from the electronic device 101.
  • the server 106 includes a group of one or more servers.
  • all or some of the operations executed on the electronic device 101 can be executed on another or multiple other electronic devices (such as the electronic devices 102 and 104 or server 106).
  • the electronic device 101 when the electronic device 101 should perform some function or service automatically or at a request, the electronic device 101, instead of executing the function or service on its own or additionally, can request another device (such as electronic devices 102 and 104 or server 106) to perform at least some functions associated therewith.
  • the other electronic device (such as electronic devices 102 and 104 or server 106) is able to execute the requested functions or additional functions and transfer a result of the execution to the electronic device 101.
  • the electronic device 101 can provide a requested function or service by processing the received result as it is or additionally.
  • a cloud computing, distributed computing, or client-server computing technique may be used, for example. While FIGURE 1 shows that the electronic device 101 includes the communication interface 170 to communicate with the external electronic device 104 or server 106 via the network 162, the electronic device 101 may be independently operated without a separate communication function according to some embodiments of this disclosure.
  • the server 106 can optionally support the electronic device 101 by performing or supporting at least one of the operations (or functions) implemented on the electronic device 101.
  • the server 106 can include a processing module or processor that may support the processor 120 implemented in the electronic device 101.
  • FIGURE 1 illustrates one example of a network configuration 100 including an electronic device 101
  • the network configuration 100 could include any number of each component in any suitable arrangement.
  • computing and communication systems come in a wide variety of configurations, and FIGURE 1 does not limit the scope of this disclosure to any particular configuration.
  • FIGURE 1 illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.
  • FIGURES 2A and 2B illustrate an example process for efficient regularized image alignment using a multi-frame fusion algorithm in accordance with this disclosure.
  • the process 200 shown in FIGURE 2A is described as being performed using the electronic device 101 shown in FIGURE 1.
  • the process 200 shown in FIGURE 2A could be used with any other suitable electronic device and in any suitable system.
  • the process 200 includes steps to capture multiple image frames of a scene at different exposures and process the image frames to generate a fused output.
  • the fused imaged is blended using an additional chroma processing to reduce ghosting and blurring in images.
  • the process 200 involves the capture of multiple image frames 205.
  • two image frames are captured and processed, although more than two image frames may also be used.
  • the image frames 205 are captured at different exposures, such as when one of the image frame 205 is captured using an automatic exposure (“auto-exposure”) or other longer exposure and the other of the image frame 205 is captured using a shorter exposure (compared to the automatic or a longer exposure).
  • An auto-exposure generally refers to an exposure that is automatically, such as without intervening human action, determined by a camera or other device, typically with little or no user input.
  • a user is allowed to specify an exposure mode (such as portrait, landscape, sports, or other modes), and the automatic exposure can be generated based on the selected exposure mode without any other user input.
  • Each exposure setting is typically associated with different settings for the camera, such as different apertures, shutter speeds, and camera sensor sensitivities.
  • a shorter-exposure image frame is typically darker, lacks image details, and has more noise compared to an auto-exposure or other longer-exposure image frame.
  • the shorter-exposure image frame may include one or more regions that are under-exposed, while the auto-exposure or other longer-exposure image frame may include one or more regions that are over-exposed.
  • the short-exposure frame may only have a shorter exposure time, but has a higher ISO to match the overall image brightness of the auto-exposure or long-exposure frame. Note that while often described below as involving the use of an auto-exposure image frame and at least one shorter-exposure image frame, embodiments of the present disclosure may be used with any suitable combination of image frames captured using different exposures.
  • the processor 120 can control the camera of the electronic device 101 so that the image frames 205 are captured rapidly, such as in a burst mode.
  • a capture request that triggers the capture of the image frames 205 represents any suitable command or input indicating a need or desire to capture an image of a scene using the electronic device 101.
  • the capture request could be initiated in response to a user’s pressing of a “soft” button presented on the display 160 or the user’s pressing of a “hard” button.
  • two image frames 205 are captured in response to the capture request, although more than two images could be captured.
  • the image frames 205 may be produced in any suitable manner, such as where each image frame is simply captured by a camera or where a multiple-frame fusion technique is used to capture multiple initial image frames and combine them into one or more of the image frames 205.
  • one image frame 205 can be used as a reference image frame, and the other image frame 205 can be used as a non-reference image frame.
  • the reference image frame may represent the auto-exposure or other longer-exposure image frame, or the reference image frame may represent the shorter-exposure image frame.
  • the auto-exposure or other longer-exposure image frame may be used as the reference image frame by default, since this typically allows the image frame with greater image details to be used more when generating a composite or final image of a scene.
  • this is not desirable (such as due to the creation of image artifacts), in which case the shorter-exposure image frame may be selected as the reference image frame.
  • the raw image frames are pre-processed in some manner to provide part of the image processing.
  • the pre-processing operation 210 can perform a white balancing function to change or correct the color balance in the raw image frames.
  • the pre-processing operation 210 also can perform a function to reconstruct full color image frames from incomplete color samples contained in the raw image frames using a mask (such as a CFA mask).
  • the image frames 205 are provided to an image alignment operation 215.
  • the image alignment operation aligns the image frames 205 and produce aligned image frames.
  • the image alignment operation 215 can modify the non-reference image frame so that particular features in the non-reference image frame align with corresponding features in the reference image frame.
  • one of the aligned image frames can represent an aligned version of the reference image frame
  • the other of the aligned image frames can represent an aligned version of the non-reference image frame.
  • Alignment may be needed to compensate for misalignment caused by the electronic device 101 moving or rotating between image capture events, which causes objects in the image frames 205 to move or rotate slightly (as is common with handheld devices).
  • the image frames 205 can be aligned both geometrically and photometrically.
  • the image alignment operation 215 can use global Oriented FAST and Rotated BRIEF (ORB) features and local features from a block search to align the image frames, although other implementations of the image registration operation could also be used.
  • ORB Rotated BRIEF
  • the reference image frame here may or may not be modified during the alignment, and the non-reference image frame could represent the only image frame that is modified during the alignment.
  • the aligned image frame (the non-reference image frame) can be histogram matched as part of the pre-processing operation 210.
  • Histogram matching matches a histogram of the non-reference image frame to a histogram of the reference image frame, such as by applying a suitable transfer function to the aligned image frame.
  • histogram matching can operate to make the brightness level generally equal for both aligned image frames. Histogram matching may involve increasing the brightness of the shorter-exposure image frame to substantially match the brightness of the auto-exposure or other longer-exposure image frame, although the converse may occur. This also results in the generation of a pre-processed aligned image frame associated with the aligned image frame. More details regarding the image alignment operation 215 are described in relation to FIGURE 2B.
  • the aligned image frame is then blended in a blending operation 220.
  • the blending operation 220 blends or otherwise combines the pixels from the image frames based on the label map(s) in order to produce at least one final image of a scene.
  • the final image generally represents a blend of the image frames, where each pixel in the final image is extracted from either the reference image frame or the non-reference image frame (depending on the corresponding value in the label map). Additional image processing operations can occur once the proper pixels are extracted from the image frames and used to form an image. Ideally, the final image has little or no artifacts and improved image details, even in areas where at least one of the image frames 205 were over-exposed or under-exposed.
  • the blended frames then are put through a post-processing operation 225.
  • the post-processing operation 225 can perform any processing to the blended image to complete the fused output image 230.
  • FIGURE 2B illustrate an example local image alignment operation 215 in accordance with this disclosure. If multiple frames are being captured, the cameras move when the lens is not kept perfectly still. Between different images captured, slight movement in the camera causes the images to be misaligned. The images need to be aligned.
  • Global alignment is the determination of how much the camera moves between each of the frames.
  • a model would be assigned for the movement of the image based on the movement of the camera.
  • the images are corrected based on the model assigned based on the determined movement of the camera from the reference frame.
  • This method would only capture the camera motion and the models are imperfect. Even though only the camera is determined to move in a single frame, this is not always true for a real camera as there are always objects moving in the scene, or other secondary effects may be present such as depth-related distortion from motion. This leads to approximating camera motion. If it appears that two images are aligned, global alignment would still have features in the images that are not correctly aligned.
  • Certain alignments start with a simple global alignment and then allow the alignment to adapt to local content.
  • One method to accomplish this is through optical flow methods.
  • An optical flow method tries to confirm or find that every location in a non-reference image is also found in the reference image. If this is performed for every pixel, a full map of how everything has moved can be generated. This method is costly for processing time and expensive, which is not practical for mobile electronics.
  • the flow techniques also have complications for images if there is no real image content (such as a wall or sky) where the output would be completely noisy or just wrong.
  • Certain alignments provide for gaining the benefits of optical flow quality in the alignment without the costs and adding in more regularization to ensure that the objects align appropriately in the final image.
  • Small motions are also a consideration that can be corrected or situations where the camera moves; but the objects in the images move independently. Small motions tend to go with the camera and appear as camera motion. Large motion will look like real scene motion and is outside the determination of alignment.
  • the image alignment operation can include a histogram matching operation 235, a coarse-to-fine tile-based motion vector estimation operation 240, an outlier removal operation 245, and a structure-guided refinement operation 250.
  • the image alignment operation 215 receives input reference frame 255 and non-reference frame 260 that have been processed in the pre-processed operation 210. During subsequent operations, one image frame can be used as a reference image frame 255, and the other image frame can be used as a non-reference image frame 260.
  • the reference image frame 255 may represent the auto-exposure or other longer-exposure image frame, or the reference image frame 255 may represent the shorter-exposure image frame.
  • the auto-exposure or other longer-exposure image frame may be used as the reference image frame 255 by default, since this typically allows the image frame with greater image details to be used more when generating a composite or final image of a scene. As described below, there may be some instances in which this is not desirable (such as due to the creation of image artifacts), in which case the shorter-exposure image frame may be selected as the reference image frame 255.
  • the image alignment operation 215 includes a histogram matching operation 235.
  • the histogram match occurs first to bring multiple images or frames with different capture configurations to a same brightness levels. Histogram matching is necessary for the following search of motion vectors.
  • the purpose of a histogram match is to take two images that may not be exposed the same way and adjust accordingly. An image that is under exposed is compared to an image that is correctly exposed, which would cause one image to be darker than the other. The darker image would need to be adjusted to the correctly exposed image to properly compare the two images. A short exposure time with high gain to achieve a same effect as a longer exposure with low gain would also need histogram matching. These pictures should look the same, but will have slightly different images, which can use the histogram matching for normalize differences between images to make the images look as similar as possible.
  • the histogram match also occurs first to bring multiple image/frames with different capture configuration to the same brightness level.
  • This histogram matching can be used for searching of motion vectors, which is unnecessary for sparse features such as oriented FAST and rotated BRIEF (ORB).
  • a histogram of the non-reference frame 260 is compared to a histogram of the reference frame 255.
  • the histogram of the non-reference frame 260 is transformed to match the histogram of the reference frame 255.
  • the transformed histogram of the non-reference frame can be used later in the image alignment operation 215, as well as in the image blending operation 220 or the post processing operation 225.
  • a tile-based motion vector image breaks up the image into multiple tiles and tries to find motion map like a flow. The goal would be to find for each tile the motion vector to find the content of tile in another frame. The result is a two-dimensional (2D) map of motion vectors of the reference frame.
  • the tile-based motion vectors can generate evenly distributed features. For example, sparse features such as ORB are sparse and sometimes biased towards local areas. The distribution property reduces the probability of registration failure.
  • the coarse-to-fine tile-based motion vector estimation (MV estimation) 240 can determine motion vectors for a patch between the reference frame 255 and the non-reference frame 260.
  • the tile-based search of motion vectors can generate evenly distributed features. Sparse features such as ORB are sparse and sometimes biased towards local areas. The distribution property reduces the probability of registration failure.
  • the search of motion vectors (features) can be performed in a coarse-to-fine scheme to reduce a search radius, which in turn decreases processing time. The selection of tile size and search radius guarantees that most common cases could be covered. Sub-pixel search improves alignment accuracy. Normalized cross correlation is adopted to be robust against poor histogram matching or different noise levels.
  • the output of the MV estimation 240 is a motion vector for each of the patches into which the reference frame 255 is divided.
  • the set of motion vectors for the reference frame 255 and non-reference frame 260 are output for use later in the image alignment operation 215, image blending operation 220, or the post processing operation 225.
  • the MV estimation 240 can use a hierarchy implementation to reduce a search radius for faster results.
  • the MV estimation 240 can search the motion vector on a sub-pixel level for greater accuracy.
  • the MV estimation 240 can search for a motion vector for each tile providing sufficient matched features for a more robust motion vector estimation.
  • the MV estimation 240 can search using an L2 distance for images with the same exposure. For images with different exposures, the L2 distance can be replaced by normalized cross-correlation. More details regarding the MV estimation 240 are described with respect to FIGURE 3.
  • the outlier removal operation 245 can receive the locally aligned non-reference frame 260 and can determine if any outliers were produced during the motion vector estimation 240.
  • the outlier removal is the concept of removing items that do not match to the global motion. For example, a person moving across the frame might show a motion vector in a substantially different direction. This motion would be labeled as an outlier.
  • Outlier rejection can reject undesirable motion vectors (motion vectors on flat areas and large motion areas) in some multi-frame fusion modes.
  • HDR multi-frame higher dynamic range
  • motion vectors are not reliable on flat areas due to an absence of features.
  • other multi-frame applications such as multi-frame noise reduction
  • motion vectors on large moving objects are not constrained during the search. Both situations could lead to non-reference image distortion, which is generally not a problem as the blending operation 220 can reject inconsistent pixels between the reference image 255 and the warped non-reference image 265.
  • the image contents in the non-reference images 260 are unique and can appear in the final composite image.
  • the outlier removal 245 can reject erroneous motion vector outliers to avoid potential image distortion for a more robust image. More details regarding the outlier removal operation 245 are described with respect to FIGURE 4.
  • the structure-guided refinement operation 250 can preserve an image structure from issues such as warping.
  • a quadratic optimization problem is set up to try to maintain structure in the image. For example, if a building with edge is in the image, the edges of the building should not to be distorted.
  • the refinement operation 250 can preserve the objects, like the edges in the building, in the scene.
  • the structure-guided mesh warping can be used in multi-frame fusion modes.
  • the refinement operation 250 can impose constraints on flat areas and motion areas with a global transform, while deforming the remaining image area with searched features. Compared to common optical flow, the quadratic optimization equation can be solved in a closed-form and the processing time is feasible on mobile platforms.
  • the refinement operation 250 can pose a constraint as a quadratic optimization problem and solve the problem using linear equations for faster results.
  • the refinement operation 250 can add similarity constraints and global constraints to reduce image contents distortion for enhanced structure preservation.
  • the refinement operation can output a warped non-reference frame 265 to the image blending operation 220 or the post processing operation 225. More details regarding the structure-guided refinement operation 250 are described with respect to FIGURE 5.
  • FIGURES 2A and 2B illustrate examples of a process for efficient regularized image alignment using a multi-frame fusion algorithm
  • various changes may be made to FIGURES 2A and 2B.
  • various operations shown in FIGURE 2A and 2B could overlap, occur in parallel, occur in a different order, or occur any number of times.
  • the specific operations shown in FIGURE 2A and 2B are examples only, and other techniques could be used to perform each of the operations shown in FIGURE 2A and 2B.
  • FIGURE 3 illustrates an example coarse-to-fine tile-based motion vector estimation 240 in accordance with this disclosure.
  • FIGURE 3 illustrates how the coarse-to-fine tile based motion vector estimation 240 shown in FIGURE 2B may be used to quickly and accurately produce motion vectors for each tile divided from the reference frame 255.
  • the generation of the motion vectors is described as being performed using the electronic device 101 of FIGURE 1, although any other suitable electronic device in any suitable system may be used.
  • the reference frame 325 and non-reference frame 330 correspond respectively to reference frame 255 and non-reference frame 260 shown in FIGURE 2B.
  • the frames 325 and 330 are different resolutions of a man waving his left hand while sitting on a ledge in an office building. Behind the man is a window with a view of the city including many natural (trees, clouds, etc.) and manmade objects (other buildings, roads, etc.). The movement of the hand is being lifted from the reference frame 325 to the non-reference frame 330. The movement of the arm causes smaller movements in the overall posture of the man.
  • the motion vector maps can accurately detect the different motions for enhanced local alignment.
  • the coarse-to-fine tile-based motion vector basically takes a high-resolution image and the resolution is reduced in steps 305a-305d to a low-resolution.
  • the frames 325 and 330 are then broken into tiles in each step.
  • the tile 310 from the reference frame 325 is found in another frame, such as the non-reference frame 330.
  • the reference image frame is reduced in resolution and then split into tiles.
  • the movement of each tile in the reference image is then found in the non-reference image. This allows a search over a large range with low-resolution covering more content.
  • the motion is found in the low-resolution first step 305a and then the search is moved to a higher resolution second step 305b.
  • the tile 310 in the low-resolution step 305a is used as a search area 315 in the second step 305b. That search area 315 is divided into different tiles 320 for searching for the current tile 310 in the different step 305a-305d for matching.
  • the search is moved up again to another higher-level resolution third step 305c.
  • the tile in which the tile 310 was found to match is used as the search area 315 in the third step 305c.
  • the electronic device 101 can divide the search area 315 in the third step 305c into multiple tiles. The tiles are then searched for the tile 310. Once the tile 310 is found in a tile of the third step 305c, the search is moved up to the highest resolution fourth step 305d.
  • the electronic device 101 can search the area in the highest resolution fourth step 305d based on the tile 320 of the third step 305c. Once the tile 310 is matched to a tile 320 in the fourth step 305d, the tile 310 is considered to be located in the non-reference image and a motion vector can be developed to determine how far the location of the tile 320 in the final image is compared to the tile 310 in the reference image 325.
  • the movement vectors 340 can be labeled in a motion vector map 335.
  • the motion vector maps 335 for each resolution show movement vectors 340 based on color and intensity of the color.
  • the color indicates the direction of the movement vector and the intensity of the color indicates the amount of movement in the specified direction. The stronger or darker colors indicate significant movement between the image frames.
  • search area 315 appears in the top left corner of each respective step 305a-305d, the search area can be located at any tile of the underlying resolution step 305b-305d.
  • the example illustrated in FIGURE 3 shows that movement for the tile 310 in the top left corner is negligible in the non-reference frame.
  • the corresponding movement vector 340 in the motion vector map 335 would be very low intensity, if there is any color at all.
  • the auto-exposure or other longer-exposure image frame is used as the reference image frame 325 during the coarse-to-fine tile-based motion vector estimation 240, and any areas of the image frame 325 containing motion could be replaced using corresponding areas of the non-reference image frame 330. It is noted that there may be various situations in which one or more saturated areas in the longer-exposure image frame are partially occluded by at least one moving object in the shorter-exposure image frame.
  • the motion vector estimation 240 can perform a coarse-to-fine alignment on four Gaussian pyramids of the input frames.
  • the electronic device 101 can split the reference image frame into multiple tiles. On each pyramid, the electronic device 101 can search for a corresponding tile in a neighborhood of a non-reference image frame 330 for each tile in the reference image frame.
  • the tile size and search radius can vary with the different levels 305a-305d.
  • the electronic device 101 can evaluate multiple hypotheses when up sampling the coarser level of motion vectors to avoid boundary issue. For same-exposed images, the electronic device 101 can search the movement vector 340 by minimizing an L2 norm distance. For different-exposed images, a search is performed by maximizing normalized cross correlation.
  • the above searching method can only generate pixel-level alignment.
  • the electronic device 101 can use a quadratic function to fit near the pixel minimum and directly compute a subpixel minimum.
  • FIGURES 4A through 4D illustrate example outlier removal of a non-reference image for use in producing an HDR image of a dynamic scene in accordance with this disclosure.
  • FIGURES 4A through 4D illustrate how the image outlier removal 245 shown in FIGURE 2B may be used to generate an aligned non-reference image for use in producing an HDR image of a dynamic scene.
  • the generation of the non-reference image 260 with outliers removed here is described as being performed using the electronic device 101 of FIGURE 1, although any other suitable electronic device in any suitable system may be used.
  • the image 405 shows the non-reference image before local alignment.
  • the background of the image appears as a featureless area.
  • the hand appears blurry due to movement and the arm appears to be wavy.
  • Motion vectors may not be reliable on “flat areas,” such as a saturated area or a featureless area.
  • the “flat areas” are detected by comparing a gradient magnitude accumulation in a tile with a pre-defined threshold. “Large motion vectors” can lead to significant image distortion if they are directly used for image warping.
  • the electronic device 101 can use motion vectors from the motion vector estimation 240 to compute a global geometric transformation, which could be described by an affine matrix. Although a transformation is described by an affine matrix, any type of global geometric transformation could be applied.
  • the affine matrix “records” or preserves straight lines in a two-dimensional image. If a distance between a motion vector and the global affine matrix exceeds a threshold, it is referred to as a “large motion vector.” If this threshold is too small, the fine alignment results can be close to the global registration. When the threshold is increased, the strength of the local alignment is correspondingly increased.
  • image 410 local alignment has been performed, but no outlier removal has been applied. While the arm appears to be aligned with slight misalignment in the fingers and arm, the background is significantly warped. The straight edge of the street is disjointed due to flaws with local alignment procedures experienced by the lack of details in the alignment image.
  • the outlier removal process corrects straight edges in the image 415 when edges are detected.
  • the image 415 still has artifacts due to motion.
  • the electronic device 101 can apply outlier removal in motion areas.
  • image 420 the artifacts due to motion in the arm and fingers are corrected. Due to the size of the fingers, the motion may not be fully corrected. The accuracy of small details in the finger is a trade off with computing complexity.
  • FIGURES 5A and 5B illustrate an example structure-guided refinement operation 255 in accordance with this disclosure.
  • FIGURE 5A illustrates a quadratic mesh of a final image generated using the structure-guided refinement 250 shown in FIGURE 2B.
  • FIGURE 5B illustrates an example before refinement image and after refinement image generated using the structure-preserving refinement operation 255 shown in FIGURE 2B.
  • the electronic device 101 can preserve image structure by imposing quadratic constraints on a mesh vertex 520.
  • the quadratic constraint can be defined by the following equation.
  • the local alignment term Ep can represent the error term for feature points 515 (represented by a bilinear combination of vertexes 520) in the non-reference frames 510 after the features points are warped to align with the corresponding feature points in the reference image frame.
  • the feature points 515 in the reference image frame 505 are centers of the tiles on the finest scale.
  • the corresponding feature points are features points in the reference image frames 505 shifted by the computed motion vectors.
  • the similarity term Es represents the error in similarity of the triangle coordinates 525 (formed by three vertexes) after warping, meaning that the shape of the triangles should be similar before warping and after warping to keep Es low.
  • the global constraint term Eg can enhance “flat area” and “large motion areas” to take the global affine transform.
  • artifacts 535 still exist.
  • the bump on the knuckle area of the hand of the person is not actually meant to be in the image; but is an artifact of the local alignment.
  • artifacts 535 that exist on the edge of the ledge at the base of the window.
  • the after-refinement image has these “bump” artifacts 535 corrected.
  • FIGURES 3 through 5B illustrate various examples of image alignment operations
  • various changes may be made to FIGURES 3 through 5B.
  • FIGURES 3 through 5B are merely meant to illustrate examples of the types of results that could be obtained using the approaches described in this disclosure.
  • images of scenes can vary widely, and the results obtained using the approaches described in this patent document can also vary widely depending on the circumstances.
  • FIGURES 6 and 7 illustrate an example enhance results 600, 700 of the image alignment operation 215 in accordance with this disclosure.
  • the embodiments of results 600 and results 700 illustrated in FIGURES 6 and 7 are for illustration only FIGURES 6 and 7 do not limit the scope of this disclosure to any particular results of an image alignment operation.
  • FIGURE 6 illustrates an example ghost artifact correction 600 for ghost artifacts 615 in accordance with this disclosure.
  • the images capture a side of a person’s head and a background through a window. In the background, several buildings can be seen. Above the buildings appear to be lights reflected in the window. A ghost artifact appears around the hair of the person of the global aligned image 605. Using the local alignment methods of this application, the ghosting artifact can be significantly reduced, if not corrected altogether as seen in the local aligned image 610.
  • FIGURE 7 illustrates an example blending issue correction 700 in accordance with this disclosure.
  • the blending issues remaining in the global aligned image 705 can be seen in the trees of the global aligned blending map 715 and the globally aligned image 705.
  • the locally aligned blending map 720 has more detail in the trees that can be deciphered.
  • the locally aligned image 710 has fewer blending issues compared to the globally aligned image.
  • FIGURE 8 illustrates an example method 800 for efficient regularized image alignment for multi-frame fusion in accordance with this disclosure.
  • the method 800 shown in FIGURE 8 is described as involving the performance of the process 200 shown in FIGURE 2A using the electronic device 101 shown in FIGURE 1.
  • the method 800 shown in FIGURE 8 could be used with any other suitable electronic device and in any suitable system.
  • the electronic device 101 can receive a reference image and a non-reference image.
  • “Receiving” in this context can mean capture using an image sensor, receive from an external device, or load from a memory.
  • the reference image and the non-reference image can be captured using a same lens at different exposures, different lenses at different exposures, different lenses at different resolutions, and the like.
  • the reference image and the non-reference image can capture the same subject matter at different resolutions, exposures, offsets, and the like.
  • the electronic device 101 can divide the reference image into a plurality of tiles.
  • the reference frame is divided into a plurality of tiles in order to search for each tile in one or more non-reference frames.
  • the tiles can be the same size, which means that the image would be evenly divided horizontally and vertically.
  • the electronic device 101 can determine a motion vector map using a local alignment for each tile using a Gaussian pyramid of the non-reference image.
  • the electronic device 101 can split a lower resolution frame of the Gaussian pyramid corresponding to the non-reference image into a plurality of search tiles. Then for each of the plurality of tiles in the reference image, the electronic device 101 can locate a matching tile from the plurality of search tiles that corresponds to a tile of the plurality of tiles and determine a low-resolution movement vector based on a change in a location of the tile from the reference image to the matching tile in the lower resolution frame of the non-reference image. The result is a generation of a low-resolution motion map based on the low-resolution movement vectors for the plurality of tiles in the reference image. This sub-process can be performed for the lowest resolution level of the Gaussian pyramid.
  • the electronic device 101 can split a search area in the non-reference image into a plurality of second search tiles, wherein the search area corresponds to the matching tile.
  • the search area in the non-reference image is split into a plurality of second search tiles, where the search area corresponds to the matching tile.
  • a second matching tile from the plurality of second search tiles is located that corresponds to the tile of the plurality of tiles.
  • the electronic device 101 determines a motion vector based on a change in location of the tile in the reference image to the second matching tile in the non-reference image.
  • the motion vector map is generated based on the movement vectors for the plurality of tiles in the reference image.
  • the electronic device can determine outlier motion vectors in the motion vector map.
  • a global affine matrix is computed using motion vectors in the motion vector map.
  • a difference is determined by comparing each of the motion vectors to the global affine matrix.
  • a large motion vector is determined when the difference is greater than a threshold and the determined large motion vector is removed from the motion vector map.
  • the electronic device 101 can also determine flat areas to remove from the motion vector map.
  • the non-reference image can be divided into a plurality of non-reference tiles.
  • a gradient magnitude accumulation in a non-reference tile can be compared with a pre-defined threshold and a flat area can be determined based on the gradient magnitude accumulation exceeding the pre-defined threshold.
  • the electronic device 101 can remove motion vectors from the motion vector map that correspond to the flat area.
  • the electronic device can impose quadratic constraints on a mesh vertex of an image structure corresponding to the non-reference image, wherein the quadratic constraints are defined by , where Ep is a local alignment term, Eg is a global constraint term, and Es is a similarity term.
  • the electronic device 101 can generate an output frame using the motion vector map with the reference image and the non-reference image.
  • an image blending operation uses the alignment output to generate a blending map.
  • the electronic device 101 performs post-processing using the reference image, the non-reference image, the motion vector map, and the blending map to generate the output image.
  • FIGURE 8 illustrates one example of a method 800 for efficient regularized image alignment for multi-frame fusion
  • various changes may be made to FIGURE 8.
  • steps in FIGURE 8 could overlap, occur in parallel, occur in a different order, or occur any number of times.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Image Analysis (AREA)

Abstract

Procédé comprenant la réception d'une image de référence et d'une image de non-référence ; la division de l'image de référence en une pluralité de tuiles ; la détermination, à l'aide d'un dispositif électronique, d'une carte de vecteur de mouvement à l'aide d'une estimation de vecteur de mouvement sur une base grossière à fine ; et la génération d'une trame de sortie à l'aide de la carte de vecteur de mouvement avec l'image de référence et l'image de non-référence.
PCT/KR2020/010083 2019-08-06 2020-07-30 Appareil et procédé d'alignement d'image régularisé efficace pour fusion multi-trame WO2021025375A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP20849055.7A EP3956863A4 (fr) 2019-08-06 2020-07-30 Appareil et procédé d'alignement d'image régularisé efficace pour fusion multi-trame
CN202080054064.3A CN114503541A (zh) 2019-08-06 2020-07-30 用于多帧融合的有效率的正则化图像对准的装置和方法

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US201962883306P 2019-08-06 2019-08-06
US62/883,306 2019-08-06
US16/727,751 US11151731B2 (en) 2019-08-06 2019-12-26 Apparatus and method for efficient regularized image alignment for multi-frame fusion
US16/727,751 2019-12-26
KR10-2020-0094683 2020-07-29
KR1020200094683A KR20210018084A (ko) 2019-08-06 2020-07-29 다중 프레임 융합을 위한 효율적인 정칙화된 이미지 정렬 기기 및 방법

Publications (1)

Publication Number Publication Date
WO2021025375A1 true WO2021025375A1 (fr) 2021-02-11

Family

ID=74503208

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2020/010083 WO2021025375A1 (fr) 2019-08-06 2020-07-30 Appareil et procédé d'alignement d'image régularisé efficace pour fusion multi-trame

Country Status (3)

Country Link
EP (1) EP3956863A4 (fr)
CN (1) CN114503541A (fr)
WO (1) WO2021025375A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023273868A1 (fr) * 2021-06-29 2023-01-05 展讯通信（上海）有限公司 Procédé et appareil de débruitage d'images, terminal et support de stockage

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6173087B1 (en) * 1996-11-13 2001-01-09 Sarnoff Corporation Multi-view image registration with application to mosaicing and lens distortion correction
US6809758B1 (en) * 1999-12-29 2004-10-26 Eastman Kodak Company Automated stabilization method for digital image sequences
US20070242900A1 (en) 2006-04-13 2007-10-18 Mei Chen Combining multiple exposure images to increase dynamic range
US20100271498A1 (en) * 2009-04-22 2010-10-28 Qualcomm Incorporated System and method to selectively combine video frame image data
US8711248B2 (en) 2011-02-25 2014-04-29 Microsoft Corporation Global alignment for high-dynamic range image generation
US20160035104A1 (en) * 2013-03-18 2016-02-04 Fotonation Limited A method and apparatus for motion estimation

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3277418B2 (ja) * 1993-09-09 2002-04-22 ソニー株式会社 動きベクトル検出装置および方法
US20100157079A1 (en) * 2008-12-19 2010-06-24 Qualcomm Incorporated System and method to selectively combine images
CN109997351B (zh) * 2016-12-22 2021-02-23 华为技术有限公司 用于生成高动态范围图像的方法和装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6173087B1 (en) * 1996-11-13 2001-01-09 Sarnoff Corporation Multi-view image registration with application to mosaicing and lens distortion correction
US6809758B1 (en) * 1999-12-29 2004-10-26 Eastman Kodak Company Automated stabilization method for digital image sequences
US20070242900A1 (en) 2006-04-13 2007-10-18 Mei Chen Combining multiple exposure images to increase dynamic range
US20100271498A1 (en) * 2009-04-22 2010-10-28 Qualcomm Incorporated System and method to selectively combine video frame image data
US8711248B2 (en) 2011-02-25 2014-04-29 Microsoft Corporation Global alignment for high-dynamic range image generation
US20160035104A1 (en) * 2013-03-18 2016-02-04 Fotonation Limited A method and apparatus for motion estimation

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3956863A4

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023273868A1 (fr) * 2021-06-29 2023-01-05 展讯通信（上海）有限公司 Procédé et appareil de débruitage d'images, terminal et support de stockage

Also Published As

Publication number Publication date
EP3956863A1 (fr) 2022-02-23
EP3956863A4 (fr) 2022-06-08
CN114503541A (zh) 2022-05-13

Similar Documents

Publication Publication Date Title
WO2020171373A1 (fr) Techniques permettant de réaliser une fusion multi-exposition fondée sur un réseau neuronal convolutif d'une pluralité de trames d'image et de corriger le brouillage d'une pluralité de trames d'image
US11151731B2 (en) Apparatus and method for efficient regularized image alignment for multi-frame fusion
WO2021201438A1 (fr) Système et procédé de déformation de mouvement utilisant des trames à multiples expositions
EP4066216A1 (fr) Système et procédé de génération de trames à expositions multiples à partir d'une entrée unique
WO2020251285A1 (fr) Appareil et procédé de création d'image à plage dynamique élevée (hdr) de scènes dynamiques à l'aide d'un étiquetage basé sur une coupe graphique
WO2022019437A1 (fr) Fusion d'images à expositions multiples guidée
WO2020171305A1 (fr) Appareil et procédé de capture et de mélange d'images multiples pour photographie flash de haute qualité à l'aide d'un dispositif électronique mobile
WO2021177784A1 (fr) Génération de carte de profondeur à super-résolution pour caméras multiples ou autres environnements
US10554890B1 (en) Apparatus and method for generating low-light images with improved bokeh using mobile electronic device
WO2021101097A1 (fr) Architecture de réseau neuronal de fusion multi-tâches
WO2022014790A1 (fr) Mise à jour de gradient de rétropropagation guidée pour tâche de traitement d'image utilisant des informations redondantes de l'image
WO2022146023A1 (fr) Système et procédé de rendu d'effet de profondeur de champ synthétique pour des vidéos
US11418766B2 (en) Apparatus and method for chroma processing for multi-frame fusion
WO2021025375A1 (fr) Appareil et procédé d'alignement d'image régularisé efficace pour fusion multi-trame
WO2021256812A1 (fr) Moteur de fusion d'images synthétiques multi-capteurs, multi-vues, multi-trames et multi-tâches pour système d'imagerie mobile
WO2022025565A1 (fr) Système et procédé de génération d'une image de bokeh pour un rendu de profondeur de champ de qualité dslr et procédé d'affinage et d'apprentissage associé
US11200653B2 (en) Local histogram matching with global regularization and motion exclusion for multi-exposure image fusion
CN113627328A (zh) 电子设备及其图像识别方法、片上系统和介质
WO2021112550A1 (fr) Système et procédé de génération de trames à expositions multiples à partir d'une entrée unique
WO2021101037A1 (fr) Système et procédé de sélection dynamique de trame d'image de référence
WO2021107592A1 (fr) Système et procédé de retouche d'image précise pour éliminer un contenu non souhaité d'images numériques
WO2021221492A1 (fr) Systèmes et procédés d'évaluation quantitative de qualité de carte optique et d'automatisation d'augmentation de données
WO2023149786A1 (fr) Procédé et dispositif électronique de synthèse de données d'apprentissage d'image et de traitement d'image à l'aide d'une intelligence artificielle
WO2021025445A1 (fr) Mise en correspondance d'histogramme local avec régularisation globale et exclusion de mouvement pour fusion d'image à expositions multiples
WO2022025741A1 (fr) Estimation de profondeur basée sur un réseau

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20849055

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020849055

Country of ref document: EP

Effective date: 20211115

NENP Non-entry into the national phase

Ref country code: DE