CN116368814A

CN116368814A - Spatial alignment transformation without FOV loss

Info

Publication number: CN116368814A
Application number: CN202180074272.4A
Authority: CN
Inventors: 任剑锋; 刘石忠; 刘伟亮; K-C·潘; N·于希; W·左
Original assignee: Qualcomm Inc
Current assignee: Qualcomm Inc
Priority date: 2020-11-10
Filing date: 2021-10-08
Publication date: 2023-06-30
Also published as: EP4245027A1; US11405563B2; US20220150420A1; WO2022103535A1

Abstract

The present disclosure provides devices, methods, computer-readable media, and units for spatial alignment transformation. In some aspects, a device may perform operations comprising: capturing a first image of the scene at a first zoom ratio using a first sensor; capturing a second image of the scene at a second zoom ratio using a second sensor, the first image having a different field of view (FOV) than the second image; determining a transformation matrix based on one or more spatial misalignments between the first image and the second image; determining a confidence level associated with the transformation matrix; determining a weighting factor based on the first zoom ratio, the second zoom ratio, and the current zoom ratio of the device in response to the confidence being greater than the confidence threshold; applying a weighting factor to the transformation matrix; and warping the first image to a second image using the weighted transformation matrix.

Description

Spatial alignment transformation without FOV loss

Cross Reference to Related Applications

This patent application claims priority from U.S. patent application Ser. No. 17/094,052 entitled "SPATIAL ALIGNMENT TRANSFORM WITHOUT FOV LOSS" filed on even 10/11/2020, which is assigned to the assignee of the present application. The disclosures of all of the prior applications are considered to be part of this patent application and are incorporated by reference into this patent application.

Technical Field

The present disclosure relates generally to image or video capture devices, including processing, by an image signal processor, image frames from a plurality of image sensors, such as for spatial alignment transformation of the image frames.

Background

Many devices include multiple image sensors that may be used to capture one or more image frames. For example, a smart phone or tablet computer includes multiple image sensors for generating images or video for different imaging applications. One or more image signal processors may process image frames from a plurality of image sensors. The processed image frames may be used by an imaging application, for example, to generate preview images, user photographs, record video, perform augmented reality operations, and so forth.

Disclosure of Invention

The systems, methods, and devices of the present disclosure each have several innovative aspects, no single one of which is solely responsible for the desirable attributes disclosed herein.

One innovative aspect of the subject matter described in this disclosure can be implemented in a device. The apparatus may include a processor and a memory coupled to the processor and storing instructions that, when executed by the processor, cause the apparatus to perform operations. The operations may include: capturing a first image of the scene at a first zoom ratio using a first sensor; capturing a second image of the scene at a second zoom ratio using a second sensor, the first image having a different field of view (FOV) than the second image; determining a transformation matrix based on one or more spatial misalignments between the first image and the second image; determining a confidence level associated with the transformation matrix; determining a weighting factor based on the first zoom ratio, the second zoom ratio, and the current zoom ratio of the device in response to the confidence being greater than the confidence threshold; applying a weighting factor to the transformation matrix; and warping the first image to a second image using the weighted transformation matrix.

In some implementations, the first image and the second image are captured simultaneously when the current zoom ratio is within an overlap region spanning between the first zoom ratio and the second zoom ratio. In some examples, the weighting factor is 0% when the current zoom ratio is equal to the first zoom ratio and 100% when the current zoom ratio is equal to the second zoom ratio. In some other examples, the weighting factor corresponds to the identity matrix when the confidence level is not greater than a confidence threshold. In some implementations, the first image has a wider FOV than the second image. In some examples, the first sensor is a wide angle image sensor and the second sensor is a remote image sensor.

In some implementations, execution of the instructions causes the device to perform operations further comprising: gradually increasing the weighting factor as the current zoom ratio increases from the first zoom ratio to the second zoom ratio; and gradually decreasing the weighting factor as the current zoom ratio decreases toward the first zoom ratio within the overlap region. In some other implementations, execution of the instructions causes the device to perform operations further comprising: generating a preview image based on the warped first image and the second image; and displaying the preview image to a user of the device.

In some implementations, execution of the instructions causes the device to perform operations further comprising: scaling the first image to match the FOV of the second image; and identifying one or more spatial misalignments between the scaled first image and the second image. In some other implementations, execution of the instructions causes the device to perform operations further comprising: when the confidence level is greater than the confidence level threshold, determining additional transformation matrices is avoided.

Another innovative aspect of the subject matter described in this disclosure can be implemented as a method for spatial alignment transformation. The method may be performed by a device and may include: capturing a first image of a scene at a first zoom ratio using a first sensor of the device; capturing a second image of the scene at a second zoom ratio using a second sensor of the device, the first image having a different field of view (FOV) than the second image; determining a transformation matrix based on one or more spatial misalignments between the first image and the second image; determining a confidence level associated with the transformation matrix; determining a weighting factor based on the first zoom ratio, the second zoom ratio, and the current zoom ratio of the device in response to the confidence being greater than the confidence threshold; applying a weighting factor to the transformation matrix; and warping the first image to a second image using the weighted transformation matrix.

In some implementations, the method further comprises: gradually increasing the weighting factor as the current zoom ratio increases from the first zoom ratio to the second zoom ratio; and gradually decreasing the weighting factor as the current zoom ratio decreases toward the first zoom ratio within the overlap region. In some other implementations, the method further comprises: generating a preview image based on the warped first image and the second image; and displaying the preview image to a user of the device.

In some implementations, the method further comprises: scaling the first image to match the FOV of the second image; and identifying one or more spatial misalignments between the scaled first image and the second image. In some other implementations, the method further comprises: when the confidence level is greater than the confidence level threshold, determining additional transformation matrices is avoided.

Another innovative aspect of the subject matter described in this disclosure can be implemented in non-transitory computer-readable media. The non-transitory computer-readable medium may store instructions that, when executed by a processor of a device, cause the device to perform operations. In some implementations, the operations include: capturing a first image of the scene at a first zoom ratio using a first sensor; capturing a second image of the scene at a second zoom ratio using a second sensor, the first image having a different field of view (FOV) than the second image; determining a transformation matrix based on one or more spatial misalignments between the first image and the second image; determining a confidence level associated with the transformation matrix; determining a weighting factor based on the first zoom ratio, the second zoom ratio, and the current zoom ratio of the device in response to the confidence being greater than the confidence threshold; applying a weighting factor to the transformation matrix; and warping the first image to a second image using the weighted transformation matrix.

In some implementations, the operations further include: gradually increasing the weighting factor as the current zoom ratio increases from the first zoom ratio to the second zoom ratio; and gradually decreasing the weighting factor as the current zoom ratio decreases toward the first zoom ratio within the overlap region. In some other implementations, the operations further comprise: generating a preview image based on the warped first image and the second image; and displaying the preview image to a user of the device.

In some implementations, the operations further include: scaling the first image to match the FOV of the second image; and identifying one or more spatial misalignments between the scaled first image and the second image. In some other implementations, the operations further comprise: when the confidence level is greater than the confidence level threshold, determining additional transformation matrices is avoided.

Another innovative aspect of the subject matter described in this disclosure can be implemented in a device. The apparatus may include: capturing a first image of a scene at a first zoom ratio using a first sensor; capturing a second image of the scene at a second zoom ratio using a second sensor, the first image having a different field of view (FOV) than the second image; means for determining a transformation matrix based on one or more spatial misalignments between the first image and the second image; determining a confidence level associated with the transformation matrix; means for determining a weighting factor based on the first zoom ratio, the second zoom ratio, and the current zoom ratio of the device in response to the confidence being greater than the confidence threshold; a unit for applying a weighting factor to the transformation matrix; and means for warping the first image to the second image using the weighted transformation matrix.

In some implementations, the first image and the second image are captured simultaneously when the current zoom ratio is within an overlap region spanning between the first zoom ratio and the second zoom ratio. In some examples, the weighting factor is 0% when the current zoom ratio is equal to the first zoom ratio and 100% when the current zoom ratio is equal to the second zoom ratio. In some other examples, the weighting factor corresponds to the identity matrix when the confidence level is not greater than a confidence threshold. In some implementations, the first image has a wider FOV than the second image. In some examples, the first sensor and the second sensor are a wide angle image sensor and a remote image sensor, respectively.

In some implementations, the apparatus further includes means for: gradually increasing the weighting factor as the current zoom ratio increases from the first zoom ratio to the second zoom ratio; and gradually decreasing the weighting factor as the current zoom ratio decreases toward the first zoom ratio within the overlap region. In some other implementations, the apparatus further includes means for: generating a preview image based on the warped first image and the second image; and displaying the preview image to a user of the device.

In some implementations, the apparatus further includes means for: scaling the first image to match the FOV of the second image; and identifying one or more spatial misalignments between the scaled first image and the second image. In some other implementations, the operations further comprise: when the confidence level is greater than the confidence level threshold, determining additional transformation matrices is avoided.

The details of one or more implementations of the subject matter described in this disclosure are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. Note that the relative dimensions of the following figures may not be drawn to scale.

Drawings

Fig. 1 illustrates a block diagram of an example apparatus for performing spatial alignment transforms of image frames.

Fig. 2 illustrates a timing diagram of an example device performing spatial alignment transformation of an image frame.

FIG. 3 illustrates an example data flow for a device performing a spatial alignment transformation of one or more image frames.

FIG. 4A illustrates an example overlap region spanning between a first zoom ratio and a second zoom ratio of a device as a current zoom ratio of the device increases.

FIG. 4B shows a block diagram depicting an example operation of a device to generate a transition matrix as a current zoom ratio of the device increases.

FIG. 5A illustrates an example overlap region spanning between a first zoom ratio and a second zoom ratio of a device as a current zoom ratio of the device decreases.

FIG. 5B shows a block diagram depicting an example operation of the device to generate a transition matrix as the current zoom ratio of the device decreases.

FIG. 6 shows a flowchart depicting example operations of a spatial alignment transformation.

FIG. 7 shows a flowchart depicting another example operation of a spatial alignment transformation.

FIG. 8 shows a flowchart depicting another example operation of a spatial alignment transformation.

FIG. 9 shows a flowchart depicting another example operation of a spatial alignment transformation.

FIG. 10 shows a flowchart depicting another example operation of a spatial alignment transformation.

Like reference numbers and designations in the various drawings indicate like elements.

Detailed Description

Aspects of the present disclosure may be used to capture image frames using multiple image sensors of a device, such as a combination of ultra-wide (high field of view (FOV), wide, remote, and ultra-remote (low FOV) sensors, some aspects include processing the captured image frames, such as by performing a spatial alignment transformation on the captured one or more image frames when the device transitions from capturing an image of a scene using a first image sensor of the multiple image sensors to capturing an image of a scene using a second image sensor of the multiple image sensors.

An example device (e.g., a smart phone) may include a configuration of two, three, four, or more cameras on the back of the device. A device having multiple image sensors includes one or more image signal processors, computer Vision Processors (CVPs), or other suitable circuitry for processing images captured by the image sensors. The one or more image signal processors may provide the processed image frames to a memory and/or processor (e.g., an application processor, an Image Front End (IFE), an Image Processing Engine (IPE), or other suitable processing circuitry) for further processing, e.g., for encoding or other manipulation.

As used herein, an image sensor may refer to the image sensor itself and any other suitable component coupled to the image sensor. For example, an image sensor may also refer to other components of a camera, including a shutter, a buffer, or other readout circuitry. The image sensor may also refer to an analog front end or other circuitry for converting an analog signal into a digital representation of a frame. Thus, the term "image sensor" herein may refer to any suitable component for capturing and reading out image frames to an image signal processor.

An example multi-sensor device may capture a "wide" image of a scene using a wide sensor when a current zoom ratio of the device is a first value, and may switch to capture a "remote" image of the scene using a remote sensor when the current zoom ratio is a second, higher value. The device may capture images of the scene using two sensors simultaneously or approximately simultaneously when the current zoom ratio is within a defined range of values, which may be referred to herein as an "overlap region. The device may use image data from one or more sensors to generate, for example, a preview image of a scene for display to a device user.

However, due to manufacturing imperfections, the plurality of sensors and/or captured images may be spatially misaligned, which may result in misalignment errors associated with the generated images when the current zoom ratio of the device is within the overlap region. Misalignment errors may result in visual defects in the corresponding preview image. Some devices may use the transformation matrix to warp (e.g., shift and/or rotate) the image into spatial alignment, which may account for some misalignment errors. Unfortunately, in doing so, conventional devices crop out portions of the higher FOV image and generate a corresponding image with reduced FOV and quality (i.e., fewer pixels).

Aspects of the present disclosure provide a multi-sensor device that can smoothly transition from capturing an image of a scene using a first one of the sensors to capturing an image of a scene using a second one of the sensors by warping one or more captured images without degrading FOV or quality in generating the corresponding images.

In the following description, numerous specific details are set forth, such as examples of specific components, circuits, and processes, in order to provide a thorough understanding of the present disclosure. The term "coupled," as used herein, means directly connected or connected through one or more intervening components or circuits. Furthermore, in the following description and for purposes of explanation, specific nomenclature is set forth to provide a thorough understanding of the present disclosure. It will be apparent, however, to one skilled in the art that these specific details may not be required to practice the teachings disclosed herein. In other instances, well-known circuits and devices are shown in block diagram form in order not to obscure the teachings of the present disclosure. Some portions of the detailed descriptions which follow are presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. In this disclosure, a procedure, logic block, process, etc., is conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present application, discussions utilizing terms such as "accessing," "receiving," "transmitting," "using," "selecting," "determining," "normalizing," "multiplying," "averaging," "monitoring," "comparing," "applying," "updating," "measuring," "deriving," "resolving," "generating," or the like, refer to the operation and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's registers, memories, or other such information storage, transmission or display devices.

In the figures, a single block may be described as performing one or more functions. However, in actual practice, one or more functions performed by the block may be performed in a single component or across multiple components, and/or may be performed using hardware, software, or a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described below generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Further, example devices may include components other than those shown, including well-known components such as processors, memory, and the like.

Aspects of the present disclosure are applicable to any suitable electronic device that includes or is coupled to two or more image sensors capable of capturing image frames (or "frames"). Further, aspects of the present disclosure may be implemented in image sensors or devices coupled to image sensors having the same or different capabilities and characteristics (e.g., resolution, shutter speed, sensor type, etc.).

The terms "device" and "apparatus" are not limited to one or a particular number of physical objects (e.g., a smart phone, a camera controller, a processing system, etc.). As used herein, a device may be any electronic device having one or more components that may implement at least some portions of the present disclosure. While the following description and examples use the term "device" to describe various aspects of the disclosure, the term "device" is not limited to a particular configuration, type, or number of objects. As used herein, an apparatus may comprise a device or portion of a device for performing the described operations.

Fig. 1 illustrates a block diagram of an example apparatus 100 for performing spatial alignment transforms of image frames. The device 100 may include or otherwise be coupled to an image signal processor 112 for processing image frames from a plurality of image sensors (e.g., the first image sensor 101 and the second image sensor 102). In some implementations, the device 100 also includes or is coupled to the processor 104 and the memory 106 that stores instructions 108. The device 100 may also include or be coupled to a display 114 and a plurality of input/output (I/O) components 116. The device 100 may also include or be coupled to a power source 118 for the device 100, such as a battery or a component that couples the device 100 to an energy source. The device 100 may also include or be coupled to additional features or components not shown. In one example, a wireless interface for a wireless communication device may be included, which may include multiple transceivers and baseband processors. In another example, one or more other sensors (e.g., a gyroscope or a Global Positioning System (GPS) receiver) may be included in or coupled to the device. In another example, an analog front end for converting analog image frame data to digital image frame data may be coupled between the

image sensors

101 and 102 and the image signal processor 112.

The first image sensor 101 and the second image sensor 102 are configured to capture one or more image frames. For example, the first image sensor 101 and the second image sensor 102 may be included in one multi-camera configuration, or in a single-camera or multi-camera configuration alone (e.g., for a smart phone or other suitable device, a two-camera configuration, a three-camera configuration, etc.). The

image sensors

101 and 102 may also include or be coupled to one or more lenses for focusing light, one or more apertures for receiving light, one or more shutters for blocking light outside of an exposure window, one or more Color Filter Arrays (CFAs) for filtering light outside of a particular frequency range, one or more analog front ends for converting analog measurements to digital information, or other suitable components for imaging. The device 100 may also include a flash, depth sensor, GPS, or other suitable components for imaging.

The image signal processor 112 processes image frames captured by the

image sensors

101 and 102. Although fig. 1 shows that device 100 includes two

image sensors

101 and 102 coupled to image signal processor 112, any number of image sensors may be coupled to image signal processor 112. Furthermore, any number of additional image sensors or image signal processors may be present for the device 100.

In some aspects, the image signal processor 112 may execute instructions from a memory, such as instructions 108 from the memory 106, instructions stored in a separate memory coupled to or contained in the image signal processor 112, or instructions provided by the processor 104. Additionally or alternatively, the image signal processor 112 may execute software and/or may include specific hardware (e.g., one or more Integrated Circuits (ICs)) to perform one or more operations described in this disclosure.

In some implementations, the memory 106 may include a non-transitory or non-transitory computer-readable medium storing computer-executable instructions 108 to perform all or part of one or more operations described in this disclosure. In some implementations, the instructions 108 include a camera application (or other suitable application) to be executed by the device 100 for generating images or video. The instructions 108 may also include other applications or programs executed by the device 100, such as an operating system and specific applications other than for image or video generation. Execution of the camera application (e.g., by the processor 104) may cause the device 100 to generate an image using the image sensor and the image signal processor 112. The memory 106 may also be accessed by the image signal processor 112 to store processed frames or may be accessed by the processor 104 to obtain processed frames. In some other implementations, the device 100 does not include the memory 106. For example, the device 100 may be a circuit including the image signal processor 112, and the memory may be external to the device 100. The device 100 may be coupled to the memory and configured to access the memory to write the processed frame.

In some implementations, the processor 104 may include one or more general-purpose processors capable of executing scripts or instructions (e.g., instructions 108 stored in the memory 106) of one or more software programs. For example, the processor 104 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or video) stored in the memory 106. In executing a camera application, the processor 104 may be configured to instruct the image signal processor 112 to perform one or more operations with reference to the

image sensor

101 or 102. Execution of the instructions 108 by the processor 104 external to the camera application may also cause the device 100 to perform any number of functions or operations. In some implementations, the processor 104 may include an ICS or other hardware in addition to the ability to execute software to cause the device 100 to perform a plurality of functions or operations (e.g., the operations described herein). In some other implementations, the device 100 does not include the processor 104.

In some implementations, at least one of the image signal processor 112 or the processor 104 may execute instructions to perform various operations described herein. For example, execution of the instructions may determine the transformation matrix based on one or more spatial misalignments between a first image of the scene captured at a first zoom ratio using the first image sensor 101 and a second image of the scene captured at a second zoom ratio using the second image sensor 102. In some aspects, the first image may have a different FOV than the second image. Execution of the instructions may also determine a confidence associated with the transformation matrix. Execution of the instructions may also determine the weighting factor based on a first zoom ratio, a second zoom ratio, and a current zoom ratio of the device. Thereafter, execution of the instructions may apply a weighting factor to the transformation matrix and warp the first image to the second image using the weighted transformation matrix.

In some implementations, the display 114 may include one or more suitable displays or screens that allow user interaction and/or presentation of items to a user (e.g., previews of image frames captured by the image sensors 101 and 102). In some aspects, the display 114 is a touch sensitive display. The I/O component 116 may be or include any suitable mechanism, interface, or device to receive input (e.g., commands) from a user and provide output to the user. For example, the I/O component 116 may include, but is not limited to, a Graphical User Interface (GUI), a keyboard, a mouse, a microphone, a speaker, a squeezable bezel, one or more buttons (e.g., power buttons), a slider, a switch, and the like.

Although shown coupled to each other via the processor 104, the memory 106, the image signal processor 112, the display 114, and the I/O component 116 may be coupled to each other in other various arrangements, such as via one or more local buses, which are not shown for simplicity. Although the image signal processor 112 is shown as being separate from the processor 104, the image signal processor 112 may be the core of the processor 104, the processor 104 being an Application Processor Unit (APU), contained in a system on a chip (SoC), or otherwise contained in the processor 104. Although reference is made in the examples herein to device 100 to perform aspects of the disclosure, some device components may not be shown in fig. 1 to prevent obscuring aspects of the disclosure. Additionally, other components, numbers of components, or combinations of components may be included in suitable devices for performing aspects of the present disclosure. Accordingly, the present disclosure is not limited to the configuration of a particular device or component, including device 100.

Fig. 2 illustrates a timing diagram 200 of an example device performing spatial alignment transformation of an image frame. An example device may be an example implementation of device 100 shown in fig. 1. Although the timing diagram 200 is described with reference to the devices and components shown in fig. 1, any suitable device or device component may perform the operations described in the timing diagram 200. The operations shown in the timing diagram 200 may be performed when the current zoom ratio of the device 100 is within an overlap region spanning between the first zoom ratio and the second zoom ratio, as described in more detail below. In some implementations, the first image sensor 101 is a wide angle image sensor and the second image sensor 102 is a remote image sensor.

The image signal processor 112 triggers the first image sensor 101 and the second image sensor 102 to capture a first image and a second image, respectively (202). In some implementations, the first image may have a wider FOV than the second image. Although timing diagram 200 illustrates providing triggers to image

sensors

101 and 102 simultaneously, triggers may be provided at different times or at any suitable time or times. Upon receiving the trigger, the first image sensor 101 may capture a first image at a first zoom ratio (204) and the second image sensor 102 may capture a second image at a second zoom ratio (206). In some implementations, the image sensors may capture images at or near the same time.

The second image sensor 102 may send the second image to the image signal processor 112 (208), and the first image sensor 101 may send the first image to the image signal processor 112 (210). In some cases, the image sensor may send the captured images to the processor at or near the same time. The image signal processor 112 receives the first image and the second image from the first image sensor 101 and the second image sensor 102, respectively (212). In some cases, the image signal processor 112 may receive images at or near the same time.

The image signal processor 112 may determine a transformation matrix based on one or more spatial misalignments between the first image and the second image (214). In some implementations not shown in the timing diagram 200, the image signal processor 112 scales the first image to match the FOV of the second image and identifies one or more spatial misalignments between the scaled first and second images. The image signal processor 112 may determine a confidence level associated with the transformation matrix (216) and compare the confidence level to a confidence threshold (218). In some implementations, the confidence threshold may be a predetermined value and/or an adjustable value.

In response to the confidence being greater than the confidence threshold, the image signal processor 112 may determine a weighting factor based on the first zoom ratio, the second zoom ratio, and the current zoom ratio of the device 100 (220). In some implementations, the weighting factor may be 0% when the current zoom ratio is equal to the first zoom ratio and 100% when the current zoom ratio is equal to the second zoom ratio. In some cases, the image signal processor 112 may avoid determining additional transformation matrices when the confidence level is greater than a confidence threshold. In some other implementations, the weighting factor may be an identity matrix when the confidence level is not greater than a confidence threshold.

The image signal processor 112 may then apply a weighting factor to the transformation matrix (222). In some implementations, the image signal processor 112 may increase the weighting factor as the current zoom ratio increases toward the second zoom ratio within the overlap region; and decreasing the weighting factor as the current zoom ratio decreases toward the first zoom ratio within the overlap region. The image signal processor 112 may send the weighted conversion matrix to the processor 104 (224), and the processor 104 may then receive the weighted conversion matrix (226). The processor 104 may then warp the first image to a second image using the weighted transformation matrix (228). In some cases, the device 100 may generate a preview image based on the warped first and second images and display the preview image to a user of the device 100.

The operations described with reference to timing diagram 200 may be extended to any number of image sensors coupled to an image signal processor and any number of frames to be provided to the image signal processor. Accordingly, the timing diagram 200 shows only two image sensors for clarity of explanation of aspects of the present disclosure, i.e., the present disclosure is not limited to a particular number of image sensors. For example, in some implementations, the device 100 may have some combination of ultra-wide sensors, remote sensors, ultra-remote sensors, and any other suitable type of sensor.

FIG. 3 illustrates an example data flow 300 for a device performing spatial alignment transformation of one or more image frames. The device may be an example implementation of the device 100 shown in fig. 1. Although data stream 300 is described with reference to the devices, components, and operations illustrated in the previous figures, any suitable device or component of a device may perform the operations described with respect to data stream 300. In some implementations, the operations shown in data stream 300 may be performed when the current zoom ratio of device 100 is within the overlap region.

The first image 302 and the second image 312 are captured by a first image sensor and a second image sensor, respectively, of the device 100, which are not shown for simplicity. In some implementations, the first and second image sensors may be example implementations of the first and

second image sensors

101 and 102, respectively, of fig. 1. As a non-limiting example, the first image 302 may be a wide-angle image captured at a first zoom ratio (e.g., 4.9X), and the second image 312 may be a remote image captured at a second zoom ratio (e.g., 5.0X). In some implementations, the first image 302 and the second image 312 may be captured at an initial resolution (e.g., 2304 x 1728) that is higher than a final output resolution of the images (e.g., 1920 x 1440). As shown in fig. 3, the device 100 may crop the image to a smaller resolution (e.g., 480 x 360) for further processing at the image signal processor 112.

In some implementations, the image signal processor 112 scales the first image to match the FOV of the second image and identifies one or more spatial misalignments between the scaled first and second images. As shown in fig. 3, the image signal processor 112 determines a transformation matrix based on the one or more spatial misalignments, determines a confidence level associated with the transformation matrix, compares the confidence level to a confidence threshold, determines a weighting factor in response to the confidence level being greater than the confidence threshold, and applies the weighting factor to the transformation matrix, as described with respect to fig. 2. Thereafter, the image signal processor 112 may send the weighted conversion matrix to the processor 104 (324).

The processor 104 may receive the first image at an initial resolution (326) and warp the first image to a second image using a weighted transformation matrix from the image signal processor 112. In some aspects, the final output resolution (e.g., 1920 x 1440) of the warped first image may be less than the initial resolution (e.g., 2304 x 1728) of the first image 302. In this way, the first image 302 may have a pixel margin between the final output resolution and the initial resolution. Thus, when the processor 104 distorts (e.g., shifts and/or rotates) the first image 302 to the second image, only pixels within the pixel margin may shift or rotate out of the frame, so the processor 104 may, for example, output (330) the distorted image to the display of the device 100 without any FOV or quality loss, as only pixels within the pixel margin may shift or rotate out of the frame. In some implementations, the device 100 may then generate the preview image based on the warped first and second images 312.

Fig. 4A illustrates an example overlap region 400 spanning between a first zoom ratio and a second zoom ratio of a device as a current zoom ratio of the device increases. The device may be an example implementation of the device 100 shown in fig. 1. Although the overlap region 400 is described with reference to the devices, components, and operations shown in the previous figures, any suitable device or device component may perform the operations described with respect to fig. 4A. In some implementations, the first sensor is an example implementation of the first image sensor 101 of fig. 1 and may be a wide angle image sensor (or "wide sensor"), and the second image sensor is an example implementation of the second sensor 102 of fig. 1 and may be a remote image sensor (or "remote sensor").

When the current zoom ratio of the device 100 is within the first sensor region 402, the device 100 may capture only a wide image using only a wide sensor. In some implementations, the first sensor region 402 may begin at the leftmost side of fig. 4A (e.g., a zoom ratio of 1.0X) and end at a "first zoom ratio" (e.g., a zoom ratio of 2.7X). Conversely, when the current zoom ratio of the device 100 is within the second sensor region 404, the device 100 may capture only remote images using only remote sensors. In some implementations, the second sensor region 404 may begin at a second zoom ratio (e.g., zoom ratio 3.0X) and end at the far right side of fig. 4A (e.g., zoom ratio 10.0X).

When the current zoom ratio of the device 100 is within the overlap region (e.g., a zoom ratio between 2.7X-3.0X), the device 100 may capture a wide image using the wide sensor while capturing a remote image using the remote sensor, as described with respect to fig. 2. The device 100 may determine that at least one of the captured wide images is spatially misaligned with at least one of the captured remote images, for example, if the wide sensor and the remote sensor are spatially misaligned. Thus, as described above, the device 100 may determine a transformation matrix M based on spatial misalignment between the wide image and the remote image for warping the wide image into spatial alignment with the remote image, determine a confidence associated with M, and compare the confidence to a confidence threshold to determine whether M may be used to align the wide image and the remote image with a level of accuracy above a certain value.

In some implementations, the weighting factor r for application to M may be an identity matrix when the confidence is not greater than the confidence threshold. In such implementations, for example, the device 100 may generate additional transformation matrices based on identifying spatial misalignments between the additional wide image and the remote image when the current zoom ratio changes. In some implementations, the device 100 may refrain from determining additional transformation matrices when the confidence level is greater than the confidence threshold.

In response to the confidence being greater than the confidence threshold, the device 100 may determine r based on the first zoom ratio, the second zoom ratio, and the current zoom ratio. The device 100 may then apply r to M to generate a weighted transformation matrix M 'and warp the wide image to the remote image using M'. In some implementations, the device 100 may increase r as the current zoom ratio increases within the overlap region 400. For example, r may be 0% when the current zoom ratio is equal to the first zoom ratio, and r may be 100% when the current zoom ratio is equal to the second zoom ratio. In this way, the device 100 may warp the wide image throughout the overlap region 400 using the same (weighted) transformation matrix M.

In some implementations, the apparatus 100 may increase r in proportion to an increase in the current zoom ratio within the overlap region 400. For illustration, the example of fig. 4A includes two vertical lines between the first zoom ratio and the second zoom ratio to show three equal portions within the overlap region 400. As a non-limiting example, the first zoom ratio may be 2.7X, the left vertical line may represent a zoom ratio of 2.8X, the right vertical line may represent a zoom ratio of 2.9X, and the second zoom ratio may be 3.0X. Thus, when the current zoom ratio is 2.7X, the apparatus 100 may calculate r as 0%; 33% at a current zoom ratio of 2.8X; 66% at a current zoom ratio of 2.9X; and 100% when the current zoom ratio is 3.0X. More specifically, the device 100 may calculate r as:

In some implementations, firstzoom ratio may represent the zoom ratio at which the device 100 begins to transition from capturing an image of a scene using a first sensor to capturing an image of a scene using a smaller second FOV sensor. In such implementations, the first zoom ratio may be considered as the beginning of the transition region. In this example, second zoom ratio may represent the device 100 completing the transition from capturing an image of the scene using the first sensor to capturing an image of the scene using the smaller second FOV sensor, and may be considered the end of the transition region (or "optical zoom ratio").

In the example of fig. 4A, firstzoom ratio may be 2.7X and second zoom ratio may be 3.0X, which may represent a zoom ratio of 1.0X from the perspective of a smaller FOV (e.g., remote) sensor. Thus, if currentzoom ratio is 2.8X, device 100 may calculate r as follows:

in some implementations, the device 100 may then weight M according to the following formula, where I is an identity matrix

And wherein M' _n A weighted transformation matrix M' representing the position of the corresponding nth location within the overlap region:

M′ _n ＝|r*M _n +(1-r)*I|

for this example, where r=33%, device 100 may compare M' _n The calculation is as follows:

the device 100 may then use M' _n The wide image is warped to the remote image. By dynamically increasing r as the current zoom ratio increases, the device 100 may warp the wide image to the remote image using the same (dynamically weighted) transformation matrix without reducing its FOV (e.g., 2.7X) in the entire overlap region 400. In this way, the device 100 may transition from capturing an image of a scene using a wide sensor to capturing an image of a scene using a remote sensor without degrading the FOV or quality of a preview image generated, for example, based on the remote image and the warped wide image.

As another illustrative example of fig. 4A, if currentzoom ratio is 2.7X, device 100 may calculate r and M 'as follows' _n ：

M′ _n ＝|0％*M _n +(100％-0％)*I|＝I

As another illustrative example of fig. 4A, if currentzoom ratio is 3.0X, device 100 may calculate r and M 'as follows' _n ：

M′ _n ＝|100％*M _n +(100％-100％)*I|＝M _n

In some implementations, not shown, the device may have more than two sensors, such as an ultra-wide sensor, a remote sensor, and an ultra-remote sensor. In such an implementation, the device may utilize the above-described procedure to make a smooth transition between the sensors as the current zoom ratio increases by a "first overlap region" that spans from a particular zoom ratio within the ultra-wide sensor zoom range to a particular zoom ratio within the wide sensor zoom range. Similarly, as the current zoom ratio increases by a "second overlap region" that spans from a particular higher zoom ratio within the wide sensor zoom range to a particular zoom ratio within the remote sensor zoom range, the device may utilize the above-described process to make a smooth transition between the sensors. Similarly, as the current zoom ratio increases by a "third overlap region" that spans from a particular higher zoom ratio within the remote sensor zoom range to a particular zoom ratio within the ultra-remote sensor zoom range, the device may utilize the above-described process to make a smooth transition between sensors.

Fig. 4B shows a block diagram 450 depicting example operations of a device to generate a transition matrix when a current zoom ratio of the device increases. The device may be an example implementation of the device 100 shown in fig. 1. Although block diagram 450 is described with reference to the devices, components, and operations illustrated in the previous figures, any suitable device or device component may perform the operations described with respect to block diagram 450.

As shown in fig. 4B, the image signal processor 112 may receive a first image captured at a first zoom ratio (e.g., 2.7X) and a second image captured at a second zoom ratio (e.g., 3.0X). In some implementations, the first image may be a wide image captured using a wide sensor and the second image may be a remote image captured using a remote sensor. In some implementations, the device 100 may adjust (e.g., crop and/or scale) the wide image to match the FOV of the remote image, and generate the transformation matrix M based on the adjusted wide image (e.g., at 3.0X FOV) and the remote image. In some cases, the device 100 may also adjust (e.g., crop and/or zoom) the remote image to a second zoom ratio (e.g., 3.0X in this example).

FIG. 5A illustrates an example overlap region 500 spanning between a first zoom ratio and a second zoom ratio of a device as a current zoom ratio of the device decreases. The device may be an example implementation of the device 100 shown in fig. 1. Although overlap region 500 is described with reference to the devices, components, and operations shown in the previous figures, any suitable device or device component may perform the operations described with respect to fig. 5A. In some implementations, the first sensor is an example implementation of the first image sensor 101 of fig. 1 and may be a wide angle image sensor (or "wide sensor"), and the second image sensor is an example implementation of the second sensor 102 of fig. 1 and may be a remote image sensor (or "remote sensor").

Similar to fig. 4A, when the current zoom ratio of the device 100 is within the first sensor region 502, the device 100 may capture only a wide image using only a wide sensor; when the current zoom ratio of the device 100 is within the second sensor region 504, the device 100 may capture only remote images using only remote sensors; and when the current zoom ratio of the device 100 is within the overlap region 500 (2.7X-3.0X in this example), the device 100 may capture a wide image using the wide sensor while capturing a remote image using the remote sensor. As described with respect to fig. 4A, the device 100 may determine a transformation matrix M for warping the wide image into spatial alignment with the remote image, and may determine a weighting factor r for weighting M to generate a weighted transformation matrix M'.

In contrast to fig. 4A, in some implementations, the device 100 may decrease r as the current zoom ratio decreases within the overlap region 500. As a non-limiting example, the first zoom ratio may be 2.7X, the left vertical line may represent a zoom ratio of 2.8X, the right vertical line may represent a zoom ratio of 2.9X, and the second zoom ratio may be 3.0X. Thus, in the opposite manner as described with reference to FIG. 4A, the apparatus 100 may calculate r as 0% when the current zoom ratio is 3.0X, 33% when the current zoom ratio is 2.9X, 2.8X when the current zoom ratio is 66%, 100% when the current zoom ratio is 2.7X, and correspondingly generate M' _n Then use M' _n The wide image is warped to the remote image.

The device 100 may then use M' _n The wide image is warped to the remote image. By dynamically decreasing r as the current zoom ratio increases, the device 100 may warp the wide image to the remote image using the same (dynamically weighted) transformation matrix without decreasing its FOV (e.g., 2.7X) throughout the overlap region 500. In this wayThe device 100 may smoothly transition from capturing an image of a scene using a wide sensor to capturing an image of a scene using a remote sensor without degrading the FOV or quality of a preview image generated based on the remote image and the warped wide image, for example.

Fig. 5B shows a block diagram 550 depicting example operations of the device to generate a transition matrix when the current zoom ratio of the device decreases. The device may be an example implementation of the device 100 shown in fig. 1. Although block 550 is described with reference to the devices, components, and operations shown in the previous figures, any suitable device or device component may perform the operations described with respect to block 550.

As shown in fig. 5B, the image signal processor 112 may receive a first image captured at a first zoom ratio (e.g., 2.7X) and a second image captured at a second zoom ratio (e.g., 3.0X). In some implementations, the first image may be a wide image captured using a wide sensor and the second image may be a remote image captured using a remote sensor. In some implementations, the device 100 may adjust (e.g., crop and/or scale) the wide image to match the FOV of the remote image, and generate the transformation matrix M based on the adjusted wide image (e.g., at 3.0X FOV) and the remote image.

Fig. 6 shows a flow chart depicting an example operation 600 for spatial alignment transformation. The operations 600 may be performed by a device, such as the device 100 shown in fig. 1. At block 602, the device captures a first image of a scene at a first zoom ratio using a first sensor. At block 604, the device captures a second image of the scene at a second zoom ratio using a second sensor, the first image having a different field of view (FOV) than the second image. At block 606, the device determines a transformation matrix based on one or more spatial misalignments between the first image and the second image. At block 608, the device determines a confidence level associated with the transformation matrix. At block 610, in response to the confidence being greater than the confidence threshold, the device determines a weighting factor based on the first zoom ratio, the second zoom ratio, and the current zoom ratio of the device. At block 612, the device applies a weighting factor to the transformation matrix. At block 614, the device distorts the first image to a second image using the weighted transformation matrix.

In some implementations, the first image and the second image are captured simultaneously when the current zoom ratio is within an overlap region spanning between the first zoom ratio and the second zoom ratio. In some cases, the weighting factor is 0% when the current zoom ratio is equal to the first zoom ratio, and 100% when the current zoom ratio is equal to the second zoom ratio. In other cases, the weighting factor corresponds to the identity matrix when the confidence level is not greater than the confidence threshold. In some implementations, the first image has a wider FOV than the second image. In some cases, the first sensor is a wide angle image sensor and the second sensor is a remote image sensor.

Fig. 7 shows a flow chart depicting another example operation 700 for spatial alignment transformation. The operations 700 may be performed by a device, such as the device 100 shown in fig. 1. In various aspects, operation 700 may be one implementation for determining the weighting factor in block 610 of operation 600 of fig. 6. In other various aspects, the operations 700 may begin after the operations 600 of fig. 6. For example, after the first image is warped to the second image in block 614 of operation 600, operation 700 may begin in block 702. At block 702, the device gradually increases the weighting factor as the current zoom ratio increases from the first zoom ratio to the second zoom ratio. At block 704, the device gradually decreases the weighting factor as the current zoom ratio decreases toward the first zoom ratio within an overlap region spanning between the first zoom ratio and the second zoom ratio.

Fig. 8 shows a flow chart depicting another example operation 800 for spatial alignment transformation. The operations 800 may be performed by a device, such as the device 100 shown in fig. 1. In various aspects, operation 800 may begin after operation 600 of fig. 6. For example, after the first image is warped to the second image in block 614 of operation 600, operation 800 may begin in block 802. At block 802, the device generates a preview image based on the warped first image and the second image. At block 804, the device displays the preview image to a user of the device.

Fig. 9 shows a flow chart depicting another example operation 900 for spatial alignment transformation. Operation 900 may be performed by a device, such as device 100 shown in fig. 1. In various aspects, after capturing a second image of the scene in block 604 of operation 600 of fig. 6, operation 900 may begin in block 902. At block 902, the device scales the first image to match the FOV of the second image. At block 904, the device identifies one or more spatial misalignments between the scaled first image and the second image.

Fig. 10 shows a flow chart depicting another example operation 1000 for spatial alignment transformation. Operation 1000 may be performed by a device, such as device 100 shown in fig. 1. In various aspects, after determining the confidence associated with the transformation matrix in block 608 of operation 600 of fig. 6, operation 1000 may begin in block 1002. At block 1002, the device refrains from determining an additional transformation matrix when the confidence is greater than a confidence threshold.

As used herein, a phrase referring to "at least one" of a list of items refers to any combination of these items, including individual members. For example, "at least one of a, b, or c" is intended to encompass: a. b, c, a-b, a-c, b-c and a-b-c.

The various illustrative logics, logical blocks, modules, circuits, and algorithm processes described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. The interchangeability of hardware and software has been described generally in terms of functionality, and is illustrated in the various illustrative components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system.

Hardware and data processing apparatus for implementing the various illustrative logics, logical blocks, modules, and circuits described in connection with the aspects disclosed herein may be implemented or performed with: a general purpose single or multi-chip processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof for performing the functions described herein. A general purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor), a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. In some implementations, particular processes and methods may be performed by circuitry that is specific to a given function.

In one or more aspects, the functions described may be implemented in hardware, digital electronic circuitry, computer software, firmware (including the structures disclosed in this specification and their equivalents), or in any combination thereof. Implementations of the subject matter described in this specification can also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by, or to control the operation of, data processing apparatus.

If implemented in software, these functions may be stored on a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. The processes of the methods or algorithms disclosed herein may be implemented in processor-executable software modules residing on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that supports transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Further, any connection is properly termed a computer-readable medium. Disk (disc) and optical disc (disc), as used herein, includes Compact Disc (CD), laser disc, optical disc, digital Versatile Disc (DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and instructions on a machine readable medium and computer readable medium, which codes and instructions may be incorporated into a computer program product

Various modifications to the implementations described in this disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other implementations without departing from the spirit or scope of the disclosure. Thus, the claims are not intended to be limited to the implementations shown herein but are to be accorded the widest scope consistent with the disclosure and the principles and novel features disclosed herein.

Claims

1. An apparatus, comprising:

a processor; and

a memory coupled to the processor and storing instructions that, when executed by the processor, cause the apparatus to perform operations comprising:

capturing a first image of the scene at a first zoom ratio using a first sensor;

capturing a second image of the scene at a second zoom ratio using a second sensor, the first image having a different field of view (FOV) than the second image;

determining a transformation matrix based on one or more spatial misalignments between the first image and the second image;

determining a confidence level associated with the transformation matrix;

determining a weighting factor based on the first, second, and current zoom ratios of the device in response to the confidence being greater than a confidence threshold;

Applying the weighting factors to the transformation matrix; and

the first image is warped to the second image using a weighted transform matrix.

2. The device of claim 1, wherein the weighting factor comprises an identity matrix when the confidence level is not greater than the confidence threshold.

3. The apparatus of claim 1, wherein the first image and the second image are captured simultaneously when the current zoom ratio is within an overlap region spanning between the first zoom ratio and the second zoom ratio.

4. The apparatus of claim 3, wherein the weighting factor is 0% when the current zoom ratio is equal to the first zoom ratio and 100% when the current zoom ratio is equal to the second zoom ratio.

5. The device of claim 3, wherein execution of the instructions causes the device to perform operations further comprising:

gradually increasing the weighting factor as the current zoom ratio increases from the first zoom ratio to the second zoom ratio; and

the weighting factor is gradually reduced as the current zoom ratio is reduced toward the first zoom ratio within the overlap region.

6. The device of claim 1, wherein execution of the instructions causes the device to perform operations further comprising:

generating a preview image based on the warped first image and the second image; and

the preview image is displayed to a user of the device.

7. The apparatus of claim 1, wherein the first image has a wider FOV than the second image.

8. The apparatus of claim 1, wherein the first sensor comprises a wide angle image sensor and the second sensor comprises a remote image sensor.

9. The device of claim 1, wherein execution of the instructions causes the device to perform operations further comprising:

scaling the first image to match the FOV of the second image; and

one or more spatial misalignments between the scaled first and second images are identified.

10. The device of claim 1, wherein execution of the instructions causes the device to perform operations further comprising:

when the confidence level is greater than the confidence level threshold, determining an additional transformation matrix is avoided.

11. A method, comprising:

Capturing a first image of a scene at a first zoom ratio using a first sensor of the device;

capturing a second image of the scene at a second zoom ratio using a second sensor of the device, the first image having a different field of view (FOV) than the second image;

determining a confidence level associated with the transformation matrix;

applying the weighting factors to the transformation matrix; and

12. The method of claim 11, wherein the weighting factor comprises an identity matrix when the confidence level is not greater than the confidence threshold.

13. The method of claim 11, wherein the first image and the second image are captured simultaneously when the current zoom ratio is within an overlap region spanning between the first zoom ratio and the second zoom ratio.

14. The method of claim 13, wherein the weighting factor is 0% when the current zoom ratio is equal to the first zoom ratio and 100% when the current zoom ratio is equal to the second zoom ratio.

15. The method of claim 13, further comprising:

16. The method of claim 11, further comprising:

the preview image is displayed to a user of the device.

17. The method of claim 11, wherein the first image has a wider FOV than the second image.

18. The method of claim 11, wherein the first sensor comprises a wide angle image sensor and the second sensor comprises a remote image sensor.

19. The method of claim 11, further comprising:

Scaling the first image to match the FOV of the second image; and

20. The method of claim 11, further comprising:

21. A non-transitory computer-readable medium storing instructions that, when executed by a processor of a device, cause the device to perform operations comprising:

determining a confidence level associated with the transformation matrix;

applying the weighting factors to the transformation matrix; and

22. The computer-readable medium of claim 21, wherein the weighting factor comprises an identity matrix when the confidence level is not greater than the confidence level threshold.

23. The computer-readable medium of claim 21, wherein the first image and the second image are captured simultaneously when the current zoom ratio is within an overlap region spanning between the first zoom ratio and the second zoom ratio.

24. The computer-readable medium of claim 23, wherein the weighting factor is 0% when the current zoom ratio is equal to the first zoom ratio and 100% when the current zoom ratio is equal to the second zoom ratio.

25. The computer-readable medium of claim 23, wherein execution of the instructions causes the device to perform operations further comprising:

26. The computer-readable medium of claim 21, wherein execution of the instructions causes the device to perform operations further comprising:

the preview image is displayed to a user of the device.

27. The computer readable medium of claim 21, wherein the first image has a wider FOV than the second image.

28. The computer readable medium of claim 21, wherein the first sensor comprises a wide angle image sensor and the second sensor comprises a remote image sensor.

29. The computer-readable medium of claim 21, wherein execution of the instructions causes the device to perform operations further comprising:

scaling the first image to match the FOV of the second image; and

30. An apparatus, comprising:

capturing a first image of a scene at a first zoom ratio using a first sensor;

Means for determining a transformation matrix based on one or more spatial misalignments between the first image and the second image;

means for determining a confidence level associated with the transformation matrix;

means for determining a weighting factor based on the first, second, and current zoom ratios of the device in response to the confidence being greater than a confidence threshold;

means for applying the weighting factor to the transformation matrix; and

and means for warping the first image to the second image using a weighted transformation matrix.