DE102017009118A1 - IMAGE BALANCE FOR SERIAL PICTURES - Google Patents

Abstract

The present disclosure relates to systems and methods for generating a new matched image from a plurality of burst images. The systems and methods divide a reference image into a plurality of regions and a subsequent image into a plurality of corresponding local regions. In addition, the systems and methods capture a plurality of feature points in each of the reference image and the subsequent image and determine matched feature point pairs between the reference image and the subsequent image. On the basis of the matched feature point pairs, the systems and methods determine at least one homography of the reference image to the subsequent image. Based on homography, the systems and methods generate a new matched image that is matched pixel by pixel to the reference image. Further, the systems and methods refine boundaries between local areas of the new matched image.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority of U.S. Provisional Patent Application No. 62 / 430,709 , filed December 6, 2016, which is incorporated herein by reference in its entirety.

BACKGROUND

Continuous shooting (e.g., taking a plurality of consecutive pictures in a relatively short period of time) is available for various types of image capture devices (e.g., digital cameras, cell phones, etc.). By picking up a plurality of images in a short period of time, a user can conventionally select an image from the burst images having the highest quality or the best timing. In some cases, however, one of the images will be of the highest quality while another will have the best timing. In other cases, none of the pictures has a very good quality. In addition, the continuous images may have deficiencies in exposure and significant blurring.

Some effort has been made to merge (e.g., unify) a plurality of images of the burst images to eliminate noise (i.e., statistical variations in brightness and / or color information) from the burst images or to form a single image without noise. Such conventional systems usually work well only when local motions are consistent (eg, motion is consistent in all bursts), and if this assumption is not true (due to camera deformation or 3D view change), the accuracy of local motion estimation (essential for merging the plurality of images). As a result, the image systems conventionally do not produce higher quality images than the burst images. Additional efforts have been made in terms of image matching (e.g., matching of burst images with each other) and providing a new matched image from the matched image. However, such imaging systems traditionally operate offline and require significant processing power and memory. As a result, the image systems are not suitable for real-time mobile applications (e.g., mobile phone applications).

Accordingly, there are these and other disadvantages with respect to conventional systems.

SHORT SUMMARY

The various embodiments described above provide utility and / or solve one or more of the foregoing or other problems in the art with systems and methods for producing an image of a plurality of enhanced quality continuous frames. In particular, the systems and methods provide for the generation of a frame (e.g., a higher overall quality picture) from the plurality of bursts that captures the instant (i.e., timing) of the burst images while attaining a quality level of typical non-burst images. In particular, the systems and methods provide efficient compensation for translation, rotation, and scaling that can occur in a plurality of burst images, and efficient generation of a new matched image that has higher quality than the quality of each of the individual burst images. In addition, the systems and methods provide for efficient generation of the new matched image on a mobile device, such as a mobile device. a mobile phone.

In one or more embodiments, the systems and methods divide a reference image and a subsequent image of the series images into a plurality of local areas and detect feature points in each of the local areas. Further, the systems and methods tune feature points between the reference image and the subsequent image according to the local regions. On the basis of the matched feature points, the systems and methods determine homography (ie, motion transformation) that enables the systems and methods to warp the subsequent image ("warp") and create a new matched image of higher quality matched pixel by pixel with the reference image. Accordingly, as the systems and methods subdivide the reference image and the subsequent image into local areas and capture feature points based on the local areas, the systems and methods determine more accurate homography compared to conventional systems to a new, higher quality, reconciled image leads.

Additional features and advantages of one or more embodiments of the present disclosure are set forth in the description which follows, and in part will be apparent from the description, or from the practice, of such example embodiments.

list of figures

• 1 ein Ablaufflussdiagramm des Erzeugen eines neuen abgeglichenen Bilds von einer Mehrzahl von Serienbildern gemäß einer oder mehrerer Ausführungsform(en) zeigt;
• 2 ein Referenzbild und ein nachfolgendes Bild einer Mehrzahl von Serienbildern zeigt, wobei das Referenzbild und das nachfolgende Bild eine Mehrzahl von erfassten Merkmalspunkten aufweist, gemäß einer oder mehrerer Ausführungsform(en);
• 3 einen Vergleich von Merkmalspunkten, die mit einem allgemeinen Erfassungssystem erfasst worden sind, und Merkmalspunkten, die mit einem lokalen Bereich-Erfassungssystem erfasst worden sind, gemäß einer oder mehrerer Ausführungsform(en) zeigt;
• 4 eine schematische Darstellung von abgestimmtes Merkmalspunkten zwischen einem Referenzbild und einem nachfolgenden Bild gemäß einer oder mehrerer Ausführungsform(en) zeigt;
• 5 eine schematische Darstellung des Vereinigens von lokalen Bereichen eines Bilds gemäß einer oder mehrerer Ausführungsform(en) der vorliegenden Offenbarung zeigt;
• 6 eine schematische Darstellung von verschiedenen Hierarchieniveaus von lokalen Bereichen innerhalb eines Bilds und dazugehörige Homographien gemäß einer oder mehrerer Ausführungsform(en) der vorliegenden Offenbarung zeigt;
• 7 einen Vergleich von Abgleichfehlern, die durch allgemeines Anwenden eines Homographiebewegungsmodells auf ein Bild erzeugt werden, und Abgleichfehlern, die durch individuelles Anwenden einer Homographie auf jeden lokalen Bereich des Bilds erzeugt werden, gemäß einer oder mehrerer Ausführungsform(en) der vorliegenden Offenbarung zeigt;
• 8A und 8B eine schematische Darstellung des Verfeinerns von Grenzen von lokalen Bereichen innerhalb eines Bilds zum Vermindern der Verformung, wenn Bilder verändert werden, gemäß einer oder mehrerer Ausführungsform(en) der vorliegenden Offenbarung zeigen;
• 9 eine schematische Darstellung von nicht-angrenzenden Eckpunkten eines veränderten Bilds und durchschnittlichen Koordinaten der nicht-angrenzenden Eckpunkte gemäß einer oder mehrerer Ausführungsform(en) der vorliegenden Offenbarung zeigt;
• 10 eine schematische Darstellung einer gemeinsamen Grenze von lokalen Bereichen eines Bilds gemäß einer oder mehrerer Ausführungsform(en) der vorliegenden Offenbarung zeigt;
• 11 ein schematisches Diagramm einer Ausführungsform einer beispielhaften Umgebung, in der ein Bildabgleichsystem gemäß einer oder mehrerer Ausführungsform(en) der vorliegenden Offenbarung arbeiten kann zeigt;
• 12 ein Flussdiagramm eines Beispielverfahrens zum Erzeugen eines neuen abgeglichenen Bilds von einer Mehrzahl von Serienbildern gemäß einer oder mehrerer Ausführungsform(en) der vorliegenden Offenbarung zeigt;
• 13 ein Flussdiagramm eines weiteren Beispielverfahrens zum Erzeugen eines neuen abgeglichenen Bilds von einer Mehrzahl von Serienbildern gemäß einer oder mehrerer Ausführungsform(en) der vorliegenden Offenbarung zeigt;
• 14 ein Flussdiagramm eines weiteren Beispielverfahrens zum Erzeugen eines neuen abgeglichenen Bilds von einer Mehrzahl von Serienbildern gemäß einer oder mehrerer Ausführungsform(en) der vorliegenden Offenbarung zeigt;
• 15 ein Diagramm eines Algorithmus zum Durchführen eines Schritts zum Abstimmen von Merkmalspunkten zwischen einem Referenzbild und dem nachfolgenden Bild gemäß einer oder mehrerer Ausführungsform(en) der vorliegenden Offenbarung zeigt;
• 16 ein Diagramm eines Algorithmus zum Durchführen eines Schritts zum Abstimmen von Merkmalspunkten zwischen einem Referenzbild und dem nachfolgenden Bild gemäß einer oder mehrerer Ausführungsform(en) der vorliegenden Offenbarung zeigt; und
• 17 ein Blockdiagramm einer Beispielrechenvorrichtung gemäß einer oder mehrerer Ausführungsform(en) der vorliegenden Offenbarung zeigt.

Various embodiments are described and illustrated with additional specificity and in more detail by the use of the accompanying drawings, in which:

• 1 10 is a flowchart diagram of generating a new matched image from a plurality of burst images according to one or more embodiments;
• 2 a reference image and a subsequent image of a plurality of series images, the reference image and the subsequent image having a plurality of detected feature points, according to one or more embodiments;
• 3 a comparison of feature points acquired with a general detection system and feature points detected with a local area detection system according to one or more embodiments;
• 4 12 shows a schematic representation of matched feature points between a reference image and a subsequent image according to one or more embodiments;
• 5 FIG. 12 is a schematic illustration of merging local areas of an image according to one or more embodiments of the present disclosure; FIG.
• 6 Figure 12 shows a schematic representation of various hierarchical levels of local areas within an image and associated homographies according to one or more embodiments of the present disclosure;
• 7 FIG. 4 shows a comparison of alignment errors generated by generally applying a homography gesture model to an image, and alignment errors produced by individually applying homography to each local region of the image, according to one or more embodiments of the present disclosure;
• 8A and 8B Figure 12 is a schematic illustration of refining boundaries of local areas within an image to reduce deformation when images are changed, according to one or more embodiments of the present disclosure;
• 9 12 shows a schematic representation of non-contiguous vertices of a modified image and average coordinates of the non-contiguous vertices, in accordance with one or more embodiments of the present disclosure;
• 10 12 shows a schematic representation of a common boundary of local areas of an image according to one or more embodiments of the present disclosure;
• 11 12 is a schematic diagram of one embodiment of an example environment in which an image matching system may operate in accordance with one or more embodiments of the present disclosure;
• 12 FIG. 10 is a flowchart of an example method for generating a new matched image from a plurality of burst images according to one or more embodiments of the present disclosure; FIG.
• 13 FIG. 10 is a flowchart of another example method for generating a new matched image from a plurality of burst images according to one or more embodiments of the present disclosure; FIG.
• 14 FIG. 10 is a flowchart of another example method for generating a new matched image from a plurality of burst images according to one or more embodiments of the present disclosure; FIG.
• 15 10 is a diagram of an algorithm for performing a step of matching feature points between a reference image and the subsequent image according to one or more embodiments of the present disclosure;
• 16 10 is a diagram of an algorithm for performing a step of matching feature points between a reference image and the subsequent image according to one or more embodiments of the present disclosure; and
• 17 FIG. 3 shows a block diagram of a sample computing device according to one or more embodiments of the present disclosure. FIG.

DETAILED DESCRIPTION

The various embodiments described herein provide an image matching system for generating a new matched image from a plurality of burst images. For example, the image matching system may generate a new aligned image from the plurality of burst images that has higher quality (e.g., higher resolution, better exposure, less noise, less blur) than any of the individual burst images of the plurality of burst images. In some embodiments, the image matching system divides images of the plurality of burst images into a plurality of local areas and applies a homography motion model to each local area of the plurality of local areas. In additional embodiments, the image matching system matches key feature points between images of the plurality of burst images according to the respective local region of the feature points. In one or more embodiments, the image matching system uses a hierarchical bottom-up approach to determine the homographies of the images. In further embodiments, the image matching system refines boundaries of the local areas of the images while creating the new matched image to reduce image distortion caused by non-uniform positional motion change. Each of the above will be discussed in more detail below.

In some embodiments, the image matching system divides images of the plurality of burst images into a plurality of local areas and applies a homography motion model to each local area of the plurality of local areas. More specifically, the image matching system divides a reference image and a subsequent image of the plurality of burst images into a plurality of local areas and a plurality of respective corresponding local areas. Further, the image matching system individually applies a homography motion model to each of the local areas and the respective corresponding local areas to determine a homography of each local area of the reference image to its respective corresponding local area of the subsequent image. In addition, as discussed in more detail below, the image matching system generates the new, higher quality image based on the particular homographies.

Since the image matching system divides the reference image and the succeeding image into local areas and corresponding local areas and individually applies homography motion models to each of the local areas (eg, determines a local motion for the local areas), the image matching system of the present disclosure determines the overall motion of the reference image to the following image more precisely (ie, a more accurate homography) compared to conventional systems which conventionally apply a general homo-motion model to all reference images and subsequent images. In particular, the image matching system of the present disclosure determines a more accurate homography for each local area, resulting in a more accurate overall tomography for the entire image.

As noted above, in additional embodiments, the image matching system adjusts feature points between images (i.e., the reference image and the subsequent image) of the plurality of burst images according to the respective local regions of the feature points. In particular, the image matching system detects feature points in each of the local areas and corresponding local areas individually. Further, assuming that a local area includes a threshold number of (e.g., 100) feature points, the image matching system detects the threshold number of feature points in each local area and corresponding local area.

In addition, the image matching system adjusts feature points between a given local area and its corresponding local area by predicting a pixel location of a given feature point within the local area. After predicting the pixel location of the feature point within the local area, the image matching system uses a k-next method to select some of the best potentially matched feature points. Further, the image matching system determines similarities between the given feature point and any potentially matched feature points. In some embodiments, the image matching system determines similarities between feature points based on a Hamming distance between descriptors of the feature points. The image matching system filters any pairs of the given feature point and potentially matched feature points with descriptor intervals above a given extent (eg 80). In addition, the image matching system determines an average displacement vector of all remaining pairs of feature points and filters any pairs of the given feature point and potentially matched feature points with a shift that deviates from the average displacement vector by more than a predetermined number of pixels.

Since the image matching system detects feature points in each of the local areas and corresponding local areas individually, rather than just generally detecting feature points, the image matching system of the present disclosure achieves a more evenly distributed number of detected feature points as compared to conventional systems that typically capture feature points generally. As a result of the foregoing, the image matching system ensures that each area of the images (i.e., the reference image and the subsequent image) is represented in the detected feature points rather than only in areas with higher concentrations of feature points.

Further, because the image matching system tunes feature points from the reference image to feature points of the subsequent image based on local areas, the image matching system of the present disclosure provides fewer matching errors than conventional systems, which typically tune feature points generally. Moreover, as will be apparent to those skilled in the art, the image matching system will eventually produce a new matched image with a higher quality as compared to conventional systems by achieving fewer matching errors.

As mentioned briefly above, the image matching system uses a bottom-up approach to determine the homographies of the images. In particular, the image matching system determines whether a given local area at a lowest hierarchical level (eg, a smallest local area resulting from dividing the images (eg, a 1/16 portion of an image)) the threshold number (eg, a predetermined number) of Feature points to ensure that an applied (eg, fitted) homography motion model provides accurate homography of the local area. If the image matching system determines that the local area does not include the threshold number of (e.g., 100) feature points, the image matching system may merge the given local area with other local areas until a resulting local area includes the threshold number of feature points. After merging the local areas, the image matching system can apply a more stable homography motion model to the unified local area.

Since the image matching system ensures that the local areas to which the image matching system applies homography motion models includes a threshold number of feature points, the image matching system ensures that each portion of the reference image is represented (e.g., considered) within the particular homography of the reference image. For example, the image matching system of the present disclosure, as compared with conventional systems that determine homographies that rely heavily on portions of the reference image (ie, determined) that have many feature points, ensures that each area (ie, local area) of the reference image in the certain homography is shown in the same way. Accordingly, the image matching system of the present disclosure results in less matching error when compared to conventional systems when creating a new aligned image from the plurality of burst images.

Further, since the image matching system of the present disclosure utilizes the bottom-up approach for determining the homographies of the images as described above, the image matching system provides improvements in the performance of a computer system. For example, because the image matching system limits feature matching to local areas and corresponding local areas while assuming relatively small amounts of motion between frames of burst images, the image matching system reduces the required processing power, memory, and communication resources needed to determine a first photograph Image (ie, the reference image) of the plurality of burst images to a second image (ie, the subsequent image) of the plurality of burst images. Further, because the image matching system independently determines homography from each local area, the image matching system reduces the processing power, memory, and communication resources required to determine homography of a first image to a second image as compared to conventional systems that use known common estimation techniques , Accordingly, the content analysis system results in less data transfer and data bandwidth utilization for a computer / communication system. In other words, the image matching system results in less required processing power and communication bandwidth compared with conventional systems. As a result, the image matching system of the present invention discloses Revelation compared with conventional systems, a more suitable system for mobile devices, such as a Mobilfunkelfon.

In addition, the image matching system of the present disclosure, by fusing serial images (ie, the reference image and the succeeding image) in the manner described above, allows additional noise reduction as compared with conventional systems, facilitates the use of high dynamic range ("HDR") with images and motion and / or removing objects within the images.

As briefly stated above, in other embodiments, the image matching system refines boundaries of local areas of the images while creating the new matched image to reduce image distortion caused by non-uniform positional motion change. For example, along common boundaries of local areas of the subsequent image, the image matching system may adjust vector streams of pixels defined by respective homographies that are close to the common boundaries. Further, the image matching system may adjust a position of adjacent corners of the subsequent image to compensate for non-uniform homographies between local areas.

The 1 FIG. 10 is a flowchart diagram showing an image matching system. FIG 100 to create a new matched image 116 from a plurality of burst images according to one or more embodiments of the present disclosure. As shown, the image matching system receives 100 a plurality of burst images from an image pickup device 102 (eg a standalone camera, a camera of a mobile device, etc.). As used herein, the term "burst" may refer to recording multiple (eg, ten, twenty, thirty, forty, etc.) images within a short period of time (eg, 1 second, 2 seconds, 5 seconds, 10 seconds, etc.). ) Respectively. In some embodiments, the image matching system includes 100 the image pickup device 102 and any software associated with the image capture device 102 related.

In response to receiving a plurality of burst images from the image capture device 102 selects the image matching system 100 a reference picture 104 and a subsequent picture 106 from the plurality of burst images to a new matched image 116 to create. For example, the image matching system 100 a first image (ie, the reference image 104 ) within the plurality of burst images and a second image (ie, the subsequent image 106 ) present after the first image (ie, taken after the first image). In some embodiments, the image matching system may 100 the reference picture 104 according to the image quality. For example, the image matching system 100 the image with the highest quality (eg, the highest resolution, the best exposure, the lowest noise, etc.) as a reference image 104 choose. In alternative embodiments, the image matching system 100 selecting a first image in the plurality of burst images (ie, a first image taken with the plurality of burst images) as the reference image. In further embodiments, as described in greater detail below, the image matching system may be used 100 a picture with the most recorded feature points 108 as a reference picture 104 choose. Furthermore, the image matching system 100 Any image that matches the selected reference image 104 follows, as the following picture 106 choose. In addition, the image matching system 100 in some embodiments, an image after the selected reference image 104 with the highest quality than the following picture 106 choose. In one or more embodiments, the reference image may 104 and the following picture 106 at least generally (eg, at least substantially) comprise the same scene.

After selecting the reference image 104 and the following picture 106 subdivides the image matching system 100 the reference picture 104 into a plurality of local areas 105 and the following picture 106 into a plurality of corresponding local areas 107 , In particular, each of the reference image 104 and the subsequent image 106 define a respective image plane and the image matching system 100 can change the picture plane into the local areas 105 (eg different sections) and divide corresponding local areas 107. Furthermore, each local area of the plurality of local areas 105 of the reference picture 104 have a corresponding local area of the plurality of corresponding local areas (eg affecting them). To facilitate the explanation of the image matching system 100 and the operation of the image matching system 100 can the local areas 105 of the reference picture 104 and the corresponding local areas 107 of the following picture 106 collectively referred to herein as "local areas").

In some embodiments, the image matching system may 100 the reference picture 104 and the following picture 106 subdivide by means of a spatial grid. In particular, the image matching system 100 the reference picture 104 and the following picture 106 eg by means of a 2 x 2 spatial grid, a 4 x 4 spatial grid, a 8 x 8 spatial grid, etc. In such embodiments, each of the local areas 105 and the corresponding areas at least generally the same shape and size. In alternative embodiments, the image matching system 100 the reference picture 104 and the following picture 106 divide into irregular areas. For example, the local areas 105 of the reference picture 104 have inconsistent sizes and random shapes.

In addition to dividing the reference image 104 and the subsequent image 106, the image matching system detects feature points 108 in the reference picture 104 and the following picture 106 , For example, the image matching system 100 a plurality of feature points 108 in each of the reference picture 104 and the following picture 106 to capture. In some cases, the image matching system captures 100 a plurality of feature points 108 in each of the local areas 105 the plurality of local areas 105 of the reference image 104 in each of the corresponding local areas 107 the plurality of corresponding local areas 107 of the following picture 106 ,

Additionally, the image matching system extracts (ie, generates) 100 Merkmalsdeskriptoren 110 for each detected feature point 108 of the reference picture 104 and the following picture 106 , For example, the image matching system generates 100 a descriptor vector for each detected feature point 108 , Each descriptor vector includes information (eg, intensity information, pixel position information, local neighborhood information) regarding its respective feature point 108 , For example, each feature descriptor 110 include a numeric "fingerprint" that differentiates a feature point 108 can be used by another. The detection of the feature points 108 and extracting feature descriptors 110 are described below with reference to the 2 and 3 discussed in more detail.

In response to detecting feature points 108 in each of the reference picture 104 and the following picture 106 determines the image matching system 100 matched pairs 112 of feature points 108 between the reference picture 104 and the following picture 106 , In particular, the image matching system determines 100 , which feature points 108 included in the reference picture 108 which feature points 108 correspond to those shown in the following image 106 be recorded. For example, the image matching system determines 100 for a given feature point 108 , which is detected in the reference image 104, a feature point shown in the subsequent image 106 is detected with the given feature point of the reference image 104 correlates (eg indicates the same feature point as the given feature point). In some embodiments, the image matching system matches the detected feature points 108 between the reference picture 104 and the following picture 106 according to the similarities of the feature descriptor 110 the feature points 108 from. The tuning of the feature points 108 between the reference image 104 and the subsequent image 106 will be described below in relation to 4 described in more detail.

After determining the matched pairs 112 of feature points 108 between the reference picture 104 and the following picture 106 determines (eg, estimates) the image matching system 100 at least one homography 114 (eg homography transformation, a motion transformation) of the reference image 104 to the following picture 106 on the basis of matched pairs 112 of feature points 108 between the reference picture 104 and the following picture 106 , As used herein, the term "homography" refers to a transformation that transfers objects and / or lines from one level to another level. For example, the term "homography" refers to a transformation, the feature points 108 from the reference picture 104 (ie, the specified plane of the reference image 104 ) the following picture 106 (ie, the specified level of the subsequent image 106 ).

In one or more embodiments, the image matching system 100 determines the homography 114 by applying (ie, adjusting) a homography gesture model to at least a portion of the reference image 104 and the following picture 106 by means of conventional optimization methods. In some cases, the image matching system 100 a homography gesture model on each of the local areas 105 of the reference picture 104 and the corresponding local areas 107 apply individually to determine a plurality of homographies representing the different sections (ie, local areas) of the reference image 104 the following picture 106 assigns. Furthermore, as will be discussed in more detail below, the image matching system 100 a hierarchical approach from bottom to top when applying a homography gesture model to the local areas 105 and the corresponding local areas 107 use to determine the majority of homographies. The determination of the homography of the reference image 104 to the following picture 106 will be described in more detail below with reference to the 5 to 7 described.

In addition to determining the plurality of homographies of the reference image 104 to the subsequent image 106 provides (eg refined) the image matching system 100 Boundaries (eg, vertices and transitions) between transformed local areas (eg, homographs determined for the local areas) of an original new image (eg, an altered image) that has been generated based on the plurality of homographies. For example, the image matching system 100 after determining the plurality of homographs and transforming the corresponding local areas 107 of the following picture 106 for generating the original new image, refining vertices of the transformed local areas, the adjacent vertices of the subsequent image 106 correspond. In particular, the image matching system 100 the transformed vertices by using an average position (ie, location) of the transformed vertices of the original new image and restoring the adjacent vertices of the subsequent image (ie, the image to be transformed) to that average position and retransforming the subsequent image 106 Refine on the basis of the newly set corner points. The above procedure may apply to any set of adjacent vertices of the subsequent image 106 be repeated.

Furthermore, the image matching system 100 as noted above, refine the boundaries of the corresponding local areas of the subsequent image while the new matched image is being generated. For example, if the image matching system 100 the following picture 106 on the basis of the plurality of homographies to produce the new matched image 116 transformed (ie, changed), the image matching system 100 the borders of local areas of the newly adjusted image 116 by interpolating homography flow vectors of the homographies close to boundaries 1002 present two or more of the corresponding local areas 107 of the following picture 106 are common. In particular, the image matching system 100 determines for each pixel position of the subsequent image 106 within a given distance from a given common boundary 1002 a corresponding local area 107 the plurality of motion vectors based on the determined homographies of the respective local areas 107 that have the given limit in common. Further, the image matching system determines 100 Weightings of each of the pixel positions within the given distance. In particular, for a given pixel position, its weight is inversely proportional to a distance of the given pixel position from the center of a corresponding local area 107 , In addition, forms the image matching system 100 on the basis of the determined plurality of motion vectors and the determined weighting of the pixel position, the mean of the motion vectors with the weighting to determine a new homography flow vector of the pixel position. Setting the boundaries between transformed local areas of the new matched image 116 is described below with reference to 8A to 10 described in more detail.

After determining boundary settings, the image adjustment system changes 100 the following picture 106 according to the predetermined plurality of homographies and the determined settings for creating the new adjusted image 116 (ie, a finished new image). The new matched image may be pixel by pixel with the reference image 104 be matched. As used herein, the term "pixel by pixel" may refer to the newly adjusted image 116 refer to the pixels of the reference picture 104 with the reference picture 104 has been adjusted. For example, if a pixel-by-pixel comparison with the reference image 104 is performed, the pixels of the newly adjusted image 116 adjusted so that they match the pixels of the reference image 104 are adjusted. Furthermore, the image matching system 100 the following picture 106 On the basis of a plurality of homographies and the default settings, the image matching system 100 may display a new matched image 116 with less blur and higher quality than images produced by conventional systems.

The 2 shows an exemplary reference image 104 and an exemplary subsequent image 106 the majority of series images, of which the image matching system 100 feature points of the present disclosure 108 and can extract feature descriptors 110 according to one or more embodiments. As it is in the 2 is shown, each of the reference picture 104 and the following picture 106 through a 2 x 2 spatial grid into four local areas 105 and four corresponding local areas 107 divided. Further, as stated above, the image matching system 100 a plurality of feature points 108a . 108b in each of the local areas 105 of the reference picture 104 and in each of the corresponding local areas 107 of the subsequent image 106.

In some embodiments, the image matching system uses 100 a feature point detector and descriptor for detecting feature points 108 and for determining feature descriptors 110 the detected feature points 108 , In particular, the feature point detector may be an image (eg, the reference image 104 ) and output positions (ie, pixel positions) of significant areas (ie, points of interest) of the image. For example, the feature point detector may detect the locations of corners, edges, ridges, textures, and blobs of objects and / or elements shown in the image. In addition, the descriptor may be the detected feature points 108 and output a descriptor vector for each detected feature point (eg, generate) that encodes information (eg, intensity information, pixel position information, local neighborhood information) regarding its respective feature point. For example, each descriptor may comprise a binary descriptor vector. In some embodiments, each descriptor may include a dimension of 256 for 32-bit binary feature points 108.

In one or more embodiments, the image matching system 100 an "Oriented FAST and Rotated LETTER (" ORB ")" key point detector and descriptor for detecting the feature points 108 and determining the feature descriptors 110 include. In particular, the image matching system 100 use the ORB keypoint detector and descriptor specified in Rublee, Ethan, et al., ORB: An Efficient Alternative to SIFT or SURF, International Conference on Computer Vision, pp. 2564-2571 (2011), the disclosure of which is fully incorporated herein by reference. In additional embodiments, the image matching system 100 may utilize other key-point detectors and descriptors, such as a Scale Invariant Feature Transform (SIFT) keypoint detector, a Speeded-Up Robust Features (SURF) keypoint detector, and / or a Histogram of Oriented Gradient (HOG) "- key point detector ,

As stated above, the image matching system 100 the feature points 108 within each local area 105 of the reference picture 104 and the feature points 108 within each corresponding local area 107 of the following picture 106 to capture. In addition, the image matching system 100 a best or highest predetermined number of feature points 108 with the best response (ie, the highest confidence) from each of the local areas 105 and from each of the corresponding local areas 107 to include as feature points (eg referred to as) 108 choose. Accordingly, by selecting a predetermined number of feature points 108 from every local area 105 and corresponding local area 107 the detected feature points 108 evenly between the local areas 105 of the reference picture 104 distributed and equally between the corresponding local areas 107 of the following image 106 distributed. Examples of given numbers of feature points 108 include 50, 100, 200, 300, or any other number of feature points 108 , For example, the predetermined number of feature points 108 depending on the size, quality and / or resolution of an image. When the image adjustment system 100 not at least the predetermined number (ie, a threshold number) of feature points 108 within a specific local area 105 (or a corresponding local area 107 ), selects the image matching system 100 all feature points 108 out in this specific local area 105 be recorded.

As a non-limiting example, such as the example given in the 2 can be shown, each of the four local areas 15 of the reference picture 104 100 recorded feature points 108 such that the reference image 104 a total of 400 recorded feature points 108 includes. Accordingly, the image matching system 100 make sure that feature points 108 in each of the local areas 105 (and corresponding local areas 107), instead of only feature point-rich areas. As a result, the image matching system 100 Make sure that items and / or items passing through a picture (eg the reference picture 104 ), better by the detected feature points 108 can be represented as by conventional systems that use a single general feature detection instead of detecting feature points within each local area 105 use.

For example, the shows 3 a comparison of feature points 108 (eg objects) detected by a general detection system and by a local detection system (ie, the image matching system 100 of the present disclosure) have been detected (and selected). Remaining feature points 108 that are not detected by the two systems (ie, feature points 108 that are not recorded and / or selected) are in the boxes 202 and 204 shown in gray and / or black. As shown, the general detection system detects significantly fewer feature points 108 as the local detection system described here. In other words, the local acquisition system (ie, the image matching system 100 ) records significantly more feature points 108 as the general detection system.

As briefly mentioned above, the tuning of the feature points becomes 108 between the reference picture 104 and the following picture 106 more detailed in the 4 described. In particular, the shows 4 a local area 105 of the reference picture 104 and a corresponding local area 107 of the subsequent image 106. To the feature points 108 between the reference picture 104 and the subsequent image 106 selects the image matching system 100 a detected feature point R in the reference image 104 and matches the feature point R with the most similar image in the subsequent image 106 from. In some embodiments, the image matching system 100 determines the similarity of two given feature points 108 by evaluating the Hamming distance between the descriptors 110 the feature points 108 , As is known, the Hamming distance is between two vectors (eg the descriptors 110 ) then the number of corresponding positions (eg values) of the two vectors that are different. For example, the Hamming distance measures a minimum number of substitutions that would be required to change a first vector from two vectors to a second vector of the second vectors. As one skilled in the art will appreciate, the Hamming distance measures a minimum number of errors that may have been transferred to the second vector by the first vector.

In one or more embodiments, the image matching system 100 determines for a given feature point in the reference image 104 (eg the feature point R) a corresponding position in the following image 106 , Further, the image matching system searches 100 only one respective corresponding local area 107 of the following picture 106 for a feature point P corresponding to the given feature point R of the local area 105 of the reference picture 104 is tuned, as it is in the 4 is shown. For example, the image matching system says 100 for a given feature point R with pixel position X _R, advance the pixel position of the matched feature point as X _R + dX _R. In some embodiments, dX _{R is} a previous motion offset detected by a motion sensor (eg, a motion sensor of the image capture device 102 ) has been recorded. In additional embodiments, dX _{R is} a previous motion displacement of previous trim results (eg, previously created new images) that has been estimated by an autoregressive model. For example, the previous motion displacement is regression-based on an earlier previous motion displacement due to analysis, with additional subsequent forming and the reference image 104 has been carried out.

Based on the pixel position of the predicted matched feature point (X _R + dX _R ), the image matching system compares 100 the feature descriptor 110 of the feature point R with the feature descriptor 110 from any feature point P with a pixel position X _P close to X _R + dX _R. For example, the image matching system 100 may include the feature descriptor 110 of the feature point R with the feature descriptor 110 of any potential feature points {P} (ie, any feature point of a set of potentially matched feature points 108 ) with a pixel position that is within a predetermined number of pixels (eg, 50 pixels, 75 pixels, 100 pixels, 200 pixels, etc.) of X _R + dX _R. For example, the image matching system 100 the feature descriptor 100 of the feature point R with the feature descriptor 110 of any potential feature points {P} within a circle extending around X _R + dX _R , where X _R + dX _{R is} the center of the circle and the circle has a radius with a given number of pixels.

In addition to comparing the feature descriptor 110 of the feature point R with the feature descriptor 110 of any potential feature points {P} (ie, sorts and / or prioritizes) the image matching system 100 the potential feature points {P} according to the similarities of feature descriptors 110 of potential feature points {P} with the feature descriptor 110 of the feature point R. As mentioned briefly above, the image matching system determines 100 in some embodiments, the similarity of two given feature points 108 by evaluating the Hamming distance between the feature descriptors 110 the feature points 108 ,

After ranking the potential feature points {P} according to their similarities with the feature point R, the image matching system filters 100 all but a predetermined best number (ie, a predetermined number) of potential feature points {P} for consideration as a matched feature point. For example, the image matching system 100 filter all but the best 3, 6, or 10 potential feature points {P} most similar to the feature point R. In other words, the image matching system 100 maintains the best potential feature points {P} considering a vote with the feature point R. For example, the image matching system retains 100 a set of potentially matched feature points 108 taking into account that there is an attunement to the feature point R. For example, the image matching system retains 100 a set of potentially matched feature points 108 taking into account a vote with the feature point R at. The Image matching system 100 retains more than the best potential feature point P since the best match could still be a wrong vote and is filtered (eg removed) for consideration as a potential feature point in later processes. Furthermore, maintaining more than the best match will result for any given feature point R of the reference image 104 to more feature points 108 of the reference picture 104 correctly with the potential feature points {P} of the following image 106 are coordinated. For example, the image matching system determines 100 more matched pairs of feature points 108 if more than one top potential feature point is preserved.

In response to the best potential feature points {P} for feature point R, the image matching system filters 100 (from the set of potentially matched feature points 108 ) any potential feature points {P} with a descriptor distance above a threshold distance from the feature point R. In some embodiments, the descriptor distance is the Hamming distance between the feature descriptor 110 of the feature point R and a feature descriptor of a given potential feature point P. In additional embodiments, the descriptor distance is a Euclidean distance feature descriptor 110 of the feature point R and a given feature point P. In some cases, after determining the descriptor distances between the feature point R and the potential feature points {P}, the image matching system filters (eg removes) 100 from consideration of any potential feature points {P} (of the set of potentially matched feature points 108 ) having a descriptor distance from the above feature point R, such as 50, 80, 100, 150 or 200.

In addition to filtering potential feature points {P} based on descriptor spacing, the image matching system filters potential feature points (P) from consideration based on a displacement (ie, X _P - X _R ) of the potentially matched pair 112 of feature points 108 , In particular, the image matching system determines 100 from all remaining potential feature points {P} (the set of potentially matched feature points 108 ), for all recorded feature points 108 {R} included in the reference image 104 (eg of all potential matched pairs 112 of feature points between the reference image 104 and the following picture 106 ) an averaged displacement vector D as the mean value of the feature displacement {X _P - X _R }. As one skilled in the art will appreciate, the mean displacement vector D is a stable approximation of a general translation between the reference image 104 and the following picture 106 , After determining the average displacement vector D, the image matching system filters 100 any potential feature points {P}, the potentially matched pairs 112 with displacements (X _P - X _R ) that deviate from the average displacement vector by more than a predetermined number of pixels. For example, the image matching system 100 any potential feature points {P}, the potentially matched pairs 112 with displacements (X _P - X _R ), which deviate from the average displacement vector D by more than 10, 20, 30, 40, 50 or 100 pixels.

In some cases, after filtering the potential feature points {P} based on the predicted pixel position, the descriptor distances, and the relative displacements, a plurality of potentially matched pairs may be used 112 feature points 108 for the same feature point R remain. In such cases, the image matching system 100 selects a potential feature point P having the greatest similarity with the feature point R. The resulting pair (R, P) is the finely tuned feature point pair 112 which, as will be discussed in more detail below, the image matching system 100 for determining a motion estimate (ie, a homography) of the reference picture 104 to the following picture 106 uses. Furthermore, the image matching system 100 the methods described above for each detected feature point 108 of the reference picture 104 for determining a plurality of matched pairs 112 of feature points 108 between the reference picture 104 and the following picture 106 carry out.

As briefly mentioned above, determining the homography of the reference image becomes 104 to the following picture 106 with reference to the following 5 and 6 described in more detail. Furthermore, in one or more embodiments, the actions related to the 5 and 6 are shown and described within a step of determining a homography for each local area 105 the majority of local areas 105 be performed. As described above, the image matching system uses 100 then, if the homography of the reference picture 104 to the following picture 106 is determined, a Homografiebewegungsmodell on each of the local areas 105 of the reference picture 104 at. In particular, the image matching system 100 a conventional homography gesture model on each local area 105 apply while using conventional optimization techniques. For example, the image matching system 100 apply the homography motion model and conventional optimization techniques described in Liu, Ziwei et al., Fast Burst Images Denoising, ACM Transaction on Graphics 33, No 6: 232 (2014) , the disclosure of which is fully incorporated herein by reference.

Furthermore, the operation of the image matching system 100 in which a conventional homography movement model is applied to each local area 105 the majority of local areas 105 is applied to algorithms, equations, or pseudocode performed by a computing device (eg, a server). In particular, the application of a conventional homography motion model to any local area 105 include the following algorithms and equations.

was ein direktes Mapping bzw. Zuordnen zwischen Punkten in zwei Bildebenen darstellt. Ferner nutzt das Bildabgleichsystem 100 bei der Verwendung von homogenen Koordinaten die folgende Beschränkung: $$\left[\begin{array}{c}{x}_2\\ y_2\\ z_2\end{array}\right]=\left[\begin{array}{ccc}{H}_{11}& H_{12}& H_{13}\\ H_{21}& H_{22}& H_{23}\\ H_{31}& H_{32}& H_{33}\end{array}\right]\left[\begin{array}{c}{x}_1\\ y_1\\ z_1\end{array}\right]\iff {\text{x}}_2=H{\text{x}}_1$$

For estimating a homography H, the image matching system starts 100 with the following: $${\text{x}}_2~\mathrm{<figure-callout\; id="1"\; label="H\; x"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">H\; </figure-callout>}{\text{x}}_{}{}_1$$

which is a direct mapping or mapping between points in two image layers. Furthermore, the image matching system uses 100 the following restriction applies when using homogeneous coordinates: $$\left[\begin{array}{c}{x}_2\\ y_2\\ z_2\end{array}\right]=\left[\begin{array}{ccc}{H}_{11}& H_{12}& H_{13}\\ H_{21}& H_{22}& H_{23}\\ H_{31}& H_{32}& H_{33}\end{array}\right]\left[\begin{array}{c}{x}_1\\ y_1\\ z_1\end{array}\right]\iff {\text{x}}_2=\mathrm{<figure-callout\; id="1"\; label="H\; x"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">H\; </figure-callout>}{\text{x}}_{}{}_1$$

nutzt das Bildabgleichsystem 100 die folgenden Beschränkungen: $$x_2^{\mathrm{\prime }}=\frac{H_{11}{x}_1+H_{12}{y}_1+H_{13}{z}_1}{H_{31}{x}_1+H_{32}{y}_1+H_{33}{z}_1}$$

$$y_2^{\mathrm{\prime }}=\frac{H_{21}{x}_1+H_{22}{y}_1+H_{23}{z}_1}{H_{31}{x}_1+H_{32}{y}_1+H_{33}{z}_1}$$

If inhomogeneous coordinates are used, such as $\left(x_2^{\mathrm{\text{'}}}=x_2\text{/}{\mathrm{<figure-callout\; id="2"\; label="z"\; filenames="DE102017009118A1\_0029.png"\; state="\{\{state\}\}">z</figure-callout>}}_2\text{and}{y}_{\text{2}}^{\mathrm{\text{'}}}={\mathrm{<figure-callout\; id="2"\; label="y"\; filenames="DE102017009118A1\_0029.png"\; state="\{\{state\}\}">y</figure-callout>}}_2\text{/}{z}_2\right)\text{.}$

uses the image matching system 100 the following restrictions: $$x_2^{\mathrm{\text{'}}}=\frac{H_{11}{x}_1+H_{12}{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">y</figure-callout>}}_1+H_{13}{z}_1}{H_{31}{x}_1+H_{32}{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">y</figure-callout>}}_1+H_{33}{\mathrm{<figure-callout\; id="1"\; label="z"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">z</figure-callout>}}_1}$$

$$y_2^{\mathrm{\text{'}}}=\frac{H_{21}{x}_1+H_{22}{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">y</figure-callout>}}_1+H_{23}{z}_1}{H_{31}{x}_1+H_{32}{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">y</figure-callout>}}_1+H_{33}{\mathrm{<figure-callout\; id="1"\; label="z"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">z</figure-callout>}}_1}$$

$$y_2^{\mathrm{\prime }}\left(H_{31}{x}_1+H_{32}{y}_1+H_{33}\right)=H_{21}{x}_1+H_{122}{y}_1+H_{23}$$

In addition, the image matching system defines 100 without loss of generality z ₁ = 1 and converts the above equation to: $$x_2^{\mathrm{\text{'}}}\left(H_{31}{x}_1+H_{32}{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">y</figure-callout>}}_1+H_{33}\right)=H_{11}{x}_1+H_{12}{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">y</figure-callout>}}_1+H_{13}$$

$$y_2^{\mathrm{\text{'}}}\left(H_{31}{x}_1+H_{32}{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">y</figure-callout>}}_1+H_{33}\right)=H_{21}{x}_1+H_{122}{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">y</figure-callout>}}_1+H_{23}$$

$${\text{a}}_y^T\text{h}=0$$

wobei $$\text{h}={\left(H_{11},H_{12},H_{13},H_{21},H_{22},H_{23},H_{31},H_{32},H_{33}\right)}^T$$

$${\text{a}}_x={\left(-x_1,-y_1,-\mathrm{1,0,0,0,}{x}_2^{\mathrm{\prime }}{x}_1,x_2^{\mathrm{\prime }}{y}_1,x_2^{\mathrm{\prime }}\right)}^T$$

$${\text{a}}_y={\left(\mathrm{0,0,0,}-x_1,-y_1,-\mathrm{1,}{y}_2^{\mathrm{\prime }}{x}_1,y_2^{\mathrm{\prime }}{y}_1,y_2^{\mathrm{\prime }}\right)}^T$$

Although the above inhomogeneous equations include the coordinates non-linearly, the coefficients of H appear linear. Accordingly, in order to resolve to H, the image matching system provides 100 the above two equations um as: $${\text{a}}_x^T\text{H}=0$$

$${\text{a}}_y^T\text{H}=0$$

in which $$\text{H}={\left(H_{11}.H_{12}.H_{13}.H_{21}.H_{22}.H_{23}.H_{31}.H_{32}.H_{33}\right)}^T$$

$${\text{a}}_x={\left(-{\mathrm{<figure-callout\; id="1"\; label="x"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">x</figure-callout>}}_1.-{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">y</figure-callout>}}_1.-\mathrm{1,0,0,0,}{x}_2^{\mathrm{\text{'}}}{\mathrm{<figure-callout\; id="1"\; label="x"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">x</figure-callout>}}_1.x_2^{\mathrm{\text{'}}}{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">y</figure-callout>}}_1.x_2^{\mathrm{\text{'}}}\right)}^T$$

$${\text{a}}_y={\left(\mathrm{0,0,0,}-{\mathrm{<figure-callout\; id="1"\; label="x"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">x</figure-callout>}}_1.-{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">y</figure-callout>}}_1.-\mathrm{1,}{y}_2^{\mathrm{\text{'}}}{\mathrm{<figure-callout\; id="1"\; label="x"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">x</figure-callout>}}_1.y_2^{\mathrm{\text{'}}}{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png"\; state="\{\{state\}\}">y</figure-callout>}}_1.y_2^{\mathrm{\text{'}}}\right)}^T$$

wobei $$A=\left(\begin{array}{c}{\text{a}}_{x1}^T\\ {\text{a}}_{y1}^T\\ ⋮\\ {\text{a}}_{xN}^T\\ {\text{a}}_{yN}^T\end{array}\right)$$

Furthermore, the image matching system 100 based on the specific feature points agreed 108 (ie, correspondences) referred to above 4 which form the following linear system of equations: $$A\text{H}=0$$

in which $$A=\left(\begin{array}{c}{\text{a}}_{x1}^T\\ {\text{a}}_{y1}^T\\ ⋮\\ {\text{a}}_{xN}^T\\ {\text{a}}_{yN}^T\end{array}\right)$$

The image matching system 100 can solve the aforementioned equations by means of homogeneous linear least squares, which is known in the art. Accordingly, the image matching system 100 in some embodiments, the aforementioned equations for determining the homologies of the local regions 105 use.

Still referring to the 5 can the image matching system 100 in some embodiments, prior to applying the homography motion model described above, determine whether a given local area 105 of the reerenzbilds 104 a threshold number (ie, a predetermined number) of detected feature points 108 includes. The threshold number may be 50, 75, 100, 200, 500, 1000, or any other number of feature points detected 108 include. For example, the threshold number of detected feature points 108 depending on the size, quality and / or resolution of an image.

Furthermore, the image matching system 100 depending on whether a given local area 105 the threshold number of detected feature points 108 involves applying a hierarchical estimation of the homography flow from bottom to top. For example, if the image matching system 100 determines that a given local area 105 the threshold number of detected feature points 108 includes, uses the image matching system 100 the homography movement model on the given local area 105 as described above. On the other hand, if the image matching system 100 determines that a given local area 105 the threshold number of detected feature points 108 not included, unites the image matching system 100 the local area 105 with a further local area with the same hierarchy level and within a local area (ie, a larger local area of a next hierarchy level) and applies the homography movement model to the unified local area 5 the reference picture 104 as it is originally divided into a plurality of local areas, being a local area 105a and a local area 105b they may be near each other (ie, sharing a boundary) and, in some embodiments, may not have a significant number of feature points 108 include. Accordingly, after determining that one or more of the local areas 105a and 105b not including the threshold number (ie, the predetermined number) of feature points, the image matching system 100 the local area 105a and the local area 105b for the formation of the local area 105g unite. As shown, however, the image matching system unites 100 the local area 105a and the local area 105b within the local area 505 (eg the next hierarchy level).

In some embodiments, the hierarchy levels are determined by local sets of quadrants of the local regions. For example, as it is in the 5 shown is local regions 105c . 105d . 105e and 105f within the same set of quadrants ( 2 x 2 set of quadrants) and are therefore within the same hierarchy level. The next hierarchy level (ie, the hierarchy level above the local regions 105c , 105d, 105e and 105f) is the local region (and associated local regions) that is divided into the set of quadrants that are the local regions 105c . 105d . 105e and 105f defines, such as the local area 507 , Should the image matching system 100 determine one or more of the local regions 105c . 105d . 105e and 105f the threshold number of detected feature points 108 not included, unites the image matching system 100 the missing local area with another of the local areas within the same set of quadrants. For example, the image matching system unites 100 the missing local area has in common another local area of the same hierarchy level and the same next level of hierarchy (eg another local area that is within the local area 507 located). For example, the image matching system would 100 the missing local area not with the local area 105c unite since the local area 105c does not have the same hierarchical level in common. In addition, the image matching system determines 100 after uniting the local areas 105c . 105d . 105e and 105f that the united local area (now the local area 507 ) not the threshold number of feature points 108 includes, the image matching system unites the local area 507 with another local area (eg the local area 505 ) of the same hierarchical level and the same hierarchical level in common (eg the whole picture in this case).

In one or more embodiments, the image matching system 100 after determining that a given local area is not the threshold number of feature points 108 includes the local area with other local areas until the resulting pooled local area includes the threshold number of feature points 108 includes. In some embodiments, the image matching system may 100 combine the local areas for up to three levels of hierarchy, eg from one-sixteenth of the image to one quadrant of the image to the entire image. In addition, the image matching system 100 in one or more embodiments, the local area that represents the threshold number of feature points 108 does not have, with another local area of the same hierarchy level with the least number of detected feature points 108 unite. In particular, the image matching system 100 the local area, which is the threshold number of feature points 108 does not merge with other local areas of the same hierarchical level in the order of the number of detected feature points in each of the other local areas, beginning with the local area having the least number of detected feature points 108 , Hierarchy levels are detailed below in relation to 6 described.

festgelegt werden. Darüber hinaus kann, wie es gezeigt ist, die Homographie $H_i^l$

die Homographieflüsse des lokalen Bereichs (i) bei dem niedrigsten Niveau (l) darstellen.In particular, the shows 6 another illustration of hierarchy levels according to one or more embodiments. As it is shown, the image matching system 100 a reference picture 104 divide into four quadrants and can divide each quadrant into four quadrants, which results in the reference image 104 is divided into sixteen local areas. Furthermore, the reference image includes 104 by having sixteen local areas, at least three levels of hierarchy. The lowest level (ie, the lower level) (l) is the level at each individual local area of the sixteen local areas and represents a first hierarchical level. Additionally, the image matching system takes into account 100 at the lowest level (L), each of the sixteen local areas individually. Further, as shown, at the lowest level (1), homography of the local area (i) at the lowest level (1) may be considered $H_i^l$

be determined. In addition, as shown, homography $H_i^l$

represent the homographic fluxes of the local region (i) at the lowest level (l).

festgelegt werden. Ferner vereinigt dann, wie es vorstehend diskutiert worden ist, wenn das Bildabgleichsystem 100 einen der lokalen Bereiche auf dem ersten Hierarchieniveau mit einem benachbarten lokalen Bereich vereinigt (z.B. einem lokalen Bereich, der eine Grenze gemeinsam hat), das Bildabgleichsystem 100 den lokalen Bereich mit dem benachbarten lokalen Bereich innerhalb dessen jeweiligen Quadranten auf dem zweiten Hierarchieniveau.The first level up (l-1) from the lowest level (l) is the level at the four quadrants of the overall image and represents a second hierarchical level. Homography of a local area (ie, a quadrant) at the first level upwards (l-1) can be considered as $H_i^{l-1}$

be determined. Further, as discussed above, if the image matching system then merges 100 merges one of the local areas at the first hierarchy level with an adjacent local area (eg, a local area that has a boundary in common), the image matching system 100 the local area with the adjacent local area within its respective quadrants at the second hierarchical level.

festgelegt werden. Ferner vereinigt dann, wie es vorstehend diskutiert worden ist, wenn das Bildabgleichsystem 100 einen der lokalen Bereiche auf dem zweiten Hierarchieniveau mit einem benachbarten lokalen Bereich vereinigt (z.B. einem lokalen Bereich, der eine Grenze gemeinsam hat), das Bildabgleichsystem 100 den lokalen Bereich mit dem benachbarten lokalen Bereich innerhalb dessen dritten Hierarchieniveau (z.B. das gesamte Bild in der gezeigten Ausführungsform).The second level up (l-2) from the lowest level (l) is the level that covers the entire reference image 104 includes and represents a third hierarchical level. A homography of the entire reference image 104 at the second level up (l-2) can be considered as $H_i^{l-2}$

be determined. Further, as discussed above, if the image matching system then merges 100 one of the local areas on the second hierarchy level associated with an adjacent local area (eg, a local area having a boundary in common), the image matching system 100 the local area with the neighboring local area within its third hierarchical level (eg, the entire image in the illustrated embodiment).

Although regarding 6 If only three levels of hierarchy are described, it will be apparent to one of ordinary skill in the art that any number of hierarchy levels are covered by the image matching system 100 of the present disclosure depending on the size and quality of the reference image 104 could be used. For example, the image matching system 100 could use four, five, ten, or twenty levels of hierarchy.

With reference to the 5 and 6 together, the image matching system 100 may be individualized in view of the foregoing, such as the image matching system 100 a homography gesture model on each area of the reference image 104 applies. In particular, as in the embodiment in the 5 shown is the image matching system 100 the local areas 105c . 105d . 105 and 105f up to the quadrant hierarchy level (eg the second hierarchy level) to the local area 105m form so that the threshold number of feature points 108 is included. In addition, the image matching system 100 the local areas 105a and 105b join together to the local area 105g to form one half of a quadrant of the reference image 104 is. Further, as shown, the image matching system 100 may be the local area 105n and 105o unite to the local area 105r to form an upper half of a quadrant of the reference image 104 is. Accordingly, the image matching system 100 the local areas 105p and 105q unite to the local area 105s to form a lower half of a quadrant of the reference image 104 is. In addition, as shown, the image matching system 100 the local areas 105h , 105i, 105j and 105k do not unite since the local areas 105h . 105i . 105j nd 105k the threshold number of feature points 108 Include without unification. In some embodiments, the image matching system may 100 apply a more stable (eg, more powerful) motion model to unified local areas that is significantly more than the threshold number of feature points detected 108 includes. Accordingly, the image matching system 100 tailor how homography models are applied to the reference image 104 on the basis of where the feature points 108 (eg concentrations of feature points 108 ) within the reference image 104 are located. As a result, the image matching system 100 Compared to conventional image systems using known related optimization techniques, computationally more efficiently results in altered images (ie, the new aligned image 116 ) with a lower balance adjustment error.

For example, the shows 7 a comparison of the residual alignment error produced by using a general (ie, single) homography and using local homographies pooled according to the hierarchy levels described above with respect to FIG 5 and 6 are described. The balance adjustment error (ie, error in adjusting / changing images) generated by the conventional systems and the image matching system 100 are in the boxes 702 respectively. 704 shown in gray and / or black. As shown, conventional image systems produced significantly more alignment errors than the image matching system 100 of the present disclosure.

Although the above regarding the 5 to 7 described processes or procedures with respect to the reference image 104 It is clear that the same processes or procedures in relation to the following image 106 can be performed. Moreover, in one or more embodiments, the actions related to the 5 to 7 are shown and described as a step for determining a homography for each local area 105 the majority of local areas 105 be executed.

As briefly discussed above, the boundaries between transformed local areas of the altered image with respect to the 8A to 10 described in more detail. For example, the shows 8A an image deformation 802 within a new reconciled image (ie, the new reconciled image 116 ), which may be caused by a nonuniform change in positional motion (eg, when adjacent homographies have different (eg, significantly different) homographic fluxes). The 8B shows the re-adjusted image after adjusting the boundaries between transformed local areas by the image matching system 100 , In one or more embodiments, since the homographies of the local regions are independently determined, as described above with respect to FIG 5 to 7 described, the homographies (ie, the estimated motion) to be uneven around boundaries of the local areas. As a result, an image (ie, the most recent matched image 116 ), which has been generated from the homographies, be uneven at the boundaries of the local areas. In some Embodiments reduces and / or eliminates the image matching system 100 Inconsistencies in the altered image by refining vertices of the transformed local regions. In additional embodiments, the image matching system reduces and / or eliminates 100 Inconsistencies in the altered image by refining the common boundaries of the transformed local areas. Each of the above will be described in more detail below.

As stated above, the image matching system refines 100 Corner points of the transformed local areas. The 9 shows an example embodiment of an originally altered image 900 with transformed vertices 902a . 902b . 902c , 902d (hereinafter collectively referred to as "902") that are not aligned after transformation (eg, not adjacent). In operation, the image adjustment system determines 100 after the original transform of the corresponding local areas 107 of the following picture 106 according to the determined homographies, whether the transformed vertices 902 the transformed local areas, the adjacent vertices of the corresponding local area 107 of the following picture 106 correspond within the altered image are continuous. When the image adjustment system 100 determines that the transformed vertices 902 are not contiguous, the image matching system determines 100 the coordinates (eg the pixel position) of each of the transformed vertices 902 within the original altered image 900 , Further, the image matching system determines 100 based on the coordinates of each of the transformed vertices 902 the average coordinates 904 the transformed vertices 902 within the original altered image 900 ,

After determining the original coordinates 904 the transformed vertices 902 sets the image matching system 100 the coordinates of the adjacent vertices of the following image 106 (ie, the image that has been transformed) to the average coordinates 904 back. The image matching system 100 repeats the process described above with respect to 9 has been described for all vertices of the corresponding local areas 107 of the following picture 106 that is adjacent to another vertex (eg vertices that have two or more corresponding local areas 107 have in common). It also applies after resetting the coordinates of all vertices of the subsequent image 106 the image matching system 100 the homography gesture model again on each corresponding local area 107 of the following picture 106 based on the reset vertices (eg, re-transforms) to the new matched image 116 to generate (eg the newly adjusted image 116 ), the pixel power to the reference image 104 is adjusted.

The 10 shows a partial subsequent image 106 with two corresponding local areas 107 , wherein each of the two local areas has a predetermined homography (H ₁ and H ₂ ). As mentioned above, to reduce and / or eliminate inconsistencies in the altered image, the image matching system 100 refines the common boundaries of the transformed local areas. Especially if the image matching system 100 the following picture 106 on the basis of the plurality of homographies to produce the new matched image (eg, the new matched image 116 ( 1 )) (ie, changed), the image matching system 100 the common boundaries 1002 of the corresponding local areas of the subsequent image 106 by interpolating homography flow vectors of the homographies close to the common boundaries 1002 lie, refine. For example, the image matching system 100 may transform the subsequent image 106 close to the common borders 1002 the corresponding local areas 107 Refine. In particular, the image matching system determines 100 for each pixel position of the subsequent image 106 within a given distance (d) from a given common boundary 1002 a plurality of motion vectors based on the predetermined homographies (H ₁ and H ₂ ) of the respective local areas which correspond to the given common boundary 1002 have in common. Further, the image matching system determines 100 Weightings of each of the pixel positions within the predetermined distance (d). In particular, for a given pixel position (P ₁ ), its weights are inversely proportional to distances (w ₁ and w ₂ ) of the given pixel position of respective centers 1004 and 1006 from local areas. In addition, the image matching system averages 100 on the basis of the determined plurality of motion vectors and the determined weighting of the pixel position, the motion vectors having the weight for determining a new homography flow vector (H _P ) of the pixel position (P ₁ ). For example, the new homography flow vector (H _P ) of the pixel position (P ₁ ) can be set as follows: $$H_P={\text{<figure-callout id="1" label="w" filenames="DE102017009118A1\_0020.png,DE102017009118A1\_0029.png" state="{{state}}">w</figure-callout>}}_1\times {\text{P}}_1\left(H_1\right)+{\text{<figure-callout id="2" label="w" filenames="DE102017009118A1\_0029.png" state="{{state}}">w</figure-callout>}}_2\times {\text{P}}_1\left(H_2\right)$$

Furthermore, the image matching system 100 new homography flow vectors for each of the pixel positions within the predetermined distance (d) of common boundaries 1002 the corresponding local areas of the following picture 106 determine. On the basis of the determined new homography flow vectors for each of the pixel positions within the predetermined distance (d) of the common boundaries 1002 the corresponding local areas of the subsequent image, the image matching system 100 the following picture 106 change, so that's new-oriented picture 116 (eg the new aligned image 116 ) is generated with refined boundaries.

The 11 shows a schematic diagram of an embodiment of an example environment 1100 in which the image matching system 100 can be operated. In one or more embodiments, the example environment includes 1100 one or more client device (s) 1102 , a network 1106 and (a) server 1104 , The client device 1102 and the server (s) 1104 stand with the network 1106 in connection. The network 1106 may be any suitable network over which the computing devices can communicate. Examples of networks are given with reference to 17 discussed in more detail. Although the 11 a special arrangement of the client device 1102 , the server 1104 and the network 1106 shows, several additional applications are possible. For example, the server 1104 directly with the client device 1102 communicate, being the network 1106 is bypassed.

Both the client device 1102 as well as the server 1104 may represent various types of computing devices that the user may interact with. For example, the client device 1102 a mobile device (eg, a mobile phone, a smartphone, a PDA, a tablet, a laptop, a watch, a portable device, etc.). In some embodiments, the client device may 1102 however, be a non-mobile device (eg, a desktop or a server). In addition, the server can 1104 be any type of server computing device. In one or more embodiments, the image matching system operates 100 on the client device 102 , Accordingly, the image matching system 100 either perform an online or offline workflow according to the principles described below. Furthermore, the client device comprises 1102 an image pickup device 102 that the image matching system 100 Can take continuous pictures. In alternative embodiments, as shown, the image matching system 100 on the server (s) 1104 are operated. Additional details regarding the client device 1102 and the server 1104 will be discussed below with regard to 17 discussed.

As indicated above, the server may or may not 1104 the entire image matching system 100 or part of it. In particular, the image matching system 100 an application running on the server (s) 114 or comprising part of a software application that may be downloaded from the server (s) 1104. For example, the image matching system 100 include a web hosting application that allows the client devices 1102 allows to interact with content hosted on the server (s) 1104. For example, in one or more embodiments of the example environment 1100 the client device 1102 to access a webpage provided by the server (s) 1104 is supported. In particular, the client device 1102 run an application that allows a user to access, view, and / or interact with a web page or web site hosted on the server (s) 1104.

In one or more embodiments, the server (s) includes or includes 1104 an image processing system, including but not limited ADOBE PHOTOSHOP ^{® ®} software, ADOBE ^® INDESIGN ^® or Adobe ^® Lightroom ^® software. The image processing system may rely on the image matching system 100 access or include this.

Accordingly, although the environment 1100 from 11 The environment can be shown as having different components 1100 have additional or alternative embodiments. For example, the image matching system 100 be implemented on a single computing device. In particular, the image matching system 100 as a whole by the client device 1102 be implemented or the image matching system 100 can be as a whole by the server or servers 1104 be implemented. Alternatively, the image matching system 100 be implemented over several devices or components (eg by means of the client device 1102 and the server (s) 1104 ).

For example, the client device 1102 in one or more embodiments, a request to the server (s) 1104 for matching an image of a plurality of burst images. The server or servers 1104 may or may be the client device 1102 an access to the image matching system 100 provide. In response to the request, the image matching system 100 the new balanced image 116 (eg the new matched image 116 ), as stated above with respect to 1 to 10 has been discussed, and the server (s) 1104 can or can the new matched image 116 to the client device 1102 send.

The 12 shows a flowchart of an example method 1200 to create a new matched image 116 from a plurality of series images. The procedure 1200 can by the image matching system described above 100 be implemented. The procedure 1200 includes a process 1210 subdividing a reference image 104 and a subsequent image 106 in a plurality of local areas. In addition, the process can 1210 the subdivision of the reference image 104 in at least four local areas and the following image 106 in at least four corresponding local areas. Furthermore, the process can 1210 the subdivision of the reference image 104 and the following picture 106 by means of a 2 × 2 spatial grid. Furthermore, the process can 1210 the subdivision of the reference image 104 and the following picture 106 by means of a 4 × 4 spatial grid. In addition, the process can 1210 any of those in relation to the 5 and 6 include operations described.

The procedure 1200 may also be a process 1220 of detecting feature points 108 within the reference image 104 and the following picture 106 include. In particular, the process can 1220 the detection of a plurality of feature points 108 in each of the local areas of the plurality of local areas of both the reference image 104 as well as the following picture 106 include. In some cases, the process can 1220 the detection of a plurality of feature points 108 by means of an aligned FAST and rotated MAIL feature point detector and descriptor for detecting the plurality of feature points 108 include. Further, act 1220 may include capturing best 100 feature points with the highest confidence values of each of the local regions of the plurality of local regions. In addition, the process can 1200 include any of the actions described above with respect to FI.

In addition, the process includes 1200 a process 1230 determining matched pairs 112 of feature points 108 , In particular, act 1230 may include determining matched pairs 112 of feature points 108 between the reference picture 104 and the following picture 106 include. In some embodiments, the process may 1230 Feature points that are most similar to one another, based at least in part on a Hamming distance between the feature descriptors 110 the feature points 108 determine. In addition, the process can 1230 include any of the actions described above relating to 2 to 4 have been described.

In addition, the procedure includes 1200 a process 1240 determining at least one homography of the reference image 104 and the following picture 106 , For example, the process can 1240 based on the matched pairs 112 of feature points 108 performing a step of determining a homography for each local region of the plurality of local regions, wherein the homography of each local region comprises feature points 108 from a local area of the reference image 104 to a corresponding local area of the subsequent image 106 assigns. In addition, the process can 1240 include any of the actions described above in relation to the steps in the 5 and 6 have been described.

Furthermore, the method comprises 1200 a process 1250 changing the subsequent image 106 , For example, the process can 1250 changing the subsequent image 106 based on the homography of each local area of the plurality of local areas to produce a new adjusted image 116 include the pixel values with the reference image 104 is balanced.

In some embodiments, the method 1200 also an operation of determining a feature descriptor 110 for each feature point of the plurality of feature points 108 include. In addition, the process can 1200 include any of the actions described above in relation to the steps in the 1 to 10 have been described.

The 13 shows a flowchart of an example method 1300 to create a new aligned image 116 from a plurality of series images. The procedure 1300 can by the image matching system described above 100 be implemented. The procedure 1300 includes a process 1310 detecting feature points 108 within a reference image 104 and a subsequent picture 106 and respective feature descriptors 110 , For example, the process can 1310 the detection of a plurality of feature points 108 both in the reference picture 104 as well as in the following picture 106 include.

The procedure 1300 also includes a process 1320 determining a feature descriptor for each feature point. For example, the process can 1320 comprising analyzing the detected feature points and generating a descriptor vector for each detected feature point, the Intensity information, pixel position information and information on the local neighborhood with respect to the respective feature point encoded.

In addition, the process includes 1300 a process 1330 determining pairs of matched feature points 108 , For example, the process can 1330 based on the feature descriptor 110 from every feature point 108 determining matched feature points 108 between the reference picture 104 and the following picture 106 include. Furthermore, the process can 1330 also determining, from the set of potentially matched feature points, an average displacement vector from the pixel location of the given feature point and filters, from the set of potentially matched feature points 108 , feature points having a shift that deviates from the average displacement vector by more than a predetermined number of pixels. In addition, the process can 1330 include any of the actions described above in relation to 4 have been described.

In addition, the process can 1330 predicting a pixel position of a given feature. In particular, the process can 1330 predicting for a given feature point a pixel position in the reference image 104 a corresponding pixel position in the subsequent image 106 include. Furthermore, the process can 1330 in some embodiments, a set of potentially matched feature points 108 identify. For example, the process can 1330 comparing a feature descriptor of the given feature point with feature descriptors 110 of feature points 108 with a pixel position within a predetermined number of pixels of the predicted corresponding pixel position in the subsequent image 106 include a set of potentially matched feature points 108 to identify. Furthermore, the process can 1330 also the sorting of feature points 108 within the predetermined number of pixels of the predicted corresponding pixel position, according to similarities of the feature points with the given feature point and filters, from the sorted feature points 108 , all but a predetermined number of most similar feature points 108 include. In addition, the process can 1330 the filtering, from the sorted feature points 108 , all but three of the most similar feature points 108 and / or sorting the feature points 108 according to a Hamming distance between the feature descriptors 110 the feature points 108 and the feature descriptor of the given feature point.

In addition, the process can 1330 in some embodiments, filtering feature points 108 from the set of potential matched feature points 108 include. For example, the process can 1330 filtering, from the set of potentially matched feature points 108 from any feature points within the predetermined number of pixels of the predicted corresponding pixel position in the subsequent image 106 with a descriptor distance above a threshold distance from the given feature point. Accordingly, the process can 1330 selecting a matched feature point. In particular, act 1360 may include selecting a matched feature point from the set of potentially matched feature points 108 with the highest similarity score as matching with the given feature point.

The procedure 1300 may also be a process 1340 determining a homography of the reference image 104 to the following picture 106 include. For example, the process can 1340 determining at least one homography, the matched feature point 108 between the reference picture 104 and the subsequent image 106. In addition, the process can 1340 include any of the actions described above in relation to 5 and 6 are described.

Accordingly, the method 1300 a process 1350 changing the reference image 104 include. For example, the process can 1350 changing the subsequent image 106 based on the at least one homography for generating a new adjusted image 116 include the pixel performance with the reference image 104 is balanced.

The 14 shows a flowchart of an example method 1400 to create a new matched image 116 from a plurality of series images. The procedure 1400 can by the image matching system described above 100 be implemented. The procedure 1400 includes a process 1410 subdividing a reference image 104 and a subsequent image 106 , For example can the process 1410 subdividing a reference image 104 in a plurality of local areas and a subsequent image 106 into a plurality of corresponding local areas. In some cases, the process can 1410 the subdivision of the reference image 104 into at least four local areas and dividing the subsequent image 106 in at least four corresponding local areas. In addition, the process can 1410 the subdivision of the reference image 104 and the following picture 106 by means of a spatial 2 x 2 grid and / or the subdivision of the reference image 104 and the following picture 106 by means of a 4 × 4 spatial grid. In addition, the process can 1410 include any of the actions described above in relation to 5 and 6 are described.

The procedure 1400 also includes a process 1420 of detecting feature points 108 both in the reference picture 104 as well as in the following picture 106 , For example, the process can 1420 the detection of a plurality of feature points 108 in each of the local areas of the plurality of local areas of the reference image 104 and in each of the respective local areas of the plurality of corresponding local areas of the subsequent image 106 include. In addition, the process can 1420 include any of the actions described above in relation to 2 and 3 are described.

In addition, the process includes 1400 a process 1430 determining matched feature points 108 , In particular, the process can 1430 determining matched feature points 108 between the reference picture 104 and the following picture 106 include. In some embodiments, the process may 1430 determining feature points that are most similar to each other based, at least in part, on a Hamming distance between the feature descriptors 110 the feature points 108 include. In addition, the process can 1430 include any of the actions described above in relation to 2 to 4 are described.

Furthermore, the method comprises 1400 a process 1440 uniting local areas that do not include a threshold number of feature points. In addition, the process can 1440 determining whether each local area of the plurality of local areas of the reference image 104 includes at least a predetermined number of feature points 108. Accordingly, the process can 1440 if a local area of the plurality of local areas of the reference image 104 not at least the predetermined number of feature points 108 comprising combining the local area with an additional local area of the plurality of local areas to form a unified local area.

Furthermore, the method comprises 1400 a process 1450 determining homography for each local area. For example, the process can 1450 based on the agreed feature points 108 between the reference picture 104 and the following picture 106 determining a homography for each local area of the plurality of local areas of the reference image 104 where the homography of each local area is feature points 108 from a respective local area of the reference image 104 a corresponding local area of the following image 106 assigns. Furthermore, the process can 1450 if a local or unified local area of the plurality of local areas of the reference image 104 at least the predetermined number of feature points 108 comprising determining the homography of the local area. In some embodiments, the process may 1440 comprise adapting a stable homography motion model to the unified local area. In addition, the process can 1440 include any of the actions described above in relation to 5 and 6 are described.

The procedure 1400 may also be a process 1460 changing the subsequent image 106 include. For example, the process can 1460 changing the subsequent image 106 based on the homography of each local area and unified local area to produce a new matched image 116 include, pixel by pixel with the reference image 104 is balanced.

In some embodiments, the method 1400 also include determining whether corners of the new matched image 116 , the adjacent corners of the plurality of local areas of the reference image 104 correspond, adjoin. If the corners of the new matched image 116 do not adjoin, become the average coordinates 904 the non-contiguous corners of the new aligned image 116 and a homography of each of the local areas of the plurality of local areas of the reference image 104 is calculated using the determined average coordinates 904 determined as coordinates of the adjacent corners. Furthermore, the method can 1400 based on the determined homography of each of the local areas of the reference image 104 and the associated flow vectors of each pixel comprise interpolating the flow vectors of pixels in the vicinity of boundaries common to a plurality of local areas of the plurality of local areas.

The 15 FIG. 10 is a diagram of an algorithm for performing a step for 1500 to tune feature points 108 between a reference picture 104 and the following picture 106 in accordance with one or more embodiments of the present disclosure. Furthermore, the step for 1500 include any of the actions described below with respect to the boxes 1502 to 1516 is described. As shown in box 1502, the algorithm includes selecting a feature point in a reference image 104 , In particular, the image matching system 100 a plurality of feature points 108 in each of the reference picture 104 and the following picture 106 include. Furthermore, the algorithm may include any of the actions described above with respect to 2 and 3 are described.

In addition, the algorithm includes predicting a corresponding pixel position of a matched feature point of the selected feature point of the reference image 104 in a subsequent picture 106 as it is in the box 1504 is shown. For example, the image matching system says 100 for a given feature point of the reference image 104 with pixel position X _R, the pixel position of the matched feature point as X _R + dX _R within the subsequent image 106 where dX _{R is} a previous motion offset as described above 4 has been described. Predicting the corresponding pixel position may include any of the actions described above with respect to FIG 4 have been described.

It also includes, as in the box 1506 described, the algorithm determining whether potentially matched feature points (ie, potential feature points) of the subsequent image 106 are present within a certain number of pixels of the predicted pixel position of the matched feature point. In particular, the image matching system 100 determine if there are potentially matched feature points within, eg, 50 pixels of the predicted pixel position of the matched feature point. Determining if potential matched feature points of the subsequent image 106 may be present within a certain number of pixels of the predicted pixel position of the matched feature point, may include any of the actions described above with respect to 4 have been described.

When the image adjustment system 100 determines that a given feature point of the potentially matched feature points of the subsequent image 106 is not within the given number of pixels, the algorithm may include filtering the given feature point from a set of potentially matched feature points, as in the box 1518 is shown. For example, the image matching system filters 100 the given feature point from the set of potentially matched feature points. For example, the image matching system removes 100 the given feature point from consideration as a matched feature point.

When the image adjustment system 100 determines that a given feature point of the subsequent image 106 Within the given number of pixels, the algorithm includes adding the given feature point to a set of potentially matched feature points, as in the box 1508 is shown. For example, the image matching system adds 100 add the given feature point to the set of potentially matched feature points. Further, the image matching system determines 100 after adding the given feature point to the set of potentially matched feature points, whether a descriptor distance of the feature point is above a threshold distance, as in the box 1510 is shown. For example, the image matching system determines 100 the similarity of two given feature points by evaluating the Hamming distance between the descriptors of the feature point.

When the image adjustment system 100 determines that the descriptor distance of the feature point is above a threshold distance filters the image matching system 100 the given feature point from the set of potentially matched feature points, as in the box 1518 is shown. When the image adjustment system 100 on the other hand determines that the descriptor distance of the feature point is not above a threshold distance determines the image matching system 100 an average displacement vector for all potentially matched pairs of feature points of the reference image 104 and the following picture 106 as it is in the box 1512 is shown. In particular, the image matching system determines 100 of all remaining potentially matched feature points for all detected feature points in the reference image 104 an average displacement vector D as the mean value of the feature displacement {X _P -X _R }.

After determining the mean value of the feature displacement {X _P -X _R }, the image matching system determines 100 whether a shift of remaining potentially matched feature points from the average displacement vector deviates by more than a predetermined number of pixels, as in the box 1514 is shown. It also filters when the image matching system 100 determines that a shift of a residual potentially matched feature point from the average displacement vector deviates by more than a predetermined number of pixels, the image matching system 100 the given feature point from the set of potentially matched feature points, as shown in box 1518. When the image adjustment system 100 on the other hand, determines that a shift of a residual potentially matched feature point from the average displacement vector does not deviate by more than a predetermined number of pixels, the image matching system selects 100 a matched feature point from the remaining potentially matched feature points (ie, the potential feature points) most closely resembling the feature point of the reference image 104 like in the box 1516 is shown. As discussed above, the matched feature point of the subsequent image 106 and the given feature point of the reference image 104 a matched feature point pair.

The 16 FIG. 12 is a diagram of an algorithm for performing a step of adjusting feature points between a reference image. FIG 104 and the following picture 106 in accordance with one or more embodiments of the present disclosure. For example, the step for 1600 may include any of the actions discussed below with respect to the boxes 1602 to 1622 are described. As it is in the box 1602 1, the algorithm includes determining whether a local area (or corresponding local area) comprises at least a threshold number of feature points. In particular, the image matching system 100 determine if a local area of the reference image 104 comprises a threshold number of detected feature pixels. As indicated above, the threshold number 50 . 75 . 100 . 200 . 500 . 1000 or any number of detected feature points. Furthermore, the algorithm may include any of the actions described above with respect to 5 and 6 have been described.

For example, if the image matching system 100 on the one hand, as in the box 1622 2, it is determined that the local area includes the threshold number of detected feature points, the algorithm may include applying a homography gesture model to the local area. For example, the image matching system 100 apply a homography gesture model to the local area in any of the methods described above with respect to the 5 to 7 have been described.

If, on the other hand, as in the box 1604 is described, the image adjustment system 100 determines that the local area does not include the threshold number of detected feature points, the algorithm may include merging the local area with another local area of the same hierarchy level. For example, the image matching system 100 include the local area with another local area of the same hierarchy level. The hierarchy levels are above in relation to the 5 and 6 described in detail. Further, uniting the local area with another local area of the same hierarchical level may comprise any of the actions described above with respect to 5 and 6 have been described.

Like in the box 1608 after combining the local area with another local area, the algorithm comprises determining whether the resulting merged local area comprises at least the threshold number of feature points. For example, the image matching system determines 100 whether the resulting merged local area comprises at least the threshold number of feature points. When the image adjustment system 100 determining that the resulting pooled local area comprises at least the threshold number of feature points, the algorithm comprises applying a homography motion model to the pooled local area, as shown in box 1622. For example, the image matching system 100 apply a homography movement model to the unified local area in any of the procedures described in relation to the 5 to 7 has been described.

When the image adjustment system 100 alternatively, determining that the resulting unified local area does not comprise at least the threshold number of feature points, the algorithm comprises merging the unified local area with an additional local area of the same hierarchy level as in the box 1610 is shown. In particular, the image matching system unites 100 the unified local area with an additional local area of the same hierarchy level. As stated above, the hierarchy levels are above in relation to the 5 and 6 described in detail.

Like in the box 1612 5, after combining the unified local area with an additional local area, the algorithm comprises determining whether the resulting new merged local area comprises at least the threshold number of feature points. For example, the image matching system determines 100 whether the resulting new merged local area comprises at least the threshold number of feature points. When the image adjustment system 100 determines that the resulting new merged local area comprises at least the threshold number of feature points, the algorithm comprises applying a homography motion model to the new merged local area, such as it in the box 1622 is shown. For example, the image matching system 100 apply a homography movement model to the unified local area in any of the procedures described in relation to the 5 to 7 has been described.

If alternatively the image matching system 100 determines that the resulting new merged local area does not include at least the threshold number of feature points, the algorithm comprises determining whether all local areas of the same hierarchy level have been merged with the new merged local area, as in the box 1614 is shown. In particular, the image matching system determines 100 Whether all local areas of the same hierarchy level have been merged with the new United Local Area. When the image adjustment system 100 determines that not all local areas of the same hierarchy level have been merged with the new merged local area, the algorithm comprises merging the unified local area with an additional local area of the same (eg, lowest level ( 6 )) Hierarchy levels, as in the box 1610 is shown. In particular, the image matching system unites 100 the new united local area with an additional local area of the same level of hierarchy. As stated above, the hierarchy levels are above in relation to the 5 and 6 described in detail.

On the other hand, if the image matching system 100 determines that all local areas of the same hierarchy level have been merged with the new merged local area, the algorithm comprises merging the new merged local area with a local area of a next hierarchy level (eg, first level up ( 6 )), as in the box 1620 is shown. In particular, the image matching system unites 100 the new United local area with a local area of the next hierarchy level. As stated above, the hierarchy levels are above in relation to the 5 and 6 described in detail.

Like in the box 1618 10, after merging the new merged local area with a local area of the next hierarchy level, the algorithm comprises determining whether the resulting merged local area comprises at least the threshold number of detected feature points. For example, the image matching system determines 100 whether the resulting merged local area comprises at least the threshold number of detected feature points. When the image adjustment system 100 determines that the resulting pooled local area comprises at least the threshold number of detected feature points, the algorithm comprises applying a homography motion model to the resulting pooled local area, as in the box 1622 is shown. For example, the image matching system 100 apply a homography gesture model in any way as described above with respect to 5 to 7 has been described.

If alternatively the image matching system 100 determines that the resulting merged local area does not comprise at least the threshold number of detected feature points, the algorithm comprises determining whether all local areas of the same hierarchy level (eg, the first level up ( 6 )) have been associated with the resulting combined local area. In particular, the image matching system determines 100 whether all local areas of the same hierarchy level have been merged with the resulting local area. When the image adjustment system 100 determines that not all local areas of the same hierarchy level have been merged with the resulting local area, the algorithm comprises merging the resulting pooled local area with an additional local area of the same hierarchy level (eg, the first level up ( 6 )) as shown in box 1610. In particular, the image matching system unites 100 the resulting pooled local area with an additional local area of the same hierarchy level. As stated above, the hierarchy levels are above in relation to the 5 and 6 described in detail.

On the other hand, if the image matching system 100 determines that all local regions of the same hierarchy level have been merged with the resulting local region, the algorithm comprises merging the resulting merged local region with a local region of a next hierarchy level (eg, the second level upwards ( 66 ), as in the box 1620 is shown. In particular, the image matching system unites 100 the resulting unified local area with a local area of the next hierarchy level. As stated above, the hierarchy levels are above in relation to the 5 and 6 described in detail.

Embodiments of the present disclosure may include or utilize a special or generic computer that includes computer hardware, such as one or more processors and a system memory, as discussed in more detail below. Embodiments within the scope of the present disclosure also include physical and other computer-readable media for containing or storing computer-executable instructions and / or data structures. In particular, one or more of the operations described herein may be implemented, at least in part, as instructions residing in a non-transitory computer-readable medium and executable by one or more computing devices (eg, any of the media content access devices described herein). In general, a processor (eg, a microprocessor) receives instructions from a non-transitory computer-readable medium (eg, a memory, etc.) and executes those instructions, thereby performing one or more operations, including one or more operations, here or there is or is, is or will be.

Computer readable media can be any available media that can be accessed by a general or specific computer system. Computer-readable media storing computer-executable instructions are non-transitory computer-readable storage media (devices). Computer-readable media containing computer-executable instructions are transmission media. Thus, by way of example and not limitation, embodiments of the disclosure may include at least two distinctly different types of computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.

Non-transitory computer-readable storage media (devices) include RAM, ROM, EEPROM, CD-ROM, SSDs (eg, based on RAM), flash memory, phase change memory ("PCM"), other types of memory, other optical disk storage, Magnetic disk storage or other magnetic storage devices, or any other medium suitable for storing desired program code means in the form of computer-executable instructions or data structures that can be accessed by a general or special computer.

Further, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures may be automatically transferred from transmission media to non-transitory computer-readable storage media (devices) (or vice versa). For example, computer-executable instructions or data structures obtained via a network or data link may be buffered in RAM within a network interface module (eg, a "NIC") and then eventually to a computer system RAM and / or less volatile computer storage media (devices). be transmitted in a computer system. Thus, it should be noted that non-transitory computer-readable storage media (devices) can be included in computer system components that also use (or even predominantly) transmission media.

Computer-executable instructions include e.g. Instructions and data that, when executed in a processor, cause a general purpose computer, computer, or processing device to perform a particular function or set of functions. In some embodiments, computer-executable instructions are executed on a general computer so that the general computer becomes a particular computer that executes elements of the disclosure. The computer-executable instructions may be e.g. binary statements ("binaries"), intermediate format statements, such as Assembly language, or even a source code. Although the subject matter has been described in language specific to structural features and / or procedural operations, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the features and processes described are disclosed as example forms for implementing the claims.

One skilled in the art will appreciate that the disclosure can be made in network computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframes, cell phones , PDAs, tablets, pagers, routers, switches, and the like. The disclosure may also be made in distributed system environments where local and remote computer systems connected through a network (either through wired data links, wireless data links, or a combination of wired and wireless data links) perform both tasks. In a distributed system environment, program modules may reside in both local and remote storage devices.

Embodiments of the present disclosure may also be implemented in cloud computing environments. In this description, "cloud computing" is defined as a model that allows network access to a common pool of configurable computing resources as needed. For example, cloud computing may be deployed in the marketplace to provide anywhere, convenient on-demand access to the common pool of configurable computing resources. The shared pool of configurable computing resources can be quickly deployed through virtualization and released with a low overhead or service provider interaction, and then scaled accordingly.

A cloud computing model may be composed of various characteristics, such as e.g. on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, etc. A cloud computing model may also include various service models, such as: Software as a Service ("SaaS"), Platform as a Service ("PaaS") and Infrastructure as a Service ("IaaS"). A cloud computing model may also be propagated by means of various distribution models, e.g. a private cloud, a community cloud, a public cloud, a hybrid cloud, etc. In this description and in the claims, a "cloud computing environment" is an environment in which cloud computing is employed.

The 17 shows in block diagram form an exemplary computing device 1700 , which may be configured to perform one or more of the above-described operations. It is clear that one or more computing devices, such as the computing device 1700 , the image matching system 100 and / or the client device may or may implement. As it is in the 17 1, the computing device may be a processor 1702 , a store 1704 , a storage device 1706 , an I / O interface 1708 and a communication interface 1710, which may be communicatively coupled by means of a communication infrastructure. While a sample computing device 1700 in the 17 are shown in the 17 not limited to components shown. Additional or alternative components may be used in other embodiments. Furthermore, the computing device 1700 in certain embodiments, comprise fewer or more components than those described in the 17 are shown. Components of the computing device 1700 used in the 17 are shown below in more detail.

In one or more embodiments, the processor includes 1702 Hardware for executing instructions, such as those that make up a computer program. By way of example and not limitation, the processor 1702 to execute instructions, the instructions from an internal register, an internal cache, the memory 1704 or the storage device 1706 query (or fetch) and decode and execute. In one or more embodiments, the processor 1702 include one or more internal cache (s) for data, instructions or addresses. By way of example and not limitation, the processor 1702 one or more instruction cache (s), one or more data cache (s) and one or more translation look aside buffers (TLBs). Instructions in the instruction caches may be copies of the instructions in the memory 1704 or the storage device 1706 be.

The computing device 1700 includes a memory 1704 that is related to the processor (s) 1702 is coupled. The memory 1704 may be used to store data, metadata and programs for execution by the processor (s). The memory 1704 may include one or more of volatile and nonvolatile memories, such as random access memory ("RAM"), read only memory ("ROM"), semiconductor disk ("SSD"), flash memory, phase change memory ("PCM"), or other types of data storage. The memory 1004 can be an internal or a distributed or external memory.

The computing device 1700 comprises a storage device 1706 which includes a memory for storing data or instructions. As an example and not by way of limitation, the memory device 1706 comprise a non-volatile storage medium as described above. The storage device 1706 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a serial universal bus (USB) drive, or a combination of two or more thereof. This storage device 1706 may optionally include removable or non-removable (fixed) media. The storage device 1706 can be internal or external to the computing device 1700 available. In one or more embodiments, the storage device is 1706 a nonvolatile solid state memory. In other embodiments, the storage device comprises 17006 a read-only memory (ROM). Optionally, this ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable ROM (EPROM), an electrically erasable PROM (EEPROM), an electrically variable ROM (EAROM) or flash memory, or a combination of two or more thereof.

The computing device 1700 also includes one or more input or output ("I / O") devices / interfaces 1708 provided to enable a user to input to the computing device 1700 perform an output from the computing device 1700 or otherwise receive data to and from the computing device 1700 transferred to. These I / O devices / interfaces 1708 may be a mouse, keypad or keyboard, touch screen, camera, optical scanner, network interface, modem or other known I / O devices, or a combination of such I / O devices. Devices / interfaces include. The touch screen can be activated with a pen or a finger.

The I / O devices / interfaces 1708 may include one or more devices for presenting an output to a user, including, but not limited to, a graphics module, a display (eg, a display screen), one or more output drivers (eg Display driver), one or more speakers, and one or more audio drivers. In certain embodiments, the devices / interfaces 1708 adapted to provide graphical data for a display for presentation to a user. The graphical data may be representative of one or more graphical user interfaces and / or any other graphical content that may serve as a particular implementation.

The computing device 1700 may further include a communication interface 1710 include. The communication interface 1710 may include hardware, software, or both. The communication interface 1710 may include one or more interfaces for communicating (such as with packet-based communication) between the computing device and one or more other computing devices. 1700 or one or more network (s). By way of example and not limitation, the communication interface 1710 a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as WI-FI. The computing device 1700 can also have a bus 1712 include. The bus 1712 may include hardware, software, or both, the components of the computing device 1700 coupled with each other.

The foregoing description has been described with reference to specific exemplary embodiments thereof. Various embodiments and aspects of the disclosure are described with reference to details discussed herein, and the accompanying drawings illustrate the various embodiments. The foregoing description and drawings are for illustrative purposes only and are not intended to be limiting. Numerous specific details are described so that various embodiments will be well understood.

The additional or alternative embodiments may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The described embodiments are to be considered in all aspects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes within the meaning and range of equivalence of the claims are intended to be embraced by their scope.

The above-described embodiments of the disclosure shown in the accompanying figures do not limit the scope of the disclosure, as these embodiments are merely examples of embodiments of the invention, which is defined by the appended claims and their legal equivalents. Any equivalent embodiments are intended to be within the scope of this invention. Various modifications of the present disclosure, in addition to those described herein, such as e.g. Alternative suitable combinations of the features described will become apparent to those skilled in the art from the description. Such modifications and embodiments are also intended to fall within the scope of the appended claims and legal equivalents.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 62/430709 [0001]

Cited non-patent literature

• Rublee, Ethan, et al., ORB: An Efficient Alternative to SIFT or SURF, International Conference on Computer Vision, pp. 2564-2571 (2011), the disclosure of which is fully incorporated herein by reference. In additional embodiments, the image matching system 100 may utilize other key-point detectors and descriptors, such as e.g. a "Scale Invariant Feature Transform (SIFT)" Key Point Detector, a "Speeded-Up Robust Features (SURF)" Key Point Detector and / or a "Histogram of Oriented Gradients (HOG)" Key Point Detector [0035]
• Liu, Ziwei et al., Fast Burst Images Denoising, ACM Transaction on Graphics 33, No 6: 232 (2014) [0047]

Claims (20)

A computer-implemented method for generating matched images from burst images, the method comprising: Detecting a plurality of feature points in a reference image and a subsequent image; Determining a feature descriptor for each feature point of the plurality of feature points; Determining matched feature points between the reference image and the subsequent image based on the feature descriptor of each feature point; Determining at least one homography associating matched feature points between the reference image and the subsequent image; and Altering or warping the subsequent image based on the at least one homography to produce a new adjusted image aligned with the reference image. Computer-implemented method according to Claim 1 wherein determining a feature descriptor for each feature point of the plurality of feature points comprises analyzing the detected feature points and generating a descriptor vector for each detected feature point comprising intensity information, pixel position information, and local neighborhood information regarding its respective feature point. Computer-implemented method according to Claim 1 or 2 in which the determining of matched feature points between the reference image and the subsequent image comprises: predicting for a given feature point having a pixel position in the reference image of a corresponding pixel position in the subsequent image; Identifying a set of potentially matched feature points from the subsequent image; Filtering the set of potentially matched feature points from any feature points having a descriptor distance above a threshold distance from the given feature point; and selecting a matched feature point from the set of potentially matched feature points having the highest similarity score in accordance with the given feature point. Computer-implemented method according to Claim 3 wherein identifying the set of potentially matched feature points from the subsequent image comprises comparing a feature descriptor of the given feature point with feature descriptors of feature points having a pixel position within a predetermined number of pixels of the predicted corresponding pixel position in the subsequent image. Computer-implemented method according to Claim 4 wherein identifying the set of potentially matched feature points comprises: sorting feature points within the predetermined number of pixels of the predicted corresponding pixel position according to the similarities of the feature points to the given feature point; and filtering, from the sorted feature points, all but a predetermined number of feature points with the closest similarity. Computer-implemented method according to Claim 5 in which the filtering, from the sorted feature points, all but a predetermined number of feature points with the strongest similarity, comprises filtering, from the sorted feature points, all but three most similarity feature points. Computer-implemented method according to Claim 5 or 6 wherein sorting feature points within the predetermined number of pixels of the predicted corresponding pixel position according to the similarities of the feature points with the given feature point comprises sorting the feature points according to a Hamming distance between the feature descriptors of the feature points and the feature descriptor of the given feature point. Computer implemented method according to one of Claims 3 to 7 wherein determining the matched feature points between the reference image and the subsequent image further comprises: determining, from the set of potentially matched feature points, an average displacement vector from the pixel location of the given feature point; and filtering, from the set of potentially matched feature points of feature points, a displacement that deviates from the average displacement vector by more than a predetermined number of pixels. The computer-implemented method of claim 1, wherein determining at least one homography associating matched feature points between the reference image and the subsequent image comprises applying a homography gesture model to each of a plurality of local regions individually to determine a homography for each local area. associates the feature points in a local area of the reference image with feature points in a corresponding local area of the subsequent image. Computer-implemented method according to Claim 8 or 9 wherein modifying the subsequent image based on the at least one homography to produce a new matched image aligned with the reference image comprises changing the subsequent image based on the homography of each local area. A system for generating matched images from burst images, comprising: a memory comprising a reference image and a subsequent image; at least one processor communicatively coupled to the memory and storing instructions thereon that, when executed by the at least one processor, cause the system to: divide the reference image into a plurality of local areas; dividing the subsequent image into a plurality of corresponding local areas; detects a plurality of feature points in the local areas of the reference image; detects a plurality of feature points of the respective local areas of the subsequent image; determines matched feature points between the reference image and the subsequent image; local areas of the reference image that do not include a threshold number of feature points are merged with another local area of the reference image; determining homography for each local area and unified area of the reference image based on the matched feature points between the reference image and the subsequent image, wherein the homography of each local area associates feature points from a respective local area of the reference image with a respective corresponding local area of the subsequent image; and the subsequent image based on the homography of each local area and the combined local area alters / warps to produce a new matched image aligned with the reference image. System after Claim 11 wherein the instructions, when executed by the at least one processor, cause the system to determine homography for each local area of the reference image by performing steps comprising: determining whether each local area of the plurality of local areas of the reference image comprises at least one threshold number of feature points; if a given local area of the reference image comprises at least the predetermined number of feature points, determining a homography of the given local area; and if a given local area of the reference image does not include at least the predetermined number of feature points, merging the given local area with an additional local area of the reference image to form a unified local area and determining the homography of the merged local area. System after Claim 12 wherein the instructions, when executed by the at least one processor, cause the system to determine the homography of the merged local area by performing steps that include adapting a stable homography motion model to the merged local area. System after Claim 12 or 13 in that the instructions, when executed by the at least one processor, cause the system to obtain the subsequent image by interpolating homography flow vectors of the homographies that are near boundaries, the two or more of the corresponding local regions of the subsequent image are common, changed. System according to one of Claims 11 to 14 method further comprising instructions that, when executed by the at least one processor, cause the system to: determine whether corners of the new matched image corresponding to adjacent corners of the plurality of local areas of the subsequent image are adjacent; if the corners of the new matched image do not abut, determining average coordinates of the non-adjacent corners of the new matched image; and determines a homography of each of the local areas of the plurality of local areas of the reference image using the average coordinates as coordinates of the adjacent corners. A computer-implemented method for generating matched images from burst images, the method comprising: Dividing a reference picture and a subsequent picture into a plurality of local areas; Detecting a plurality of feature points in the local areas of the reference image and the subsequent image; Determining matched pairs of feature points between the reference image and the subsequent image; performing, based on the matched pairs of feature points, a step of determining a homography for each local area of the plurality of local areas, the homography of each local area associating feature points from a local area of the reference image with a corresponding local area of the subsequent image; and Changing or warping the subsequent image based on the homography of each local area of the plurality of local areas to produce a new matched image aligned with the reference image. Computer-implemented method according to Claim 16 wherein determining matched pairs of feature points between the reference image and the subsequent image comprises determining feature points that are most similar to one another based at least in part on a Hamming distance between the feature descriptors of the feature points. Computer-implemented method according to Claim 16 or 17 wherein dividing each of a reference image and a subsequent image comprises dividing the reference image into at least four local regions and the subsequent image into at least four corresponding local regions. Computer-implemented method according to Claim 18 in which dividing the reference image into at least four local regions and the subsequent image into at least four corresponding local regions comprises subdividing the reference image and the subsequent image by means of a 2 × 2 spatial grid. Computer implemented method according to one of Claims 16 to 19 wherein detecting a plurality of feature points comprises selecting a best number of feature points having the highest confidence values from each local area of the plurality of local areas.

Images