WO2023098045A1

WO2023098045A1 - Image alignment method and apparatus, and computer device and storage medium

Info

Publication number: WO2023098045A1
Application number: PCT/CN2022/100804
Authority: WO
Inventors: 蒋海峰
Original assignee: 上海闻泰信息技术有限公司
Priority date: 2021-12-02
Filing date: 2022-06-23
Publication date: 2023-06-08
Also published as: CN114298902A

Abstract

Provided in the embodiments of the present disclosure are an image alignment method and apparatus, and a computer device and a storage medium, which are applied to the technical field of image processing, and solve the problem in the prior art of low image alignment precision caused by unobvious feature points. The method comprises: acquiring a first image and a second image; extracting a plurality of first feature points from a first region of interest of the first image, and extracting a plurality of second feature points from a second region of interest of the second image; performing feature matching on the plurality of first feature points and the plurality of second feature points, determining the coordinates of at least one first target feature point among the plurality of first feature points, and determining the coordinates of at least one second target feature point among the plurality of second feature points; calculating a first homography matrix for pixel mapping; and on the basis of the first homography matrix, mapping a pixel value of each pixel point in the first region of interest to a corresponding pixel point in the second region of interest, so as to obtain the first region of interest after image alignment.

Description

Image alignment method, device, computer equipment and storage medium

This disclosure claims the priority of the Chinese patent application with the application number 2021114598720 and the title of the invention "an image alignment method, device, electronic device and storage medium" submitted to the China Patent Office on December 02, 2021, the entire content of which is passed References are incorporated in this disclosure.

technical field

The present disclosure relates to an image alignment method, device, computer equipment and storage medium.

Background technique

Image alignment plays an important role in image fusion, image stitching, and computer vision. It is to align two or more images at different times or from different angles to determine the mapping relationship between the spatial positions and intensities of the images. In this way, multiple frames of images shot continuously in a very short period of time are fused into one frame of image. The feature-based image alignment method in the related art first finds the feature points in the two pictures and performs feature matching, and then calculates the homography matrix to uniformly transform the pixel coordinates in the images to perform image alignment. There are uncertain factors such as illumination, depth of field, and angle in the process, which makes it difficult to extract obvious feature points from the image to be processed, which makes the method's scene adaptability poor and the accuracy of image alignment low.

Contents of the invention

(1) Technical problems to be solved

The method solves the problem in the related art that the accuracy of image alignment is not high due to inconspicuous feature points.

(2) Technical solutions

According to various embodiments of the present disclosure, an image alignment method, device, computer equipment, and storage medium are provided.

An image alignment method, the method comprising:

Acquiring a first image and a second image, the first image being the first image after global image alignment processing based on the second image;

Extract a plurality of first feature points from the first region of interest of the first image, and extract a plurality of second feature points from the second region of interest of the second image; the size of the first region of interest and the second region of interest The same, and the corresponding positions in different images;

performing feature matching on multiple first feature points and multiple second feature points, determining the coordinates of at least one first target feature point among the multiple first feature points, and determining at least one second target among the multiple second feature points Coordinates of feature points, at least one first target feature point matches at least one second target feature point;

calculating a first homography matrix for pixel mapping according to the coordinates of at least one first target feature point and the coordinates of at least one second target feature point;

Based on the first homography matrix, the pixel value of each pixel in the first region of interest is mapped to the corresponding pixel in the second region of interest, so as to obtain the first region of interest after image alignment.

As an optional implementation manner of the embodiment of the present disclosure, acquiring the first image includes: acquiring a histogram of the second image, and determining the pixel value of each pixel in the second image; Set the pixel value of the pixel point corresponding to the pixel position in the third image; according to the pixel value of each pixel point in the second image and the pixel value of each pixel point in the third image after setting, extract At least one first corner point in the third image, and extract at least one second corner point in the second image; perform feature matching on at least one first corner point and at least one second corner point, and determine at least one first corner point A first target corner point among the points, and a second target corner point among at least one second corner point, the first target corner point matches the second target corner point according to the coordinates of the first target corner point and the second target corner point The coordinates of the second homography matrix are calculated; based on the second homography matrix, the pixel value of each pixel point in the third image is mapped to the corresponding pixel point of the second image, and the first one after global image alignment is obtained image.

As an optional implementation manner of the embodiment of the present disclosure, the number of at least one feature matching is multiple, and performing feature matching on at least one first corner point and at least one second corner point includes: combining at least one first corner point Perform feature matching with at least one second corner point in units of N corner points; perform feature matching again with at least one first corner point and at least one second corner point in units of M corner points; Wherein, N is a positive integer greater than M.

As an optional implementation of this embodiment of the present disclosure, calculating the second homography matrix according to the coordinates of the first target corner and the coordinates of the second target corner includes: if the number of the first target corner is greater than or equal to the preset number, then calculate the second homography matrix according to the coordinates of the first target corner point and the coordinates of the second target corner point; the method also includes: if the number of the first target corner points is less than the preset number, then Extract a plurality of first ORB feature points from the third image, and extract a plurality of second ORB feature points from the second image; perform feature matching with a plurality of first ORB feature points and a plurality of second ORB feature points, and determine the plurality of ORB feature points The coordinates of at least one first target ORB feature point among the first ORB feature points, and the coordinates of at least one second target ORB feature point among a plurality of second ORB feature points, at least one first target ORB feature point and at least one Matching the second target ORB feature points; calculating a second homography matrix according to the coordinates of at least one first target ORB feature point and at least one second target ORB feature point.

As an optional implementation manner of an embodiment of the present disclosure, multiple first feature points are extracted from a first region of interest in the first image, and multiple second feature points are extracted from a second region of interest in the second image, Including: performing target object detection on the first region of interest and the second region of interest; determining the first target object graph from the first region of interest, and determining the second target object graph from the second region of interest; determining the second target object graph from the first target region A plurality of first feature points are extracted from the object graph, and a plurality of second feature points are extracted from the second target object.

As an optional implementation manner of an embodiment of the present disclosure, extracting a plurality of first feature points from the first target object map, and extracting a plurality of second feature points from the second target object includes: identifying the first target object Whether the target object is included in the figure and the second target object figure; if the target object is contained in the first target object figure and the second target object figure, then a plurality of first ECC feature points are extracted from the first target object figure, and from the second target object figure A plurality of second ECC feature points are extracted from the target object graph.

As an optional implementation manner of the embodiment of the present disclosure, after determining the first target object graph from the first region of interest and determining the second target object graph from the second region of interest, further includes: acquiring the second target object graph and determine the pixel value of each pixel in the second object image; according to the pixel value of each pixel in the second object image, set the pixel corresponding to the pixel position in the first object image value.

As an optional implementation manner of the embodiment of the present disclosure, the target object detection is portrait detection, determining the first target object map from the first region of interest, and determining the second target object map from the second region of interest, including: obtaining A first portrait mask of the first region of interest obtained after the portrait detection, and a second portrait mask of the second region of interest; determining the first portrait image from the first image according to the first portrait mask, And determining a second portrait image from the second image according to the second portrait mask.

As an optional implementation manner of this embodiment of the present disclosure, the method further includes: extracting a third region of interest in the first image and a fourth region of interest in the second image according to a default size; If the size of the ROI is smaller than the preset size, expand the third ROI according to the preset size to obtain the first ROI; if the size of the fourth ROI is smaller than the preset size, expand the third ROI according to the preset size Extend the fourth ROI to obtain the second ROI.

An image alignment device, comprising:

An acquisition module configured to acquire a first image and a second image, where the first image is a first image after global image alignment processing based on the second image;

An extraction module configured to extract a plurality of first feature points from a first region of interest in the first image, and extract a plurality of second feature points from a second region of interest in the second image; the first region of interest and the second The ROIs have the same size and correspond to positions in different images;

A feature matching module configured to perform feature matching on a plurality of first feature points and a plurality of second feature points, determine the coordinates of at least one first target feature point in the plurality of first feature points, and determine a plurality of second feature points Coordinates of at least one second target feature point, at least one first target feature point matches at least one second target feature point;

A calculation module configured to calculate a first homography matrix for pixel mapping according to the coordinates of at least one first target feature point and the coordinates of at least one second target feature point;

A mapping module configured to map the pixel value of each pixel in the first region of interest to the corresponding pixel in the second region of interest based on the first homography matrix, so as to obtain the first image of interest after image alignment area.

As an optional implementation manner of the embodiment of the present disclosure, the acquiring module is configured to acquire a histogram of the second image, and determine the pixel value of each pixel in the second image; according to the histogram of each pixel in the second image Pixel value, set the pixel value of the pixel point corresponding to the pixel position in the third image; according to the pixel value of each pixel point in the second image, and the pixel value of each pixel point in the third image after setting, extract the third At least one first corner point in the image, and extracting at least one second corner point in the second image; performing feature matching on at least one first corner point and at least one second corner point, and determining at least one of the first corner points The first target corner point of , and the second target corner point of at least one second corner point, the first target corner point matches the second target corner point; according to the coordinates of the first target corner point and the second target corner point coordinates, calculate the second homography matrix; based on the second homography matrix, map the pixel value of each pixel point in the third image to the corresponding pixel point in the second image, and obtain the first image after global image alignment .

As an optional implementation manner of the embodiment of the present disclosure, the acquisition module is configured to perform feature matching once with at least one first corner point and at least one second corner point in units of N corner points; One corner point and at least one second corner point, with M corner points as the unit, perform feature matching again;

Wherein, N is a positive integer greater than M.

As an optional implementation manner of the embodiment of the present disclosure, the obtaining module is configured to, under the condition that the number of the first target corner points is greater than or equal to the preset number, according to the coordinates of the first target corner point and the second target corner point The coordinates of , calculate the second homography matrix; under the condition that the number of the first target corner points is less than the preset number, extract a plurality of first directional fast rotation ORB feature points from the third image, and extract from the second image Extracting a plurality of second ORB feature points; performing feature matching on a plurality of first ORB feature points with a plurality of second ORB feature points, determining the coordinates of at least one first target ORB feature point in the plurality of first ORB feature points, and determining the coordinates of at least one second target ORB feature point among the plurality of second target ORB feature points, at least one first target ORB feature point matching at least one second target ORB feature point; according to the coordinates of at least one first target ORB feature point , and the coordinates of at least one second target ORB feature point to calculate a second homography matrix.

As an optional implementation manner of this embodiment of the present disclosure, the extraction module is configured to perform target object detection on the first region of interest and the second region of interest; determine the first target object graph from the first region of interest, and obtain the The second region of interest determines a second object map; extracts a plurality of first feature points from the first object map, and extracts a plurality of second feature points from the second object map.

As an optional implementation manner of the embodiment of the present disclosure, the first feature point is a first entropy correlation coefficient ECC feature point, and the second feature point is a second ECC feature point;

An extraction module configured to identify whether the first target object graph and the second target object graph contain the target object; under the condition that the first target object graph and the second target object graph contain the target object, extract the A plurality of first ECC feature points are extracted, and a plurality of second ECC feature points are extracted from the second target object map.

As an optional implementation manner of the embodiment of the present disclosure, the extraction module is configured to obtain a histogram of the second target object graph, and determine the pixel value of each pixel in the second target object graph; according to the second target object graph The pixel value of each pixel in the set, set the pixel value of the pixel corresponding to the pixel position in the first object map.

As an optional implementation manner of the embodiment of the present disclosure, the target object detection is portrait detection; the extraction module is configured to obtain the first portrait mask of the first region of interest obtained after the portrait detection, and the second region of interest A second portrait mask; determining a first portrait image from the first image according to the first portrait mask, and determining a second portrait map from the second image according to the second portrait mask.

As an optional implementation manner of the embodiment of the present disclosure, the extraction module is configured to extract the third region of interest in the first image and the fourth region of interest in the second image according to a default size; in the third region of interest If the size of the area is smaller than the preset size, expand the third ROI according to the preset size to obtain the first ROI; if the size of the fourth ROI is smaller than the preset size, expand according to the preset size The fourth ROI is used to obtain the second ROI.

A computer device comprising a memory and one or more processors, the memory configured as a module storing computer readable instructions; the computer readable instructions, when executed by the one or more processors, cause the One or more processors execute the steps of providing the image alignment method in any one embodiment of the present disclosure.

One or more non-volatile storage media storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, one or more processors execute the image provided in any one of the embodiments of the present disclosure. Align the steps of the method.

Additional features and advantages of the disclosure will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the disclosure. The objectives and other advantages of the disclosure will be realized and attained by the structure particularly pointed out in the written description, claims hereof as well as the accompanying drawings, the details of one or more embodiments of the disclosure being set forth in the accompanying drawings and the description below.

In order to make the above objects, features and advantages of the present disclosure more comprehensible, optional embodiments are given below and described in detail in conjunction with the accompanying drawings.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, In other words, other drawings can also be obtained from these drawings without paying creative labor.

FIG. 1 is an application scenario diagram 1 of an image alignment method provided by one or more embodiments of the present disclosure;

FIG. 2 is a flow chart 1 of steps of an image alignment method provided by one or more embodiments of the present disclosure;

FIG. 3 is the second application scene diagram of the image alignment method provided by one or more embodiments of the present disclosure;

FIG. 4 is a schematic diagram of extracting a region of interest provided by one or more embodiments of the present disclosure;

FIG. 5 is a second flow chart of the steps of the image alignment method provided by one or more embodiments of the present disclosure;

FIG. 6 is a flowchart three of steps of an image alignment method provided by one or more embodiments of the present disclosure;

Fig. 7 is a structural block diagram of an image alignment device in one or more embodiments of the present disclosure;

FIG. 8 is an internal structural diagram of a computer device in one or more embodiments of the present disclosure.

Detailed ways

In order to more clearly understand the above objects, features and advantages of the present disclosure, the solutions of the present disclosure will be further described below. It should be noted that, in the case of no conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

In the following description, many specific details are set forth in order to fully understand the present disclosure, but the present disclosure can also be implemented in other ways than described here; obviously, the embodiments in the description are only some of the embodiments of the present disclosure, and Not all examples.

The terms "first" and "second" and the like in the specification and claims of the present disclosure are used to distinguish different objects, rather than to describe a specific order of objects. For example, the first camera and the second camera are used to distinguish different cameras, not to describe a specific order of the cameras. .

In the embodiments of the present disclosure, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present disclosure shall not be construed as being preferred or advantageous over other embodiments or designs. To be precise, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific manner. In addition, in the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "plurality" refers to two one or more.

In some cases, the feature-based image alignment method first finds the feature points in the two pictures and performs feature matching, and then calculates the homography matrix to uniformly transform the pixel coordinates in the image for image alignment, but in the actual operation process, due to There are uncertain factors such as illumination, depth of field, and angle in the image shooting process, which lead to obvious time stamps of images that need to be processed, obvious depth of field, large brightness differences, large field of view differences, and large changes in image space, making it difficult to extract obvious feature points. This method has poor scene adaptability and low accuracy of image alignment.

In order to solve the above problems, the present disclosure first performs global image alignment on the image to be processed, and then extracts feature points from the first region of interest in the image to be processed and the second region of interest in the reference image for feature matching to obtain the matched feature points The coordinates of the homography matrix are calculated to map the first region of interest to the second region of interest according to the homography matrix to achieve local alignment of the image to be processed by aligning all regions of interest in the image to be processed The precise alignment of the images to be processed reduces the influence of uncertain factors in the image shooting process, and can extract more feature points for image alignment, which is applicable to various scenarios and improves the accuracy of image alignment.

The image alignment method provided in the present disclosure can be applied to the application environment shown in FIG. 1 . The image alignment method is applied in an image alignment system. The image alignment system includes a terminal 102 and a server 104 . Wherein, the terminal 102 communicates with the server 104 through a network. Specifically, the image alignment method is applied to the terminal 102. The terminal 102 performs global image alignment on the image to be processed, and then extracts feature points from the first region of interest in the image to be processed and the second region of interest in the reference image to perform feature Matching to obtain the coordinates of the matched feature points, thereby calculating the homography matrix to map the first region of interest to the second region of interest according to the homography matrix to achieve local alignment of the image to be processed, by aligning the image to be processed All regions of interest in the image to achieve precise alignment of the image to be processed.

In addition, the image alignment method is applied to the server 104, the server 104 receives the image to be processed 106 sent by the terminal 102, and accurately aligns the image through the above image alignment method, and then the server 104 returns the precisely aligned image 108 to the terminal 102, the The method also includes steps S202-S210:

Wherein, the terminal 102 can be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices, and the server 104 can be realized by an independent server or a server cluster composed of multiple servers.

Referring to FIG. 2 , FIG. 2 is a flow chart 1 of steps of an image alignment method provided by one or more embodiments of the present disclosure. The present disclosure is mainly illustrated by taking the method applied to the terminal 102 in FIG. 1 as an example.

S202. Acquire a first image and a second image.

Among them, the first image is an image obtained by performing global image alignment processing on the third image with the second image as a reference image; the third image is an image obtained from a terminal device as a comparison device, which requires global image alignment processing The second image is an image obtained from the terminal device as a reference device; the second image and the third image are images of the same scene at different angles, or images of the same scene at different times.

In some embodiments, before acquiring the first image, the second image and the third image are acquired from different terminal devices. Wherein, the second image is a reference image, and the third image is an image to be processed that requires global image alignment processing.

Exemplarily, as shown in FIG. 3 , the devices included in the figure include a mobile phone 301 , a mobile phone 303 and a server 305 ; the images include a second image 302 , a third image 304 , and a first image 306 . First, the second image 302 is obtained from the mobile phone 301 as the reference machine, and the third image 304 is obtained from the mobile phone 304 as the comparison machine, and the server 305 uses the second image 302 as the reference image to perform global image alignment processing on the third image to obtain the second image. An image 306 .

The following will introduce the process of using the second image as a reference image to perform global image processing on the third image to obtain the first image.

In some embodiments, after the second image and the third image are obtained from different terminal devices, the histogram of the second image is obtained. Since the histogram represents the number of pixels of each brightness level of the image, it shows the The distribution in , intuitively reflects the brightness of the picture, so the pixel value of each pixel in the second image can be determined according to the histogram of the second image; then according to the pixel value of each pixel in the second image, use Set the pixel value of the pixel point corresponding to the pixel position in the third image by histogram equalization method, histogram specification, etc., so as to adjust the brightness of the third image according to the second image, so that the pixel value of the pixel point corresponding to the pixel position in the third image The pixel value is set to be the same as the pixel value of the pixel point corresponding to the corresponding pixel position in the second image, so that the brightness of the third image is the same as that of the first image, so as to extract feature points from the two pictures.

In some embodiments, the brightness of the third image may be adjusted by methods such as smoothing and denoising, illumination equalization processing, etc., including but not limited to the above methods, which are not limited in the present disclosure.

Further, according to the pixel value of each pixel in the second image and the set pixel value of each pixel in the third image, extract at least one corner point in the brightness-adjusted third image, and extract the second At least one corner point in the image, and match the corner points of the two extracted images.

Among them, the corner point is the isolated point with the greatest or smallest intensity on certain attributes, the end point of the line segment, or the point with the largest local curvature on the curve, which is used to describe the corners, boundary points and other information in the image. In this disclosure, the corner points are extracted by the Harris corner detection method. It should be noted that the basic idea of the Harris corner detection method is to use a fixed window to slide in any direction on the image, and compare the before and after sliding. In both cases, the pixel grayscale changes in the window. If there is a sliding in any direction and there is a large grayscale change, it is considered that there is a corner in the window. The corner points are extracted by the Harris corner point detection method, which improves the robustness to noise and improves the efficiency and stability of corner point detection.

In some embodiments, during the process of corner point matching, the corner points in the extracted third image and the corner points in the second image are subjected to feature matching at least once, and firstly, when the image with a smaller scale is blurred , perform feature matching on N corner points to obtain a rough matching result of the corner points, and then increase the scale to perform feature matching on M corner points until the scale is transformed to the size of the original image to complete the precise matching of the image, where , N is a positive integer greater than M.

Exemplarily, in the process of corner point matching, rough matching is first performed in units of 10 corner points, and then the scale is increased to perform precise matching in units of 4 corner points.

Through at least one feature matching, the coarse matching and precise matching of the third image and the second image are realized, the false matching rate is reduced, and the matching accuracy is improved.

In some embodiments, after exact matching to obtain the matched corners, optionally, if the number of the first target corners is greater than or equal to the preset number, then according to the coordinates of the first target corners and the second target corners coordinates, calculate the second homography matrix. Wherein, the second homography matrix is a homography matrix used for global image processing on the third image.

If the number of corner points of the first target is less than the preset number, a plurality of Oriented Fast and Rotated Brief (Oriented Fast and Rotated Brief, ORB) feature points, and extract a plurality of second ORB feature points from the second image, then carry out feature matching with a plurality of first ORB feature points and a plurality of second ORB feature points, and determine among a plurality of first ORB feature points coordinates of at least one first target ORB feature point, and determining coordinates of at least one second target ORB feature point among a plurality of second target ORB feature points, the at least one first target ORB feature point matching at least one second target ORB feature point , and then, according to the coordinates of at least one first target ORB feature point and the coordinates of at least one second target ORB feature point, calculate the second homography matrix.

Through ORB feature matching, ORB feature points are extracted for feature matching, the matched ORB feature points are determined, and the coordinates corresponding to the ORB feature points are obtained, thereby calculating the homography matrix, which improves the accuracy and efficiency of feature matching.

Optionally, the present disclosure uses a random sampling consensus algorithm to calculate the homography matrix and perform iterative update to eliminate mismatching points and determine the optimal homography matrix.

By adjusting the brightness of the third image, the problem of the difference in brightness between the image to be processed and the reference image is solved, and the convenience of the feature matching process is improved; the feature points of the third image and the second image are matched multiple times to reduce errors. The matching rate improves the accuracy of matching; the optimal homography matrix is iteratively calculated through the random consensus algorithm, so as to obtain the first image after global image alignment, and achieve a better effect of global image alignment.

S204, extracting a plurality of first feature points from the first region of interest of the first image, and extracting a plurality of second feature points from the second region of interest of the second image;

Wherein, the size of the first ROI is the same as that of the second ROI, and their positions in different images correspond to each other. The first ROI is extracted from the first image according to the default size. The default size is set according to the user’s experience value. For example, the default size is A×A, and the unit is pixel or centimeter. Correspondingly, the extraction principle of the second ROI is the same as The principle of extracting the first region of interest is the same, and the present disclosure will not repeat them here.

As shown in Figure 4, Figure 4 is a schematic diagram of extracting a region of interest from an image, setting a window 402 with a size of A×A, and extracting from the upper left corner of the image 401 to obtain a region of interest 403 with a size of A×A .

It should be noted that in the present disclosure, multiple regions of interest of the image can be obtained by traversing the image according to a default size or a preset size. The mosaic combination of multiple ROIs is the original image before ROI extraction.

In some embodiments, the third region of interest in the first image after global image alignment processing and the fourth region of interest of a corresponding size in the second image are extracted according to a default size. Since the region of interest is selected according to the default size, the region of interest may be smaller and the feature points less, and the user needs to set the preset size. An embodiment of the present disclosure provides an implementation method. First, determine the third region of interest and Whether the fourth ROI is smaller than the preset size, if the third ROI and the fourth ROI are smaller than the preset size, expand the third ROI to obtain the first ROI, expand the fourth ROI to obtain the 2. The region of interest. Optionally, expand the boundary of the region of interest to obtain a region of interest with a larger size for feature point extraction.

In some embodiments, since there is an obvious depth difference between the scene in the image and the background, the scene in the image is detected as the target object, and the target object is first detected for the first region of interest and the second region of interest, from Determining a first target object graph from a first region of interest, and determining a second target object graph from a second region of interest; then identifying whether the first region of interest and the second region of interest contain the target object;

When the target object is detected in the first region of interest and the second region of interest, feature extraction is performed on the first object map and the second object map, and a plurality of first object images are extracted from the first object map. feature points, and extract a plurality of second feature points from the second target object graph. The first feature point is the first corner point, the second feature point is the second corner point, or the first feature point is the first ECC feature point, and the second feature point is the second ECC feature point.

In some embodiments, the scene in the image may be a plant, an animal, a building, and other objects of concern in the field of image processing. The following will introduce the local alignment process in the case that the scene in the image is a person.

In some embodiments, portrait detection is performed on the first region of interest and the second region of interest; after the portrait detection, the first portrait mask of the first region of interest and the portrait mask of the second region of interest are obtained; according to The first portrait mask and the second portrait mask determine the first portrait image from the first region of interest, and determine the second portrait image from the second region of interest, and then extract multiple A first feature point, and extracting a plurality of second feature points from the second portrait. Taking the portrait as the key information in the image, the deep learning algorithm is used to perform portrait detection on the region of interest extracted from the image, and the portrait image is extracted to achieve image alignment according to the portrait feature points in the portrait image.

In some embodiments, after the first portrait image is determined from the first region of interest, and the second portrait image is determined from the second region of interest, it is necessary to further verify whether the portrait image contains a portrait. The embodiments of the present disclosure provide One embodiment: identify whether the first portrait image and the second portrait image contain portraits; if the first portrait image and the second portrait image contain portraits, then extract a plurality of first features from the first portrait image points, and extract a plurality of second feature points from the second portrait, since the first portrait image and the second portrait image contain portraits, and the feature points in the portraits are obvious and large in number, in order to improve the efficiency of feature matching , optionally, the first feature point is the first corner point, and the second feature point is the second corner point. Using feature points as corner points ensures that when the number of feature points is large and obvious, the speed of feature matching is improved, and the reliability of feature matching is improved.

In some embodiments, if the first portrait image and the second portrait image contain portraits, after performing feature matching on the extracted corner points, the number of matched corner points obtained is less than the preset number, then according to the first person Calculate the first entropy correlation coefficient (Entropy Corrleation Coefficient, ECC), and then extract a plurality of first ECC feature points from the first portrait image, and extract a plurality of second ECC feature points from the second portrait image. In the case of identifying and determining that the portrait image contains a portrait, in order to avoid too few corner points used to calculate the homography matrix after feature matching, it is necessary to use the ECC feature matching method to ensure that the photometric distortion of contrast and brightness remains unchanged. , to extract ECC feature points to better perform feature matching on the basis of including portraits in the image, to calculate a homography matrix with higher accuracy, thereby improving the accuracy of image alignment and achieving better images Alignment effect.

In some embodiments, after the first portrait image is determined from the first region of interest, and the second portrait image is determined from the second region of interest, optionally, the histogram of the second portrait image is obtained, and the second portrait image is determined. The pixel value of each pixel in the two-person portrait; according to the pixel value of each pixel in the second portrait, set the pixel value of the pixel corresponding to the pixel position in the first portrait. After determining the portrait image in the image, adjust the brightness of the portrait image in the image to be processed according to the histogram of the portrait image extracted from the reference image, so as to reduce the influence of brightness difference on feature point extraction and improve the accuracy of feature matching, thereby improving The precision of image alignment.

When it is detected that there is no target object in the first region of interest and the second region of interest, first obtain the histogram of the second region of interest, determine the pixel value of each pixel in the second region of interest, Accordingly, the pixel value of each pixel in the first region of interest is set to adjust the brightness of the first region of interest using the second region of interest as a reference, and further extract at least one corner point from the first region of interest, and extract at least one corner point from the second region of interest, match the corner points of the extracted first region of interest and the second region of interest, and then calculate the first homography matrix, so that the first homography The property matrix is used for image alignment processing.

In some embodiments, when the extracted corner points of the first ROI and the second ROI are less than the preset number, since there is no target object angle between the first ROI and the second ROI The number of points is small. In order to avoid the error of the first homography matrix calculated when the number of matched corner points is less than the preset number is too large, which is not conducive to image alignment, the embodiment of the present disclosure provides an implementation mode. The region of interest extracts ECC feature points, and extracts ECC feature points from the second region of interest. The ECC feature points are feature points with high feature matching accuracy, so that there is no obvious difference between the first region of interest and the second ECC feature points. In the case of the target object, precise feature matching is carried out, and when there is no obvious scene in the image, feature matching is performed by extracting ECC feature points, which improves the accuracy of image alignment and obtains a better image alignment effect.

In the above embodiment, feature points are extracted by performing target object detection in the first image to obtain the target object map, wherein, when the target object map contains the target object, corner points are used as feature points for extraction, in order to ensure feature matching Accuracy, using ECC feature points as feature points for extraction; in the case that the target object map does not contain the target object, adjust the brightness according to the reference area of interest, and extract the corner points of the area of interest for subsequent use At least one feature match. Whether or not the image contains the target object is processed separately to ensure the accuracy of the extracted feature points for subsequent feature point matching, thereby improving the accuracy of image alignment.

S206. Perform feature matching on multiple first feature points and multiple second feature points, determine the coordinates of at least one first target feature point among the multiple first feature points, and determine at least one first target feature point among the multiple second feature points The coordinates of the two target feature points.

Wherein at least one first target feature point matches at least one second target feature point.

In some embodiments, the present disclosure performs feature matching according to the corner points extracted after the target object is detected.

In some embodiments, the determined number of target feature points is more than 3 pairs required for calculating the homography matrix.

In some embodiments, when the number of corner points does not reach the 3 pairs required to calculate the homography matrix, the present disclosure extracts ECC feature points according to the above steps for feature matching. The feature matching method includes but is not limited to accelerated robust features (Speeded Up Robust Feature, SURF), Scale Invariant Feature Transform (SIFT), Accelerated Segmentation Test Acquired Features (Features from Accelerated Segment Test, FAST), which are not limited in this disclosure.

In some embodiments, the present disclosure performs feature matching based on the corner points extracted after the target object is not detected, and performs at least one feature matching that is the same as in the global image alignment process, first in the case of a smaller-scale image that is blurred Next, perform feature matching on N corner points to obtain a rough matching result of the corner points, and then increase the scale to perform feature matching on M corner points until the scale is transformed to the size of the original image to complete the precise matching of the image. Wherein, N is a positive integer greater than M.

S208. Calculate a first homography matrix for pixel mapping according to the coordinates of at least one first target feature point and the coordinates of at least one second target feature point.

In some embodiments, after identifying the target object contained in the first target object map and the target object contained in the second target object map, the coordinates of at least one target feature point and the coordinates of at least one second target feature point are determined, wherein, The at least one first target feature point is at least one first ECC target feature point, and the at least one second target feature point is at least one second ECC target feature point determined by an ECC feature matching method.

In some embodiments, after identifying the target object contained in the first target object map and the target object contained in the second target object map, according to the target corner coordinates determined by Harris corner point detection and feature matching in the region of interest, use The random sampling consensus algorithm calculates the homography matrix and iteratively updates it, eliminates the wrong matching points, and determines the optimal homography matrix.

S210, based on the first homography matrix, map the pixel value of each pixel in the first region of interest to the corresponding pixel in the second region of interest, so as to obtain the first region of interest after image alignment.

In some embodiments, the aligned first region of interest is cropped and output. If the first ROI is an expanded ROI, clipping is performed according to the size of the expanded ROI, and the expanded ROI is output.

In some embodiments, the region of interest is extracted multiple times from the first image and the second image, and then steps S204-S210 are performed until each region of interest in the first image is image-aligned, and then the output All the images of the first image are aligned to the processed region of interest, so as to obtain the first image after complete image alignment. By dividing the first image into a plurality of regions of interest, and performing the image alignment operation on each of these regions of interest, precise image alignment is realized, and the accuracy of image alignment is improved.

To sum up, the present disclosure first performs global image alignment on the image to be processed, and then extracts feature points from the first region of interest in the image to be processed and the second region of interest in the reference image for feature matching, and obtains the coordinates of the matched feature points , so as to calculate the homography matrix, so as to map the first region of interest to the second region of interest according to the homography matrix to realize the local alignment of the image to be processed, and realize the to-be-processed image by aligning all regions of interest in the image to be processed The precise alignment of images reduces the influence of uncertain factors in the image shooting process, and can extract more feature points for image alignment, which is applicable to various scenarios and improves the accuracy of image alignment.

Fig. 5 is a flow chart 2 of the steps of the image alignment method provided by one or more embodiments of the present disclosure. Fig. 5 is a further expansion and optimization of the present disclosure based on the embodiment shown in Fig. 2 , wherein a part of S202 A possible implementation is as follows:

S502. Acquire a histogram of the second image, and determine a pixel value of each pixel in the second image.

S504. Set the pixel value of the pixel corresponding to the pixel position in the third image according to the pixel value of each pixel in the second image.

S506. According to the pixel value of each pixel in the second image and the set pixel value of each pixel in the third image, extract at least one first corner point in the third image, and extract at least one second corner point of .

S508. Perform a feature matching on at least one first corner point and at least one second corner point in units of N corner points.

S510. Perform feature matching again on at least one first corner point and at least one second corner point, using M corner points as a unit.

Wherein, N is a positive integer greater than M.

S512. Determine a first target corner point in at least one first corner point, and a second target corner point in at least one second corner point.

S514. Calculate a second homography matrix according to the coordinates of the first target corner point and the coordinates of the second target corner point;

S516. Based on the second homography matrix, map the pixel value of each pixel in the third image to the corresponding pixel in the second image to obtain the first image after global image alignment.

The first image is obtained by adjusting the brightness of the third image, which solves the problem of brightness difference between the image to be processed and the reference image, and improves the convenience of the feature matching process; iteratively calculates the optimal homography matrix through the random consensus algorithm, and thus obtains The first image after the global image alignment achieves a better effect of global image alignment; the third image is matched with the second image for multiple feature points, which reduces the false matching rate and improves the matching accuracy.

Fig. 6 is a flowchart three of steps of an image alignment method provided by one or more embodiments of the present disclosure. Fig. 6 is a further expansion and optimization of the present disclosure on the basis of the embodiment shown in Fig. 2 , wherein one of S205 A possible implementation is as follows:

S602. Perform target object detection on the first region of interest and the second region of interest.

S604. Determine a first target object graph from the first region of interest, and determine a second target object graph from the second region of interest.

S606. Acquire a histogram of the second target object map, and determine a pixel value of each pixel in the second target object map.

S608. Set the pixel value of the pixel corresponding to the pixel position in the first object image according to the pixel value of each pixel in the second object image.

S610. Identify whether the first target object graph and the second target object graph contain the target object.

S612, if the first target object graph and the second target object graph contain the target object, then extract a plurality of first ECC feature points from the first target object graph, and extract a plurality of second ECC feature points from the second target object graph Feature points.

By detecting the target object in the region of interest of the image, the target object map is obtained, and the brightness of the target object map corresponding to the image to be processed is adjusted to determine whether the important information of the target object is contained in the region of interest, and then extract the relevant information of the target object. ECC feature points, so as to extract obvious feature points for image alignment, reduce the influence of other interference factors in the image shooting process, and improve the accuracy of image alignment.

It should be understood that although the various steps in the flow charts in FIGS. 2-4 are displayed sequentially as indicated by the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in Figures 2-4 may include a plurality of sub-steps or stages, these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, these sub-steps or stages The order of execution is not necessarily performed sequentially, but may be performed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

Based on the same inventive concept, as the implementation of the above method, the embodiment of the present disclosure also provides an image alignment device, the device embodiment corresponds to the foregoing method embodiment, for the sake of easy reading, the present device embodiment does not implement the foregoing method The details in the examples are described one by one, but it should be clear that the device in this embodiment can correspondingly implement all the content in the foregoing method embodiments.

FIG. 7 is a structural block diagram of an image alignment device in one or more embodiments of the present disclosure. As shown in FIG. Module 708 and mapping module 710, wherein:

The acquisition module 702 is configured to acquire a first image and a second image, the first image is the first image after global image alignment processing based on the second image;

Extraction module 704, configured to extract a plurality of first feature points from the first region of interest of the first image, and extract a plurality of second feature points from the second region of interest of the second image; the first region of interest and the first region of interest The two regions of interest have the same size and correspond to the positions in different images;

The feature matching module 706 is configured to perform feature matching on a plurality of first feature points and a plurality of second feature points, determine the coordinates of at least one first target feature point in the plurality of first feature points, and determine a plurality of second feature points coordinates of at least one second target feature point in the points, at least one first target feature point matches at least one second target feature point;

The calculation module 708 is configured to calculate a first homography matrix configured as a pixel map according to the coordinates of at least one first target feature point and the coordinates of at least one second target feature point;

The mapping module 710 is configured to map the pixel value of each pixel in the first region of interest to the corresponding pixel in the second region of interest based on the first homography matrix, so as to obtain the first sense image after image alignment area of interest.

As an optional implementation of this embodiment of the present disclosure, the acquisition module 702 is configured to acquire the histogram of the second image, and determine the pixel value of each pixel in the second image; Set the pixel value of the pixel point corresponding to the pixel position in the third image; according to the pixel value of each pixel point in the second image and the pixel value of each pixel point in the third image after setting, extract the first At least one first corner point in the three images, and extracting at least one second corner point in the second image; performing feature matching on at least one first corner point and at least one second corner point, and determining at least one first corner point The first target corner point in , and the second target corner point in at least one second corner point, the first target corner point matches the second target corner point; according to the coordinates of the first target corner point and the second target corner point The coordinates of the second homography matrix are calculated; based on the second homography matrix, the pixel value of each pixel point in the third image is mapped to the corresponding pixel point of the second image, and the first one after global image alignment is obtained image.

As an optional implementation manner of this embodiment of the present disclosure, the acquisition module 702 is configured to perform feature matching on at least one first corner point and at least one second corner point in units of N corner points; The first corner point and at least one second corner point are used to perform feature matching again in units of M corner points; wherein, N is a positive integer greater than M.

As an optional implementation of this embodiment of the present disclosure, the acquisition module 702 is configured to, under the condition that the number of first target corner points is greater than or equal to the preset number, according to the coordinates of the first target corner point and the second target angle The coordinates of the points are used to calculate the second homography matrix; under the condition that the number of the first target corner points is less than the preset number, a plurality of first orientation fast rotation ORB feature points are extracted from the third image, and from the second image Extracting a plurality of second ORB feature points; performing feature matching with a plurality of first ORB feature points and a plurality of second ORB feature points, determining the coordinates of at least one first target ORB feature point in a plurality of first ORB feature points, and determining the coordinates of at least one second target ORB feature point among the plurality of second target ORB feature points, at least one first target ORB feature point matching with at least one second target ORB feature point; according to the at least one first target ORB feature point coordinates, and the coordinates of at least one second target ORB feature point, to calculate a second homography matrix.

As an optional implementation manner of this embodiment of the present disclosure, the extraction module 704 is configured to perform target object detection on the first region of interest and the second region of interest; determine the first target object graph from the first region of interest, and Determining a second object map from the second region of interest; extracting a plurality of first feature points from the first object map, and extracting a plurality of second feature points from the second object map.

Extraction module 704, configured to identify whether the first target object graph and the second target object graph contain the target object; under the condition that the first target object graph and the second target object graph contain the target object, extract A plurality of first ECC feature points are extracted from the image, and a plurality of second ECC feature points are extracted from the second target object graph.

As an optional implementation of this embodiment of the present disclosure, the extraction module 704 is configured to: acquire the histogram of the second target object map, and determine the pixel value of each pixel in the second target object map; For the pixel value of each pixel in the object map, set the pixel value of the pixel corresponding to the pixel position in the first target object map.

As an optional implementation of the embodiment of the present disclosure, the target object detection is portrait detection; the extraction module 704 is configured to obtain the first portrait mask of the first region of interest obtained after portrait detection, and the second A second portrait mask of the region; determining a first portrait map from the first image according to the first portrait mask, and determining a second portrait map from the second image according to the second portrait mask.

As an optional implementation manner of this embodiment of the present disclosure, the extracting module 704 is configured to extract the third ROI in the first image and the fourth ROI in the second image according to a default size; If the size of the ROI is smaller than the preset size, expand the third ROI according to the preset size to obtain the first ROI; if the size of the fourth ROI is smaller than the preset size, expand the third ROI according to the preset size Extend the fourth ROI to obtain the second ROI.

The image alignment device provided by the present disclosure first performs global image alignment on the image to be processed, and then extracts feature points from the first region of interest in the image to be processed and the second region of interest in the reference image to perform feature matching to obtain the matched feature points The coordinates of the homography matrix are calculated to map the first region of interest to the second region of interest according to the homography matrix to achieve local alignment of the image to be processed by aligning all regions of interest in the image to be processed The precise alignment of the images to be processed reduces the influence of uncertain factors in the image shooting process, and more feature points can be extracted for image alignment. It is suitable for various scenarios and improves the accuracy of image alignment.

The image alignment device provided in this embodiment can execute the image alignment method provided in the above method embodiment, and its implementation principle and technical effect are similar, and will not be repeated here. Each module in the above-mentioned image alignment device can be fully or partially realized by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of one or more processors in the computer device in the form of hardware, and can also be stored in the memory of the computer device in the form of software, so that one or more processors can call and execute the above-mentioned The operation corresponding to the module.

It is worth noting that in the above embodiment of the image alignment device, the included units and modules are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, each The specific names of the functional units are only for the convenience of distinguishing each other, and are not configured to limit the protection scope of the present disclosure.

In one or more embodiments, a computer device is provided. The computer device may be a terminal device, and its internal structure may be as shown in FIG. 8 . The computer device includes one or more processors, memory, communication interfaces, databases, display screens, and input devices connected by a system bus. Wherein, one or more processors of the computer device are configured as modules providing computing and control capabilities. The memory of the computer equipment includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer readable instructions. The internal memory provides an environment for the execution of the operating system and computer readable instructions in the non-volatile storage medium. The communication interface of the computer device is configured as a wired or wireless communication module with an external terminal, and the wireless mode can be realized through WIFI, operator network, near field communication (NFC) or other technologies. When the computer-readable instructions are executed by one or more processors, the image alignment method provided by the above-mentioned embodiments can be realized. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, a trackball or a touch pad provided on the casing of the computer device , and can also be an external keyboard, touchpad, or mouse.

Those skilled in the art can understand that the structure shown in FIG. 8 is only a block diagram of a partial structure related to the disclosed solution, and does not constitute a limitation to the computer equipment on which the disclosed solution is applied. The specific computer equipment can be More or fewer components than shown in the figures may be included, or some components may be combined, or have a different arrangement of components.

In one or more embodiments, the apparatus for displaying preview images provided by the present disclosure may be implemented in the form of computer-readable instructions, and the computer-readable instructions may be run on a computer device as shown in FIG. 8 . Various program modules that make up the computer device can be stored in the memory of the computer device, such as the acquisition module 702 , extraction module 704 , feature matching module 706 , calculation module 708 and mapping module 710 shown in FIG. 7 . Computer-readable instructions constituted by various program modules enable one or more processors to execute the steps in the image alignment method of any one embodiment of the present disclosure described in this specification.

For example, the computer device shown in FIG. 8 can execute the acquisition of the first image and the second image through the acquisition module 702 in the image alignment device as shown in FIG. first image. The computer device can perform extracting a plurality of first feature points from a first region of interest of a first image through an extraction module 704, and extracting a plurality of second feature points from a second region of interest of a second image; the first region of interest It is the same size as the second ROI and corresponds to its position in different images. The computer device may use the feature matching module 706 to perform feature matching on multiple first feature points and multiple second feature points, determine the coordinates of at least one first target feature point among the multiple first feature points, and determine multiple first feature points. The coordinates of at least one second target feature point among the two feature points, at least one first target feature point matches at least one second target feature point. The computer device can calculate the first homography matrix for pixel mapping according to the coordinates of at least one first target feature point and the coordinates of at least one second target feature point through the calculation module 708 . The computer device may use the mapping module 710 to perform mapping of the pixel value of each pixel in the first region of interest to the corresponding pixel in the second region of interest based on the first homography matrix, so as to obtain the first aligned image. a region of interest.

In one or more embodiments, there is provided a computer device comprising a memory and one or more processors, the memory storing computer readable instructions which, when executed by the one or more processors, cause One or more processors execute the steps of the image alignment method provided in any one embodiment of the present disclosure.

The computer device provided in this embodiment can implement the image alignment method provided in the above method embodiment, and its implementation principle is similar to the technical effect, and will not be repeated here.

Wherein, the computer-readable storage medium may be a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and the like.

The computer-readable instructions stored on the computer-readable storage medium provided by this embodiment can realize the image alignment method provided by the above-mentioned method embodiment, and its implementation principle is similar to the technical effect, and will not be repeated here.

Those of ordinary skill in the art can understand that the implementation of all or part of the processes in the above method embodiments can be completed by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a non-volatile computer-readable When the computer-readable instructions are executed, the computer-readable instructions may include the processes of the embodiments of the above-mentioned methods. Wherein, any reference to storage, database or other media used in the various embodiments provided by the present disclosure may include at least one of non-volatile and volatile storage. Non-volatile memory may include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms such as Static Random Access Memory (SRAM) and Dynamic Random Access Memory (DRAM), among others.

The technical features of the above embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be It is considered to be within the range described in this specification.

The above examples only express several implementations of the present disclosure, and the descriptions thereof are more specific and detailed, but should not be construed as limiting the scope of the patent for the invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present disclosure, and these all belong to the protection scope of the present disclosure. Therefore, the scope of protection of the disclosed patent should be based on the appended claims.

Industrial Applicability

The image alignment method provided in this disclosure achieves precise alignment of the image to be processed by aligning all regions of interest in the image to be processed, reduces the influence of uncertain factors in the image shooting process, and can extract more feature points for image alignment , is applicable to various scenarios, improves the accuracy of image alignment, and has strong industrial applicability.

Claims

A method for image alignment, characterized in that the method comprises:

Acquiring a first image and a second image, the first image being a first image after global image alignment processing based on the second image;

extracting a plurality of first feature points from a first region of interest in the first image, and extracting a plurality of second feature points from a second region of interest in the second image; the first region of interest and the The sizes of the second regions of interest are the same, and their positions in different images correspond to each other;

performing feature matching on the plurality of first feature points and the plurality of second feature points, determining the coordinates of at least one first target feature point in the plurality of first feature points, and determining the plurality of second feature points coordinates of at least one second target feature point among the feature points, the at least one first target feature point matching the at least one second target feature point;

calculating a first homography matrix for pixel mapping according to the coordinates of the at least one first target feature point and the coordinates of the at least one second target feature point;

Based on the first homography matrix, the pixel value of each pixel point in the first region of interest is mapped to the corresponding pixel point in the second region of interest, so as to obtain the first sense image after image alignment. area of interest.
The method according to claim 1, wherein said obtaining the first image comprises:

Obtain the histogram of the second image, and determine the pixel value of each pixel in the second image;

Set the pixel value of the pixel corresponding to the pixel position in the third image according to the pixel value of each pixel in the second image;

Extract at least one first corner point in the third image according to the pixel value of each pixel point in the second image and the set pixel value of each pixel point in the third image, and extract the at least one second corner point in the second image;

performing feature matching on the at least one first corner point and the at least one second corner point, determining a first target corner point in the at least one first corner point, and a target corner point in the at least one second corner point a second target corner, the first target corner matching the second target corner;

calculating a second homography matrix according to the coordinates of the first target corner point and the coordinates of the second target corner point;

Based on the second homography matrix, the pixel value of each pixel in the third image is mapped to the corresponding pixel in the second image to obtain the first image after global image alignment.
The method according to claim 2, wherein the at least one feature matching is multiple times, and performing feature matching on the at least one first corner point and the at least one second corner point includes:

performing a feature matching on the at least one first corner point and the at least one second corner point in units of N corner points;

performing feature matching on the at least one first corner point and the at least one second corner point in units of M corner points;

Wherein, N is a positive integer greater than M.
The method according to claim 2, wherein said calculating a second homography matrix according to the coordinates of the first target corner point and the coordinates of the second target corner point comprises:

Under the condition that the number of the first target corner points is greater than or equal to a preset number, the second homography matrix is calculated according to the coordinates of the first target corner point and the coordinates of the second target corner point ;

The method also includes:

Under the condition that the number of the first target corner points is less than a preset number, extracting a plurality of first directional fast rotation ORB feature points from the third image, and extracting a plurality of second ORB feature points from the second image ORB feature points;

performing feature matching on the plurality of first ORB feature points and the plurality of second ORB feature points, determining the coordinates of at least one first target ORB feature point among the plurality of first ORB feature points, and determining the coordinates of at least one second target ORB feature point among the plurality of second ORB feature points, the at least one first target ORB feature point matching the at least one second target ORB feature point;

The second homography matrix is calculated according to the coordinates of the at least one first target ORB feature point and the coordinates of the at least one second target ORB feature point.
The method according to claim 1, wherein said extracting a plurality of first feature points from a first region of interest of said first image, and extracting a plurality of first feature points from a second region of interest of said second image Two feature points, including:

performing target object detection on the first region of interest and the second region of interest;

determining a first target object map from the first region of interest, and determining a second target object map from the second region of interest;

A plurality of first feature points are extracted from the first target object graph, and a plurality of second feature points are extracted from the second target object graph.
The method according to claim 5, wherein, the first feature point is a first entropy correlation coefficient ECC feature point, the second feature point is a second ECC feature point, and the Extract a plurality of first feature points from the image, and extract a plurality of second feature points from the second target object map, including:

identifying whether a target object is contained in the first target object graph and the second target object graph;

Under the condition that the first target object graph and the second target object graph contain a target object, extract a plurality of the first ECC feature points from the first target object graph, and extract a plurality of the first ECC feature points from the second target object graph A plurality of the second ECC feature points are extracted from the target object map.
The method according to claim 5, wherein, after determining the first object map from the first region of interest and determining the second object map from the second region of interest, further comprising:

Acquiring a histogram of the second target object map, and determining the pixel value of each pixel in the second target object map;

Set the pixel value of the pixel corresponding to the pixel position in the first object image according to the pixel value of each pixel in the second object image.
The method according to claim 5, wherein said target object detection is human portrait detection, said determining a first target object map from said first region of interest, and determining a second target from said second region of interest Object diagrams, including:

Acquiring the first portrait mask of the first region of interest obtained after the portrait detection, and the second portrait mask of the second region of interest;

A first portrait image is determined from the first image according to the first portrait mask, and a second portrait image is determined from the second image according to the second portrait mask.
The method according to claim 1, wherein the method further comprises:

extracting a third region of interest in the first image and a fourth region of interest in the second image according to a default size;

If the size of the third region of interest is smaller than a preset size, expanding the third region of interest according to the preset size to obtain the first region of interest;

If the size of the fourth region of interest is smaller than the preset size, the fourth region of interest is expanded according to the preset size to obtain the second region of interest.
An image alignment device, characterized in that it comprises:

An acquisition module configured to acquire a first image and a second image, the first image being a first image after global image alignment processing based on the second image;

An extraction module configured to extract a plurality of first feature points from a first region of interest of the first image, and extract a plurality of second feature points from a second region of interest of the second image; the first The size of the region of interest is the same as that of the second region of interest, and the positions in different images correspond to each other;

A feature matching module configured to perform feature matching on the plurality of first feature points and the plurality of second feature points, determine the coordinates of at least one first target feature point among the plurality of first feature points, and determine coordinates of at least one second target feature point among the plurality of second feature points, the at least one first target feature point matching the at least one second target feature point;

A calculation module configured to calculate a first homography matrix for pixel mapping according to the coordinates of the at least one first target feature point and the coordinates of the at least one second target feature point;

A mapping module configured to map the pixel value of each pixel in the first region of interest to a corresponding pixel in the second region of interest based on the first homography matrix, so as to obtain an image alignment After the first region of interest.
The device according to claim 10, wherein the acquisition module is configured to:

Obtain the histogram of the second image, and determine the pixel value of each pixel in the second image;

Set the pixel value of the pixel corresponding to the pixel position in the third image according to the pixel value of each pixel in the second image;

Extract at least one first corner point in the third image according to the pixel value of each pixel point in the second image and the set pixel value of each pixel point in the third image, and extract the at least one second corner point in the second image;

performing feature matching on the at least one first corner point and the at least one second corner point, determining a first target corner point in the at least one first corner point, and a target corner point in the at least one second corner point a second target corner, the first target corner matching the second target corner;

calculating a second homography matrix according to the coordinates of the first target corner point and the coordinates of the second target corner point;

Based on the second homography matrix, the pixel value of each pixel in the third image is mapped to the corresponding pixel in the second image to obtain the first image after global image alignment.
The device according to claim 11, wherein the acquisition module is configured to:

performing a feature matching on the at least one first corner point and the at least one second corner point in units of N corner points;

performing feature matching on the at least one first corner point and the at least one second corner point in units of M corner points;

Wherein, N is a positive integer greater than M.
The device according to claim 11, wherein the acquisition module is configured to:

Under the condition that the number of the first target corner points is greater than or equal to a preset number, the second homography matrix is calculated according to the coordinates of the first target corner point and the coordinates of the second target corner point ;

Under the condition that the number of the first target corner points is less than a preset number, extracting a plurality of first directional fast rotation ORB feature points from the third image, and extracting a plurality of second ORB feature points from the second image ORB feature points;

performing feature matching on the plurality of first ORB feature points and the plurality of second ORB feature points, determining the coordinates of at least one first target ORB feature point among the plurality of first ORB feature points, and determining the coordinates of at least one second target ORB feature point among the plurality of second ORB feature points, the at least one first target ORB feature point matching the at least one second target ORB feature point;

The second homography matrix is calculated according to the coordinates of the at least one first target ORB feature point and the coordinates of the at least one second target ORB feature point.
The device according to claim 10, wherein the extraction module is configured to:

performing target object detection on the first region of interest and the second region of interest;

determining a first target object map from the first region of interest, and determining a second target object map from the second region of interest;

A plurality of first feature points are extracted from the first target object graph, and a plurality of second feature points are extracted from the second target object graph.
The device according to claim 14, wherein the first feature point is a first entropy correlation coefficient ECC feature point, and the second feature point is a second ECC feature point;

The extraction module is configured as:

identifying whether a target object is contained in the first target object graph and the second target object graph;

Under the condition that the first target object graph and the second target object graph contain a target object, extract a plurality of the first ECC feature points from the first target object graph, and extract a plurality of the first ECC feature points from the second target object graph A plurality of the second ECC feature points are extracted from the target object map.
The device according to claim 14, wherein the extraction module is configured to:

Acquiring a histogram of the second target object map, and determining the pixel value of each pixel in the second target object map;

Set the pixel value of the pixel corresponding to the pixel position in the first object image according to the pixel value of each pixel in the second object image.
The device according to claim 14, wherein the target object detection is portrait detection;

The extraction module is configured to: acquire a first portrait mask of the first region of interest obtained after portrait detection, and a second portrait mask of the second region of interest;

A first portrait image is determined from the first image according to the first portrait mask, and a second portrait image is determined from the second image according to the second portrait mask.
The device according to claim 10, wherein the extraction module is configured to:

extracting a third region of interest in the first image and a fourth region of interest in the second image according to a default size;

If the size of the third region of interest is smaller than a preset size, expanding the third region of interest according to the preset size to obtain the first region of interest;

If the size of the fourth region of interest is smaller than the preset size, the fourth region of interest is expanded according to the preset size to obtain the second region of interest.
A computer device comprising: a memory and one or more processors, the memory having computer-readable instructions stored therein; when executed by the one or more processors, the computer-readable instructions cause the one or more A plurality of processors execute the steps of the image alignment method described in any one of claims 1-9.
One or more non-transitory computer-readable storage media storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform claim 1 The steps of the image alignment method described in any one of -9.