WO2023098045A1 - Procédé et appareil d'alignement d'image, et dispositif informatique et support de stockage - Google Patents

Procédé et appareil d'alignement d'image, et dispositif informatique et support de stockage Download PDF

Info

Publication number
WO2023098045A1
WO2023098045A1 PCT/CN2022/100804 CN2022100804W WO2023098045A1 WO 2023098045 A1 WO2023098045 A1 WO 2023098045A1 CN 2022100804 W CN2022100804 W CN 2022100804W WO 2023098045 A1 WO2023098045 A1 WO 2023098045A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
target
interest
region
point
Prior art date
Application number
PCT/CN2022/100804
Other languages
English (en)
Chinese (zh)
Inventor
蒋海峰
Original Assignee
上海闻泰信息技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海闻泰信息技术有限公司 filed Critical 上海闻泰信息技术有限公司
Publication of WO2023098045A1 publication Critical patent/WO2023098045A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions

Definitions

  • the present disclosure relates to an image alignment method, device, computer equipment and storage medium.
  • Image alignment plays an important role in image fusion, image stitching, and computer vision. It is to align two or more images at different times or from different angles to determine the mapping relationship between the spatial positions and intensities of the images. In this way, multiple frames of images shot continuously in a very short period of time are fused into one frame of image.
  • the feature-based image alignment method in the related art first finds the feature points in the two pictures and performs feature matching, and then calculates the homography matrix to uniformly transform the pixel coordinates in the images to perform image alignment. There are uncertain factors such as illumination, depth of field, and angle in the process, which makes it difficult to extract obvious feature points from the image to be processed, which makes the method's scene adaptability poor and the accuracy of image alignment low.
  • the method solves the problem in the related art that the accuracy of image alignment is not high due to inconspicuous feature points.
  • an image alignment method, device, computer equipment, and storage medium are provided.
  • An image alignment method comprising:
  • the pixel value of each pixel in the first region of interest is mapped to the corresponding pixel in the second region of interest, so as to obtain the first region of interest after image alignment.
  • acquiring the first image includes: acquiring a histogram of the second image, and determining the pixel value of each pixel in the second image; Set the pixel value of the pixel point corresponding to the pixel position in the third image; according to the pixel value of each pixel point in the second image and the pixel value of each pixel point in the third image after setting, extract At least one first corner point in the third image, and extract at least one second corner point in the second image; perform feature matching on at least one first corner point and at least one second corner point, and determine at least one first corner point A first target corner point among the points, and a second target corner point among at least one second corner point, the first target corner point matches the second target corner point according to the coordinates of the first target corner point and the second target corner point
  • the coordinates of the second homography matrix are calculated; based on the second homography matrix, the pixel value of each pixel point in the third image is mapped to the corresponding pixel point of the second image, and
  • the number of at least one feature matching is multiple, and performing feature matching on at least one first corner point and at least one second corner point includes: combining at least one first corner point Perform feature matching with at least one second corner point in units of N corner points; perform feature matching again with at least one first corner point and at least one second corner point in units of M corner points; Wherein, N is a positive integer greater than M.
  • calculating the second homography matrix according to the coordinates of the first target corner and the coordinates of the second target corner includes: if the number of the first target corner is greater than or equal to the preset number, then calculate the second homography matrix according to the coordinates of the first target corner point and the coordinates of the second target corner point; the method also includes: if the number of the first target corner points is less than the preset number, then Extract a plurality of first ORB feature points from the third image, and extract a plurality of second ORB feature points from the second image; perform feature matching with a plurality of first ORB feature points and a plurality of second ORB feature points, and determine the plurality of ORB feature points The coordinates of at least one first target ORB feature point among the first ORB feature points, and the coordinates of at least one second target ORB feature point among a plurality of second ORB feature points, at least one first target ORB feature point and at least one Matching the second target ORB feature points; calculating
  • multiple first feature points are extracted from a first region of interest in the first image
  • multiple second feature points are extracted from a second region of interest in the second image
  • a plurality of first feature points are extracted from the object graph
  • a plurality of second feature points are extracted from the second target object.
  • extracting a plurality of first feature points from the first target object map, and extracting a plurality of second feature points from the second target object includes: identifying the first target object Whether the target object is included in the figure and the second target object figure; if the target object is contained in the first target object figure and the second target object figure, then a plurality of first ECC feature points are extracted from the first target object figure, and from the second target object figure A plurality of second ECC feature points are extracted from the target object graph.
  • after determining the first target object graph from the first region of interest and determining the second target object graph from the second region of interest further includes: acquiring the second target object graph and determine the pixel value of each pixel in the second object image; according to the pixel value of each pixel in the second object image, set the pixel corresponding to the pixel position in the first object image value.
  • the target object detection is portrait detection, determining the first target object map from the first region of interest, and determining the second target object map from the second region of interest, including: obtaining A first portrait mask of the first region of interest obtained after the portrait detection, and a second portrait mask of the second region of interest; determining the first portrait image from the first image according to the first portrait mask, And determining a second portrait image from the second image according to the second portrait mask.
  • the method further includes: extracting a third region of interest in the first image and a fourth region of interest in the second image according to a default size; If the size of the ROI is smaller than the preset size, expand the third ROI according to the preset size to obtain the first ROI; if the size of the fourth ROI is smaller than the preset size, expand the third ROI according to the preset size Extend the fourth ROI to obtain the second ROI.
  • An image alignment device comprising:
  • An acquisition module configured to acquire a first image and a second image, where the first image is a first image after global image alignment processing based on the second image;
  • An extraction module configured to extract a plurality of first feature points from a first region of interest in the first image, and extract a plurality of second feature points from a second region of interest in the second image; the first region of interest and the second The ROIs have the same size and correspond to positions in different images;
  • a feature matching module configured to perform feature matching on a plurality of first feature points and a plurality of second feature points, determine the coordinates of at least one first target feature point in the plurality of first feature points, and determine a plurality of second feature points Coordinates of at least one second target feature point, at least one first target feature point matches at least one second target feature point;
  • a calculation module configured to calculate a first homography matrix for pixel mapping according to the coordinates of at least one first target feature point and the coordinates of at least one second target feature point;
  • a mapping module configured to map the pixel value of each pixel in the first region of interest to the corresponding pixel in the second region of interest based on the first homography matrix, so as to obtain the first image of interest after image alignment area.
  • the acquiring module is configured to acquire a histogram of the second image, and determine the pixel value of each pixel in the second image; according to the histogram of each pixel in the second image Pixel value, set the pixel value of the pixel point corresponding to the pixel position in the third image; according to the pixel value of each pixel point in the second image, and the pixel value of each pixel point in the third image after setting, extract the third At least one first corner point in the image, and extracting at least one second corner point in the second image; performing feature matching on at least one first corner point and at least one second corner point, and determining at least one of the first corner points The first target corner point of , and the second target corner point of at least one second corner point, the first target corner point matches the second target corner point; according to the coordinates of the first target corner point and the second target corner point coordinates, calculate the second homography matrix; based on the second homography matrix, map the pixel value of each pixel point in
  • the acquisition module is configured to perform feature matching once with at least one first corner point and at least one second corner point in units of N corner points; One corner point and at least one second corner point, with M corner points as the unit, perform feature matching again;
  • N is a positive integer greater than M.
  • the obtaining module is configured to, under the condition that the number of the first target corner points is greater than or equal to the preset number, according to the coordinates of the first target corner point and the second target corner point The coordinates of , calculate the second homography matrix; under the condition that the number of the first target corner points is less than the preset number, extract a plurality of first directional fast rotation ORB feature points from the third image, and extract from the second image Extracting a plurality of second ORB feature points; performing feature matching on a plurality of first ORB feature points with a plurality of second ORB feature points, determining the coordinates of at least one first target ORB feature point in the plurality of first ORB feature points, and determining the coordinates of at least one second target ORB feature point among the plurality of second target ORB feature points, at least one first target ORB feature point matching at least one second target ORB feature point; according to the coordinates of at least one first target ORB feature point , and the coordinates of
  • the extraction module is configured to perform target object detection on the first region of interest and the second region of interest; determine the first target object graph from the first region of interest, and obtain the The second region of interest determines a second object map; extracts a plurality of first feature points from the first object map, and extracts a plurality of second feature points from the second object map.
  • the first feature point is a first entropy correlation coefficient ECC feature point
  • the second feature point is a second ECC feature point
  • An extraction module configured to identify whether the first target object graph and the second target object graph contain the target object; under the condition that the first target object graph and the second target object graph contain the target object, extract the A plurality of first ECC feature points are extracted, and a plurality of second ECC feature points are extracted from the second target object map.
  • the extraction module is configured to obtain a histogram of the second target object graph, and determine the pixel value of each pixel in the second target object graph; according to the second target object graph The pixel value of each pixel in the set, set the pixel value of the pixel corresponding to the pixel position in the first object map.
  • the target object detection is portrait detection
  • the extraction module is configured to obtain the first portrait mask of the first region of interest obtained after the portrait detection, and the second region of interest A second portrait mask; determining a first portrait image from the first image according to the first portrait mask, and determining a second portrait map from the second image according to the second portrait mask.
  • the extraction module is configured to extract the third region of interest in the first image and the fourth region of interest in the second image according to a default size; in the third region of interest If the size of the area is smaller than the preset size, expand the third ROI according to the preset size to obtain the first ROI; if the size of the fourth ROI is smaller than the preset size, expand according to the preset size The fourth ROI is used to obtain the second ROI.
  • a computer device comprising a memory and one or more processors, the memory configured as a module storing computer readable instructions; the computer readable instructions, when executed by the one or more processors, cause the One or more processors execute the steps of providing the image alignment method in any one embodiment of the present disclosure.
  • One or more non-volatile storage media storing computer-readable instructions.
  • the computer-readable instructions are executed by one or more processors, one or more processors execute the image provided in any one of the embodiments of the present disclosure. Align the steps of the method.
  • FIG. 1 is an application scenario diagram 1 of an image alignment method provided by one or more embodiments of the present disclosure
  • FIG. 2 is a flow chart 1 of steps of an image alignment method provided by one or more embodiments of the present disclosure
  • FIG. 3 is the second application scene diagram of the image alignment method provided by one or more embodiments of the present disclosure.
  • FIG. 4 is a schematic diagram of extracting a region of interest provided by one or more embodiments of the present disclosure
  • FIG. 5 is a second flow chart of the steps of the image alignment method provided by one or more embodiments of the present disclosure.
  • FIG. 6 is a flowchart three of steps of an image alignment method provided by one or more embodiments of the present disclosure.
  • Fig. 7 is a structural block diagram of an image alignment device in one or more embodiments of the present disclosure.
  • FIG. 8 is an internal structural diagram of a computer device in one or more embodiments of the present disclosure.
  • first and second and the like in the specification and claims of the present disclosure are used to distinguish different objects, rather than to describe a specific order of objects.
  • first camera and the second camera are used to distinguish different cameras, not to describe a specific order of the cameras. .
  • words such as “exemplary” or “for example” are used as examples, illustrations or illustrations. Any embodiment or design described as “exemplary” or “for example” in the embodiments of the present disclosure shall not be construed as being preferred or advantageous over other embodiments or designs. To be precise, the use of words such as “exemplary” or “for example” is intended to present related concepts in a specific manner. In addition, in the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "plurality” refers to two one or more.
  • the feature-based image alignment method first finds the feature points in the two pictures and performs feature matching, and then calculates the homography matrix to uniformly transform the pixel coordinates in the image for image alignment, but in the actual operation process, due to There are uncertain factors such as illumination, depth of field, and angle in the image shooting process, which lead to obvious time stamps of images that need to be processed, obvious depth of field, large brightness differences, large field of view differences, and large changes in image space, making it difficult to extract obvious feature points.
  • This method has poor scene adaptability and low accuracy of image alignment.
  • the present disclosure first performs global image alignment on the image to be processed, and then extracts feature points from the first region of interest in the image to be processed and the second region of interest in the reference image for feature matching to obtain the matched feature points
  • the coordinates of the homography matrix are calculated to map the first region of interest to the second region of interest according to the homography matrix to achieve local alignment of the image to be processed by aligning all regions of interest in the image to be processed.
  • the precise alignment of the images to be processed reduces the influence of uncertain factors in the image shooting process, and can extract more feature points for image alignment, which is applicable to various scenarios and improves the accuracy of image alignment.
  • the image alignment method provided in the present disclosure can be applied to the application environment shown in FIG. 1 .
  • the image alignment method is applied in an image alignment system.
  • the image alignment system includes a terminal 102 and a server 104 . Wherein, the terminal 102 communicates with the server 104 through a network. Specifically, the image alignment method is applied to the terminal 102.
  • the terminal 102 performs global image alignment on the image to be processed, and then extracts feature points from the first region of interest in the image to be processed and the second region of interest in the reference image to perform feature Matching to obtain the coordinates of the matched feature points, thereby calculating the homography matrix to map the first region of interest to the second region of interest according to the homography matrix to achieve local alignment of the image to be processed, by aligning the image to be processed All regions of interest in the image to achieve precise alignment of the image to be processed.
  • the image alignment method is applied to the server 104, the server 104 receives the image to be processed 106 sent by the terminal 102, and accurately aligns the image through the above image alignment method, and then the server 104 returns the precisely aligned image 108 to the terminal 102, the The method also includes steps S202-S210:
  • the terminal 102 can be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices, and the server 104 can be realized by an independent server or a server cluster composed of multiple servers.
  • FIG. 2 is a flow chart 1 of steps of an image alignment method provided by one or more embodiments of the present disclosure.
  • the present disclosure is mainly illustrated by taking the method applied to the terminal 102 in FIG. 1 as an example.
  • the first image is an image obtained by performing global image alignment processing on the third image with the second image as a reference image
  • the third image is an image obtained from a terminal device as a comparison device, which requires global image alignment processing
  • the second image is an image obtained from the terminal device as a reference device
  • the second image and the third image are images of the same scene at different angles, or images of the same scene at different times.
  • the second image and the third image are acquired from different terminal devices.
  • the second image is a reference image
  • the third image is an image to be processed that requires global image alignment processing.
  • the devices included in the figure include a mobile phone 301 , a mobile phone 303 and a server 305 ; the images include a second image 302 , a third image 304 , and a first image 306 .
  • the second image 302 is obtained from the mobile phone 301 as the reference machine
  • the third image 304 is obtained from the mobile phone 304 as the comparison machine
  • the server 305 uses the second image 302 as the reference image to perform global image alignment processing on the third image to obtain the second image.
  • An image 306 uses the second image 302 as the reference image to perform global image alignment processing on the third image to obtain the second image.
  • the following will introduce the process of using the second image as a reference image to perform global image processing on the third image to obtain the first image.
  • the histogram of the second image is obtained. Since the histogram represents the number of pixels of each brightness level of the image, it shows the The distribution in , intuitively reflects the brightness of the picture, so the pixel value of each pixel in the second image can be determined according to the histogram of the second image; then according to the pixel value of each pixel in the second image, use Set the pixel value of the pixel point corresponding to the pixel position in the third image by histogram equalization method, histogram specification, etc., so as to adjust the brightness of the third image according to the second image, so that the pixel value of the pixel point corresponding to the pixel position in the third image
  • the pixel value is set to be the same as the pixel value of the pixel point corresponding to the corresponding pixel position in the second image, so that the brightness of the third image is the same as that of the first image, so as to extract feature points from the two pictures.
  • the brightness of the third image may be adjusted by methods such as smoothing and denoising, illumination equalization processing, etc., including but not limited to the above methods, which are not limited in the present disclosure.
  • each pixel in the second image and the set pixel value of each pixel in the third image extract at least one corner point in the brightness-adjusted third image, and extract the second At least one corner point in the image, and match the corner points of the two extracted images.
  • the corner point is the isolated point with the greatest or smallest intensity on certain attributes, the end point of the line segment, or the point with the largest local curvature on the curve, which is used to describe the corners, boundary points and other information in the image.
  • the corner points are extracted by the Harris corner detection method. It should be noted that the basic idea of the Harris corner detection method is to use a fixed window to slide in any direction on the image, and compare the before and after sliding. In both cases, the pixel grayscale changes in the window. If there is a sliding in any direction and there is a large grayscale change, it is considered that there is a corner in the window.
  • the corner points are extracted by the Harris corner point detection method, which improves the robustness to noise and improves the efficiency and stability of corner point detection.
  • the corner points in the extracted third image and the corner points in the second image are subjected to feature matching at least once, and firstly, when the image with a smaller scale is blurred , perform feature matching on N corner points to obtain a rough matching result of the corner points, and then increase the scale to perform feature matching on M corner points until the scale is transformed to the size of the original image to complete the precise matching of the image, where , N is a positive integer greater than M.
  • the coarse matching and precise matching of the third image and the second image are realized, the false matching rate is reduced, and the matching accuracy is improved.
  • the second homography matrix is a homography matrix used for global image processing on the third image.
  • a plurality of Oriented Fast and Rotated Brief Oriented Fast and Rotated Brief, ORB
  • Oriented Fast and Rotated Brief ORB
  • second ORB feature points from the second image
  • carry out feature matching with a plurality of first ORB feature points and a plurality of second ORB feature points and determine among a plurality of first ORB feature points coordinates of at least one first target ORB feature point, and determining coordinates of at least one second target ORB feature point among a plurality of second target ORB feature points, the at least one first target ORB feature point matching at least one second target ORB feature point , and then, according to the coordinates of at least one first target ORB feature point and the coordinates of at least one second target ORB feature point, calculate the second homography matrix.
  • ORB feature points are extracted for feature matching, the matched ORB feature points are determined, and the coordinates corresponding to the ORB feature points are obtained, thereby calculating the homography matrix, which improves the accuracy and efficiency of feature matching.
  • the present disclosure uses a random sampling consensus algorithm to calculate the homography matrix and perform iterative update to eliminate mismatching points and determine the optimal homography matrix.
  • the problem of the difference in brightness between the image to be processed and the reference image is solved, and the convenience of the feature matching process is improved; the feature points of the third image and the second image are matched multiple times to reduce errors.
  • the matching rate improves the accuracy of matching; the optimal homography matrix is iteratively calculated through the random consensus algorithm, so as to obtain the first image after global image alignment, and achieve a better effect of global image alignment.
  • the size of the first ROI is the same as that of the second ROI, and their positions in different images correspond to each other.
  • the first ROI is extracted from the first image according to the default size.
  • the default size is set according to the user’s experience value.
  • the default size is A ⁇ A, and the unit is pixel or centimeter.
  • the extraction principle of the second ROI is the same as The principle of extracting the first region of interest is the same, and the present disclosure will not repeat them here.
  • Figure 4 is a schematic diagram of extracting a region of interest from an image, setting a window 402 with a size of A ⁇ A, and extracting from the upper left corner of the image 401 to obtain a region of interest 403 with a size of A ⁇ A .
  • multiple regions of interest of the image can be obtained by traversing the image according to a default size or a preset size.
  • the mosaic combination of multiple ROIs is the original image before ROI extraction.
  • the third region of interest in the first image after global image alignment processing and the fourth region of interest of a corresponding size in the second image are extracted according to a default size. Since the region of interest is selected according to the default size, the region of interest may be smaller and the feature points less, and the user needs to set the preset size.
  • An embodiment of the present disclosure provides an implementation method. First, determine the third region of interest and Whether the fourth ROI is smaller than the preset size, if the third ROI and the fourth ROI are smaller than the preset size, expand the third ROI to obtain the first ROI, expand the fourth ROI to obtain the 2. The region of interest. Optionally, expand the boundary of the region of interest to obtain a region of interest with a larger size for feature point extraction.
  • the scene in the image is detected as the target object, and the target object is first detected for the first region of interest and the second region of interest, from Determining a first target object graph from a first region of interest, and determining a second target object graph from a second region of interest; then identifying whether the first region of interest and the second region of interest contain the target object;
  • first object map and the second object map When the target object is detected in the first region of interest and the second region of interest, feature extraction is performed on the first object map and the second object map, and a plurality of first object images are extracted from the first object map. feature points, and extract a plurality of second feature points from the second target object graph.
  • the first feature point is the first corner point
  • the second feature point is the second corner point
  • the first feature point is the first ECC feature point
  • second feature point is the second ECC feature point.
  • the scene in the image may be a plant, an animal, a building, and other objects of concern in the field of image processing.
  • the following will introduce the local alignment process in the case that the scene in the image is a person.
  • portrait detection is performed on the first region of interest and the second region of interest; after the portrait detection, the first portrait mask of the first region of interest and the portrait mask of the second region of interest are obtained; according to The first portrait mask and the second portrait mask determine the first portrait image from the first region of interest, and determine the second portrait image from the second region of interest, and then extract multiple A first feature point, and extracting a plurality of second feature points from the second portrait.
  • the deep learning algorithm is used to perform portrait detection on the region of interest extracted from the image, and the portrait image is extracted to achieve image alignment according to the portrait feature points in the portrait image.
  • the embodiments of the present disclosure provide One embodiment: identify whether the first portrait image and the second portrait image contain portraits; if the first portrait image and the second portrait image contain portraits, then extract a plurality of first features from the first portrait image points, and extract a plurality of second feature points from the second portrait, since the first portrait image and the second portrait image contain portraits, and the feature points in the portraits are obvious and large in number, in order to improve the efficiency of feature matching , optionally, the first feature point is the first corner point, and the second feature point is the second corner point. Using feature points as corner points ensures that when the number of feature points is large and obvious, the speed of feature matching is improved, and the reliability of feature matching is improved.
  • the number of matched corner points obtained is less than the preset number, then according to the first person Calculate the first entropy correlation coefficient (Entropy Corrleation Coefficient, ECC), and then extract a plurality of first ECC feature points from the first portrait image, and extract a plurality of second ECC feature points from the second portrait image.
  • ECC Entropy Corrleation Coefficient
  • the histogram of the second portrait image is obtained, and the second portrait image is determined.
  • the pixel value of each pixel in the two-person portrait according to the pixel value of each pixel in the second portrait, set the pixel value of the pixel corresponding to the pixel position in the first portrait.
  • the pixel value of each pixel in the first region of interest is set to adjust the brightness of the first region of interest using the second region of interest as a reference, and further extract at least one corner point from the first region of interest, and extract at least one corner point from the second region of interest, match the corner points of the extracted first region of interest and the second region of interest, and then calculate the first homography matrix, so that the first homography
  • the property matrix is used for image alignment processing.
  • the embodiment of the present disclosure provides an implementation mode.
  • the region of interest extracts ECC feature points, and extracts ECC feature points from the second region of interest.
  • the ECC feature points are feature points with high feature matching accuracy, so that there is no obvious difference between the first region of interest and the second ECC feature points.
  • precise feature matching is carried out, and when there is no obvious scene in the image, feature matching is performed by extracting ECC feature points, which improves the accuracy of image alignment and obtains a better image alignment effect.
  • feature points are extracted by performing target object detection in the first image to obtain the target object map, wherein, when the target object map contains the target object, corner points are used as feature points for extraction, in order to ensure feature matching Accuracy, using ECC feature points as feature points for extraction; in the case that the target object map does not contain the target object, adjust the brightness according to the reference area of interest, and extract the corner points of the area of interest for subsequent use At least one feature match. Whether or not the image contains the target object is processed separately to ensure the accuracy of the extracted feature points for subsequent feature point matching, thereby improving the accuracy of image alignment.
  • S206 Perform feature matching on multiple first feature points and multiple second feature points, determine the coordinates of at least one first target feature point among the multiple first feature points, and determine at least one first target feature point among the multiple second feature points The coordinates of the two target feature points.
  • At least one first target feature point matches at least one second target feature point.
  • the present disclosure performs feature matching according to the corner points extracted after the target object is detected.
  • the determined number of target feature points is more than 3 pairs required for calculating the homography matrix.
  • the present disclosure extracts ECC feature points according to the above steps for feature matching.
  • the feature matching method includes but is not limited to accelerated robust features (Speeded Up Robust Feature, SURF), Scale Invariant Feature Transform (SIFT), Accelerated Segmentation Test Acquired Features (Features from Accelerated Segment Test, FAST), which are not limited in this disclosure.
  • the present disclosure performs feature matching based on the corner points extracted after the target object is not detected, and performs at least one feature matching that is the same as in the global image alignment process, first in the case of a smaller-scale image that is blurred Next, perform feature matching on N corner points to obtain a rough matching result of the corner points, and then increase the scale to perform feature matching on M corner points until the scale is transformed to the size of the original image to complete the precise matching of the image.
  • N is a positive integer greater than M.
  • the coordinates of at least one target feature point and the coordinates of at least one second target feature point are determined, wherein, The at least one first target feature point is at least one first ECC target feature point, and the at least one second target feature point is at least one second ECC target feature point determined by an ECC feature matching method.
  • the random sampling consensus algorithm calculates the homography matrix and iteratively updates it, eliminates the wrong matching points, and determines the optimal homography matrix.
  • the aligned first region of interest is cropped and output. If the first ROI is an expanded ROI, clipping is performed according to the size of the expanded ROI, and the expanded ROI is output.
  • the region of interest is extracted multiple times from the first image and the second image, and then steps S204-S210 are performed until each region of interest in the first image is image-aligned, and then the output All the images of the first image are aligned to the processed region of interest, so as to obtain the first image after complete image alignment.
  • the present disclosure first performs global image alignment on the image to be processed, and then extracts feature points from the first region of interest in the image to be processed and the second region of interest in the reference image for feature matching, and obtains the coordinates of the matched feature points , so as to calculate the homography matrix, so as to map the first region of interest to the second region of interest according to the homography matrix to realize the local alignment of the image to be processed, and realize the to-be-processed image by aligning all regions of interest in the image to be processed
  • the precise alignment of images reduces the influence of uncertain factors in the image shooting process, and can extract more feature points for image alignment, which is applicable to various scenarios and improves the accuracy of image alignment.
  • Fig. 5 is a flow chart 2 of the steps of the image alignment method provided by one or more embodiments of the present disclosure.
  • Fig. 5 is a further expansion and optimization of the present disclosure based on the embodiment shown in Fig. 2 , wherein a part of S202 A possible implementation is as follows:
  • N is a positive integer greater than M.
  • S512 Determine a first target corner point in at least one first corner point, and a second target corner point in at least one second corner point.
  • the first image is obtained by adjusting the brightness of the third image, which solves the problem of brightness difference between the image to be processed and the reference image, and improves the convenience of the feature matching process; iteratively calculates the optimal homography matrix through the random consensus algorithm, and thus obtains The first image after the global image alignment achieves a better effect of global image alignment; the third image is matched with the second image for multiple feature points, which reduces the false matching rate and improves the matching accuracy.
  • Fig. 6 is a flowchart three of steps of an image alignment method provided by one or more embodiments of the present disclosure.
  • Fig. 6 is a further expansion and optimization of the present disclosure on the basis of the embodiment shown in Fig. 2 , wherein one of S205 A possible implementation is as follows:
  • S602. Perform target object detection on the first region of interest and the second region of interest.
  • the target object map is obtained, and the brightness of the target object map corresponding to the image to be processed is adjusted to determine whether the important information of the target object is contained in the region of interest, and then extract the relevant information of the target object.
  • ECC feature points so as to extract obvious feature points for image alignment, reduce the influence of other interference factors in the image shooting process, and improve the accuracy of image alignment.
  • steps in the flow charts in FIGS. 2-4 are displayed sequentially as indicated by the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in Figures 2-4 may include a plurality of sub-steps or stages, these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, these sub-steps or stages The order of execution is not necessarily performed sequentially, but may be performed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.
  • the embodiment of the present disclosure also provides an image alignment device, the device embodiment corresponds to the foregoing method embodiment, for the sake of easy reading, the present device embodiment does not implement the foregoing method.
  • the details in the examples are described one by one, but it should be clear that the device in this embodiment can correspondingly implement all the content in the foregoing method embodiments.
  • FIG. 7 is a structural block diagram of an image alignment device in one or more embodiments of the present disclosure. As shown in FIG. Module 708 and mapping module 710, wherein:
  • the acquisition module 702 is configured to acquire a first image and a second image, the first image is the first image after global image alignment processing based on the second image;
  • Extraction module 704 configured to extract a plurality of first feature points from the first region of interest of the first image, and extract a plurality of second feature points from the second region of interest of the second image; the first region of interest and the first region of interest The two regions of interest have the same size and correspond to the positions in different images;
  • the feature matching module 706 is configured to perform feature matching on a plurality of first feature points and a plurality of second feature points, determine the coordinates of at least one first target feature point in the plurality of first feature points, and determine a plurality of second feature points coordinates of at least one second target feature point in the points, at least one first target feature point matches at least one second target feature point;
  • the calculation module 708 is configured to calculate a first homography matrix configured as a pixel map according to the coordinates of at least one first target feature point and the coordinates of at least one second target feature point;
  • the mapping module 710 is configured to map the pixel value of each pixel in the first region of interest to the corresponding pixel in the second region of interest based on the first homography matrix, so as to obtain the first sense image after image alignment area of interest.
  • the acquisition module 702 is configured to acquire the histogram of the second image, and determine the pixel value of each pixel in the second image; Set the pixel value of the pixel point corresponding to the pixel position in the third image; according to the pixel value of each pixel point in the second image and the pixel value of each pixel point in the third image after setting, extract the first At least one first corner point in the three images, and extracting at least one second corner point in the second image; performing feature matching on at least one first corner point and at least one second corner point, and determining at least one first corner point The first target corner point in , and the second target corner point in at least one second corner point, the first target corner point matches the second target corner point; according to the coordinates of the first target corner point and the second target corner point The coordinates of the second homography matrix are calculated; based on the second homography matrix, the pixel value of each pixel point in the third image is mapped to the corresponding pixel point of the second image, and
  • the acquisition module 702 is configured to perform feature matching on at least one first corner point and at least one second corner point in units of N corner points;
  • the first corner point and at least one second corner point are used to perform feature matching again in units of M corner points; wherein, N is a positive integer greater than M.
  • the acquisition module 702 is configured to, under the condition that the number of first target corner points is greater than or equal to the preset number, according to the coordinates of the first target corner point and the second target angle The coordinates of the points are used to calculate the second homography matrix; under the condition that the number of the first target corner points is less than the preset number, a plurality of first orientation fast rotation ORB feature points are extracted from the third image, and from the second image Extracting a plurality of second ORB feature points; performing feature matching with a plurality of first ORB feature points and a plurality of second ORB feature points, determining the coordinates of at least one first target ORB feature point in a plurality of first ORB feature points, and determining the coordinates of at least one second target ORB feature point among the plurality of second target ORB feature points, at least one first target ORB feature point matching with at least one second target ORB feature point; according to the at least one first target ORB feature point coordinates, and the coordinates of
  • the extraction module 704 is configured to perform target object detection on the first region of interest and the second region of interest; determine the first target object graph from the first region of interest, and Determining a second object map from the second region of interest; extracting a plurality of first feature points from the first object map, and extracting a plurality of second feature points from the second object map.
  • the first feature point is a first entropy correlation coefficient ECC feature point
  • the second feature point is a second ECC feature point
  • Extraction module 704 configured to identify whether the first target object graph and the second target object graph contain the target object; under the condition that the first target object graph and the second target object graph contain the target object, extract A plurality of first ECC feature points are extracted from the image, and a plurality of second ECC feature points are extracted from the second target object graph.
  • the extraction module 704 is configured to: acquire the histogram of the second target object map, and determine the pixel value of each pixel in the second target object map; For the pixel value of each pixel in the object map, set the pixel value of the pixel corresponding to the pixel position in the first target object map.
  • the target object detection is portrait detection
  • the extraction module 704 is configured to obtain the first portrait mask of the first region of interest obtained after portrait detection, and the second A second portrait mask of the region; determining a first portrait map from the first image according to the first portrait mask, and determining a second portrait map from the second image according to the second portrait mask.
  • the extracting module 704 is configured to extract the third ROI in the first image and the fourth ROI in the second image according to a default size; If the size of the ROI is smaller than the preset size, expand the third ROI according to the preset size to obtain the first ROI; if the size of the fourth ROI is smaller than the preset size, expand the third ROI according to the preset size Extend the fourth ROI to obtain the second ROI.
  • the image alignment device provided by the present disclosure first performs global image alignment on the image to be processed, and then extracts feature points from the first region of interest in the image to be processed and the second region of interest in the reference image to perform feature matching to obtain the matched feature points
  • the coordinates of the homography matrix are calculated to map the first region of interest to the second region of interest according to the homography matrix to achieve local alignment of the image to be processed by aligning all regions of interest in the image to be processed.
  • the precise alignment of the images to be processed reduces the influence of uncertain factors in the image shooting process, and more feature points can be extracted for image alignment. It is suitable for various scenarios and improves the accuracy of image alignment.
  • the image alignment device provided in this embodiment can execute the image alignment method provided in the above method embodiment, and its implementation principle and technical effect are similar, and will not be repeated here.
  • Each module in the above-mentioned image alignment device can be fully or partially realized by software, hardware and a combination thereof.
  • the above-mentioned modules can be embedded in or independent of one or more processors in the computer device in the form of hardware, and can also be stored in the memory of the computer device in the form of software, so that one or more processors can call and execute the above-mentioned The operation corresponding to the module.
  • the included units and modules are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, each The specific names of the functional units are only for the convenience of distinguishing each other, and are not configured to limit the protection scope of the present disclosure.
  • a computer device is provided.
  • the computer device may be a terminal device, and its internal structure may be as shown in FIG. 8 .
  • the computer device includes one or more processors, memory, communication interfaces, databases, display screens, and input devices connected by a system bus. Wherein, one or more processors of the computer device are configured as modules providing computing and control capabilities.
  • the memory of the computer equipment includes a non-volatile storage medium and an internal memory.
  • the non-volatile storage medium stores an operating system and computer readable instructions.
  • the internal memory provides an environment for the execution of the operating system and computer readable instructions in the non-volatile storage medium.
  • the communication interface of the computer device is configured as a wired or wireless communication module with an external terminal, and the wireless mode can be realized through WIFI, operator network, near field communication (NFC) or other technologies.
  • the computer-readable instructions are executed by one or more processors, the image alignment method provided by the above-mentioned embodiments can be realized.
  • the display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, a trackball or a touch pad provided on the casing of the computer device , and can also be an external keyboard, touchpad, or mouse.
  • FIG. 8 is only a block diagram of a partial structure related to the disclosed solution, and does not constitute a limitation to the computer equipment on which the disclosed solution is applied.
  • the specific computer equipment can be More or fewer components than shown in the figures may be included, or some components may be combined, or have a different arrangement of components.
  • the apparatus for displaying preview images provided by the present disclosure may be implemented in the form of computer-readable instructions, and the computer-readable instructions may be run on a computer device as shown in FIG. 8 .
  • Various program modules that make up the computer device can be stored in the memory of the computer device, such as the acquisition module 702 , extraction module 704 , feature matching module 706 , calculation module 708 and mapping module 710 shown in FIG. 7 .
  • Computer-readable instructions constituted by various program modules enable one or more processors to execute the steps in the image alignment method of any one embodiment of the present disclosure described in this specification.
  • the computer device shown in FIG. 8 can execute the acquisition of the first image and the second image through the acquisition module 702 in the image alignment device as shown in FIG. first image.
  • the computer device can perform extracting a plurality of first feature points from a first region of interest of a first image through an extraction module 704, and extracting a plurality of second feature points from a second region of interest of a second image; the first region of interest It is the same size as the second ROI and corresponds to its position in different images.
  • the computer device may use the feature matching module 706 to perform feature matching on multiple first feature points and multiple second feature points, determine the coordinates of at least one first target feature point among the multiple first feature points, and determine multiple first feature points.
  • the coordinates of at least one second target feature point among the two feature points, at least one first target feature point matches at least one second target feature point.
  • the computer device can calculate the first homography matrix for pixel mapping according to the coordinates of at least one first target feature point and the coordinates of at least one second target feature point through the calculation module 708 .
  • the computer device may use the mapping module 710 to perform mapping of the pixel value of each pixel in the first region of interest to the corresponding pixel in the second region of interest based on the first homography matrix, so as to obtain the first aligned image. a region of interest.
  • a computer device comprising a memory and one or more processors, the memory storing computer readable instructions which, when executed by the one or more processors, cause One or more processors execute the steps of the image alignment method provided in any one embodiment of the present disclosure.
  • the computer device provided in this embodiment can implement the image alignment method provided in the above method embodiment, and its implementation principle is similar to the technical effect, and will not be repeated here.
  • One or more non-volatile storage media storing computer-readable instructions.
  • the computer-readable instructions are executed by one or more processors, one or more processors execute the image provided in any one of the embodiments of the present disclosure. Align the steps of the method.
  • the computer-readable storage medium may be a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and the like.
  • ROM Read-Only Memory
  • RAM Random Access Memory
  • magnetic disk or an optical disk and the like.
  • the computer-readable instructions stored on the computer-readable storage medium provided by this embodiment can realize the image alignment method provided by the above-mentioned method embodiment, and its implementation principle is similar to the technical effect, and will not be repeated here.
  • Non-volatile memory may include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory or optical memory, etc.
  • Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory.
  • RAM Random Access Memory
  • SRAM Static Random Access Memory
  • DRAM Dynamic Random Access Memory
  • the image alignment method provided in this disclosure achieves precise alignment of the image to be processed by aligning all regions of interest in the image to be processed, reduces the influence of uncertain factors in the image shooting process, and can extract more feature points for image alignment , is applicable to various scenarios, improves the accuracy of image alignment, and has strong industrial applicability.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)

Abstract

Les modes de réalisation de la présente divulgation concernent un procédé et un appareil d'alignement d'image, ainsi qu'un dispositif informatique et un support de stockage, qui sont appliqués au domaine technique du traitement d'image et résolvent le problème de l'état de la technique, à savoir une faible précision d'alignement d'image provoquée par des points caractéristiques non évidents. Le procédé consiste à : acquérir une première image et une seconde image ; extraire une pluralité de premiers points caractéristiques à partir d'une première zone d'intérêt de la première image, puis extraire une pluralité de seconds points caractéristiques à partir d'une seconde zone d'intérêt de la seconde image ; effectuer une mise en correspondance de caractéristiques sur la pluralité de premiers points caractéristiques et la pluralité de seconds points caractéristiques, déterminer les coordonnées d'au moins un premier point caractéristique cible parmi la pluralité de premiers points caractéristiques, puis déterminer les coordonnées d'au moins un second point caractéristique cible parmi la pluralité de seconds points caractéristiques ; calculer une première matrice d'homographie pour le mappage de pixels ; et d'après la première matrice d'homographie, mapper une valeur de pixel de chaque point de pixel dans la première zone d'intérêt avec un point de pixel correspondant dans la seconde zone d'intérêt afin d'obtenir la première zone d'intérêt après alignement de l'image.
PCT/CN2022/100804 2021-12-02 2022-06-23 Procédé et appareil d'alignement d'image, et dispositif informatique et support de stockage WO2023098045A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111459872.0 2021-12-02
CN202111459872.0A CN114298902A (zh) 2021-12-02 2021-12-02 一种图像对齐方法、装置、电子设备和存储介质

Publications (1)

Publication Number Publication Date
WO2023098045A1 true WO2023098045A1 (fr) 2023-06-08

Family

ID=80964942

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/100804 WO2023098045A1 (fr) 2021-12-02 2022-06-23 Procédé et appareil d'alignement d'image, et dispositif informatique et support de stockage

Country Status (2)

Country Link
CN (1) CN114298902A (fr)
WO (1) WO2023098045A1 (fr)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114298902A (zh) * 2021-12-02 2022-04-08 上海闻泰信息技术有限公司 一种图像对齐方法、装置、电子设备和存储介质
WO2023206475A1 (fr) * 2022-04-29 2023-11-02 北京小米移动软件有限公司 Procédé et appareil de traitement d'image, dispositif électronique et support de stockage
CN114998097A (zh) * 2022-07-21 2022-09-02 深圳思谋信息科技有限公司 图像对齐方法、装置、计算机设备和存储介质
CN116309760B (zh) * 2023-05-26 2023-09-19 安徽高哲信息技术有限公司 谷物的图像对齐方法及谷物检测设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120281921A1 (en) * 2011-05-02 2012-11-08 Los Alamos National Security, Llc Image alignment
CN104376548A (zh) * 2014-11-07 2015-02-25 中国电子科技集团公司第二十八研究所 一种基于改进型surf算法的图像快速拼接方法
CN109886878A (zh) * 2019-03-20 2019-06-14 中南大学 一种基于由粗到精配准的红外图像拼接方法
CN110610457A (zh) * 2018-06-14 2019-12-24 杭州海康威视数字技术股份有限公司 全景图像拼接方法、装置以及系统
CN114298902A (zh) * 2021-12-02 2022-04-08 上海闻泰信息技术有限公司 一种图像对齐方法、装置、电子设备和存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120281921A1 (en) * 2011-05-02 2012-11-08 Los Alamos National Security, Llc Image alignment
CN104376548A (zh) * 2014-11-07 2015-02-25 中国电子科技集团公司第二十八研究所 一种基于改进型surf算法的图像快速拼接方法
CN110610457A (zh) * 2018-06-14 2019-12-24 杭州海康威视数字技术股份有限公司 全景图像拼接方法、装置以及系统
CN109886878A (zh) * 2019-03-20 2019-06-14 中南大学 一种基于由粗到精配准的红外图像拼接方法
CN114298902A (zh) * 2021-12-02 2022-04-08 上海闻泰信息技术有限公司 一种图像对齐方法、装置、电子设备和存储介质

Also Published As

Publication number Publication date
CN114298902A (zh) 2022-04-08

Similar Documents

Publication Publication Date Title
WO2023098045A1 (fr) Procédé et appareil d'alignement d'image, et dispositif informatique et support de stockage
US9275281B2 (en) Mobile image capture, processing, and electronic form generation
US10284789B2 (en) Dynamic generation of image of a scene based on removal of undesired object present in the scene
WO2017107700A1 (fr) Procédé et terminal d'alignement d'images
US9609221B2 (en) Image stabilization method and electronic device therefor
WO2019042419A1 (fr) Procédé et dispositif d'acquisition de point de suivi d'image et support d'informations
AU2017232186A1 (en) Fast and robust image alignment for burst mode
CN106296570B (zh) 图像处理方法及装置
US20130129213A1 (en) Non-Rigid Dense Correspondence
US10122912B2 (en) Device and method for detecting regions in an image
CN110136198B (zh) 图像处理方法及其装置、设备和存储介质
US9147226B2 (en) Method, apparatus and computer program product for processing of images
US11620730B2 (en) Method for merging multiple images and post-processing of panorama
WO2022160857A1 (fr) Procédé et appareil de traitement d'images, support de stockage lisible par ordinateur et dispositif électronique
US20170351932A1 (en) Method, apparatus and computer program product for blur estimation
US9418446B2 (en) Method and apparatus for determining a building location based on a building image
WO2018184255A1 (fr) Procédé et dispositif de correction d'image
CN114332183A (zh) 图像配准方法、装置、计算机设备和存储介质
US10861138B2 (en) Image processing device, image processing method, and program
US11216961B2 (en) Aligning digital images by selectively applying pixel-adjusted-gyroscope alignment and feature-based alignment models
US20230016350A1 (en) Configurable keypoint descriptor generation
WO2022206679A1 (fr) Procédé et appareil de traitement d'image, dispositif informatique et support de stockage
CN113056905A (zh) 用于拍摄类长焦图像的系统和方法
CN115514887A (zh) 视频采集的控制方法、装置、计算机设备和存储介质
JP5929379B2 (ja) 画像処理装置、画像処理方法及びプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22899865

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE