WO2022095596A1 - Procédé d'alignement d'image, appareil d'alignement d'image et dispositif terminal - Google Patents

Procédé d'alignement d'image, appareil d'alignement d'image et dispositif terminal Download PDF

Info

Publication number
WO2022095596A1
WO2022095596A1 PCT/CN2021/117471 CN2021117471W WO2022095596A1 WO 2022095596 A1 WO2022095596 A1 WO 2022095596A1 CN 2021117471 W CN2021117471 W CN 2021117471W WO 2022095596 A1 WO2022095596 A1 WO 2022095596A1
Authority
WO
WIPO (PCT)
Prior art keywords
grid area
image
grid
matching point
point
Prior art date
Application number
PCT/CN2021/117471
Other languages
English (en)
Chinese (zh)
Inventor
林枝叶
Original Assignee
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oppo广东移动通信有限公司 filed Critical Oppo广东移动通信有限公司
Publication of WO2022095596A1 publication Critical patent/WO2022095596A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods

Definitions

  • the present application belongs to the technical field of image processing, and in particular, relates to an image alignment method, an image alignment apparatus, a terminal device and a computer-readable storage medium.
  • Image alignment technology is a very important and basic technology in image processing, which can be applied to many image processing tasks.
  • terminals such as mobile phones, AR glasses, and virtual reality devices are often integrated with multiple cameras, and the imaging principles used by different cameras may also be different. For example, there may be infrared cameras and RGB imaging cameras on the terminal. The images collected by cameras with different imaging principles can be considered as images of different modalities. At this time, images of different modalities need to be aligned to achieve stitching fusion.
  • image alignment technology should also be used to achieve different modalities such as Computed Tomography (CT) images and Magnetic Resonance Imaging (MRI). Stitching and fusion between images.
  • CT Computed Tomography
  • MRI Magnetic Resonance Imaging
  • the embodiments of the present application provide an image alignment method, an image alignment device, a terminal device, and a computer-readable storage medium, which can solve the problem that the existing method cannot find accurate matching point pairs between images of different modalities, so that different modalities cannot find accurate matching point pairs.
  • an embodiment of the present application provides an image alignment method, including:
  • the first target pixel corresponding to the first candidate matching point is searched from the second modal image, wherein the first target pixel
  • the cross-correlation information between the point and the corresponding first candidate matching point meets a preset cross-correlation condition
  • the first candidate matching point and the first matching point corresponding to the first candidate matching point is used as a set of matching point pairs between the first modal image and the second modal image;
  • a grid transformation matrix between the first grid area and the second grid area corresponding to the first grid area in the second modal image is obtained, wherein, The position of the first grid area in the first modal image is the same as the position of the second grid area corresponding to the first grid area in the second modal image;
  • an image alignment device including:
  • a determining module configured to determine at least two first candidate matching points in the first grid area for each first grid area pre-divided in the first modal image
  • a search module is configured to search, for each first candidate matching point in the first grid area, the first target pixel point corresponding to the first candidate matching point from the second modal image, wherein the The cross-correlation information between the first target pixel point and the corresponding first candidate matching point meets a preset cross-correlation condition;
  • the first processing module is configured to, if the first target pixel corresponding to the first candidate matching point is found from the second modal image, compare the first candidate matching point with the first candidate matching point.
  • the first target pixel point corresponding to the matching point is used as a set of matching point pairs between the first modal image and the second modal image;
  • a second processing module configured to obtain, according to the matching point pair, the difference between the first grid area and the second grid area corresponding to the first grid area in the second modal image A grid transformation matrix, wherein the position of the first grid area in the first modal image and the second grid area corresponding to the first grid area are in the second modal image the same location;
  • a transformation module configured to transform the second modal image into a target image aligned with respect to the first modal image according to each grid transformation matrix.
  • an embodiment of the present application provides a terminal device, including a memory, a processor, a display, and a computer program stored in the memory and running on the processor, characterized in that the processor executes the computer
  • the image alignment method as described above in the first aspect is implemented during the program.
  • an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, implements the image alignment method described in the first aspect.
  • an embodiment of the present application provides a computer program product that, when the computer program product runs on a terminal device, enables the terminal device to execute the image alignment method described above in the first aspect.
  • FIG. 1 is a schematic flowchart of an image alignment method provided by an embodiment of the present application.
  • FIG. 2 is an exemplary schematic diagram of a distribution manner of a first candidate matching point in a first grid area provided by an embodiment of the present application;
  • FIG. 3 is a schematic flowchart of step S102 provided by an embodiment of the present application.
  • FIG. 4 is an exemplary schematic diagram of aligning the first modality image and the second modality image provided by an embodiment of the present application
  • FIG. 5 is a schematic structural diagram of an image alignment apparatus provided by an embodiment of the present application.
  • FIG. 6 is a schematic structural diagram of a terminal device provided by an embodiment of the present application.
  • the term “if” may be contextually interpreted as “when” or “once” or “in response to determining” or “in response to detecting “.
  • the phrases “if it is determined” or “if the [described condition or event] is detected” may be interpreted, depending on the context, to mean “once it is determined” or “in response to the determination” or “once the [described condition or event] is detected. ]” or “in response to detection of the [described condition or event]”.
  • references in this specification to "one embodiment” or “some embodiments” and the like mean that a particular feature, structure or characteristic described in connection with the embodiment is included in one or more embodiments of the present application.
  • appearances of the phrases “in one embodiment,” “in some embodiments,” “in other embodiments,” “in other embodiments,” etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean “one or more but not all embodiments” unless specifically emphasized otherwise.
  • the terms “including”, “including”, “having” and their variants mean “including but not limited to” unless specifically emphasized otherwise.
  • the image alignment method provided by the embodiments of the present application can be applied to servers, desktop computers, mobile phones, tablet computers, wearable devices, in-vehicle devices, augmented reality (AR)/virtual reality (VR) devices, and notebook computers , ultra-mobile personal computer (ultra-mobile personal computer, UMPC), netbook, personal digital assistant (personal digital assistant, PDA) and other terminal equipment, the embodiment of the present application does not make any restrictions on the specific type of the terminal equipment.
  • the similarity between pixels can be measured through the cross-correlation information, so as to find the exact matching point pair between the first modal image and the second modal image, which can reduce the number of different modalities.
  • the interference caused by the difference in the gradient direction of the images in the structurally similar regions also ensures the alignment accuracy of the final target image accordingly.
  • FIG. 1 shows a flowchart of an image alignment method provided by an embodiment of the present application, and the image alignment method can be applied to a terminal device.
  • the image alignment method may include:
  • Step S101 for each first grid area pre-divided in the first modal image, determine at least two first candidate matching points in the first grid area.
  • the first modality image and the second modality image may be considered to be images of different modalities.
  • images of different modalities can be considered as images collected by cameras with different imaging principles.
  • the infrared image collected by the infrared camera and the image collected by the RGB imaging camera can be considered as images of different modalities.
  • Computed Tomography (CT) images, Magnetic Resonance Imaging (Magnetic Resonance Imaging, MRI) and ultrasound images can also be considered as images of different modalities.
  • the imaging principles respectively corresponding to the first modal image and the second modal image may be determined according to actual scene requirements, which are not limited herein.
  • the first modality image may be an RGB image
  • the second modality image may be a modality image other than an RGB image, such as an infrared image.
  • the size and division method of the first grid area may also be determined according to the actual scene.
  • Each of the first grid regions in the first modality image may be uniformly distributed. If the resolution of the first modal image is W*H, where the width is W, the height is H, and the size of the first grid area is w*h, then the number of the first grid area is W/w*H/h.
  • the position of the first mesh region in the first modal image may be identified by the coordinates of four vertices.
  • the pixel points obtained by uniform sampling in the first grid area may be used as the first candidate matching points.
  • the number of the first candidate matching points in the first grid area may also be determined according to the image conditions in the first grid area, so as to determine the coordinate position of each first candidate matching point.
  • the first modality image and the second modality image may be aligned in coplanar rows.
  • the image plane corresponding to the first modal image and the image plane corresponding to the second modal image are parallel to each other, thereby reducing the stereo parallax, and reducing the complexity of matching in the subsequent search for matching point pairs And the amount of calculation, improve the accuracy of the grid transformation matrix determined according to the matching point pair.
  • the method before determining at least two first candidate matching points in the first grid area for each first grid area pre-divided in the first modal image, the method further includes:
  • the first camera parameters may include internal parameters, external parameters and/or distortion parameters of the first camera.
  • the second camera parameters may include internal parameters, external parameters and/or distortion parameters of the second camera.
  • the internal parameters may be parameters related to the characteristics of the corresponding camera, such as the focal length and pixel distribution of the corresponding camera.
  • the external parameters may indicate the pose of the corresponding camera in the world coordinate system, which is determined by the relative pose relationship between the camera and the world coordinate system. Exemplarily, the external parameters may include a rotation vector and a translation vector.
  • the distortion parameters may include radial distortion parameters and/or tangential distortion parameters.
  • first camera parameters and the second camera parameters it is possible to determine the difference between the image coordinate system of the first original image captured by the first camera and the image coordinate system of the second original image captured by the second camera relative relationship.
  • the first camera and the second camera may be calibrated in advance to determine the first camera parameters of the first camera and the second camera parameters of the second camera.
  • the first camera and the second camera may be calibrated respectively by Zhang Zhengyou's calibration method.
  • the separately correcting the first original image and the second original image may include performing distortion correction and/or stereo correction on the first original image, and performing distortion on the second original image.
  • Correction and/or Stereo Correction may be to correct the image distortion of the corresponding image through distortion parameters, such as correcting the radial distortion of the image and the tangential distortion of the image.
  • the stereo correction can align the non-coplanar lines of the two images into a coplanar line alignment.
  • the plane of the corrected first original image is parallel to the plane of the corrected second original image, corresponding to The optical axes of the cameras are concentric, and the poles of the corrected first original image and the corrected second original image are at infinity.
  • the first original image and the second original image can be corrected to obtain the first modal image and the second modal image aligned in a coplanar row, so that the subsequent search can be performed.
  • search in images whose image planes are parallel to each other to avoid errors caused by differences in image distortion and image shooting angle improve the search efficiency and accuracy of matching point pairs, and thus improve the accuracy of image alignment .
  • determining at least two first candidate matching points in the first grid area for each pre-divided first grid area in the first modal image includes:
  • the gradient information of the pixel point may reflect the change of the pixel value of the pixel point relative to the pixel value of the surrounding pixel points.
  • the pixel point gradient information of the first grid area itself may indicate the distribution of image content in the first grid area. Therefore, if the pixel point gradient variation range of the first grid area itself If the value is larger and the gradient value is larger, the amount of information in the first grid area may be large, and the number of the first candidate matching points in the first grid area can be increased.
  • the first grid area may be determined by combining the pixel point gradient information of the first grid area and the pixel point gradient information in the second grid area corresponding to the first grid area.
  • the number of first candidate matching points in the grid area For example, if the difference between the gradient information of the pixel points in the second grid area corresponding to the first grid area and the gradient information of the pixel points in the first grid area is large, it can be considered that the first grid area The similarity between a grid area and the second grid area corresponding to the first grid area is poor, and the degree of coincidence is poor. Therefore, the first candidate matching point in the first grid area can be improved. to obtain as many matching point pairs as possible between the first grid area and the second grid area corresponding to the first grid area, thereby improving the accuracy of image alignment.
  • At least two first candidate matching points of including:
  • a first candidate matching point in the first grid area is determined according to the number of the first candidate matching points.
  • the alignment error may reflect a difference in pixel point gradient information between the first grid area and a second grid area corresponding to the first grid area. If the alignment error between the first grid area and the second grid area corresponding to the first grid area is large, it can be considered that the first grid area and the first grid area There is no alignment between the second grid regions corresponding to the regions, and ghost images are likely to occur during subsequent fusion. Therefore, if the alignment error is large, the first grid region in the first grid region can be increased. A number of candidate matching points, so as to obtain a more accurate transformation relationship between the first grid area and the second grid area corresponding to the first grid area later. Specifically, the alignment error may be calculated and obtained by comparing the difference between the first gradient value of each first pixel point and the second gradient value of the corresponding second pixel point.
  • the number of the first candidate matching points in the first grid area may be determined according to the size of the alignment error. For example, if the alignment error is greater than a preset error threshold, determine that the number of first candidate matching points in the first grid area is M, and if the alignment error is not greater than a preset error threshold, determine The number of the first candidate matching points in the first grid area is N, where N and M are positive integers respectively, and M is greater than N.
  • each first candidate matching point may be determined according to information such as the size of the first grid area and the number of the first candidate matching points distribution of points in the first grid area, so as to determine the position of the first candidate matching point in the first grid area.
  • FIG. 2 it is an exemplary schematic diagram of the distribution manner of the first candidate matching points in the first grid area.
  • the number of the first candidate matching points in the first grid area is determined to be 9
  • the number of the first candidate matching points is determined to be 9.
  • the number of the first candidate matching points in the first grid area is four.
  • a first gradient value of the first pixel point and the second gradient value of the second pixel point corresponding to the first pixel point are set between The absolute value of the difference is taken as the first absolute value, wherein the position of the first pixel in the first modal image and the second pixel corresponding to the first pixel are in the second The position in the modal image is the same;
  • the coordinate range of the first grid area A is x ⁇ [0,w 1 ],y ⁇ [0,h 1 ], then the alignment error of the first grid area A is ⁇ 1 for:
  • grad rgb (x, y) is the first gradient value of any first pixel point
  • grad spectral (x, y) is the second gradient value of any second pixel point
  • the number of the first candidate matching points may be (3*n) 2 , otherwise the number of the first candidate matching points may be n 2 , and the threshold is a preset error threshold.
  • Step S102 for each first candidate matching point in the first grid area, search for the first target pixel point corresponding to the first candidate matching point from the second modal image, wherein the first candidate matching point is
  • the cross-correlation information between a target pixel point and the corresponding first candidate matching point conforms to a preset cross-correlation condition.
  • the cross-correlation information may include a normalized cross-correlation (Normalized Cross Correlation, NCC) value, a cross-correlation value calculated according to a preset cross-correlation function, etc., which may measure the cross-correlation of two related pixels. value of sex.
  • the preset cross-correlation condition may be determined according to the type of the cross-correlation information. For example, if the cross-correlation information includes the normalized cross-correlation value, the preset cross-correlation condition may be that the normalized cross-correlation value is greater than a preset cross-correlation threshold.
  • the step S102 includes:
  • Step S301 for each first candidate matching point in the first grid area, determine the region of interest corresponding to the first candidate matching point from the second modal image;
  • Step S302 for each pixel in the region of interest, calculate the cross-correlation metric value between the pixel and the first candidate matching point;
  • Step S303 if the maximum value of the cross-correlation metric values corresponding to each pixel in the region of interest is greater than a preset threshold, then the pixel corresponding to the maximum value in the region of interest is used as the maximum value in the region of interest.
  • the region of interest may be considered as a search range for searching for a first target pixel point corresponding to the first candidate matching point.
  • the size of the region of interest may be determined according to scene requirements, for example, may be predetermined according to information such as computing resources, the positional relationship between the cameras corresponding to the first modal image and the second modal image respectively .
  • the cross-correlation metric value may be a normalized cross-correlation (Normalized Cross Correlation, NCC) value, a cross-correlation value calculated according to a preset cross-correlation function, and the like.
  • the cross-correlation metric value is a normalized cross-correlation value
  • the calculating, for each pixel point in the region of interest, a cross-correlation metric value between the pixel point and the first candidate matching point including:
  • the normalized cross-correlation value between the pixel and the first candidate matching point is calculated according to the specified correlation region, where the specified correlation region is the specified correlation region. a designated area centered on the pixel in the second modal image.
  • the pixel value of the first candidate matching point (x 1 , y 1 ) is R rgb (x 1 , y 1 ).
  • the region of interest is As the starting point, a rectangular area with both width and height k.
  • the designated associated area of the pixel point R spectral (x,y) is the pixel point R spectral (x,y) as the center, with d is a rectangular area D of width and height.
  • u rgb_roi represents the expectation of the pixel value of each pixel in the corresponding region of the region of interest in the first modal image
  • u spectral_roi represents the expectation of the pixel value of each pixel in the region of interest
  • ⁇ rgb_roi represents The variance of the pixel value of each pixel in the corresponding region of the region of interest in the first modal image
  • ⁇ spectral_roi represents the variance of the pixel value of each pixel in the region of interest, when the NCC is larger, it means that the first A candidate matching point is more similar to the pixel point R spectral (x, y).
  • NCC NCC (x m , y m ) ⁇ T ncc , then the pixel in the region of interest corresponding to the NCC(x m , y m ) is the first target pixel of the first candidate matching point.
  • T ncc can be a preset threshold.
  • Step S103 if the first target pixel point corresponding to the first candidate matching point is found from the second modal image, then the first candidate matching point and the first candidate matching point corresponding to the first candidate matching point are found.
  • the first target pixel point of is used as a set of matching point pairs between the first modal image and the second modal image.
  • Using the cross-correlation information to measure the similarity between pixels to find the first target pixel corresponding to the first candidate matching point can reduce the gradient direction of images of different modalities in structurally similar regions Compared with the existing feature point detection and matching based on SIFT and other methods, the accuracy of the obtained matching point pairs can be improved.
  • the distance between the first grid area and the second grid area corresponding to the first grid area in the second modal image is obtained according to each set of matched point pairs.
  • the mesh transformation matrix also include:
  • a second target pixel corresponding to the second candidate matching point is searched from the first modal image, wherein the second target pixel
  • the cross-correlation information between the point and the corresponding second candidate matching point meets a preset cross-correlation condition
  • the second candidate matching point and the second matching point corresponding to the second candidate matching point are used as a set of matching point pairs between the first modal image and the second modal image.
  • the image coordinate system of the first modal image be used as a benchmark to search for the matching point pair according to the first candidate matching points
  • the image coordinate system of the second modal image can be used as a benchmark
  • the matching point pair is searched according to the second candidate matching point, so that the number of the matching point pair can be increased.
  • Step S104 obtaining a grid transformation matrix between the first grid area and the second grid area corresponding to the first grid area in the second modal image according to the matching point pair , wherein the position of the first grid area in the first modal image is the same as the position of the second grid area corresponding to the first grid area in the second modal image.
  • the distribution manner of each of the first grid areas in the first modal image is the same as the distribution manner of each of the second grid areas in the second modal image.
  • the grid transformation matrix may indicate a translational transformation relationship and/or a rotational transformation relationship of the image in the first grid area with respect to the image in the corresponding second grid area, and the like.
  • the specific calculation method of the grid transformation matrix may be determined according to the situation such as the number of the matching point pairs. If the number of matching point pairs between the first grid area and the second grid area corresponding to the first grid area in the second modal image is not less than a preset number, for example , not less than 4, then according to the matching point pair, through affine transformation, homography transformation, etc., to calculate the difference between the first grid area and the first grid area in the second modal image The grid transformation matrix between the corresponding second grid regions.
  • the matching points of other points in the first grid area in the second grid area corresponding to the first grid area in the second modal image may be further calculated according to the matching point pair , and then calculate the grid transformation matrix.
  • Step S105 Transform the second modal image into a target image aligned with the first modal image according to each grid transformation matrix.
  • each second grid region may be transformed according to each grid transformation matrix, and then the transformed second grid regions may be combined to obtain the target image.
  • the transformation of each second grid area can be independent of each other, instead of realizing the transformation of the entire second modal image through a unified transformation matrix, which can improve the accuracy of the transformation of each local area, thereby greatly improving the image. Alignment precision.
  • the obtaining between the first grid area and the second grid area corresponding to the first grid area in the second modal image according to the matching point pair Grid transformation matrix including:
  • the coordinates of the designated vertex in the first mesh area, and the coordinate of the designated vertex in the first mesh area in the second mesh area corresponding to the first mesh area Expected coordinates, build a least squares model
  • the second mesh area corresponding to the first mesh area obtains the second mesh area corresponding to the first mesh area and the first mesh area a homography matrix between grid regions, and using the homography matrix as the grid transformation matrix;
  • the transforming the second modal image into a target image aligned relative to the first modal image according to each grid transformation matrix includes:
  • a target image aligned with respect to the first modality image is obtained according to each perspective transformed second grid area.
  • the designated vertices may be four vertices of the first mesh region.
  • the specified vertices can be four vertices of the first mesh area, and the relationship between the first mesh area and the second mesh area corresponding to the first mesh area can be solved.
  • the homography matrix between may also be less than 3. In this case, the desired coordinates of the specified vertices and the matching points may be combined Calculate the homography matrix for the pair.
  • the construction and solution of the least squares model can be implemented according to the prior art.
  • the matching point pair the coordinates of the designated vertex in the first mesh area, and the second corresponding to the first mesh area of the designated vertex in the first mesh area. desired coordinates in the grid area
  • constructing the least squares model that can indicate the first graph and the second graph by calculating a preset error between vertices in the first graph and The shape similarity between the second graphics.
  • the first graph is a first candidate matching point in the matching point pair and a designated vertex in the first grid area
  • the second graph is a center between the desired coordinate and the matching point
  • the preset error can be optimized to minimize the preset error, so as to obtain the relationship between the specified vertex in the first mesh area in the second mesh area corresponding to the first mesh area desired coordinates.
  • the least squares model can be solved by means of Gauss-Newton method, gradient descent method, LM (Levenberg-Marquart) method, or the like.
  • the specific solution method is not limited here.
  • the target image may be stored in the form of a binary file for subsequent image reading and processing.
  • the target image can also be stored in other formats according to the needs of the scene.
  • FIG. 4(a) is the first modal image
  • Fig. 4(b) is the second modal image. If the first modal image and the second modal image are aligned in a coplanar row, then, after directly merging the first modal image and the second modal image, FIG. 4( c ) is obtained. At this time, a clear positional deviation appears in Fig. 4(c).
  • first modal image and the second modal image are obtained, for each first grid area pre-divided in the first modal image, according to each of the first grid areas
  • the first gradient value of the first pixel point, and the second gradient value of each second pixel point in the second grid area corresponding to the first grid area determine the relationship between the first grid area and the The alignment error between the second grid areas corresponding to the first grid area.
  • the number of the corresponding first candidate matching points in the first grid area may be larger, for example, 9; and if the homogeneous error is not equal to is greater than the preset error threshold, then, the number of the corresponding first candidate matching points in the first grid area may be less, for example, four.
  • the first grid area with a large number of first candidate matching points and the first grid area with a large number of first candidate matching points The distribution of grid areas.
  • each grid transformation matrix is obtained according to the matching point pairs. Then, according to the grid transformation matrix, according to the matching point pair, the coordinates of the four vertices of each first grid area in the first modal image as shown in FIG. 4(e), and the The coordinates of four vertices in a grid area are expected coordinates in the second grid area corresponding to the first grid area, and a least squares model is constructed. Solve the least squares model to obtain the expectation of the coordinates of the 4 vertices in the first grid area as shown in Figure 4(f) in the second grid area corresponding to the first grid area coordinate.
  • the first target pixel corresponding to the first candidate matching point is searched from the second modal image, wherein the first target pixel corresponds to the corresponding
  • the cross-correlation information between the first candidate matching points conforms to a preset cross-correlation condition. At this time, the similarity between the pixel points can be measured by the cross-correlation information, so as to find the first target pixel point corresponding to the first candidate matching point.
  • the first candidate matching point and the first matching point corresponding to the first candidate matching point is used as a set of matching point pairs between the first modal image and the second modal image; at this time, for each first grid area, the cross-correlation information between the pixels can be used
  • the matching point pair between the first grid area and the second grid area corresponding to the first grid area in the second modal image is obtained, so as to obtain the matching point pair according to the matching point pair.
  • the grid transformation matrix between the first grid region and the second grid region corresponding to the first grid region in the second modal image and then, according to each grid transformation matrix, the The second modality image is transformed into a target image aligned with respect to the first modality image, thereby realizing image alignment between images of different modality.
  • the matching point pairs required for the image alignment are determined based on the cross-correlation information between the pixels, the interference caused by the difference in the gradient directions of the images of different modalities in the structurally similar regions can be reduced, Therefore, the accuracy of matching point pairs is high, and accordingly, the alignment accuracy of the final target image is also ensured, which avoids that the existing feature point detection and matching based on SIFT and other methods cannot be found between images of different modalities. Accurately match point pairs, so that the image alignment accuracy between images of different modalities is poor.
  • FIG. 5 shows a structural block diagram of an image alignment apparatus provided by the embodiment of the present application. For convenience of description, only the part related to the embodiment of the present application is shown.
  • the image alignment device 5 includes:
  • a determination module 501 configured to determine at least two first candidate matching points in the first grid area for each first grid area pre-divided in the first modal image
  • a search module 502 configured to search for the first target pixel point corresponding to the first candidate matching point from the second modal image for each first candidate matching point in the first grid area, wherein, The cross-correlation information between the first target pixel point and the corresponding first candidate matching point meets a preset cross-correlation condition;
  • the first processing module 503 is configured to, if the first target pixel corresponding to the first candidate matching point is found from the second modal image, compare the first candidate matching point with the first matching point.
  • the first target pixel point corresponding to the candidate matching point is used as a set of matching point pairs between the first modal image and the second modal image;
  • the second processing module 504 is configured to obtain, according to the matching point pair, the distance between the first grid area and the second grid area corresponding to the first grid area in the second modal image The grid transformation matrix of , wherein the position of the first grid area in the first modal image and the second grid area corresponding to the first grid area in the same position;
  • the transformation module 505 is configured to transform the second modal image into a target image aligned with respect to the first modal image according to each grid transformation matrix.
  • the determining module 501 is specifically used for:
  • the determining module 501 specifically includes:
  • a first determining unit configured to, for each first grid area pre-divided in the first modal image, according to the first gradient value of each first pixel point in the first grid area, and the second gradient value of each second pixel in the second grid area corresponding to the first grid area, to determine the first grid area and the second grid area corresponding to the first grid area Alignment error between;
  • a second determining unit configured to determine the number of the first candidate matching points in the first grid area according to the alignment error
  • a third determining unit configured to determine a first candidate matching point in the first grid area according to the number of the first candidate matching points.
  • the first determining unit specifically includes:
  • a first processing subunit configured to, for each first pixel in the first grid area, compare the first gradient value of the first pixel with the second pixel corresponding to the first pixel The absolute value of the difference between the second gradient values of the point is taken as the first absolute value, wherein the position of the first pixel point in the first modal image and the first pixel point corresponding to the first pixel point The positions of the two pixel points in the second modal image are the same;
  • the second processing subunit is configured to use the sum of the absolute values of the first gradient values in the first grid area as the first summation result, and use the second grid area corresponding to the first grid area The sum of the absolute values of the respective second gradient values in the grid region is used as the second summation result;
  • a determination subunit configured to determine the relationship between the first grid area and the Alignment error between a second grid area corresponding to a grid area.
  • the image alignment device 5 further includes:
  • an acquisition module for acquiring the first original image captured by the first camera and the second original image captured by the second camera
  • a correction module configured to correct the first original image and the second original image respectively according to the pre-calibrated first camera parameters of the first camera and the second camera parameters of the second camera;
  • a third processing module configured to take the corrected first original image as the first modality image, and take the corrected second original image as the second modality image, wherein the first modality image and The second modality images are aligned in coplanar rows.
  • the image alignment device 5 further includes:
  • the second determining module is configured to, for each second grid area pre-divided in the second modal image, determine the grid area in the second grid area according to the gradient information of the second grid area at least two second candidate matching points;
  • the second search module is configured to, for each second candidate matching point in the second grid area, search for the second target pixel point corresponding to the second candidate matching point from the first modal image, wherein , the cross-correlation information between the second target pixel point and the corresponding second candidate matching point meets a preset cross-correlation condition;
  • the fourth processing module is configured to, if the second target pixel corresponding to the second candidate matching point is found from the first modal image, compare the second candidate matching point with the second candidate matching point.
  • the second target pixel points corresponding to the matching points are used as a set of matching point pairs between the first modal image and the second modal image.
  • the search module 502 specifically includes:
  • a fourth determining unit configured to, for each first candidate matching point in the first grid area, determine a region of interest corresponding to the first candidate matching point from the second modal image
  • a calculation unit for calculating the cross-correlation metric value between the pixel and the first candidate matching point for each pixel in the region of interest
  • the first processing unit is configured to, if the maximum value of the cross-correlation metric values corresponding to each pixel in the region of interest is greater than a preset threshold, then the maximum value in the region of interest corresponding to the The pixel point is used as the first target pixel point of the first candidate matching point.
  • the cross-correlation metric value is a normalized cross-correlation value
  • the computing unit is specifically used for:
  • the normalized cross-correlation value between the pixel and the first candidate matching point is calculated according to the specified correlation region, where the specified correlation region is the specified correlation region. a designated area centered on the pixel in the second modal image.
  • the second processing module 504 specifically includes:
  • the construction unit is configured to, according to the matching point pair, the coordinates of the designated vertex in the first mesh area, and the second corresponding to the first mesh area of the designated vertex in the first mesh area the desired coordinates in the grid area to construct a least squares model;
  • a solving unit configured to solve the least squares model, and obtain the desired coordinates of the specified vertex in the first grid region in the second grid region corresponding to the first grid region;
  • the second processing unit is configured to obtain the first mesh area and the first mesh area according to the expected coordinates and the coordinates of the specified vertex in the second mesh area corresponding to the first mesh area a homography matrix between the second grid regions corresponding to the grid regions, and using the homography matrix as the grid transformation matrix;
  • the transformation module 505 specifically includes:
  • a transformation unit configured to, for each second grid region, perform perspective transformation on the second grid region according to the homography matrix corresponding to the second grid region;
  • the third processing unit is configured to obtain a target image aligned relative to the first modal image according to each perspective transformed second grid area.
  • FIG. 6 is a schematic structural diagram of a terminal device provided by an embodiment of the present application.
  • the terminal device 6 in this embodiment includes: at least one processor 60 (only one is shown in FIG. 6 ), a memory 61 , and is stored in the above-mentioned memory 61 and can run on the above-mentioned at least one processor 60
  • the computer program 62 when the processor 60 executes the computer program 62, implements the steps in any of the above image alignment method embodiments.
  • the above-mentioned terminal device 6 may be a server, a mobile phone, a wearable device, an augmented reality (AR)/virtual reality (VR) device, a desktop computer, a notebook, a desktop computer, a handheld computer and other computing devices.
  • the terminal device may include, but is not limited to, a processor 60 and a memory 61 .
  • FIG. 6 is only an example of the terminal device 6, and does not constitute a limitation on the terminal device 6, and may include more or less components than the one shown, or combine some components, or different components , for example, may also include input devices, output devices, network access devices, and so on.
  • the above-mentioned input devices may include keyboards, touchpads, fingerprint collection sensors (for collecting user's fingerprint information and fingerprint direction information), microphones, cameras, etc.
  • output devices may include displays, speakers, and the like.
  • the above-mentioned processor 60 can be a central processing unit (Central Processing Unit, CPU), and the processor 60 can also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the above-mentioned memory 61 may be an internal storage unit of the above-mentioned terminal device 6 in some embodiments, such as a hard disk or a memory of the terminal device 6 .
  • the above-mentioned memory 61 may also be an external storage device of the above-mentioned terminal device 6 in other embodiments, such as a plug-in hard disk equipped on the above-mentioned terminal device 6, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital) , SD) card, flash memory card (Flash Card), etc.
  • the above-mentioned memory 61 may also include both the internal storage unit of the above-mentioned terminal device 6 and an external storage device.
  • the above-mentioned memory 61 is used to store an operating system, an application program, a boot loader (Boot Loader), data, and other programs, for example, program codes of the above-mentioned computer programs, and the like.
  • the above-mentioned memory 61 can also be used to temporarily store data that has been output or is to be output.
  • the above-mentioned terminal device 6 may also include a network connection module, such as a Bluetooth module, a Wi-Fi module, a cellular network module, etc., which will not be repeated here.
  • a network connection module such as a Bluetooth module, a Wi-Fi module, a cellular network module, etc., which will not be repeated here.
  • the processor 60 executes the computer program 62 to implement the steps in any of the above image alignment method embodiments, for each first grid area pre-divided in the first modal image, determine at least two first candidate matching points in the first grid area; then, for each first candidate matching point in the first grid area, searching for the first candidate matching point from the second modal image The first target pixel point corresponding to the candidate matching point, wherein the cross-correlation information between the first target pixel point and the corresponding first candidate matching point meets a preset cross-correlation condition. At this time, the similarity between the pixel points can be measured by the cross-correlation information, so as to find the first target pixel point corresponding to the first candidate matching point.
  • the first candidate matching point and the first matching point corresponding to the first candidate matching point is used as a set of matching point pairs between the first modal image and the second modal image; at this time, for each first grid area, the cross-correlation information between the pixels can be used
  • the matching point pair between the first grid area and the second grid area corresponding to the first grid area in the second modal image is obtained, so as to obtain the matching point pair according to the matching point pair.
  • the grid transformation matrix between the first grid region and the second grid region corresponding to the first grid region in the second modal image and then, according to each grid transformation matrix, the The second modality image is transformed into a target image aligned relative to the first modality image, thereby realizing image alignment between images of different modality.
  • the matching point pairs required for the image alignment are determined based on the cross-correlation information between the pixels, the interference caused by the difference in the gradient directions of the images of different modalities in the structurally similar regions can be reduced, Therefore, the accuracy of matching point pairs is high, and accordingly, the alignment accuracy of the final target image is also guaranteed, and the existing feature point detection and matching based on SIFT and other methods cannot be found between images of different modalities. Accurately match point pairs, which leads to the problem of poor image alignment accuracy between images of different modalities.
  • Embodiments of the present application further provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the foregoing method embodiments can be implemented.
  • the embodiments of the present application provide a computer program product, when the computer program product runs on a terminal device, so that the terminal device can implement the steps in the foregoing method embodiments when executed.
  • the above-mentioned integrated units are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on this understanding, the present application realizes all or part of the processes in the methods of the above-mentioned embodiments, which can be completed by instructing the relevant hardware through a computer program.
  • the above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program is in When executed by the processor, the steps of the foregoing method embodiments can be implemented.
  • the above-mentioned computer program includes computer program code, and the above-mentioned computer program code may be in the form of source code, object code form, executable file or some intermediate form.
  • the above-mentioned computer-readable medium may include at least: any entity or device capable of carrying the computer program code to the photographing device/terminal device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), electrical carrier signals, telecommunication signals, and software distribution media.
  • ROM read-only memory
  • RAM Random Access Memory
  • electrical carrier signals telecommunication signals
  • software distribution media for example, U disk, mobile hard disk, disk or CD, etc.
  • computer readable media may not be electrical carrier signals and telecommunications signals.
  • the disclosed apparatus/network device and method may be implemented in other manners.
  • the apparatus/network device embodiments described above are only illustrative.
  • the division of the above modules or units is only a logical function division.
  • the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.
  • the units described above as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Apparatus For Radiation Diagnosis (AREA)
  • Measuring And Recording Apparatus For Diagnosis (AREA)
  • Magnetic Resonance Imaging Apparatus (AREA)

Abstract

L'invention concerne un procédé d'alignement d'image, consistant : pour chaque première zone de grille obtenue au moyen d'une prédivision dans une première image modale, à déterminer au moins deux premiers points de correspondance candidats dans la première zone de grille ; pour chaque premier point de correspondance candidat dans la première zone de grille, à rechercher, dans une seconde image modale, un premier point de pixel cible correspondant au premier point de correspondance candidat, des informations de corrélation croisée entre le premier point de pixel cible et le premier point de correspondance candidat correspondant étant conformes à une condition de corrélation croisée prédéfinie ; à déterminer une paire de points de correspondance entre la première image modale et la seconde image modale selon les premiers points de pixel cible trouvés ; en fonction de la paire de points de correspondance, à obtenir une matrice de transformation de grille entre la première zone de grille et une seconde zone de grille qui correspond à la première zone de grille dans la seconde image modale ; et, selon chaque matrice de transformation de grille, à transformer la seconde image modale en une image cible qui est alignée par rapport à la première image modale.
PCT/CN2021/117471 2020-11-09 2021-09-09 Procédé d'alignement d'image, appareil d'alignement d'image et dispositif terminal WO2022095596A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011237926.4 2020-11-09
CN202011237926.4A CN112348863B (zh) 2020-11-09 2020-11-09 图像对齐方法、图像对齐装置及终端设备

Publications (1)

Publication Number Publication Date
WO2022095596A1 true WO2022095596A1 (fr) 2022-05-12

Family

ID=74429333

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/117471 WO2022095596A1 (fr) 2020-11-09 2021-09-09 Procédé d'alignement d'image, appareil d'alignement d'image et dispositif terminal

Country Status (2)

Country Link
CN (1) CN112348863B (fr)
WO (1) WO2022095596A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116625385A (zh) * 2023-07-25 2023-08-22 高德软件有限公司 路网匹配方法、高精地图构建方法、装置及设备
CN117173439A (zh) * 2023-11-01 2023-12-05 腾讯科技（深圳）有限公司 基于gpu的图像处理方法、装置、存储介质及电子设备
CN117745537A (zh) * 2024-02-21 2024-03-22 微牌科技(浙江)有限公司 隧道设备温度检测方法、装置、计算机设备和存储介质

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112348863B (zh) * 2020-11-09 2022-06-21 Oppo广东移动通信有限公司 图像对齐方法、图像对齐装置及终端设备
CN113538606B (zh) * 2021-08-17 2022-07-22 数坤(北京)网络科技股份有限公司 图像关联方法、装置、计算机可读存储介质及电子设备
US20230075297A1 (en) * 2021-09-03 2023-03-09 Kla Corporation Wafer alignment improvement through image projection-based patch-to-design alignment
CN114491131B (zh) * 2022-01-24 2023-04-18 北京至简墨奇科技有限公司 对候选图像进行重新排序的方法、装置和电子设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103325085A (zh) * 2012-03-02 2013-09-25 索尼公司 自动图像对齐
CN103679714A (zh) * 2013-12-04 2014-03-26 中国资源卫星应用中心 一种基于梯度互相关的光学和sar图像自动配准方法
CN108154526A (zh) * 2016-12-06 2018-06-12 奥多比公司 突发模式图像的图像对准
US10055898B1 (en) * 2017-02-22 2018-08-21 Adobe Systems Incorporated Multi-video registration for video synthesis
CN112348863A (zh) * 2020-11-09 2021-02-09 Oppo广东移动通信有限公司 图像对齐方法、图像对齐装置及终端设备

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH09274659A (ja) * 1996-04-08 1997-10-21 Kobe Steel Ltd 画像位置合わせ方法
EP2245592B1 (fr) * 2008-01-18 2013-03-27 Koninklijke Philips Electronics N.V. Métrique d'alignement de superposition d'image
CN103065323B (zh) * 2013-01-14 2015-07-15 北京理工大学 一种基于单应性变换矩阵的分段空间对准方法
US9245201B1 (en) * 2013-03-15 2016-01-26 Excelis Inc. Method and system for automatic registration of images

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103325085A (zh) * 2012-03-02 2013-09-25 索尼公司 自动图像对齐
CN103679714A (zh) * 2013-12-04 2014-03-26 中国资源卫星应用中心 一种基于梯度互相关的光学和sar图像自动配准方法
CN108154526A (zh) * 2016-12-06 2018-06-12 奥多比公司 突发模式图像的图像对准
US10055898B1 (en) * 2017-02-22 2018-08-21 Adobe Systems Incorporated Multi-video registration for video synthesis
CN112348863A (zh) * 2020-11-09 2021-02-09 Oppo广东移动通信有限公司 图像对齐方法、图像对齐装置及终端设备

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116625385A (zh) * 2023-07-25 2023-08-22 高德软件有限公司 路网匹配方法、高精地图构建方法、装置及设备
CN116625385B (zh) * 2023-07-25 2024-01-26 高德软件有限公司 路网匹配方法、高精地图构建方法、装置及设备
CN117173439A (zh) * 2023-11-01 2023-12-05 腾讯科技（深圳）有限公司 基于gpu的图像处理方法、装置、存储介质及电子设备
CN117745537A (zh) * 2024-02-21 2024-03-22 微牌科技(浙江)有限公司 隧道设备温度检测方法、装置、计算机设备和存储介质
CN117745537B (zh) * 2024-02-21 2024-05-17 微牌科技(浙江)有限公司 隧道设备温度检测方法、装置、计算机设备和存储介质

Also Published As

Publication number Publication date
CN112348863B (zh) 2022-06-21
CN112348863A (zh) 2021-02-09

Similar Documents

Publication Publication Date Title
WO2022095596A1 (fr) Procédé d'alignement d'image, appareil d'alignement d'image et dispositif terminal
WO2021115071A1 (fr) Procédé et appareil de reconstruction tridimensionnelle pour image d'endoscope monoculaire, et dispositif terminal
WO2020207191A1 (fr) Procédé et appareil permettant de déterminer une zone masquée d'un objet virtuel, et dispositif terminal
US7554575B2 (en) Fast imaging system calibration
WO2017181899A1 (fr) Procédé et dispositif de vérification faciale in vivo
US10645364B2 (en) Dynamic calibration of multi-camera systems using multiple multi-view image frames
WO2021136386A1 (fr) Procédé de traitement de données, terminal et serveur
US10915998B2 (en) Image processing method and device
WO2021196548A1 (fr) Procédé, appareil, et système de détermination de la distance
CN111160178A (zh) 图像处理方法及装置、处理器、电子设备及存储介质
CN112085771B (zh) 图像配准方法、装置、终端设备及计算机可读存储介质
WO2018184255A1 (fr) Procédé et dispositif de correction d'image
CN113362445B (zh) 基于点云数据重建对象的方法及装置
CN112233189B (zh) 多深度相机外部参数标定方法、装置及存储介质
CN113793387A (zh) 单目散斑结构光系统的标定方法、装置及终端
CN114298902A (zh) 一种图像对齐方法、装置、电子设备和存储介质
CN111210506A (zh) 一种三维还原方法、系统、终端设备和存储介质
WO2016208404A1 (fr) Dispositif et procédé de traitement d'informations, et programme
CN114078165A (zh) 光学成像模组的标定方法、畸变校正的方法及相关设备
WO2023010565A1 (fr) Procédé et appareil d'étalonnage d'un système de lumière structurée à chatoiement monoculaire, et terminal
CN112200842B (zh) 一种图像配准方法、装置、终端设备及存储介质
WO2022204953A1 (fr) Procédé et appareil pour déterminer un angle d'inclinaison et dispositif terminal
CN112102378B (zh) 图像配准方法、装置、终端设备及计算机可读存储介质
CN113838151A (zh) 相机标定方法、装置、设备及介质
WO2024002186A1 (fr) Procédé et appareil de fusion d'images, et support de stockage

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21888295

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21888295

Country of ref document: EP

Kind code of ref document: A1