CN117291958A - Image matching method, device, equipment and medium - Google Patents

Abstract

The embodiment of the disclosure relates to an image matching method, device, equipment and medium, wherein the method comprises the following steps: determining a target image block corresponding to the first feature point in the first image; acquiring a camera rotation matrix of a first image, and acquiring a gravity vector under a camera coordinate system; determining a horizontal unit vector and a vertical unit vector under a camera coordinate system according to the camera rotation matrix; acquiring a rotation angle of the first image relative to the center point according to the vertical unit vector, the gravity vector and the horizontal unit vector; and determining first feature description information of the first feature points according to the rotation angle and the target image block, and performing image matching processing on the first image according to the first feature description information. Therefore, the rotation angle of the image is determined based on the gravity direction irrelevant to the image, and the extraction direction of the extracted feature description information is determined based on the rotation angle, so that the accuracy and the uniqueness of the extracted feature description information are ensured, and the robustness of image matching is improved.

Description

Image matching method, device, equipment and medium

Technical Field

The disclosure relates to the technical field of image processing, and in particular relates to an image matching method, device, equipment and medium.

Background

In scenes such as visual map positioning, it is common to perform similarity matching processing on two images, for example, in a map positioning scene, a scene image is photographed, and the photographed scene image is subjected to similarity matching with a pre-stored image to determine the current position in the map.

In the related art, similarity matching is performed on two images according to feature points, a descriptor algorithm (Binary Robust Independent Elementary Features, BRIEF) is used, N point pairs are selected in a certain mode around the feature points to be matched, and comparison results of the N point pairs are combined to serve as description information of the feature points. In order to ensure the rotation invariance of N point pairs selection, the direction information of a characteristic point is needed, the direction information is determined by using a gray centroid method based on the direction information as the extraction direction of the extracted point pairs in a traditional mode, the gray centroid method uses an image gray value as the center of weight, the distance of an image block is determined, the centroid of the image block is further determined based on the distance of the image block, and the geometric centers (characteristic point positions) O and the centroid C of the linked image blocks are used for obtaining a direction vectorWith the direction vector +.>As the extraction direction of the N pairs of points.

However, when the extraction directions of extracting N pairs of points are determined based on the gray value centroid method, the gray value of the image needs to be taken as a reference to determine the center of the weight, so that depending on the presentation of the gray value of the image itself, if the noise in the image is too large, the extraction directions of the N pairs of points are obviously caused to have errors, thereby causing the accuracy of the extracted feature description information to be not high and further affecting the matching accuracy of the image.

Disclosure of Invention

In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides an image matching method, apparatus, device, and medium, which determine a rotation angle of an image based on a gravity direction unrelated to the image itself as a reference, and determine an extraction direction of extracting feature description information based on the rotation angle, thereby ensuring accuracy and uniqueness of the extracted feature description information, and improving robustness of image matching.

The embodiment of the disclosure provides an image matching method, which comprises the following steps: determining a target image block corresponding to a first feature point in a first image, wherein the center point of the target image block is the first feature point; acquiring a camera rotation matrix of the first image, and acquiring a gravity vector under a camera coordinate system; determining a horizontal unit vector and a vertical unit vector under a camera coordinate system according to the camera rotation matrix; acquiring a rotation angle of the first image relative to the center point according to the vertical unit vector, the gravity vector and the horizontal unit vector; and determining first feature description information of the first feature point according to the rotation angle and the target image block, and performing image matching processing on the first image according to the first feature description information.

The embodiment of the disclosure also provides an image matching device, which comprises: the first determining module is used for determining a target image block corresponding to a first feature point in a first image, wherein the center point of the target image block is the first feature point; the first acquisition module is used for acquiring a camera rotation matrix of the first image and acquiring a gravity vector under a camera coordinate system; the second determining module is used for determining a horizontal unit vector and a vertical unit vector under a camera coordinate system according to the camera rotation matrix; the second acquisition module is used for acquiring the rotation angle of the first image relative to the center point according to the vertical unit vector, the gravity vector and the horizontal unit vector; a third determining module, configured to determine first feature description information of the first feature point according to the rotation angle and the target image block; and the image matching module is used for carrying out image matching processing on the first image according to the first characteristic description information.

The embodiment of the disclosure also provides an electronic device, which comprises: a processor; a memory for storing the processor-executable instructions; the processor is configured to read the executable instructions from the memory and execute the instructions to implement an image matching method as provided in an embodiment of the disclosure.

The present disclosure also provides a computer-readable storage medium storing a computer program for executing the image matching method as provided by the embodiments of the present disclosure.

Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has the following advantages:

according to the image matching scheme provided by the embodiment of the disclosure, a target image block corresponding to a first feature point in a first image is determined, wherein the center point of the target image block is the first feature point, a camera rotation matrix of the first image is obtained, a gravity vector under a camera coordinate system is obtained, a horizontal unit vector and a vertical unit vector under the camera coordinate system are determined according to the camera rotation matrix, further, the rotation angle of the first image relative to the center point is obtained according to the vertical unit vector, the gravity vector and the horizontal unit vector, first feature description information of the first feature point is determined according to the rotation angle and the target image block, and image matching processing is performed on the first image according to the first feature description information. Therefore, the rotation angle of the image is determined based on the gravity direction irrelevant to the image, and the extraction direction of the extracted feature description information is determined based on the rotation angle, so that the accuracy and the uniqueness of the extracted feature description information are ensured, and the robustness of image matching is improved.

Drawings

The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale.

Fig. 1 is a schematic view of an image matching scene in the prior art according to an embodiment of the present disclosure;

fig. 2 is a schematic diagram of an image matching scenario provided in an embodiment of the present disclosure;

fig. 3 is a schematic flow chart of an image matching method according to an embodiment of the disclosure;

fig. 4 is a schematic diagram of another image matching scenario provided in an embodiment of the present disclosure;

fig. 5 is a schematic view of another image matching scenario provided in an embodiment of the present disclosure;

fig. 6 is a flowchart of another image matching method according to an embodiment of the present disclosure;

fig. 7 is a schematic diagram of another image matching scenario provided in an embodiment of the present disclosure;

fig. 8 is a flowchart of another image matching method according to an embodiment of the present disclosure;

fig. 9 is a schematic structural diagram of an image matching device according to an embodiment of the present disclosure;

fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure.

Detailed Description

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure have been shown in the accompanying drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.

It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order and/or performed in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.

The term "including" and variations thereof as used herein are intended to be open-ended, i.e., including, but not limited to. The term "based on" is based at least in part on. The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments. Related definitions of other terms will be given in the description below.

It should be noted that the terms "first," "second," and the like in this disclosure are merely used to distinguish between different devices, modules, or units and are not used to define an order or interdependence of functions performed by the devices, modules, or units.

It should be noted that references to "one", "a plurality" and "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be understood as "one or more" unless the context clearly indicates otherwise.

The names of messages or information interacted between the various devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

As described above, when determining the extraction direction of the descriptors of the feature points in the image based on the gray centroid method, the matching effect tends to be poor under the condition that the image noise is large depending on the image itself. Even, as shown in fig. 1, if two symmetric feature points exist in the image, the two feature points cannot be distinguished based on the gray centroid method, for example, for the symmetric 2 feature points a and B, the determined gray centroids are Q1 and Q2 respectively, and based on fig. 1, it is known that the gray values of the point pairs determined under the direction vectors of the feature points a to Q1 are the same as those of the point pairs determined under the direction vectors of the feature points B to Q2, so it is obvious that the extracted feature description information of the feature points a and B is the same, and the two feature points a and B cannot be distinguished, thereby further affecting the accuracy of image matching.

In order to solve the above technical problems, the consistency of the extraction directions of the feature description information of the extracted feature points is ensured, and in the embodiment of the disclosure, the extraction directions of the feature description information of the extracted feature points are determined by means of gravity directions which are irrelevant to the images, rather than relying on the images themselves, so that the extraction directions of the feature description information among the images are irrelevant to the images, and are not affected by image noise.

In addition, as shown in fig. 2, in the scene shown in fig. 1, when the direction of extracting the feature description information is determined based on the gravity direction as a reference, since the gray values of the point pairs extracted by even the symmetrical feature points a and B are different in the gravity direction, it is possible to ensure that the corresponding feature description information is different for the symmetrical feature points, and the accuracy of image matching is ensured.

In order to solve the above-mentioned problems, embodiments of the present disclosure provide an image matching method, which is described below with reference to specific embodiments.

Fig. 3 is a flowchart of an image matching method according to an embodiment of the present disclosure, where the method may be performed by an image matching device, and the device may be implemented by software and/or hardware, and may be generally integrated in an electronic device. As shown in fig. 3, the method includes:

In step 301, a target image block corresponding to the first feature point in the first image is determined, where a center point of the target image block is the first feature point.

The first image may be any one of images to be matched, and in this embodiment, since matching of images is achieved based on feature points as a matching object, in an embodiment of the present disclosure, a method for acquiring feature description information of a first feature point in the first image is mainly disclosed. The first feature point may be any feature point in the first image.

In this embodiment, based on the feature registration principle of algorithms such as BRIEF, in order to obtain feature description information of a first feature point, a target image block corresponding to the first feature point is obtained, where a center point of the target image block is the first feature point.

In some possible embodiments, the target image block is determined in the first image based on the preset size and the preset shape, centered on the first feature point.

The preset size and the preset shape can be calibrated according to a scene, and the preset size is used for limiting the size of the preset shape.

In the actual implementation process, in order to ensure the rotation consistency, the preset shape can be a circle, the round target image blocks rotate around the circle center anyway, and the front round target image blocks and the rear round target image blocks are completely overlapped. This ensures that the pixels participating in calculating the feature description information after the rotation of the target image block are identical to those before the rotation.

Step 302, a camera rotation matrix of the first image is acquired, and a gravity vector in a camera coordinate system is acquired.

The gravity vector is understood to be the direction vector of the gravity direction in the world coordinate system in the camera coordinate system.

It is easy to understand that the gravitational direction is fixed, whatever the shooting angle of the image, the camera coordinate system is, and the gravitational direction is constant under the world coordinate system, and therefore, in the embodiment of the present disclosure, the gravitational vector of the gravitational direction under the camera coordinate system determined with the gravitational direction as a reference is used to further determine the rotation angle of the image based on the gravitational vector.

In some possible embodiments, gravity direction information in the world coordinate system may be acquired, which may be acquired by a sensor in the camera device that reads the captured first image, and further, from the camera rotation matrix and the gravity direction information, a gravity vector of the gravity direction information in the camera coordinate system under the action of the camera rotation matrix is determined.

In step 303, a horizontal unit vector and a vertical unit vector in the world camera coordinate system are determined according to the camera rotation matrix.

The camera rotation matrix is understood as an external parameter of a camera capturing the first image, and is a rotation matrix 3*3 for converting feature points in the world coordinate system into the camera coordinate system, so that a horizontal unit vector (X-axis direction) and a vertical unit vector (Z-axis direction) exhibited by the coordinate system in the world coordinate system due to the influence of the rotation angle at the time of capturing the camera in the camera coordinate system can be acquired based on the camera rotation coordinates.

In one embodiment of the present disclosure, a horizontal rotation parameter and a vertical rotation parameter corresponding to a camera rotation matrix are determined, a horizontal unit vector is determined according to the horizontal rotation parameter, and a vertical unit vector is determined according to the vertical rotation parameter.

Wherein the rotation parameter represents a conversion of the world coordinate system into the camera coordinate system, and therefore, a unit vector in the camera coordinate system can be determined based on the rotation parameter.

For example, if the rotation angle of the camera coordinate system in the X-axis with respect to the world coordinate system is α, the rotation matrix in the X-axis is represented by the following formula (1), the rotation matrix in the Y-axis is represented by the following formula (2), and the rotation matrix in the Z-axis is represented by the following formula (3).

The camera rotation matrix is shown in the following equation (4):

if it isThe horizontal unit vector can be determined as +.>Determining the vertical unit vector as +.>

Step 304, according to the vertical unit vector, the gravity vector and the horizontal unit vector, the rotation angle of the first image relative to the center point is obtained.

In one embodiment of the present disclosure, since the vertical unit vector and the horizontal unit vector represent the vertical direction vector of the world coordinate system and the actual presentation result of the horizontal direction vector under the camera coordinate system, and the gravity vector represents the theoretical presentation result of the vertical direction vector of the world coordinate system under the camera coordinate system, the rotation angle of the first image with respect to the gravity direction may be determined based on the comparison of the actual presentation result and the theoretical presentation result.

Because the above-mentioned rotation angle determination process is irrelevant to the gray value of the target image block itself, the rotation angle of the first image relative to the center point is obtained based on the vertical unit vector, the gravity vector and the horizontal unit vector, and is not affected by some parameters of the image itself, for example, is not affected by noise of the image itself, so that the consistency of rotation angles between images during image matching is ensured.

And 305, determining first feature description information of the first feature points according to the rotation angle and the target image block, and performing image matching processing on the first image according to the first feature description information.

In an embodiment of the present disclosure, after determining the rotation angle, determining first feature description information of a first feature point according to the rotation angle and the target image block, where the first feature description information is used to describe the first feature point, and performing feature matching on a feature serving as the first feature point and feature points of other images, the rotation angle in this embodiment may be used as an extraction direction of N pairs of points in the BRIEF algorithm.

In this embodiment, as shown in fig. 4, if the first feature point is a and the target image block is rectangular, the initial coordinate system of the target image block is OXY (the original direction of the corresponding target image block is the solid line direction), after determining the gravity vector OG, the target image block is rotated to the position indicated by the dashed line based on the calculated rotation angle γ calculated based on the OG, and at this time, the coordinate system of the rotated target image block is OX yg, and since the gravity direction of OX yg is determined to be fixed, the feature description information of the first feature point is extracted based on the coordinate system in this direction, so that the rotation consistency can be ensured.

When extracting the feature description information of the first feature point, a mode of extracting N pairs (N may be 218, 516, etc., which are not listed here) of point pairs in any BRIEF algorithm may be used. In one embodiment of the present disclosure, an original image coordinate system of a target image block may be modified according to a rotation angle to obtain a target image coordinate system, a center point is taken as an origin point under the target image coordinate system, a plurality of pairs of pixel points in the target image block are extracted according to a preset extraction strategy, a pixel value of each pair of pixel points is extracted, comparison result information of each pair of feature points is determined according to the pixel value, and first feature description information of a first feature point is determined according to a plurality of comparison result information corresponding to the plurality of pairs of pixel points.

For example, as shown in fig. 5, each pixel may be numbered according to the coordinate system after the rotation angle is rotated (each circle in the figure identifies a pixel point), a pixel point pair is extracted based on the extraction mode, the comparison result of the gray value of the pixel point pair is determined to be 1 according to the comparison result of the gray value of the pixel point pair (for example, if the gray value of 12 is greater than the gray value of 119 in the extracted gray values of the two pixel points 12 and 119, and if the gray value of 12 is less than the gray value of 119, the comparison result is determined to be 0), and the first feature description information of the first feature point is determined according to the comparison result of the N point pairs.

It is easy to understand that, since we observe an object from different distances, different directions, different angles and different lighting conditions, the size, shape and brightness distribution of the object may be different, but we can still regard it as an object in the brain, and the ideal feature description information should also be the same feature point should have similar feature description information in two different images of size, direction rotation and gray brightness variation, so in the embodiment of the present disclosure, on the premise of ensuring the spatial position consistency of the pixel point pairs extracted by the first feature description information, the reliability of performing the image matching processing on the first image based on the first feature description information can be ensured. In addition, when the feature description information is extracted based on the gravity direction, even the gray values of the point pairs extracted by the symmetrical feature points are different in the gravity direction, so that the difference of the corresponding feature description information can be ensured for the symmetrical feature points, and the accuracy of image matching is further ensured.

In summary, according to the image matching method of the embodiment of the present disclosure, a target image block corresponding to a first feature point in a first image is determined, a center point of the target image block is the first feature point, a camera rotation matrix of the first image is obtained, a gravity vector under a camera coordinate system is obtained, a horizontal unit vector and a vertical unit vector under the camera coordinate system are determined according to the camera rotation matrix, further, a rotation angle of the first image relative to the center point is obtained according to the vertical unit vector, the gravity vector and the horizontal unit vector, first feature description information of the first feature point is determined according to the rotation angle and the target image block, and image matching processing is performed on the first image according to the first feature description information. Therefore, the rotation angle of the image is determined based on the gravity direction irrelevant to the image, and the extraction direction of the extracted feature description information is determined based on the rotation angle, so that the accuracy and the uniqueness of the extracted feature description information are ensured, and the robustness of image matching is improved.

It should be noted that, in different application scenarios, the manner of acquiring the rotation angle of the first image relative to the center point according to the vertical unit vector, the gravity vector and the horizontal unit vector is different, and examples are as follows:

In one embodiment of the present disclosure, a deep learning model is trained in advance according to a large number of sample data, and a vertical unit vector, a gravity vector, and a horizontal unit vector are input into the trained deep learning model to obtain a rotation angle output by the deep learning model.

In one embodiment of the present disclosure, as shown in fig. 6, acquiring a rotation angle of a first image with respect to a center point according to a vertical unit vector, a gravity vector, and a horizontal unit vector includes:

in step 601, a rotational reference vector is determined from the vertical unit vector and the gravity vector.

As shown in fig. 7, it can be understood that the vertical unit vector can be regarded as an actual unit vector and an ideal unit vector of the world coordinate system in the Z-axis direction under the camera coordinate system (the first feature point is taken as a center point of rotation, and consistency of the unit vector on the Z-axis needs to be ensured), the rotational reference vector can be regarded as an actual unit vector of the world coordinate system in the X-axis direction under the camera coordinate system, and the gravity vector can be regarded as an ideal unit vector of the gravity direction under the world coordinate system in the Y-axis direction under the camera coordinate system, and thus, based on the structure of the coordinate system, the rotational reference vector (ideal unit vector on the X-axis) can be determined based on the vertical unit vector (ideal unit vector on the Z-axis) and the gravity vector (ideal unit vector on the Y-axis).

In some possible embodiments, a vector product of the vertical unit vector and the gravity vector is calculated, and the vector product is determined to be the rotation reference vector, where if the vertical unit vector isGravity vector is->The rotation reference vector is +.>Then the calculation can be performed using the following equation (5):

in step 602, a rotation angle is obtained according to the rotation reference vector and the horizontal unit vector.

In this embodiment, after the rotation reference vector is determined, since the rotation reference vector is an ideal unit vector in the X-axis when the world coordinate system is converted into the camera coordinate system, and the horizontal unit vector is an actual unit vector in the X-axis in the world coordinate system under the camera coordinate system, the rotation angle of the current display angle of the target image block in the world coordinate system can be obtained from the rotation reference vector and the horizontal unit vector.

In some possible embodiments, the inner product and the outer product of the rotation reference vector and the horizontal unit vector can be calculated respectively, the ratio of the outer product and the inner product is calculated, and the arcsine function of the ratio is calculated to obtain the rotation angle, for example, if the rotation reference vector isThe horizontal unit vector is->And referring to FIG. 5, it can be seen that->Therefore, the rotation angle γ can be calculated using the following formula (6).

In other possible embodiments, based on the image rotation principle, the inner product and the outer product of the gravity direction vector and the vertical unit vector can be calculated respectively, wherein the vertical unit vector can be the camera rotation matrixAnd respectively calculating an inner product and an outer product of the gravity direction vector and the vertical unit vector, calculating the ratio of the outer product and the inner product, and calculating an arcsine function of the ratio to obtain the rotation angle.

In summary, the image matching method of the embodiment of the present disclosure may combine the gravity vector of the gravity direction information under the camera coordinates as a reference to determine the rotation angle of the first image, and determine the rotation angle by adopting parameters unrelated to the image itself, thereby ensuring the rotation invariance of similar feature points in image matching, and improving the accuracy of image matching.

Based on the above embodiment, after the first feature description information of the first feature point is determined from the rotation angle and the target image block, since the first feature information is extracted with the gravitational direction as the reference direction, since the gravitational direction is specified in the world coordinate system, the rotation uniformity of the position of the feature point in space at the time of image matching can be ensured based on the rotation direction determined by the gravitational direction. For example, we have a seal, on which some straight lines can be arranged, when the seal is used to seal on a picture, some point on the picture is extracted, under the condition that the seal is not changed, the picture is rotated, and then a seal is applied, then the point pair extracted at this time is different from the previous point pair. In order to make the 2 extracted dots the same, we need to rotate the stamp also by an angle. In this embodiment, the purpose of the rotation angle is to rotate the "stamp", so after the rotation angle is determined, the rotation angle is used as the feature description information extracted by the extraction angle, so that rotation invariance can be ensured, and the accuracy of image matching is improved. The following is a detailed description in connection with the image matching process:

In one embodiment of the present disclosure, as shown in fig. 8, performing image matching processing on a first image according to first feature description information, includes:

step 801, second feature description information of each reference feature point in the second image is acquired.

In one embodiment of the disclosure, a reference image block corresponding to each reference feature point in the second image is determined, a center point of the reference image block is a corresponding reference feature point, a reference camera rotation matrix of the reference image block is obtained, a reference gravity vector under a camera coordinate system is obtained, a reference horizontal unit vector and a reference vertical unit vector under a world coordinate system are determined according to the reference camera rotation matrix, a rotation angle of the second image is determined according to the reference vertical unit vector, the reference gravity vector and the reference horizontal unit vector, and second feature description information of the corresponding reference feature point is obtained according to the rotation angle and the reference image block.

Because the second feature description information and the first feature description information are extracted based on the gravity direction as the rotation angle, the second feature description information and the first feature description information have rotation invariance, and if the reference feature points and the first feature points are similar feature points in the same space, the first feature description information and the second feature description information are similar and cannot be influenced by different rotation angles of pictures.

Step 802, determining a second feature point matching the first feature point in the second image according to the first feature description information and the second feature description information.

In this embodiment, based on the feature point matching principle, the second feature point matched with the first feature point in the second image is determined according to the first feature description information and the second feature description information, and since the first feature description information and the second feature description information are extracted in the same spatial direction, if the second feature description information of the second feature point is successfully matched with the first feature description information, the first feature point and the second feature point belong to similar feature points in the same spatial position.

Step 803, performing image matching processing on the first image and the second image according to the first feature point and the second feature point.

The image matching processing manner of the first image and the second image according to the first feature point and the second feature point can be different in different application scenes:

in one embodiment of the present disclosure, whether the first image and the second image match may be determined according to the number of matches of the first feature point and the second feature point, or the like.

In one embodiment of the present disclosure, whether the first image and the second image match or not may be determined according to a matching weight of the first feature point and the second feature point or the like.

In summary, according to the image matching method disclosed by the embodiment of the disclosure, the gravity direction under the world coordinate system is taken as a reference to determine the conversion angle of the image, and the rotation angle is taken as the extraction direction of the description feature information during image matching, so that the rotation invariance of the feature description information during image matching is ensured, and the reliability of image matching is improved.

In order to implement the above embodiment, the present disclosure further proposes an image matching apparatus.

Fig. 9 is a schematic structural diagram of an image matching apparatus according to an embodiment of the present disclosure, where the apparatus may be implemented by software and/or hardware, and may be generally integrated in an electronic device to perform image matching. As shown in fig. 9, the apparatus includes: a first determination module 910, a first acquisition module 920, a second determination module 930, a second acquisition module 940, a third determination module 950, and an image matching module 960, wherein,

a first determining module 910, configured to determine a target image block corresponding to a first feature point in a first image, where a center point of the target image block is the first feature point;

a first acquisition module 920, configured to acquire a camera rotation matrix of the first image, and acquire a gravity vector in a camera coordinate system;

A second determining module 930 configured to determine a horizontal unit vector and a vertical unit vector in a world coordinate system according to the camera rotation matrix;

a second obtaining module 940, configured to obtain a rotation angle of the first image relative to the center point according to the vertical unit vector, the gravity vector, and the horizontal unit vector;

a third determining module 950, configured to determine first feature description information of the first feature point according to the rotation angle and the target image block;

and an image matching module 960, configured to perform image matching processing on the first image according to the first feature description information.

The image matching device provided by the embodiment of the disclosure can execute the image matching method provided by any embodiment of the disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

To achieve the above embodiments, the present disclosure also proposes a computer program product comprising a computer program/instruction which, when executed by a processor, implements the image matching method in the above embodiments.

Fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure.

Referring now in particular to fig. 10, a schematic diagram of an electronic device 1000 suitable for use in implementing embodiments of the present disclosure is shown. The electronic device 1000 in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and stationary terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 10 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.

As shown in fig. 10, the electronic device 1000 may include a processor (e.g., a central processing unit, a graphics processor, etc.) 1001 that may perform various appropriate actions and processes according to programs stored in a Read Only Memory (ROM) 1002 or programs loaded from a memory 1008 into a Random Access Memory (RAM) 1003. In the RAM 1003, various programs and data necessary for the operation of the electronic apparatus 1000 are also stored. The processor 1001, the ROM 1002, and the RAM 1003 are connected to each other by a bus 1004. An input/output (I/O) interface 1005 is also connected to bus 1004.

In general, the following devices may be connected to the I/O interface 1005: input devices 1006 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, and the like; an output device 1007 including, for example, a Liquid Crystal Display (LCD), speaker, vibrator, etc.; memory 1008 including, for example, magnetic tape, hard disk, etc.; and communication means 1009. The communication means 1009 may allow the electronic device 1000 to communicate wirelessly or by wire with other devices to exchange data. While fig. 10 shows an electronic device 1000 having various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead.

In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a non-transitory computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication device 1009, or installed from the memory 1008, or installed from the ROM 1002. The above-described functions defined in the image matching method of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1001.

It should be noted that the computer readable medium described in the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.

In some implementations, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.

The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to:

determining a target image block corresponding to a first feature point in a first image, wherein the center point of the target image block is the first feature point, acquiring a camera rotation matrix of the first image, acquiring a gravity vector under a camera coordinate system, determining a horizontal unit vector and a vertical unit vector under a world coordinate system according to the camera rotation matrix, further acquiring a rotation angle of the first image relative to the center point according to the vertical unit vector, the gravity vector and the horizontal unit vector, determining first feature description information of the first feature point according to the rotation angle and the target image block, and performing image matching processing on the first image according to the first feature description information. Therefore, the rotation angle of the image is determined based on the gravity direction irrelevant to the image, and the extraction direction of the extracted feature description information is determined based on the rotation angle, so that the accuracy and the uniqueness of the extracted feature description information are ensured, and the robustness of image matching is improved.

The electronic device may write computer program code for performing the operations of the present disclosure in one or more programming languages, including, but not limited to, an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units involved in the embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware. Wherein the names of the units do not constitute a limitation of the units themselves in some cases.

The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by persons skilled in the art that the scope of the disclosure referred to in this disclosure is not limited to the specific combinations of features described above, but also covers other embodiments which may be formed by any combination of features described above or equivalents thereof without departing from the spirit of the disclosure. Such as those described above, are mutually substituted with the technical features having similar functions disclosed in the present disclosure (but not limited thereto).

Moreover, although operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are example forms of implementing the claims.

Claims (13)

1. An image matching method, comprising the steps of:

determining a target image block corresponding to a first feature point in a first image, wherein the center point of the target image block is the first feature point;

acquiring a camera rotation matrix of the first image, and acquiring a gravity vector under a camera coordinate system;

determining a horizontal unit vector and a vertical unit vector under a camera coordinate system according to the camera rotation matrix;

acquiring a rotation angle of the first image relative to the center point according to the vertical unit vector, the gravity vector and the horizontal unit vector;

and determining first feature description information of the first feature point according to the rotation angle and the target image block, and performing image matching processing on the first image according to the first feature description information.

2. The method of claim 1, wherein determining the target image block in the first image corresponding to the first feature point comprises:

and taking the first characteristic point as a center, and determining the target image block in the first image according to a preset size and a preset shape.

3. The method of claim 1, wherein the acquiring a gravity vector in a camera coordinate system comprises:

acquiring gravity direction information under a world coordinate system;

and acquiring the gravity vector under a camera coordinate system according to the camera rotation matrix and the gravity direction information.

4. The method of claim 1, wherein the determining horizontal and vertical unit vectors in a camera coordinate system from the camera rotation matrix comprises:

determining a horizontal rotation parameter and a vertical rotation parameter corresponding to the camera rotation matrix;

determining the horizontal unit vector according to the horizontal rotation parameter;

and determining the vertical unit vector according to the vertical rotation parameter.

5. The method of claim 1, wherein the obtaining the rotation angle of the first image with respect to the center point based on the vertical unit vector, the gravity vector, and the horizontal unit vector comprises:

Determining a rotation reference vector from the vertical unit vector and the gravity vector;

and acquiring the rotation angle according to the rotation reference vector and the horizontal unit vector.

6. The method of claim 5, wherein said determining a rotation reference vector from said vertical unit vector and said gravity vector comprises:

calculating a vector product of the vertical unit vector and the gravity vector;

the vector product is determined to be the rotational reference vector.

7. The method of claim 5 or 6, wherein the obtaining the rotation angle from the rotation reference vector and the horizontal unit vector comprises:

respectively calculating an inner product and an outer product of the rotation reference vector and the horizontal unit vector;

calculating the ratio of the outer product and the inner product, and calculating an arcsine function of the ratio to obtain the rotation angle.

8. The method of claim 1, wherein the determining the first feature description information of the first feature point according to the rotation angle and the target image block comprises:

correcting an original image coordinate system of the target image block according to the rotation angle to obtain a target image coordinate system;

Extracting a plurality of pairs of pixel points in the target image block according to a preset extraction strategy by taking the central point as an origin in the target image coordinate system;

extracting pixel values of each pair of pixel points, and determining comparison result information of each pair of characteristic points according to the pixel values;

and determining first feature description information of the first feature points according to the comparison result information corresponding to the pairs of pixel points.

9. The method of claim 1, wherein performing image matching processing on the first image according to the first feature description information comprises:

acquiring second feature description information of each reference feature point in a second image;

determining a second feature point matched with the first feature point in the second image according to the first feature description information and the second feature description information;

and carrying out image matching processing on the first image and the second image according to the first characteristic point and the second characteristic point.

10. The method of claim 9, wherein the acquiring second feature description information of each reference feature point in the second image includes:

determining a reference image block corresponding to each reference feature point in the second image, wherein the center point of the reference image block is the corresponding reference feature point;

Acquiring a reference camera rotation matrix of the reference image block, and acquiring a reference gravity vector under a camera coordinate system;

determining a reference horizontal unit vector and a reference vertical unit vector under a camera coordinate system according to the reference camera rotation matrix;

determining a rotation angle of the second image according to the reference vertical unit vector, the reference gravity vector and the reference horizontal unit vector;

and acquiring second characteristic description information of the corresponding reference characteristic point according to the rotation angle and the reference image block.

11. An image matching apparatus, comprising:

the first determining module is used for determining a target image block corresponding to a first feature point in a first image, wherein the center point of the target image block is the first feature point;

the first acquisition module is used for acquiring a camera rotation matrix of the first image and acquiring a gravity vector under a camera coordinate system;

the second determining module is used for determining a horizontal unit vector and a vertical unit vector under a camera coordinate system according to the camera rotation matrix;

the second acquisition module is used for acquiring the rotation angle of the first image relative to the center point according to the vertical unit vector, the gravity vector and the horizontal unit vector;

A third determining module, configured to determine first feature description information of the first feature point according to the rotation angle and the target image block;

and the image matching module is used for carrying out image matching processing on the first image according to the first characteristic description information.

12. An electronic device, the electronic device comprising:

a processor;

a memory for storing the processor-executable instructions;

the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the image matching method of any one of the preceding claims 1-10.

13. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program for executing the image matching method according to any one of the preceding claims 1-10.