CN110335315B

CN110335315B - Image processing method and device and computer readable storage medium

Info

Publication number: CN110335315B
Application number: CN201910570613.1A
Authority: CN
Inventors: 杨宇尘; 陈岩; 方攀; 金珂
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2019-06-27
Filing date: 2019-06-27
Publication date: 2021-11-02
Anticipated expiration: 2039-06-27
Also published as: WO2020259365A1; CN110335315A

Abstract

The embodiment of the application discloses an image processing method and device and a computer readable storage medium, which can improve the accuracy of image processing, and the method can comprise the following steps: acquiring a current frame image to be processed; extracting current image feature points of the current frame image from the current frame image, and acquiring reference feature points corresponding to a reference frame based on the current image feature points; matching the current image characteristic points with the reference characteristic points based on a first preset matching threshold value to obtain first matching point pairs; amplifying the first preset matching threshold to obtain a second preset matching threshold, and obtaining a constraint condition based on the first matching point pair; under the constraint condition, matching the current image characteristic points with the reference characteristic points based on a second preset matching threshold to obtain second matching point pairs; and performing an image processing function based on the second matching point pairs.

Description

Image processing method and device and computer readable storage medium

Technical Field

The present application relates to the field of software engineering, and in particular, to an image processing method and apparatus, and a computer-readable storage medium.

Background

Currently, in the field of instant positioning and mapping (SLAM), repositioning is usually done in a way that restores the relative pose, i.e. the geometric relationship between the cameras of the two images. A commonly used method for recovering the relative pose is to violently match the image features of the current scene with the image features of the offline map, then calculate a preset basic matrix or homography matrix using matching point pairs obtained according to a violent matching method, and use the matrix calculation result to recover the relative pose between the two images.

In practical application, the matching point pairs obtained according to the violent matching method often have the problem of errors of the matching point pairs or the problem of too few matching point pairs, so that a basic matrix or a homography matrix calculated according to the matching point pairs obtained according to the violent matching method is not accurate enough, so that the relative pose between two recovered images has a large error, and finally the accuracy of the repositioning position recovered according to the relative pose is low. Further, the basis matrix is suitable for recovering a stereoscopic scene, the homography matrix is suitable for recovering a planar scene or an approximately planar scene, and if the preset basis matrix or homography matrix is not suitable for the type of the scene where the actual repositioning is performed, a large error can be caused in the relative pose recovered according to the preset basis matrix or homography matrix, and finally the accuracy of the repositioning position recovered according to the relative pose is low.

Disclosure of Invention

The embodiment of the application provides an image processing method, which can improve the accuracy of image processing, and finally improve the accuracy of relocation when the image processing function is a relocation function.

The technical scheme of the application is realized as follows:

the embodiment of the application discloses an image processing method, which comprises the following steps:

acquiring a current frame image to be processed;

extracting current image feature points of the current frame image from the current frame image, and acquiring reference feature points corresponding to a reference frame based on the current image feature points;

matching the current image characteristic points with the reference characteristic points based on a first preset matching threshold value to obtain first matching point pairs;

amplifying the first preset matching threshold to obtain a second preset matching threshold, and obtaining a constraint condition based on the first matching point pair;

under the constraint condition, matching the current image characteristic points with the reference characteristic points based on the second preset matching threshold to obtain second matching point pairs;

and performing an image processing function based on the second matching point pair.

In the above method, the matching, under the constraint condition, the current image feature point and the reference feature point based on the second preset matching threshold to obtain a second matching point pair includes:

matching the image characteristic points with the reference characteristic points based on the second preset matching threshold to obtain intermediate matching point pairs;

and screening out the second matching point pairs meeting the constraint condition from the intermediate matching point pairs.

In the above method, the screening out, from the intermediate matching point pairs, the second matching point pairs that satisfy the constraint condition includes:

determining a first mapping value based on the three-dimensional coordinates of each first feature point in the intermediate matching point pair and the constraint condition;

determining a first matching error based on the first mapping value;

and determining the second matching point pairs which are less than or equal to a preset error threshold value based on the first matching error.

screening out sub-matching point pairs except the first matching point pair from the intermediate matching point pair;

determining a second mapping value based on the three-dimensional coordinates of each second feature point in the sub-matching point pair and the constraint condition;

determining a second matching error based on the second mapping value;

determining a final sub-matching point pair smaller than or equal to a preset error threshold value based on the second matching error;

and taking the first matching point pair and the final sub-matching point pair as the second matching point pair.

In the above method, obtaining a constraint condition based on the first matching point pair includes:

acquiring at least one preset initial constraint condition;

calculating the at least one preset initial constraint condition by using the first matching point pair to obtain at least one sub-constraint condition;

determining at least one third mapping value based on the three-dimensional coordinates of each third feature point in the first matching point pair and the at least one sub-constraint condition;

determining at least one third matching error based on the at least one third mapping value;

determining the first matching point pair corresponding to the at least one sub-constraint condition and smaller than or equal to the preset error threshold value as a correct matching point pair based on the at least one third matching error;

respectively selecting one sub-constraint condition containing the most correct matching point pairs from the at least one preset initial constraint condition as at least one intermediate sub-constraint condition corresponding to each preset initial constraint condition;

selecting, as the constraint, an intermediate sub-constraint that contains the most correct matching point pairs from the at least one intermediate sub-constraint.

In the above method, the calculating the at least one preset initial constraint condition by using the first matching point pair to obtain at least one sub-constraint condition includes:

selecting a preset number of first matching point pairs from the first matching point pairs for combination according to each preset initial constraint condition, and repeating preset times to obtain first matching point pair combinations equal to the preset times in number, wherein each combination of the first matching point pair combinations comprises the preset number of the first matching point pairs;

calculating each preset initial constraint condition by using each combination in the first matching point pair combination to obtain the preset times of the sub-constraint conditions corresponding to each preset initial constraint condition, wherein the preset times of the sub-constraint conditions are used as final sub-constraint conditions of each preset initial constraint condition;

and determining a final sub-constraint condition of each preset initial constraint condition as the at least one sub-constraint condition in the at least one preset initial constraint condition.

In the above method, when the image processing function is a relocation function, the performing an image processing function based on the second matching point pair includes:

restoring at least one piece of relative pose information according to the second matching point pair;

for each piece of relative pose information, calculating a first image position coordinate of the current image feature point in a first shooting coordinate system and a second image position coordinate in a second shooting coordinate system from the second matching point pair; the first shooting coordinate system is a shooting coordinate system corresponding to the current image feature point, and the second shooting coordinate system is a shooting coordinate system corresponding to the reference feature point;

calculating a first reference position coordinate of the reference feature point in the first shooting coordinate system from the second matching point pair aiming at each piece of relative pose information; and second reference position coordinates in the second photographing coordinate system;

respectively verifying each piece of relative pose information based on the first image position coordinate, the second image position coordinate, the first reference position coordinate and the second reference position coordinate to obtain a verification result corresponding to at least one piece of relative pose information;

determining target pose information from the at least one relative pose information based on the verification result;

and realizing the repositioning function according to the target pose information.

In the above method, the recovering at least one piece of relative pose information according to the second matching point pair includes:

calculating at least one translation amount and at least one rotation amount in the first shooting coordinate system and the second shooting coordinate system according to the corresponding relation between the preset constraint condition and the camera pose and the second matching point pair;

and correspondingly combining the at least one translation amount and the at least one rotation amount to obtain the at least one relative pose information.

In the above method, the checking each piece of relative pose information based on the first image position coordinate, the second image position coordinate, the first reference position coordinate, and the second reference position coordinate, respectively, to obtain a check result corresponding to at least one piece of relative pose information includes:

obtaining a first image position sub-coordinate and a second image position sub-coordinate corresponding to each current image sub-feature point in the current image feature points and a first reference position sub-coordinate and a second reference position sub-coordinate corresponding to each reference sub-feature point in the reference feature points according to the first image position coordinate and the second image position coordinate;

regarding each piece of relative pose information, when the first image position sub-coordinate of each current image sub-feature point meets a first preset condition and the second image position sub-coordinate of each current image sub-feature point also meets the first preset condition, taking each current image sub-feature point as a correct current image sub-feature point, wherein the first preset condition is that the depth value of the three-dimensional coordinate is a preset value;

determining the correct current image sub-feature point in each current image sub-feature point from the current image feature points, thereby obtaining the correct current image feature point of the current image feature points;

for each piece of relative pose information, when the first reference position sub-coordinate of each reference sub-feature point meets the first preset condition and the second reference position sub-coordinate of each reference sub-feature point also meets the first preset condition, taking each reference sub-feature point as a correct reference sub-feature point;

determining the correct reference sub-feature point in each reference sub-feature point from the reference feature points, thereby obtaining the correct reference feature point of the reference feature points;

taking the correct current image feature point and the correct reference feature point as a verification result corresponding to each piece of relative pose information, so as to obtain a verification result corresponding to the at least one piece of relative pose information;

and determining target pose information from the at least one relative pose information based on the checking result.

In the above method, the determining target pose information from the at least one relative pose information based on the verification result includes:

counting the sum of the number of the correct sub current image feature points and the correct sub reference feature points in the verification result corresponding to each relative pose information from the verification result corresponding to the at least one relative pose information, so as to obtain the number of the correct results corresponding to each relative pose information;

and determining the target pose information with the maximum number of correct results from the number of correct results corresponding to each piece of relative pose information.

In the above method, the extracting a current image feature point of the current frame image from the current frame image, and obtaining a reference feature point corresponding to a reference frame based on the current image feature point includes:

extracting first pixel points representing angle boundaries with preset angular point quantity from the current frame image;

extracting features of second pixel points within a preset range of the first pixel points to obtain first image feature points;

extracting features of third pixel points outside the preset range of the first pixel points to obtain second image feature points;

taking the first image feature point and the second image feature point as the current image feature point;

matching the current image feature points with a preset map feature library, wherein the preset map feature library stores preset map feature points of at least one frame of preset map image frame;

and taking the preset map image frame which is most matched with the current image feature point in the preset map feature library as the reference frame, and taking the preset map feature point contained in the reference frame as the reference feature point.

In the above method, the first preset matching threshold includes: presetting a similarity threshold and a significance threshold; the matching the current image feature point and the reference feature point based on a first preset matching threshold to obtain a first matching point pair, including:

combining each current image sub-feature point in the current image feature points with the reference feature point one by one to obtain at least one sub-feature point pair;

aiming at the at least one sub-feature point pair, calculating the Hamming distance from each current image sub-feature point to the reference feature point to obtain at least one Hamming distance, wherein the at least one Hamming distance is in one-to-one correspondence with the at least one sub-feature point pair;

determining a minimum hamming distance and a next smallest hamming distance from the at least one hamming distance;

comparing the minimum hamming distance to the preset similarity threshold when the minimum hamming distance is less than the preset significance threshold multiplied by the next minimum hamming distance;

and when the minimum Hamming distance is smaller than or equal to the preset similarity threshold, determining a sub-feature point pair corresponding to each current image sub-feature point corresponding to the minimum Hamming distance from the at least one sub-feature point pair, so as to obtain the first matching point pair corresponding to the current image feature point.

An embodiment of the present application provides an image processing apparatus, including:

the acquisition unit is used for acquiring a current frame image to be processed;

the extraction unit is used for extracting the current image characteristic points of the current frame image from the current frame image and acquiring the reference characteristic points corresponding to the reference frame based on the current image characteristic points;

the matching unit is used for matching the current image characteristic points with the reference characteristic points based on a first preset matching threshold value to obtain first matching point pairs;

the calculation unit is used for amplifying the first preset matching threshold value to obtain a second preset matching threshold value and obtaining a constraint condition based on the first matching point pair;

the matching unit is further configured to match the current image feature point with the reference feature point based on the second preset matching threshold under the constraint condition to obtain a second matching point pair;

and the processing unit is used for performing an image processing function based on the second matching point pair.

a memory for storing executable instructions;

and the processor is used for realizing the image processing method provided by the embodiment of the application when the executable instructions stored in the memory are executed.

The embodiment of the application provides a computer-readable storage medium, which is used for storing executable instructions and is applied to an image processing device, and the executable instructions are executed by a processor to realize the image processing method provided by the embodiment of the application.

The embodiment of the application discloses an image processing method and device and a computer readable storage medium, wherein the method comprises the following steps: the image processing device acquires a current frame image to be processed; the image processing device extracts current image feature points of the current frame image from the current frame image and acquires reference feature points corresponding to a reference frame based on the current image feature points; the image processing device matches the current image characteristic points with the reference characteristic points based on a first preset matching threshold value to obtain first matching point pairs; the image processing device amplifies the first preset matching threshold to obtain a second preset matching threshold, and the image processing device obtains a constraint condition based on the first matching point pair; under the constraint condition, the image processing device matches the current image characteristic points with the reference characteristic points based on a second preset matching threshold value to obtain second matching point pairs; and performing an image processing function based on the second matching point pairs. By adopting the method, when the image matching is carried out, the image processing device firstly calculates the constraint condition according to the first matching point pair obtained by matching once, then amplifies the first preset matching threshold value under the constraint condition and then matches once, thereby developing more accurate second matching point pairs, and finally, the image processing device carries out image processing according to the second matching point pairs, thereby improving the number and the matching accuracy of the matching point pairs, and finally improving the accuracy of image processing.

Drawings

Fig. 1 is a flowchart of a first image processing method according to an embodiment of the present disclosure;

fig. 2 is a flowchart of a second image processing method according to an embodiment of the present application;

fig. 3 is a first schematic structural diagram of an image processing apparatus according to an embodiment of the present disclosure;

fig. 4 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application.

Detailed Description

In order to more clearly illustrate the embodiments of the present application or technical solutions in the prior art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

An embodiment of the present application provides an image processing method, as shown in fig. 1, the method may include:

and S101, acquiring a current frame image to be processed.

The image processing method provided by the embodiment of the application is suitable for initialization and relocation scenes of an SLAM system, such as scenes of Augmented Reality (AR) relocation, autonomous mobile robot relocation, unmanned relocation and the like, and is also suitable for recognition and matching of non-rigid deformation of an image, such as scenes of image matching between an object after the non-rigid deformation and an object in an original state and the like.

In the embodiment of the present application, when performing image processing, an image processing apparatus first needs to acquire a current frame image to be processed.

Illustratively, when the image processing function is a repositioning function, the image processing apparatus calls a shooting function to shoot a scene at the current position, and obtains a current frame image to be processed.

S102, extracting current image feature points of the current frame image from the current frame image, and acquiring reference feature points corresponding to a reference frame based on the current image feature points.

In the embodiment of the application, after the image processing device acquires a current frame image to be processed, the image processing device extracts feature points from the current frame image as current image feature points, then finds a reference frame which is most matched with the current image feature points from a reference image frame based on the current image feature points, and finally extracts the feature points from the reference frame as reference feature points.

In the embodiment of the application, the image processing device extracts, from the obtained current frame image, a point which can effectively reflect the essential features of the image and can identify an object in the image in the current frame image as a current image feature point.

In this embodiment of the present application, the image processing apparatus may extract the current image feature point from the current frame image by using a fast (features from obtained segment) algorithm, or may use another algorithm, which is specifically selected according to an actual situation, and this embodiment of the present application is not specifically limited.

In the embodiment of the application, after obtaining the current image feature point, the image processing device matches with the reference image frame according to the current image feature point, and finds a frame of reference image frame which is most matched with the current image feature point and is used as a reference frame; the image processing apparatus acquires feature points from a reference frame as reference feature points.

In this embodiment of the application, when the image processing apparatus performs matching in the reference image frame according to the feature points of the current image, the bag-of-words model of the current image may be calculated according to the image description feature points in the feature points of the current image, and then the bag-of-words model of the current image is used for matching.

Illustratively, when the image processing function is a repositioning function, the image processing apparatus calculates bag-of-words model features of the current frame image according to image description feature points in the feature points of the current frame image, finds a frame that best matches the bag-of-words model features of the current frame image in a preset map feature library according to the bag-of-words model features of the current frame image, and takes the found frame of the preset scene map as a repositioned reference frame. The preset map feature library is formed by feature points extracted from at least one preset map image frame by an image processing device during map construction, and each preset map image frame is stored in the preset map feature library in the form of preset map feature points.

S103, matching the current image characteristic points with the reference characteristic points based on a first preset matching threshold value to obtain first matching point pairs.

In the embodiment of the application, after the image processing device extracts the current image feature point of the current frame image from the current frame image and acquires the reference feature point corresponding to the reference frame based on the current image feature point, the image processing device matches the current frame image feature point with the reference frame feature point, and uses the current frame image feature point and the reference frame feature point which meet a first preset matching threshold as a first matching point pair. Each first matching point pair comprises a current frame image feature point and a reference frame feature point, and represents the position coordinates of pixel points in the same object in a current frame image coordinate system and a reference frame image coordinate system respectively.

In the embodiment of the present application, when the image processing apparatus matches the current image feature point with the reference feature point, a brute force matching method may be used. Other matching algorithms can also be used, and the selection is specifically performed according to the actual situation, and the embodiment of the application is not limited.

In the embodiment of the application, the image processing device matches the current image feature point with the reference feature point, and judges whether the similarity degree and the significance degree of the current image feature point and the reference feature point meet a first preset matching threshold, if the similarity degree and the significance degree meet the first preset matching threshold, the current image feature point and the reference feature point are possibly pixel points of the same part in the same object and are respectively present in different positions in the current frame image and the reference frame. The image processing apparatus thus determines the current image feature point and the reference feature point as a first matching point pair including at least one matching point pair of the current image feature point and the reference feature point.

S104, amplifying the first preset matching threshold to obtain a second preset matching threshold, and obtaining a constraint condition based on the first matching point pair.

In the embodiment of the application, the image processing device matches the current image feature point with the reference feature point based on a first preset matching threshold, after the first matching point pair is obtained, the image processing device amplifies the first preset matching threshold to obtain a second preset matching threshold, and the image processing device calculates a mathematical model satisfied by the first matching point pair according to the first matching point pair as a constraint condition.

In the embodiment of the application, the number of the first matching point pairs obtained according to the first preset matching threshold is generally small, and the wrong matching point pairs often exist, so that if the mathematical model is directly calculated according to the first matching point pairs to perform image processing, the accuracy of the calculation result is greatly influenced, and the accuracy of the image processing is influenced. Therefore, the image processing apparatus enlarges the first preset matching threshold to obtain the second preset matching threshold, and thus uses the broadened threshold condition to obtain more matching point pairs from the current image feature point and the reference feature point. Further, in order to ensure the accuracy of the matching point pairs, the image processing apparatus may calculate, according to the first matching point pair, a mathematical model that the first matching point pair satisfies, and use the mathematical model as a constraint condition to further screen the matching point pairs that are matched after the threshold condition is relaxed.

In this embodiment of the application, when the image processing function is a repositioning function, the constraint relationship may be algebraic representation of constraint relationships existing in image points of the same three-dimensional point in the three-dimensional scene at different viewing angles, for example, a basic matrix formula corresponding to the first matching point pair may be calculated by the first matching point pair to serve as a constraint condition; the constraint relationship may also be a constraint relationship between image points when points on one projection plane are mapped onto another projection plane, for example, a homography matrix formula corresponding to a first matching point pair may be calculated through the first matching point pair as a constraint condition.

In the embodiment of the present application, when the image processing function is an image matching function, an image recognition function, and the like, the constraint relationship may also be a geometric constraint or a mathematical model in other forms, which is specifically selected according to an actual situation, and the embodiment of the present application is not limited.

And S105, under the constraint condition, matching the current image characteristic points with the reference characteristic points based on a second preset matching threshold value to obtain second matching point pairs.

In the embodiment of the application, the image processing device amplifies a first preset matching threshold to obtain a second preset matching threshold, and after a constraint condition is obtained based on the first matching point pair, the image processing device performs matching on the current image feature point and the reference feature point again under the second preset threshold, and the second preset threshold is larger than the screening range of the first matching preset threshold, so that the image processing device obtains more matching point pairs than the first matching; then, the image processing apparatus verifies all the matching point pairs obtained by the secondary matching using the constraint condition, takes the matching point pairs satisfying the constraint condition as correct matching point pairs, takes the matching point pairs not satisfying the constraint condition as wrong matching point pairs, and finally takes all the correct matching point pairs as second matching point pairs.

In the embodiment of the present application, since the matching point pairs obtained by the second matching usually include the first matching point pair, the image processing apparatus may also verify whether only the newly added matching point pair in the second matching satisfies the constraint condition, and then use the newly added matching point pair and the first matching point pair that satisfy the constraint condition as the second matching point pair.

In the embodiment of the application, since the second preset matching threshold is greater than the first preset matching threshold, the image processing apparatus may match more matching point pairs using the second preset matching threshold, and since the image processing apparatus uses the constraint condition calculated according to the first matching point pair, the matching point pair obtained according to the second preset matching threshold is verified, thereby ensuring that the accurate matching rate of the secondary matching point pair satisfying the constraint condition is not lower than that of the first matching point pair, and therefore, the image processing apparatus may obtain more accurate second matching point pairs.

For example, after performing a violent matching once by the image processing apparatus to obtain a first matching point pair of N matching point pairs, the image processing apparatus may further perform a violent matching again on the current image feature point and the reference feature point based on a second preset matching threshold to obtain N + M matching point pairs, then the image processing apparatus may calculate only the M matching point pairs to determine M1 matching point pairs satisfying the constraint condition among the M matching point pairs, and finally, the image processing apparatus may use the N + M1 matching point pairs as the second matching point pair.

In the embodiment of the present application, in order to verify the accuracy of the matching result in the embodiment of the present application, the image processing apparatus may calculate the accuracy and the recall rate of the matching result corresponding to the brute force matching and the embodiment of the present application, respectively. Wherein, the accuracy is the percentage of the correct matching point pairs in all the matched point pairs. The recall is the percentage of matching point pairs that are correctly matched to all point pairs. The higher the accuracy, the more accurate the result. The higher the recall, the more complete the correct match found.

For example, the image processing apparatus selects the same 10 groups of pictures, and calculates the accuracy and recall ratio of the matching result of the violent matching and the matching result of the embodiment of the present application under the same parameters, and the results are shown in table 1:

TABLE 1

As can be seen from table 1, the accuracy and the recall ratio of the embodiment of the present application are higher than that of violent matching, so that the embodiment of the present application can solve the problems of too few matching point pairs and more wrong matching point pairs, thereby improving the number and the accuracy of the matching point pairs and finally improving the accuracy of image processing.

And S106, performing an image processing function based on the second matching point pair.

In this embodiment, after the image processing apparatus obtains the second matching point pair, the image processing apparatus may perform a corresponding image processing function based on the second matching point pair. When the image processing function is a repositioning function, the image processing apparatus may restore the relative pose between the current image frame and the reference frame using the second matching point pair, and finally the image processing apparatus may implement the repositioning function by the relative pose.

In this embodiment, when the image processing function is identification and matching of non-rigid deformation of an image, the image processing apparatus may calculate an image matching degree between the object with non-rigid deformation and the object in the original state using the second matching point pair to identify a correspondence between the object with non-rigid deformation and the object in the original state.

It can be understood that the image processing device can expand more matches by calculating a small number of reliable matches first and then estimating the mathematical model through the matches, and meanwhile, the accuracy of the matched point pairs is ensured, the problems that the number of the matched point pairs is too small and more wrong matched point pairs exist are solved, the number and the accuracy of the matched point pairs are improved, and finally, the accuracy of the image processing is improved.

In some embodiments of the present application, the present application further provides an image processing method, as shown in fig. 2, the method may include:

s201, the image processing device acquires a current frame image to be processed.

S202, the image processing device extracts the current image feature point of the current frame image from the current frame image and acquires the reference feature point corresponding to the reference frame based on the current image feature point.

In the embodiment of the application, after the image processing device acquires a current frame image to be processed, the image processing device extracts feature points from the current frame image to serve as current image feature points, then based on the current image feature points, finds a frame which is most matched with the current image feature points from a reference image frame to serve as a reference frame, and finally extracts the feature points from the reference frame to serve as the reference feature points.

In this embodiment of the application, the extracting, by the image processing apparatus, the current image feature point of the current frame image from the current frame image may specifically include: s2021 to S2024. The following were used:

s2021, extracting the first pixel points representing the corner boundaries with the preset number of corner points from the current frame image.

In the embodiment of the application, after the image processing device acquires the current frame image to be processed, the image processing device extracts an angular point from the current frame image as a first pixel point.

In the embodiment of the application, the image processing device extracts pixel points, namely corner points, of objects such as table corners and corners of the objects in the current frame image, and the pixel points at the corners, edges and the like in the image are high in identification degree and are more representative in image identification. In some embodiments of the present application, the image processing apparatus extracts 150 corner points.

S2022, the image processing device extracts features of second pixel points in the preset range of the first pixel points to obtain first image feature points.

In the embodiment of the application, after the image processing device extracts the first pixel points representing the angle boundaries in the preset angular point quantity from the current frame image, the image processing device extracts the features of the second pixel points within the preset range of the first pixel points to obtain the first image feature points.

In the embodiment of the application, the image processing device extracts features from the pixels within the preset range of the first pixel to serve as the first image feature point, and the first image feature point is used for describing the first pixel.

In this embodiment of the present application, the feature point may be a Binary robust feature (BRIEF) or other data format, and is specifically selected according to an actual situation, and this embodiment of the present application is not specifically limited.

Illustratively, the image processing apparatus extracts the binary robust feature BRIEF from the pixel points within 31 × 31 around the corner point as the first image feature point.

S2023, the image processing device extracts features of third pixel points outside the preset range of the first pixel points to obtain second image feature points.

In the embodiment of the application, after the image processing device extracts the features of the second pixel point within the preset range of the first pixel point to obtain the first image feature point, the image processing device extracts the features of the third pixel point outside the preset range of the first pixel point to obtain the second image feature point.

In the embodiment of the application, the image processing device can additionally extract more feature points in the current image frame so as to better represent the current image frame.

Illustratively, the image processing device extracts more feature points for scenes with complex textures, and extracts fewer feature points for general weak texture scenes, such as white walls, as the second image feature points. In some embodiments of the present application, the image processing apparatus may extract 500 to 1500 second image feature points.

S2024, the image processing apparatus regards the first image feature point and the second image feature point as current image feature points.

In the embodiment of the application, after the image processing device extracts the features of the third pixel point outside the preset range of the first pixel point to obtain the second image feature point, the image processing device takes the first image feature point and the second image feature point as the current image feature point.

In the embodiment of the present application, the current image feature points include a first feature point and a second feature point.

In this embodiment of the application, the obtaining, by the image processing apparatus, the reference feature point corresponding to the reference frame based on the current image feature point may specifically include: s2025 to S2026. The following were used:

and S2025, the image processing device matches the current image feature points with a preset map feature library, wherein the preset map feature library stores preset map feature points of at least one frame of preset map image frame.

In the embodiment of the application, after the image processing device takes the first image feature point and the second image feature point as the current image feature point, the image processing device searches for a best matching preset map image frame in a preset map feature library according to the current image feature point, wherein the preset map feature library stores at least one preset map feature point in one preset map image frame.

In this embodiment of the application, when the image processing apparatus matches the preset map feature library according to the feature points of the current image, the feature matching of the bag-of-words model of the current image may be adopted, or other algorithms may be adopted, for example, an accumulation algorithm vlad (vector of collected descriptors), a traversal method, or the like, which is specifically selected according to the actual situation, and this embodiment of the application is not limited.

In the embodiment of the application, when the image processing device matches the bag-of-word model of the current frame image with the preset map feature library, firstly, the image processing device calculates the bag-of-word model of the current frame image according to the first image feature point in the current image feature points. Wherein, the bag-of-words model feature is an expression model which uses a bag filled with specific words to represent the first image feature point of the current frame image; and then, the image processing device compares the word bag model of the current frame image with preset map feature points contained in each frame of preset map image frame in a preset map feature library.

In the embodiment of the application, when the image processing device matches the current image feature point with the preset map feature library, the current image feature point may be matched with the preset map image frame, or the current image feature point may be matched with a preset key frame in the preset map image frame, which is specifically selected according to an actual situation, which is not specifically limited in the embodiment of the application.

And S2026, taking a preset map image frame which is most matched with the current image feature point in the preset map feature library as a reference frame by the image processing device, and taking the preset map feature point contained in the reference frame as a reference feature point.

In the embodiment of the application, after the image processing device matches the current image feature point with the preset map feature library, the image processing device finds the preset map image frame which is most matched with the current image feature point and serves as a reference frame; after the image processing device finds the reference frame, the image processing device obtains preset map feature points contained in the reference frame as reference feature points.

In the embodiment of the present application, the preset map feature points are feature points extracted from at least one preset map image frame by the image processing device during map construction, and the principle of extracting the feature points from the preset map image frame by the image processing device is the same as that in steps S2021 to S2024.

S203, the image processing device matches the current image characteristic point with the reference characteristic point based on the first preset matching threshold value to obtain a first matching point pair.

In the embodiment of the application, after the image processing device extracts the current image feature point of the current frame image from the current frame image and acquires the reference feature point corresponding to the reference frame based on the current image feature point, the image processing device matches the current image feature point with the reference feature point based on the first preset matching threshold to obtain the first matching point pair.

In this embodiment of the application, the matching, by the image processing apparatus, the current image feature point with the reference feature point based on the first preset matching threshold, and obtaining the first matching point pair specifically may include: s2031 to S2035. The following were used:

s2031, the image processing device combines each current image sub-feature point in the current image feature points with the reference feature point one by one to obtain at least one sub-feature point pair.

In the embodiment of the application, after the image processing device uses a preset map image frame which is most matched with the current image feature points in a preset map feature library as a reference frame and uses the preset map feature points contained in the reference frame as reference feature points, the image processing device performs one-to-one combination by using one current image sub-feature point in the current image feature points to correspond to each reference sub-feature point one by one until all current image sub-feature points and all reference sub-feature points are combined completely, and the image processing device obtains at least one sub-feature point pair.

In the embodiment of the present application, each sub-feature point pair includes a current frame sub-image feature point and a reference sub-feature point.

Illustratively, the current image feature points include current image sub-feature points a1, a2, A3. The image processing apparatus combines the current image sub-feature point a1 with all 3 reference feature points B, C, D one by one, and can obtain three sub-feature point pairs (a1, B), (a1, C), and (a1, D). The image processing apparatus combines all the current image feature points with all the reference feature points, and may further obtain sub-feature point pairs (a2, B), (a2, C) and (a2, D), (A3, B), (A3, C) and (A3, D), the image processing apparatus regarding the 9 sub-feature point pairs as at least one sub-feature point pair.

In some embodiments of the present application, only 150 corner points of the current frame image feature points are used for combination with the reference feature points and for matching with the reference feature points. If 500 to 1500 additional feature points are also used for matching, additional computation is required, and the number of first matching point pairs and matching accuracy are not greatly improved.

S2032, the image processing device calculates the Hamming distance from each current image sub-feature point to the reference feature point aiming at least one sub-feature point pair to obtain at least one Hamming distance, and the at least one Hamming distance is in one-to-one correspondence with the at least one sub-feature point pair.

In the embodiment of the application, after the image processing device combines each current image sub-feature point in the current image feature points with the reference feature point one by one to obtain at least one sub-feature point pair, the image processing device calculates the hamming distance from one current image sub-feature point to one reference sub-feature point in each sub-feature point pair until the hamming distance of at least one sub-feature point pair is calculated, the image processing device obtains at least one hamming distance, and the at least one hamming distance is in one-to-one correspondence with the at least one sub-feature point pair.

In the embodiment of the present application, for each sub-feature point pair, the image processing apparatus calculates a hamming distance between the sub-feature point of the current image and the sub-reference feature point contained in the sub-feature point pair, and obtains a hamming distance corresponding to each sub-feature point pair.

In the embodiment of the present application, specifically, the hamming distance represents the direct different digits of the two binary codes. The image processing device calculates different direct bit numbers of binary codes between the second image characteristic point in the current image sub-characteristic points and the characteristic corresponding to the reference characteristic point, and obtains the Hamming distance of each sub-characteristic point pair.

In the embodiment of the present application, the image processing apparatus performs the same calculation for at least one sub-feature point pair, so as to obtain at least one hamming distance of the at least one sub-feature point pair.

S2033, the image processing apparatus determines a minimum hamming distance and a next minimum hamming distance from the at least one hamming distance.

In the embodiment of the application, the image processing device calculates the hamming distance from each current image sub-feature point to the reference feature point for at least one sub-feature point pair, and after obtaining at least one hamming distance, the image processing device finds the minimum hamming distance calculation value and the second minimum hamming distance calculation value from at least one hamming distance as the minimum hamming distance and the second minimum hamming distance.

S2034, when the minimum hamming distance is less than the preset significance threshold multiplied by the second minimum hamming distance, the image processing apparatus compares the minimum hamming distance with the preset similarity threshold.

In an embodiment of the present application, after the image processing apparatus determines the minimum hamming distance and the second minimum hamming distance from the at least one hamming distance, the image processing apparatus determines whether the minimum hamming distance is smaller than a preset significance threshold value multiplied by the second minimum hamming distance, and when the minimum hamming distance is smaller than the preset significance threshold value multiplied by the second minimum hamming distance, the image processing apparatus compares the minimum hamming distance with the preset similarity threshold value.

In this embodiment of the present application, the first preset matching threshold includes a preset similarity threshold and a preset significance threshold.

In this embodiment of the present application, the first preset matching threshold includes a preset similarity threshold and a preset significance threshold. When the Hamming distance is smaller than a preset similarity threshold, the current image sub-feature point corresponding to the Hamming distance is sufficiently similar to the reference sub-feature point; when the minimum Hamming distance is smaller than the preset significance threshold value multiplied by the second minimum Hamming distance, the current image sub-feature point corresponding to the Hamming distance is obviously matched with the reference sub-feature point, and other similar matches are not available.

For example, the preset significance threshold may be 0.9, the preset similarity threshold may be 50, or other thresholds may be selected according to actual conditions, which is not limited in the embodiment of the present application.

In the embodiment of the application, the image processing device calculates the preset significance threshold value multiplied by the second-time small hamming distance, then compares the calculation result with the minimum hamming distance, and if the minimum hamming distance is still smaller than the calculation result, it indicates that the matching degree between the current image sub-feature point corresponding to the minimum hamming distance and the reference sub-feature point is significant, and there is no other similar matching.

Illustratively, the minimum hamming distance is 40, the second smallest hamming distance is 50, the image processing apparatus calculates a preset significance threshold value multiplied by the second smallest hamming distance, that is, 50 × 0.9, to obtain a calculation result 45, the image processing apparatus compares the calculation result 45 with the minimum hamming distance 40, the minimum hamming distance 40 is still smaller than the calculation result 45, and it indicates that the significance of the matching between the current image sub-feature point and the reference sub-feature point corresponding to the minimum hamming distance 40 is high.

In the embodiment of the present application, after verifying the significance of the hamming distance, the image processing apparatus needs to verify the approximation of the hamming distance. Therefore, when the minimum hamming distance is less than the preset significance threshold multiplied by the next smallest hamming distance, the image processing apparatus needs to further verify whether the minimum hamming distance is less than or equal to the preset similarity threshold.

S2035, when the minimum Hamming distance is less than or equal to the preset similarity threshold, the image processing device determines a sub-feature point pair corresponding to each current image sub-feature point corresponding to the minimum Hamming distance from at least one sub-feature point pair, so that the image processing device obtains a first matching point pair corresponding to the current image feature point.

In this embodiment of the application, when the minimum hamming distance is smaller than the preset significance threshold multiplied by the second minimum hamming distance, after the image processing device compares the minimum hamming distance with the preset similarity threshold, the image processing device further determines whether the minimum hamming distance is smaller than or equal to the preset similarity threshold, and when the minimum hamming distance is smaller than or equal to the preset similarity threshold, the image processing device determines a sub-feature point pair corresponding to the minimum hamming distance from at least one sub-feature point pair, so that the image processing device obtains a first matching point pair corresponding to the current image feature point.

In this embodiment of the present application, if the minimum hamming distance is smaller than the preset significance threshold multiplied by the next minimum hamming distance, and the minimum hamming distance is smaller than or equal to the preset similarity threshold, it indicates that the sub-feature point pair corresponding to the minimum hamming distance simultaneously satisfies the preset similarity threshold and the preset significance threshold, and therefore the sub-feature point pair corresponding to the minimum hamming distance is a matched sub-feature point pair. The image processing apparatus sequentially determines each sub-feature point pair, so that in at least one sub-feature point pair, the image processing apparatus can determine all matched sub-feature point pairs corresponding to at least one sub-feature point pair as a first matched point pair.

Illustratively, the preset similarity threshold is 50 and the preset significance threshold is 0.9. The image processing apparatus calculates the hamming distances between the current image sub-feature point a to the 3 reference feature points B, C, D, resulting in a hamming distance from a to B40, a hamming distance from a to C50, and a hamming distance from a to D60, respectively. The image processing apparatus determines the minimum hamming distance 40 and the next smallest hamming distance 50 from which the hamming distance significance of a to B is significantly higher than the hamming distance significance of a to C or a to D, since 50 is still greater than the minimum hamming distance 40 after being multiplied by the preset significance threshold of 0.9. The image processing apparatus compares the minimum hamming distance 40 with a preset similarity threshold value smaller than 50, so that the minimum hamming distance 40 is still smaller than the preset similarity threshold value, and the image processing apparatus can determine that the sub-feature point pairs (a, B) are matched. The image processing apparatus may continue to determine, as the first matching point pair, a sub-feature point pair matching among all the sub-feature point pairs in at least one of the sub-feature point pairs.

S204, the image processing device amplifies the first preset matching threshold value to obtain a second preset matching threshold value.

In the embodiment of the application, the image processing device matches the current image feature point with the reference feature point based on the first preset matching threshold to obtain the first matching point pair, and then amplifies the first preset matching threshold to obtain the second preset matching threshold.

In this embodiment of the application, when the minimum hamming distance is less than or equal to the preset similarity threshold, the image processing device determines a sub-feature point pair corresponding to each current image sub-feature point corresponding to the minimum hamming distance from at least one sub-feature point pair, so as to obtain a first matching point pair corresponding to the current image feature point, and then the image processing device amplifies the preset similarity threshold and the preset significance threshold in the first preset matching threshold to obtain a second preset matching threshold.

Illustratively, the image processing apparatus enlarges a preset similarity threshold value of the first preset matching threshold value to 100, enlarges a preset significance threshold value of the first preset matching threshold value to 0.95, and the image processing apparatus takes the preset similarity threshold value 100 and the preset significance threshold value 0.95 as the second preset matching threshold value.

S205, the image processing device acquires at least one preset initial constraint condition.

In this embodiment, after the image processing apparatus amplifies the first preset matching threshold to obtain the second preset matching threshold, the image processing apparatus calculates the constraint condition according to at least one preset initial constraint condition, and the image processing apparatus may obtain the at least one preset initial condition according to the difference of the image processing functions.

In this embodiment of the application, the image processing apparatus amplifies the first preset matching threshold to obtain the second preset matching threshold, so as to obtain more matching point pairs, and in order to further ensure the correctness of the obtained matching point pairs, the image processing apparatus further needs to calculate constraint conditions that are satisfied between the first matching point pairs, to verify whether the matching point pairs obtained by secondary matching satisfy the same constraint conditions, and the calculation constraint conditions first need to obtain preset initial constraint conditions.

In the embodiment of the present application, according to different image processing functions, the image processing apparatus needs to obtain at least one preset initial constraint condition.

For example, when the image processing function is the relocation, the image processing apparatus may use a matrix equation of the basis matrix and the homography matrix as the preset initial constraint condition.

S206, the image processing device calculates at least one preset initial constraint condition by using the first matching point pair to obtain at least one sub-constraint condition.

In the embodiment of the application, after the image processing device obtains at least one preset initial constraint condition, the image processing device calculates the at least one preset initial constraint condition by using the first matching point pair to obtain at least one sub-constraint condition.

In this embodiment of the application, the step of calculating, by the image processing apparatus, at least one preset initial constraint condition by using the first matching point pair to obtain at least one sub-constraint condition may specifically include: s2061 to S2063. The following were used:

s2061, the image processing apparatus selects a preset number of first matching point pairs from the first matching point pairs for combination according to each preset initial constraint condition, and repeats the preset times to obtain first matching point pair combinations equal to the preset times, where each of the first matching point pair combinations includes a preset number of first matching point pairs.

In this embodiment of the application, after the image processing apparatus obtains at least one preset initial constraint condition, in order to perform calculation according to the preset initial constraint condition, the image processing apparatus may select a preset number of first matching point pairs from the first matching point pairs to perform combination, and repeat the preset times to obtain the first matching point pair combinations equal to the preset times in number.

Illustratively, when the initial constraint condition is preset as a basic matrix calculation formula, at least 8 pairs of matching point pairs are required as calculation parameters, so that the image processing device randomly selects 8 pairs of first matching point pairs in a first matching point pair for combination to calculate the basic matrix; in order to improve the calculation accuracy and reduce the error, the image processing apparatus may further continue to randomly select 8 pairs of matching point pairs from the first matching point pairs, and repeat the selection 100 times, and the image processing apparatus may obtain 100 sets of randomly selected first matching point pair combinations, each of which contains 8 pairs of matching point pairs.

Exemplarily, when the initial constraint condition is preset as a basic matrix calculation formula, at least 4 pairs of matching point pairs are required to be used as calculation parameters, so that the image processing apparatus randomly selects 4 pairs of first matching point pairs in a first matching point pair to combine the first matching point pairs for calculating the homography matrix; in order to improve the calculation accuracy and reduce the error, the image processing apparatus may further continue to randomly select 4 pairs of matching point pairs from the first matching point pairs, and repeat the selection 100 times, and the image processing apparatus may obtain 100 sets of randomly selected first matching point pair combinations, each of the combinations including 4 pairs of matching point pairs.

S2062, the image processing apparatus calculates each preset initial constraint condition using each of the first matching point pair combinations, to obtain a sub-constraint condition of a preset number of times corresponding to each preset initial constraint condition, as a final sub-constraint condition of each preset initial constraint condition.

In this embodiment of the application, for each preset initial constraint condition, the image processing apparatus selects a preset number of first matching point pairs from the first matching point pairs to combine, repeats the preset times, and after obtaining the first matching point pair combinations equal to the preset times, the image processing apparatus calculates each preset initial constraint condition using each combination of the first matching point pair combinations, where one first matching point pair combination is to be calculated for each different preset initial condition once, so that the image processing apparatus can obtain, according to the preset number of first matching point pair combinations, a preset number of sub-constraint conditions corresponding to each preset initial constraint condition as a final sub-constraint condition of each preset initial constraint condition.

In this embodiment of the present application, when the preset initial condition is a basis matrix calculation formula, the image processing apparatus uses the first matching point pair combination, and the method for calculating the basis matrix includes:

illustratively, the fundamental matrix calculation formula is:

F＝K^-TEK^-1 (1)

in (1), E is an essential matrix which can be obtained by the combined calculation of the first matching point pairs, K is an internal reference matrix of the camera and is marked_TRepresenting transposing the matrix, K^-TRepresenting transposing and inverting the intra-reference matrix, K^-1Representing the inversion of the intra-parameter matrix. Wherein K may be represented as:

wherein f is_xAnd f_yFocal lengths of the camera in the x and y directions, c, respectively_xAnd c_yThe x, y coordinates, respectively, of the center of the image to the origin of the image coordinates, in pixels.

In the embodiment of the present application, as shown in (2), the camera internal reference matrix may be directly determined according to the camera parameters, that is, in formula (1), K is a known parameter, the image processing apparatus needs to calculate the essential matrix E by combining the first matching point pairs, and then substitute E and K of the known parameter into formula (1) calculated according to the basic matrix, and use formula (1) calculated according to the basic matrix with known E and K as the sub-constraint condition.

In the embodiment of the present application, the method for the image processing apparatus to calculate the essential matrix E by using the coordinates of the first matching point pair combination midpoint includes:

according to the epipolar constraint formula:

wherein x is₁And x₂For normalized three-dimensional coordinates x of a pair of matching point pairs in a first matching point pair combination₁＝[u₁,v₁,1]^T，x₂＝[u₂,v₂,1]^TThe image processing apparatus converts x₁And x₂Substituting equation (3) results in:

wherein e is₁To e₉Represents the value of each matrix element when the essential matrix E is written as a3 x 3 matrix. The image processing device is used for solving e in (4)₁To e₉Can be transformed to E by expanding the matrix E to solve the column equation to obtain:

e＝[e₁ e₂ e₃ e₄ e₅ e₆ e₇ e₈ e₉] (5)

the image processing apparatus substitutes equation (5) into equation (4), and multiplies the coordinate points in equation (4), the image processing apparatus can obtain an equation listed according to a pair of matching point pairs, as shown in (6):

[u₁u₂ u₁v₂ u₁ v₁u₂ v₁v₂ v₁ u₂ v₂ 1]·e＝0 (6)

by analogy, 8 pairs of matching point pairs in a first matching point pair combination can be obtained by the image processing apparatus as shown in (7):

since both sides of the equation are multiplied by any real number in the polar constraint formula (3), the equation is satisfied, which indicates that the essential matrix E has a scale equivalence. That is to say that the essential matrix E lacks scale information. That is, the 9 unknowns contained in the essential matrix E can be derived from 8 equations. Therefore, the image processing apparatus can solve an intrinsic matrix E by 8 for the first pair of matching points, i.e. the image processing apparatus can solve an intrinsic matrix E by a set of matching points.

In this embodiment, when the preset initial constraint condition is a basic matrix formula, the image processing apparatus calculates the preset times of the basic matrix by using the preset times of the first matching point pair combination to obtain a preset times intrinsic matrix E, and then substitutes the calculated intrinsic matrix E into the preset initial condition, that is, the formula (1), to obtain a preset times of basic matrix calculation formula F ═ K of different parameters E^-TEK^-1And the image processing device takes the basic matrix calculation formula with different parameters E of preset times as a final sub-constraint condition corresponding to the basic matrix formula with the preset initial condition.

In this embodiment of the present application, when the preset initial condition is a homography matrix calculation formula, the image processing apparatus uses the first matching point pair combination, and the method for calculating the homography matrix includes:

the homography matrix calculation formula is as follows:

p₂＝Hp₁ (8)

wherein H represents a homography matrix, p₁And p₂For a pair of matching point pairs, p, contained in the first combination of matching point pairs₁Representing the coordinates p in the pixel coordinate system of the current frame image₁＝(u₁,v₁，1)，p₂Coordinates p in the pixel coordinate system representing the reference frame₂＝(u₂,v₂，1)。

The image processing apparatus converts p₁And p₂The pixel coordinates of (a) are substituted into the calculation formula (8) of the homography matrix, and the calculation formula is developed to obtain:

in (9), h₁To h₉Represents the value of each matrix element when the homography matrix is written as a3 x 3 matrix, and s represents a non-zero factor. Since the third row of the matrix is multiplied by p₁Is 1, i.e. h₇u₁+h₈v₁+h₉1, so the equation needs to be satisfied by a non-zero factor s. In some embodiments of the present application, the image processing apparatus makes h by s₉By again removing this non-zero factor by (9), 1, we can get:

finishing to obtain:

h₁u₁+h₂v₁+h₃-h₇u₁u₂-h₈v₁u₂＝u₂ (12)

h₄u₁+h₅v₁+h₆-h₇u₁v₂-h₈v₁v₂＝v₂ (13)

as can be seen from (12) and (13), the image processing apparatus can obtain 2 constraint equations for calculating the homography matrix H from 1 pair of matching point pairs. The image processing apparatus expands the homography matrix H as a vector:

h＝[h₁ h₂ h₃ h₄ h₅ h₆ h₇ h₈ h₉] (14)

h₉＝1(15)

by analogy, 4 pairs of matching point pairs exist in a first matching point pair combination, and the image processing device can obtain 8 equations about the homography matrix through the 4 pairs of first matching point pairs, wherein h is₉Is known to be equal to 1. Thus, the image processing apparatus can obtain the homography matrix by solving the system of linear equations, and then the image processing apparatus calculates p of the known parameter H₂＝Hp₁The formula is used as a sub-constraint condition, that is, the image processing device can obtain p of a known parameter H through a group of first matching point pairs₂＝Hp₁The formula serves as a sub-constraint.

In the embodiment of the present application, a Direct Linear Transform (DLT) is usually used as a calculation method for solving the Linear equation set, and other methods may also be used according to actual situations, which is not specifically limited in the embodiment of the present application.

In this embodiment, when the initial constraint condition is a homography matrix formula, the image processing apparatus calculates a predetermined number of times for the homography matrix using a predetermined number of first matching point pair combinations to obtain a predetermined number of homography matrices H, and substitutes the calculated predetermined number of times for the homography matrix calculation formula p₂＝Hp₁The homography matrix calculation formulas with different parameters H of the preset times can be obtained, and the image processing device takes the homography matrix calculation formulas with different parameters H of the preset times as the final sub-constraint conditions corresponding to the homography matrix formulas of the preset initial conditions.

S2063, the image processing apparatus determines a final sub-constraint of each of the at least one preset initial constraint as at least one sub-constraint.

In this embodiment of the application, the image processing apparatus calculates each preset initial constraint condition by using the first matching point for each combination in the combinations to obtain sub-constraint conditions corresponding to each preset initial constraint condition for a preset number of times, and after the sub-constraint conditions are used as final sub-constraint conditions of each preset initial constraint condition, the image processing apparatus uses the final sub-constraint conditions corresponding to each preset initial constraint condition in at least one preset initial constraint condition as at least one sub-constraint condition.

Illustratively, the image processing apparatus calculates the combination according to 100 groups of first matching points by using a basic matrix calculation formula as a preset initial condition, to obtain 100 basic matrix formulas with different known parameters as 100 sub-constraint conditions corresponding to the preset initial condition as a basic matrix; the image processing device calculates the combination according to 100 groups of first matching points by taking a homography matrix calculation formula as a preset initial condition to obtain 100 homography matrix formulas with different known parameters as 100 sub-constraint conditions of which the preset initial condition is corresponding to the homography matrix; the image processing device uses the 200 sub-constraints together as at least one sub-constraint corresponding to at least one preset initial condition.

S207, the image processing device determines at least one third mapping value based on the three-dimensional coordinates of each third feature point in the first matching point pair and at least one sub-constraint condition.

In this embodiment of the application, after the image processing apparatus calculates at least one preset initial constraint condition by using the first matching point pair to obtain at least one sub-constraint condition, the image processing apparatus determines at least one third mapping value according to the three-dimensional coordinates of each third feature point in the first matching point pair and the sub-constraint condition of each different known parameter.

In this embodiment, the image processing apparatus may use the three-dimensional coordinates of each current image feature point in the first matching point pair and the corresponding reference feature point to obtain a value of a corresponding basis matrix or homography matrix according to a basis matrix formula of a known parameter or a homography matrix formula of a known parameter in the at least one sub-constraint condition, and use the value as the third mapping value.

In this embodiment, the image processing apparatus may calculate all of the at least one sub-constraint condition until the processing is completed, and the image processing apparatus may determine values of the at least one basis matrix and the homography matrix as the at least one third mapping value.

S208, the image processing device determines at least one third matching error based on the at least one third mapping value.

In this embodiment of the application, after the image processing apparatus determines at least one third mapping value based on the three-dimensional coordinates of each third feature point in the first matching point pair and at least one sub-constraint condition, the image processing apparatus calculates an error of the first matching point pair under the action of the third mapping value as at least one third matching error based on the at least one third mapping value according to an error calculation formula.

In this embodiment, when the third mapping value is the third mapping value corresponding to the basic matrix, the image processing apparatus calculates an error of the first matching point pair combination under the action of the basic matrix, which may be according to formula (16):

in the step (16), the step (c),

representative pair coordinate p₁＝(u₁,v₁1) transposing, F representing the base matrix, F being here a known third mapping value, the image processing apparatus calculates the relative error according to equation (16), which can be obtained:

in (17), a₁，b₁And c₁For unknown intermediate parameters, the image processing device determines a according to the formula (17)₁，b₁And c₁Then, the image processing apparatus calculates the value of error according to the error calculation formula (18):

wherein the error of the matching point pair under the action of the basis matrix is represented by the value of error calculated according to the formula (18), and the image processing apparatus regards the error of a group of matching point pairs under the action of a third mapping value corresponding to a basis matrix as a third matching error.

In this embodiment, when the third mapping value is the third mapping value corresponding to the homography matrix, the image processing apparatus calculates an error of the first matching point pair combination under the action of the homography matrix, which may be according to the definition:

p₂＝Hp₁ (19)

the image processing apparatus calculates the relative error according to equation (19) to obtain:

in (20) and (21), u'₂And v'₂The image processing device obtains u 'for unknown intermediate parameters'₂And v'₂Then, the image processing apparatus calculates the value of error using a homography matrix error calculation formula (22):

error＝(u₂-u'₂)²+(v₂-v'₂)² (22)

the error of the matching point pair under the action of the homography matrix is represented by the value of error calculated according to the formula (22), and the error of a group of matching point pairs under the action of a third mapping value corresponding to the homography matrix is also taken as a third matching error by the image processing device.

In this embodiment, the image processing apparatus may determine, as the at least one third matching error, an error of each of the at least one first matching point pair combination under the action of the at least one third mapping value, based on the at least one third mapping value in the same manner.

S209, the image processing device determines a first matching point pair which is corresponding to at least one sub-constraint condition and is less than or equal to a preset error threshold value as a correct matching point pair based on at least one third matching error.

In this embodiment of the application, after the image processing apparatus determines at least one third matching error based on at least one third mapping value, the image processing apparatus performs screening according to a preset error threshold based on the at least one third matching error, and determines a first matching point pair corresponding to at least one sub-constraint condition and smaller than or equal to the preset error threshold as a correct matching point pair.

In this embodiment of the application, when error is less than or equal to a preset error threshold, the image processing apparatus determines that the corresponding first matching point pair is a correct matching point pair, and when error is greater than the preset error threshold, the image processing apparatus determines that the corresponding first matching point pair is an incorrect matching point pair.

In this embodiment of the application, the image processing apparatus performs verification of the third matching error for each of the at least one sub-constraint condition, to obtain a correct matching point pair corresponding to each sub-constraint condition.

S210, the image processing device selects one sub-constraint condition containing the most correct matching point pairs from at least one preset initial constraint condition respectively to serve as at least one middle sub-constraint condition corresponding to each at least one preset initial constraint condition respectively.

In this embodiment, after the image processing apparatus determines, based on the at least one third matching error, a first matching point pair corresponding to the at least one sub-constraint condition and smaller than or equal to a preset error threshold, as a correct matching point pair, the image processing apparatus selects, from the at least one preset initial constraint condition, one sub-constraint condition that each preset initial constraint condition contains the most correct matching point pairs, as at least one intermediate sub-constraint condition corresponding to each at least one preset initial constraint condition.

In this embodiment of the application, after the image processing apparatus obtains the correct matching point pair corresponding to each sub-constraint condition, the image processing apparatus counts, under each preset initial constraint condition, at least one sub-constraint condition corresponding to the preset initial condition, where the at least one sub-constraint condition includes one sub-constraint condition with the largest number of correct matching point pairs, and the at least one sub-constraint condition is used as an intermediate sub-constraint condition. The image processing device carries out statistics on at least one preset initial constraint condition to obtain at least one intermediate sub-constraint condition.

Illustratively, the image processing apparatus counts 100 basic matrix sub-constraint conditions under the basic matrix formula to obtain one basic matrix sub-constraint condition corresponding to the most correct matching point pairs as a middle sub-constraint condition corresponding to the basic matrix formula, and the image processing apparatus counts 100 homography matrix sub-constraint conditions under the homography matrix formula to obtain one homography matrix sub-constraint condition corresponding to the most correct matching point pairs as a middle sub-constraint condition corresponding to the homography matrix formula. The image processing device takes the intermediate sub-constraint condition corresponding to the basic matrix formula and the intermediate sub-constraint condition corresponding to the homography matrix formula as at least one intermediate sub-constraint condition.

S211, the image processing apparatus selects, as a constraint, an intermediate sub-constraint that includes the most correct matching point pairs from among the at least one intermediate sub-constraint.

In this embodiment of the application, after the image processing apparatus selects, from at least one preset initial constraint condition, one sub-constraint condition that contains the most correct matching point pairs, as at least one intermediate sub-constraint condition corresponding to each of the at least one preset initial constraint condition, the image processing apparatus selects, from the at least one intermediate sub-constraint condition, an intermediate sub-constraint condition that contains the most correct matching point pairs, as a constraint condition.

Illustratively, the image processing apparatus finds one basic matrix sub-constraint F containing the most correct matching point pairs among the 100 basic matrix sub-constraints₁＝K^-TE₁K^-1The intermediate sub-constraint condition corresponding to the basic matrix formula is used; the image processing device finds one containing the most correct matching point pairs in the 100 homography sub-constraintsSingle homography matrix sub-constraint p₂＝H₁p₁As the intermediate sub-constraint condition corresponding to the homography matrix formula; the image processing apparatus further converts F₁＝K^-TE₁K^-1And p₂＝H₁p₁Making a comparison, if F₁＝K^-TE₁K^-1More than p correctly matched point pairs in₂＝H₁p₁In (3), the image processing apparatus constrains the base matrix to a condition F₁＝K^-TE₁K^-1As a constraint.

S212, the image processing device matches the image characteristic points with the reference characteristic points based on a second preset matching threshold value to obtain intermediate matching point pairs.

In this embodiment of the present application, the image processing apparatus selects an intermediate sub-constraint condition that includes the most correct matching point pairs from at least one intermediate sub-constraint condition, and after the intermediate sub-constraint condition is used as the constraint condition, the image processing apparatus performs matching on the image feature points and the reference feature points by using the second preset matching threshold after being amplified, and since the threshold is amplified, the image processing apparatus obtains more matching point pairs, and the image processing apparatus uses the obtained matching point pairs obtained by secondary matching as intermediate matching point pairs.

And S213, screening out a second matching point pair meeting the constraint condition from the intermediate matching point pair by the image processing device.

In the embodiment of the application, the image processing device matches the image feature points with the reference feature points based on the second preset matching threshold, and after the intermediate matching point pairs are obtained, the image processing device screens out the second matching point pairs meeting the constraint conditions from the intermediate matching point pairs according to the constraint conditions.

In this embodiment of the application, the screening, by the image processing apparatus, the second matching point pair that satisfies the constraint condition from the intermediate matching point pair specifically includes:

the image processing device determines a first mapping value based on the three-dimensional coordinates and the constraint conditions of each first feature point in the intermediate matching point pair; the image processing device determines a first matching error based on the first mapping value; the image processing device determines a second matching point pair smaller than or equal to a preset error threshold value based on the first matching error.

In the embodiment of the application, after the image processing device matches the image feature points with the reference feature points based on the second preset matching threshold to obtain the intermediate matching point pairs, the image processing device determines the first mapping value based on the three-dimensional coordinates and the constraint conditions of each first feature point in the intermediate matching point pairs.

In the embodiment of the present application, the image processing apparatus uses the image feature point and the reference feature point in the intermediate matching point pair as the first feature point, and calculates the calculation result of the constraint condition as the first mapping value by using the three-dimensional coordinates of the first feature point and the constraint condition finally obtained in step S211.

Illustratively, the image processing apparatus obtains a basis matrix calculation formula as the constraint condition, and when the image processing apparatus obtains the intermediate matching point pair, the image processing apparatus calculates a basis matrix calculation result corresponding to the basis matrix calculation formula according to the method in step S2062 using the three-dimensional coordinates of each first feature point in the intermediate matching point pair, and the image processing apparatus uses the basis matrix calculation result calculated according to the second matching point pair as the first mapping value.

In the embodiment of the present application, after the image processing apparatus determines the first mapping value based on the three-dimensional coordinates and the constraint condition of each first feature point in the intermediate matching point pair, the image processing apparatus determines the first matching error based on the first mapping value.

In the embodiment of the present application, the principle of determining the first matching error based on the first mapping value is the same as S208.

In the embodiment of the application, after the image processing apparatus determines the first matching error based on the first mapping value, the image processing apparatus determines the second matching point pair which is less than or equal to the preset error threshold based on the first matching error.

The principle of determining the second matching point pair smaller than or equal to the preset error threshold based on the first matching error in the embodiment of the present application is the same as that in S209.

In practical applications, since the intermediate matching point pairs obtained by the secondary matching usually include the first matching point pair, the image processing apparatus may further perform a process of screening out the second matching point pair satisfying the constraint condition from the intermediate matching point pairs:

the image processing device screens out the sub-matching point pairs except the first matching point pair from the intermediate matching point pair; the image processing device determines a second mapping value based on the three-dimensional coordinates and the constraint conditions of each second feature point in the sub-matching point pair; the image processing device determines a second matching error based on the second mapping value; the image processing device determines a final sub-matching point pair smaller than or equal to a preset error threshold value based on the second matching error; the image processing apparatus takes the first matching point pair and the final sub-matching point pair as a second matching point pair.

In the embodiment of the application, the image processing device matches the image feature points with the reference feature points based on the second preset matching threshold, and after obtaining the intermediate matching point pair, the image processing device screens out the sub-matching point pairs except the first matching point pair from the intermediate matching point pair.

In the embodiment of the present application, the image processing apparatus screens out the sub-matching point pairs other than the first matching point pair to obtain matching point pairs newly added in the secondary matching, and the matching point pairs are used as the sub-matching point pairs. The image processing apparatus processes only the sub-matching point pairs, and can reduce the amount of calculation.

In the embodiment of the present application, the principle of determining the second mapping value by the image processing apparatus based on the three-dimensional coordinates and the constraint condition of each second feature point in the sub-matching point pair is the same as that in step S2062.

In the embodiment of the present application, the principle of determining the second matching error based on the second mapping value by the image processing apparatus is the same as that in step S208.

In this embodiment of the application, after the image processing apparatus determines the second matching error based on the second mapping value, the image processing apparatus determines the final sub-matching point pair that is less than or equal to the preset error threshold based on the second matching error.

In the embodiment of the present application, the principle of determining the final sub-matching point pair equal to or smaller than the preset error threshold by the image processing apparatus based on the second matching error is the same as that in step S209.

In this embodiment, after the image processing apparatus determines, based on the second matching error, a final sub-matching point pair that is less than or equal to a preset error threshold, the image processing apparatus uses the first matching point pair and the final sub-matching point pair as the second matching point pair.

It can be understood that, in the embodiment of the present application, after the hamming distance threshold and the significance threshold are amplified, the feature points of the current frame image and the feature points of the map reference frame are matched for the first time, so that the image processing apparatus may obtain more matching point pairs, the image processing apparatus uses the constraint condition calculated by the first matching, and then verifies the matching point pairs obtained by the second matching of the amplified threshold, thereby ensuring the accuracy of the matching point pairs, solving the problems of too few matching point pairs and more wrong matching point pairs, and obtaining more and more accurate matching point pairs.

And S214, when the image processing function is a repositioning function, restoring at least one piece of relative pose information according to the second matching point pair.

In the embodiment of the application, after the image processing device screens out the second matching point pair meeting the constraint condition from the intermediate matching point pair, when the image processing function is the repositioning function, the image processing device restores at least one piece of relative pose information according to the second matching point pair and the preset corresponding relation between the constraint condition and the pose of the camera.

In this embodiment of the application, the restoring, by the image processing apparatus, at least one piece of relative pose information according to the second matching point pair specifically includes: s2141 to S2142. The following were used:

s2141, the image processing device calculates at least one translation amount and at least one rotation amount in the first shooting coordinate system and the second shooting coordinate system according to the corresponding relation between the preset constraint condition and the camera pose and the second matching point pair.

In the embodiment of the present application, when the constraint condition is a fundamental matrix calculation formula, the correspondence between the preset constraint condition and the camera pose is the correspondence between the essential matrix E corresponding to the fundamental matrix and the camera relative pose:

E＝t^R (23)

in (23), E is an essential matrix, t ^ represents an antisymmetric matrix of the amount of translation between the first photographing coordinate system and the second photographing coordinate system, and R represents the amount of rotation between the first photographing coordinate system and the second photographing coordinate system. The image processing apparatus may calculate a Value of the intrinsic matrix E through the second matching point pair, and obtain values of t and R by using a Singular Value Decomposition (SVD) method for the Value of the intrinsic matrix E, and further obtain a relative pose of the camera, where the SVD method is as shown in formula (24):

E＝U∑V^T (24)

in (24), U is an orthogonal matrix, V^TAnd sigma is a singular value matrix.

The singular value form of E is [ σ σ σ 0 ] according to the intrinsic properties of the intrinsic matrix E]^TWhere σ represents an arbitrary number. Then in the singular value decomposition, for any one E, there are two possible t, R corresponding to it, as in equations (24-1) to (24-4):

in (24-1) to (24-4),

representing a rotation matrix rotated 90 deg. about the Z axis.

In the embodiment of the present application, when the constraint condition is the basis matrix, the image processing apparatus compares t₁A and t₂R as at least one translation quantity₁And R₂As the at least one rotation amount.

In the embodiment of the present application, when the constraint condition is a homography matrix calculation formula, the correspondence between the preset constraint condition and the camera pose is:

A＝dR+tn^T＝dK^-1HK (25)

wherein, H is a homography matrix, the image processing device can calculate the value of the homography matrix H through the second matching point pair, d represents the up-down translation value of a plane and a zero point plane in the space, and A is an intermediate value matrix calculated according to the value H of the homography matrix and the camera internal reference matrixes K and d.

In space, since at least three points can form a plane, a point on one plane satisfies formula (26):

ax+by+cz＝d (26)

in (26), a, b and c are known plane normal vectors, a group of parallel plane families can be determined according to a, b and c, x, y and z are three-dimensional coordinates of points, and a unique plane in the group of parallel planes can be located through d.

The image processing apparatus obtains a unit normal vector of a plane:

n＝[a b c]^T (27)

then a three-dimensional coordinate point X on the plane satisfies equation (28), which may be the three-dimensional coordinates of the current image feature point or the reference feature point:

the image processing apparatus multiplies equation (28) into t of equation (29):

X₂＝RX₁+t (29)

the image processing apparatus may obtain equation (30):

wherein, X₁And X₂Three-dimensional coordinates of the same point are respectively in a first shooting coordinate system and a second shooting coordinate systemAnd (4) coordinates.

Wherein, the three-dimensional coordinates of the same point are the coordinates x under the current frame image coordinate system and the reference frame image coordinate system₁，x₂Satisfies the formula:

αx₂＝Hx₁ (31)

in the step (31), the step (c),_αthe ratio of the depths of the three-dimensional coordinate point X in the first shooting coordinate system and the second shooting coordinate system is equivalent to the non-zero factor S in step S2062.

For example, the coordinates of one three-dimensional coordinate point in the first photographing coordinate system may be represented as (32), and the coordinates of the three-dimensional coordinate point in the second photographing coordinate system may be represented as (33):

X₁＝[x₍₁₎ y₍₁₎ z₍₁₎] (32)

X₂＝[x₍₂₎ y₍₂₎ z₍₂₎] (33)

wherein, X₁And X₂The coordinate of the same three-dimensional coordinate point in the two different coordinate systems is obtained.

The image processing apparatus may obtain a value of α as:

the image processing apparatus substitutes the value of α in (34) into equation (31), and multiplies the camera internal reference matrix K on both sides of the equation at the same time, and the equation is still true, the image processing apparatus can obtain:

finishing to obtain:

X₂＝K^-1HKX₁ (36)

the image processing apparatus multiplies α to equation (30) to obtain:

the image processing apparatus performs singular value decomposition on a according to the following formula:

A＝UΛV^T (38)

Λ＝diag(d₁,d₂,d₃),d₁≥d₂≥d₃ (39)

where U is an orthogonal matrix, V^TAnd Λ is a singular value matrix, and is the transpose of an orthogonal matrix. According to the definition of the orthogonal matrix, the image processing device can obtain:

U^TU＝V^TV＝I (40)

wherein I is an identity matrix.

The image processing apparatus carries out a term shift by substituting U and V into (38) of the singular value decomposition, and can obtain:

Λ＝U^TAV＝dU^TRV+(U^Tt)(V^Tn)^T (41)

the image processing apparatus sets s to detUdetV to ± 1. Where det represents determinant. Since U, V are unit orthogonal matrices, the determinant is 1. S here is different from the non-zero factor S in step S2062.

Order:

the image processing apparatus can solve the values of R 'and t'.

Substituting the values of R 'and t' into Λ yields:

the image processing apparatus takes a set of orthogonal bases: e.g. of the type₁＝(1 0 0)^T,e₂＝(0 1 0)^T,e₃＝(0 0 1)^T。

n'＝(x₁ x₂ x₃)^T＝x₁e₁+x₂e₂+x₃e₃ (44)

The image processing device substitutes (44) into Λ to obtain:

since n is a unit normal vector and V is a unit orthogonal matrix, it can also be regarded as a rotation matrix. Vn can be viewed as turning each component in n to three unit orthogonal bases in V. So n' is also a unit normal vector. The image processing apparatus can thus obtain the following equation:

the image processing apparatus eliminates the following equation, and eliminates t', and obtains:

in the embodiment of the present application, the elimination method may multiply x by two sides of the first equation₂Then multiply x by both sides of the second equation₁The two equation sides are then subtracted simultaneously.

Since R ' is also a rotation matrix, R ' is satisfied '^TSince I is given by | | | R' X | | | | X | |, the image processing apparatus deforms (48), and takes the two-norm to obtain:

thereby, the image processing apparatus obtains a result

The homogeneous equation set of (a) necessarily has a non-zero solution, and the number of rows and columns is necessarily zero, so that d' can be obtained first, then:

depending on the magnitude of the singular values, there are three cases:

d₁>d₂>d₃，d₁＝d₂>d₃or d₁>d₂＝d₃And d₁＝d₂＝d₃

In any case, the solution d ═ d₂A homogeneous system of equations can be made. And d ═ d₂And d ═ d₂The above three cases are also respectively corresponded, so there are 6 possibilities (actually 8, d)₁＝d₂>d₃Or d₁>d₂＝d₃There are two more cases). Only one solution of the actual homography matrix is true, and the image processing apparatus needs to find this solution by:

when d ═ d₂，d₁>d₂>d₃In the case of (1):

the homogeneous system of equations becomes:

from the first two equations: x is the number of₂When x is equal to 0, x is₂Carry 0 into (52):

can obtain e₂＝R'e₂I.e. R' is a surrounding e₂A rotation matrix of rotations. The image processing apparatus may write R' as:

the image processing apparatus substitutes R ' into d ' R ' (x) of formula 48₁e₃-x₃e₁)＝d₃x₁e₃-d₁x₃e₁In (b), one can obtain:

the image processing apparatus is based on d₁≠d₃And

it is possible to obtain:

the image processing apparatus thereby obtains cos θ and sin θ:

substituting cos theta and sin theta into equation (53)

In (3), the image processing apparatus may obtain R'.

Wherein, R' represents a rotation amount between the first photographing coordinate system and the second photographing coordinate system recovered from the homography matrix.

d₁e₁＝d'R'e₁+t'x₁

At the imageThe physical device brings R' back to d of (52)₂e₂＝d'R'e₂+t'x₂In (b) can obtain

d₃e₃＝d'R'e₃+t'x₃

And t' represents the translation amount between the first shooting coordinate system and the second shooting coordinate system recovered by the homography matrix.

In the embodiment of the present application, when the constraint condition is a homography matrix, the image processing apparatus takes t 'as at least one translation amount and takes R' as at least one translation amount.

S2142, the image processing device correspondingly combines the at least one translation amount and the at least one rotation amount to obtain at least one piece of relative pose information.

In the embodiment of the application, after the image processing device calculates at least one translation amount and at least one rotation amount in the first shooting coordinate system and the second shooting coordinate system according to the corresponding relation between the preset constraint condition and the camera pose and the second matching point pair, the image processing device performs one-to-one corresponding combination on the at least one translation amount and the at least one rotation amount, and each combination mode of the translation amount and the rotation amount is used as at least one piece of relative pose information.

When the constraint condition is a basic matrix, according to the different values of t and R obtained in S2141, the image processing apparatus combines the different values of t and R, respectively, and the image processing apparatus can obtain 4 different relative camera poses, each of which includes a value of t and a value of R. When the constraint condition is a homography matrix, the image processing apparatus can obtain 8 possible cases based on the cases of different singular value sizes from the different values of t 'and R' obtained in S2141, as shown in table 2:

TABLE 2

S215, the image processing device calculates a first image position coordinate of the current image feature point in a first shooting coordinate system and a second image position coordinate in a second shooting coordinate system from the second matching point pair aiming at each piece of relative pose information; the first shooting coordinate system is a shooting coordinate system corresponding to the current image characteristic point, and the second shooting coordinate system is a shooting coordinate system corresponding to the reference characteristic point.

In an embodiment of the application, after the image processing apparatus recovers at least one piece of relative pose information according to the second matching point pair, the image processing apparatus calculates, for each piece of relative pose information, a first image position coordinate of the current image feature point in the shooting coordinate system corresponding to the current image feature point and a second image position coordinate of the shooting coordinate system corresponding to the reference feature point from the second matching point pair.

In the embodiment of the application, for each piece of relative pose information, the image processing device calculates the position coordinates of each current image sub-feature point in the second matching point pair under the current image frame shooting coordinate system one by one; and the image processing device calculates the position coordinates of the same current image sub-feature point in the reference frame shooting coordinate system until the current image feature point in the second matching point pair is calculated, the image processing device takes the position coordinates of all the current image feature points in the current image frame shooting coordinate system as the first image position coordinates, and the image processing device takes the position coordinates of all the current image feature points in the reference frame shooting coordinate system as the second image position coordinates.

In the embodiment of the present application, the method for calculating the position coordinates of each current image feature point in the current image frame shooting coordinate system and the position coordinates of the same current image sub-feature point in the reference frame shooting coordinate system by the image processing device is as follows: the image processing device converts the coordinates of the current sub-image feature point in the image pixel coordinate system into three-dimensional coordinates of the current sub-image feature point in a space respectively taking a camera corresponding to the current frame image and a camera corresponding to the reference frame as the origin of the coordinate system according to the relative pose of each camera, and exemplarily, the coordinates of the first image position point are (x, y, z), wherein x represents the coordinates of the current image sub-feature point in the horizontal direction of the imaging interface of the camera shooting the current image frame, y represents the coordinates of the current image sub-feature point in the horizontal direction of the imaging interface of the camera shooting the current image frame, and z represents the imaging focal length, i.e. the depth value, of the camera shooting the current image frame.

S216, calculating a first reference position coordinate of the reference feature point in the first shooting coordinate system from the second matching point pair aiming at each piece of relative pose information by the image processing device; and second reference position coordinates in a second photographing coordinate system.

In the embodiment of the application, the image processing device calculates, for each piece of relative pose information, a first image position coordinate of the current image feature point in the first shooting coordinate system from the second matching point pair, and after a second image position coordinate of the second shooting coordinate system, the image processing device calculates, for each piece of relative pose information, a first reference position coordinate of the reference feature point in the shooting coordinate system corresponding to the current image feature point from the second matching point pair; and second reference position coordinates of the photographing coordinate system corresponding to the reference feature points.

In the embodiment of the application, for each piece of relative pose information, the image processing device calculates the position coordinates of each reference sub-feature point in the second matching point pair in the current image frame shooting coordinate system one by one; and the image processing device calculates the position coordinates of the same reference sub-feature point in the reference frame shooting coordinate system until the second matching point pair finishes calculating the reference feature point, the image processing device takes the position coordinates of the reference feature point in the current image frame shooting coordinate system as first reference position coordinates, and the image processing device takes the position coordinates of the reference feature point in the reference frame shooting coordinate system as second reference position coordinates.

In the embodiment of the present application, the method for calculating the position coordinate of each reference sub-feature point in the current image frame shooting coordinate system and the position coordinate of the same reference sub-feature point in the reference frame shooting coordinate system by the image processing device is as follows: the image processing device converts the coordinates of the reference sub-feature point in the image pixel coordinate system into three-dimensional coordinates of the reference sub-feature point in a space respectively taking a camera corresponding to the current frame image and a camera corresponding to the reference frame as the origin of the coordinate system, illustratively, the first reference position coordinates are (x, y, z), wherein x represents the coordinates of the reference sub-feature point in the horizontal direction of the camera imaging interface shot by the current image frame, y represents the coordinates of the reference sub-feature point in the horizontal direction of the camera imaging interface shot by the current image frame, and z represents the camera imaging focal length, i.e. the depth value, shot by the current image frame.

S217, the image processing device checks each piece of relative pose information respectively based on the first image position coordinate, the second image position coordinate, the first reference position coordinate and the second reference position coordinate, and a checking result corresponding to at least one piece of relative pose information is obtained.

In the embodiment of the application, the image processing device calculates a first reference position coordinate of the reference feature point in the first shooting coordinate system from the second matching point pair aiming at each piece of relative pose information; and after the second reference position coordinate of the second shooting coordinate system, the image processing device respectively verifies whether the position coordinate recovered by each relative pose information is correct or not based on the first image position coordinate, the second image position coordinate, the first reference position coordinate and the second reference position coordinate, and obtains a verification result corresponding to at least one relative pose information.

In this embodiment of the application, the image processing apparatus checks each piece of relative pose information based on the first image position coordinate, the second image position coordinate, the first reference position coordinate, and the second reference position coordinate, respectively, and obtaining a check result corresponding to at least one piece of relative pose information may specifically include: s2171 to S2176. The following were used:

s2171, the image processing device obtains a first image position sub-coordinate and a second image position sub-coordinate corresponding to each current image sub-feature point in the current image feature points and a first reference position sub-coordinate and a second reference position sub-coordinate corresponding to each reference sub-feature point in the reference feature points according to the first image position coordinate and the second image position coordinate.

And S2172, when the first image position sub-coordinate of each current image sub-feature point meets a first preset condition and the second image position sub-coordinate of each current image sub-feature point also meets the first preset condition, the image processing device takes each current image sub-feature point as a correct current image sub-feature point according to each relative pose information, wherein the first preset condition is that the depth value of the three-dimensional coordinate is a preset value.

In the embodiment of the application, after the image processing device obtains a first image position sub-coordinate and a second image position sub-coordinate corresponding to each current image sub-feature point in the current image feature point and a first reference position sub-coordinate and a second reference position sub-coordinate corresponding to each reference sub-feature point in the reference feature point according to the first image position coordinate and the second image position coordinate, for each piece of relative pose information, when the first image position sub-coordinate of each current image sub-feature point meets a first preset condition and the second image position sub-coordinate of each current image sub-feature point also meets the first preset condition, the image processing device takes each current image sub-feature point as a correct current image sub-feature point, and the first preset condition is that a depth value of a three-dimensional coordinate is a preset value.

In the embodiment of the present application, the first image position sub-coordinate is (x, y, z), where x represents a coordinate of the first image position sub-coordinate in a horizontal direction of the camera imaging interface, y represents a coordinate of the first image position sub-coordinate in the horizontal direction of the camera imaging interface, and z represents a camera imaging focal length, i.e. a depth value; the corresponding second image position sub-coordinate is (x)₁,y₁,z₁). If z and z₁If the current image sub-feature point is correct, the image processing device determines that the current image sub-feature point is correct. If z or z₁If one of the values is a negative value, the point is behind the camera, namely the corresponding camera relative pose is incorrect, and the image processing device confirms that the current image sub-feature point is the wrong current image sub-feature point.

S2173 the image processing apparatus determines a correct current image sub-feature point in each current image sub-feature point from the current image feature points, thereby obtaining a correct current image feature point of the current image feature points.

In the embodiment of the application, for each piece of relative pose information, when a first image position sub-coordinate of each current image sub-feature point meets a first preset condition and a second image position sub-coordinate of each current image sub-feature point also meets the first preset condition, the image processing device takes each current image sub-feature point as a correct current image sub-feature point, and after the first preset condition is that a depth value of a three-dimensional coordinate is a preset value, the image processing device determines a correct current image sub-feature point in each current image sub-feature point from the current image feature points, so as to obtain a correct current image feature point of the current image feature points.

And S2174, regarding each piece of relative pose information, when the first reference position sub-coordinate of each reference sub-feature point meets a first preset condition and the second reference position sub-coordinate of each reference sub-feature point also meets the first preset condition, the image processing device takes each reference sub-feature point as a correct reference sub-feature point.

In the embodiment of the application, after the image processing device determines the correct current image sub-feature point in each current image sub-feature point from the current image feature points, so as to obtain the correct current image feature point of the current image feature point, the image processing device regards each reference sub-feature point as the correct reference sub-feature point when the first reference position sub-coordinate of each reference sub-feature point meets a first preset condition and the second reference position sub-coordinate of each reference sub-feature point also meets the first preset condition for each relative pose information.

In the embodiment of the present application, for example, the first reference position sub-coordinate is (x, y, z), where x represents a coordinate of the first reference position sub-coordinate in a horizontal direction of the camera imaging interface, y represents a coordinate of the first reference position sub-coordinate in the horizontal direction of the camera imaging interface, and z represents a camera imaging focal length, that is, a depth value; to pairThe corresponding second reference position sub-coordinate is (x)₁,y₁,z₁). If z and z₁If the positions of the reference sub-feature points are positive, the image processing device determines that the relative pose of the camera is solved correctly, and then the image processing device determines that the reference sub-feature points are correct reference sub-feature points. If z or z₁If one of the values is a negative value, indicating that the point is behind the camera, i.e., the corresponding camera relative pose is incorrect, the image processing apparatus confirms that the reference sub-feature point is the wrong reference sub-feature point.

S2175 the image processing apparatus determines a correct reference sub-feature point in each reference sub-feature point from the reference feature points, thereby obtaining a correct reference feature point of the reference feature points.

In the embodiment of the application, for each piece of relative pose information, when the first reference position sub-coordinate of each reference sub-feature point meets a first preset condition and the second reference position sub-coordinate of each reference sub-feature point also meets the first preset condition, the image processing device determines the correct reference sub-feature point in each reference sub-feature point from the reference feature points after taking each reference sub-feature point as the correct reference sub-feature point, so as to obtain the correct reference feature point of the reference feature point.

And S2176, the image processing device takes the correct current image feature point and the correct reference feature point as a verification result corresponding to each piece of relative pose information, so that at least one verification result corresponding to the relative pose information is obtained.

In the embodiment of the application, the image processing device calculates a first reference position coordinate of the reference feature point in the first shooting coordinate system from the second matching point pair aiming at each piece of relative pose information; and after the second reference position coordinate of the second shooting coordinate system, the image processing device respectively verifies each piece of relative pose information based on the first image position coordinate, the second image position coordinate, the first reference position coordinate and the second reference position coordinate to obtain a verification result corresponding to at least one piece of relative pose information.

S218, the image processing device determines target pose information from at least one piece of relative pose information based on the verification result.

In the embodiment of the application, the image processing device checks each piece of relative pose information respectively based on the first image position coordinate, the second image position coordinate, the first reference position coordinate and the second reference position coordinate, and after a checking result corresponding to at least one piece of relative pose information is obtained, the image processing device finds out the relative pose information with the most recovered correct position coordinates from at least one piece of relative pose information based on the checking result to serve as the target pose information.

In an embodiment of the present application, the determining, by the image processing apparatus, the target pose information from the at least one piece of relative pose information based on the verification result specifically includes: s2181 to S2182. The following were used:

s2181, the image processing device counts the sum of the number of the correct sub current image feature points and the correct sub reference feature points in the verification result corresponding to each relative pose information from the verification result corresponding to at least one piece of relative pose information, so that the number of the correct results corresponding to each piece of relative pose information is obtained.

In the embodiment of the application, the image processing device takes the correct current image feature point and the correct reference feature point as the verification result corresponding to each piece of relative pose information, so that after the verification result corresponding to at least one piece of relative pose information is obtained, the image processing device counts the sum of the number of correct sub-current image feature points and correct sub-reference feature points in the verification result corresponding to each piece of relative pose information from the verification result corresponding to at least one piece of relative pose information, and thus the number of correct results corresponding to each piece of relative pose information is obtained.

And S2182, the image processing device determines the target pose information with the maximum correct result number from the correct result number corresponding to each piece of relative pose information.

In the embodiment of the application, the image processing device counts the sum of the number of correct sub current image feature points and correct sub reference feature points in the verification result corresponding to each relative pose information from the verification result corresponding to at least one piece of relative pose information, so that after the number of correct results corresponding to each piece of relative pose information is obtained, the image processing device determines the target pose information with the largest number of correct results from the number of correct results corresponding to each piece of relative pose information.

And S219, the image processing device realizes a repositioning function according to the target pose information.

In the embodiment of the application, after the image processing device determines the target pose information with the largest number of correct results from the number of correct results corresponding to each piece of relative pose information, the image processing device realizes the repositioning function according to the target pose information.

In the embodiment of the application, the image processing device obtains the translation and rotation of the current frame image relative to the reference frame according to the final output pose, so that the position of the current frame image in the reference frame is obtained, and the repositioning is realized.

It can be understood that the image processing device will acquire the current frame image to be processed; then the image processing device extracts the current image feature point of the current frame image from the current frame image, and is used for finding a reference frame which is most matched with the current frame image feature point according to the current image feature point and acquiring the reference feature point corresponding to the reference frame; then, the image processing device carries out first matching on the current image characteristic points and the reference characteristic points based on a first preset matching threshold value to obtain first matching point pairs; the image processing device amplifies the first preset matching threshold to obtain a second preset matching threshold, and constraint conditions are obtained based on the first matching point pairs; and finally, under the constraint condition, the image processing device matches the current image characteristic points with the reference characteristic points based on a second preset matching threshold value to obtain second matching point pairs. By the method provided by the embodiment of the application, image processing can be realized by firstly calculating a small number of reliable matches and then estimating the mathematical model through the matches, so that more matches are developed, the accuracy of the matched point pairs is ensured, the problems that the number of the matched point pairs is too small and more wrong matched point pairs exist are solved, the number and the accuracy of the matched point pairs are improved, and finally the accuracy of the image processing is improved.

Further, the image processing apparatus may calculate at least one sub-constraint simultaneously when calculating the constraint, and determine the constraint according to the number of correctly matched point pairs in the at least one sub-constraint. When the image processing function is a repositioning function, the image processing device can automatically select the most suitable basic matrix or homography matrix of the current scene to restore the relative pose, so that the repositioning accuracy is improved.

The embodiment of the application provides an image processing device 1, corresponding to an image processing method; fig. 3 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application, where as shown in fig. 3, the image processing apparatus 3 includes:

an obtaining unit 10, configured to obtain a current frame image to be processed;

an extracting unit 11, configured to extract a current image feature point of the current frame image from the current frame image, and obtain a reference feature point corresponding to a reference frame based on the current image feature point;

the matching unit 12 is configured to match the current image feature point with the reference feature point based on a first preset matching threshold to obtain a first matching point pair;

a calculating unit 13, configured to amplify the first preset matching threshold to obtain a second preset matching threshold, and obtain a constraint condition based on the first matching point pair;

the matching unit 12 is further configured to match the current image feature point with the reference feature point based on the second preset matching threshold under the constraint condition, so as to obtain a second matching point pair;

and the processing unit 14 is used for performing an image processing function based on the second matching point pairs.

In some embodiments of the present application, the matching unit 12 is further configured to match the image feature point with the reference feature point based on the second preset matching threshold to obtain an intermediate matching point pair; and screening out the second matching point pairs meeting the constraint condition from the intermediate matching point pairs.

In some embodiments of the present application, the matching unit 12 is further configured to determine a first mapping value based on the three-dimensional coordinates of each first feature point in the intermediate matching point pair and the constraint condition; determining a first matching error based on the first mapping value; and determining the second matching point pairs which are less than or equal to a preset error threshold value based on the first matching error.

In some embodiments of the present application, the matching unit 12 is further configured to filter out sub-matching point pairs other than the first matching point pair from the intermediate matching point pairs; determining a second mapping value based on the three-dimensional coordinates of each second feature point in the sub-matching point pair and the constraint condition; determining a second matching error based on the second mapping value; determining a final sub-matching point pair smaller than or equal to a preset error threshold value based on the second matching error; and taking the first matching point pair and the final sub-matching point pair as the second matching point pair.

In some embodiments of the present application, the calculating unit 13 is further configured to obtain at least one preset initial constraint; calculating the at least one preset initial constraint condition by using the first matching point pair to obtain at least one sub-constraint condition; determining at least one third mapping value based on the three-dimensional coordinates of each third feature point in the first matching point pair and the at least one sub-constraint condition; determining at least one third matching error based on the at least one third mapping value; determining the first matching point pair corresponding to the at least one sub-constraint condition and smaller than or equal to the preset error threshold value as a correct matching point pair based on the at least one third matching error; respectively selecting one sub-constraint condition containing the most correct matching point pairs from the at least one preset initial constraint condition as at least one intermediate sub-constraint condition corresponding to each preset initial constraint condition; selecting, as the constraint, an intermediate sub-constraint that contains the most correct matching point pairs from the at least one intermediate sub-constraint.

In some embodiments of the present application, the calculating unit 13 is further configured to select, for each preset initial constraint condition, a preset number of the first matching point pairs from the first matching point pairs to be combined, and repeat a preset number of times to obtain first matching point pair combinations equal to the preset number of times, where each combination in the first matching point pair combinations includes the preset number of the first matching point pairs; calculating each preset initial constraint condition by using each combination in the first matching point pair combination to obtain the preset times of the sub-constraint conditions corresponding to each preset initial constraint condition, wherein the preset times of the sub-constraint conditions are used as final sub-constraint conditions of each preset initial constraint condition; and determining a final sub-constraint condition of each preset initial constraint condition as the at least one sub-constraint condition in the at least one preset initial constraint condition.

In some embodiments of the present application, the processing unit 14 is further configured to recover at least one piece of relative pose information according to the second matching point pair; for each piece of relative pose information, calculating a first image position coordinate of the current image feature point in a first shooting coordinate system and a second image position coordinate in a second shooting coordinate system from the second matching point pair; the first shooting coordinate system is a shooting coordinate system corresponding to the current image feature point, and the second shooting coordinate system is a shooting coordinate system corresponding to the reference feature point; calculating a first reference position coordinate of the reference feature point in the first shooting coordinate system from the second matching point pair aiming at each piece of relative pose information; and second reference position coordinates in the second photographing coordinate system; respectively verifying each piece of relative pose information based on the first image position coordinate, the second image position coordinate, the first reference position coordinate and the second reference position coordinate to obtain a verification result corresponding to at least one piece of relative pose information; determining target pose information from the at least one relative pose information based on the verification result; and realizing the repositioning function according to the target pose information.

In some embodiments of the present application, the processing unit 14 is further configured to calculate at least one translation amount and at least one rotation amount in the first shooting coordinate system and the second shooting coordinate system according to a corresponding relationship between preset constraint conditions and camera poses and the second matching point pairs; and correspondingly combining the at least one translation amount and the at least one rotation amount to obtain the at least one relative pose information.

In some embodiments of the present application, the processing unit 14 is further configured to obtain, according to the first image position coordinate and the second image position coordinate, a first image position sub-coordinate and a second image position sub-coordinate corresponding to each current image sub-feature point in the current image feature points, and a first reference position sub-coordinate and a second reference position sub-coordinate corresponding to each reference sub-feature point in the reference feature points; regarding each piece of relative pose information, when the first image position sub-coordinate of each current image sub-feature point meets a first preset condition and the second image position sub-coordinate of each current image sub-feature point also meets the first preset condition, taking each current image sub-feature point as a correct current image sub-feature point, wherein the first preset condition is that the depth value of the three-dimensional coordinate is a preset value; determining the correct current image sub-feature point in each current image sub-feature point from the current image feature points, thereby obtaining the correct current image feature point of the current image feature points; for each piece of relative pose information, when the first reference position sub-coordinate of each reference sub-feature point meets the first preset condition and the second reference position sub-coordinate of each reference sub-feature point also meets the first preset condition, taking each reference sub-feature point as a correct reference sub-feature point; determining the correct reference sub-feature point in each reference sub-feature point from the reference feature points, thereby obtaining the correct reference feature point of the reference feature points; taking the correct current image feature point and the correct reference feature point as a verification result corresponding to each piece of relative pose information, so as to obtain a verification result corresponding to the at least one piece of relative pose information; and determining target pose information from the at least one relative pose information based on the checking result.

In some embodiments of the present application, the processing unit 14 is further configured to count a sum of the number of the correct sub current image feature points and the correct sub reference feature points in the verification result corresponding to each relative pose information from the verification result corresponding to the at least one relative pose information, so as to obtain the number of correct results corresponding to each relative pose information; and determining the target pose information with the maximum number of correct results from the number of correct results corresponding to each piece of relative pose information.

In some embodiments of the present application, the extracting unit 11 is further configured to extract, from the current frame image, first pixel points representing corner boundaries, where the number of the corner points is preset; extracting features of second pixel points within a preset range of the first pixel points to obtain first image feature points; extracting features of third pixel points outside the preset range of the first pixel points to obtain second image feature points; taking the first image feature point and the second image feature point as the current image feature point; matching the current image feature points with a preset map feature library, wherein the preset map feature library stores preset map feature points of at least one frame of preset map image frame; and taking the preset map image frame which is most matched with the current image feature point in the preset map feature library as the reference frame, and taking the preset map feature point contained in the reference frame as the reference feature point.

In some embodiments of the present application, the matching unit 12 is further configured to combine each current image sub-feature point in the current image feature points with the reference feature point one by one to obtain at least one sub-feature point pair; aiming at the at least one sub-feature point pair, calculating the Hamming distance from each current image sub-feature point to the reference feature point to obtain at least one Hamming distance, wherein the at least one Hamming distance is in one-to-one correspondence with the at least one sub-feature point pair; determining a minimum hamming distance and a next smallest hamming distance from the at least one hamming distance; comparing the minimum hamming distance to the preset similarity threshold when the minimum hamming distance is less than the preset significance threshold multiplied by the next minimum hamming distance; and when the minimum Hamming distance is smaller than or equal to the preset similarity threshold, determining a sub-feature point pair corresponding to each current image sub-feature point corresponding to the minimum Hamming distance from the at least one sub-feature point pair, so as to obtain the first matching point pair corresponding to the current image feature point.

The embodiment of the application provides an image processing device, which corresponds to an image processing method; fig. 4 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application, and as shown in fig. 4, the image processing apparatus includes: a processor 20, a memory 21, and a communication bus 22. In a Specific embodiment, the Processor 20 may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a CPU, a controller, a microcontroller, and a microprocessor. It is understood that the electronic devices for implementing the above processor functions may be other devices, and the embodiments of the present application are not limited in particular.

In an embodiment of the present application, the memory 21 is configured to store executable instructions; the processor 20 is configured to execute executable instructions stored in the memory 21 to implement the operation steps of the acquiring unit 10, the extracting unit 11, the matching unit 12, the calculating unit 13 and the processing unit 14 in the above embodiments.

In the embodiment of the present application, the communication bus 22 is used for realizing connection communication between the processor 20 and the memory 21.

The image processing device provided by the embodiment of the application can acquire a current frame image to be processed; then the image processing device extracts the current image feature point of the current frame image from the current frame image, and is used for finding a reference frame which is most matched with the current frame image feature point according to the current image feature point and acquiring the reference feature point corresponding to the reference frame; then, the image processing device carries out first matching on the current image characteristic points and the reference characteristic points based on a first preset matching threshold value to obtain first matching point pairs; the image processing device amplifies the first preset matching threshold to obtain a second preset matching threshold, and constraint conditions are obtained based on the first matching point pairs; and finally, under the constraint condition, the image processing device matches the current image characteristic points with the reference characteristic points based on a second preset matching threshold value to obtain second matching point pairs. By the method provided by the embodiment of the application, image processing can be realized by firstly calculating a small number of reliable matches and then estimating the mathematical model through the matches, so that more matches are developed, the accuracy of the matched point pairs is ensured, the problems that the number of the matched point pairs is too small and more wrong matched point pairs exist are solved, the number and the accuracy of the matched point pairs are improved, and finally the accuracy of the image processing is improved. Further, the image processing apparatus may calculate at least one sub-constraint simultaneously when calculating the constraint, and determine the constraint according to the number of correctly matched point pairs in the at least one sub-constraint. When the image processing function is a repositioning function, the image processing device can automatically select the most suitable basic matrix or homography matrix of the current scene to restore the relative pose, so that the repositioning accuracy is improved. The embodiment of the application provides a computer readable storage medium, which stores one or more programs, wherein the one or more programs can be executed by one or more processors and are applied to an image processing device, and when the programs are executed by the processors, the programs realize the image processing method provided by the embodiment of the application.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.

Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a network function deployment system, an air conditioner, or a network device) to execute the method according to the embodiments of the present application.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present application should be covered within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image processing method, comprising:

acquiring a current frame image to be processed;

matching the current image characteristic points with the reference characteristic points based on a first preset matching threshold value to obtain first matching point pairs; the first preset matching threshold comprises a preset similarity threshold;

amplifying the first preset matching threshold to obtain a second preset matching threshold, and obtaining a constraint condition based on the first matching point pair; the constraint condition is at least one of a basis matrix formula and a homography matrix formula corresponding to the first matching point pair;

performing an image processing function based on the second matching point pair;

obtaining a constraint condition based on the first matching point pair comprises: acquiring at least one preset initial constraint condition; determining at least one sub-constraint condition based on the first matching point pair and the at least one preset initial constraint condition, determining the first matching point pair which is corresponding to the at least one sub-constraint condition and is less than or equal to a preset error threshold value based on the three-dimensional coordinates of each third feature point in the first matching point pair to serve as a correct matching point pair, determining at least one intermediate sub-constraint condition based on the correct matching point pair, and determining the constraint condition from the at least one intermediate sub-constraint condition.

2. The method according to claim 1, wherein the matching, under the constraint condition, the current image feature point and the reference feature point based on the second preset matching threshold to obtain a second matching point pair comprises:

3. The method according to claim 2, wherein said screening out the second matching point pair satisfying the constraint condition from the intermediate matching point pairs comprises:

determining a first matching error based on the first mapping value;

4. The method according to claim 2, wherein said screening out the second matching point pair satisfying the constraint condition from the intermediate matching point pairs comprises:

determining a second matching error based on the second mapping value;

5. The method according to claim 1, wherein the determining at least one sub-constraint based on the first matching point pair and the at least one preset initial constraint, determining the first matching point pair corresponding to the at least one sub-constraint being equal to or less than a preset error threshold based on three-dimensional coordinates of each third feature point in the first matching point pair as a correct matching point pair, determining at least one intermediate sub-constraint based on the correct matching point pair, and determining the constraint from the at least one intermediate sub-constraint comprises:

calculating the at least one preset initial constraint condition by using the first matching point pair to obtain the at least one sub-constraint condition;

determining the first matching point pair corresponding to the at least one sub-constraint condition and smaller than or equal to the preset error threshold value as the correct matching point pair based on the at least one third matching error;

respectively selecting one sub-constraint condition containing the most correct matching point pairs from the at least one preset initial constraint condition as the at least one intermediate sub-constraint condition corresponding to each preset initial constraint condition;

6. The method according to claim 5, wherein said calculating the at least one preset initial constraint using the first matching point pair to obtain the at least one sub-constraint comprises:

7. The method according to claim 1, wherein performing an image processing function based on the second matching point pair when the image processing function is a relocation function comprises:

obtaining a correct current image feature point of the current image feature point based on the first image position coordinate and the second image position coordinate, obtaining a correct reference feature point of the reference feature point based on the first reference position coordinate and the second reference position coordinate, and taking the correct current image feature point and the correct reference feature point as a verification result corresponding to each piece of relative pose information, thereby obtaining a verification result corresponding to at least one piece of relative pose information;

8. The method of claim 7, wherein the recovering at least one relative pose information from the second pair of matching points comprises:

9. The method according to claim 7, wherein the obtaining a correct current image feature point of the current image feature point based on the first image position coordinate and the second image position coordinate, obtaining a correct reference feature point of the reference feature point based on the first reference position coordinate and the second reference position coordinate, and using the correct current image feature point and the correct reference feature point as a verification result corresponding to each piece of relative pose information to obtain a verification result corresponding to the at least one piece of relative pose information comprises:

for each piece of relative pose information, when the first reference position sub-coordinate of each reference sub-feature point meets the first preset condition and the second reference position sub-coordinate of each reference sub-feature point also meets the first preset condition, taking each reference sub-feature point as a correct reference sub-feature point; when the depth value in any one of the first image position sub-coordinate, the second image position sub-coordinate, the first reference position sub-coordinate and the second reference position sub-coordinate is a positive number, representing that any one sub-coordinate meets the first preset condition;

the method further comprises the following steps:

10. The method of claim 9, wherein determining target pose information from the at least one relative pose information based on the verification comprises:

counting the sum of the number of the correct current image sub-feature points and the correct reference sub-feature points in the verification result corresponding to each relative pose information from the verification result corresponding to the at least one relative pose information, so as to obtain the number of correct results corresponding to each relative pose information;

11. The method according to claim 1, wherein the extracting a current image feature point of the current frame image from the current frame image and obtaining a reference feature point corresponding to a reference frame based on the current image feature point comprises:

12. The method of claim 1, wherein the first preset matching threshold further comprises: presetting a significance threshold; the matching the current image feature point and the reference feature point based on a first preset matching threshold to obtain a first matching point pair, including:

13. An image processing apparatus characterized by comprising:

the matching unit is used for matching the current image characteristic points with the reference characteristic points based on a first preset matching threshold value to obtain first matching point pairs; the first preset matching threshold comprises a preset similarity threshold;

the calculation unit is used for amplifying the first preset matching threshold value to obtain a second preset matching threshold value and obtaining a constraint condition based on the first matching point pair; the constraint condition is at least one of a basis matrix formula and a homography matrix formula corresponding to the first matching point pair;

a processing unit, configured to perform an image processing function based on the second matching point pair;

the computing unit is further configured to obtain at least one preset initial constraint condition; determining at least one sub-constraint condition based on the first matching point pair and the at least one preset initial constraint condition, determining the first matching point pair which is corresponding to the at least one sub-constraint condition and is less than or equal to a preset error threshold value based on the three-dimensional coordinates of each third feature point in the first matching point pair to serve as a correct matching point pair, determining at least one intermediate sub-constraint condition based on the correct matching point pair, and determining the constraint condition from the at least one intermediate sub-constraint condition.

14. An image processing apparatus characterized by comprising:

a memory for storing executable instructions;

a processor for implementing the method of any one of claims 1 to 12 when executing executable instructions stored in the memory.

15. A computer-readable storage medium having stored thereon executable instructions for use in an image processing apparatus, wherein the executable instructions when executed by a processor implement the method of any one of claims 1 to 12.