CN111986313A

CN111986313A - Loop detection method and device, electronic equipment and storage medium

Info

Publication number: CN111986313A
Application number: CN202010851685.6A
Authority: CN
Inventors: 孙瀚; 周立阳; 冯友计; 姜翰青
Original assignee: Zhejiang Shangtang Technology Development Co Ltd
Current assignee: Zhejiang Shangtang Technology Development Co Ltd; Zhejiang Sensetime Technology Development Co Ltd
Priority date: 2020-08-21
Filing date: 2020-08-21
Publication date: 2020-11-24

Abstract

The application provides a loop detection method and device, electronic equipment and a storage medium. The method comprises the following steps: screening out images with similarity scores reaching a first similarity score threshold value from images in front of the images to be detected in the image sequence to be detected, and creating a loop image candidate set corresponding to the images to be detected based on the screened images. And determining whether a loop image candidate set corresponding to the image to be detected and a loop image candidate set corresponding to at least one continuous frame image before the image to be detected respectively have a common image. The common images are images with similarity scores reaching a second similarity score threshold value, wherein the similarity scores of the images in the image sets reach the second similarity score threshold value. And if the shared image exists, determining the shared image as a loop image corresponding to the image to be detected.

Description

Loop detection method and device, electronic equipment and storage medium

Technical Field

The present application relates to computer vision technologies, and in particular, to a loop detection method and apparatus, an electronic device, and a storage medium.

Background

In the field of computer vision technology, Structure from motion (SfM) is a commonly used three-dimensional reconstruction technique. The technology can determine the camera pose corresponding to each frame of image by analyzing the image sequence, after the camera pose is determined, the motion track of the camera can be determined based on the camera pose corresponding to at least one continuous frame of image, and the scene is subjected to three-dimensional reconstruction based on the motion track.

When the motion trail is determined, errors exist in the determined camera pose caused by light, camera parameters and the like of each frame of image in the continuous frames, and the camera pose corresponding to the next frame of image is generally estimated according to the previous frame of image, so that errors are easily accumulated when the camera pose is calculated based on the continuous frames of image, accumulated errors are caused, the accuracy of the finally determined camera pose is low, and the robustness is poor.

Disclosure of Invention

In view of the above, the present application at least discloses a loop detection method and apparatus, an electronic device, and a storage medium.

According to a first aspect of embodiments of the present application, a loop detection method is provided, including:

screening out an image with the similarity score reaching a first similarity score threshold value from images in front of an image to be detected in an image sequence to be detected, and creating a loop image candidate set corresponding to the image to be detected based on the screened image;

determining whether a loop image candidate set corresponding to the image to be detected and a loop image candidate set corresponding to at least one continuous frame image before the image to be detected have common images; wherein, the common image is the image with the similarity score reaching a second similarity score threshold value;

and if the shared image exists, determining the shared image as a loop image corresponding to the image to be detected.

In an embodiment, the method further includes:

determining the first similarity score threshold value based on the similarity score of each frame image in the image adjacent range corresponding to the image to be detected; the image adjacent range is at least one frame image sequence range which is continuous before and after the image to be detected.

In an embodiment shown in the above, the determining the first similarity score threshold value based on the similarity score of each frame image in the image neighboring range corresponding to the image to be detected includes:

determining the highest similarity score in the similarity scores of each frame of image in the image adjacent range and the image to be detected;

calculating the product of the highest similarity score and a preset first score coefficient to obtain a first similarity score threshold value; wherein, the first scoring coefficient is a natural number between 0 and 1.

In an embodiment, the method further includes:

determining whether the highest similarity score reaches a preset threshold value;

and if the highest similarity score does not reach the preset threshold value, terminating the loop detection aiming at the image to be detected.

In an embodiment shown in the above, the screening out, from the images in the image sequence before the image to be detected, an image whose similarity score with the image to be detected reaches a first similarity score threshold value includes:

screening out an image with the similarity score reaching the first similarity score threshold value from the images in the loop image detection range corresponding to the image to be detected in the image sequence; the loop image detection range is a sequence range from a first frame image in the image sequence to an image in the image sequence, which is located before the image to be detected and has a distance with the image to be detected exceeding a preset number of frames.

In an embodiment shown in the above, determining whether a loop image candidate set corresponding to the image to be detected and a loop image candidate set corresponding to at least one frame of image before the image to be detected include common images, further includes:

combining images meeting continuity conditions in the frame images included in the loop image candidate set respectively to divide the frame images included in the loop image candidate set into a plurality of groups; the continuity condition refers to an image in which the number of matched image feature matching pairs between images reaches a preset threshold;

and screening out the images with the similarity scores reaching a third similarity score threshold value in each group, and replacing the images in the loop image candidate set with the screened images.

In an embodiment, the method further includes:

determining the highest similarity score in the similarity scores of each frame image in each group and the image to be detected;

respectively calculating the product of the highest similarity score and a preset second score coefficient to obtain the third similarity score threshold corresponding to each group; wherein, the second scoring coefficient is a natural number between 0 and 1.

In an embodiment, the method further includes:

counting the number of images in the group corresponding to each group;

comparing the number of the images in the group corresponding to each group with a preset first safety threshold value one by one;

and if the number of the images in the group corresponding to any one group does not reach the first safety threshold, deleting all the images in the group.

In an embodiment shown, the method further comprises:

and if the number of the remaining images in any group is lower than a preset second safety threshold after the images with the similarity scores reaching the third similarity score threshold are screened out, deleting the images in the whole group of the group.

In an embodiment shown, a system applying the method comprises a common image set for storing common images included in a loop image candidate set corresponding to at least one frame image continuous before a current frame image to be detected;

the determining whether a loop image candidate set corresponding to the image to be detected and a loop image candidate set corresponding to at least one continuous frame image before the image to be detected have a common image includes:

determining whether the set of common images includes any images;

if the common image set comprises any image, determining whether the common image set and the loop image candidate set corresponding to the image to be detected comprise common images or not;

and if the common image set and the loop image candidate set corresponding to the image to be detected comprise any common image, determining that the loop image candidate set corresponding to the image to be detected and the loop image candidate set corresponding to at least one continuous frame image before the image to be detected respectively have common images.

In an embodiment shown, the system further comprises a counter indicating the number of updates to the common image set;

if the common image exists, determining the common image as a loop image corresponding to the image to be detected, and the method comprises the following steps:

covering the original images in the common image set with the common images existing in the loop image candidate set corresponding to the to-be-detected image in the common image set so as to update the common image set;

adding 1 to the count of the counter;

determining whether the count of the counter reaches a preset iteration number;

and if the counting number of the counter reaches the preset iteration number, determining the image included in the common image set as a loop image corresponding to the image to be detected.

In an embodiment, determining the common image as a loop image corresponding to the image to be detected includes:

carrying out geometric verification on the common image and the image to be detected;

and if the verification is successful, determining the common image as a loop image corresponding to the image to be detected.

According to a second aspect of embodiments of the present application, there is provided a loop detection apparatus, comprising:

the first screening module screens out images with similarity scores reaching a first similarity score threshold value from images in front of the images to be detected in the image sequence to be detected, and creates a loop image candidate set corresponding to the images to be detected based on the screened images;

a loop image determining module which determines whether a loop image candidate set corresponding to the image to be detected and a loop image candidate set corresponding to at least one continuous frame image before the image to be detected have common images;

In an embodiment, the apparatus further includes:

a similarity degree determination scoring module which determines the first similarity degree scoring threshold value based on the similarity degree scoring of each frame image in the image adjacent range corresponding to the image to be detected; the image adjacent range is at least one frame image sequence range which is continuous before and after the image to be detected.

In an embodiment, the apparatus further includes:

the counting module is used for counting the number of images in the group corresponding to each group;

the comparison module compares the number of the images in the group corresponding to each group with a preset first safety threshold value one by one;

and the deleting module deletes all images in the group if the number of the images in the group corresponding to any one group does not reach the first safety threshold.

In an embodiment shown, the apparatus further comprises:

the loop image determining module comprises:

determining whether the set of common images includes any images;

the module for determining the image of the loop further comprises:

adding 1 to the count of the counter;

determining whether the count of the counter reaches a preset iteration number;

According to a third aspect of embodiments of the present application, the present application provides a computer-readable storage medium storing a computer program for executing the loop detection method according to any one of the first aspect.

According to a fourth aspect of embodiments of the present application, there is provided a loop detection apparatus, the apparatus including: a processor; a memory for storing the processor-executable instructions; wherein the processor is configured to call executable instructions stored in the memory to implement the loop detection method of any one of the first aspect.

According to the technical scheme, the method determines the common images of the loop image candidate set corresponding to the current frame image (for example, the first frame image) and the loop image candidate sets corresponding to at least one frame image (for example, the N-1 th frame image, the N-2 th frame image and the N-3 th frame image) which is continuous before the frame image as the loop image corresponding to the current frame image.

Therefore, on the one hand, if the current frame image itself is abnormal, the corresponding loop image candidate set will not have a common image with the loop image candidate set respectively corresponding to at least one frame image that is continuous before the current frame image. At this time, the loop relation of the current abnormal frame will not be introduced in the process of calculating the camera track, so that the introduction of wrong loop relation due to the abnormality of the current frame can be avoided, and the correctness of the loop detection result is improved.

On the other hand, when the loop image is determined, a more accurate loop image can be determined by combining the incidence relation between the current image to be detected and at least one frame of image before the current image to be detected, so that a more accurate camera pose and camera motion track can be determined based on the loop image, and the accuracy and robustness of three-dimensional reconstruction are improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

In order to more clearly illustrate one or more embodiments of the present application or technical solutions in the related art, the drawings needed to be used in the description of the embodiments or the related art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in one or more embodiments of the present application, and other drawings can be obtained by those skilled in the art without inventive exercise.

FIG. 1 is a diagram of a dictionary structure in the form of a K-ary tree shown in the present application;

FIG. 2 is a flow chart illustrating a method of loop detection according to the present application;

FIG. 3 is a flow chart illustrating a method of loop detection according to the present application;

FIG. 4 is a flow chart illustrating a method of determining an adaptive similarity score threshold according to the present application;

FIG. 5 is a flowchart illustrating a method of a loop image screening method according to the present application;

FIG. 6 is a block diagram of a loop detection device shown in the present application;

fig. 7 is a hardware configuration diagram of a loop detection apparatus according to the present application.

Detailed Description

In order to make those skilled in the art better understand the technical solutions in one or more embodiments of the present application, the technical solutions in one or more embodiments of the present application will be clearly and completely described below with reference to the drawings in one or more embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all embodiments. All other embodiments that can be derived by one of ordinary skill in the art from one or more embodiments of the disclosure without inventive faculty are intended to be within the scope of the disclosure.

The motion trajectory is generally determined based on the camera pose corresponding to at least one continuous image. On the other hand, each frame of image in the continuous frames has certain errors in the camera pose due to light, camera parameters and the like; on the other hand, the camera pose corresponding to the next frame of image is usually obtained by performing pose estimation according to the previous frame of image. Therefore, when the camera motion trajectory is determined based on the camera pose corresponding to at least one continuous frame of image, error accumulation may be easily caused, and the finally determined camera pose is low in accuracy and poor in robustness.

In practical applications, in order to eliminate the accumulated error, a loop image corresponding to a current frame image may be determined in at least one frame image before the current frame image. The loop image may be an image whose similarity score with the current frame image reaches a score threshold in at least one frame image before the current frame. After the loop image corresponding to the frame image is determined, the camera pose of the determined current frame image may be corrected based on the determined loop image.

The process of determining a loop image corresponding to a frame image in at least one frame image before a current frame may be referred to as loop detection. The correspondence between the frame image and the loop image may be referred to as a loop relationship.

For example, if SfM outputs a sequence of pictures with 100 consecutive pictures. Wherein the similarity score of the 10 th to 15 th frame images and the 100 th frame image exceeds a score threshold value. In this case, the 10 th to 15 th frame images may be referred to as loop images corresponding to the 100 th frame image. After the loop image corresponding to the 100 th frame image is determined, the camera pose corresponding to the 100 th frame image may be corrected based on the above 10 th to 15 th frame images.

In the method, because the positions of the loop images in the image sequence are usually forward, the corresponding camera poses of the loop images are also usually less affected by errors; therefore, the camera pose corrected based on the loop image is relatively accurate, and the purpose of eliminating the accumulated error on the whole is achieved.

Through the analysis process, if a three-dimensional reconstruction result with high accuracy and good robustness needs to be obtained, a motion track with high accuracy and good robustness needs to be obtained firstly; the camera track with high accuracy and robustness is obtained, the relatively accurate camera pose needs to be determined, and the relatively accurate camera pose is determined and depends on the correct loop detection result.

Therefore, in the three-dimensional reconstruction in the technical field of computer vision, an accurate loop detection result is obtained through loop detection, and the accuracy and the robustness of the three-dimensional reconstruction are improved.

In a related technique, in order to determine a corresponding loop image of a current frame image in an image sequence, a first frame image of the image sequence, to each frame image in a section formed by a previous frame image of the current frame image, may be directly matched with the current frame image in terms of image features (e.g., SIFT features). Then, the number of matched image feature pairs between two frames of images is calculated, and the ratio of the number of the matched image feature pairs to the number of the image features of the current frame of image is determined as the similarity score of each frame of image and the current frame of image.

After the similarity score between each frame of image and the current image is determined, the similarity scores may be matched with a preset score threshold, and an image with a similarity score reaching the score threshold is determined as a loop image corresponding to the current frame of image.

In the above technical solutions, since the process of directly matching the image features of each frame of image generally requires a relatively large amount of computation and is time-consuming; therefore, the loop detection efficiency is very low by adopting the method.

In order to improve the efficiency and accuracy of loop detection, another related art proposes another way of calculating the similarity score of each frame image based on the bag-of-words tree (hereinafter referred to as bag-of-words tree way).

First, a dictionary (e.g., a dictionary in the form of a K-ary tree) may be constructed based on a subset of the feature descriptions included in the image sequence. After the dictionary is built, the number of words included in each frame image can be determined based on each word (for example, leaf nodes of a K-ary tree) included in the dictionary and the feature descriptors of each frame image, and a corresponding word vector can be built.

Then, determining the similarity score of each frame image before the current frame image and the current frame image based on the word vector corresponding to each frame image;

finally, the similarity scores can be matched with a preset score threshold, and the image with the similarity score reaching the score threshold is determined as the loop image corresponding to the current frame image.

Referring to fig. 1, fig. 1 is a block diagram of a dictionary in a K-ary tree form according to the present application. As shown in fig. 1, when constructing the dictionary, a dictionary in the form of a K-ary tree may be constructed. The K-ary tree has a total of D layers, wherein the nth layer has KⁿAnd the leaf node layer stores words, wherein n is more than or equal to 1 and less than or equal to D. In constructing the K-ary tree, the number of layers (the size of the D value) of the K-ary tree and the number of child nodes (the size of the K value) connected to each node of the K-ary tree may be specified.

The subset of feature descriptions comprised by the image sequence (corresponding to the root node of the K-ary tree) is then subjected to a clustering algorithm (e.g., K-means or K-means + +) to be divided into K classes (each class corresponding to a node of the first level of the K-ary tree). Then, the clustering algorithm is continued to be carried out for each class to be divided into K subclasses. And repeating the steps until the K-ary tree is built. At this time, the feature description subsets included in the leaf nodes of the K-ary tree may be used as words of the dictionary. When words are inquired, the words can be searched layer by layer based on the K-ary tree, and the searching efficiency is high.

It should be noted that the dictionary may be in other forms, and the clustering algorithm may be in other algorithms, which are merely exemplary and not limited herein.

After the K-ary tree dictionary is constructed, word vector description may be performed on each frame of image in the image sequence based on the dictionary.

For example, 256 words are determined based on each image feature in the image sequence, and a dictionary in the form of a K-ary tree is constructed based on the 256 words. At this time, when constructing the word vector corresponding to each frame image, the feature description subset included in each word may be determined based on the dictionary. Then, the number corresponding to each word is determined according to the feature descriptors included in each frame image. After the number of words corresponding to each frame image is determined, a 256-dimensional word vector F { (a1, word 1), (a2, word 2), (a3, word 3), … (ai, word i) …, (a256, word 256) } can be constructed for each frame image; where ai may represent the ith number of words included in each frame image.

After the word corresponding to each frame image is constructed, when determining the loop image corresponding to the current frame image, the similarity score between each frame image before the current frame image and the current frame image can be determined. Specifically, the euclidean distance between the word vector of each frame image and the word vector of the current frame image may be calculated. After the euclidean distance is calculated, the calculated euclidean distance may be used as a similarity score of two images, so as to obtain a similarity score between each image of a previous frame and the current frame.

The method for calculating the similarity score may be a method based on the weight of each word (in this case, ai represents the weight of the ith word included in each frame image) (in a specific process, refer to the related art), and is not particularly limited herein.

After determining the similarity score between each frame image before the current frame image and the current frame image, matching each similarity score with a preset score threshold, and determining the image with the similarity score reaching the score threshold as the loop image corresponding to the current frame image.

In the scheme for calculating the similarity score of each frame of image by using the bag-of-words tree based on the description, when the similarity score between the images is calculated, the direct matching of the image characteristics is replaced by calculating the Euclidean distance of a word vector, so that the calculation complexity is simplified, and the detection efficiency of loop detection is greatly improved;

on one hand, however, when determining the loop image corresponding to the current frame image, only the image with the similarity score reaching the first similarity score threshold value with the current frame image is still used as the loop image, and the association relationship between the current image to be detected and the previous and next frame images is not combined; therefore, assuming that the current frame image itself is an abnormal frame (for example, an abnormal image photographed due to camera shake), the loop image determined by the above-described bag-of-words tree manner will also be an abnormal image. At this time, an erroneous loop-back relationship is introduced in the loop-back detection;

on the other hand, the loop image detected by the method is low in accuracy, so that the robustness of the camera motion track determined based on the loop image is poor, and the three-dimensional reconstruction effect is poor.

Based on this, the application provides a loop detection method for avoiding introducing an erroneous loop relation by combining the incidence relation between the current image to be detected and the previous and next frame images.

In implementation, when determining a loop image corresponding to an image to be detected, an image whose similarity score with the image to be detected reaches a first similarity score threshold may be screened from images in front of the image to be detected in an image sequence to be detected, and a loop image candidate set corresponding to the image to be detected may be created based on the screened images. And sequentially determining the images in the image sequence to be detected as the images to be detected.

After the loop image candidate set corresponding to the image to be detected is obtained, it may be determined whether the loop image candidate set corresponding to the image to be detected and the loop image candidate sets corresponding to at least one frame of images before the image to be detected respectively include a common image. The common images are images with similarity scores reaching a second similarity score threshold value.

If so, the common image may be determined as a loop image corresponding to the image to be detected.

According to the technical scheme, the method determines the loop image candidate set corresponding to the current frame image and the common images of the loop image candidate sets respectively corresponding to at least one frame image continuous before the frame image as the loop image corresponding to the current frame image;

on one hand, for the corresponding loop image candidate set, the loop image candidate set corresponding to at least one frame of image before the frame of image does not have an abnormal frame of a common image, and the loop relation of the loop image candidate set cannot be introduced into the process of calculating the camera track, so that the introduction of an error loop relation due to the abnormality of the current frame can be avoided, and the correctness of the loop detection result is improved;

on the other hand, the loop image is detected by combining the incidence relation between the current image to be detected and the previous and next frame images, so that the detection accuracy of the loop image can be improved, more accurate camera pose and camera motion track can be determined based on the loop image, and the accuracy and robustness of three-dimensional reconstruction are improved.

The method of detecting a loop described in the present application will be described in detail below with reference to specific examples.

Example one

Referring to fig. 2, fig. 2 is a flowchart illustrating a method of loop detection according to the present application.

As shown in fig. 2, the method includes:

s201, screening out an image of which the similarity score with the image to be detected reaches a first similarity score threshold value from images in front of the image to be detected in an image sequence to be detected, and creating a loop image candidate set corresponding to the image to be detected based on the screened image.

S202, determining whether a loop image candidate set corresponding to the image to be detected and a loop image candidate set corresponding to at least one continuous frame image before the image to be detected have common images. The common images are images with similarity scores reaching a second similarity score threshold value.

And S203, if the shared image exists, determining the shared image as a loop image corresponding to the image to be detected.

The image sequence to be detected specifically refers to an image sequence which needs to be subjected to loop detection and is output in the SfM process.

For example, when performing SfM, it is often necessary to calculate the pose and motion trajectory of the camera based on the image sequence output by it. In order to obtain an accurate pose and motion trajectory, loop detection is usually performed on the images in the image sequence to correct the pose and motion trajectory of the camera. In this case, the image sequence is an image sequence to be detected.

The image to be detected is specifically an image which is determined to be currently required to be subjected to loop detection in the image sequence to be detected. When determining the image to be detected, the images in the image sequence to be detected can be sequentially determined as the image to be detected. For example, it is assumed that the sequence to be sequenced is a sequence of 100 consecutive frames. When determining the images to be detected, the current frame image may be determined as the images to be detected in sequence from the first frame image to the end of the 100 th frame image, and the above S201 to S203 are performed for each image to be detected.

The first similarity score threshold may be a fixed threshold that is pre-specified based on development experience, or may be a self-adaptive threshold that is calculated based on a certain algorithm (the algorithm is specifically described in embodiment two). For example, in determining the first similarity score threshold, a fixed threshold (e.g., 0.9) may be used as the first similarity score threshold based on development experience. Or, a set of algorithm may be designed, so that the first similarity score threshold may change according to the change of the image to be detected, and the first similarity score threshold more suitable for the current image to be detected is determined, thereby determining the candidate set of the looped images more matched with the current image to be detected.

The loop image candidate set specifically refers to a set of images, from images in front of an image to be detected in an image sequence to be detected, screened out of which the similarity score with the image to be detected reaches a first similarity score threshold. For example, it is assumed that the sequence to be detected is a sequence of 100 consecutive frames, the image to be detected is a 50 th frame image, and the first similarity score threshold is 0.9. The images of the 1 st to 49 th frames in the sequence to be detected and the image of the 50 th frame are respectively subjected to similarity score calculation to obtain the 1 st to 5 th frames, the 11 th to 15 th frames and the 45 th to 49 th frames with the similarity score reaching 0.9. At this time, the images included in the loop image candidate set are the 1 st to 5 th frames, the 11 th to 15 th frames, and the 45 th to 49 th frames.

The common image is specifically an image with a similarity score reaching a second similarity score threshold value included in the plurality of image sets. When the second similarity score threshold is full, the common image may be the same image common to a plurality of image sets.

For example, assuming that the sequence to be detected is a sequence of 100 consecutive frames, the candidate set of loop images corresponding to the images to be detected, i.e., the images of the 50 th frame, the 51 th frame and the 52 th frame, is A, B, C. In determining the common images included in the set A, B, C, the similarity between the images in a and the images in B may be calculated, and then the images with the calculated similarity score reaching the second similarity score threshold may be combined into a set D. And then calculating the similarity between the image in the D and the image in the C. At this time, the images whose similarity scores have reached the second similarity score threshold are the common images of the set A, B, C. It is to be understood that, when the second similarity score threshold is full, the common image refers to the same image that is common to the A, B, C. The determination manner of the second similarity score threshold may refer to the determination manner of the first similarity score threshold, which is not described herein.

The following is a detailed description of the procedure of example one.

Before loop detection, an image to be detected can be determined from the image sequence to be detected. In an embodiment, each image may be determined as the image to be detected in turn according to the arrangement order of the images in the image sequence to be detected. For example, assuming that the sequence to be detected is a sequence of 100 consecutive frames, after the 50 th frame image is executed in steps S201-S203, the 51 st frame image can be used as the image to be detected.

Of course, it can be understood by those skilled in the art that S201-S203 may be continuously performed by taking the next frame image of the current image to be detected as the image to be detected, assuming that the loop detection process of the current image to be detected is terminated by the program. For example, it is assumed that the sequence to be sequenced is a sequence of 100 consecutive frames. When the 50 th frame image is executed in S202, the loop image candidate set corresponding to the frame image, and the loop image candidate sets respectively corresponding to at least one frame image that is consecutive before the frame image do not include any common image, the loop detection for the 50 th frame image may be terminated, and S201-S203 may be executed continuously with the 51 st frame image as the image to be detected.

After the image to be detected is determined, S201 may be executed, an image whose similarity score with the image to be detected reaches a first similarity score threshold is screened out from images located before the image to be detected in the image sequence to be detected, and a loop image candidate set corresponding to the image to be detected is created based on the screened out images.

When this step is performed, a loop image detection range corresponding to the image to be detected may be determined first.

In an embodiment, a sequence range from a first frame image of the sequence to be detected to a frame image before the frame image to be detected is determined as the loop image detection range. For example, the image to be detected is a 50 th frame image. At this time, the corresponding loop image detection range is 1-49 frame images.

In another embodiment, in order to improve the loop detection effect, when determining the loop image detection range, a sequence range from a first frame image in the image sequence to an image in the image sequence, which is located before the image to be detected and has a distance from the image to be detected exceeding a preset number of frames, may be determined as the loop image detection range. For example, the preset number of frames is 5, the image to be detected is the 50 th frame image, and in this case, the corresponding loop image detection range may be the 1 st to 45 th frame images.

Since the determined loop image detection range does not include images within the preset frame number range adjacent to the current image to be detected, the determined loop image in the loop image detection range will not include images adjacent to the current image to be detected. At the moment, when the loop image corresponding to the image to be detected is determined, the situation that the corrected camera pose error is large due to the fact that the determined loop image is too close to the image to be detected can be avoided, and therefore the loop detection effect is good.

After the detection range of the loop image is determined, the similarity score between each frame of image in the detection range of the loop image and the image to be detected can be calculated based on a preset similarity score algorithm.

In an embodiment, the preset similarity scoring algorithm may be an image feature matching method. In practical applications, the following steps may be performed in sequence from the first frame image in the loop image detection range to the end of the last frame image: s2011, calculating image features (for example, SIFT features) included in the image within the loop image detection range and the current image to be detected, and constructing a feature vector. S2012, a distance (e.g., euclidean distance) between the image features included in the two images is calculated. And S2013, determining the number of matching pairs of which the distance is smaller than a reference threshold value in the image characteristics of the two images. And S2014, calculating the ratio of the number of the matching pairs to the number of the image features of the current image to be detected, and calculating the ratio according to a mapping algorithm to obtain the similarity score of the two images.

For example, the image to be detected includes 100 image features. By the image feature matching method, if the determined feature matching pair of the image a and the image to be detected is 90, the similarity score between the image a and the image to be detected is 0.9.

In another embodiment, the preset similarity score algorithm may be a similarity score algorithm based on the bag-of-words tree method (please refer to the above-mentioned contents related to the bag-of-words tree method, which will not be repeated here). It should be noted that the image feature matching method and the similarity scoring algorithm based on the bag-of-words tree method are only exemplary similarity scoring algorithms, and in practical applications, other algorithms for calculating a similarity score (for example, a similarity scoring algorithm based on a neural network or deep learning) may also be used as the preset similarity scoring algorithm, which is not limited herein.

After calculating the similarity score between each frame of image in the detection range of the loop image and the image to be detected, comparing the similarity score between each frame of image in the detection range of the loop image and the image to be detected with a first similarity score threshold; and forming a loop image candidate set corresponding to the image to be detected by the images with the similarity scores reaching the first similarity score threshold value.

Up to this point, a loop image candidate set corresponding to the image to be detected has been determined by step S201.

After determining the loop image candidate set corresponding to the image to be detected, S202-S203 may be continuously performed to determine whether the loop image candidate set corresponding to the image to be detected and the loop image candidate sets respectively corresponding to at least one continuous frame of image before the image to be detected have a common image. And if so, determining the common image as a loop image corresponding to the image to be detected.

In an embodiment, in the step S202, loop image candidate sets respectively corresponding to at least one frame of image before the image to be detected may be obtained. For example, in one case, the loop image candidate sets respectively corresponding to the at least one continuous frame image are stored in advance in a storage medium of the loop detection system. At this time, the collection may be read from the medium when the collection is acquired. In another case, the loop image candidate sets corresponding to the at least one continuous frame image respectively need to be calculated. In this case, when acquiring the set, the set corresponding to each of the at least one continuous frame image may be calculated based on an algorithm (an image feature matching method or another algorithm) used in S201 for calculating the similarity score.

After the loop image candidate sets corresponding to the at least one continuous frame of image are obtained, common images of the loop image candidate set corresponding to the image to be detected and the loop image candidate sets corresponding to the at least one continuous frame of image can be continuously determined.

In practical application, two loop image candidate sets can be selected at will, and a similarity score between images included in the two loop image candidate sets is calculated based on a preset similarity score algorithm. And then, combining the images with the calculated similarity scores reaching the second similarity score threshold value into a new set. After the new set is formed, the new set and other loop image candidate sets may be subjected to the foregoing steps until common images of the loop image candidate set corresponding to the image to be detected and the loop image candidate sets corresponding to the at least one continuous frame of image are determined.

Here, as will be understood by those skilled in the art, when the second similarity score threshold is a similarity score that can be calculated by the similarity calculation method, the common image is an image that is common to the loop image candidate set corresponding to the image to be detected and the loop image candidate sets corresponding to the respective at least one continuous frame image. For example, when the preset similarity calculation method is the image feature matching algorithm, the similarity score that can be calculated by the similarity calculation method is 1. In this case, when the second similarity score threshold is 1, the images selected by the selection based on the threshold are the same images shared by the plurality of image sets.

And determining a loop image candidate set corresponding to the image to be detected, a loop image candidate set corresponding to at least one continuous frame image before the image to be detected, and if the loop image candidate set corresponding to the image to be detected and the loop image candidate set corresponding to at least one continuous frame image before the image to be detected comprise shared images, determining the shared images as the loop images corresponding to the image to be detected.

For example, assume that the image to be detected is the 52 th frame image, the corresponding loop image candidate set includes {1, 2, 3, 4, 5}, the loop image candidate set corresponding to the 51 st frame includes {1, 2, 3, 4}, and the loop image candidate set corresponding to the 50 th frame includes {3, 4 }. In this case, according to the method described in the above embodiment, if the common images included in the loop image candidate sets corresponding to the 50 th to 52 th frames are {3, 4}, the loop image corresponding to the 52 th frame image is determined to be {3, 4 }.

In another embodiment, the loop detection system may maintain a common image set for storing common images included in a loop image candidate set corresponding to at least one frame of image that is consecutive before a current frame of image to be detected, and a counter indicating the number of iterations of the common image set. At this time, please refer to fig. 3, fig. 3 is a flowchart illustrating a method of a loop detection method according to the present application, and as shown in fig. 3, the steps of S202 to S203 may include:

s2021, determining whether the common image set includes any image. If not, adding the image in the loop image candidate set corresponding to the current image to be detected into the common image set, and updating the count of the counter to be 0 (which means that the current calculation is the first calculation and the iteration number is 0). Then, the loop detection process aiming at the current image to be detected is stopped, and the loop detection step is executed on the next frame image of the frame. Here, it is understood by those skilled in the art that if the above-mentioned common image set is an empty set, it can be stated that the current frame may be the first frame in the image sequence to be detected. Therefore, the candidate set of the loop image corresponding to the current frame image can be used as the common image set, so that the scheme can be ensured to be continuously executed.

If the common image set includes several images during the execution of S2021, the execution of S2022 may be continued to determine whether the loop image candidate set corresponding to the to-be-detected image in the common image set includes a common image.

If the common image set and the loop image candidate set corresponding to the to-be-detected image do not include any common image, S2023 may be continuously performed to delete the image in the common set, add the image in the loop image candidate set corresponding to the current to-be-detected image into the common image set, update the count of the counter to 0, then terminate the loop detection process for the current to-be-detected image, and perform the loop detection for the next frame image of the frame.

If the candidate set of loop images corresponding to the to-be-detected image in the common image set includes a common image, S2024 may be executed to overwrite the included common image with the original image in the common image set to update the common image set. Further, the count of the counter may be increased by 1.

Then, S2025 is further performed to determine whether the count of the counter exceeds a preset number of iterations. And if so, determining the images in the common image set as loop images corresponding to the current frame image.

If not, determining the image to be detected as the next frame image of the current frame image, and continuing to execute S201-S203.

Example two

In an embodiment two, a method for determining an adaptive first similarity score threshold is provided for the first similarity score threshold disclosed in embodiment one. According to the method, the first similarity score threshold value is determined based on the similarity score of each frame of image in the image adjacent range corresponding to the image to be detected, so that the determined first similarity score can be more accurate, and therefore when the loop image corresponding to the image to be detected is screened based on the first similarity score threshold value, a more accurate loop image can be screened, and the three-dimensional reconstruction accuracy and robustness are further improved.

Referring to fig. 4, fig. 4 is a flowchart illustrating a method for determining an adaptive similarity score threshold according to the present application. As shown in fig. 4, S401 determines the highest similarity score among the similarity scores of each frame image in the image neighboring range and the to-be-detected image.

S402, calculating the product of the highest similarity score and a preset first score coefficient to obtain the first similarity score threshold. Wherein, the first scoring coefficient is a natural number between 0 and 1.

The image proximity range specifically refers to a range of at least one frame of image sequence that is continuous before and after the image to be detected. In practical applications, the image proximity range can be established according to development experience. For example, it is assumed that the sequence to be detected is a sequence of 100 consecutive frames, and the image to be detected is a 50 th frame image. At this time, when determining the image proximity range corresponding to the image to be detected, the image sequence range of 5 frames before and after the frame image to be detected may be used as the image proximity range. I.e. the 45-49 and 51-55 frame image sequences are the image neighborhood.

The first scoring coefficient may be a natural number between 0 and 1 manually specified according to development experience, or may be derived by the loop detection system based on machine learning, which is not limited herein.

The execution steps of S401-S402 are described in detail below.

In executing S401, an image proximity range corresponding to an image to be detected may be confirmed first. In practical application, the image sequence range of 5 frames before and after the image to be detected can be used as the image proximity range.

After the image proximity range is determined, the similarity score between the image in the image proximity range and the image to be detected can be obtained.

In practical applications, the similarity score may be pre-calculated and stored in a storage medium of the loop detection system. At this time, when the similarity score is acquired, it may be read from the storage medium. For example, in S201, the similarity score between the image in the vicinity of the image and the image to be detected may be calculated and stored in the storage medium. In this case, the similarity score may be read from the storage medium.

In one case, if the similarity score is not stored in advance, the similarity score between the image in the image proximity range and the image to be detected may be directly calculated based on a preset similarity score algorithm when the similarity score is obtained.

After the similarity scores of the image in the image proximity range and the image to be detected are obtained, the highest similarity score in the similarity scores can be determined.

In practical applications, the similarity scores may be ranked from large to small, and the top similarity score is used as the highest similarity score.

After determining the highest similarity score, S402 may be executed to calculate a product of the highest similarity score and a preset first score coefficient, so as to obtain the first similarity score threshold.

The first similarity score threshold determined based on the technical scheme is determined based on the similarity score condition of the images in the adjacent range of the current image to be detected, so that the method is more suitable for the actual situation. The candidate loopback images screened out through the similarity score are more accurate, and the three-dimensional reconstruction accuracy and robustness are further improved.

It should be noted that the adaptive score threshold determination method disclosed in the second embodiment can be used in conjunction with any embodiment disclosed in this application, and is not described in detail herein.

When the adaptive score threshold determination method is applied to the display loop detection method in an embodiment, whether the current image to be detected is an abnormal image or not may also be determined by the determined highest similarity score. If the image to be detected is determined to be an abnormal image, loop detection of the image to be detected can be stopped, so that the introduction of an error loop relation is avoided, and the influence on a three-dimensional reconstruction result is avoided.

In practical applications, after the highest similarity score in the image neighboring range corresponding to the current image to be detected is calculated through S401, it may be further determined whether the highest similarity score reaches a preset threshold (which may be a preset numerical value based on development experience or machine learning). If the highest similarity score does not reach the preset threshold value, loop detection aiming at the image to be detected is stopped, and a next frame image of the image to be detected is taken as the image to be detected to execute a corresponding loop detection step. If the highest similarity score reaches the preset threshold, the process continues to execute step S402.

Based on the method, if the highest similarity score does not reach the preset threshold, it can be determined that the similarity between the current frame image and at least one frame image before and after the current frame image is not high, and the current frame image may be an abnormal image. At this time, the loop detection of the frame image is stopped, so that the introduction of an erroneous loop relation can be effectively avoided, and the accuracy of the loop detection is improved.

EXAMPLE III

In the process of performing loop detection (including performing loop detection by using a correlation technique or the technical scheme disclosed in the present application), after an image with a similarity score reaching a similarity score threshold value with an image to be detected is screened out and the images are combined into a loop image candidate set, in order to obtain a more accurate loop image to increase the accuracy and robustness of a three-dimensional reconstruction result, further screening with higher accuracy may be performed on the images in the loop image candidate set.

Referring to fig. 5, fig. 5 is a flowchart illustrating a method of a loop image screening method according to the present application. As shown in fig. 5, in S501, images satisfying continuity in the frame images included in the loop image candidate set are combined to divide the frame images included in the loop image candidate set into a plurality of groups. The images meeting the continuity condition refer to a group of images meeting the condition that the number of image feature matching pairs between the images reaches a preset threshold value.

S502, screening out the images of which the similarity scores in all the groups reach a third similarity score threshold value, and replacing the images in the loop image candidate set with the screened images.

The image satisfying the continuity is specifically a group of images satisfying that the number of image feature matching pairs between the images reaches a preset threshold. In practical applications, after images in a group of images are sorted according to the size of the number of frames, if the number of image feature matching pairs between any two frames of images reaches a preset threshold, the group of images can be called as images satisfying continuity.

For example, assuming that the preset threshold is 70, assume that a group of images sorted by frame number size includes 5 images {1, 2, 3, 4, 5 }. At this time, if the number of matched image feature pairs calculated by the image feature matching method between four pairs of images {1, 2}, {2, 3}, {3, 4}, {4, 5} reaches 70, it is considered that the group of images satisfies continuity.

The third similarity score threshold may be a fixed threshold pre-specified based on development experience; or may be a self-adaptive threshold calculated based on an algorithm. For example, in determining the third similarity score threshold, a fixed threshold (e.g., 60) may be used as the third similarity score threshold based on development experience. A set of algorithm (which is described in the following content) may also be designed, so that the third similarity score threshold may be determined according to the images in each group, and the third similarity score threshold adapted to each group is determined, so that the images in each group are more accurately screened, and a more accurate candidate image of the loop image is determined.

When loop detection is performed on an image to be detected (including loop detection performed by using a correlation technique or the technical scheme disclosed in the present application), S501 may be executed after an image whose similarity score reaches a similarity score threshold with the image to be detected is screened out and the images are combined into a loop image candidate set.

When the image grouping is performed in S501, the images in the candidate set of loop images may be sorted by the size of the number of frames. When the sorting is completed, the image with the smallest sequence number (or the largest sequence number) may be taken as the first image (e.g., image a) of the first packet. Then, the number of matching image feature pairs corresponding to the image of its next rank (for example, image B) is calculated based on the image feature matching method.

After the number of the matched image feature pairs is calculated, the number of the matched image feature pairs may be compared with a preset threshold, and it is determined whether the number of the matched image feature pairs reaches the preset threshold. If so, it can be determined that the image a and the image B satisfy the continuity condition. At this point, the image B may be added to the first packet, and the number of matching image feature pairs of the image B and the next sequential image (e.g., image C) may be calculated.

When the number of the matching image feature pairs is calculated, it may be determined whether the number of the matching image feature pairs reaches a preset threshold. If yes, the image B and the image C can be determined to meet the continuity condition, the image C is added into the first group, and whether the following images meet the continuity condition or not is continuously determined.

If the number of the matched image feature pairs corresponding to the image B and the image C does not reach the preset threshold, it can be determined that the image B and the image C do not satisfy the continuity condition. At this time, it may be determined that the first group has been grouped, a group (e.g., a second group) may be newly added, and the image C may be used as a first image of the second group, and it may be continuously determined whether an image next in sequence thereto (e.g., the image D) satisfies a continuity condition. And repeating the steps until the images in the loop image candidate set are grouped.

For example, assume that the loop image candidate set includes { A, B, C, D, E, F }, where A, B satisfies the continuity condition, B, C does not satisfy the continuity condition, C, D, E satisfies the continuity condition, and E, F does not satisfy the continuity condition. In this case, the grouping result according to the above-described embodiment is three groups { A, B }, { C, D, E }, and { F }.

After grouping the loop image candidate set, S502 may be executed to perform more precise screening on the groups.

In step S502, the third similarity score threshold may be obtained first. In one case, the third similarity score threshold may be a static threshold preset based on development experience. At this time, when the third similarity score threshold is acquired, the threshold may be read from the storage medium storing the third similarity score threshold.

In another case, the third similarity score threshold is an adaptive threshold, and when the third similarity score threshold is obtained, the third similarity score threshold may be calculated based on an algorithm preset in the loop detection system. In an embodiment, the predetermined algorithm may be to determine the highest similarity score among similarity scores of each frame image in each group and the image to be detected. And then, respectively calculating the product of the highest similarity score and a preset second score coefficient to obtain the third similarity score threshold corresponding to each group. Wherein, the second scoring coefficient is a natural number between 0 and 1.

The second scoring coefficient may be a natural number between 0 and 1 manually specified according to development experience, or may be derived by the loop detection system based on machine learning, which is not limited herein.

After determining the third similarity score threshold, S502 may be executed to screen out the images in each group whose similarity scores reach the third similarity score threshold, and replace the images in the candidate set of loop images with the screened images.

When the above steps are executed, the similarity scores of the images in each group and the current image to be detected may be respectively compared with the third similarity score threshold corresponding to each group, and the image reaching the third similarity score may be used as a screened image to replace the image in the candidate set of loop images.

According to the technical scheme, the third similarity score threshold is calculated based on the similarity score condition of the images in the groups of the groups and the current image to be detected, so that the similarity score threshold is more suitable for the actual situation, and the images in the candidate set of the loop images are more accurately screened. Therefore, more accurate loop images can be obtained through the scheme, and the accuracy and the robustness of the three-dimensional reconstruction result are improved.

In one case, there may still be false candidate images in the loop image candidate set corresponding to the image to be detected. At this time, although the similarity score between the error image and the image to be detected reaches the first similarity score threshold, the images adjacent to the error image before and after the error image are not selected from the candidate set of loop images. This indicates that the error image may be an image that happens to satisfy the similarity score, although reaching the first similarity score threshold, but is not actually associated with the image to be detected.

In order to delete such erroneous images, in an embodiment, after grouping the images in the candidate set of loop images, the number of images in the group corresponding to each group may be counted; and comparing the number of the images in the group corresponding to each group with a preset first safety threshold value one by one. And if the number of the images in the group corresponding to any one group does not reach the first safety threshold, deleting all the images in the group.

The first safety threshold may be a threshold determined based on development experience or a machine learning algorithm. If the number of images in a group corresponding to a certain grouping is lower than the first safety threshold (for example, 2), it indicates that there is no image satisfying the continuity condition with the images in the grouping in the loop image candidate set, and it indicates that the image may be an erroneous loop image and needs to be deleted.

In another embodiment, in the step of performing S502, if the number of images remaining after screening out the images with the similarity score reaching the third similarity score threshold is lower than a preset second safety threshold, the images of the whole group are deleted.

The second safety threshold may be a threshold determined based on development experience or a machine learning algorithm. If the number of images remaining after a group is screened out images with similarity scores reaching the third similarity score threshold is lower than the second safety threshold (for example, 2), it indicates that the number of images meeting the continuity condition in the group is not enough, and may belong to wrong candidate images, and need to be deleted.

Therefore, the method can be used for further screening the images in the loop image candidate set with higher precision, so that a more precise loop image is obtained to increase the precision and the robustness of a three-dimensional reconstruction result.

It should be noted that the method disclosed in the third embodiment can be used together with any embodiment disclosed in combination with the present application, and is not described in detail herein.

Example four

In order to further obtain a more accurate loop image, the common images determined in S202 may be subjected to set verification, and the common images successfully verified may be determined as the loop images corresponding to the to-be-detected images.

Specifically, if the loop image candidate set corresponding to the image to be detected and the loop image candidate sets corresponding to at least one continuous frame image before the image to be detected respectively include a common image, geometric verification can be performed on the common image and the image to be detected;

In the technical scheme, the determined common image is subjected to geometric verification, and the successfully verified common image is used as the loop image corresponding to the image to be detected, so that the determined loop image is more accurate, and the accuracy and the robustness of a three-dimensional reconstruction result are improved.

In practical applications, the following steps may be performed on the determined common images in step S202 one by one:

a plurality of matching image feature pairs corresponding to the common image and the image to be detected are randomly extracted, and a relative pose transformation matrix (R, T matrix) corresponding to the common image and the image to be detected is calculated based on the extracted matching image feature pairs.

Then, based on the relative pose transformation matrix, it is determined whether the ratio of the number of image features that can be described by the relative pose transformation between the two images to the number of feature points included in the current frame image reaches a reference threshold. If yes, the current common image can be determined to pass the verification; otherwise, it may be determined that the current common image verification failed.

According to the scheme, due to the fact that the determined common images are subjected to geometric verification, the successfully verified common images serve as loop images corresponding to the images to be detected, the determined loop images are more accurate, and accuracy and robustness of three-dimensional reconstruction results are improved.

It should be noted that the method disclosed in the fourth embodiment can be used in conjunction with any embodiment disclosed in this application, and will not be described in detail here.

EXAMPLE five

In the fifth embodiment, a loop detection method is shown, which can be applied to a loop detection system in which a common image set and a counter can be maintained. Wherein the counter indicates a number of iterations of the common image set.

It is assumed that the image sequence to be detected output in the SfM process includes 100 frames of images. Wherein the last 50 frames of the sequence are subjected to loop back detection. At this time, when determining the loop image of the image to be detected, the common image set may be cleared first, and the counter may be cleared by 0.

Then, the current frame image is determined as an image to be detected in order from the 50 th frame image, and the following steps are performed.

S601, determining an image adjacent range and a loop detection range corresponding to the image to be detected.

S602, calculating similarity scores of each image in the image proximity range and the loopback detection range and the image to be detected based on a similarity scoring algorithm in a word bag tree mode.

And S603, determining the highest similarity score in the image adjacent range.

And S604, determining whether the highest similarity score reaches a preset reference threshold value. If not, stopping the loop detection operation of the current image to be detected, clearing the counting of the common image set and the counter, and performing the loop detection operation of the next frame of image.

And S605, if yes, multiplying the determined highest similarity score by a preset first score coefficient to obtain a first similarity score threshold value.

And S606, screening out the images with the similarity scores reaching the first similarity score threshold value in the loop detection range, and forming a loop image candidate set corresponding to the images to be detected.

S607, grouping the images satisfying the continuity condition with respect to the images in the loop image candidate set.

And S608, determining the highest similarity score in each group according to each group, and multiplying the highest similarity score by a preset second score coefficient to obtain a third similarity score threshold corresponding to each group.

And S609, deleting the images of which the similarity scores reach the third similarity score threshold value in each group.

S610, determining whether the number of the images left in each group reaches a safety threshold value; if not, deleting the group of images.

S611, the images in the candidate set of loop images are replaced with the remaining images in each group.

S612, determining whether the common image set maintained by the current loop detection system comprises any image. If not, adding the images in the loop image candidate set into the shared image set, updating the count in the counter to be 0, and stopping the loop detection process of the current image to be detected.

S613, if the common image set includes any image, further determining whether the common image set and the loop image candidate set include common images.

And if not, clearing the common image set, and adding the images in the loop image candidate set into the common image set. Then, updating the count in the counter to 0, and stopping the loop detection process of the current image to be detected;

and S614, if the common image set and the loop image candidate set include common images, replacing the images in the common image set with the included common images, and adding 1 to the count of the counter.

And S615, determining whether the counting number of the counter reaches a preset iteration number. If not, the next frame image of the current image to be detected is continuously taken as the image to be detected to execute the steps.

And S616, if the counting number of the counter reaches the preset iteration number, checking the images in the common image set and the current image to be detected by using the two-view set, and taking the image which is successfully checked as the loop image corresponding to the image to be detected.

So far, the loop detection method is completed.

In the above steps, on one hand, the adaptive first similarity score threshold value is used to replace a similarity score threshold value preset according to development experience in the related art, so that the screened similar images are more accurate.

On the other hand, continuous image grouping is carried out based on the determined loop image candidate set, and more accurate screening is carried out on each group, so that more accurate candidate images, more accurate candidate images and more accurate image selection are obtained,

On the other hand, after the method is used for more accurately screening each group, the images of the groups with the residual images lower than the safety threshold value can be deleted, so that the wrong loop-back images can be filtered,

On the other hand, because the method determines the loop image candidate set corresponding to the current frame image and the loop image candidate set corresponding to at least one frame image which is continuous before the current frame image as the common images which are possessed by the loop image candidate set corresponding to the current frame image, the introduction of abnormal loop relation can be avoided, and more accurate loop images can be determined.

In conclusion, by the technical scheme described in the fifth embodiment, an accurate loop image can be obtained, so that a more accurate camera pose and camera motion trajectory can be determined based on the loop image, and the accuracy and robustness of three-dimensional reconstruction can be improved.

Fig. 6 is a structural diagram of a loop detection device shown in the present application. As shown in fig. 6, the apparatus 600 includes,

the first filtering module 610 filters out an image whose similarity score with the image to be detected reaches a first similarity score threshold value from images in front of the image to be detected in the image sequence to be detected, and creates a loop image candidate set corresponding to the image to be detected based on the filtered image.

The loop image determining module 620 determines whether a loop image candidate set corresponding to the image to be detected and a loop image candidate set corresponding to at least one frame of image before the image to be detected have a common image. The common images are images with similarity scores reaching a second similarity score threshold value.

In an embodiment, the apparatus 600 further includes:

and the similarity determining and scoring module is used for determining the first similarity scoring threshold value based on the similarity scoring of each frame of image in the image adjacent range corresponding to the image to be detected. The image adjacent range is at least one frame image sequence range which is continuous before and after the image to be detected.

and determining the highest similarity score in the similarity scores of the images of the frames in the image adjacent range and the image to be detected.

And calculating the product of the highest similarity score and a preset first score coefficient to obtain the first similarity score threshold. Wherein, the first scoring coefficient is a natural number between 0 and 1.

In an embodiment, the apparatus 600 further includes:

and determining whether the highest similarity score reaches a preset threshold value.

and screening out the images with the similarity scores reaching the first similarity score threshold value from the images in the loop image detection range corresponding to the images to be detected in the image sequence. The loop image detection range is a sequence range from a first frame image in the image sequence to an image in the image sequence, which is located before the image to be detected and has a distance with the image to be detected exceeding a preset number of frames.

and combining images meeting continuity conditions in the frame images included in the loop image candidate set respectively so as to divide the frame images included in the loop image candidate set into a plurality of groups. The continuity condition is an image having a number of matching pairs of image features that match between images, which reaches a preset threshold.

In an embodiment, the apparatus 600 further includes:

and determining the highest similarity score in the similarity scores of the frame images in each group and the image to be detected.

And respectively calculating the product of the highest similarity score and a preset second score coefficient to obtain the third similarity score threshold corresponding to each group. Wherein, the second scoring coefficient is a natural number between 0 and 1.

In an embodiment, the apparatus 600 further includes:

and the counting module is used for counting the number of the images in the group corresponding to each group.

And the comparison module compares the number of the images in the group corresponding to each group with a preset first safety threshold value one by one.

In an embodiment shown, the apparatus further comprises:

the determine loop image module 620 includes:

determining whether the set of common images includes any images;

the loop image determining module 620 further includes:

adding 1 to the count of the counter;

determining whether the count of the counter reaches a preset iteration number;

and carrying out geometric verification on the common image and the image to be detected.

The embodiment of the loop detection device shown in the application can be applied to loop detection equipment. The device embodiments may be implemented by software, or by hardware, or by a combination of hardware and software. Taking a software implementation as an example, as a logical device, the device is formed by reading, by a processor of the electronic device where the device is located, a corresponding computer program instruction in the nonvolatile memory into the memory for operation. In terms of hardware, as shown in fig. 7, a hardware structure diagram of a loopback detection device shown in this application is shown, except for the processor, the memory, the network interface, and the nonvolatile memory shown in fig. 7, an electronic device where a device is located in an embodiment may also include other hardware according to an actual function of the electronic device, which is not described again.

Please refer to fig. 7, which illustrates a loop detection apparatus, the apparatus includes: a processor.

A memory for storing processor-executable instructions.

Wherein, the processor executes the executable instructions to implement the loop detection method as shown in any of the above embodiments.

The present application proposes a computer-readable storage medium storing a computer program for executing the loop detection method as shown in any one of the above embodiments.

One skilled in the art will recognize that one or more embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, one or more embodiments of the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

As used herein, "and/or" means having at least one of two, for example, "A and/or B" includes three scenarios: A. b, and "A and B".

The embodiments in the present application are described in a progressive manner, and the same and similar parts among the embodiments can be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the data processing apparatus embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to part of the description of the method embodiment.

The foregoing description of specific embodiments of the present application has been presented. Other embodiments are within the scope of the following claims. In some cases, the acts or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

Embodiments of the subject matter and functional operations described in this application may be implemented in the following: digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this application and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this application can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a tangible, non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode and transmit information to suitable receiver apparatus for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

The processes and logic flows described in this application can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows described above can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

Computers suitable for executing computer programs include, for example, general and/or special purpose microprocessors, or any other type of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory and/or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer does not necessarily have such a device. Moreover, a computer may be embedded in another device, e.g., a mobile telephone, a Personal Digital Assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device such as a Universal Serial Bus (USB) flash drive, to name a few.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., an internal hard disk or a removable disk), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

Although this application contains many specific implementation details, these should not be construed as limiting the scope of any disclosure or of what may be claimed, but rather as merely describing features of particular disclosed embodiments. Certain features that are described in this application in the context of separate embodiments can also be implemented in combination in a single embodiment. In other instances, features described in connection with one embodiment may be implemented as discrete components or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Further, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.

The foregoing is merely a preferred embodiment of one or more embodiments of the present application and is not intended to limit the scope of the one or more embodiments of the present application, such that any modifications, equivalents, improvements and the like which come within the spirit and principle of one or more embodiments of the present application are included within the scope of the one or more embodiments of the present application.

Claims

1. A loop back detection method, the method comprising:

screening an image with the similarity score reaching a first similarity score threshold value from an image positioned in front of an image to be detected in an image sequence to be detected, and creating a loop image candidate set corresponding to the image to be detected based on the screened image;

determining whether a loop image candidate set corresponding to the image to be detected and a loop image candidate set corresponding to at least one continuous frame image before the image to be detected have common images; the common images are images with similarity scores reaching a second similarity score threshold value, wherein the similarity scores of the images in the image sets reach the second similarity score threshold value;

and if the common image exists, determining the common image as a loop image corresponding to the image to be detected.

2. The method of claim 1, further comprising:

determining the first similarity score threshold value based on the similarity score of each frame of image in the image adjacent range corresponding to the image to be detected; the image adjacent range is at least one frame of image sequence range which is continuous before and after the image to be detected.

3. The method according to claim 2, wherein the determining the first similarity score threshold value based on the similarity score of each frame image in the image neighboring range corresponding to the image to be detected comprises:

calculating the product of the highest similarity score and a preset first score coefficient to obtain a first similarity score threshold value; wherein the first scoring coefficient is a natural number between 0 and 1.

4. The method of claim 3, further comprising:

determining whether the highest similarity score reaches a preset threshold;

5. The method according to any one of claims 1 to 4, wherein screening out an image from the image sequence to be detected that precedes the image to be detected, the image having a similarity score with the image to be detected that reaches a first similarity score threshold value comprises:

screening out images with similarity scores reaching the first similarity score threshold value from the images in the loop image detection range corresponding to the images to be detected in the image sequence; the detection range of the loop image is a sequence range from a first frame image in the image sequence to an image which is positioned in front of the image to be detected in the image sequence and exceeds a preset frame number with the image to be detected.

6. The method according to any one of claims 1 to 5, wherein determining whether a loop image candidate set corresponding to the image to be detected and a loop image candidate set corresponding to at least one frame image in succession before the image to be detected include a common image further comprises:

combining images meeting continuity conditions in the frame images included in the loop image candidate set respectively to divide the frame images included in the loop image candidate set into a plurality of groups; the continuity condition refers to an image with the number of matched image feature matching pairs reaching a preset threshold value;

7. The method of claim 6, further comprising:

determining the highest similarity score in the similarity scores of the frame images in each group and the image to be detected;

respectively calculating the product of the highest similarity score and a preset second score coefficient to obtain a third similarity score threshold corresponding to each group; wherein the second scoring coefficient is a natural number between 0 and 1.

8. The method according to claim 6 or 7, characterized in that the method further comprises:

counting the number of images in the group corresponding to each group;

9. The method according to any one of claims 6-8, further comprising:

10. The method according to claim 1, wherein the system applying the method comprises a common image set for storing common images included in the loop image candidate set corresponding to at least one frame image continuous before the current frame image to be detected;

determining whether the set of common images includes any images;

11. The method of claim 10, wherein the system further comprises a counter indicating a number of updates to the common image set;

adding 1 to the count of the counter;

determining whether the count of the counter reaches a preset iteration number;

12. The method according to any one of claims 1 to 11, wherein determining the common image as a loop image corresponding to the image to be detected comprises:

performing geometric verification on the common image and the image to be detected;

13. A loop detection apparatus, comprising:

a loop image determining module which determines whether a loop image candidate set corresponding to the image to be detected and a loop image candidate set corresponding to at least one continuous frame image before the image to be detected have common images; wherein the common image is an image with a similarity score reaching a second similarity score threshold;

14. A computer-readable storage medium, characterized in that the storage medium stores a computer program for executing the loop detection method according to any one of claims 1 to 12.

15. An electronic device, characterized in that the device comprises:

a processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to invoke executable instructions stored in the memory to implement the loop detection method of any one of claims 1 to 12.