KR20230043958A - Image grouping method and apparatus during 3D reconstruction, electronic device and computer readable storage medium - Google Patents

Abstract

The present invention proposes an image grouping method and apparatus, an electronic device, and a computer readable storage medium during 3D reconstruction. The method includes the steps of calculating modularity corresponding to the image set to be grouped by setting each image of the image set to be grouped as one image group; obtaining an incremental set by calculating an increment of modularity corresponding to each combination, assuming that a plurality of image groups included in the image set to be grouped are combined by two; Determining the maximum increment of the incremental set and merging two image groups corresponding to the maximum increment to obtain a merged image group, repeatedly performing until the image set to be grouped includes a unique image group. ; and when a unique image group is included in the set of images to be grouped, a maximum modularity among the updated modularities is determined in each recorded iteration, and an image combination method corresponding to the maximum modularity is selected as an image for the set of images to be grouped. It includes the step of confirming as a grouping result.

Description

Image grouping method and apparatus during 3D reconstruction, electronic device and computer readable storage medium

CROSS REFERENCES OF RELATED APPLICATIONS

The present invention is submitted based on a Chinese patent application with application number 202010852560.5 and an application date of August 21, 2020, claiming priority to the Chinese patent application, and citing all contents of the Chinese patent application in the present invention by reference. do.

In the field of computer vision technology, Structure from Motion (SfM) is a commonly used 3D reconstruction technique. The technology may determine a camera pose corresponding to a scene image of each frame by analyzing an image sequence corresponding to a scene. After determining the above-mentioned camera pose, the technology determines the motion trajectory of the camera based on the camera pose corresponding to the scene image of several consecutive frames, and 3D reconstruction of the above-mentioned scene based on the above-described motion trajectory can be performed.

In a real situation, on the one hand, the algorithm of the above-described SfM is very complicated, so when performing 3D reconstruction for a large-scale scene, it takes a very long time and the 3D reconstruction efficiency is not high. On the other hand, since computational resources consumed in performing 3D reconstruction on a large-scale scene are enormous, 3D reconstruction may not be implemented in extreme situations.

In order to solve the above problem, in the related art, when performing 3D reconstruction on a large-scale scene, generally first, an image sequence corresponding to the above-mentioned large-scale scene is divided into several groups of image sequences, 3D reconstruction is performed on the image sequence of each group through the SfM algorithm. 3D reconstruction is performed on each group of image sequences to obtain a reconstruction model corresponding to each group, and then a final reconstruction model corresponding to the large-scale scene is obtained by fusing the reconstruction models corresponding to each group. do.

In the process of performing the 3D reconstruction on the above-described large-scale scene, it can be seen that the effect of the reconstruction model of each group may affect the robustness of the final reconstruction model. However, when image grouping is performed in the related art, images are clustered and grouped mainly through a normalized cut algorithm. Although this grouping method can ensure uniformity of segmentation, the accuracy of image grouping is lowered due to weak internal connectivity of image groups obtained by segmentation, and thus the robustness of performing 3D reconstruction based on image grouping is lowered.

Embodiments of the present invention propose an image grouping method and apparatus, an electronic device, and a computer readable storage medium in 3D reconstruction that can improve the accuracy of image grouping and the robustness of 3D reconstruction based on image grouping.

An embodiment of the present invention discloses an image grouping method during 3D reconstruction, the method comprising:

Calculating modularity corresponding to the image set to be grouped by taking each image among the image sets to be grouped as one image group;

obtaining an incremental set by calculating an increment of modularity corresponding to each combination, assuming that a plurality of image groups included in the image set to be grouped are combined by two;

determining a maximum increment in the incremental set, merging two image groups corresponding to the maximum increment to obtain a merged image group, and updating the modularity corresponding to the image set to be grouped and the incremental set; performing iteratively until a unique image group is included in the to-be-image set; and

When a unique image group is included in the set of images to be grouped, the maximum modularity among the updated modularities is determined in each recorded iteration, and an image combination method corresponding to the maximum modularity is selected as image grouping for the set of images to be grouped. It includes the step of confirming the result.

In the illustrated embodiment, the step of determining the image combination method corresponding to the maximum modularity as an image grouping result for the image set to be grouped,

determining whether the maximum modularity has reached a preset threshold;

determining an image combination method corresponding to the maximum modularity as an image grouping result for the image set to be grouped, when the maximum modularity reaches the preset threshold; and

and making each image included in the image set to be grouped into an independent image group when the maximum modularity does not reach the preset threshold.

In one illustrated embodiment, the step of combining a plurality of image groups included in the set of images to be grouped by two,

Combining two image groups having a connection relationship among the plurality of image groups, and recording a correspondence between group identifiers of the image groups having a connection relationship;

The two image groups having a connection relationship indicate that a similarity between images of a first image group and a second image group of the two image groups reaches a preset similarity threshold.

In an illustrated embodiment, the incremental set includes a correspondence between a group identifier of two image groups assumed to be combined and the incremental obtained by calculating and assuming merging the two image groups;

Updating the incremental set comprises:

Searching for a correspondence between group identifiers of image groups having a recorded connection relationship, and determining another image group having a connection relationship with one of the two image groups corresponding to the maximum increment;

After assuming that the different image groups are merged with the two images respectively, the obtained increment is determined, and the result obtained by adding the determined increment is assumed to merge the merged image group with the other image group. incrementing the modularity; and

and recording a correspondence between a group identifier of each of the merged image group and the other image group and the increment of the module obtained assuming that both are merged, in the increment set.

In one illustrated embodiment, the method comprises:

In each iteration, after updating the incremental set, the step of recording a correspondence between the group identifier of the merged image group and the group identifier of the other image group is further included.

In one illustrated embodiment, the method comprises:

In each repetition process, the step of recording the merged image group obtained by merging and the correspondence between the group identifier of the merged image group and the group identifier of the two merged image groups is further included.

In one illustrated embodiment, the merge image group corresponds to an updated modularity in an iteration of creating the merge image group; a group identifier of the image group corresponds to a corresponding image identifier indicating each image included in the image group;

In each repetition of the recording, the step of determining the maximum modularity among the updated modularities and determining the image combination method corresponding to the maximum modularity as the image grouping result for the image set to be grouped,

Starting from the group of merged images recorded in the last iteration, and in reverse order for the group of merged images recorded in each iteration, according to the order of occurrence of the iterations,

It is determined whether the group identifier of the merged image group is included in the identifier set for checking the image combination method, and if the identifier set does not include the group identifier of the merged image group, the group identifier of the merged image group is determined. writing to the set of identifiers;

determining whether an updated modularity corresponding to the merged image group is a maximum modularity among updated modularities in each recorded iteration; and

If the updated modularity corresponding to the merged image group is the maximum modularity among the updated modularity in each recording iteration, the image indicated by the image identifier corresponding to each group identifier included in the identifier set is converted into one image combination. and determining each obtained image combination as an image grouping result for the set of images to be grouped.

In one illustrated embodiment, the method comprises:

When the updated modularity corresponding to the merged image group is not the largest modularity among the updated modularities in each recording iteration, a group identifier of the merged image group recorded in the identifier set is selected as the merged image group corresponding to the merged image group. and replacing with the group identifiers of the previous two image groups.

In the illustrated embodiment, the step of determining whether the updated modularity corresponding to the recorded merged image group is the maximum modularity among the updated modularities in each repetition,

determining whether the modularity updated in this iteration has reached the modularity updated in the previous iteration corresponding to this iteration;

if reached, determining the modularity updated in this iteration as the maximum modularity.

In one illustrated embodiment, assuming that the plurality of image groups are combined in twos, obtaining an incremental set by calculating each increment of modularity corresponding to each combination comprises:

For each of the above increments obtained by calculating and assuming a combination,

Assume combining is for each group of two images hypothesized to combine, determine a binary tree corresponding thereto, and set the increment, and the group identifier of the two images hypothesized to combine, to the determined binary tree. performing the step of writing to the tree;

At the root node of the binary tree, the maximum increment among all increments recorded in the binary tree and a group identifier corresponding to the maximum increment are recorded.

In one illustrated embodiment, ascertaining the maximum increment included in the set of increments comprises:

pushing a root node pointer of the binary tree corresponding to each image group into a maximum heap to determine a maximum increment;

determining a pointer recorded in a root node of the maximum heap; and

and determining the increment recorded at the node indicated by the recorded pointer as the maximum increment included in the increment set.

In one illustrated embodiment, the method comprises:

In each acquired image group, determining an image group in which the number of included images is less than a first preset number threshold, and determining the determined image group as an image group to be merged, wherein the first preset threshold is the minimum of each group - Indicates the number of images;

determining remaining groups other than the image group to be merged among the image groups as the remaining groups;

calculating a connection weight between each two image groups among the to-be-fused groups and constructing a connection weight set, wherein the connection weight indicates the number of images matched between the two image groups; and

A step of fusing the to-be-fused image groups according to the set of connection weights is further included.

In one illustrated embodiment, the fusing the image group to be fused according to the set of connection weights comprises:

Until the connection weight is not included in the connection weight set,

determining a maximum connection weight among the set of connection weights;

determining whether or not the number of images in one of the two image groups corresponding to the maximum concatenation weight reaches a second preset number threshold, wherein the second preset number threshold is set in each image group; Indicates the maximum number of images - ;

terminating this iteration if the second preset number threshold is reached;

merging the two image groups to obtain a merged image group and updating the concatenation weight set when the second preset number threshold is not reached; and

When the connection weight set does not include the connection weight, repeatedly performing the step of setting an image group included in the group to be merged and an image group among the remaining groups as a final image group for the image set to be grouped. includes

In one illustrated embodiment, the step of calculating connection weights between each two image groups among the to-be-fused groups and configuring a connection weight set,

generating a binary tree corresponding to each image group on a one-to-one basis for each image group among the to-be-fused groups; and

Calculating a connection weight between each of two image groups among the groups to be fused, and recording the connection weight and a group identifier of the two image groups in a node of the binary tree - each image in the root node of the binary tree Among the connection weights corresponding to the group, a maximum connection weight and a group identifier corresponding to the maximum connection weight are recorded.

In one illustrated embodiment, the step of determining the maximum connection weight of the connection weight set,

pushing a root node pointer of a binary tree corresponding to each image group to a maximum heap for deriving the maximum connection weight; and

and reading a pointer recorded in a root node of the maximum heap, and determining a connection weight recorded in a binary tree node indicated by the recorded pointer as the maximum connection weight.

In one illustrated embodiment, after determining the maximum connection weight of the set of connection weights, the method:

determining whether a binary tree corresponding to one of the two image groups corresponding to the maximum connection weight is null;

if not null, continuing with subsequent steps; and

If null, further comprising terminating the current iteration.

In one illustrated embodiment, updating the set of connection weights comprises:

deleting a binary tree corresponding to each of the two merged image groups;

calculating a connection weight between the merged image group and another group among the groups to be merged;

writing the calculated connection weight and group identifiers of the merged image group and the other group into a binary tree corresponding to the other group; and

and deleting a node recording a connection weight between the other group and the merged two image groups from the binary tree corresponding to the other group.

An embodiment of the present invention further proposes an image grouping method during 3D reconstruction, the method comprising:

Calculating modularity corresponding to the set of images to be grouped when each image among the set of images to be grouped is regarded as one image group;

obtaining an incremental set by calculating an increment of modularity corresponding to each combination when combining a plurality of image groups included in the image set to be grouped by two;

Determine the maximum increment included in the set of increments, further determine whether the maximum increment reaches 0, and merge the two image groups recorded corresponding to the maximum increment if the maximum increment reaches 0. obtaining a merged image group and updating the incremental set repeatedly until a unique image group is included in the to-be-grouped image set;

In each iteration, when the maximum increment does not reach 0, terminating the above iterative step, and making the image combination method merged in the previous iteration of this iteration the final image grouping result for the image set to be grouped. ; and

and dividing each image included in the image set to be grouped into one image group when the set of images to be grouped includes a unique image group.

An embodiment of the present invention further provides an image grouping device during three-dimensional reconstruction, the device comprising:

a first calculation unit configured to calculate modularity corresponding to the image set to be grouped by taking each image among the image sets to be grouped as one image group;

a first generation unit configured to obtain an incremental set by calculating an increment of modularity corresponding to each combination, assuming that a plurality of image groups included in the image set to be grouped are combined in pairs;

determining a maximum increment in the incremental set, merging two image groups corresponding to the maximum increment to obtain a merged image group, and updating the modularity corresponding to the image set to be grouped and the incremental set; a first merging unit configured to perform iteratively until a unique image group is included in the image set;

When a unique image group is included in the set of images to be grouped, the maximum modularity among the updated modularities is determined in each recorded iteration, and an image combination method corresponding to the maximum modularity is selected as image grouping for the set of images to be grouped. and a first group determination unit configured to determine a result.

In one illustrated embodiment, the first group determination unit also,

determine whether the maximum modularity has reached a preset threshold;

when the maximum modularity reaches the preset threshold, determining an image combining method corresponding to the maximum modularity as an image grouping result for the image set to be grouped;

When the maximum modularity does not reach the preset threshold, each image included in the set of images to be grouped is configured to be an independent image group.

In one illustrated embodiment, the first generation unit also,

Among the plurality of image groups, image groups having a connection relationship are combined by two, and a correspondence relationship between group identifiers of the image groups having a connection relationship is recorded.

The two image groups having a connection relationship indicate that a similarity between images of a first image group and a second image group of the two image groups reaches a preset similarity threshold.

In an illustrated embodiment, the incremental set includes a correspondence between a group identifier of two image groups assumed to be combined and the incremental obtained by calculating and assuming merging the two image groups;

The first merger unit also,

Searching for a correspondence between group identifiers of image groups having a recorded connection relationship, determining another image group having a connection relationship with one of the two image groups corresponding to the maximum increment;

Assuming that the different image groups are merged with the two images respectively, and the obtained increment is determined, and the result obtained by adding the determined increment is assumed, the merged image group is assumed to be merged with the other image group, and then as an increment of the modularity;

and record a correspondence between a group identifier of each of the merged image group and the other image group, and the increment of the module obtained assuming that both are merged, in the increment set.

In one illustrated embodiment, the device,

and a first recording unit, configured to record a correspondence between the group identifier of the merged image group and the group identifier of the other image group after updating the incremental set in each iteration process.

In one illustrated embodiment, the device,

A second recording unit configured to record a merged image group obtained by merging and a correspondence between a group identifier of the merged image group and a group identifier of the merged two image groups in each iteration process.

In one illustrated embodiment, the merge image group corresponds to an updated modularity in an iteration of creating the merge image group; a group identifier of the image group corresponds to a corresponding image identifier indicating each image included in the image group;

The first group determination unit also,

Starting from the group of merged images recorded in the last iteration, and in reverse order for the group of merged images recorded in each iteration, according to the order of occurrence of the iterations,

It is determined whether the group identifier of the merged image group is included in the identifier set for checking the image combination method, and if the identifier set does not include the group identifier of the merged image group, the group identifier of the merged image group is determined. writing to the set of identifiers;

determining whether an updated modularity corresponding to the merged image group is a maximum modularity among updated modularities in each recorded iteration; and

If the updated modularity corresponding to the merged image group is the maximum modularity among the updated modularity in each recording iteration, the image indicated by the image identifier corresponding to each group identifier included in the identifier set is converted into one image combination. and determining each obtained image combination as an image grouping result for the set of images to be grouped.

In one illustrated embodiment, the device,

When the updated modularity corresponding to the merged image group is not the largest modularity among the updated modularities in each recording iteration, a group identifier of the merged image group recorded in the identifier set is selected as the merged image group corresponding to the merged image group. and a replacement unit configured to replace the group identifiers of the previous two image groups.

In one illustrated embodiment, the replacement unit also,

determine whether the modularity updated in this iteration has reached the modularity updated in the previous iteration corresponding to this iteration;

If reached, it is configured to determine the modularity updated in this iteration as the maximum modularity.

In one illustrated embodiment, the first generating unit,

For each of the above increments obtained by calculating and assuming a combination,

Assume combining is for each group of two images hypothesized to combine, determine a binary tree corresponding thereto, and set the increment, and the group identifier of the two images hypothesized to combine, to the determined binary tree. configured to perform the step of writing to the tree;

At the root node of the binary tree, the maximum increment among all increments recorded in the binary tree and a group identifier corresponding to the maximum increment are recorded.

In one illustrated embodiment, the first generation unit also,

push the root node pointer of the binary tree corresponding to each image group to the maximum heap to determine the maximum increment;

determine a pointer recorded in a root node of the maximum heap;

and determine the increment recorded at the node pointed by the recorded pointer as the largest increment included in the set of increments.

In one illustrated embodiment, the device,

an image group to be merged determining unit, configured to determine, in each acquired image group, an image group whose number of included images is smaller than a first preset number threshold, and determine the determined image group as an image group to be merged - the first dictionary The set threshold indicates the minimum number of images in each group;

a remaining group determining unit configured to determine remaining groups other than the to-be-fused image group among the image groups as remaining groups;

a connection weight calculation unit configured to calculate a connection weight between each two image groups among the groups to be fused and configure a connection weight set, wherein the connection weight indicates the number of images matched between the two image groups; and

and a group fusion unit configured to fuse the to-be-fused image groups according to the set of connection weights.

In one illustrated embodiment, the group fusion unit also,

Until the connection weight is not included in the connection weight set,

determining a maximum connection weight among the set of connection weights;

determining whether or not the number of images in one of the two image groups corresponding to the maximum concatenation weight reaches a second preset number threshold, wherein the second preset number threshold is set in each image group; Indicates the maximum number of images - ;

terminating this iteration if the second preset number threshold is reached;

merging the two image groups to obtain a merged image group and updating the concatenation weight set when the second preset number threshold is not reached; and

When the connection weight set does not include the connection weight, the image group included in the group to be merged and the image group among the remaining groups are configured to repeatedly perform the step of making the final image group for the image set to be grouped. do.

In one illustrated embodiment, the connection weight calculation unit also,

for each image group of the to-be-fused group, a binary tree corresponding to each image group on a one-to-one basis is generated;

Calculate a connection weight between each two image groups of the group to be fused, and record the connection weight and a group identifier of the two image groups in a node of the binary tree, wherein each root node of the binary tree Among the linkage weights corresponding to the image group, a maximum linkage weight and a group identifier corresponding to the maximum linkage weight are recorded.

In one illustrated embodiment, the group fusion unit also,

push a root node pointer of the binary tree corresponding to each image group to a maximum heap for deriving the maximum connection weight;

Read a pointer recorded in a root node of the maximum heap, and determine a connection weight recorded in a binary tree node indicated by the recorded pointer as the maximum connection weight.

In one illustrated embodiment, the device,

Determine whether a binary tree corresponding to one of the two image groups corresponding to the maximum connection weight is null,

If not null, continue with subsequent steps;

If null, further comprising a data determination unit, configured to end the current iteration.

In one illustrated embodiment, the group fusion unit also,

delete a binary tree corresponding to each of the two merged image groups;

calculate connection weights between the merged image group and other groups among the groups to be merged;

writing the calculated connection weight and group identifiers of the merged image group and the other group into a binary tree corresponding to the other group;

A node recording a connection weight between the other group and the merged two image groups is deleted from the binary tree corresponding to the other group.

An embodiment of the present invention proposes an image grouping device during 3D reconstruction, the device comprising:

a second calculation unit configured to calculate modularity corresponding to the image set to be grouped when each image among the image sets to be grouped is regarded as one image group;

a second generation unit configured to obtain an incremental set by calculating an increment of modularity corresponding to each combination when combining the plurality of image groups included in the image set to be grouped by two;

Determine the maximum increment included in the set of increments, further determine whether the maximum increment reaches 0, and merge the two image groups recorded corresponding to the maximum increment if the maximum increment reaches 0. a second merging unit, configured to repeatedly perform the step of obtaining a merged image group and updating the incremental set until a unique image group is included in the to-be-grouped image set; and

In each iteration, if the maximum increment does not reach 0, the above iteration steps are terminated, and the image combination method merged in the previous iteration of this iteration is taken as the final image grouping result for the image set to be grouped;

and a second group determination unit configured to divide each image included in the image set to be grouped into one image group when the set of images to be grouped includes a unique image group.

An embodiment of the present invention proposes an electronic device, the device comprising:

processor; and

A memory for storing instructions executable by the processor,

The processor is configured to invoke an executable instruction stored in the memory to implement the image grouping method during 3D reconstruction of any one of the above-described embodiments.

An embodiment of the present invention submits a computer readable storage medium storing a computer program for performing the image grouping method during 3D reconstruction of any one of the above embodiments.

Embodiments of the present invention further provide a computer program including computer readable code, and when the computer readable code is executed in an electronic device, when executed by a processor of the electronic device, the 3D image of any one of the embodiments described above is executed. Implement an image grouping method during reconstruction.

In the above scheme, on the one hand, the method merges the two image groups corresponding to the increment of the maximum modularity in each iteration step, ensuring that the image combination method after each merge is the optimal combination method that can be achieved in this merger. do.

On the other hand, if only one image group remains after merging, the method tracks the updated modularity in each recorded iteration, determines the maximum modularity among the modularities, and determines the image combination method corresponding to the maximum modularity for the grouping. Since the result of image grouping for the set of images to be grouped can be determined, it is possible to ensure that the optimal image combination method shown in the process of merging image groups can be determined as the result of grouping the image for the set of images to be grouped. Through the division of groups, the connection between the images in each group is relatively strong and the connection between the images inside each group and the images in other groups is relatively weak, thereby improving the accuracy of image grouping, thereby enabling 3D reconstruction based on image grouping. improve performance robustness.

It should be understood that the above general description and the detailed description that follows are illustrative and interpretative only and are not intended to limit the invention.

BRIEF DESCRIPTION OF THE DRAWINGS To more clearly describe technical solutions among one or more embodiments or related technologies according to an embodiment of the present invention, the following briefly introduces the accompanying drawings used in the description of the embodiments or related technologies. It is obvious to those skilled in the art that the accompanying drawings in the following description are only some embodiments described in one or a plurality of situations of the present invention, and other accompanying drawings may be obtained according to these accompanying drawings without creative labor. .
1 is an exemplary division diagram of a network community exemplified in an embodiment of the present invention.
2 is a method flow chart of an image grouping method during 3D reconstruction exemplified in an embodiment of the present invention;
3 is an exemplary view of an image grouping scene exemplified in an embodiment of the present invention.
4 is an exemplary diagram of an incremental set illustrated in an embodiment of the present invention.
5 is an exemplary diagram of a balanced binary tree illustrated in an embodiment of the present invention.
6 is an exemplary diagram of image grouping merging exemplified in an embodiment of the present invention.
7 is a method flowchart of an incremental set updating method illustrated in an embodiment of the present invention;
8 is an exemplary diagram of image grouping merging exemplified in an embodiment of the present invention.
9 is a method flowchart of an image grouping method during 3D reconstruction exemplified in an embodiment of the present invention.
10 is an exemplary flow diagram of applying an image grouping method during 3D reconstruction exemplified in an embodiment of the present invention to an actual scene.
11 is a configuration diagram of an image grouping apparatus during 3D reconstruction exemplified by an embodiment of the present invention.
12 is a block diagram of an image grouping apparatus during 3D reconstruction exemplified by an embodiment of the present invention.
13 is a hardware configuration diagram of an electronic device exemplified by an embodiment of the present invention.

In order for those skilled in the art to better understand the technical solutions in one or more situations according to the present invention, the accompanying drawings in one or more situations according to the present invention are combined with Clearly and completely describe the technical options in the situation. It is obvious that the described embodiments are only some embodiments of the present invention, but not all embodiments. Based on one or more embodiments according to the present invention, all other embodiments obtained by a person skilled in the art without creative labor shall all fall within the protection scope of the present invention.

Large-scale 3D reconstruction technology has broad application prospects in the fields of cultural heritage protection, positioning and navigation, scene roaming, augmented reality, industrial measurement, etc., and how to effectively reconstruct large-scale scenes is an important research topic in the field of 3D vision. Due to the size of the scene and the complexity of the reconstruction algorithm, it often takes several days to reconstruct a large scene, and computational resources are also a big problem.

Current 3D reconstruction systems can only process small-scale scenes, or the processing time is too long for large scenes, such as indoor scenes of 80,000 square meters, so the current best SfM systems need to process over 40 hours. this is not efficient In order to solve the above problems, some systems improve reconstruction efficiency through parallel optimization techniques or distributed computation methods. This approach sacrifices computational accuracy, does not scale well, and the time consumption increases exponentially as the scene gets larger. Some other systems group scenes, put the data of each group of scenes on an independent server, perform model reconstruction processing, and then fuse the reconstructed models. However, this method mainly clusters and groups images through a normalized cut algorithm when performing image grouping. The grouping method of normalized segmentation can only guarantee the uniformity of segmentation and cannot guarantee the internal connectivity of image groups obtained by segmentation, resulting in low accuracy of image grouping, and therefore a rover that performs 3D reconstruction based on image grouping str is lowered

An embodiment of the present invention is to propose an image grouping method during 3D reconstruction, and the connection relationship between images in each group determined through the method is relatively strong, and the connection relationship with images in other groups is relatively weak, so image grouping It improves the accuracy of , and further improves the robustness of performing 3D reconstruction based on image grouping. The image grouping method presented in the present invention can be applied to scenes such as 3D modeling, augmented reality, image processing, photography, games, animation, film and television, e-commerce, education, real estate, home decoration, etc., specifically in real situations It can be selected according to, and the embodiment of the present invention is not limited.

The method presented in the embodiment of the present invention first configures each image into an independent group, and at this time, the modularity corresponding to the set of images to be grouped is determined. Next, according to the merging principle in which the modularity increment is the largest after each merger, the image groups included in the image groups are merged, and after each merger, the modularity corresponding to the set of images to be grouped is recorded.

When the images to be grouped include only one image group, the maximum modularity among the modularities recorded in each meeting merging process is found, and the image combination method corresponding to the maximum modularity is determined as the image grouping result for the image set to be grouped. do.

As can be seen from the above scheme, on the one hand, the method merges the two image groups corresponding to the increment of the maximum modularity in each iteration step, so that the image combination method after each merger can be achieved in this merger. Ensure that this is the optimal combination.

On the other hand, if only one image group remains after merging, the method tracks the updated modularity in each recorded iteration, determines the maximum modularity among the modularities, and recalls the image combination method corresponding to the maximum modularity. Since the image grouping result for the image set to be grouped can be determined, it is possible to ensure that the optimal image combination method shown in the process of merging image groups can be determined as the image grouping result for the image set to be grouped. Through the segmentation of groups, the connection between the images in each group is relatively strong and the connection between the images inside each group and the images in other groups is relatively weak, thereby improving the accuracy of image grouping. Improve the robustness of performing reconstructions.

Before describing the embodiments described in the present invention, the applicant first introduces the modularity, the increment of the modularity, and the calculation formula of the modularity and the increment of the modularity in detail.

The modularity is also called the modularity metric, which is a method of measuring the network community segmentation standard. It is explained that the higher the modularity, the more rational the partitioning result for the network. That is, the higher the modularity, the stronger the connection between nodes within each community in the current network and the weaker the connection between nodes within other communities.

The formula for calculating the modularity is as follows.

공식 (1)

Formula (1)

Here, to be explained, the derivation process of Formula (1) can refer to Newman's related descriptions in Finding and evaluating community structure in networks.

Hereinafter, the meaning of each element included in the formula (1) will be described with reference to the accompanying drawings.

Referring to FIG. 1 , FIG. 1 is a diagram illustrating a division of a network community exemplified in an embodiment of the present invention.

As shown in Figure 1, nodes 1-7 constitute one network. Nodes 1, 2, 3, and 4 are connected to each other between adjacent nodes (in other words, nodes 1, 2, 3, and 4 are connected to each other between adjacent nodes), and nodes 5, 6, and 7 are connected to each other between adjacent nodes. (In other words, nodes 5, 6, and 7 have edges connected between adjacent nodes), and nodes 4 and 5 are connected to each other (ie, nodes 4 and 5 have edges connected).

In the above formula (1), the Q represents the modularity corresponding to the current community division situation of the network.

Here, n indicates that the network is divided into n communities. For example, see FIG. 1 . When nodes 1, 2, 3, and 4 included in the network shown in FIG. 1 are regarded as one community and nodes 5, 6, and 7 are regarded as one community, the network is divided into two communities. At this time, n is 2. When each of nodes 1-7 is independently regarded as one community, the network is divided into 7 communities. At this time, n is 7.

The i represents community i among the n communities.

는 상기 커뮤니티 i 내부에 포함된 각 변에 대응하는 부가 가중 인자의 합과, 상기 네트워크에 포함된 모든 변에 대응하는 부가 가중 인자의 합의 비율을 나타낸다. remind

represents a ratio between the sum of additional weighting factors corresponding to each edge included in the community i and the sum of additional weighting factors corresponding to all edges included in the network.

Here, the side may indicate whether a connection relationship exists between two nodes. For example, if the node v and the node w represent two images, a similarity between v and w may be calculated when determining whether or not there is a connection relationship between the nodes v and w. When the similarity between the nodes v and w reaches a preset similarity threshold, it may be considered that a connection relationship exists between the nodes v and w, and it may be considered that two sides exist between v and w. At this time, the v and w are the two end points of the corresponding two sides.

The additional weighting factor may quantitatively represent a connection relationship between the two nodes. For example, when an edge exists between the nodes v and w, the matching number of image feature points between v and w may be mapped as an additional weighting factor of the edge between v and w. It is explained that the connection relationship between the node v and the node w is stronger as the additional weighting factor increases.

를 계산할 수 있다. 공식 (2)에 나타낸 바와 같으며,

공식 (2)이다. 여기서, 상기

는 커뮤니티 i 내부에 포함된 변의 개수를 가리키고, 상기 m은 상기 네트워크에 포함된 변의 개수를 가리킨다. A person skilled in the art can easily understand that in practical applications, additional weighting factors corresponding to sides should be taken into account when calculating modularity, in order to calculate modularity more accurately. However, in order to conveniently describe the embodiment, the value of the additional weighting factor is set to 1 in the present invention. In the above situation, when modularity is calculated, the ratio of the number of edges included in the community i to the number of edges included in the network is calculated,

can be calculated. As shown in formula (2),

This is formula (2). here, above

denotes the number of edges included in community i, and m denotes the number of edges included in the network.

가 0.6인 것을 얻을 수 있다. i 가 2인 경우, 상기

는 0.3이다. For example, continued reference is made to FIG. 1 . Assume that the network is divided into two communities, Community 1 and Community 2 (Community 1 is composed of nodes 1, 2, 3, and 4, and Community 2 is composed of nodes 5, 6, and 7). At this time, the number of sides included in community 1 is 6. The number of sides included in Community 2 is 3. The number of edges included in the network is 10. When i is 1, according to formula (2) above

It can be obtained that is 0.6. When i is 2, the above

is 0.3.

Continuing, the above formula (1) is introduced.

은, 커뮤니티 i 내에 포함된 각 노드의 진입 차수의 합과, 상기 네트워크에 포함된 변의 개수의 2 배의 비율을 가리킨다.

는 아래와 같이 공식 (3)으로 계산할 수 있다. In formula (1) above

denotes the ratio of the sum of the entry degrees of each node included in community i to twice the number of edges included in the network.

can be calculated by Formula (3) as follows.

공식 (3). 여기서, 상기

는 상기 커뮤니티 i 에 포함된 각 노드의 진입 차수의 합을 나타내고, 상기 m은 상기 네트워크에 포함된 변의 개수를 가리킨다.

Formula (3). here, above

denotes the sum of entry degrees of each node included in the community i, and m denotes the number of edges included in the network.

The entry degree of the node indicates the number of end points having the node as an edge. For example, continued reference is made to FIG. 1 . Since four edges are connected to node 4 in FIG. 1 , the number of end points having node 4 as an edge is 4, and at this time, it can be determined that the entry degree of node 4 is 4. On the other hand, since three edges are connected to node 5, the entry degree of node 5 is 3.

는 0.65이다. i 가 2인 경우, 상기

는 0.35이다.With continued reference to FIG. 1, the network is divided into two communities, Community 1 and Community 2 (Community 1 is composed of Nodes 1, 2, 3, and 4, and Community 2 is composed of Nodes 5, 6, and 7). Assume it has been At this time, the sum of the entry degrees of each node included in Community 1 is 13. The sum of the in-degrees of each node included in Community 2 is 7. Twice the number of edges included in the network is 20. When i is 1, the above

is 0.65. When i is 2, the above

is 0.35.

을 소개한다. Then, on and on, incremental modularity

Introduce

The increment of modularity refers to a difference between modularity corresponding to the network after merging two communities included in the network into one community and modularity corresponding to the network before merging.

At this time, a person skilled in the art can understand that, when the increment is an integer, the larger the increment, the greater the value of the modularity corresponding to the network after the merger, that is, the greater the value of this merger. do. When the increment is a negative number, the larger the increment, the smaller the reduced value of the modularity corresponding to the network after the merger, that is, the smaller the negative impact of this merger. Therefore, if it is not possible to determine whether the modularity of the network increases or decreases after the current merger, when merging the communities included in the network, the increment can be made the largest in the combination of all possible two communities in the network. By merging into an existing merging combination, it is possible to ensure that the modularity of the network increases the most or decreases the least after the merger.

로 표시할 수 있다.For example, with continuing reference to FIG. 1, it is assumed that nodes 1 to 7 of the network are each divided into one community. At this time, the modularity corresponding to the network is

can be displayed as

로 표시할 수 있고, 상기 모듈성의 증분의 표현식은

이다.In the above situation, when merging any two communities of the network into one community, the modularity corresponding to the network is

, where the expression for the increment of modularity is

am.

공식의 도출 과정을 소개한다.Hereinafter, the increment of modularity

The derivation process of the formula is introduced.

인 것을 알 수 있다.Assuming that the network currently includes n communities and that the network includes community i and community j, according to Formula (1), the modularity corresponding to the current network is

It can be seen that

It is assumed that community i and community j included in the network are merged, and community p is acquired.

을 획득할 수 있다. 여기서,

는 각각 커뮤니티 i 및 커뮤니티 j에 포함된 변의 개수를 나타낸다. 상기

는 커뮤니티 i에 포함된 각 노드와, 커뮤니티 j에 포함된 각 노드 간에 연결된 변의 개수를 나타낸다. 상기 m은 상기 네트워크에 포함된 변의 개수를 가리킨다. According to formula (2),

can be obtained. here,

denotes the number of edges included in community i and community j, respectively. remind

represents the number of edges connected between each node included in community i and each node included in community j. The m indicates the number of edges included in the network.

이다. 여기서, 상기

는 각각 상기 커뮤니티 i, 커뮤니티 j에 포함된 각 노드의 진입 차수의 합을 나타내고, 상기 m은 상기 네트워크에 포함된 변의 개수를 가리킨다.According to formula (3),

am. here, above

denotes the sum of entry degrees of each node included in community i and community j, respectively, and m denotes the number of edges included in the network.

According to formula (1), the modularity corresponding to the current network is,

이다.

am.

이다.According to the expression of the modularity increment, after merging community i and community j, the increment of modularity corresponding to the network is

am.

으로 커뮤니티 i에 포함된 각 노드와 커뮤니티 j에 포함된 각 노드 사이에 연결된 변의 개수와, 상기 그루핑될 이미지에 포함된 각 이미지 사이에 연결된 변의 개수의 2 배 비율, 즉

를 가리키는 것으로 가정하면,

공식 (4)이다.

, the double ratio of the number of edges connected between each node included in community i and each node included in community j and the number of edges connected between each image included in the image to be grouped, i.e.

Assuming it points to

This is formula (4).

인 것을 도출할 수 있다. 여기서,

커뮤니티 i에 포함된 각 노드와 커뮤니티 j에 포함된 각 노드 사이에 연결된 변의 개수와, 상기 그루핑될 이미지에 포함된 각 이미지 사이에 연결된 변의 개수의 2 배 비율을 가리킨다.

는 각각 커뮤니티 i, 커뮤니티 j 내에 포함된 각 노드의 진입 차수의 합과, 상기 네트워크에 포함된 변의 개수의 2 배의 비율을 가리킨다. Accordingly, when community i and community j are merged, the increment of the module corresponding to the network

which can be derived. here,

It indicates a ratio of 2 times the number of edges connected between each node included in community i and each node included in community j and the number of edges connected between each image included in the image to be grouped.

denotes a ratio of the sum of the entry degrees of each node included in community i and community j, respectively, to twice the number of edges included in the network.

In the above situation, if it is assumed that a community p obtained by merging communities i and j is merged with another community q, then an increment of modularity corresponding to the network needs to be calculated:

을 산출할 수 있다. According to formula (4) above, the increment of the modularity

can be calculated.

공식 (5）임을 알 수 있다.Deriving from the above formula

It can be seen that Formula (5) is

So far, we have introduced modularity, increment of modularity, and related calculation formulas. Hereinafter, embodiments of the present invention will be introduced.

In some embodiments according to the present invention, referring to FIG. 2 , FIG. 2 is a method flow chart of a method for image grouping during 3D reconstruction illustrated in an embodiment of the present invention.

As shown in FIG. 2 , the method may include the following steps.

In step S202, each image among the image sets to be grouped is regarded as one image group, and modularity corresponding to the image set to be grouped is calculated.

In step S204, a plurality of image groups included in the image set to be grouped are combined in pairs, and incremental increments of modularity corresponding to each combination are respectively calculated to obtain an incremental set.

In step S206, determining a maximum increment of the incremental set, merging two image groups corresponding to the maximum increment to obtain a merged image group, and updating the modularity corresponding to the image set to be grouped and the incremental set. is repeatedly performed until a unique image group is included in the set of images to be grouped.

In step S208, when the set of images to be grouped includes a unique image group, in each iteration, the maximum modularity among the updated modularities is determined, and an image combination method corresponding to the maximum modularity is assigned to the set of images to be grouped. It is confirmed by the image grouping result for

It should be explained that the image grouping method can be installed in any software or hardware environment capable of performing the above method. For example, the software and hardware environment may be an electronic device such as a personal PC, PAD, and server. Alternatively, the software and hardware environment may be a virtual system or the like. Not limited here. Hereinafter, an example in which an execution subject of the above-described method is an electronic device (hereinafter, abbreviated as a device) will be described.

In some embodiments, the set of images to be grouped may include an image sequence to be grouped in a 3D reconstruction scene. Understandably, when calculating modularity, the set of images to be grouped can be considered as a network.

It can be understood that, in practical application, when the device maintains the inclusive relationship between the set of images to be grouped and the images included therein, generally, after maintaining one set of image identifiers corresponding to the set of images to be grouped, By writing an image identifier corresponding to an image into the image identifier set to be grouped, an inclusive relationship between the image set to be grouped and the images included in the set is maintained. The image identifier may be a node identifier of a data node corresponding to the image.

The image group may be composed of some images of the set of images to be grouped. It should be understood that when calculating modularity, the group of images can be considered as a community corresponding to the network. The image may be regarded as a node of the network.

It should be understood that, in practical application, when the device maintains an inclusive relationship between the set of images to be grouped and the image groups included therein, generally, after maintaining one set of image group identifiers corresponding to the set of images to be grouped, By writing a group identifier of an image group into the image group identifier set, an inclusion relationship between the image set to be grouped and the image groups included therein is maintained. The group identifier may be a node identifier of a data node corresponding to the image group.

At this time, on the other hand, if it is necessary to determine the image group included in the set of images to be grouped, first read each group identifier included in the set of image group identifiers, and then select the image group corresponding to each group identifier to be grouped. You can output as a group of images included in an image set.

On the other hand, when performing a merging operation on any two image groups included in the set of images to be grouped, the group identifiers of the two merged image groups are deleted from the set of image group identifiers, and the merging operation The group identifier of the merged image group generated by the image group identifier may be written into the image group identifier set to complete the merge operation.

When maintaining an inclusion relationship between the image group and images included in the image group, the device may generally maintain a correspondence relationship between an identifier of the image group and an image identifier corresponding to an image included in the image group. For example, the device may create one data node for the image group, and set a group identifier of the image group as a node identifier (node index) of the data node. Next, the device may write an image identifier corresponding to an image included in the image group to the data node to maintain an inclusion relationship between the image group and the images included in the image group.

In some embodiments, the image combination method may include a method of combining each image included in the set of images to be grouped after completing the merging step once in each iteration.

In practical applications, it is generally possible to define a group identifier corresponding to each image group, and an image identifier corresponding to an image. Next, the image combination method may be recorded by recording a correspondence between each group identifier and an image identifier corresponding to each image.

For example, it is assumed that the set of images to be grouped includes 10 images, each corresponding to image identifiers 0-9. After one-time merging is completed, identifiers of image groups included in the set of images to be grouped are a, b, c, and d, the group identifier a corresponds to the image identifiers 0-3, and the group identifier b It is assumed that the image identifier 4-5 corresponds, the group identifier c corresponds to the image identifier 6, and the group identifier d corresponds to the image identifier 7-9.

In the above situation, when it is assumed that each image combination method included in the current set of images to be grouped must be determined, a group identifier included in the set of images to be currently grouped can be determined. After determining the currently included group identifier, searching for a correspondence between the group identifier and the image identifier, the image group method currently included in the image set to be grouped is: image 0-3 is one image group, image 4 It can be determined that -5 is one image group, image 6 is one image group, and images 7-9 are one image group.

When performing image grouping on a set of images to be grouped, the device first performs step S202 of calculating modularity corresponding to the set of images to be grouped by setting each image among the set of images to be grouped as one image group. can

In practical application, the device may first traverse each image in the set of images to be grouped, and match each image with a unique image identifier on a one-to-one basis. Next, the device can match a unique group identifier to each image identifier. In this case, the device may record a correspondence between the image group and the image by recording a correspondence between each group identifier and the image identifier.

It should be understood that in this step, since it is assumed that each image included in the set of images to be grouped independently constitutes a group, the image identifier corresponding to each image becomes the group identifier of the image group corresponding to each image. can

For example, the set of images to be grouped includes 10 images. In implementing step S202, the device may first map image identifiers 0-9 sequentially to the 10 images on a one-to-one basis. Next, the device may generate the same number of data nodes as the number of images (here, the data nodes may be recorded in the form of an array), and correspond the data nodes to the 10 images on a one-to-one basis . Then, the device sets the image identifier corresponding to each image as a node identifier (node index) of a data node corresponding to each image, and assigns an image identifier corresponding to each image to a data node corresponding to each image. can be entered.

It should be understood that, at this time, each data node actually corresponds to the 10 images one-to-one or to the 10-image group one-to-one, and the group identifier of the 10 image groups is 0-9.

Referring to FIG. 3 , FIG. 3 is an exemplary view of an image grouping scene according to an embodiment of the present invention.

As shown in FIG. 3, the illustrated image group scene is to group 10 images included in the set to be grouped. A node identifier (node index) of each data node shown in FIG. 3 is a group identifier of an image group corresponding to each data node. An image identifier corresponding to an image included in an image group corresponding to each data node is recorded in each data node.

What will be described herein is that the data node configuration shown in FIG. 3 is only exemplary, and may include various data node configurations in actual applications, but is not limited thereto.

을 계산할 수 있다.After assuming each image among the set of images to be grouped as one image group, the device determines the modularity corresponding to the network in the current image combination method according to the above formula (1).

can be calculated.

을 계산하기 전에, 상기 기기는 현재 이미지 조합 방식에서 각 이미지 그룹 간의 연결관계를 먼저 확정할 수 있다.In practical applications, the modularity

Before calculating , the device may first determine the connection relationship between each image group in the current image combination method.

Here, it will be described that, since each image group currently includes one image, determining the connection relationship between each image group corresponds to determining the connection relationship between the respective images included in the image set to be grouped.

When determining the connection relationship between each image included in the above-described image to be grouped, each image included in the image to be grouped is set as a target image in turn, and the similarity between the target image and other images included in the image to be grouped is determined. can be calculated

After obtaining the similarity between the target image and any other image, it is determined whether the similarity between the target image and the other image is greater than a preset similarity threshold (wherein the preset similarity threshold may be set according to experience). It can be further determined, and if it is large, it may be considered that a connection relationship exists between the target image and the corresponding other image, or that the target image and the corresponding other image are connected. At this time, the target image and the corresponding other image are the two end points of the side.

When it is determined that there is a side between any two images, a correspondence relationship between group identifiers of image groups corresponding to each of the two images may be recorded in a contiguous table maintained by the device. For example, in the image grouping scene corresponding to FIG. 3 , data nodes 0 to 9 correspond to each image group 0 to 9 included in an image set to be grouped on a one-to-one basis. In this case, when it is determined that a side exists between image 0 and image 1, a correspondence between group identifier 0 and group identifier 1 may be recorded in the adjacency table.

What will be described here is that in the method of maintaining the above-mentioned adjacency table, since the correspondence of group identifiers of two image groups having a connection relationship can be recorded twice, a duplicate removal operation is performed here, and the adjacency table It is possible to ensure that the correspondence of group identifiers of two image groups having a connection relationship is recorded only once.

As will be understood by those skilled in the art, after checking the connection relationship between each image included in the set to be grouped, the adjacency table records all correspondence relationships between group identifiers of image groups having current connection relationships.

In this case, when it is necessary to determine whether a connection relationship exists between two arbitrary image groups, it may be searched whether a correspondence relationship between group identifiers of the two image groups is recorded in the adjacency table. If a correspondence relationship is recorded, it can be considered that the two image groups have a connection relationship.

When it is necessary to calculate the entry degree of an image group, the number of correspondences of group identifiers of the image group among correspondences recorded in the adjacency table may be counted.

을 계산하는 경우, 먼저 하나의 카운터(초기값은 0)를 설정할 수 있다. 다음, 상기 인접 테이블에 기록된 그룹 식별자 대응관계가 그룹 식별자 0을 포함하는지 여부를 트래버스한다. 상기 그룹 식별자 0의 대응관계가 검색될 때마다, 상기 카운터의 카운트는 1 씩 증가될 수 있다. 상기 인접 테이블의 검색을 완료한 후, 상기 카운터의 카운트를 판독하고, 상기 카운트를 이미지 그룹 0의 진입 차수

로 할 수 있다. For example, the in-order of image group 0

In the case of calculating , one counter (initial value is 0) can be set first. Next, whether or not the group identifier correspondence relationship recorded in the neighbor table includes the group identifier 0 is traversed. Whenever the correspondence of the group identifier 0 is searched, the count of the counter may be increased by one. After completing the search of the adjacency table, the count of the counter is read, and the count is the entry order of image group 0

can be done with

를 계산할 수 있으며, 상기

은 이미지 그룹 i의 진입 차수를 가리킨다.When the confirmation of the connection relationship between each image included in the image to be grouped is completed, the device determines the entry degree corresponding to each image group.

can be calculated, and the

denotes the entry order of image group i.

에 따라, 상기 그루핑될 이미지 집합에 포함된 변의 개수를 산출할 수 있다. When you have finished calculating the in-order of each group of images, the formula

According to, the number of sides included in the image set to be grouped can be calculated.

를 산출할 수 있으며, i는 0보다 크고 상기 그루핑될 이미지 집합에 포함된 이미지 수보다 작거나 같은 자연수이다. Next, according to the above formula (3), corresponding to each image group

Can be calculated, i is a natural number greater than 0 and less than or equal to the number of images included in the set of images to be grouped.

는 0이다.Currently, each image group contains one image, so at this time

is 0.

와

의 계산이 완료된 경우, 공식 (1)에 따라, 상기 현재 이미지 조합 방식에서 상기 네트워크에 대응하는 모듈성

을 계산할 수 있다.

and

When the calculation of is completed, according to Formula (1), the modularity corresponding to the network in the current image combination method

can be calculated.

을 산출한 경우, 상기 기기는 상기

을 기록할 수 있다. 실제 응용에서, 상기

은 임의의 형태로 기록될 수 있다. 예를 들어, 상기

은 도 3에 도시된 데이터 노드에 기록되거나, 하나의 어레이를 전문적으로 유지하여 상기

을 기록할 수 있다.remind

When calculating , the device

can be recorded. In practical applications, the

can be written in any form. For example, the

is written to the data node shown in FIG. 3, or specially maintained one array to

can be recorded.

When step S202 is completed, assuming that a plurality of image groups included in the image set to be grouped are combined by two, the device calculates increments of modularity corresponding to each combination to obtain an incremental set S204. can continue to do so.

In actual application, it is assumed that the device combines each image group of the set of images to be grouped two by two, and for each obtained combination, merges the image group corresponding to the combination, then calculates the modularity increment; , a set of increments may be generated based on the increments obtained by calculating .

In an embodiment of the present invention, when calculating the increment of modularity, the device may first combine any two image groups included in the to-be-grouped image set, and then obtain all possible combination results. After obtaining the combination result, for each combination result, it is first determined whether the two image groups corresponding to the combination result have a connection relationship (for example, in the adjacency table, the two image groups It is possible to determine whether a connection relationship between two image groups is provided by searching whether a correspondence relationship between group identifiers is recorded).

는 1로 확정할 수 있다. 반면, 연결관계가 없는 상기 두 개의 이미지 그룹 사이의 변의 개수

는 0으로 확정할 수 있다.At this time, if the two image groups corresponding to the combination result have a connection relationship, if the connection relationship is provided, the number of sides between the two image groups having a connection relationship

can be determined as 1. On the other hand, the number of sides between the two image groups without a connection

can be set to 0.

를 확정한 후, 상기 기기는 상기 공식 (4)에 따라, 상기 조합에 대응하는 이미지 그룹을 합병하는 것으로 가정한 후 상기 모듈성의 증분

을 계산할 수 있으며, 상기 i, j는 0보다 크거나 같고 9보다 작거나 같은 정수이다.Number of sides between the two image groups

After determining , it is assumed that the device merges the image group corresponding to the combination, according to the above formula (4), and then the increment of the modularity.

Can be calculated, wherein i and j are integers greater than or equal to 0 and less than or equal to 9.

의 계산을 완료한 후, 상기 기기는 계산하여 획득된 상기 증분을 기반으로 증분 집합을 생성하고, 조합하는 것으로 가정된 두 개의 이미지 그룹의 그룹 식별자와, 상기 두 개의 이미지 그룹을 합병한 후, 계산하여 획득된 상기 증분의 대응관계를 기록할 수 있다.above increment

After completing the calculation of , the device generates an incremental set based on the incremental obtained by calculating, obtains a group identifier of two image groups assumed to be combined, merges the two image groups, and then calculates It is possible to record the correspondence of the increments obtained by doing so.

의 대응관계를 상기 증분과 그룹 식별자 대응관계 테이블에 기록한다.In actual application, the device may maintain a table of correspondence between increments and group identifiers, and the increments calculated after assuming that all possible combinations and all possible image groups corresponding to the combinations are merged.

The correspondence of is recorded in the increment and group identifier correspondence table.

Thus, the preparation work for the above-described image grouping method has been completed.

Enter the next iteration step. In some embodiments, the device performs step S206 to determine the maximum increment included in the increment set, merge two groups of recorded images corresponding to the maximum increment, and update the modularity after merging. , the step of updating the incremental set after merging is repeatedly performed until a unique image group is included in the set of images to be grouped.

In an embodiment of the present invention, when determining the maximum increment included in the increment set, the device may search the increment set to obtain all increments. Next, the device may compare every increment two by two to determine the largest increment of the set of increments.

After determining the maximum increment, the device may search for a correspondence between the increment and the group identifier, and determine two image groups corresponding to the maximum increment. After determining the two image groups, the device may merge the two image groups.

In practical application, when merging the two image groups, the device may merge the two image groups to obtain a merged image group. Next, the device deletes the group identifiers of the two merged image groups from the image group identifier set corresponding to the image set to be grouped, and assigns the group identifier of the merged image group generated by the merging operation to the image group identifier set. can be entered in It should be understood that, at this time, the image group identifier set includes the group identifier of the merged image group and the group identifier of other non-merged image groups.

For example, assume that the set of images to be grouped includes 10 image groups with group identifiers 0-9, and the group identifiers of the two image groups corresponding to the maximum increment are 0 and 1. A group identifier of the merged image group obtained after merging the two image groups is a. At this time, after merging the two image groups, the image groups currently included in the image to be grouped include group a and groups 2-9.

After completing the merging, since a change has occurred in the image group included in the set to be grouped, the modularity of the set to be grouped needs to be updated.

에 이번 합병에 대응하는 최대 증분을 더하여, 업데이트된 상기 모듈성을 획득할 수 있다. When updating the modularity of the set to be grouped, the device records the

The updated modularity can be obtained by adding the maximum increment corresponding to this merger to .

When updating of the current modularity is completed, the device may record the image combination type indicated by each image group and the modularity.

When the above merging is completed, since a new merged image group is created, the incremental set needs to be updated.

In a practical application, when updating the incremental set, the increment of the modularity may be first calculated assuming that the merged image group and each other image group included in the image set to be grouped are merged.

After calculating the increment, the device may record the increment in the increment set, and delete the increment corresponding to the group identifier of any one of the two merged image groups.

After updating the increment, the device further establishes a correspondence between group identifiers of the merged image group and the other image group and the increment calculated assuming that the merged image group and the other image group are merged; It can be recorded in a table of correspondence between increments and group identifiers maintained by the device.

When updating of the modularity and incremental sets is completed, the step S206 may be repeated until a unique image group is included in the set of images to be grouped.

When a unique image group is included in the set of images to be grouped, the device determines the maximum modularity among the updated modularities in each recorded iteration, and assigns an image combination method corresponding to the maximum modularity to the set of images to be grouped. It can be confirmed as the result of image grouping for

In this step, when determining the maximum modularity, the device may first determine a corresponding iteration corresponding to the maximum modularity. Next, the device performs a merging operation in the corresponding iteration, reads each group identifier included in the image group identifier set, and determines an image combining method corresponding to an image group corresponding to each group as the set of images to be grouped. It can be determined as an image grouping result for .

As can be seen from the above technical solution, on the one hand, the method merges the two image groups corresponding to the increment of the maximum modularity in each iteration step, so that the image combination method after each merge is the optimum that can be achieved in this merger. Assured that it is a combination method.

On the other hand, if only one image group remains after merging, the method tracks the updated modularity in each recorded iteration, determines the maximum modularity among the modularities, and determines the image combination method corresponding to the maximum modularity for the grouping. Since the result of image grouping for the set of images to be grouped can be determined, it is possible to ensure that the optimal image combination method shown in the process of merging image groups can be determined as the result of grouping the image for the set of images to be grouped. Through the division of groups, the connection between the images in each group is relatively strong and the connection between the images inside each group and the images in other groups is relatively weak, thereby improving the accuracy of image grouping, thereby enabling 3D reconstruction based on image grouping. Improve the robustness you perform.

In some embodiments according to the present invention, in the step of determining an image combination method corresponding to the maximum modularity as an image grouping result for the set of images to be grouped, in order to improve the stability of the image group, the maximum modularity is preset. It is determined whether a threshold is reached, and when the maximum modularity reaches the preset threshold, an image combination method corresponding to the maximum modularity may be determined as an image grouping result for the image set to be grouped. What will be described here is that reaching the preset threshold may refer to being greater than or equal to or greater than the preset threshold, which may be set according to actual circumstances.

In some embodiments, the preset threshold may be a threshold related to a stability evaluation standard of a grouping result set by a developer according to experience. The higher the threshold value, the higher the stability requirement for grouping results.

Therefore, when determining whether the maximum modularity reaches the preset threshold, if the modularity reaches the preset threshold, it is explained that the current image combination method has reached the preset stability criterion; An image combination method corresponding to modularity may be determined as an image grouping result for the image set to be grouped.

If the modularity does not reach the preset threshold, in this grouping process, it is explained that the image grouping method corresponding to the maximum modularity does not reach the preset safety standard, and at this time, each image included in the image set to be grouped can be made into an independent image group.

In some embodiments according to the present invention, in order to improve image grouping efficiency by reducing the amount of calculation of the image grouping method, the above-described steps S204 to S208 may be implemented by performing the following process.

In some embodiments, when performing step S204 to combine a plurality of image groups of the image set to be grouped by two, the device may combine image groups having a connection relationship by two.

을 계산한다(상기

를 계산하는 단계는 상술한 상황과 유사함). 설명할 것은, 상술한 상황에서, 연결관계가 없는 두 개의 이미지 그룹을 합병한 후, 상기 모듈성의 증분을 0으로 설정할 수 있다.Next, for each combination obtained, after merging the image group corresponding to the combination, the increment

Calculate (above

The step of calculating is similar to the situation described above). It should be noted that, in the above situation, after merging two image groups having no connection relationship, the increment of the modularity may be set to zero.

After merging two image groups having no connection through the above step, it is not necessary to calculate the increment of the modularity, so it is not difficult to see that the amount of calculation for calculating the increment is reduced and the image grouping efficiency can be increased.

When the calculation of the increment is completed, an increment set is generated based on the increment obtained by the calculation, and a correspondence relationship between the group identifier of each combined two image groups and each combined increment of the modularity is recorded. can

In this embodiment, the incremental set may be stored in matrix form. In practical application, the device may maintain one n*n matrix when generating the incremental set.

Here, each element of the n*n matrix may indicate the increment obtained by merging and calculating image groups respectively corresponding to the row and column belonging to the element. The n is the number of images included in the image set to be grouped. It will be explained that since the number of image pairs having a connection relationship is generally small, the matrix may be a sparse matrix for convenience of calculation.

는, 상기 요소의 소속 행 및 상기 열에 각각 대응하는 이미지 그룹 i와 이미지 그룹 j를 합병하는 것으로 가정한 후, 계산하여 획득된 상기 증분

를 가리키며, 여기서 상기 i, j는 0보다 크거나 같고 9보다 작거나 같은 정수이다. For example, in the image grouping scene shown in FIG. 3, the incremental set may be recorded through a 10*10 sparse matrix T. Referring to Fig. 4, Fig. 4 is an exemplary diagram of an incremental set illustrated in an embodiment of the present invention. As shown in FIG. 4, elements included in the sparse matrix T

is the increment obtained by calculating after assuming that image group i and image group j corresponding to the row and column belonging to the element are merged, respectively.

, where i and j are integers greater than or equal to 0 and less than or equal to 9.

In this embodiment, when maintaining an n*n matrix to record the increments, the device may first generate a corresponding binary tree (eg, a balanced binary tree) for each group of images. In addition, when each increment of the set of increments is calculated, two image groups assumed to be combined when calculating the increment are determined. Then, for a first image group of the two image groups assumed to be combined, the device first determines a binary tree corresponding to the first image group, and determines the increment and the group identifier of the two image groups. write to the node of the binary tree corresponding to the first image group; For a second image group of the two image groups assumed to be combined, a binary tree corresponding to the second image group is determined, and the increment and the group identifier of the two image groups correspond to the second image group. write to the node of the binary tree that

When recording is completed for all the increments, the largest increment among the increments recorded in each binary tree and the group identifier corresponding to the maximum increment are recorded at the root node of the binary tree corresponding to each image group.

을 산출하였다고 가정하면, 상기 기기는 현재 조합되는 것으로 가정된 것이 이미지 그룹 i 및 이미지 그룹 j인 것을 확정할 수 있다. 다음, 상기 기기는 이미지 그룹 i 및 이미지 그룹 j에 대응하는 이진 트리를 각각 확정할 수 있다. 상기 이진 트리를 확정한 후, 상기 기기는 상기 증분

이 상기 이진 트리의 루트 노드에 기록된 증분보다 큰지 여부를 확정할 수 있으며, 큰 경우, 상기 증분

, 및 그룹 식별자 i 및 그룹 식별자 j를 상기 이진 트리의 루트 노드에 기입할 수 있다. For example, currently one increment

Assuming that n has been calculated, the device can determine that image group i and image group j are currently assumed to be combined. Next, the device may determine binary trees corresponding to image group i and image group j, respectively. After determining the binary tree, the device increments the

is greater than the increment recorded at the root node of the binary tree; if greater, the increment

, and group identifier i and group identifier j can be written to the root node of the binary tree.

What will be described here is that, in the case of maintaining other nodes of the binary tree except for the root node, they can be maintained according to an arbitrary method, but are not limited thereto. Of course, for the convenience of node search, when maintaining other nodes of the binary tree except for the root node, among the two group identifiers corresponding to the increment, other group identifiers other than group identifier i may be maintained according to the size order. .

, 및 상기 증분

을 계산할 때 조합되는 것으로 가정된 두 개의 이미지 그룹의 그룹 식별자 i 및 j가 기록된다.Referring to Figure 5, Figure 5 is an exemplary diagram of a balanced binary tree illustrated in an embodiment of the present invention. As shown in Fig. 5, the balanced binary tree corresponds to the image group i. At the root node of the balanced binary tree, the largest increment among the increments corresponding to the group identifier i

, and the increment

The group identifiers i and j of the two image groups assumed to be combined when calculating ? are recorded.

To be explained, the balanced binary tree shown in Fig. 5 and the configuration of nodes of the balanced binary tree are merely exemplary. In actual application, the balanced binary tree and the configuration of the nodes may be set according to actual situations, but are not limited thereto.

For each of the combinations, it is assumed that the image group corresponding to the combination is merged, and the increment is calculated, and each increment obtained by the calculation is written into the corresponding binary tree, and then the generation of the increment set is completed. do.

In this embodiment, it is not difficult to see that by generating incremental sets in the above-described manner, at least the following effects are obtained.

On the one hand, we create a corresponding binary tree for each group of images, respectively, and actually write to the binary tree of each of the two image groups corresponding to the increment, for the same increment. At this time, after merging two image groups of each image group into one merged image group, and then recording the increment corresponding to the merged image group, there is no need to maintain a new binary tree for the merged image group. , The increment can be written to a binary tree of another image group corresponding to the merged image group, thereby reducing the amount of calculation when recording the increment corresponding to the merged image group, and improving image grouping efficiency. can be further improved.

On the other hand, what is written to the root node of the binary tree is that each image group corresponds to the maximum of the increment, so when determining the maximum increment of the increment set in the subsequent iteration step, the binary tree corresponding to each image group Since only the increment recorded in the root node needs to be compared, the efficiency of determining the maximum increment can be improved, and thus the image grouping efficiency can be further improved.

Thus, the preparation work for the above-described image grouping method has been completed.

Enter the next iteration step. In some embodiments, the device performs step S206 to determine a maximum increment in the increment set, merge two image groups corresponding to the maximum increment to obtain a merged image group, and Modularity and the step of updating the incremental set may be performed repeatedly until a unique image group is included in the set of images to be grouped.

In this embodiment, when determining the maximum increment included in the set of increments, use the maximum heap (the values corresponding to each parent node of the maximum heap are all greater than or equal to the values corresponding to the left and right child nodes) The maximum increment can be derived.

For example, upon ascertaining the maximum increment included in the set of increments, the device may push a pointer corresponding to the root node of each binary tree to the maximum heap maintained. Next, the device reads a point recorded in the root node of the maximum heap, and when the pointer is read, an increment recorded in a node indicated by the pointer may be determined as the maximum increment.

Due to the nature of the maximum heap, where the values corresponding to each parent node are greater than or equal to the values corresponding to the left and right child nodes, it is not difficult to see that the maximum value recorded in the root node of the maximum heap is the maximum value maintained in the maximum heap. there is. As can be seen from this, the point recorded in the root node of the maximum heap is read, and when the pointer is read, the increment recorded in the node pointed to by the pointer is determined, thereby determining the maximum increment among the set of increments. there is.

When the maximum increment included in the increment set is determined, the device may first determine two image groups corresponding to the maximum increment.

In this embodiment, the device may determine the two image groups by searching for a group identifier recorded in a node indicated by the pointer.

For example, it is assumed that the node indicated by the pointer is the root node of the binary tree shown in FIG. 5 . Continuing to refer to FIG. 5 , at this time, two image groups corresponding to the maximum increment may be determined as group i and group j.

After determining the two image groups corresponding to the maximum increment, the device merges the two images, and after merging, the merged image group obtained after the merge and the group identifier of the merged image group and the merged image group. Record the correspondence between the group identifiers of the two image groups.

In practical applications, after merging two image groups corresponding to the maximum increment among the set of images to be grouped, one data node may be allocated for the merged image group generated by the merge, and the node identifier of the data node (node index) may be a group identifier of the merged image group.

At this time, when recording a correspondence between a group identifier of the merged image group and a group identifier indicating the two image groups, the device may record the correspondence in a data node corresponding to the image merged group. .

For example, in the image group scene shown in FIG. 3, it is assumed that the two image groups determined corresponding to the maximum increment are 0 and 1. At this time, the device merges the image group 0 and the image group 1 to obtain a merged image group. Upon acquiring the merged image group, the device may generate a corresponding data node for the merged image group, and set a node identifier (node index) of the data node as a group identifier of the merged image group. Next, the device may write group identifier 0 and group identifier 1 to the data node.

Referring to Fig. 6, Fig. 6 is an exemplary view of image group merging according to an embodiment of the present invention.

As shown in Fig. 6, image group 0 and image group 1 are merged into merged image group a. Data node a corresponds to merged image group a. The node identifier (node index) of the data node a is the group identifier a of the merged image group, and group identifiers 0 and 1 are recorded in the data node a. It should be understood that the scheme described in this embodiment may record the group identifier of the merged image group and the group identifier of the merged two image groups through the data node, and thus the merged image group and the merged two image groups. Record the correspondence between

In the present invention, the device may, for each image group of the two merged image groups, determine a corresponding data node, and read image identifier information recorded in each determined data node. After obtaining the image identifier information recorded in each data node, the device may write the image identifier information into a data node corresponding to the merged image group, thereby creating a relationship between the merged image group and each included image. Record correspondence.

Referring continuously to FIG. 6 , image identifiers 0 and 1 are recorded in data node a corresponding to merged image group a shown in FIG. 6 . At this time, if the image identifier recorded in node a is read, it can be determined that image 0 and image 1 are included in image group a.

When the merging is completed, the device may update the modularity corresponding to the image set to be grouped.

를 획득한 후, 이번 반복에 대응하는 최대 증분을 더하여, 상기 그루핑될 이미지 집합에 대응하는 모듈성을 획득할 수 있다. 상기 모듈성을 획득한 후, 상기 모듈성을 이번 반복에서 생성된 합병 이미지 그룹에 대응하는 데이터 노드에 기록할 수 있다. 계속하여 도 6을 참조하면, 도 6에 도시된 상기 합병 이미지 그룹에 대응하는 데이터 노드 a에는 이번 반복에서 업데이트된 모듈성

이 기록된다.In this embodiment, the device first

After obtaining , modularity corresponding to the set of images to be grouped may be obtained by adding the maximum increment corresponding to this iteration. After obtaining the modularity, the modularity can be written to the data node corresponding to the merged image group created in this iteration. Continuing to refer to FIG. 6 , the data node a corresponding to the merged image group shown in FIG. 6 has the updated modularity in this iteration.

this is recorded

When the merging is complete, the device needs to update the incremental set.

Referring to FIG. 7 , FIG. 7 is a method flowchart of an incremental set updating method illustrated in an embodiment of the present invention.

As shown in FIG. 7, the incremental set updating method searches for a correspondence between group identifiers of image groups having a recorded connection relationship, and searches for one image group among two image groups corresponding to the maximum increment. Step S702 of determining another image group having a connection relationship;

Assuming that the different image groups are merged with the two images respectively, and the obtained increment is determined, and the result obtained by adding the determined increment is assumed, the merged image group is assumed to be merged with the other image group, and then Step S704 of incrementing the modularity;

and a step S706 of recording, in the incremental set, a correspondence between group identifiers of the merged image group and the other image groups, respectively, and the increments of the module obtained assuming that they are merged.

In this embodiment, when the device performs the step S702, the device searches for a correspondence including a group identifier of any one of the two image groups in a neighbor table maintained by the device, and the searched correspondence. It is possible to determine the other image group in the relationship.

The device may perform step S704 whenever one of the different image groups is determined. In this embodiment, when performing step S704, the device may calculate the modularity increment after assuming that the merged image group and the other image group are merged according to the above formula (5); A merged image group is a merged image group obtained by merging the two image groups.

For example, it is assumed that image group 0 and image group 1 are merged into merged image group a. Based on the adjacency table, it is found that image group i and image group 0 have a connection relationship. At this time, assuming that the merged image group a is merged with the image group i, the increment of the modularity should be calculated.

=

+

인 것을 알 수 있다.According to formula (5) above,

=

+

it can be seen that

와

는 각각 상기 이미지 그룹 0 및 상기 이미지 그룹 1에 대응하는 이진 트리를 검색하는 방식으로 획득된 것일 수 있다. 통상의 기술자가 알 수 있듯이, 본 실시예에서 연결관계가 없는 이미지 그룹을 합병하여 획득된 상기 증분을 0으로 설정하므로, 상기 다른 이미지 그룹 i와 이미지 그룹 1이 연결관계가 없는 경우, 상기

가 기록한 것은 실제 모듈성의 증분이 아니라 0이다. 이때, 정확한

를 획득하기 위해, 상기 공식 (3)에 따라 상기

를 계산할 수 있다. in one way,

and

may be obtained by searching binary trees corresponding to image group 0 and image group 1, respectively. As can be appreciated by those skilled in the art, since the increment obtained by merging image groups having no connection relationship in this embodiment is set to 0, when the other image group i and image group 1 have no connection relationship, the

What is recorded is a zero, not an increment of actual modularity. At this time, accurate

In order to obtain

can be calculated.

와

는 모두 상기 공식 (4)를 통해 계산될 수 있다. 구체적인 계산 과정은 전술한 내용을 참조할 수 있다.in another way,

and

can all be calculated through the above formula (4). A detailed calculation process may refer to the above description.

Assuming that the merged image group and other image groups are merged and after calculating the increment of the module, the device may continue to step S706.

In this embodiment, when performing the step S706, a binary tree corresponding to the other image group may be first determined, and the increment may be recorded in the determined binary tree.

이다.Assume that image group 0 and image group 1 are merged into merged image group a. Based on the adjacency table, it may be searched that image group i and image group 0 have a connection relationship. Assuming to merge the merged image group a with the image group i, the increment of modularity is

am.

를 상기 이진 트리에 삽입할 수 있다. 상기

를 상기 이진 트리에 삽입한 후, 상기 이진 트리는 자신이 기록한 증분 중 최대 증분을 다시 확정하고, 상기 최대 증분이 위치한 노드를 상기 이진 트리의 루트 노드에 이동시킬 수 있다. At this time, the device may first determine a binary tree corresponding to the image group i. After confirming the binary tree, the above

can be inserted into the binary tree. remind

After inserting into the binary tree, the binary tree may re-determine the maximum increment among the increments recorded by itself, and move the node where the maximum increment is located to the root node of the binary tree.

When the calculated increment has been recorded, the device may also delete the increment corresponding to the group identifier of any one of the two image groups.

In this embodiment, the device may delete data recorded at the root node from the binary tree corresponding to each of the two image groups.

After updating the binary tree corresponding to each image group, the device may push the root node point corresponding to each image group to the maximum heap to complete the next iteration.

This completes the update to the incremental set.

In this embodiment, in each iteration, the connection relationship between the merged image group generated in this iteration and the other image group is recorded in the adjacency table, and any image group of the two image groups merged in the adjacency table is recorded. connection can be deleted.

This completes the iteration once. The device may continue to perform the repeating step (S206) until a unique image group is included in the set of images to be grouped.

It should be understood that in each iteration, both image groups are merged to obtain one merged image group, so assuming that the image set to be grouped includes the first N (N is a natural number greater than 0) image groups. , it takes (N-1) merging iterations to merge N image groups into one image group.

In the above situation, a counter having a value of 1 may be set before starting the repetition step, and the value of the counter may be increased by 1 each time repetition is performed. At this time, the device may determine whether or not to end the repetition step (S206) by determining whether the value of the counter reaches N-1 before performing the repetition step each time.

When the image set to be grouped includes a unique image group, the device may perform step S208.

In this embodiment, the device starts from the merged image group recorded in the last iteration, and proceeds to step S2082 below in the reverse order for the merged image group recorded in each iteration, according to the order of occurrence of the repetition, i.e.,

It is determined whether the group identifier of the merged image group is included in the identifier set for checking the image combination method, and if the identifier set does not include the group identifier of the merged image group, the merged image group recorded in this iteration writing a group identifier of the group identifier to the identifier set;

determining whether an updated modularity corresponding to each recorded merged image group is a maximum modularity among updated modularities in each recorded iteration;

If the updated modularity corresponding to the merged image group is the maximum modularity among the updated modularity in each recording iteration, the image indicated by the image identifier corresponding to each group identifier included in the identifier set is converted into one image combination. and determining each obtained image combination as an image grouping result for the set of images to be grouped.

When the updated modularity corresponding to the merged image group is not the largest modularity among the updated modularities in each recording iteration, a group identifier of the merged image group recorded in the identifier set is selected as the merged image group corresponding to the merged image group. Replace with the group identifier of the previous two image groups.

Referring to FIG. 8 , FIG. 8 is an exemplary view of image group merging according to an embodiment of the present invention.

Nodes shown in FIG. 8 are data nodes corresponding to image groups, nodes 0-9 are data nodes corresponding to image groups that correspond one-to-one to each image included in the set to be grouped, and nodes a-i in each iteration. This is a data node corresponding to the created merged image group. 8 shows a correspondence between two image groups and a merged image group created by merging the two image groups.

It should be understood that each data node generally includes a current time stamp when it is created, so determining the time stamp information included in each data node can determine the order of creation of each data node. In FIG. 8, it is assumed that the index of each data node indicates the order of creation of each data node. For example, the creation time of node a is faster than node b, and the creation time of node g is faster than node i.

When the device performs step S2082 for the merged image group recorded last, since the identifier set for confirming the image combination method is initially null, the image recorded in node i and corresponding to node i A group identifier of a group may be written in the identifier set. At this time, the identifier set includes group identifier i.

Next, the device may determine whether the modularity recorded in the node i is the maximum modularity among the modularities updated in each recorded iteration.

Hereinafter, a method for determining whether the modularity recorded in the node i is the maximum modularity will be introduced.

A person skilled in the art will understand that in a real situation, when modularity is the greatest, there are three ways of combining images included in an image set to be grouped. That is, in Method 1, all images are combined into one group. In scheme 2, each image is a group respectively. In scheme 3, several images are combined into one group.

Regarding the first method, in practice, the current image combination form tends to be stable whenever merged, and the corresponding modularity gradually increases, and at this time, the change trend of modularity monotonically increases with each iteration.

Regarding the second method, in practice, the current image combination format tends to be unstable every time it is merged, and the corresponding modularity becomes smaller and smaller.

Regarding the third scheme, actually starting from the first image combining scheme until the image combining scheme with the greatest modularity appears and ends, the current image combining scheme tends to be stable every time it is merged. After the image combination method with the greatest modularity appears, the current image combination form tends to be unstable each time it is merged. Decrease.

It will be appreciated by those of ordinary skill in the art that for the above three schemes, starting from the last iteration, in reverse order according to the order in which iterations occur, at each iteration the modularity updated in this iteration corresponds to the previous iteration that corresponds to this iteration. It is determined whether updated modularity has been reached in In the above situation, if it is confirmed that the modularity updated in this iteration has reached the modularity updated in the previous iteration corresponding to this iteration, it is described that the modularity updated in this iteration is the maximum modularity. In this case, if modularity that satisfies the determination condition is not found after performing the determining step for each repetition, it will be described that the maximum modularity is the modularity corresponding to the image combination method before each repetition.

를 상기 노드 h에 기록된 모듈성

와 비교할 수 있다. 상기

가 상기

에 도달한 경우, 상기 노드 i를 상기 최대 모듈성으로 확정한다. Therefore, when the device determines whether the modularity recorded in the node i is the maximum modularity among the modularities updated in each recorded iteration, the modularity recorded in the node i

Modularity written to the node h

can be compared with remind

remind

is reached, the node i is determined as the maximum modularity.

When the device determines whether the modularity recorded in the node i is the maximum modularity among the modularities updated in each recorded iteration, the modularity recorded in the node i is the largest among the modularities updated in each recorded iteration. If modular, at this time, the device sets an image indicated by an image identifier corresponding to each group identifier included in the identifier set as one image combination, and converts each obtained image combination into an image grouping result of the image set to be grouped. can be confirmed

When the modularity recorded in the node i is not the maximum modularity among the modularities updated in each recorded iteration, a group identifier of the recorded merged image group recorded in the identifier set is selected as a group of the recorded merged image group. Replace with a group identifier indicating the two image groups corresponding to the identifier.

Continuing to refer to FIG. 8, it is assumed that the modularity recorded at node f in FIG. 8 is the maximum modularity. In the above situation, when performing step S2082 for node i, the device may determine that the modularity recorded in node i is not the maximum modularity. In this case, the device may replace group identifier i of the identifier set with group identifiers g and h of two image groups merged into image group i. At this time, the identifier set includes group identifiers g and h.

After performing the above-described steps for node i, the above-described step S2082 may be continuously performed for node h. Since the modularity recorded in node h is not the maximum modularity, the device may replace group identifier h in the identifier set with group identifiers e and f of two image groups merged into image group h. At this time, the identifier set includes group identifiers g, e, and f.

After performing the above-described steps for node h, the above-described step S2082 may be continuously performed for node g. Since the modularity recorded in node g is not the maximum modularity, the device may replace group identifier g in the identifier set with group identifiers c and 4 of two image groups merged into image group g. At this time, the identifier set includes group identifiers c, 4, e, and f.

After performing the above-described steps for node g, the above-described step S2082 may be continuously performed for node f. Since the modularity recorded at node f is the maximum modularity, the device sets the image grouping result of the set of images to be grouped, image 0-3 as one group, image 4 as one group independently, and images 5-6 as one group. It can be determined that the group of , images 7-9 are one group.

As can be seen through the method described in this embodiment, the method described in this embodiment can obtain the following effects based on rationally grouping images to be grouped.

On the one hand, we create a corresponding binary tree for each group of images, respectively, and actually write to the binary tree of each of the two image groups corresponding to the increment, for the same increment. At this time, after merging two image groups of each image group into one merged image group, and then recording the increment corresponding to the merged image group, there is no need to maintain a new binary tree for the merged image group. , The increment can be written to a binary tree of another image group corresponding to the merged image group, thereby reducing the amount of calculation when recording the increment corresponding to the merged image group, and improving image grouping efficiency. can be further improved.

On the other hand, what is written to the root node of the binary tree is that each image group corresponds to the maximum of the increment, so when determining the maximum increment of the increment set in the subsequent iteration step, the binary tree corresponding to each image group Since only the increment recorded in the root node needs to be compared, the efficiency of determining the maximum increment can be improved, and thus the image grouping efficiency can be further improved.

On the other hand, since the root node point of the binary tree is pushed to the maximum heap and the maximum increment is derived by the maximum heap, the efficiency of determining the maximum increment among the set of increments can be improved, and the image grouping efficiency can be further improved. can improve

In some embodiments according to the present invention, after grouping images from a set of images to be grouped, an image group with a small number of included images may appear. At this time, if 3D reconstruction is performed on an image group with a small number of included images, poor reconstruction results or 3D reconstruction may not be performed because the number of images included in the image group is small.

In a related art, several image groups whose number of included images is less than a minimum threshold are selected and merged to obtain a merged image group, thereby increasing the number of images included in the acquired image group.

However, since the related art does not consider the connection relationship between each image group to be fused when image groups are fused, the connection relationship between each image in the image group obtained by fusion is not very strong, and the image group obtained by fusion is not very strong. It is easy to find that the robustness of the reconstruction result performing 3D reconstruction on .

In the technical scheme presented in this embodiment, when fusing an image group with a small number of images, first fuses two image groups having the strongest connection relationship, performs fusion on the image group with a small number of images, and simultaneously obtains Since the connection relationship between each image in the image group is relatively strong, it is possible to ensure that a reconstruction result obtained by performing 3D reconstruction on the image group obtained by fusion is robust.

Before introducing the detailed steps of this embodiment, we first introduce the index-linked weights that evaluate the strength of the linking relationship between image groups.

In some embodiments, the connection weight may include an index for quantifying a connection relationship between two image groups.

When calculating the connection weight between two image groups, first, each image included in one image group (eg, the first image group) and the other image group (eg, the second image group) The similarity between each included image can be calculated. Next, it is determined whether each calculated similarity reaches a preset similarity threshold, and two images corresponding to similarities reaching the preset similarity threshold are determined as matching images. The number of matching images between the two image groups is counted, and the number of images is used as a connection weight between the two image groups.

As can be seen from this, the connection weight can quantify the connection relationship between two image groups.

Next, the detailed steps of this embodiment are introduced.

First, in each image group obtained, the device determines an image group whose number of included images is smaller than a first preset number threshold, and determines the determined image group as an image group to be merged, wherein the first preset threshold is Indicates the minimum number of images in each group. Next, among the image groups, other groups excluding the image group to be merged are determined as the remaining groups.

In some embodiments, the first preset threshold may be the minimum number of images in each group. The minimum number of images may be a preset minimum number of images that may be included in one image group.

In this step, the device may first assign an image group identifier set to be merged to the image group to be merged. Next, the device determines the number of images included in each image group in turn, puts group identifiers of image groups whose number of images is less than the first preset number threshold into the merging image group identifier set, and sets the image groups to be merged. You can complete the step of confirming.

Here, the step of determining the remaining groups may refer to the step of determining image groups to be merged.

When the image group to be merged is determined, the device may calculate a connection weight between each two image groups among the groups to be merged and configure a connection weight set, wherein the connection weight is the number of matching images between the two image groups. instruct

In this step, the device may generate a binary tree (eg, a balanced binary tree) corresponding to each image group on a one-to-one basis for each image group among the to-be-fused groups. Next, the device may calculate a connection weight between each two image groups of the to-be-fused group, and record the connection weight and a group identifier of the two image groups in a node of the binary tree. In the root node of the tree, a maximum connection weight among connection weights corresponding to each image group and a group identifier corresponding to the maximum connection weight are recorded.

At this time, generation of the connection weight set is completed.

Next, the device may merge the image group to be merged according to the connection weight set.

In some embodiments, when fusing the image group to be fused according to the connection weight set, the device continues until no connection weight is included in the connection weight set,

The step of determining the maximum connection weight among the connection weight set may be repeatedly performed.

In this step, when determining the maximum linkage weight, the root node pointer of the binary tree corresponding to each image group may be pushed to the maximum heap for deriving the maximum linkage weight. Next, a pointer recorded in the root node of the maximum heap may be read, and a connection weight recorded in a binary tree node indicated by the recorded pointer may be determined as the maximum connection weight.

After determining the maximum concatenation weight, the device may determine two image groups corresponding to the maximum concatenation weight through the group identifier recorded in the binary tree node.

After determining the two image groups corresponding to the maximum concatenation weight, the device may determine whether the number of images in one image group having a greater number of images in them reaches a second preset number threshold; The second preset number threshold indicates a maximum number of images in each image group.

When the second preset number threshold is reached, this iteration ends.

When the second preset number threshold is not reached, the two image groups are merged to obtain a merged image group and the concatenation weight set is updated.

The second preset number threshold specifically indicates the maximum number of images in each image group. The maximum number of images is the maximum number of images that can be included in each image group.

In this step, if a merged image group is obtained by merging the two image groups, the device generates one data node for the merged image group, and a data node corresponding to each of the two merged image groups. The recorded image identifier may be written to the corresponding data node. Next, the device may delete the group identifiers of the two merged image groups from the set of image group identifiers to be merged, and add the group identifiers of the merged image group.

When the merging is completed, the device may update the connection weight set.

In this step, the device may first delete data recorded in the binary tree corresponding to each of the two merged image groups.

Next, the device calculates a connection weight between the merged image group and another group among the groups to be merged, and converts the calculated connection weight and a group identifier of the merged image group and the other group into a binary number corresponding to the other group. You can write to the tree. At this time, the device also deletes a node recording a connection weight between the other group and the merged two image groups from the binary tree corresponding to the other group.

It should be understood that, in order to improve image fusion efficiency, when calculating the connection weight between the merged image group and other groups among the to-be-fused groups, the merged image group and other image groups having a connection relationship with the merged image group are calculated. Only connection weights can be calculated.

Thus, the device has completed updating the connection weight set.

Next, the device may repeatedly perform the above step until no connection weight is included in the connection weight set.

In this embodiment, when the number of images in one of the two image groups corresponding to the maximum concatenated weight is greater than the number of included images reaches the second preset number threshold, the device performs an operation to stop this iteration. In addition to performing, a binary tree corresponding to an image group having a larger number may be deleted, and a node recording a connection weight between the two image groups may be deleted from a binary tree corresponding to an image group having a smaller number. there is.

What will be described here is that when two image groups are fused or the number of images included in any one image group is greater than the second preset threshold, the corresponding binary tree is deleted, so that the binary tree of the image group corresponding to the maximum concatenation weight is If null, it explains that the image group cannot be fused.

As can be seen, in order to accelerate the image group fusion efficiency, after determining the maximum concatenation weight in each iteration, the device also determines that the binary tree corresponding to any one of the two image groups corresponding to the maximum concatenation weight is null. It can be determined whether or not In the case of null, this iteration may be terminated, and a node recording a connection weight between the two image groups may be deleted from a non-null binary tree, thereby reducing invalid repetitions and improving image grouping fusion efficiency.

When the connection weight set does not include the connection weight, the device may set the image group included in the group to be merged and the image group among the remaining groups as the final image group for the set of images to be grouped.

In some embodiments, the device reads each group identifier included in the set of image group identifiers to be merged, and for each group identifier, an image indicated by an image identifier corresponding to each group identifier is divided into one image group. can be divided Next, the device may output each image group acquired by division and the image group among the remaining groups as a final image group for the image set to be grouped.

In this embodiment, since two image groups corresponding to the maximum connection weight can be fused in each iteration, when fusing an image group with a small number of images, first, the two image groups having the strongest connection relationship are fused, At the same time as performing fusion on a small number of image groups, the connection between each image in the image group obtained by fusion is relatively strong, so that the robustness of the reconstruction result of performing 3D reconstruction on the image group obtained by fusion is good. can be guaranteed

Based on the inventive concept of the foregoing embodiment, the embodiment of the present invention further presents an image grouping method during three-dimensional reconstruction. Referring to FIG. 9 , FIG. 9 is a method flowchart of an image grouping method during 3D reconstruction exemplified in an embodiment of the present invention. As shown in Fig. 9, the method includes the following steps.

In step S902, when each image among the image sets to be grouped is regarded as one image group, modularity corresponding to the image set to be grouped is calculated.

In step S904, when a plurality of image groups included in the image set to be grouped are combined in pairs, an increment of modularity corresponding to each combination is calculated to obtain an incremental set.

In step S906, the maximum increment included in the set of increments is determined, further determining whether the maximum increment has reached 0, and if the maximum increment has reached 0, the recorded two image groups corresponding to the maximum increment are selected. The steps of merging to obtain a merged image group and updating the incremental set are repeatedly performed until a unique image group is included in the to-be-grouped image set.

In step S908, in each iteration, if the maximum increment does not reach 0, the above iteration steps are terminated, and the merged image combination method in the previous iteration of this iteration is used as the final image for the set of images to be grouped. As a result of grouping.

In step S910, when the image set to be grouped includes a unique image group, each image included in the image set to be grouped is divided into one image group.

In the above embodiment, after the merging operation in each iteration process, based on the analysis of the change trend of modularity corresponding to the current image combination format, when the modularity reaches the maximum value, the modularity monotonically decreases when iteration is performed thereafter.

Therefore, before performing the merging operation each time, it is judged whether this merging will result in a reduction in modularity, and if it does not result in a reduction in modularity, the merging operation is performed; otherwise, the merging operation is terminated, and the previous The image combination method after repetition may be used as a final image grouping result for the set of images to be grouped.

In some embodiments, it may be determined whether the maximum modularity increment has reached 0 in the current iteration, and if it has reached 0, this demonstrates that the modularity is at least not reduced after this merger. However, if we find that the maximum modularity increment in the current iteration is negative, we explain that after this merging, the modularity is reduced, and then every time a merging operation is performed, the modularity continues to be reduced.

As can be seen from the above technical solution, on the one hand, the method merges the two image groups corresponding to the increment of the maximum modularity in each iteration step, so that the image combination method after each merge is the optimum that can be achieved in this merger. Make sure it's a combination.

On the other hand, before performing the merging operation each time, it is judged whether this merging will result in a reduction in modularity, and if it does not result in a reduction in modularity, the merging operation is performed; otherwise, the merging is terminated; The image combination method after the previous iteration may be used as the final image grouping result for the set of images to be grouped. Therefore, in the iterative process, the corresponding image grouping method when the modularity is the largest is determined as the image grouping result for the image set to be grouped, so that the connection relationship between the images inside each divided group is relatively strong and different from the image inside each group. By ensuring that the connection relationship between groups is relatively weak, the accuracy of image grouping is improved, and furthermore, the robustness of performing 3D reconstruction based on image grouping is improved.

In some embodiments, referring to FIG. 10 , FIG. 10 is a flow diagram illustrating application of the image grouping method during 3D reconstruction exemplified in an embodiment of the present invention to a real scene. 10 shows three processes (processes of constructing a scene diagram, generating an image group, and merging image groups) of the image group method in this embodiment, which will be described in detail below.

Regarding the process of constructing the scene map, the device may implement feature extraction and feature matching on a plurality of true scene images collected from the true scene. Specifically, in the feature extraction process, a plurality of feature points are extracted from each true scene image. Exemplarily, the harris corner is extracted as a feature point, a gray histogram of a preset area around each feature point is statistic, a feature vector describing the feature point is generated, and exemplarily, a 128-dimensional may be a feature vector of Furthermore, a corresponding feature vector of each feature point in each true scene image is obtained. In the feature matching process, a matching relationship between two true scene images may be calculated according to a set of feature vectors corresponding to each true scene image. Specifically, for one feature point of one true scene image, finding a feature point similar to the corresponding feature point among other true scene images to form a pair of matching points. The degree of similarity between feature points can be evaluated by the Euclidean distance between feature vectors corresponding to each. If the number of matching point pairs matched between the two true scene images is greater than a preset matching number threshold, the two true scene images are determined as matching images.

In some embodiments, a scale-invariant feature transform (SIFT) feature extraction method may be applied to feature extraction, which is specifically selected according to actual circumstances, and embodiments of the present invention are not limited thereto.

In some embodiments, when performing feature matching, non-replacement suppression and bi-directional matching consistency may be applied to eliminate false matches. Alternatively, for some scenes with weak texture or large visual variance, when the number of matched matching point pairs is less than the preset minimum matching number threshold, the basis between the two corresponding true scene images is obtained through previously matched matching point pairs. A matrix can be obtained, and by providing epipolar geometric constraints between images through a default matrix, the search range of similar features is limited to epipolar lines, and matching is greatly improved.

The image group generation process is repeated through image grouping, modularity calculation, and merging based on modularity until a unique image group is obtained through the method of steps S202 to S208 described above or the method of steps S902 to S910 described above. , and the image grouping method corresponding to the maximum modularity obtained in step S208 is a rudimentary result of image grouping, or each image included in the set to be grouped obtained in step S209 is grouped into one image group. By segmentation, it is possible to obtain rudimentary results of image grouping.

Here, in the process of performing step S204 or step S904, the connection relationship between each region is obtained from the generated scene map, and when image grouping is performed, only image groups having a connection relationship are combined to reduce the amount of computation of modularity increments. and improve the efficiency of generating image groups.

In the image group fusion process, first, in the initial result of image grouping, according to the number of images in each image group, an image group to be fused with a small number of images is determined, and the connection weight between each two image groups to be fused is set as a connection weight can be computed as a set. In addition, by using a binary tree corresponding to each image group to be merged one-to-one, the number of images among two image groups to be merged corresponding to the maximum connection weight is greater than the number of images included in the second preset image group. If the number threshold is not reached, the two to-be-fused image groups corresponding to the maximum connection weight are fused, the binary tree and the connection weight are updated according to the fusion result, and the connection weight is not included in the connection weight set. At this time, the image group obtained by merging the image group to be merged and the remaining groups excluding the image group to be merged are used as the final image grouping result.

As can be seen from the above technical scheme, the process of generating image groups based on modularity can implement strong connectivity of images in the same group and weak connectivity between different groups, thereby improving the accuracy of image grouping and , the image reconstruction success rate and overall reconstruction efficiency within the same group are higher, and improve the robustness of three-dimensional reconstruction. In addition, in the image group fusion process, when merging image groups with a small number of images, first fusion of two image groups having the strongest connection relationship is performed, and accordingly, image groups with a small number of images are fused to ensure uniformity of the image groups. At the same time, it is possible to ensure that the connection relationship between each image in the final image group obtained by fusion is relatively strong, thereby ensuring robustness of the reconstruction result of performing 3D reconstruction on the final image group obtained by fusion.

Embodiments of the present invention further provide an apparatus for grouping images during 3D reconstruction, and referring to FIG. 11 , FIG. 11 is a configuration diagram of an apparatus for grouping images during 3D reconstruction exemplified by the present invention. As shown in Figure 11, the device 100

a first calculator 101 configured to calculate modularity corresponding to the image set to be grouped by taking each image of the image set to be grouped as one image group;

a first generation unit 102 configured to combine a plurality of image groups included in the image set to be grouped by two, and to obtain an incremental set by calculating each increment of modularity corresponding to each combination;

determining a maximum increment in the incremental set, merging two image groups corresponding to the maximum increment to obtain a merged image group, and updating the modularity corresponding to the image set to be grouped and the incremental set; a first merging unit 103 configured to perform iteratively until a unique image group is included in the image set;

When a unique image group is included in the set of images to be grouped, the maximum modularity among the updated modularities is determined in each recorded iteration, and an image combination method corresponding to the maximum modularity is selected as image grouping for the set of images to be grouped. and a first group determination unit 104 configured to determine a result.

In the illustrated embodiment, the first group determination unit 104 also,

determine whether the maximum modularity has reached a preset threshold;

when the maximum modularity reaches the preset threshold, determining an image combining method corresponding to the maximum modularity as an image grouping result for the image set to be grouped;

When the maximum modularity does not reach the preset threshold, each image included in the set of images to be grouped is configured to be an independent image group.

In one illustrated embodiment, the first generator 102 also,

Among the plurality of image groups, image groups having a connection relationship are combined by two, and a correspondence relationship between group identifiers of the image groups having a connection relationship is recorded.

The two image groups having a connection relationship indicate that a similarity between images of a first image group and a second image group of the two image groups reaches a preset similarity threshold.

In an illustrated embodiment, the incremental set includes a correspondence between a group identifier of two image groups assumed to be combined and the incremental obtained by calculating and assuming merging the two image groups;

The first merger 103 also,

Searching for a correspondence between group identifiers of image groups having a recorded connection relationship, determining another image group having a connection relationship with one of the two image groups corresponding to the maximum increment;

Assuming that the different image groups are merged with the two images respectively, and the obtained increment is determined, and the result obtained by adding the determined increment is assumed, the merged image group is assumed to be merged with the other image group, and then as an increment of the modularity;

and record a correspondence between a group identifier of each of the merged image group and the other image group, and the increment of the module obtained assuming that both are merged, in the increment set.

In one illustrated embodiment, the device 100,

and a first recording unit, configured to record a correspondence between the group identifier of the merged image group and the group identifier of the other image group after updating the incremental set in each iteration process.

In one illustrated embodiment, the device 100,

A second recording unit configured to record a merged image group obtained by merging and a correspondence between a group identifier of the merged image group and a group identifier of the merged two image groups in each repetition process.

In one illustrated embodiment, the merge image group corresponds to an updated modularity in an iteration of creating the merge image group; a group identifier of the image group corresponds to a corresponding image identifier indicating each image included in the image group;

The first group determination unit 104 also,

Starting from the group of merged images recorded in the last iteration, and in reverse order for the group of merged images recorded in each iteration, according to the order of occurrence of the iterations,

It is determined whether the group identifier of the merged image group is included in the identifier set for checking the image combination method, and if the identifier set does not include the group identifier of the merged image group, the group identifier of the merged image group is determined. writing to the set of identifiers;

determining whether an updated modularity corresponding to the merged image group is a maximum modularity among updated modularities in each recorded iteration;

If the updated modularity corresponding to the merged image group is the maximum modularity among the updated modularity in each recording iteration, the image indicated by the image identifier corresponding to each group identifier included in the identifier set is converted into one image combination. and determining each obtained image combination as an image grouping result for the set of images to be grouped.

In one illustrated embodiment, the device 100,

When the updated modularity corresponding to the merged image group is not the largest modularity among the updated modularities in each recording iteration, a group identifier of the merged image group recorded in the identifier set is selected as the merged image group corresponding to the merged image group. and a replacement unit configured to replace the group identifiers of the previous two image groups.

In one illustrated embodiment, the replacement unit also,

determine whether the modularity updated in this iteration has reached the modularity updated in the previous iteration corresponding to this iteration;

If reached, it is configured to determine the modularity updated in this iteration as the maximum modularity.

In one illustrated embodiment, the first generator 102 also,

For each of the above increments obtained by calculating and assuming a combination,

Assume combining is for each group of two images hypothesized to combine, determine a binary tree corresponding thereto, and set the increment, and the group identifier of the two images hypothesized to combine, to the determined binary tree. configured to perform the step of writing to the tree;

At the root node of the binary tree, the maximum increment among all increments recorded in the binary tree and a group identifier corresponding to the maximum increment are recorded.

In one illustrated embodiment, the first generator 102 also,

push the root node pointer of the binary tree corresponding to each image group to the maximum heap to determine the maximum increment;

determine a pointer recorded in a root node of the maximum heap;

and determine the increment recorded at the node pointed by the recorded pointer as the largest increment included in the set of increments.

In one illustrated embodiment, the device 100,

an image group to be merged determining unit, configured to determine, in each acquired image group, an image group whose number of included images is smaller than a first preset number threshold, and determine the determined image group as an image group to be merged - the first dictionary The set threshold indicates the minimum number of images in each group;

a remaining group determining unit configured to determine remaining groups other than the to-be-fused image group among the image groups as remaining groups;

a connection weight calculation unit configured to calculate a connection weight between each two image groups among the groups to be fused and configure a connection weight set, wherein the connection weight indicates the number of images matched between the two image groups;

A group fusion unit for merging the image groups to be fused according to the set of connection weights is further included.

In one illustrated embodiment, the group fusion unit also,

Until no connection weight is included in the connection weight set,

determining a maximum connection weight among the set of connection weights;

determining whether the number of images in one of the two image groups corresponding to the maximum concatenation weight has reached a second preset number threshold, wherein the second preset number threshold is set for each image group - Indicates the maximum number of images among ;

terminating this iteration if the second preset number threshold is reached;

merging the two image groups to obtain a merged image group and updating the concatenation weight set when the second preset number threshold is not reached;

When no connection weight is included in the connection weight set, to repeatedly perform the step of making an image group included in the group to be fused and an image group among the remaining groups as a final image group for the set of images to be grouped. It consists of

In one illustrated embodiment, the connection weight calculation unit,

for each image group of the to-be-fused group, a binary tree corresponding to each image group on a one-to-one basis is generated;

Calculate a connection weight between each two image groups of the group to be fused, and record the connection weight and a group identifier of the two image groups in a node of the binary tree, wherein each root node of the binary tree Among the linkage weights corresponding to the image group, a maximum linkage weight and a group identifier corresponding to the maximum linkage weight are recorded.

In one illustrated embodiment, the group fusion unit also,

push a root node pointer of the binary tree corresponding to each image group to a maximum heap for deriving the maximum connection weight;

Read a pointer recorded in a root node of the maximum heap, and determine a connection weight recorded in a binary tree node indicated by the recorded pointer as the maximum connection weight.

In one illustrated embodiment, the device 100,

Determine whether a binary tree corresponding to any one of the two image groups corresponding to the maximum connection weight is null;

If not null, continue with subsequent steps;

If null, further comprising a data determination unit, configured to end the current iteration.

In one illustrated embodiment, the group fusion unit also,

delete a binary tree corresponding to each of the two merged image groups;

calculate connection weights between the merged image group and other groups among the groups to be merged;

writing the calculated connection weight and group identifiers of the merged image group and the other group into a binary tree corresponding to the other group;

A node recording a connection weight between the other group and the merged two image groups is deleted from the binary tree corresponding to the other group.

Embodiments of the present invention further provide an apparatus for grouping images during 3D reconstruction, and referring to FIG. 12 , FIG. 12 is a configuration diagram of an apparatus for grouping images during 3D reconstruction exemplified by an embodiment of the present invention. As shown in FIG. 12, the device 110,

a second calculator 111 configured to calculate modularity corresponding to the image set to be grouped by taking each image among the set of images to be grouped as one image group;

a second generation unit 112 configured to obtain an incremental set by calculating an increment of modularity corresponding to each combination when combining the plurality of image groups included in the image set to be grouped by two;

Determine the maximum increment included in the set of increments, further determine whether the maximum increment reaches 0, and merge the two image groups recorded corresponding to the maximum increment if the maximum increment reaches 0. a second merging unit (113) configured to repeatedly perform the step of acquiring a merged image group and updating the incremental set until a unique image group is included in the to-be-grouped image set;

In each iteration, if the maximum increment does not reach 0, the above iteration steps are terminated, and the image combination method merged in the previous iteration of this iteration is taken as the final image grouping result for the image set to be grouped;

and a second group determination unit 114 configured to divide each image included in the image set to be grouped into one image group when the set of images to be grouped includes a unique image group.

It should be noted that the description of the device embodiments above is similar to that of the foregoing method embodiments, and has similar beneficial effects to those of the method embodiments. Technical details not disclosed in the device embodiments according to the present invention may be understood by referring to the description of the method embodiments according to the present invention.

The device embodiments exemplified by the present invention can be applied to electronic devices. Device embodiments may be implemented through software, hardware, or a combination of software and hardware. Taking a software implementation as an example, as a device in a logical sense, it is formed by reading and executing a computer program command corresponding to a non-volatile memory into a RAM through a processor of an electronic device located there. In terms of hardware, as shown in Fig. 13, Fig. 13 is a hardware configuration diagram of an electronic device exemplified by an embodiment of the present invention. Except for the processor, RAM, network interface, and non-volatile memory shown in FIG. 13, the electronic device in which the device is located may generally further include other hardware according to the actual function of the electronic device. don't explain

Referring to the electronic device shown in FIG. 13 , the electronic device includes a processor;

A recording device for recording instructions executable by the processor,

The processor executes the executable instructions to implement the image grouping method during 3D reconstruction exemplified in any one of the embodiments described above.

An embodiment of the present invention submits a computer readable storage medium on which a computer program for performing the image grouping method during 3D reconstruction exemplified in any one of the above embodiments is recorded.

Embodiments of the present invention further provide a computer program product including computer readable code, and the computer readable code, when executed in an electronic device, when executed by a processor in the electronic device, is exemplified in any one of the above embodiments. We implement an image grouping method during 3D reconstruction.

As will be appreciated by those skilled in the art, one or more embodiments according to the present invention may be provided as a method, system or computer program product. Therefore, one or more embodiments according to the present invention may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment of the aspect of combining software and hardware. In addition, one or a plurality of embodiments according to the present invention include one or a plurality of computer usable recording media (including but not limited to magnetic disk recording devices, CD-ROMs, optical recording devices, etc.) containing computer usable program codes. The format of the computer program product implemented in (not specified) can be applied.

'And ... described in the present invention. 'At least one of' denotes including at least one of both, for example, 'at least one of A and B' includes three schemes: A, B, and 'A and B'.

Each embodiment of the present invention is described in a progressive manner, the same or similar parts between each embodiment can be referred to each other, and the focus of each embodiment is the difference from other embodiments. In particular, in the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant part may refer to the partial description of the method embodiment.

In the above, specific embodiments according to the present invention are described. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order than the circumstances and still achieve desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require a specific order or sequence shown in order to achieve a desired result. In some implementations, multitasking processing and parallel processing may also be possible or advantageous.

Embodiments of the themes and functional operations described in the present invention may be digital electronic circuits, computer software or firmware embodied in tangible form, computer hardware including the structures and structural equivalents disclosed herein, or combinations of one or a plurality thereof. can be implemented as Embodiments of the themes described in the present invention are one or a plurality of computer programs, that is, one or a plurality of computer program instructions coded in a tangible non-temporary program carrier and executed by a data processing device or controlling the operation of the data processing device. It can be implemented as a module. Alternatively or additionally, program instructions may be coded into an artificially generated radio signal (eg, an electrical, optical or electromagnetic signal generated by a machine), which signal is to be executed by a data processing device. , the information can be coded and transmitted to a suitable receiver device. The computer recording medium may be a machine-readable recording device, a machine-readable recording board, a random or serial access recording device, or a combination of one or more of these.

The processing and logic flow described in the present invention can be performed by one or more programmable computers running one or more computer programs, operating according to input data to generate outputs, and performing corresponding functions. The processing and logic flow may be performed with dedicated logic circuitry - for example, an FPGA (Field Programmable Gate Array) or an ASIC (Special Purpose Integrated Circuit), and the apparatus may be implemented with dedicated logic circuitry.

A computer suitable for carrying out a computer program includes, for example, a general purpose and/or special purpose microprocessor, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only recording devices and/or random access recording devices. The basic components of a computer include a central processing unit for executing or executing instructions, and one or a plurality of recording units for recording instructions and data. Generally, a computer includes one or a plurality of mass storage devices (eg, disks, magnetic disks, optical disks, etc.) for recording data, or a computer is operatively coupled to the mass storage devices to send and receive data. Or, it could be both cases. However, the computer is not necessarily equipped with such a device. Computers may also be embedded in other devices such as cell phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, GPS receivers, or portable recording devices such as Universal Serial Bus (USB) flash drives. can be, these are just a few examples.

Computer-readable storage media suitable for recording computer program instructions and data include all types of volatile storage media and non-volatile storage media, and non-volatile storage media may include non-volatile memories, media, and recording devices. Examples include semiconductor recording devices (eg, EPROM, EEPROM, and flash devices), disks (eg, internal hard disks or mobile disks), magnetic disks, and CD ROM and DVD-ROM disks. The processor and recording device may be supplemented by or incorporated into dedicated logic circuitry.

Although this invention contains many specific implementation details, these are not to be construed as limiting the scope of any disclosed or claimed, but primarily set forth features of the specific embodiment specifically disclosed. Within this invention, some features described in multiple contexts may also be implemented in combination in a single context. On the other hand, various features that are described in a single context may also be implemented in multiple contexts separately or in any suitable subcombination. Further, while features may function in some combinations as above or even be originally claimed as such, one or a plurality of features in a claimed combination may in some cases be eliminated from that combination, and the claimed combination may be a subcombination or subcombination. transformation can be indicated.

Similarly, although operations may be described in a specific order in the accompanying drawings, it should not be understood as requiring that such operations be performed in the specific order shown or performed sequentially, or that all illustrated operations be performed in order to achieve a desired result. Can not be done. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of each system module and component among the above-mentioned situations should not be understood as such separation in all situations. It should also be understood that the described program components and systems are generally aggregated together into a single software product or may be packaged into multiple software products.

Thus, specific embodiments of the theme have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desired results. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve desired results. In some implementations, multitasking and parallel processing may be advantageous.

The above is only a preferred embodiment of one or a plurality of embodiments according to the present invention, but does not limit the one or plurality of embodiments according to the present invention, and any progress made within the spirit and principles of one or a plurality of embodiments according to the present invention Modifications, equivalent replacements, and improvements should all fall within the protection scope of one or more embodiments according to the present invention.

In an embodiment of the present invention, on the one hand, in each iteration step, the two image groups corresponding to the increment of maximum modularity are merged, ensuring that the image combination method after each merge is the optimal combination method that can be achieved in this merger. .

On the other hand, if only one image group remains after merging, the updated modularity is tracked in each recorded iteration, the maximum modularity among the modularities is determined, and the image combination method corresponding to the maximum modularity is determined as the set of images to be grouped. Since it can be determined as the image grouping result for , it is possible to ensure that the optimal image combination method shown in the process of merging image groups can be determined as the image grouping result for the image set to be grouped. Alternatively, before performing the merging operation each time, it is judged whether this merging will result in a reduction in modularity, and if it does not result in a reduction in modularity, the merging operation is performed; The image combination method can be the final image grouping result for the image set to be grouped, and accordingly, when the modularity is the highest in the iterative process, the corresponding image grouping method is determined as the image grouping result for the image set to be grouped. can be guaranteed Through the division of the groups described above, the connection relationship between the images inside each group is relatively strong and the connection relationship between the images inside each group and the images of other groups is relatively weak, thereby improving the accuracy of image grouping, and further based on image grouping Improve the robustness of performing 3D reconstruction.

Overall, in a large-scale 3D reconstruction system, the image grouping method presented by the embodiment of the present invention can be used to realize effective grouping of image sets and effective reconstruction of large-scale scenes. In addition, image clustering may be implemented using this algorithm in an image classification application scene. In addition, in the image network segmentation scene, effective segmentation of the image network configuration may be implemented using the image grouping method presented by the present embodiment.

Claims (24)

As an image grouping method during three-dimensional reconstruction,
Calculating modularity corresponding to the image set to be grouped by taking each image among the image sets to be grouped as one image group;
obtaining an incremental set by calculating an increment of modularity corresponding to each combination, assuming that a plurality of image groups included in the image set to be grouped are combined by two;
determining a maximum increment in the incremental set, merging two image groups corresponding to the maximum increment to obtain a merged image group, and updating the modularity corresponding to the image set to be grouped and the incremental set; performing iteratively until a unique image group is included in the to-be-image set; and
When a unique image group is included in the image set to be grouped, the maximum modularity among the updated modularities is determined in each iteration, and an image combination method corresponding to the maximum modularity is determined as an image grouping result for the image set to be grouped. Image grouping method during 3D reconstruction, characterized in that it comprises the step of determining. According to claim 1,
The step of determining the image combination method corresponding to the maximum modularity as an image grouping result for the image set to be grouped,
determining whether the maximum modularity has reached a preset threshold;
determining an image combination method corresponding to the maximum modularity as an image grouping result for the image set to be grouped, when the maximum modularity reaches the preset threshold; and
When the maximum modularity does not reach the preset threshold, grouping each image included in the set of images to be grouped into an independent image group. According to claim 1,
Combining the plurality of image groups included in the image set to be grouped by two,
Combining two image groups having a connection relationship among the plurality of image groups, and recording a correspondence between group identifiers of the image groups having a connection relationship;
Image grouping during 3D reconstruction, characterized in that, in the two image groups having a connection relationship, the similarity between the images of the first image group and the second image group of the two image groups reaches a preset similarity threshold. method. According to claim 3,
the incremental set includes a correspondence between group identifiers of two image groups assumed to be combined and the increments obtained by calculation assuming that the two image groups are merged;
Updating the incremental set comprises:
Searching for a correspondence between group identifiers of image groups having a recorded connection relationship, and determining another image group having a connection relationship with one of the two image groups corresponding to the maximum increment;
After assuming that the different image groups are merged with the two images respectively, the obtained increment is determined, and the result obtained by adding the determined increment is assumed to merge the merged image group with the other image group. incrementing the modularity; and
and recording, in the increment set, a correspondence between a group identifier of each of the merged image group and the other image group and the increment of the module obtained assuming that the two are merged. Among the image grouping methods. According to claim 4,
The image grouping method during the 3D reconstruction,
In each iteration process, after updating the incremental set, recording a correspondence between the group identifier of the merged image group and the group identifier of the other image group; method. According to claim 1,
The image grouping method during the 3D reconstruction,
In each repetition process, recording the merged image group obtained by merging and the correspondence between the group identifier of the merged image group and the group identifier of the two merged image groups. Image grouping method during reconstruction. According to claim 6,
the merge image group corresponds to an updated modularity in an iteration of creating the merge image group; a group identifier of the image group corresponds to a corresponding image identifier indicating each image included in the image group;
In each repetition of the recording, the step of determining the maximum modularity among the updated modularities and determining the image combination method corresponding to the maximum modularity as the image grouping result for the image set to be grouped,
Starting from the group of merged images recorded in the last iteration, and in reverse order for the group of merged images recorded in each iteration, according to the order of occurrence of the iterations,
It is determined whether the group identifier of the merged image group is included in the identifier set for checking the image combination method, and if the identifier set does not include the group identifier of the merged image group, the group identifier of the merged image group is determined. writing to the set of identifiers;
determining whether an updated modularity corresponding to the merged image group is a maximum modularity among updated modularities in each recorded iteration; and
If the updated modularity corresponding to the merged image group is the maximum modularity among the updated modularity in each recording iteration, the image indicated by the image identifier corresponding to each group identifier included in the identifier set is converted into one image combination. and performing a step of determining each obtained image combination as an image grouping result for the set of images to be grouped. According to claim 7,
The image grouping method during the 3D reconstruction,
When the updated modularity corresponding to the merged image group is not the largest modularity among the updated modularities in each recording iteration, a group identifier of the merged image group recorded in the identifier set is selected as the merged image group corresponding to the merged image group. Image grouping method during three-dimensional reconstruction, characterized in that it further comprises the step of replacing the group identifier of the previous two image groups. According to claim 8,
The step of determining whether the updated modularity corresponding to the recorded merged image group is the maximum modularity among the updated modularities in each repetition,
determining whether the modularity updated in this iteration has reached the modularity updated in the previous iteration corresponding to this iteration;
If it is reached, determining the modularity updated in this iteration as the maximum modularity. According to claim 3,
Assuming that the plurality of image groups are combined by two, obtaining an incremental set by calculating each increment of modularity corresponding to each combination,
For each of the above increments obtained by calculating and assuming a combination,
Assume combining is for each group of two images hypothesized to combine, determine a binary tree corresponding thereto, and set the increment, and the group identifier of the two images hypothesized to combine, to the determined binary tree. performing the step of writing to the tree;
Image grouping method during three-dimensional reconstruction, characterized in that the maximum increment among all increments recorded in the binary tree and a group identifier corresponding to the maximum increment are recorded at the root node of the binary tree. According to claim 10,
Determining the maximum increment included in the increment set comprises:
pushing a root node pointer of the binary tree corresponding to each image group into a maximum heap to determine a maximum increment;
determining a pointer recorded in a root node of the maximum heap; and
and determining the increment recorded at the node indicated by the recorded pointer as the maximum increment included in the increment set. According to any one of claims 1 to 11,
In each acquired image group, determining an image group in which the number of included images is less than a first preset number threshold, and determining the determined image group as an image group to be merged, wherein the first preset threshold is the minimum of each group - Indicates the number of images;
determining remaining groups other than the image group to be merged among the image groups as the remaining groups;
calculating a connection weight between each two image groups among the to-be-fused groups and constructing a connection weight set, wherein the connection weight indicates the number of images matched between the two image groups; and
Image grouping method during 3D reconstruction, characterized in that it further comprises the step of fusing the image group to be fused according to the set of connection weights. According to claim 12,
The step of fusing the image groups to be fused according to the set of connection weights,
Until the connection weight is not included in the connection weight set,
determining a maximum connection weight among the set of connection weights;
determining whether or not the number of images in one of the two image groups corresponding to the maximum concatenation weight reaches a second preset number threshold, wherein the second preset number threshold is set in each image group; Indicates the maximum number of images - ;
terminating this iteration if the second preset number threshold is reached;
merging the two image groups to obtain a merged image group and updating the concatenation weight set when the second preset number threshold is not reached; and
When the connection weight set does not include the connection weight, repeatedly performing the step of setting an image group included in the group to be merged and an image group among the remaining groups as a final image group for the image set to be grouped. Image grouping method during 3D reconstruction comprising a. According to claim 12,
The step of calculating connection weights between each two image groups among the groups to be fused and configuring a connection weight set,
generating a binary tree corresponding to each image group on a one-to-one basis for each image group among the to-be-fused groups; and
Calculating a connection weight between each of two image groups among the groups to be fused, and recording the connection weight and a group identifier of the two image groups in a node of the binary tree - each image in the root node of the binary tree A maximum connection weight among each connection weight corresponding to a group and a group identifier corresponding to the maximum connection weight are recorded. According to claim 14,
Determining the maximum connection weight among the set of connection weights,
pushing a root node pointer of a binary tree corresponding to each image group to a maximum heap for deriving the maximum connection weight; and
Reading a pointer recorded in the root node of the maximum heap, and determining a connection weight recorded in a binary tree node indicated by the recorded pointer as the maximum connection weight. grouping method. According to claim 14,
After determining the maximum connection weight of the connection weight set, the image grouping method during the 3D reconstruction,
determining whether a binary tree corresponding to one of the two image groups corresponding to the maximum connection weight is null;
if not null, continuing with subsequent steps; and
If null, the image grouping method during 3D reconstruction further comprising the step of terminating the current iteration. According to claim 14,
Updating the connection weight set comprises:
deleting a binary tree corresponding to each of the two merged image groups;
calculating a connection weight between the merged image group and another group among the groups to be merged;
writing the calculated connection weight and group identifiers of the merged image group and the other group into a binary tree corresponding to the other group; and
and deleting a node recording a connection weight between the other group and the merged two image groups in the binary tree corresponding to the other group. As an image grouping method during three-dimensional reconstruction,
Calculating modularity corresponding to the set of images to be grouped when each image among the set of images to be grouped is regarded as one image group;
obtaining an incremental set by calculating an increment of modularity corresponding to each combination when combining a plurality of image groups included in the image set to be grouped by two;
Determine the maximum increment included in the set of increments, further determine whether the maximum increment reaches 0, and merge the two image groups recorded corresponding to the maximum increment if the maximum increment reaches 0. obtaining a merged image group and updating the incremental set repeatedly until a unique image group is included in the to-be-grouped image set;
In each iteration, when the maximum increment does not reach 0, terminating the above iterative step, and making the image combination method merged in the previous iteration of this iteration the final image grouping result for the image set to be grouped. ; and
When the image set to be grouped includes a unique image group, image grouping method during 3D reconstruction comprising the step of dividing each image included in the image set to be grouped into one image group. As an image grouping device during three-dimensional reconstruction,
a first calculation unit configured to calculate modularity corresponding to the image set to be grouped by taking each image among the image sets to be grouped as one image group;
a first generation unit configured to obtain an incremental set by calculating an increment of modularity corresponding to each combination, assuming that a plurality of image groups included in the image set to be grouped are combined in pairs;
determining a maximum increment in the incremental set, merging two image groups corresponding to the maximum increment to obtain a merged image group, and updating the modularity corresponding to the image set to be grouped and the incremental set; a first merging unit configured to perform iteratively until a unique image group is included in the image set;
When a unique image group is included in the set of images to be grouped, the maximum modularity among the updated modularities is determined in each recorded iteration, and an image combination method corresponding to the maximum modularity is selected as image grouping for the set of images to be grouped. An image grouping device during 3D reconstruction comprising a first group determining unit configured to determine a result. According to claim 19,
The image grouping device during the 3D reconstruction,
an image group to be merged determining unit, configured to determine, in each acquired image group, an image group whose number of included images is smaller than a first preset number threshold, and determine the determined image group as an image group to be merged - the first dictionary The set threshold indicates the minimum number of images in each group;
a remaining group determining unit configured to determine remaining groups other than the to-be-fused image group among the image groups as remaining groups;
a connection weight calculation unit configured to calculate a connection weight between each two image groups among the groups to be fused and configure a connection weight set, wherein the connection weight indicates the number of images matched between the two image groups; and
and a group convergence unit configured to fuse the image groups to be merged according to the set of connection weights. As an image grouping device during three-dimensional reconstruction,
a second calculation unit configured to calculate modularity corresponding to the image set to be grouped when each image among the image sets to be grouped is regarded as one image group;
a second generation unit configured to obtain an incremental set by calculating an increment of modularity corresponding to each combination when combining the plurality of image groups included in the image set to be grouped by two;
determine the maximum increment included in the set of increments, further determine whether the maximum increment reaches 0, and if the maximum increment reaches 0, merge the recorded two image groups corresponding to the maximum increment; a second merging unit, configured to repeatedly perform the step of obtaining a group of merged images and updating the incremental set until a unique image group is included in the set of images to be grouped; and
In each iteration, if the maximum increment does not reach 0, the above iteration steps are terminated, and the image combination method merged in the previous iteration of this iteration is taken as the final image grouping result for the image set to be grouped;
and a second group determining unit configured to divide each image included in the image set to be grouped into one image group when the set of images to be grouped includes a unique image group. grouping device. As an electronic device,
processor;
A memory for storing instructions executable by the processor,
The processor is configured to implement the image grouping method during 3D reconstruction of any one of claims 1 to 17 or the image grouping method during 3D reconstruction of claim 18 by calling an executable command stored in the memory. electronic devices made of. As a computer readable storage medium,
A computer program is stored, wherein the computer program is for performing the image grouping method during 3D reconstruction or the image grouping method during 3D reconstruction according to claim 18 according to any one of claims 1 to 17. storage medium. As a computer program,
When the computer readable code is included and the computer readable code is executed in an electronic device, the processor of the electronic device may use the image grouping method during 3D reconstruction of any one of claims 1 to 17 or the 3 of claim 18. A computer program characterized by performing an image grouping method during dimensional reconstruction.

Images