KR102638632B1 - Methods, devices, electronic devices, storage media and programs for building point cloud models - Google Patents

Abstract

Embodiments of the present invention relate to a point cloud model building method, apparatus, device, and storage medium, wherein the point cloud model building method includes: acquiring a first feature of a panoramic image in a panoramic image set; determining at least one group of image pairs and a corresponding matching result in the panoramic image set according to the first feature, wherein the image pair includes two panoramic images to which the first feature is matched, and the matching result is: Indicates the correspondence relationship between the first features of two panoramic images - ; and constructing a point cloud model according to the at least one group of image pairs and the corresponding matching result. Since matching was performed using panoramic images and a point cloud model was built with the additional matching results, the number of images in the image set can be reduced, thereby improving matching efficiency and modeling efficiency, and the panoramic image Since the range of the corresponding space is relatively large, the matching effect between panoramic images can be increased, thereby improving the accuracy and quality of point cloud model construction.

Description

Methods, devices, electronic devices, storage media and programs for building point cloud models

Cross-reference to related applications

The present invention was filed based on a Chinese patent application with application number 202110240320.4, application date March 4, 2021, and patent title “Method, Apparatus, Electronic Device, and Storage Media for Building Point Cloud Model,” filed in China. Priority of the patent application is claimed, and the entire content of the above Chinese patent application is hereby incorporated by reference.

The present invention relates to three-dimensional modeling, and relates to, but is not limited to, a point cloud model construction method, device, electronic device, computer storage medium, and computer program.

With the development of artificial intelligence technology, spatial modeling technology becomes increasingly richer and more accurate. Motion recovery structure technology can build a three-dimensional point cloud model using multiple images, which can be widely used in fields such as physical space digitization, high-precision map construction, and augmented reality. Therefore, in the field of three-dimensional modeling technology, methods to improve the accuracy and quality of point cloud model construction are very important.

Embodiments of the present invention provide a point cloud model building method, apparatus, electronic device, computer storage medium, and computer program.

According to a first aspect of an embodiment of the present invention, a point cloud model building method applied to electronic devices is provided, the point cloud model building method comprising:

Obtaining first features of a panoramic image in a panoramic image set;

determining at least one group of image pairs and a corresponding matching result in the panoramic image set according to the first feature, wherein the image pair includes two panoramic images to which the first feature is matched, and the matching result is: Indicates the correspondence relationship between the first features of two panoramic images - ; and

and building a point cloud model according to the at least one group of image pairs and the corresponding matching result.

In one embodiment, obtaining a first feature of a panoramic image in the panoramic image set comprises:

determining a plurality of perspective images corresponding to the panoramic image, wherein a set of spaces corresponding to the plurality of perspective images is a space corresponding to the panoramic image;

acquiring a second feature of at least one perspective image from the plurality of perspective images;

determining a first sub-feature of a corresponding position of the panoramic image according to a second characteristic of the perspective image, wherein the corresponding position of the perspective image and the panoramic image correspond to the same space; and

and determining a first feature of the panoramic image according to at least one first sub-feature.

In one embodiment, the step of determining a plurality of perspective images corresponding to the panoramic image includes:

Obtaining a unit sphere corresponding to the panoramic image and determining a first mapping relationship between pixel point coordinates of the panoramic image and point coordinates of the unit sphere;

determining a plurality of perspective images according to the unit sphere, and determining a second mapping relationship between pixel point coordinates of the perspective image and point coordinates of the unit sphere - a set of spherical points corresponding to the plurality of perspective images is a unit sphere lim - ; and

Determine a third mapping relationship between the pixel point coordinates of the panoramic image and the pixel point coordinates of the perspective image according to the first mapping relationship and the second mapping relationship, and apply the third mapping relationship to the pixel information of the pixel point of the panoramic image and the third mapping relationship. Accordingly, determining the pixel information of the pixel point of the perspective image.

In one embodiment, the second feature of the perspective image includes a second feature point and a corresponding second descriptor;

Determining a first sub-feature of a corresponding position of the panoramic image according to the second feature of the perspective image includes:

determining coordinates of a first feature point of the panoramic image according to the coordinates of a second feature point of the perspective image and the third mapping relationship; and

and determining a first descriptor corresponding to a first feature point of the panoramic image according to a second descriptor corresponding to a second feature point of the perspective image.

In one embodiment, the first feature includes a first feature point and a corresponding first descriptor;

Determining at least one group of image pairs and a corresponding matching result in the panoramic image set according to the first characteristic comprises:

determining a plurality of groups of image pairs according to the panoramic images to be matched corresponding to each panoramic image;

determining a plurality of groups of feature point pairs according to the first descriptor of the two panoramic images of the image pair, wherein the feature point pair of each group has two corresponding matching first feature points belonging to the two panoramic images. Contains - ; and

Determine a first essential matrix according to the feature point pairs of the plurality of groups, perform filtering on the feature point pairs of the plurality of groups using the first essential matrix, and produce a matching result corresponding to the image pair. Includes steps to obtain

In one embodiment, after the step of determining a plurality of groups of feature point pairs according to the first descriptor of the two panoramic images of the image pairs of each group, the step includes:

Obtaining the number of feature point pairs of two panoramic images of the image pair; and

It further includes filtering image pairs in which the number of feature point pairs meets a preset first condition.

In one embodiment, determining a first essential matrix according to the plurality of groups of feature point pairs includes:

Determining an angular error of the feature point pair according to the angular error of two first feature points in the feature point pair, wherein the angular error of the first feature point is a spherical surface of a unit sphere corresponding to the first feature point. - Included angle between the point and the line connecting the optical center of the unit sphere and the external polar plane - ;

determining an essential matrix several times according to a preset number of feature point pairs, using the angular error of the corresponding feature point pair as a residual term; and

It includes determining the number of internal points of each essential matrix and determining the essential matrix with the largest number of internal points as the first essential matrix.

In one embodiment, the step of determining the number of internal points corresponding to each of the essential matrices includes:

determining an angular error of a feature point pair of each group of the image pairs according to the essential matrix;

determining a feature point pair whose angular error meets a preset second condition as an internal point; and

and determining the number of internal points corresponding to the essential matrix according to all internal points.

In one embodiment, the step of filtering a plurality of groups of feature point pairs using the first essential matrix includes:

determining an angular error of a feature point pair of each group of the image pairs according to the first essential matrix; and

and filtering feature point pairs whose angle error meets a preset third condition.

In one embodiment, the point cloud model building method includes:

obtaining coordinates of two first feature points of the feature point pair of the image pair in a panoramic image;

According to a third mapping relationship corresponding to the panoramic image and the coordinates of a first feature point belonging to the feature point pair in the panoramic image, determining a perspective image associated with the feature point pair - a perspective image associated with the feature point pair is a perspective image in which a second feature point corresponding to the first feature point belonging to the feature point pair exists; and

It further includes filtering the image pair using a perspective image related to the feature point pair.

In one embodiment, filtering the image pair using a perspective image related to the feature point pair includes:

In response to the perspective image associated with the feature point pair corresponding to at least one panoramic image of the image pair being a plurality of consecutive images, and the number of the plurality of consecutive images being less than a preset filtering threshold, the image pair It includes a filtering step.

In one embodiment, the step of building a point cloud model according to the at least one group of image pairs and the corresponding matching result includes,

Determine that a group of image pairs is an initial image pair according to preset initialization conditions, feature point pairs of each group of image pairs, and a first essential matrix, determine a camera pose of each panoramic image of the initial image pair, and determine the initial image pair. performing triangulation on the first feature point pair of the image pair to form initial three-dimensional points;

According to the matching relationship between the first feature point corresponding to the first three-dimensional point and the first feature point of each unregistered image, one unregistered image is registered until each panoramic image in the panoramic image set becomes a registered image. A step of determining an image multiple times - the unregistered image is a panoramic image in which all first feature points are not triangulated, the registered image is a panoramic image in which first feature points are triangulated, and the first three-dimensional point is the initial Includes a three-dimensional point, or includes a three-dimensional point formed by triangulating the initial three-dimensional point and the first feature point of the registered image; and

After determining a registered image each time, a camera pose of the registered image is determined, triangulation is performed on the first feature point of the registered image to form a corresponding three-dimensional point, and a third feature point of the registered image is performed. performing triangulation on to form a corresponding three-dimensional point, wherein the third feature point is a first feature point in the registered image that matches the first feature point of the registered image.

In one embodiment, the step of determining that a group of image pairs is an initial image pair according to the preset initialization conditions, feature point pairs of each group of image pairs, and a first essential matrix includes:

Select image pairs sequentially in descending order of the number of feature point pairs, and after each image pair selection, all according to the feature point pair and the first essential matrix, until the selected image pair satisfies the initialization condition. It includes determining whether the image pair satisfies the initialization conditions and determining the selected image pair as the initial image pair.

In one embodiment, determining whether the image pair satisfies the initialization condition according to the feature point pair and the first essential matrix includes:

Determine at least one group of displacement variables according to the first essential matrix of the image pair, triangulate the feature points of each pair of feature points for the displacement variable of each group, and create a three-dimensional point corresponding to the displacement variable of each group. forming, and filtering the three-dimensional points according to the reprojection error and triangulation angle of each group of three-dimensional points, wherein the displacement variable includes a rotation variable and a translation variable;

In response to the number of three-dimensional points in a group having the largest number being greater than a preset first number threshold, determining a corresponding displacement variable as a first displacement variable;

From the essential matrices obtained by calculating multiple times, select an essential matrix where the number of internal points is greater than or equal to the point number threshold, determine the displacement variables of at least one group according to each essential matrix, and determine the characteristics of the displacement variables of each group. Each feature point of a point pair is triangulated to form a three-dimensional point corresponding to the displacement variable of each group, and the three-dimensional points are filtered according to the reprojection error and triangulation angle of the three-dimensional point of each group, and each essential maintaining a displacement variable corresponding to a three-dimensional point of the group with the largest number of mattresses; and

When the difference between the displacement variable held by each essential matrix and the first displacement variable satisfies a preset range, determining that the image pair satisfies the initialization condition.

In one embodiment, the point cloud model building method includes:

optimizing the camera pose and position of the initial three-dimensional point in each panoramic image through minimizing the reprojection error of the initial three-dimensional point in the two panoramic images of the initial image pair;

After determining the camera pose of the registered image each time, optimizing the camera pose of the registered image by minimizing the reprojection error of the three-dimensional point in the registered image; and

Each time, triangulation is performed on the feature point of the registered image to form a corresponding three-dimensional point, and triangulation is performed on the third feature point of the registered image to form a corresponding three-dimensional point, and then each three-dimensional point and optimizing the position of each three-dimensional point and the camera pose of each registered image through minimizing reprojection error in each registered image.

In one embodiment, the point cloud model building method includes:

determining a panoramic image to be matched according to a space corresponding to each panoramic image; or

It includes determining a panoramic image to be matched corresponding to each panoramic image according to a preset matching rule.

According to a second aspect of an embodiment of the present invention, a point cloud model building device is provided, the point cloud model building device comprising:

an acquisition module configured to acquire first features of a panoramic image in the panoramic image set;

Determine at least one group of image pairs and a corresponding matching result in the panoramic image set according to the first feature, wherein the image pair comprises two panoramic images to which the first feature is matched, and the matching result is two panoramic images. a matching module configured to indicate correspondence between first features of the panoramic image; and

and a building module configured to build a point cloud model according to the at least one group of image pairs and the corresponding matching result.

In one embodiment, the acquisition module specifically:

determining a plurality of perspective images corresponding to the panoramic image, wherein a set of spaces corresponding to the plurality of perspective images is a space corresponding to the panoramic image;

acquire a second feature of at least one perspective image in the plurality of perspective images;

determining a first sub-feature of a corresponding position of the panoramic image according to a second characteristic of the perspective image, wherein the corresponding position of the perspective image and the panoramic image correspond to the same space;

and determine a first feature of the panoramic image according to the at least one first sub-feature.

In one embodiment, when the acquisition module is configured to determine a plurality of perspective images corresponding to the panoramic image, specifically,

Obtain a unit sphere corresponding to the panoramic image, and determine a first mapping relationship between pixel point coordinates of the panoramic image and point coordinates of the unit sphere;

Determine a plurality of perspective images according to the unit sphere, determine a second mapping relationship between pixel point coordinates of the perspective image and point coordinates of the unit sphere, and - a set of spherical points corresponding to the plurality of perspective images is a unit sphere. - ;

Determine a third mapping relationship between the pixel point coordinates of the panoramic image and the pixel point coordinates of the perspective image according to the first mapping relationship and the second mapping relationship, and apply the third mapping relationship to the pixel information of the pixel point of the panoramic image and the third mapping relationship. Accordingly, it is configured to determine pixel information of the pixel point of the perspective image.

In one embodiment, the second feature of the perspective image includes a second feature point and a corresponding second descriptor;

When the acquisition module is configured to determine a first sub-feature of a corresponding position of the panoramic image according to the second feature of the perspective image, specifically,

determine coordinates of a first feature point of the panoramic image according to the coordinates of a second feature point of the perspective image and the third mapping relationship;

and determine a first descriptor corresponding to a first feature point of the panoramic image according to a second descriptor corresponding to a second feature point of the perspective image.

In one embodiment, the first feature includes a first feature point and a corresponding first descriptor;

The matching module specifically,

determining a plurality of groups of image pairs according to the panoramic images to be matched corresponding to each panoramic image;

Determining a plurality of groups of feature point pairs according to a first descriptor of the two panoramic images of the image pair of each group, wherein the feature point pair of each group corresponds to two matching first features belonging to the two panoramic images. Contains points - ;

Determine a first essential matrix according to the feature point pairs of the plurality of groups, perform filtering on the feature point pairs of the plurality of groups using the first essential matrix, and produce a matching result corresponding to the image pair. It is designed to obtain

In one embodiment, after the matching module is configured to determine a plurality of groups of feature point pairs according to the first descriptor of the two panoramic images of the image pair of each group, further:

Obtain the number of feature point pairs of the two panoramic images of the image pairs in each group;

It is configured to filter image pairs in which the number of feature point pairs meets a preset first condition.

In one embodiment, when the matching module is configured to determine a first essential matrix according to the plurality of groups of feature point pairs, specifically,

Determine the angular error of the feature point pair according to the angular error of the two first feature points in the feature point pair, and the angular error of the first feature point is a spherical point of the unit sphere corresponding to the first feature point. - the included angle between the line connecting the optical center of the unit sphere and the outer polar plane;

Using the angular error of the corresponding feature point pair as a residual term, an essential matrix is determined several times according to a preset number of feature point pairs;

It is configured to determine the number of internal points corresponding to each essential matrix, and determine the essential matrix with the largest number of internal points as the first essential matrix.

In one embodiment, when the matching module is configured to determine the number of internal points corresponding to each of the essential matrices, specifically,

determine the angular error of the feature point pair of each group of the image pairs according to the essential matrix;

determining a feature point pair whose angle error satisfies a preset second condition as an internal point;

It is configured to determine the number of internal points corresponding to the essential matrix according to all internal points.

In one embodiment, when the matching module is configured to perform filtering on a plurality of groups of feature point pairs using the first essential matrix, specifically,

determine the angular error of the feature point pair of each group of the image pairs according to the first essential matrix;

It is configured to filter feature point pairs whose angle error meets a preset third condition.

In one embodiment, the matching module also:

Two first feature points of the feature point pair of the image pair obtain coordinates in the panoramic image;

According to a third mapping relationship corresponding to the panoramic image and the coordinates of a first feature point belonging to the feature point pair in the panoramic image, determine a perspective image associated with the feature point pair, wherein the perspective image associated with the feature point pair is A perspective image in which a second feature point corresponding to a first feature point belonging to a feature point pair exists - ;

It is configured to perform filtering on the image pair using a perspective image related to the feature point pair.

In one embodiment, when the matching module is configured to perform filtering on the image pair using a perspective image related to the feature point pair, specifically,

In response to the perspective image associated with the feature point pair corresponding to at least one panoramic image of the image pair being a plurality of consecutive images, and the number of the plurality of consecutive images being less than a preset filtering threshold, the image pair It is configured to filter.

In one embodiment, the construction module specifically,

Determine that a group of image pairs is an initial image pair according to preset initialization conditions, feature point pairs of each group of image pairs, and a first essential matrix, determine a camera pose of each panoramic image of the initial image pair, and determine the initial image pair. Perform triangulation on the first feature point pair of the image pair to form initial three-dimensional points;

According to the matching relationship between the first feature point corresponding to the first three-dimensional point and the first feature point of each unregistered image, one unregistered image is registered until each panoramic image in the panoramic image set becomes a registered image. The image is determined several times - the unregistered image is a panoramic image in which all first feature points are not triangulated, the registered image is a panoramic image in which first feature points are triangulated, and the first three-dimensional point is the initial three-dimensional point. Includes a point, or includes a three-dimensional point formed by triangulating the initial three-dimensional point and the first feature point of the registered image;

After determining a registered image each time, a camera pose of the registered image is determined, triangulation is performed on the first feature point of the registered image to form a corresponding three-dimensional point, and a third feature point of the registered image is performed. Triangulation is performed to form a corresponding three-dimensional point - the third feature point is a first feature point in the registered image that matches the first feature point of the registered image.

In one embodiment, when the building module is configured to determine that a group of image pairs is an initial image pair according to preset initialization conditions and feature point pairs of each group of image pairs and a first essential matrix, specifically,

Select image pairs sequentially in descending order of the number of feature point pairs, and after each image pair selection, all according to the feature point pair and the first essential matrix, until the selected image pair satisfies the initialization condition. It is configured to determine whether the image pair satisfies the initialization conditions and determine the selected image pair as the initial image pair.

In one embodiment, when the building module is configured to determine whether the image pair satisfies the initialization condition according to the feature point pair and the first essential matrix, specifically:

Determine at least one group of displacement variables according to the first essential matrix of the image pair, triangulate the feature points of each pair of feature points for the displacement variable of each group, and create a three-dimensional point corresponding to the displacement variable of each group. forming, and filtering the three-dimensional points according to the reprojection error and triangulation angle of each group of three-dimensional points, wherein the displacement variable includes a rotation variable and a translation variable;

In response to the number of three-dimensional points in a group having the largest number being greater than a preset first number threshold, determining the corresponding displacement variable as the first displacement variable;

From the essential matrices obtained by calculating multiple times, select an essential matrix where the number of internal points is greater than or equal to the point number threshold, determine the displacement variables of at least one group according to each essential matrix, and determine the characteristics of the displacement variables of each group. Each feature point of a point pair is triangulated to form a three-dimensional point corresponding to the displacement variable of each group, and the three-dimensional points are filtered according to the reprojection error and triangulation angle of the three-dimensional point of each group, and each essential It holds the displacement variable corresponding to the three-dimensional point of the group with the largest number of mattresses;

When the difference between the displacement variable held by each essential matrix and the first displacement variable satisfies a preset range, determining that the image pair satisfies the initialization condition.

In one embodiment, the building module also:

optimizing the camera pose and position of the initial three-dimensional point in each panoramic image through minimizing the reprojection error of the initial three-dimensional point in the two panoramic images of the initial image pair;

After determining the camera pose of the registered image each time, optimizing the camera pose of the registered image through minimizing the three-dimensional point reprojection error in the registered image;

Each time, triangulation is performed on the feature point of the registered image to form a corresponding three-dimensional point, and triangulation is performed on the third feature point of the registered image to form a corresponding three-dimensional point, and then each three-dimensional point is configured to perform at least one of optimizing the position of each three-dimensional point and the camera pose of each registered image through minimizing reprojection error in each registered image.

In one embodiment, the matching module also:

determining a corresponding panoramic image to be matched according to a space corresponding to each panoramic image; or

It is configured to determine a panoramic image to be matched corresponding to each panoramic image according to a preset matching rule.

According to a third aspect of an embodiment of the present invention, an electronic device is provided, the device includes a memory and a processor, the memory is for storing computer instructions operable on the processor, and the processor executes the computer instructions. The purpose is to build a point cloud model based on the method according to the first aspect.

According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided storing a computer program, wherein the program implements the method according to the first aspect when executed by a processor.

According to a fifth aspect of an embodiment of the present invention, a computer program including computer-readable code is provided, and when the computer-readable code is operated in an electronic device, the processor of the electronic device performs any one of the above-described methods. Implement.

As can be seen from the above-described embodiments, a panoramic image set is formed using the panoramic images, the panoramic images in the panoramic image set are further matched according to the first feature, and the two panoramic images for which the first feature matching is completed are combined into one panoramic image. By using it as a group image pair, at least one group of image pairs and the corresponding matching result are determined, and finally, a point cloud model is built according to the determined at least one group of image pairs and the corresponding matching result. Since matching was performed using panoramic images and a point cloud model was built with the additional matching results, the number of images in the image set can be reduced, thereby improving matching efficiency and modeling efficiency, and the panoramic image Since the corresponding spatial range is relatively large, the matching effect between panoramic images can be increased, thereby improving the accuracy and mass of point cloud model construction.

It is to be understood that the foregoing general description and the following detailed description are illustrative and interpretive only and are not intended to limit the invention.

The drawings of this text are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present invention, and are used together with this specification to explain the principles of the present invention.
1 is a flowchart of a point cloud model construction method according to an embodiment of the present invention.
Figure 2 is an exemplary diagram of a panoramic image according to an embodiment of the present invention.
Figure 3 is an exemplary diagram of a method for obtaining the first feature of a panoramic image according to an embodiment of the present invention.
Figure 4 is an exemplary diagram of a method for determining image pairs and corresponding matching results according to an embodiment of the present invention.
Figure 5 is an example diagram of angle error according to an embodiment of the present invention.
Figure 6 is an exemplary diagram of a method for comparing initialization conditions, feature point pairs, and a first essential matrix according to an embodiment of the present invention.
Figure 7 is a structural diagram of an apparatus for building a point cloud model according to an embodiment of the present invention.
Figure 8 is an exemplary structural diagram of an electronic device according to an embodiment of the present invention.

Exemplary embodiments will be described in detail below, examples of which are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, like numbers in different drawings refer to the same or similar elements. The embodiments described in the example examples below do not represent all embodiments consistent with the invention. On the contrary, these are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the claims.

The terms used in the present invention are only for describing specific embodiments and are not intended to limit the present invention. As used in the specification and claims of the present invention, the singular forms “a” and “the” include the plural forms, unless the text clearly indicates otherwise. It should also be understood that the term “a” as used in the text. “and/or” means and includes any combination or all possible combinations of one or more related listed items.

It should be understood that although the present invention uses terms such as “first,” “second,” “third,” etc. to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, without departing from the scope of the present invention, first information may be referred to as second information; Similarly, the second information may also be referred to as first information. Depending on the context, the term "...." as used in the text may be understood to mean "when...." or "if...." or "in response to a decision," as the case may be. there is.

With the development of artificial intelligence technology, spatial modeling technology becomes increasingly richer and more accurate. Motion recovery structure technology can build a three-dimensional point cloud model using multiple images, which can be widely used in fields such as physical space digitization, high-precision map construction, and augmented reality. In existing technology, the matching efficiency between multiple images is poor, resulting in low accuracy and poor quality of the sequentially constructed point cloud model.

Based on this, in a first aspect, at least one embodiment of the present invention provides a method for building a point cloud model, and referring to Figure 1, the flow of the method including steps S101 to S103 is shown.

Here, the point cloud model is a three-dimensional model corresponding to space, and space may mean the real world. Each object in the real world is represented by a corresponding point cloud in the model, and a point cloud refers to a set of points made up of three-dimensional points.

In addition, the point cloud model construction method can be executed by an electronic device such as a terminal device or server, and the terminal device is a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a wireless phone, and personal information. It may be a terminal (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle device, a wearable device, etc., and the point cloud model construction method may be implemented in such a way that a computer-readable command stored in a memory is called by a processor. there is. Alternatively, the point cloud model building method may be executed through a server, and the server may be a local server, a cloud server, or the like.

In step S101, the first feature of the panoramic image in the panoramic image set is obtained.

Here, the panoramic image set includes a plurality of panoramic images, and the panoramic images may be panoramic images from various angles, for example, a 360° panoramic image; The panoramic image may be an image acquired by a spherical camera, or an image obtained by equidistant rectangular projection of spliced images acquired by a plurality of fisheye cameras. Referring to FIG. 2, one panoramic image is exemplarily shown. do. Each panoramic image corresponds to one local subspace of the modeling space, the size of the local subspace corresponding to the panoramic image is larger than the size of the local subspace corresponding to a normal image at the same parameters, and all of the panoramic image sets are The local subspace corresponding to the panoramic image can constitute the entire modeling space; Local subspaces corresponding to different panoramic images may have overlapping areas.

In this step, the first feature of the panoramic image may be acquired using a pre-trained neural network, and the first feature of the panoramic image may be acquired using another method, and the present invention does not specifically describe the acquisition method. is not limited to The first feature of each panoramic image in the panoramic image set may be obtained.

In step S102, determine at least one group of image pairs and a corresponding matching result in the panoramic image set according to the first feature, wherein the image pair includes two panoramic images to which the first feature is matched. And, the matching result indicates a correspondence relationship between the first features of the two panoramic images.

Here, the first feature matching means that there is a first feature point corresponding to the same space in the two panoramic images, that is, at least one first feature point of one panoramic image and at least one first feature point of the other panoramic image. This means that the first feature point corresponds to the same space in the real world.

In this step, matching can be performed on the panoramic images in the panoramic image set using a brute-force matching method. In other words, matching can be performed using a one-by-one matching method. For example, each panoramic image can be matched sequentially. By selecting, you can proceed with matching with each other panoramic image; When matching two panoramic images, matching is performed using the first feature of the two panoramic images. In other words, the matching result of the panoramic image is determined according to the matching result of the first feature of the two panoramic images. When the first feature matching of the two panoramic images is completed, that is, when the two panoramic images complete the feature matching, these two images can be determined as a group of image pairs, and at the same time, the first feature of the two panoramic images The corresponding relationship can be determined as a matching result.

Here, each panoramic image may form an image pair with another panoramic image, and each panoramic image may form a plurality of different panoramic images and a plurality of groups of image pairs. In other words, after each panoramic image forms an image pair, , If not locked, you can continue to construct new image pairs with other panoramic images.

In step S103, a point cloud model is constructed according to the at least one group of image pairs and the corresponding matching result.

In this step, a point cloud model is constructed using the matching result in step S102 described above. The modeling process, that is, the motion recovery architecture process, includes camera registration and point cloud reconstruction, where camera registration can be expressed using the camera motion parameters (e.g. camera pose) of each single panoramic image within the panoramic image set. It is a process of restoring the point cloud, and point cloud reconstruction is restoring the point cloud of the three-dimensional structure of the corresponding local subspace (i.e., the local subspace proposed in step S102).

In one example, a point cloud model can be built using an incremental reconstruction approach.

In an embodiment of the present invention, a panoramic image set is formed using panoramic images, the panoramic images in the panoramic image set are matched according to the first feature, and the two panoramic images for which feature matching has been completed are converted into one group image. By using them in pairs, at least one group of image pairs and the corresponding matching results are determined, and finally, a point cloud model is built according to the at least one group of image pairs and the corresponding matching results determined. Since matching was performed using panoramic images and a point cloud model was built with the additional matching results, the number of images in the image set can be reduced, thereby improving matching efficiency and modeling efficiency, and the panoramic image Since the range of the corresponding space is relatively large, the matching effect between panoramic images can be increased, thereby improving the accuracy and quality of point cloud model construction.

In some embodiments of the present invention, the first feature of the panoramic image of the panoramic image set may be acquired using the following method, referring to Figure 3, which illustrates the flow of the acquisition method, steps S301 to S304. Includes.

In step S301, a plurality of perspective images corresponding to the panoramic image are determined, where a set of spaces corresponding to the plurality of perspective images is a space corresponding to the panoramic image.

In one example, a plurality of perspective images can be determined using the method below, first, obtaining a unit sphere corresponding to the panoramic image, and performing a first mapping between the pixel point coordinates of the panoramic image and the point coordinates of the unit sphere. determine relationships; Next, determine a plurality of perspective images according to the unit sphere, determine a second mapping relationship between the pixel point coordinates of the perspective image and the point coordinates of the unit sphere - a set of spherical points corresponding to the plurality of perspective images is a unit Spherical - ; Finally, determine a third mapping relationship between the pixel point coordinates of the panoramic image and the pixel point coordinates of the perspective image according to the first mapping relationship and the second mapping relationship, and determine the pixel information of the pixel point of the panoramic image and the third mapping relationship. Determine the pixel information of the pixel point of the perspective image according to the mapping relationship.

Here, obtaining the unit sphere corresponding to the panoramic image may be back-projecting the panoramic image onto the unit sphere. Determining a plurality of perspective images according to the unit sphere may mean that a virtual perspective camera optically centered on the center of the unit sphere photographs the unit sphere and determines the obtained image as a perspective image. When photographing the unit sphere, The virtual camera is rotated at a certain angle to capture one perspective image each rotation, and the entire spherical surface is captured until multiple perspective images covering all perspectives are obtained, for example, If one perspective image is taken by rotating 60° each time, the entire spherical surface can be captured only by taking six perspective images; The field of view (FOV) and focal length of the virtual camera can be set to the parameters of a general image collection device (e.g., mobile phone, digital camera, etc.) to approximate the image acquired by a general image collection device.

The first mapping relationship between the pixel point coordinates of the panoramic image and the point coordinates of the unit sphere can be expressed according to formula (1) below.

(One)

here, is the point coordinate of the unit sphere, is the pixel point coordinate of the panoramic image, w is the width of the panoramic image, and h is the height of the panoramic image.

The second mapping relationship between the pixel point coordinates of the perspective image acquired with the same virtual camera as the coordinate system of the unit sphere and the point coordinates of the unit sphere can be expressed according to formula (2) below,

(2)

here, is the point coordinate of the unit sphere, is the pixel point coordinate of the perspective image, w~ is the width of the perspective image, h~ is the height of the perspective image, and f is the focal length of the virtual camera.

The second mapping relationship between the pixel point coordinates of a perspective image acquired by a virtual camera rotated at a certain angle based on the coordinate system of the unit sphere and the point coordinates of the unit sphere can be expressed according to formula (3) below.

(3)

here, is the coordinate point of the unit sphere are the coordinates transformed through the virtual camera's reverse rotation matrix R, are the coordinates in the perspective image, w~ is the width of the perspective image, h~ is the height of the perspective image, and f is the focal length of the virtual camera.

In an embodiment of the present invention, in a plurality of shooting angles of the virtual camera, the coordinate system of one angle and the unit sphere is the same, so that at this angle, there is a third difference between the pixel point coordinates of the panoramic image and the pixel point coordinates of the perspective image. The mapping relationship can be obtained through combining formula (1) and formula (2) above; From another angle, a third mapping relationship between the pixel point coordinates of the panoramic image and the pixel point coordinates of the perspective image can be obtained through combining the above formulas (1) and (3).

Here, determining the pixel information of the pixel point of the panoramic image and the pixel information of the pixel point of the perspective image according to the third mapping relationship includes pixel information of the pixel point of the panoramic image and pixel information of the pixel point corresponding to the perspective image. can be determined directly, and pixel information of the perspective image can be obtained by performing at least one of sampling and bilinear interpolation on the pixel points of the panoramic image. The pixel information may be the brightness value of the pixel point, or the value of each color channel (eg, red, green, and blue channels) of the pixel point.

In step S302, a second feature of at least one perspective image in the plurality of perspective images is acquired.

In this step, the second feature of the perspective image may be extracted using a pre-trained neural network, and the second feature of the perspective image may be extracted using another method, and the present invention provides specific details about the extraction method. No restrictions. The second feature of each perspective image corresponding to the panoramic image can be obtained.

In one example, the second feature is a second feature point and a corresponding second descriptor, that is, all the second feature points and the corresponding second descriptor in the perspective image constitute a second feature of the perspective image.

In step S303, a first sub-feature of the corresponding position of the panoramic image is determined according to the second feature of the perspective image, where the corresponding position of the perspective image and the panoramic image corresponds to the same space.

Here, the corresponding positions of the perspective image and the panoramic image correspond to the same space, that is, the corresponding positions of the perspective image and the panoramic image correspond to the same set of spherical points on the unit sphere.

Here, the first sub-feature may include all first feature points and a corresponding first descriptor within the corresponding location of the panoramic image.

In an example corresponding to the example of step S302, the first sub-feature of the corresponding position of the panoramic image may be determined using the method below, first, the coordinates of the second feature point of the perspective image and the third mapping relationship determine the coordinates of a first feature point of the panoramic image according to; Next, a first descriptor corresponding to the first feature point of the panoramic image is determined according to the second descriptor corresponding to the second feature point of the perspective image.

Here, the point in the panoramic image corresponding to the second feature point is the first feature point, in other words, the first feature point and the second feature point correspond to each other, or are spherical points of the unit sphere corresponding to the first feature point. and the spherical point of the unit sphere corresponding to the second feature point coincides. The second descriptor corresponding to the second feature point can be directly used as the first descriptor of the corresponding first feature point.

In step S304, a first feature of the panoramic image is determined according to the at least one first sub-feature.

Here, the first feature of the panoramic image includes all first feature points and the corresponding first descriptor in the panoramic image.

In an embodiment of the present invention, the mapping relationship between the panoramic image and the perspective image is determined through the mapping relationship between the panoramic image and the unit sphere and the mapping relationship between the unit sphere and the perspective image, that is, the panoramic image is generated using the unit sphere as a medium. Extraction of the first feature of the panoramic image is implemented by dividing it into a plurality of perspective images, further extracting the second feature of the perspective image, and reverse mapping the second feature point to the first feature point of the panoramic image. .

In some embodiments of the present invention, the first feature includes a first feature point and a corresponding first descriptor, and correspondingly, at least one feature in the panoramic image set is selected according to the first feature through the following manner. Group image pairs and corresponding matching results can be determined. Referring to FIG. 4, the flow of the decision method including steps S401 to S403 is illustrated.

In step S401, a plurality of groups of image pairs are determined according to the panoramic image to be matched corresponding to each panoramic image.

In this step, two panoramic images are formed into a group of image pairs. The panoramic image to be matched can be determined according to the space corresponding to each panoramic image, or the panoramic image to be matched can be determined according to a preset matching rule. In other words, when determining a panoramic image to be matched with one panoramic image, an overlapping panoramic image existing in the corresponding space of the panoramic image can be used as a panoramic image to be matched, and can also be determined according to preset matching rules. , a preset matching rule may be determined according to the above principles, for example, numbering the panoramic images according to the corresponding spatial order, and then matching a preset number (e.g., 10) of panoramic images after each panoramic image. can be used as a panoramic image to be matched. Additionally, all panoramic images other than one panoramic image can be used as a panoramic image to be matched with the panoramic image.

In step S402, determine a plurality of groups of feature point pairs according to the first descriptor of the two panoramic images of the image pair, wherein the feature point pair of each group is correspondingly matched to two panoramic images. Includes a first feature point belonging to

In this step, first, for each first descriptor in the first panoramic image of the image pair, the first descriptor with the closest Euclidean distance is searched in the second panoramic image, and then, conversely, for each first descriptor in the first panoramic image of the image pair, the first descriptor with the closest Euclidean distance is searched. For each first descriptor, the first descriptor with the closest Euclidean distance is searched in the first panoramic image, and the specified first descriptor in the first panoramic image and the specified first descriptor in the second panoramic image are different panoramic images. If the Euclidean distance in is the closest first descriptor, the two first descriptors are considered to match, thereby determining that the two first feature points corresponding to the two first descriptors are matched, in other words, the two first descriptors are considered to match. The first feature points complete feature matching, forming a feature point pair.

In this step, after determining all feature point pairs between two panoramic images of an image pair according to the method above, the number of feature point pairs is also statistic, the first condition is preset again, and the first condition is used to A plurality of groups of image pairs determined in step S401 may be filtered, that is, some image pairs may be removed using the first condition. In one example, the first condition may be less than the second number threshold, that is, the number of filtered feature point pairs may be less than the image pairs of the second number threshold, and the number of feature point pairs removed may be the second number threshold. The number threshold may be smaller than the image pair, for example, the second number threshold may be set to 5 or 10, etc., and embodiments of the present invention are specifically limited to the specific value of the second number threshold. I never do that. By filtering some image pairs, subsequent operations on image pairs with low matching can be reduced, thereby reducing the complexity of the operation and improving processing efficiency.

Here, a pair of feature points in each group indicates a correspondence relationship between two first feature points, and a pair of feature points in a plurality of groups constitutes a matching result of an image pair.

In step S403, a first essential matrix is determined according to the plurality of groups of feature point pairs, filtering is performed on the plurality of groups of feature point pairs using the first essential matrix, and the image pair is Obtain the corresponding matching result.

In this step, the first essential matrix can be determined using the method below, first, determine the angular error of the feature point pair according to the angular error of the two first feature points in the feature point pair, where , the angular error of the first feature point is the included angle between the line connecting the spherical point of the unit sphere corresponding to the first feature point and the optical center of the unit sphere and the outer polar plane, and then the corresponding pair of feature points Using the angle error as a residual term, the essential matrix is determined several times according to a preset number of the feature point pairs; Finally, the number of internal points corresponding to each essential matrix is determined, and the essential matrix with the largest number of internal points is determined as the first essential matrix.

Here, the point coordinates of the unit sphere corresponding to the two first feature points of each group of feature point pairs and the essential matrix satisfy the relationship shown in formula (4) below,

(4)

In the above formula (4), E is the essential matrix, is the point coordinate of the unit sphere corresponding to the first feature point in the first panoramic image, is the point coordinate of the unit sphere corresponding to the first feature point in the second panoramic image.

Here, the relatively larger angle error of the two first feature points is determined as the angle error of the feature point pair. In other words, the angle error of the feature point pair is determined according to formula (5) below.

(5)

In the above formula (5), is the angular error of the first feature point of the second panoramic image, that is, the included angle between the line connecting the spherical point S' corresponding to the first feature point in the second panoramic image and the optical center O ₂ and the corresponding external polar plane, , where the corresponding external polar plane consists of a line connecting the optical center O ₁ and a spherical point S corresponding to the first feature point in the first panoramic image and a line connecting the two optical centers Q ₁ and Q ₂ It is flat. Correspondingly, referring to Figure 5, is the angular error of the first feature point of the first panoramic image, that is, is the included angle between the line connecting the optical center and the spherical point corresponding to the first feature point in the first panoramic image and the corresponding external polar plane, where The external polar plane is a plane composed of a line connecting the optical center and a spherical point corresponding to the first feature point in the second panoramic image and a line connecting two optical centers. In embodiments of the present invention, angular error and spherical error can be used to better adapt the camera model of the panoramic image.

Here, multiple essential matrices can be calculated through RANSAC (Random Sample Consensus) and the 5-point algorithm by using the angular error of the feature point pair as a residual term, and each of the five groups of feature point pairs can calculate one essential matrix. And, using the above method, a plurality of essential matrices can be obtained.

Here, the number of internal points corresponding to the essential matrix can be determined according to the following method: first, calculate the angular error of the feature point pair of each group of the image pair according to the essential matrix; Next, a feature point pair whose angular error meets a preset second condition is determined as an internal point; Finally, the number of internal points corresponding to the essential matrix is determined according to all internal points. In other words, determine the angle error of each pair of feature points in each group using the above formula (5) and the essential matrix; A second condition is preset, and internal points are selected using the second condition. In one embodiment, the second condition may be that the angle error is less than the first angle threshold, that is, the first angle threshold. The feature point pair corresponding to the angle error smaller than the value is determined as the internal point.

In this step, filtering can be performed on the feature point pairs of the plurality of groups using the first essential matrix through the method below. First, the features of each group of the image pairs are filtered according to the first essential matrix. determine the angular error of the point pair; Next, a pair of feature points whose angle error meets the preset third condition is determined.

Here, the angular error of each pair of feature points in each group can be calculated using the formula (5) and the first essential matrix; A third condition is preset, and feature point pairs are selected using the third condition. In one embodiment, the third condition is that the angle error is greater than or equal to the second angle threshold (for example, 0.4 degrees). may be greater than or equal to, that is, filter feature point pairs corresponding to an angular error greater than or equal to the second angle threshold, i.e. remove feature point pairs corresponding to an angular error greater than or equal to the second angle threshold. And, a pair of feature points corresponding to an angle error smaller than the second angle threshold is maintained.

In an embodiment of the present invention, feature matching is performed on image pairs to determine feature point pairs between image pairs, further determine a first essential matrix according to the feature point pairs, and finally, the first essential When filtering the feature point pair with a matrix, determining the first essential matrix, and filtering the feature point pair, the angle error is used, so compared to other essential matrices, the feature point pair matching the first essential matrix is the best. The filtering step not only improves the accuracy of the first essential matrix by removing feature point pairs that do not match the first essential matrix, but also maximizes the number of feature point pairs under the premise of removing incorrect feature point pairs. , improves the matching precision and accuracy of two panoramic images of image pairs. Using a 360-degree visual cover of a panoramic image, the number of feature matches between images is improved and the possibility of camera registration failure in weak texture areas is reduced.

In some embodiments of the present invention, after completing the calculation of the first essential matrix and filtering of the feature point pairs of each group of image pairs, matching of the two panoramic images using the distribution of the first feature points is performed to create an overlapping texture. By determining whether or not it occurred due to the above, filtering can be performed on a plurality of groups of image pairs. Specifically, the method below can be used. First, the two first feature points of the feature point pair of the image pair obtain coordinates in the panoramic image; Next, according to the third mapping relationship corresponding to the panoramic image and the coordinates of the first feature point belonging to the feature point pair in the panoramic image, a perspective image associated with the feature point pair is determined, wherein the feature point pair and The related perspective image is a perspective image in which there is a second feature point corresponding to the first feature point belonging to the feature point pair; Finally, filtering is performed on the image pair using the perspective image associated with the feature point pair.

In addition, a third number threshold may be preset, and the number of second feature points corresponding to a pair of feature points included in the perspective image may be determined, and if the number is greater than or equal to the above-described third number threshold, the perspective image If the image is determined to be a perspective image associated with a pair of feature points, the third number threshold can be set to, for example, 5, thereby avoiding incorrect statistics due to matching a small amount of noise.

Here, in response to the perspective image associated with the feature point pair corresponding to at least one panoramic image of the image pair being a plurality of consecutive images, and the number of the plurality of consecutive images being less than a preset filtering threshold, Filter image pairs. In other words, if the second feature points corresponding to the first feature point associated with the feature point pair are all concentrated in some perspective images, and the number of these some perspective images is less than the preset filtering threshold, the two panoramic images The match is a bad match due to duplicate textures, so the image pair is filtered, i.e. the image pair is removed. The filtering threshold may be determined according to the total number of perspective images and a preset first ratio. For example, if the total number of perspective images is 6 and the preset first ratio is 0.5, the number of the perspective images associated with the feature point pair is 0.5. If the number is less than 3, filter the image pair.

In an embodiment of the present invention, the first feature points for which feature matching has been completed all correspond to the second feature points of the perspective image, and thus the matching of the two panoramic images is duplicated by determining the distribution of the second feature points. By determining whether it is caused by a texture and removing matching noise, the accuracy of the determination can be further improved, and image pairs with incorrect matching can be removed. Most of the repeated textures are local, and the overall matching situation between panoramic images is used to eliminate incorrect matching as much as possible and avoid the occurrence of error rates in camera registration.

In some embodiments of the present invention, a point cloud model can be built according to the at least one group of image pairs and the corresponding matching result through the following method, first, preset initialization conditions and image pairs of each group Determine that a group of image pairs are initial image pairs according to the feature point pairs and the first essential matrix, determine the camera pose of each panoramic image of the initial image pair, and triangulate for the first feature point pair of the initial image pair. perform processing to form an initial three-dimensional point; Next, according to the matching relationship between the first feature point corresponding to the first three-dimensional point and the first feature point of each unregistered image, one unregistered image is selected until each panoramic image in the panoramic image set becomes a registered image. is determined several times as a registered image, wherein the unregistered image is a panoramic image in which all first feature points are not triangulated, the registered image is a panoramic image in which first feature points are triangulated, and the first three-dimensional point includes the initial three-dimensional point, or is a three-dimensional point formed by triangulating the initial three-dimensional point and the first feature point of the registered image; After determining a registered image each time, a camera pose of the registered image is determined, triangulation is performed on the first feature point of the registered image to form a corresponding three-dimensional point, and a third feature point of the registered image is performed. Triangulation is performed on to form corresponding three-dimensional points, and the third feature point is a first feature point that matches the first feature point of the registered image in the registered image.

Here, when determining the initial image pair, image pairs can be selected sequentially in descending order of the number of feature point pairs, and after each image pair is selected, the selected image is selected according to the feature point pair and the first essential matrix. Until the pair satisfies the initialization condition, a decision is made as to whether the image pair satisfies the initialization condition, and the selected image pair is determined as the initial image pair.

In addition, it is possible to determine whether the image pair satisfies the initialization condition according to the feature point pair and the first essential matrix using the method below. Referring to FIG. 6, the flow of the method is illustrated. , including steps S601 to S604.

In step S601, the displacement variable of at least one group (for example, four groups) is determined according to the first essential matrix of the image pair, and the feature points of the feature point pair are triangulated for the displacement variable of each group. Thus, three-dimensional points corresponding to the displacement variables of each group are formed, and the three-dimensional points are filtered according to the reprojection error and triangulation angle of the three-dimensional points of each group, where the displacement variables are rotation variables and translation variables. Contains variables.

Here, the rotation variable can be expressed as a 3*3 matrix R, and the translation variable can be expressed as a three-dimensional vector T. For each group of three-dimensional points

(6)

In the above formula, is the point coordinate of the unit sphere corresponding to the first feature point, is the camera matrix of the ith panoramic image, and for initialization, is the unit matrix, is the zero vector, , , and R and T are rotation variables and translation variables, respectively, corresponding to the three-dimensional points of each group.

Here, when filtering the three-dimensional points according to the reprojection error and triangulation angle of each group of three-dimensional points, set the third angle threshold and the fourth angle threshold, and then set the reprojection error in the two panoramic images. It can be maintained that all are smaller than the third angle threshold and the triangulated angle is larger than the three-dimensional point of the fourth angle threshold.

In step S602, in response to the fact that the number of three-dimensional points in a group with the largest number is greater than the preset first number threshold, the corresponding displacement variable is determined as the first displacement variable.

In step S603, an essential matrix whose internal point number is greater than or equal to the point number threshold is selected from the essential matrix obtained by calculating multiple times, and at least one group (for example, four groups) of displacement variables is selected according to each essential matrix. respectively, triangulating the feature points of the feature point pair for the displacement variables of each group to form three-dimensional points corresponding to the displacement variables of each group, and reprojection error and triangulation of the three-dimensional points of each group. The three-dimensional points are filtered according to the angle, and a displacement variable corresponding to the three-dimensional point of the group with the largest number of each essential mattress is held.

Here, after calculating and obtaining a plurality of essential matrices and determining the first essential matrix in step S403, the remaining essential matrices are retained and used in this step, or in this step again using the same method as in step S403, a plurality of essential matrices are obtained. It is obtained by calculating the matrix.

Here, the second ratio can be preset, and then the point number threshold is determined using the internal point number of the first essential matrix and the second ratio; The product of the number of internal points of the first essential matrix and the second ratio can be used as a point number threshold. For example, the second ratio can be preset to 0.6, and the present invention is limited to the specific value of the second ratio. I never do that.

The operation performed on the essential matrix selected in this step is the same as that of steps S601 to S602, and holds one displacement vector for each essential matrix.

In step S604, when the difference between the displacement variable held by each essential matrix and the first displacement variable satisfies a preset range, it is determined that the image pair satisfies the initialization condition.

Here, the difference between the displacement variable held by the essential matrix and the first displacement variable can be expressed using the orientation angle of the two displacement variables, and the orientation angle is obtained by multiplying the rotation matrix of the two displacement variables; The preset range can be displayed using one preset fifth angle threshold, that is, if it is less than the fifth angle threshold, the preset range is satisfied; Accordingly, if the direction angles of the position vector and the first displacement vector held by each essential matrix are both smaller than the fifth angle threshold, it is determined that the image pair satisfies the initialization condition.

If the image pair does not satisfy the initialization condition, steps S601 to S604 are used to continue determining whether another image pair satisfies the initialization condition.

In the process of determining the initialization conditions, determination of multiple interpretations of each essential matrix makes the determination result of the initialization conditions more stable.

In addition, the camera pose of each panoramic image and the position of the initial three-dimensional point can be optimized by minimizing the reprojection error of the initial three-dimensional point in the two panoramic images of the initial image pair;

In addition, after determining the camera pose of the registered image each time, the three-dimensional point can optimize the camera pose of the registered image by minimizing reprojection error in the registered image;

In addition, each time, triangulation is performed on the feature point of the registered image to form a corresponding three-dimensional point, and after triangulation is performed on the third feature point of the registered image to form a corresponding three-dimensional point, further By minimizing the reprojection error of each three-dimensional point in each registered image, the camera pose of each registered image and the position of each three-dimensional point can be optimized.

The above optimization can be performed using the loss function below,

(7)

In the above formula (7), is the point coordinate of the unit sphere corresponding to the first feature point, Is it i times?? is a vector of three-dimensional point coordinates, f is the focal length of the virtual perspective camera, is the camera matrix of the ith panoramic image, and for initialization, is the unit matrix, is the zero vector, , , R and T are rotation variables and translation variables corresponding to each three-dimensional point, respectively, m is the number of panoramic images, and n is the number of three-dimensional points.

Determining the camera pose of the image can be done using RANSAC (Random Sample Consensus) and P3P algorithms.

As can be seen based on the description of the point cloud model building method, the point cloud model building method can proceed with camera registration and point cloud reconstruction using a panoramic image, thereby completing construction of the point cloud model and panorama Point cloud models built based on images have higher accuracy, better robustness for overlapping textures, and more comprehensive scenario reconstruction than point cloud models built based on existing general perspective images. By constructing a high-precision visual map using the point cloud model building method, it is possible to provide visual features and three-dimensional landmark points for positioning for autonomous driving and AR, and also using the point cloud model building method, a specific By building a three-dimensional model of a given scenario, scenario exhibitions and VR applications are possible. For example, AR/VR tours of tourist attractions, museums, exhibition halls, or building a 3D model of a specific building, a specific street, or a specific city, AR. It can be applied to special effects.

According to a second aspect of an embodiment of the present invention, a point cloud model building device is provided. Referring to FIG. 7, the structure of the device is illustrated, and the point cloud model building device includes:

an acquisition module 701 configured to acquire first features of a panoramic image in the panoramic image set;

Determine at least one group of image pairs and a corresponding matching result in the panoramic image set according to the first feature, wherein the image pair comprises two panoramic images to which the first feature is matched, and the matching result is two panoramic images. a matching module 702 configured to indicate correspondence between first features of the panoramic image; and

and a building module 703 configured to build a point cloud model according to the at least one group of image pairs and the corresponding matching result.

In one embodiment, the acquisition module specifically:

determining a plurality of perspective images corresponding to the panoramic image, wherein a set of spaces corresponding to the plurality of perspective images is a space corresponding to the panoramic image;

acquire a second feature of at least one perspective image in the plurality of perspective images;

determining a first sub-feature of a corresponding position of the panoramic image according to a second characteristic of the perspective image, wherein the corresponding position of the perspective image and the panoramic image correspond to the same space; and

and determine a first feature of the panoramic image according to the at least one first sub-feature.

In one embodiment, when the acquisition module is configured to determine a plurality of perspective images corresponding to the panoramic image, specifically,

Obtain a unit sphere corresponding to the panoramic image, and determine a first mapping relationship between pixel point coordinates of the panoramic image and point coordinates of the unit sphere;

Determine a plurality of perspective images according to the unit sphere, determine a second mapping relationship between pixel point coordinates of the perspective image and point coordinates of the unit sphere, and - a set of spherical points corresponding to the plurality of perspective images is a unit sphere. - ;

Determine a third mapping relationship between the pixel point coordinates of the panoramic image and the pixel point coordinates of the perspective image according to the first mapping relationship and the second mapping relationship, and apply the third mapping relationship to the pixel information of the pixel point of the panoramic image and the third mapping relationship. Accordingly, it is configured to determine pixel information of the pixel point of the perspective image.

In one embodiment, the second feature of the perspective image includes a second feature point and a corresponding second descriptor;

When the acquisition module is configured to determine a first sub-feature of a corresponding position of the panoramic image according to a second feature of the perspective image, specifically,

determine coordinates of a first feature point of the panoramic image according to the coordinates of a second feature point of the perspective image and the third mapping relationship;

and determine a first descriptor corresponding to a first feature point of the panoramic image according to a second descriptor corresponding to a second feature point of the perspective image.

In one embodiment, the first feature includes a first feature point and a corresponding first descriptor;

The matching module specifically,

determining a plurality of groups of image pairs according to the panoramic images to be matched corresponding to each panoramic image;

Determining a plurality of groups of feature point pairs according to a first descriptor of the two panoramic images of the image pair of each group, wherein the feature point pair of each group corresponds to two matching first features belonging to the two panoramic images. Contains points - ;

Determine a first essential matrix according to the feature point pairs of the plurality of groups, perform filtering on the feature point pairs of the plurality of groups using the first essential matrix, and produce a matching result corresponding to the image pair. It is designed to obtain

In one embodiment, after the matching module is configured to determine a plurality of groups of feature point pairs according to the first descriptor of the two panoramic images of the image pair of each group, further,

Obtain the number of feature point pairs of the two panoramic images of the image pairs in each group;

It is configured to filter image pairs in which the number of feature point pairs meets a preset first condition.

In one embodiment, when the matching module is configured to determine a first essential matrix according to the plurality of groups of feature point pairs, specifically,

Determine the angular error of the feature point pair according to the angular error of the two first feature points in the feature point pair, and the angular error of the first feature point is a spherical point of the unit sphere corresponding to the first feature point. - the included angle between the line connecting the optical center of the unit sphere and the external polar plane;

Using the angular error of the corresponding feature point pair as a residual term, an essential matrix is determined several times according to a preset number of feature point pairs;

It is configured to determine the number of internal points corresponding to each essential matrix, and determine the essential matrix with the largest number of internal points as the first essential matrix.

In one embodiment, when the matching module is configured to determine the number of internal points corresponding to each of the essential matrices, specifically,

determine the angular error of the feature point pair of each group of the image pairs according to the essential matrix;

determining a feature point pair whose angle error satisfies a preset second condition as an internal point;

It is configured to determine the number of internal points corresponding to the essential matrix according to all internal points.

In one embodiment, when the matching module is configured to perform filtering on a plurality of groups of feature point pairs using the first essential matrix, specifically,

determine the angular error of the feature point pair of each group of the image pairs according to the first essential matrix;

It is configured to filter feature point pairs whose angle error meets a preset third condition.

In one embodiment, the matching module also:

Two first feature points of the feature point pair of the image pair obtain coordinates in the panoramic image;

According to a third mapping relationship corresponding to the panoramic image and the coordinates of a first feature point belonging to the feature point pair in the panoramic image, determine a perspective image associated with the feature point pair, wherein the perspective image associated with the feature point pair is It is a perspective image in which there is a second feature point corresponding to the first feature point belonging to the feature point pair - ;

It is configured to perform filtering on the image pair using a perspective image related to the feature point pair.

In one embodiment, when the matching module is configured to perform filtering on the image pair using a perspective image related to the feature point pair, specifically,

In response to the perspective image associated with the feature point pair corresponding to at least one panoramic image of the image pair being a plurality of consecutive images, and the number of the plurality of consecutive images being less than a preset filtering threshold, the image pair It is configured to filter.

In one embodiment, the construction module specifically,

Determine that a group of image pairs is an initial image pair according to preset initialization conditions, feature point pairs of each group of image pairs, and a first essential matrix, determine a camera pose of each panoramic image of the initial image pair, and determine the initial image pair. Perform triangulation on the first feature point pair of the image pair to form initial three-dimensional points;

According to the matching relationship between the first feature point corresponding to the first three-dimensional point and the first feature point of each unregistered image, one unregistered image is registered until each panoramic image in the panoramic image set becomes a registered image. The image is determined several times - the unregistered image is a panoramic image in which all first feature points are not triangulated, the registered image is a panoramic image in which first feature points are triangulated, and the first three-dimensional point is the initial three-dimensional point. Includes a point, or includes a three-dimensional point formed by triangulating the initial three-dimensional point and the first feature point of the registered image;

After determining a registered image each time, a camera pose of the registered image is determined, triangulation is performed on the first feature point of the registered image to form a corresponding three-dimensional point, and a third feature point of the registered image is performed. Triangulation is performed to form a corresponding three-dimensional point - the third feature point is a first feature point in the registered image that matches the first feature point of the registered image.

In one embodiment, when the building module is configured to determine that a group of image pairs is an initial image pair according to preset initialization conditions and feature point pairs of each group of image pairs and a first essential matrix, specifically,

Select image pairs sequentially in descending order of the number of feature point pairs, and after each image pair selection, all according to the feature point pair and the first essential matrix, until the selected image pair satisfies the initialization condition. It is configured to determine whether the image pair satisfies the initialization conditions and determine the selected image pair as the initial image pair.

In one embodiment, when the building module is configured to determine whether the image pair satisfies the initialization condition according to the feature point pair and the first essential matrix, specifically:

Determine at least one group of displacement variables according to the first essential matrix of the image pair, triangulate the feature points of each pair of feature points for the displacement variable of each group, and create a three-dimensional point corresponding to the displacement variable of each group. forming, and filtering the three-dimensional points according to the reprojection error and triangulation angle of each group of three-dimensional points, wherein the displacement variable includes a rotation variable and a translation variable;

In response to the number of three-dimensional points in a group having the largest number being greater than a preset first number threshold, determining the corresponding displacement variable as the first displacement variable;

From the essential matrices obtained by calculating multiple times, select an essential matrix where the number of internal points is greater than or equal to the point number threshold, determine the displacement variables of at least one group according to each essential matrix, and determine the characteristics of the displacement variables of each group. Each feature point of a point pair is triangulated to form a three-dimensional point corresponding to the displacement variable of each group, and the three-dimensional points are filtered according to the reprojection error and triangulation angle of the three-dimensional point of each group, and each essential It holds the displacement variable corresponding to the three-dimensional point of the group with the largest number of mattresses;

If the difference between the displacement variable held by each essential matrix and the first displacement variable satisfies a preset range, it is configured to determine that the image pair satisfies the initialization condition.

In one embodiment, the building module also:

Optimize the camera pose and the position of the initial three-dimensional point of each panoramic image through minimizing the reprojection error of the initial three-dimensional point in the two panoramic images of the initial image pair;

After determining the camera pose of the registered image each time, three-dimensional points optimize the camera pose of the registered image through minimizing reprojection error in the registered image;

Each time, triangulation is performed on the feature point of the registered image to form a corresponding three-dimensional point, and triangulation is performed on the third feature point of the registered image to form a corresponding three-dimensional point, and then each three-dimensional point consists of at least one of optimizing the camera pose of each registered image and the position of each three-dimensional point by minimizing the reprojection error in each registered image.

In one embodiment, the matching module also:

determining a corresponding panoramic image to be matched according to a space corresponding to each panoramic image; or

It is configured to determine a panoramic image to be matched corresponding to each panoramic image according to a preset matching rule. Regarding the devices in the above embodiments, the specific form in which each module executes operations has already been described in detail in the embodiments related to the point cloud model construction method in the first aspect, and will not be repeated here.

Referring to FIG. 8, according to a third aspect of an embodiment of the present invention, an electronic device is provided, the electronic device includes a memory 801 and a processor 802, and the memory 801 includes a processor 802. It is for storing computer instructions operable on the computer, and the processor is for building a point cloud model based on the method according to the first aspect when executing the computer instructions. The memory 801 may use volatile or non-volatile storage media.

According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided storing a computer program, wherein the program implements the method according to the first aspect when executed by a processor. Computer-readable storage media may be volatile or non-volatile storage media.

The present invention also provides a computer program product including computer-readable code, and when the computer-readable code is operated in a device, the processor in the device implements the point cloud model building method provided in any of the above-described embodiments. Execute the command to:

The computer program product may be specifically implemented through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically implemented on a computer storage medium, and in another alternative example, the computer program product is specifically implemented in software such as a software development kit (SDK), etc. implemented as a product.

In the present invention, the terms “first” and “second” are only for explaining the purpose and cannot be understood as indicating or implying relative importance. The term “plural” means two or more than two, unless explicitly limited otherwise.

Those skilled in the art will be able to easily think of other implementation methods of the present invention, considering the specification and the invention disclosed in the main text. The present invention is intended to cover any modifications, uses, or adaptations of the present invention, and such modifications, uses, or adaptability changes are in accordance with the general principles of the present disclosure and are not disclosed in the present invention or are known in the art. Includes conventional technical means. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the invention is indicated by the following claims.

It should be understood that the present invention is not limited to the exact structure described above and shown in the drawings, and may be subject to various modifications and changes without departing from its scope. The scope of the present invention is limited only by the appended claims.

Embodiments of the present invention provide a point cloud model building method, device, device, and storage medium, wherein the point cloud model building method includes: acquiring a first feature of a panoramic image in a panoramic image set; determining at least one group of image pairs and a corresponding matching result in the panoramic image set according to the first feature, wherein the image pair includes two panoramic images to which the first feature is matched, and the matching result is: Indicates the correspondence relationship between the first features of two panoramic images - ; and constructing a point cloud model according to the at least one group of image pairs and the corresponding matching result. Since matching was performed using panoramic images and a point cloud model was built with the additional matching results, the number of images in the image set can be reduced, thereby improving matching efficiency and modeling efficiency, and the panoramic image Since the range of the corresponding space is relatively large, the matching effect between panoramic images can be increased, thereby improving the accuracy and quality of point cloud model construction.

Claims (35)

As a method for building a point cloud model,
Applied to electronic devices, the point cloud model construction method is,
Obtaining first features of a panoramic image in a panoramic image set;
determining at least one group of image pairs and a corresponding matching result in the panoramic image set according to the first feature, wherein the image pair includes two panoramic images to which the first feature is matched, and the matching result is: Indicates the correspondence relationship between the first features of two panoramic images - ; and
Constructing a point cloud model according to the at least one group of image pairs and the corresponding matching result;
The step of acquiring a first feature of a panoramic image in the panoramic image set includes:
determining a plurality of perspective images corresponding to the panoramic image, wherein a set of spaces corresponding to the plurality of perspective images is a space corresponding to the panoramic image;
acquiring a second feature of at least one perspective image from the plurality of perspective images;
determining a first sub-feature of a corresponding position of the panoramic image according to a second characteristic of the perspective image, wherein the corresponding position of the perspective image and the panoramic image correspond to the same space; and
A point cloud model building method comprising determining a first feature of the panoramic image according to at least one first sub-feature. delete According to paragraph 1,
The step of determining a plurality of perspective images corresponding to the panoramic image,
Obtaining a unit sphere corresponding to the panoramic image and determining a first mapping relationship between pixel point coordinates of the panoramic image and point coordinates of the unit sphere;
determining a plurality of perspective images according to the unit sphere, and determining a second mapping relationship between pixel point coordinates of the perspective image and point coordinates of the unit sphere - a set of spherical points corresponding to the plurality of perspective images is a unit sphere lim - ; and
Determine a third mapping relationship between the pixel point coordinates of the panoramic image and the pixel point coordinates of the perspective image according to the first mapping relationship and the second mapping relationship, and apply the third mapping relationship to the pixel information of the pixel point of the panoramic image and the third mapping relationship. A point cloud model building method comprising the step of determining pixel information of a pixel point of a perspective image. According to paragraph 3,
The second feature of the perspective image includes a second feature point and a corresponding second descriptor;
Determining a first sub-feature of a corresponding position of the panoramic image according to the second feature of the perspective image includes:
determining coordinates of a first feature point of the panoramic image according to the coordinates of a second feature point of the perspective image and the third mapping relationship; and
A point cloud model building method comprising determining a first descriptor corresponding to a first feature point of the panoramic image according to a second descriptor corresponding to a second feature point of the perspective image. According to paragraph 4,
The first feature includes a first feature point and a corresponding first descriptor;
Determining at least one group of image pairs and a corresponding matching result in the panoramic image set according to the first characteristic comprises:
determining a plurality of groups of image pairs according to the panoramic images to be matched corresponding to each panoramic image;
determining a plurality of groups of feature point pairs according to a first descriptor of the two panoramic images of the image pairs of each group, wherein the feature point pairs of each group are correspondingly matched and belong to two panoramic images; Contains feature points - ; and
Determine a first essential matrix according to the feature point pairs of the plurality of groups, perform filtering on the feature point pairs of the plurality of groups using the first essential matrix, and produce a matching result corresponding to the image pair. A point cloud model building method comprising the steps of obtaining. According to clause 5,
After the step of determining a plurality of groups of feature point pairs according to the first descriptor of the two panoramic images of the image pairs of each group, the step includes:
Obtaining the number of feature point pairs of two panoramic images of the image pair in each group; and
A method for building a point cloud model, further comprising filtering image pairs in which the number of feature point pairs meets a preset first condition. According to claim 5 or 6,
The step of determining a first essential matrix according to the plurality of groups of feature point pairs includes:
Determining an angular error of the feature point pair according to the angular error of two first feature points in the feature point pair, wherein the angular error of the first feature point is a spherical surface of a unit sphere corresponding to the first feature point. - Included angle between the point and the line connecting the optical center of the unit sphere and the external polar plane - ;
determining an essential matrix several times according to a preset number of feature point pairs, using the angular error of the corresponding feature point pair as a residual term; and
A point cloud model building method comprising determining the number of internal points corresponding to each of the essential matrices, and determining the essential matrix with the largest number of internal points as the first essential matrix. In clause 7,
The step of determining the number of internal points corresponding to each of the essential matrices is:
determining an angular error of a feature point pair of each group of the image pairs according to the essential matrix;
determining a feature point pair whose angular error meets a preset second condition as an internal point; and
A method for building a point cloud model, comprising the step of determining the number of internal points corresponding to the essential matrix according to all internal points. In clause 7,
The step of filtering a plurality of groups of feature point pairs using the first essential matrix includes:
determining an angular error of a feature point pair of each group of the image pairs according to the first essential matrix; and
A method for building a point cloud model, comprising filtering feature point pairs whose angular errors meet a preset third condition. According to claim 5 or 6,
The point cloud model construction method is,
obtaining coordinates of two first feature points of the feature point pair of the image pair in a panoramic image;
According to a third mapping relationship corresponding to the panoramic image and the coordinates of a first feature point belonging to the feature point pair in the panoramic image, determining a perspective image associated with the feature point pair - a perspective image associated with the feature point pair is a perspective image in which a second feature point corresponding to the first feature point belonging to the feature point pair exists; and
A point cloud model building method comprising filtering the image pair using a perspective image related to the feature point pair. According to clause 10,
The step of filtering the image pair using a perspective image related to the feature point pair,
In response to the perspective image associated with the feature point pair corresponding to at least one panoramic image of the image pair being a plurality of consecutive images, and the number of the plurality of consecutive images being less than a preset filtering threshold, the image pair A point cloud model building method comprising the step of filtering. According to claim 5 or 6,
The step of building a point cloud model according to the at least one group of image pairs and the corresponding matching result,
Determine that a group of image pairs is an initial image pair according to preset initialization conditions, feature point pairs of each group of image pairs, and a first essential matrix, determine a camera pose of each panoramic image of the initial image pair, and determine the initial image pair. performing triangulation on the first feature point pair of the image pair to form initial three-dimensional points;
According to the matching relationship between the first feature point corresponding to the first three-dimensional point and the first feature point of each unregistered image, one unregistered image is registered until each panoramic image in the panoramic image set becomes a registered image. A step of determining an image multiple times - the unregistered image is a panoramic image in which all first feature points are not triangulated, the registered image is a panoramic image in which first feature points are triangulated, and the first three-dimensional point is the initial Includes a three-dimensional point, or includes a three-dimensional point formed by triangulating the initial three-dimensional point and the first feature point of the registered image; and
After determining a registered image each time, a camera pose of the registered image is determined, triangulation is performed on the first feature point of the registered image to form a corresponding three-dimensional point, and a third feature point of the registered image is performed. Forming a corresponding three-dimensional point by performing triangulation on - wherein the third feature point is a first feature point in the registered image that matches the first feature point of the registered image. How to build a point cloud model. According to clause 12,
The step of determining that a group of image pairs is an initial image pair according to the preset initialization conditions, feature point pairs of each group of image pairs, and a first essential matrix includes:
Select image pairs sequentially in descending order of the number of feature point pairs, and after each image pair selection, all according to the feature point pair and the first essential matrix, until the selected image pair satisfies the initialization condition. A point cloud model building method comprising determining whether an image pair satisfies the initialization condition and determining the selected image pair as an initial image pair. According to clause 13,
According to the feature point pair and the first essential matrix, determining whether the image pair satisfies the initialization condition includes:
Determine at least one group of displacement variables according to the first essential matrix of the image pair, triangulate the feature points of each pair of feature points for the displacement variable of each group, and create a three-dimensional point corresponding to the displacement variable of each group. forming, and filtering the three-dimensional points according to the reprojection error and triangulation angle of each group of three-dimensional points, wherein the displacement variable includes a rotation variable and a translation variable;
In response to the number of three-dimensional points in a group having the largest number being greater than a preset first number threshold, determining a corresponding displacement variable as a first displacement variable;
From the essential matrices obtained by calculating multiple times, select an essential matrix where the number of internal points is greater than or equal to the point number threshold, determine the displacement variables of at least one group according to each essential matrix, and determine the characteristics of the displacement variables of each group. Each feature point of a point pair is triangulated to form a three-dimensional point corresponding to the displacement variable of each group, and the three-dimensional points are filtered according to the reprojection error and triangulation angle of the three-dimensional point of each group, and each essential maintaining a displacement variable corresponding to a three-dimensional point of the group with the largest number of mattresses; and
If the difference between the displacement variable held by each essential matrix and the first displacement variable satisfies a preset range, determining that the image pair satisfies the initialization condition. According to clause 12,
The point cloud model construction method is,
optimizing the camera pose and position of the initial three-dimensional point in each panoramic image through minimizing the reprojection error of the initial three-dimensional point in the two panoramic images of the initial image pair;
After determining the camera pose of the registered image each time, optimizing the camera pose of the registered image by minimizing the reprojection error of the three-dimensional point in the registered image; and
Each time, triangulation is performed on the feature point of the registered image to form a corresponding three-dimensional point, and triangulation is performed on the third feature point of the registered image to form a corresponding three-dimensional point, and then each three-dimensional point A method for building a point cloud model, further comprising at least one of optimizing the camera pose of each registered image and the location of each three-dimensional point by minimizing the reprojection error in each registered image. According to clause 5,
The point cloud model construction method is,
determining a panoramic image to be matched according to a space corresponding to each panoramic image; or
A point cloud model building method comprising the step of determining a panoramic image to be matched corresponding to each panoramic image according to a preset matching rule. As a point cloud model building device,
The point cloud model building device,
an acquisition module configured to acquire first features of a panoramic image in the panoramic image set;
Determine at least one group of image pairs and a corresponding matching result in the panoramic image set according to the first feature, wherein the image pair comprises two panoramic images to which the first feature is matched, and the matching result is two panoramic images. a matching module configured to indicate correspondence between first features of the panoramic image; and
A building module configured to build a point cloud model according to the at least one group of image pairs and the corresponding matching result;
The acquisition module also:
determining a plurality of perspective images corresponding to the panoramic image, wherein a set of spaces corresponding to the plurality of perspective images is a space corresponding to the panoramic image;
acquire a second feature of at least one perspective image in the plurality of perspective images;
determining a first sub-feature of a corresponding position of the panoramic image according to a second characteristic of the perspective image, wherein the corresponding position of the perspective image and the panoramic image correspond to the same space;
A point cloud model building device, characterized in that it is configured to determine a first feature of the panoramic image according to at least one first sub-feature. As an electronic device,
Comprising a memory and a processor, the memory is for storing computer instructions operable in the processor, and the processor executes the computer instruction according to any one of claims 1, 3 to 6. An electronic device characterized for implementing a construction method. A computer-readable storage medium, comprising:
A computer-readable storage medium that stores a computer program and implements the point cloud model building method according to any one of claims 1, 3 to 6 when the program is executed by a processor. A computer program recorded on a computer-readable storage medium,
It includes a computer-readable code, and when the computer-readable code is operated in an electronic device, the processor in the electronic device is a method of building a point cloud model according to any one of claims 1, 3 to 6. A computer program characterized by implementing. delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete