KR20220125714A - Methods, devices, electronic devices, storage media and programs for building point cloud models - Google Patents

Abstract

An embodiment of the present invention relates to a method, apparatus, apparatus and storage medium for building a point cloud model, the method comprising: acquiring a first characteristic of a panoramic image in a panoramic image set; determining, according to the first characteristic, at least one group of image pairs in the panoramic image set and a corresponding matching result, wherein the image pair includes two panoramic images to which the first characteristic is matched, the matching result is indicating the correspondence between the first features of the two panoramic images; and constructing a point cloud model according to the at least one group of image pairs and a corresponding matching result. Since the matching was performed using the panoramic image and a point cloud model was additionally built with the matched result, the number of images in the image set can be reduced, thereby improving the matching efficiency and modeling efficiency, and Since the range of the corresponding space is relatively large, it is possible to increase the matching effect between the panoramic images, thereby improving the accuracy and quality of the point cloud model construction.

Description

Methods, devices, electronic devices, storage media and programs for building point cloud models

Cross-referencing of related applications

The present invention has been filed based on a Chinese patent application with an application number of 202110240320.4, an application date of March 4, 2021, and a patent title of “Method for constructing point cloud model, apparatus, electronic device, and storage medium”, said China The priority of the patent application is claimed, and all contents of the above Chinese patent application are incorporated herein by reference.

The present invention relates to three-dimensional modeling, and to a method, an apparatus, an electronic device, a computer storage medium, and a computer program for constructing a point cloud model, but is not limited thereto.

With the development of artificial intelligence technology, spatial modeling technology becomes more and more rich and more accurate. The motion recovery structure technology can build a three-dimensional point cloud model using a plurality of images, which can be widely used in fields such as physical space digitization, high-precision map construction, and augmented reality. Therefore, it is very important to improve the accuracy and quality of point cloud model construction in the field of 3D modeling technology.

An embodiment of the present invention provides a method, an apparatus, an electronic device, a computer storage medium, and a computer program for building a point cloud model.

According to a first aspect of an embodiment of the present invention, there is provided a point cloud model building method applied to an electronic device, the point cloud model building method comprising:

obtaining a first characteristic of a panoramic image in the panoramic image set;

determining, according to the first characteristic, at least one group of image pairs in the panoramic image set and a corresponding matching result, wherein the image pair includes two panoramic images to which the first characteristic is matched, the matching result is indicating the correspondence between the first features of the two panoramic images; and

and constructing a point cloud model according to the at least one group of image pairs and a corresponding matching result.

In an embodiment, obtaining a first characteristic of a panoramic image in the panoramic image set comprises:

determining a plurality of perspective images corresponding to the panoramic image, wherein a set of spaces corresponding to the plurality of perspective images is a space corresponding to the panoramic image;

acquiring a second characteristic of at least one perspective image in the plurality of perspective images;

determining a first sub-feature of a corresponding position of the panoramic image according to a second characteristic of the perspective image, wherein the corresponding position of the perspective image and the panoramic image correspond to the same space; and

and determining a first characteristic of the panoramic image according to at least one of the first sub-features.

In one embodiment, the step of determining a plurality of perspective images corresponding to the panoramic image,

obtaining a unit sphere corresponding to the panoramic image, and determining a first mapping relationship between pixel point coordinates of the panoramic image and point coordinates of the unit sphere;

determining a plurality of perspective images according to the unit sphere, and determining a second mapping relationship between pixel point coordinates of the perspective image and point coordinates of the unit sphere - A set of spherical points corresponding to the plurality of perspective images is a unit sphere lim - ; and

A third mapping relationship between pixel point coordinates of a panoramic image and pixel point coordinates of a perspective image is determined according to the first mapping relationship and the second mapping relationship, and the pixel information of the pixel points of the panoramic image and the third mapping relationship and determining the pixel information of the pixel points of the perspective image according to the method.

In an embodiment, the second feature of the perspective image comprises a second feature point and a corresponding second descriptor;

Determining the first sub-feature of the corresponding position of the panoramic image according to the second feature of the perspective image comprises:

determining the coordinates of the first feature point of the panoramic image according to the third mapping relationship and the coordinates of the second feature point of the perspective image; and

and determining a first descriptor corresponding to the first feature point of the panoramic image according to a second descriptor corresponding to the second feature point of the perspective image.

In one embodiment, the first characteristic comprises a first characteristic point and a corresponding first descriptor;

The step of determining at least one group of image pairs and a corresponding matching result in the panoramic image set according to the first feature comprises:

determining a plurality of groups of image pairs according to a panoramic image to be matched corresponding to each panoramic image;

determining a plurality of groups of feature point pairs according to a first descriptor of the two panoramic images of the image pair, wherein the feature point pairs of each group have two corresponding matching and first feature points belonging to the two panoramic images Included - ; and

A first essential matrix is determined according to the pair of feature points of the plurality of groups, and filtering is performed on the pair of feature points of the plurality of groups using the first essential matrix to obtain a matching result corresponding to the image pair. step to obtain.

In an embodiment, after determining a pair of feature points in a plurality of groups according to a first descriptor of two panoramic images of the pair of images in each group, the step comprises:

obtaining the number of feature point pairs of the two panoramic images of the image pair; and

The method further includes filtering image pairs in which the number of feature point pairs satisfies a predetermined first condition.

In one embodiment, the step of determining the first essential matrix according to the feature point pair of the plurality of groups,

determining the angular error of the feature point pair according to the angular error of the two first feature points in the feature point pair - The angular error of the first feature point is a spherical surface of a unit sphere corresponding to the first feature point The included angle between the line connecting the point and the optical center of the unit sphere and the outer polar plane - ;

determining an essential matrix multiple times according to a preset number of the feature point pairs by using the angular error of the corresponding feature point pair as a residual term; and

and determining the number of internal points of each of the essential matrices, and determining the essential matrix having the largest number of internal points as the first essential matrix.

In one embodiment, the step of determining the number of internal points corresponding to each of the essential matrix,

determining an angular error of a feature point pair of each group of the image pair according to the essential matrix;

determining, as an internal point, a pair of feature points in which the angular error satisfies a predetermined second condition; and

and determining the number of internal points corresponding to the essential matrix according to all internal points.

In one embodiment, the step of performing filtering on a pair of feature points in a plurality of groups using the first essential matrix,

determining an angular error of a feature point pair of each group of the image pair according to the first essential matrix; and

and filtering the feature point pairs in which the angular error meets a preset third condition.

In one embodiment, the point cloud model building method comprises:

obtaining coordinates in a panoramic image of two first feature points of a feature point pair of the image pair;

determining a perspective image associated with a feature point pair according to a third mapping relationship corresponding to the panoramic image and coordinates of a first feature point belonging to the feature point pair in the panoramic image - a perspective image associated with the feature point pair is a perspective image in which a second feature point corresponding to a first feature point belonging to a feature point pair exists; and

The method further includes performing filtering on the image pair by using the perspective image associated with the feature point pair.

In one embodiment, the step of filtering the image pair by using the perspective image related to the feature point pair,

In response to a perspective image associated with a feature point pair corresponding to at least one panoramic image of the image pair being a plurality of consecutive images, and the number of the plurality of consecutive images being smaller than a preset filtering threshold, the image pair filtering the .

In an embodiment, the step of constructing a point cloud model according to the at least one group of image pairs and a corresponding matching result comprises:

Determine that a group of image pairs is an initial image pair according to a preset initialization condition and a feature point pair and a first essential matrix of each group of image pairs, and determine a camera pose of each panoramic image of the initial image pair, the initial performing triangulation on the first feature point pair of the image pair to form an initial three-dimensional point;

According to the matching relationship between the first feature point corresponding to the first three-dimensional point and the first feature point of each unregistered image, one unregistered image is registered until each panoramic image in the panoramic image set becomes a registered image. determining as an image multiple times - the unregistered image is a panoramic image in which all first feature points are not triangulated, the registered image is a panoramic image in which the first feature point is triangulated, and the first three-dimensional point is the initial including a three-dimensional point, or including a three-dimensional point formed by triangulating the initial three-dimensional point and a first feature point of the registered image; and

After each registration image is determined, the camera pose of the registered image is determined, triangulation is performed on the first feature point of the registered image to form a corresponding three-dimensional point, and a third feature point of the registered image performing triangulation to form a corresponding three-dimensional point, wherein the third feature point is a first feature point matching the first feature point of the registered image in the registered image.

In one embodiment, the step of determining that a group of image pairs is an initial image pair according to the preset initialization condition and a first essential matrix and a feature point pair of each group of image pairs comprises:

Select image pairs sequentially according to the descending order of the number of feature point pairs, and after each image pair selection, all according to the feature point pair and the first essential matrix, until the selected image pair satisfies the initialization condition, the determining whether the image pair satisfies the initialization condition, and determining the selected image pair as the initial image pair.

In one embodiment, according to the feature point pair and the first essential matrix, determining whether the image pair satisfies the initialization condition comprises:

Determining at least one group of displacement variables according to the first essential matrix of the image pair, triangulating the characteristic points of the pair of characteristic points for each group of displacement variables, respectively, to obtain three-dimensional points corresponding to the displacement parameters of each group forming, and filtering the three-dimensional points according to a triangulation angle and a reprojection error of each group of three-dimensional points, wherein the displacement variable includes a rotation variable and a translational variable;

determining a corresponding displacement variable as a first displacement variable in response to the number of three-dimensional points in a group having the largest number being greater than a predetermined first number threshold;

An essential matrix in which the number of internal points is greater than or equal to the point count threshold is selected from the essential matrix obtained by multiple calculations, and at least one group of displacement variables is determined according to each essential cell matrix, respectively, and a characteristic for each group of displacement variables Each of the feature points of the point pair is triangulated to form a three-dimensional point corresponding to the displacement variable of each group, and the three-dimensional point is filtered according to the re-projection error and triangulation angle of the three-dimensional point of each group, each essential retaining a displacement variable corresponding to the three-dimensional point of the group having the largest number of mattresses; and

and determining that the image pair satisfies the initialization condition when a difference between the displacement variable held by each essential matrix and the first displacement variable satisfies a preset range.

In one embodiment, the point cloud model building method comprises:

optimizing the camera pose and the position of the initial three-dimensional point of each panoramic image, such that the initial three-dimensional point minimizes the re-projection error in the two panoramic images of the initial image pair;

after determining the camera pose of the registered image each time, optimizing the camera pose of the registered image through the three-dimensional point minimizing the re-projection error in the registered image; and

After triangulating the feature points of the registered image each time to form a corresponding three-dimensional point, and performing triangulation on the third feature point of the registered image to form a corresponding three-dimensional point, each three-dimensional point and optimizing a camera pose of each registered image and a position of each three-dimensional point through minimizing a re-projection error in each registered image.

In one embodiment, the point cloud model building method comprises:

determining a panoramic image to be matched according to a space corresponding to each panoramic image; or

and determining a panoramic image to be matched corresponding to each panoramic image according to a preset matching rule.

According to a second aspect of an embodiment of the present invention, there is provided a point cloud model building apparatus, the point cloud model building apparatus comprising:

an acquiring module, configured to acquire a first characteristic of a panoramic image in the panoramic image set;

determine, according to the first characteristic, at least one group of image pairs and a corresponding matching result in the panoramic image set, wherein the image pair includes two panoramic images to which the first characteristic is matched, and wherein the matching result is two a matching module configured to indicate a correspondence between the first features of the panoramic image; and

and a building module, configured to build a point cloud model according to the at least one group of image pairs and a corresponding matching result.

In one embodiment, the acquiring module specifically,

determine a plurality of perspective images corresponding to the panoramic image, wherein a set of spaces corresponding to the plurality of perspective images is a space corresponding to the panoramic image;

obtain a second characteristic of at least one perspective image in the plurality of perspective images;

determine a first sub-feature of a corresponding position of the panoramic image according to a second characteristic of the perspective image, wherein the corresponding position of the perspective image and the panoramic image correspond to the same space;

and determine a first characteristic of the panoramic image according to at least one of the first sub-features.

In an embodiment, when the acquisition module is configured to determine a plurality of perspective images corresponding to the panoramic image, specifically,

acquiring a unit sphere corresponding to the panoramic image, and determining a first mapping relationship between pixel point coordinates of the panoramic image and point coordinates of the unit sphere;

determine a plurality of perspective images according to the unit sphere, and determine a second mapping relationship between pixel point coordinates of the perspective image and point coordinates of the unit sphere; a set of spherical points corresponding to the plurality of perspective images is a unit sphere - ;

A third mapping relationship between pixel point coordinates of a panoramic image and pixel point coordinates of a perspective image is determined according to the first mapping relationship and the second mapping relationship, and the pixel information of the pixel points of the panoramic image and the third mapping relationship and determine the pixel information of the pixel point of the perspective image according to it.

In an embodiment, the second feature of the perspective image comprises a second feature point and a corresponding second descriptor;

When the acquiring module is configured to determine the first sub-feature of the corresponding position of the panoramic image according to the second feature of the perspective image, specifically,

determine the coordinates of the first feature point of the panoramic image according to the third mapping relationship and the coordinates of the second feature point of the perspective image;

and determine a first descriptor corresponding to the first feature point of the panoramic image according to a second descriptor corresponding to the second feature point of the perspective image.

In one embodiment, the first characteristic comprises a first characteristic point and a corresponding first descriptor;

The matching module is specifically,

determining a plurality of groups of image pairs according to the panoramic images to be matched corresponding to each panoramic image;

determine a plurality of groups of feature point pairs according to a first descriptor of the two panoramic images of the image pairs of each group, wherein the feature point pairs of each group are two correspondingly matched first features belonging to the two panoramic images including points - ;

A first essential matrix is determined according to the pair of feature points of the plurality of groups, and filtering is performed on the pair of feature points of the plurality of groups using the first essential matrix to obtain a matching result corresponding to the image pair. configured to get

In an embodiment, after the matching module is configured to determine a plurality of groups of feature point pairs according to a first descriptor of two panoramic images of the image pairs of each group, further,

obtaining the number of feature point pairs of the two panoramic images of the image pairs of each group;

The number of feature point pairs is configured to filter image pairs that meet a first preset condition.

In an embodiment, when the matching module is configured to determine the first essential matrix according to the plurality of groups of feature point pairs, specifically,

determine the angular error of the feature point pair according to the angular error of the two first feature points in the feature point pair, wherein the angular error of the first feature point is a spherical point of a unit sphere corresponding to the first feature point - is the included angle between the line connecting the optical center of the unit sphere and the outer polar plane;

determining an essential matrix multiple times according to a preset number of the feature point pairs by using the angular error of the corresponding feature point pair as a residual term;

It is configured to determine the number of internal points corresponding to each of the essential matrices, and to determine the essential matrix having the largest number of internal points as the first essential matrix.

In one embodiment, when the matching module is configured to determine the number of internal points corresponding to each of the essential matrix, specifically,

determine an angular error of a feature point pair of each group of the image pair according to the essential matrix;

determining a pair of feature points in which the angular error satisfies a predetermined second condition as an internal point;

and to determine the number of internal points corresponding to the essential matrix according to all internal points.

In one embodiment, when the matching module is configured to perform filtering on a plurality of groups of feature point pairs using the first essential matrix, specifically,

determine an angular error of a feature point pair of each group of the image pair according to the first essential matrix;

and the angular error is configured to filter feature point pairs that meet a third preset condition.

In one embodiment, the matching module is also

two first feature points of the feature point pair of the image pair acquire coordinates in the panoramic image;

determine a perspective image associated with a feature point pair according to a third mapping relationship corresponding to the panoramic image and coordinates of a first feature point belonging to the feature point pair in the panoramic image, wherein the perspective image associated with the feature point pair comprises: a perspective image in which a second feature point corresponding to a first feature point belonging to a feature point pair exists;

and perform filtering on the image pair by using the perspective image associated with the feature point pair.

In an embodiment, when the matching module is configured to perform filtering on the image pair using a perspective image related to the feature point pair, specifically,

In response to a perspective image associated with a feature point pair corresponding to at least one panoramic image of the image pair being a plurality of consecutive images, and the number of the plurality of consecutive images being smaller than a preset filtering threshold, the image pair is configured to filter.

In one embodiment, the building module specifically,

Determine that a group of image pairs is an initial image pair according to a preset initialization condition and a feature point pair and a first essential matrix of each group of image pairs, and determine a camera pose of each panoramic image of the initial image pair, the initial triangulate the first pair of feature points of the image pair to form an initial three-dimensional point;

According to the matching relationship between the first feature point corresponding to the first three-dimensional point and the first feature point of each unregistered image, one unregistered image is registered until each panoramic image in the panoramic image set becomes a registered image. determine multiple times as an image - the unregistered image is a panoramic image in which all first feature points are not triangulated, the registered image is a panoramic image in which the first feature point is triangulated, and the first three-dimensional point is the initial three-dimensional including a point, or including a three-dimensional point formed by triangulating the initial three-dimensional point and a first feature point of the registered image;

After each registration image is determined, the camera pose of the registered image is determined, triangulation is performed on the first feature point of the registered image to form a corresponding three-dimensional point, and a third feature point of the registered image and triangulate to form a corresponding three-dimensional point, wherein the third feature point is a first feature point matching the first feature point of the registered image in the registered image.

In an embodiment, when the building module is configured to determine that a group of image pairs is an initial image pair according to a preset initialization condition and a first essential matrix and a feature point pair of each group of image pairs, specifically,

Select image pairs sequentially according to the descending order of the number of feature point pairs, and after each image pair selection, all according to the feature point pair and the first essential matrix, until the selected image pair satisfies the initialization condition, the and determine whether the image pair satisfies the initialization condition, and determine the selected image pair as the initial image pair.

In an embodiment, when the building module is configured to determine, according to the feature point pair and the first essential matrix, whether the image pair satisfies the initialization condition, specifically

Determining at least one group of displacement variables according to the first essential matrix of the image pair, triangulating the characteristic points of the pair of characteristic points for each group of displacement variables, respectively, to obtain three-dimensional points corresponding to the displacement parameters of each group forming, and filtering the three-dimensional points according to a triangulation angle and a reprojection error of each group of three-dimensional points, wherein the displacement variable includes a rotation variable and a translation variable;

in response to the number of three-dimensional points in one group having the highest number being greater than a first preset number threshold, determine a corresponding displacement variable as the first displacement variable;

An essential matrix in which the number of internal points is greater than or equal to the point count threshold is selected from the essential matrix obtained by multiple calculations, and at least one group of displacement variables is determined according to each essential cell matrix, respectively, and a characteristic for each group of displacement variables Each of the feature points of the point pair is triangulated to form a three-dimensional point corresponding to the displacement variable of each group, and the three-dimensional point is filtered according to the re-projection error and triangulation angle of the three-dimensional point of each group, each essential a displacement variable corresponding to the three-dimensional point of the group having the largest number of mattresses;

and determining that the image pair satisfies the initialization condition when a difference between the displacement variable held by each essential matrix and the first displacement variable satisfies a preset range.

In one embodiment, the building module further comprises:

optimizing the camera pose and position of the initial three-dimensional point of each panoramic image, through which the initial three-dimensional point minimizes the re-projection error in the two panoramic images of the initial image pair;

after determining the camera pose of the registered image each time, optimizing the camera pose of the registered image through a three-dimensional point minimizing the re-projection error in the registered image;

After triangulating the feature points of the registered image each time to form a corresponding three-dimensional point, and performing triangulation on the third feature point of the registered image to form a corresponding three-dimensional point, each three-dimensional point is configured to perform at least one of optimizing a camera pose of each registered image and a position of each three-dimensional point through minimizing a reprojection error in each registered image.

In one embodiment, the matching module is also

determine a corresponding panoramic image to be matched according to a space corresponding to each panoramic image; or

and determine a panoramic image to be matched corresponding to each panoramic image according to a preset matching rule.

According to a third aspect of an embodiment of the present invention, there is provided an electronic device, the device comprising a memory and a processor, the memory for storing computer instructions operable on the processor, the processor to execute the computer instructions case to build a point cloud model based on the method according to the first aspect.

According to a fourth aspect of an embodiment of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method according to the first aspect.

According to a fifth aspect of an embodiment of the present invention, there is provided a computer program comprising computer readable code, wherein when the computer readable code is run in an electronic device, the processor of the electronic device performs any one of the above methods to implement

As is known according to the above embodiment, a panoramic image set is constructed from the panoramic image, and the panoramic image in the panoramic image set is further matched according to the first feature, and two panoramic images for which the first feature matching is completed are combined into one. By using as a group image pair of , at least one group of image pairs and a corresponding matching result are determined, and a point cloud model is built according to the finally determined at least one group of image pairs and a corresponding matching result. Since the matching was performed using the panoramic image and a point cloud model was additionally built with the matched result, the number of images in the image set can be reduced, thereby improving the matching efficiency and modeling efficiency, and Since the range of the corresponding space is relatively large, the matching effect between the panoramic images can be increased, thereby improving the accuracy and mass of the point cloud model construction.

It should be understood that the above general description and the following detailed description are illustrative and interpretative only, and are not intended to limit the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention, and together with this specification serve to explain the principles of the invention.
1 is a flowchart of a method for constructing a point cloud model according to an embodiment of the present invention.
2 is an exemplary view of a panoramic image according to an embodiment of the present invention.
3 is an exemplary diagram of a method of acquiring a first feature of a panoramic image according to an embodiment of the present invention.
4 is an exemplary diagram of a method of determining an image pair and a corresponding matching result according to an embodiment of the present invention.
5 is an exemplary diagram of an angle error according to an embodiment of the present invention.
6 is an exemplary diagram of a method of comparing an initialization condition, a feature point pair, and a first essential matrix according to an embodiment of the present invention.
7 is a structural diagram of an apparatus for constructing a point cloud model according to an embodiment of the present invention.
8 is a structural diagram of an electronic device according to an embodiment of the present invention.

Exemplary embodiments will be described in detail below, examples of which are shown in the drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS When the following description refers to drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The embodiments described in the illustrative examples below do not represent all embodiments consistent with the present invention. To the contrary, these are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the claims.

The terminology used in the present invention is only for describing specific embodiments, and is not intended to limit the present invention. As used in the specification and claims of the present invention, the singular forms "a thing" and "the said" include the plural forms unless the text clearly indicates another meaning. It should also be understood that the term " and/or” means and includes any or all possible combinations of one or more related listed items.

It should be understood that, although the present invention uses terms such as “first”, “second”, “third” and the like to describe various pieces of information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may be referred to as second information without departing from the scope of the present invention; Similarly, the second information may also be referred to as first information. Depending on the context, the term "...if" as used in the text may be understood to mean "when..." or "when..." or "in response to a decision" depending on the context. have.

With the development of artificial intelligence technology, spatial modeling technology becomes more and more rich and more accurate. The motion recovery structure technology can build a three-dimensional point cloud model using a plurality of images, which can be widely used in fields such as physical space digitization, high-precision map construction, and augmented reality. In the existing technology, the matching efficiency between a plurality of images is not good, so that the accuracy of the sequentially constructed point cloud model is low and the quality is deteriorated.

Based on this, in a first aspect, at least one embodiment of the present invention provides a method for building a point cloud model, and with reference to FIG. 1 , the flow of the method including steps S101 to S103 is shown.

Here, the point cloud model may be a three-dimensional model corresponding to space, and space may mean the real world. Each object in the real world is represented by a point cloud corresponding to the model, and the point cloud means a set of points composed of three-dimensional points.

In addition, the point cloud model building method may be executed by an electronic device such as a terminal device or a server, and the terminal device is a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a wireless telephone, and personal information. It may be a terminal (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle device, a wearable device, etc., and the method for building the point cloud model may be implemented in such a way that computer readable instructions stored in the memory are called by the processor. have. Alternatively, the point cloud model building method may be executed through a server, and the server may be a local server, a cloud server, or the like.

In step S101, a first characteristic of a panoramic image in the panoramic image set is acquired.

Here, the panoramic image set includes a plurality of panoramic images, and the panoramic image may be, for example, a panoramic image of various angles, such as a 360° panoramic image; The panoramic image may be an image obtained by a spherical camera or an image obtained by using an equidistant rectangular projection method using splicing images obtained by a plurality of fisheye cameras. Referring to FIG. 2 , one panoramic image is illustratively do. Each panoramic image corresponds to one local subspace of the modeling space, the size of the local subspace corresponding to the panoramic image is greater than the size of the local subspace corresponding to the general image in the same parameter, and all of the panoramic image sets The local subspace corresponding to the panoramic image may constitute the entire modeling space; Local subspaces corresponding to different panoramic images may have overlapping regions.

In this step, the first feature of the panoramic image may be acquired using a pre-trained neural network, and the first feature of the panoramic image may be acquired using another method, and the present invention provides a specific method for acquiring not limited to A first characteristic of each panoramic image in the panoramic image set may be acquired.

In step S102, at least one group of image pairs and a corresponding matching result in the panoramic image set are determined according to the first characteristic, wherein the image pair includes two panoramic images to which the first characteristic is matched. and the matching result indicates a correspondence relationship between the first features of the two panoramic images.

Here, the first feature matching means that the first feature point corresponding to the two panoramic images in the same space exists, that is, at least one first feature point of one panoramic image and at least one of the other panoramic image. It means that the first feature point corresponds to the same space in the real world.

In this step, matching may be performed on the panoramic images in the panoramic image set using the indiscriminate matching method, in other words, matching may be performed using the matching method individually, for example, sequentially By selecting, matching can be performed with each other panoramic image; When matching two panoramic images, the matching is performed using the first feature of the two panoramic images, that is, the matching result of the panoramic image is determined according to the matching result of the first feature of the two panoramic images. When the first feature matching of the two panoramic images is completed, that is, when the two panoramic images complete feature matching, the two images may be determined as a group of image pairs, and the first features of the two panoramic images at the same time may be determined as a matching result.

Here, each panoramic image may constitute another panoramic image and an image pair, and may constitute a plurality of different panoramic images and a plurality of groups of image pairs, that is, after each panoramic image constitutes an image pair , if not locked, you can go ahead and construct a new image pair with another panoramic image.

In step S103, a point cloud model is constructed according to the at least one group of image pairs and a corresponding matching result.

In this step, the point cloud model is constructed using the matching result in step S102 described above. The modeling process, that is, the motion recovery structure process, includes camera registration and point cloud reconstruction, wherein camera registration can be represented using camera motion parameters (e.g., camera poses) of each one panoramic image within a set of panoramic images. exist), and the point cloud reconstruction is to restore the point cloud of a three-dimensional structure of the corresponding local subspace (that is, the local subspace proposed in step S102).

In one example, a point cloud model may be built using a method of progressive reconstruction.

In an embodiment of the present invention, a panoramic image set is configured through the panoramic image, and the panoramic image in the panoramic image set is further matched according to the first feature, and two panoramic images for which feature matching is completed are combined into one group image. By using the pair, at least one group of image pairs and a corresponding matching result are determined, and a point cloud model is built according to the finally determined at least one group of image pairs and the corresponding matching result. Since the matching was performed using the panoramic image and a point cloud model was additionally built with the matched result, the number of images in the image set can be reduced, thereby improving the matching efficiency and modeling efficiency, and Since the range of the corresponding space is relatively large, it is possible to increase the matching effect between the panoramic images, thereby improving the accuracy and quality of the point cloud model construction.

In some embodiments of the present invention, the first characteristic of the panoramic image of the panoramic image set may be acquired using the following scheme, and with reference to FIG. 3 , the flow of the acquisition scheme is illustrated, steps S301 to S304 includes

In step S301, a plurality of perspective images corresponding to the panoramic image is determined, wherein a set of spaces corresponding to the plurality of perspective images is a space corresponding to the panoramic image.

In one example, a plurality of perspective images may be determined using the following method, first, a unit sphere corresponding to the panoramic image is obtained, and a first mapping between pixel point coordinates of the panoramic image and point coordinates of the unit sphere determine the relationship; Next, a plurality of perspective images are determined according to the unit sphere, and a second mapping relationship between pixel point coordinates of the perspective image and point coordinates of the unit sphere is determined - a set of spherical points corresponding to the plurality of perspective images is a unit Spherical - ; Finally, according to the first mapping relationship and the second mapping relationship, a third mapping relationship between the pixel point coordinates of the panoramic image and the pixel point coordinates of the perspective image is determined, and the pixel information of the pixel points of the panoramic image and the third mapping relationship are determined. Determine the pixel information of the pixel point of the perspective image according to the mapping relationship.

Here, the obtaining of the unit sphere corresponding to the panoramic image may be backward-projecting the panoramic image onto the unit sphere. Determining the plurality of perspective images according to the unit sphere may be that a virtual perspective camera with the center of the unit sphere as an optical center shoots the unit sphere, and determines the obtained image as a perspective image, and when the unit sphere is taken, Let the virtual camera rotate according to a certain angle so that one perspective image is taken with each rotation, and the entire sphere is photographed until a plurality of perspective images covering all views are obtained, for example, If one perspective image is taken by rotating 60° each time, six perspective images must be taken to capture all spheres; By setting the field of view (FOV) and focal length of the virtual camera as parameters of a general image acquisition device (eg, a mobile phone, a digital camera, etc.), it is possible to approximate an image acquired by a general image acquisition device.

The first mapping relationship between the pixel point coordinates of the panoramic image and the point coordinates of the unit sphere may be expressed according to the following formula (1).

(1)

(One)

는 단위 구면의 포인트 좌표이고,

는 파노라마 이미지의 픽셀 포인트 좌표이며, w는 파노라마 이미지의 너비이며, h는 파노라마 이미지의 높이이다.here,

is the point coordinates of the unit sphere,

is the pixel point coordinates of the panoramic image, w is the width of the panoramic image, and h is the height of the panoramic image.

The second mapping relationship between the pixel point coordinates of the perspective image acquired with the same virtual camera as the coordinate system of the unit sphere and the point coordinates of the unit sphere can be expressed according to formula (2) below,

(2)

(2)

는 단위 구면의 포인트 좌표이고,

는 투시도 이미지의 픽셀 포인트 좌표이며, w~는 투시 이미지의 너비이며, h~는 투시 이미지의 높이이며, f는 가상 카메라의 초점 거리이다.here,

is the point coordinates of the unit sphere,

is the pixel point coordinates of the perspective image, w is the width of the perspective image, h is the height of the perspective image, and f is the focal length of the virtual camera.

A second mapping relationship between pixel point coordinates of a perspective image obtained by a virtual camera rotating a predetermined angle based on the coordinate system of the unit sphere and the point coordinates of the unit sphere may be expressed according to Formula (3) below.

(3)

(3)

는 단위 구면의 좌표 포인트

가 가상 카메라의 역회전 매트릭스 R을 통해 변환된 좌표이고,

는 투시 이미지에서의 좌표이며, w~는 투시 이미지의 너비이며, h~는 투시 이미지의 높이이며, f는 가상 카메라의 초점 거리이다.here,

is the coordinate point of the unit sphere

is the coordinates transformed through the reverse rotation matrix R of the virtual camera,

is the coordinates in the perspective image, w is the width of the perspective image, h is the height of the perspective image, and f is the focal length of the virtual camera.

In an embodiment of the present invention, in a plurality of shooting angles of the virtual camera, one angle and the coordinate system of the unit sphere are the same, and therefore, at the angle, the third between the pixel point coordinates of the panoramic image and the pixel point coordinates of the perspective image The mapping relationship can be obtained by combining the above formulas (1) and (2); At another angle, a third mapping relationship between the pixel point coordinates of the panoramic image and the pixel point coordinates of the perspective image can be obtained by combining the above formulas (1) and (3).

Here, the determining of the pixel information of the pixel point of the perspective image according to the pixel information of the pixel point of the panoramic image and the third mapping relationship comprises converting the pixel information of the pixel point of the panoramic image to the pixel information of the pixel point corresponding to the perspective image. can be directly determined, and pixel information of the perspective image can be obtained through at least one of sampling and bilinear interpolation for pixel points of the panoramic image. The pixel information may be a brightness value of a pixel point or a value of each color channel (eg, red, green, and blue channels) of a pixel point.

In step S302, a second characteristic of at least one perspective image in the plurality of perspective images is acquired.

In this step, the second feature of the perspective image may be extracted using a pre-trained neural network, and the second feature of the perspective image may be extracted using another method. no restrictions A second characteristic of each perspective image corresponding to the panoramic image may be acquired.

In an example, the second feature is a second feature point and a corresponding second descriptor, ie, all second feature points and corresponding second descriptors in the perspective image constitute the second feature of the perspective image.

In step S303, a first sub-feature of a corresponding position of the panoramic image is determined according to a second characteristic of the perspective image, wherein the corresponding position of the perspective image and the panoramic image corresponds to the same space.

Here, the corresponding positions of the perspective image and the panoramic image correspond to the same space, that is, the corresponding positions of the perspective image and the panoramic image correspond to the same set of spherical points in a unit sphere.

Here, the first sub-feature may include all first feature points and corresponding first descriptors in corresponding positions of the panoramic image.

In an example corresponding to the example of step S302, the first sub-feature of the corresponding position of the panoramic image may be determined using the following method, and first, the coordinates of the second feature point of the perspective image and the third mapping relationship determine the coordinates of the first feature point of the panoramic image according to Next, a first descriptor corresponding to the first feature point of the panoramic image is determined according to a second descriptor corresponding to the second feature point of the perspective image.

Here, a point in the panoramic image corresponding to the second feature point is a first feature point, that is, the first feature point and the second feature point correspond to each other or a spherical point of a unit sphere corresponding to the first feature point. and the spherical point of the unit sphere corresponding to the second feature point coincide. The second descriptor corresponding to the second feature point may be directly used as the first descriptor of the corresponding first feature point.

In step S304, a first characteristic of the panoramic image is determined according to at least one of the first sub-features.

Here, the first feature of the panoramic image includes all first feature points in the panoramic image and a corresponding first descriptor.

In an embodiment of the present invention, the mapping relationship between the panoramic image and the perspective image is determined through the mapping relationship between the panoramic image and the unit sphere and the mapping relationship between the unit sphere and the perspective image, that is, the panoramic image is used as the medium of the unit sphere. By dividing into a plurality of perspective images, further extracting second features of the perspective image, and inversely mapping the second feature points to the first feature points of the panoramic image, extraction of the first features of the panoramic image is implemented .

In some embodiments of the present invention, the first feature comprises a first feature point and a corresponding first descriptor, and correspondingly, at least one in the panoramic image set according to the first feature through the following manner. A group image pair and a corresponding matching result may be determined. Referring to FIG. 4 , the flow of the determination method including steps S401 to S403 is exemplified.

In step S401, a plurality of groups of image pairs are determined according to a panoramic image to be matched with each panoramic image.

In this step, two panoramic images are configured as a group of image pairs. A panoramic image to be matched may be determined according to a space corresponding to each panoramic image, or a panoramic image to be matched may be determined according to a preset matching rule. In other words, when determining a panoramic image to be matched with one panoramic image, the overlapping panoramic image existing in the corresponding space of the panoramic image may be used as the to-be-matched panoramic image, and may also be determined according to a preset matching rule, , a preset matching rule may be determined according to the above principle, for example, after numbering the panoramic images according to the corresponding spatial order, and then following a preset number of each panoramic image (for example, 10) of panoramic images can be used as the panoramic image to be matched. In addition, all panoramic images other than one panoramic image may be used as the panoramic image to be matched with the panoramic image.

In step S402, a plurality of groups of feature point pairs are determined according to the first descriptor of the two panoramic images of the image pair, wherein the feature point pairs of each group correspond to two corresponding matching and two panoramic images. and a first feature point belonging to it.

In this step, first, for each first descriptor in the first panoramic image of an image pair, the first descriptor with the closest Euclidean distance in the second panoramic image is searched, and then, conversely, in the second panoramic image of the image pair, the first descriptor is searched for. For each first descriptor, the first descriptor with the closest Euclidean distance in the first panoramic image is retrieved, and the first specified descriptor in the first panoramic image and the specified first descriptor in the second panoramic image are different from each other. If the Euclidean distance in is the closest first descriptor, it is determined that two first feature points corresponding to the two first descriptors are matched by considering that the two first descriptors are matched, that is, two first descriptors are matched. The first feature points of n complete feature matching to form a feature point pair.

In this step, after determining all the feature point pairs between the two panoramic images of the image pair according to the above method, the number of feature point pairs is also statistic, the first condition is preset again, and the first condition is used The plurality of groups of image pairs determined in step S401 may be filtered, that is, some image pairs may be removed using the first condition. In one example, the first condition may be less than the second number threshold, that is, the number of filtered feature point pairs may be less than the image pair of the second number threshold, and the number of feature point pairs removed is the first. It may be smaller than the image pair of the two number threshold, for example, the second number threshold may be set to 5 or 10, etc., the embodiment of the present invention specifically limits the specific value of the second number threshold I never do that. By filtering some image pairs, it is possible to reduce subsequent operations for image pairs with low matching, thereby reducing the complexity of operations and improving processing efficiency.

Here, a pair of feature points in each group indicates a corresponding relationship between two first feature points, and a pair of feature points in a plurality of groups constitutes a matching result of the image pair.

In step S403, a first essential matrix is determined according to the feature point pairs of the plurality of groups, and filtering is performed on the feature point pairs of the plurality of groups using the first essential matrix, so that the image pair is Get the corresponding matching result.

In this step, the first essential matrix may be determined using the following method, and first, the angular error of the feature point pair is determined according to the angular error of the two first feature points in the feature point pair, where , the angular error of the first feature point is the included angle between the line connecting the spherical point of the unit sphere corresponding to the first feature point and the optical center of the unit sphere and the external polar plane, and then, the corresponding feature point pair using the angular error as a residual term to determine an essential matrix multiple times according to a preset number of pairs of the feature points; Finally, the number of internal points corresponding to each of the essential matrices is determined, and an essential matrix having the largest number of internal points is determined as the first essential matrix.

Here, the point coordinates of the unit sphere corresponding to the two first feature points of the feature point pair of each group, and the essential matrix satisfy the relationship shown in Formula (4) below,

(4)

(4)

는 첫 번째 파노라마 이미지에서의 제1 특징 포인트에 대응되는 단위 구면의 포인트 좌표이며,

는 두 번째 파노라마 이미지에서의 제1 특징 포인트에 대응되는 단위 구면의 포인트 좌표이다.In the above formula (4), E is the essential matrix,

is the point coordinate of the unit sphere corresponding to the first feature point in the first panoramic image,

is the point coordinate of the unit sphere corresponding to the first feature point in the second panoramic image.

Here, the relatively large angular error of the two first feature points is determined as the angular error of the feature point pair, that is, the angular error of the feature point pair is determined according to the following formula (5).

(5)

(5)

는 두 번째 파노라마 이미지의 제1 특징 포인트의 각도 오차이고, 다시 말해 두 번째 파노라마 이미지에서 제1 특징 포인트에 대응되는 구면 포인트 S' 및 광학적 중심 O₂을 연결하는 선과 대응되는 외부 극평면간의 끼인각이고, 여기서, 대응되는 외부 극평면은 첫 번째 파노라마 이미지에서의 제1 특징 포인트에 대응되는 구면 포인트 S와 광학적 중심 O₁을 연결하는 선 및 두 개의 광학적 중심 Q₁ 및 Q₂을 연결하는 선으로 구성된 평면이다. 대응되게 도 5를 참고하면,

는 첫 번째 파노라마 이미지의 제1 특징 포인트의 각도 오차이고, 다시 말해 첫 번째 파노라마 이미지에서 제1 특징 포인트에 대응되는 구면 포인트 및 광학적 중심을 연결하는 선과 대응되는 외부 극평면 간의 끼인각이고, 여기서, 대응되는 외부 극평면은 두 번째 파노라마 이미지에서 제1 특징 포인트에 대응되는 구면 포인트와 광학적 중심을 연결하는 선 및 두 개의 광학적 중심을 연결하는 선으로 구성된 평면이다. 본 발명의 실시예에 있어서 각도 오차 및 구면 오차를 사용하여 파노라마 이미지의 카메라 모델에 더 잘 적응할 수 있다.In the above formula (5),

is the angular error of the first feature point of the second panoramic image, that is, the included angle between the line connecting the spherical point S' corresponding to the first feature point and the optical center O ₂ in the second panoramic image and the corresponding external polar plane, , where the corresponding external polar plane consists of a line connecting the optical center O ₁ and the spherical point S corresponding to the first feature point in the first panoramic image, and a line connecting the two optical centers Q ₁ and Q ₂ . It is flat. Correspondingly, referring to Figure 5,

is the angular error of the first feature point of the first panoramic image, that is, the included angle between the line connecting the optical center and the spherical point corresponding to the first feature point in the first panoramic image and the corresponding external polar plane, where The external polar plane to be formed is a plane composed of a line connecting the optical center and the spherical point corresponding to the first feature point in the second panoramic image and a line connecting the two optical centers. The angular error and spherical error can be used in the embodiment of the present invention to better adapt to the camera model of the panoramic image.

Here, a plurality of essential matrices can be calculated through RANSAC (Random Sample Consensus) and a five-point algorithm by using the angular error of the feature point pair as a residual term, and one essential matrix can be calculated for each five group feature point pair. and a plurality of essential matrices can be obtained by using the above method.

Here, the number of internal points corresponding to the essential matrix can be determined according to the following method, and first, the angular error of the feature point pair of each group of the image pair is calculated according to the essential matrix; next, determining a feature point pair in which the angle error satisfies a preset second condition as an internal point; Finally, the number of internal points corresponding to the essential matrix is determined according to all internal points. In other words, using the above formula (5) and the essential matrix to determine the angular error of the pair of feature points in each group; A second condition is preset, and an inner point is selected using the second condition. In an embodiment, the second condition may be that the angle error is smaller than the first angle threshold, that is, the first angle threshold. A feature point pair corresponding to an angular error smaller than a value is determined as an internal point.

In this step, filtering may be performed on the feature point pairs of the plurality of groups using the first essential matrix through the following method, and first, the features of each group of the image pairs according to the first essential matrix determine the angular error of the point pair; Next, a pair of feature points satisfying the third condition in which the angular error is set is determined.

Here, the angular error of the feature point pair of each group can be calculated using the above formula (5) and the first essential matrix; A third condition is preset, and a feature point pair is selected using the third condition, and in an embodiment, the third condition is that an angle error is greater than or equal to a second angle threshold (eg, greater than or equal to 0.4 degrees). Filtering feature point pairs corresponding to angular errors greater than or equal to, that is, greater than or equal to, the second angular threshold, ie, removing feature point pairs corresponding to angular errors greater than or equal to the second angular threshold and a feature point pair corresponding to an angle error smaller than the second angle threshold value.

In an embodiment of the present invention, by performing feature matching on image pairs, a feature point pair between image pairs is determined, and a first essential matrix is further determined according to the feature point pair, and finally the first essential When the feature point pair is filtered with a matrix, the first essential matrix is determined and the feature point pair is filtered, angle error is used for both, so compared to other essential matrices, the feature point pair matching the first essential matrix is the most There are many, and the filtering step removes feature point pairs that do not match the first essential matrix, so not only improves the accuracy of the first essential matrix, but also by maximizing the number of feature point pairs under the premise of removing false feature point pairs. , to improve the matching precision and accuracy of two panoramic images of an image pair. By using the 360 degree visual cover of the panoramic image, the number of feature matching between images is improved, and the possibility of camera registration failure in the weak texture area is reduced.

In some embodiments of the present invention, after completing the first essential matrix calculation and feature point pair filtering of each group of image pairs, the matching of two panoramic images is also overlapped by using the distribution of the first feature points By determining whether or not it occurs due to , it is possible to further filter the image pairs of a plurality of groups, specifically, the following method can be used, and first, two first feature points of the feature point pair of the image pair obtain coordinates in the panoramic image; Next, a perspective image associated with a feature point pair is determined according to a third mapping relationship corresponding to the panoramic image and coordinates of a first feature point belonging to a feature point pair in the panoramic image, wherein the feature point pair and the related perspective image is a perspective image in which a second feature point corresponding to a first feature point belonging to a feature point pair exists; Finally, filtering is performed on the image pair using a perspective image related to the feature point pair.

In addition, the third number threshold may be preset, the number of second feature points corresponding to the feature point pair included in the perspective image may be determined, and when the number is greater than or equal to the third number threshold, the perspective The image may be determined as a perspective image associated with a feature point pair, for example, the third number threshold may be set to 5, thereby avoiding false statistics according to a small amount of noise matching.

Here, the perspective image associated with the feature point pair corresponding to at least one panoramic image of the image pair is a plurality of continuous images, and in response to the number of the plurality of continuous images being smaller than a preset filtering threshold, the Filter image pairs. In other words, if all the second feature points corresponding to the first feature point related to the feature point pair are concentrated on a partial perspective image, and the number of the partial perspective images is less than a preset filtering threshold, A match is an erroneous match due to duplicate textures, and thus filters the image pair, ie removes the image pair. The filtering threshold may be determined according to the total number of perspective images and a preset first ratio, for example, if the total number of perspective images is 6 and the preset first ratio is 0.5, If the number is less than 3, the image pair is filtered.

In an embodiment of the present invention, all the first feature points for which feature matching is completed correspond to the second feature points of the perspective image, and thus the matching of two panoramic images is overlapped by determining the distribution of the second feature points. By judging whether or not it is caused by a texture, and removing noise matching, it is possible to further improve the accuracy of the determination, and to remove image pairs of erroneous matching. Most of the repeated textures are local, and by using the overall matching situation between the panoramic images, erroneous matching is eliminated as much as possible, thereby avoiding the occurrence of an error rate of camera registration.

In some embodiments of the present invention, a point cloud model can be built according to the at least one group of image pairs and a corresponding matching result through the following method, and first, preset initialization conditions and image pairs of each group Determine that a group of image pairs is an initial image pair according to the feature point pair and the first essential matrix, determine a camera pose of each panoramic image of the initial image pair, and triangulate for the first feature point pair of the initial image pair performing an image to form an initial three-dimensional point; Next, according to the matching relationship between the first feature point corresponding to the first three-dimensional point and the first feature point of each unregistered image, until each panoramic image in the panoramic image set becomes a registered image, one unregistered image is determined several times as a registered image, wherein the unregistered image is a panoramic image in which all first feature points are not triangulated, the registered image is a panoramic image in which the first feature point is triangulated, and the first three-dimensional point is a three-dimensional point including the initial three-dimensional point or formed by triangulating the initial three-dimensional point and the first feature point of the registered image; After each registration image is determined, the camera pose of the registered image is determined, triangulation is performed on the first feature point of the registered image to form a corresponding three-dimensional point, and a third feature point of the registered image is triangulated to form a corresponding three-dimensional point, and the third feature point is a first feature point matching the first feature point of the registered image in the registered image.

Here, when determining the initial image pair, image pairs may be sequentially selected according to the descending order of the number of feature point pairs, and after each image pair is selected, all selected images according to the feature point pair and the first essential matrix Until the pair satisfies the initialization condition, it is determined whether the image pair satisfies the initialization condition, and the selected image pair is determined as the initial image pair.

In addition, according to the feature point pair and the first essential matrix, it can be determined whether or not the image pair satisfies the initialization condition using the following method. Referring to FIG. 6 , the flow of the method is illustrated and , including steps S601 to S604.

In step S601, at least one group (for example, four groups) of displacement variables is determined according to the first essential matrix of the image pair, and for each group of displacement variables, the feature points of the feature point pair are triangulated, respectively. to form a three-dimensional point corresponding to a displacement variable of each group, and filter the three-dimensional point according to a reprojection error and a triangulation angle of the three-dimensional point of each group, wherein the displacement variable is a rotation variable and a translation contains variables.

Here, the rotation variable can be represented by a 3*3 matrix R, and the translation variable can be represented by a three-dimensional vector T. If the three-dimensional point X in each group computes the reprojection error in the ith panoramic image (i.e. the first or second panoramic image), the formula below can be used,

(6)

(6)

는 제1 특징 포인트에 대응되는 단위 구면의 포인트 좌표이고, X는 삼차원 포인트 좌표 벡터이며, f는 가상 투시 카메라의 초점 거리이며,

는 i 번째 파노라마 이미지의 카메라 매트릭스이며, 초기화의 경우,

는 단위 매트릭스이고,

는 영 벡터이며,

,

이며, R 및 T는 각각 각 그룹의 삼차원 포인트에 대응되는 회전 변수 및 평행 이동 변수이다.In the above formula,

is the point coordinate of the unit sphere corresponding to the first feature point, X is the three-dimensional point coordinate vector, f is the focal length of the virtual perspective camera,

is the camera matrix of the i-th panoramic image, for initialization,

is the unit matrix,

is the zero vector,

,

where R and T are rotational and translational variables corresponding to three-dimensional points of each group, respectively.

Here, when filtering the three-dimensional points according to the re-projection error and triangulation angle of the three-dimensional points of each group, a third angle threshold value and a fourth angle threshold value are set, and then the re-projection error in two panoramic images may hold that both are smaller than the third angle threshold and the triangulated angle is greater than the three-dimensional point of the fourth angle threshold.

In step S602, a corresponding displacement variable is determined as the first displacement variable in response to the number of three-dimensional points in one group having the highest number being greater than a predetermined first number threshold.

In step S603, an essential matrix in which the number of internal points is greater than or equal to the point number threshold is selected from the essential matrix obtained by multiple calculations, and displacement variables of at least one group (for example, four groups) are selected according to each essential cell matrix. respectively, and triangulate the feature points of the feature point pair for each group of displacement variables to form three-dimensional points corresponding to the displacement variables of each group, and reprojection error and triangulation of the three-dimensional points of each group The three-dimensional point is filtered according to the angle, and a displacement variable corresponding to the three-dimensional point of the group having the largest number of each essential mattress is stored.

Here, after calculating and obtaining a plurality of essential matrices in step S403 and determining the first essential matrix, the remaining essential matrices are retained and used in this step, or in this step, using the same method as in step S403 again, a plurality of essentials It is obtained by calculating the matrix.

Here, the second ratio may be preset, and then the number of points inside the first essential matrix and the second ratio are used to determine the number of points threshold; The product of the number of internal points of the first essential matrix and the second ratio may be used as a threshold value for the number of points, for example, the second ratio may be preset to 0.6, and the present invention is limited to a specific value of the second ratio I never do that.

The operation performed on the essential matrix selected in this step is the same as the operation in steps S601 to S602, and one displacement vector is held for each essential matrix.

In step S604, when the difference between the displacement variable held by each essential matrix and the first displacement variable satisfies a preset range, it is determined that the image pair satisfies the initialization condition.

Here, the difference between the displacement variable held by the essential matrix and the first displacement variable may be expressed using the direction angles of the two displacement variables, the direction angle being obtained as a product of the rotation matrix of the two displacement variables; The preset range may be displayed using one preset fifth angle threshold, that is, if it is less than the fifth angle threshold, the preset range is satisfied; Accordingly, if the orientation angles of the position vector and the first displacement vector held by each essential matrix are both smaller than the fifth angle threshold, it is determined that the image pair satisfies the initialization condition.

If the image pair does not satisfy the initialization condition, steps S601 to S604 are used to determine whether another image pair satisfies the initialization condition or not.

In the process of judging the initialization condition, the determination regarding a plurality of analysis of each essential matrix makes the determination result of the initialization condition more stable.

In addition, the initial three-dimensional point minimizes the re-projection error in the two panoramic images of the initial image pair, so that the camera pose of each panoramic image and the position of the initial three-dimensional point can be optimized;

In addition, after determining the camera pose of the registered image each time, the three-dimensional point may optimize the camera pose of the registered image by minimizing the re-projection error in the registered image;

In addition, each time, after triangulating the feature points of the registered image to form a corresponding three-dimensional point, and performing triangulation on the third feature point of the registered image to form a corresponding three-dimensional point, also By minimizing the re-projection error of each three-dimensional point in each registered image, the camera pose of each registered image and the position of each three-dimensional point can be optimized.

The optimization can be performed using the following loss function,

(7)

(7)

는 제1 특징 포인트에 대응되는 단위 구면의 포인트 좌표이고,

는 i 번?? 삼차원 포인트 좌표의 벡터이며, f는 가상 투시 카메라의 초점 거리이며,

는 i 번째 파노라마 이미지의 카메라 매트릭스이며, 초기화의 경우,

는 단위 매트릭스이고,

는 영 벡터이며,

,

이며, R 및 T는 각각 각 삼차원 포인트에 대응되는 회전 변수 및 평행 이동 변수이며, m은 파노라마 이미지의 개수이고, n은 삼차원 포인트의 개수이다.In the above formula (7),

is the point coordinate of the unit sphere corresponding to the first feature point,

is i times?? is a vector of three-dimensional point coordinates, f is the focal length of the virtual perspective camera,

is the camera matrix of the i-th panoramic image, for initialization,

is the unit matrix,

is the zero vector,

,

where R and T are rotation and translation variables corresponding to each three-dimensional point, respectively, m is the number of panoramic images, and n is the number of three-dimensional points.

Determining the camera pose of the image may proceed using a RANSAC (Random Sample Consensus) and P3P algorithm.

As you know based on the description of the point cloud model construction method, the point cloud model construction method completes the construction of the point cloud model and completes the construction of the point cloud model by being able to proceed with camera registration and point cloud reconstruction using the panoramic image. The point cloud model built based on the image has higher accuracy than the point cloud model built based on the existing general perspective image, the robustness of the overlapping texture is better, and the scenario reconstruction is more comprehensive. By building a high-precision visual map using the point cloud model building method, it is possible to provide visual features and three-dimensional landmark points for positioning for autonomous driving and AR, and also using the point cloud model building method, a specific By building a three-dimensional model of a given scenario, scenario display and VR applications are possible, for example, AR/VR tours of tourist attractions, museums, exhibition halls, or 3D model construction of a specific building, a specific street or a specific city, AR It can be applied to special effects.

According to a second aspect of the embodiment of the present invention, a point cloud model building apparatus is provided, and with reference to FIG. 7 , the structure of the apparatus is illustrated, the point cloud model building apparatus comprising:

an acquiring module 701, configured to acquire a first characteristic of a panoramic image in the panoramic image set;

determine, according to the first characteristic, at least one group of image pairs and a corresponding matching result in the panoramic image set, wherein the image pair includes two panoramic images to which the first characteristic is matched, and wherein the matching result is two a matching module 702 configured to indicate a correspondence between the first features of the panoramic image; and

and a building module 703, configured to build a point cloud model according to the at least one group of image pairs and a corresponding matching result.

In one embodiment, the acquiring module specifically,

determine a plurality of perspective images corresponding to the panoramic image, wherein a set of spaces corresponding to the plurality of perspective images is a space corresponding to the panoramic image;

obtain a second characteristic of at least one perspective image in the plurality of perspective images;

determine a first sub-feature of a corresponding position of the panoramic image according to a second characteristic of the perspective image, wherein the corresponding position of the perspective image and the panoramic image correspond to the same space; and

and determine a first characteristic of the panoramic image according to at least one of the first sub-features.

In an embodiment, when the acquisition module is configured to determine a plurality of perspective images corresponding to the panoramic image, specifically,

acquiring a unit sphere corresponding to the panoramic image, and determining a first mapping relationship between pixel point coordinates of the panoramic image and point coordinates of the unit sphere;

determine a plurality of perspective images according to the unit sphere, and determine a second mapping relationship between pixel point coordinates of the perspective image and point coordinates of the unit sphere; a set of spherical points corresponding to the plurality of perspective images is a unit sphere - ;

A third mapping relationship between pixel point coordinates of a panoramic image and pixel point coordinates of a perspective image is determined according to the first mapping relationship and the second mapping relationship, and the pixel information of the pixel points of the panoramic image and the third mapping relationship and determine the pixel information of the pixel point of the perspective image according to it.

In an embodiment, the second feature of the perspective image comprises a second feature point and a corresponding second descriptor;

When the acquiring module is configured to determine the first sub-feature of the corresponding position of the panoramic image according to the second feature of the perspective image, specifically,

determine the coordinates of the first feature point of the panoramic image according to the third mapping relationship and the coordinates of the second feature point of the perspective image;

and determine a first descriptor corresponding to the first feature point of the panoramic image according to a second descriptor corresponding to the second feature point of the perspective image.

In one embodiment, the first characteristic comprises a first characteristic point and a corresponding first descriptor;

The matching module is specifically,

determining a plurality of groups of image pairs according to the panoramic images to be matched corresponding to each panoramic image;

determine a plurality of groups of feature point pairs according to a first descriptor of the two panoramic images of the image pairs of each group, wherein the feature point pairs of each group are two correspondingly matched first features belonging to the two panoramic images including points - ;

A first essential matrix is determined according to the pair of feature points of the plurality of groups, and filtering is performed on the pair of feature points of the plurality of groups using the first essential matrix to obtain a matching result corresponding to the image pair. configured to get

In an embodiment, after the matching module is configured to determine a plurality of groups of feature point pairs according to a first descriptor of two panoramic images of the image pairs of each group, further,

obtaining the number of feature point pairs of the two panoramic images of the image pairs of each group;

The number of feature point pairs is configured to filter image pairs that meet a first preset condition.

In one embodiment, when the matching module is configured to determine the first essential matrix according to the feature point pairs of the plurality of groups, specifically,

determine the angular error of the feature point pair according to the angular error of the two first feature points in the feature point pair, wherein the angular error of the first feature point is a spherical point of a unit sphere corresponding to the first feature point - is the included angle between the line connecting the optical center of the unit sphere and the outer polar plane;

determining an essential matrix multiple times according to a preset number of the feature point pairs by using the angular error of the corresponding feature point pair as a residual term;

It is configured to determine the number of internal points corresponding to each of the essential matrices, and to determine the essential matrix having the largest number of internal points as the first essential matrix.

In one embodiment, when the matching module is configured to determine the number of internal points corresponding to each of the essential matrix, specifically,

determine an angular error of a feature point pair of each group of the image pair according to the essential matrix;

determining a pair of feature points in which the angular error satisfies a predetermined second condition as an internal point;

and to determine the number of internal points corresponding to the essential matrix according to all internal points.

In one embodiment, when the matching module is configured to perform filtering on a plurality of groups of feature point pairs using the first essential matrix, specifically,

determine an angular error of a feature point pair of each group of the image pair according to the first essential matrix;

and the angular error is configured to filter feature point pairs that meet a third preset condition.

In one embodiment, the matching module is also

two first feature points of the feature point pair of the image pair acquire coordinates in the panoramic image;

determine a perspective image associated with a feature point pair according to a third mapping relationship corresponding to the panoramic image and coordinates of a first feature point belonging to the feature point pair in the panoramic image, wherein the perspective image associated with the feature point pair comprises: a perspective image in which a second feature point corresponding to a first feature point belonging to a feature point pair exists;

and perform filtering on the image pair by using the perspective image associated with the feature point pair.

In an embodiment, when the matching module is configured to perform filtering on the image pair using a perspective image related to the feature point pair, specifically,

In response to a perspective image associated with a feature point pair corresponding to at least one panoramic image of the image pair being a plurality of consecutive images, and the number of the plurality of consecutive images being smaller than a preset filtering threshold, the image pair is configured to filter.

In one embodiment, the building module specifically,

Determine that a group of image pairs is an initial image pair according to a preset initialization condition and a feature point pair and a first essential matrix of each group of image pairs, and determine a camera pose of each panoramic image of the initial image pair, the initial triangulate the first pair of feature points of the image pair to form an initial three-dimensional point;

According to the matching relationship between the first feature point corresponding to the first three-dimensional point and the first feature point of each unregistered image, one unregistered image is registered until each panoramic image in the panoramic image set becomes a registered image. determine multiple times as an image - the unregistered image is a panoramic image in which all first feature points are not triangulated, the registered image is a panoramic image in which the first feature point is triangulated, and the first three-dimensional point is the initial three-dimensional including a point, or including a three-dimensional point formed by triangulating the initial three-dimensional point and a first feature point of the registered image;

After each registration image is determined, the camera pose of the registered image is determined, triangulation is performed on the first feature point of the registered image to form a corresponding three-dimensional point, and a third feature point of the registered image and triangulate to form a corresponding three-dimensional point, wherein the third feature point is a first feature point matching the first feature point of the registered image in the registered image.

In an embodiment, when the building module is configured to determine that a group of image pairs is an initial image pair according to a preset initialization condition and a first essential matrix and a feature point pair of each group of image pairs, specifically,

Select image pairs sequentially according to the descending order of the number of feature point pairs, and after each image pair selection, all according to the feature point pair and the first essential matrix, until the selected image pair satisfies the initialization condition, the and determine whether the image pair satisfies the initialization condition, and determine the selected image pair as the initial image pair.

In an embodiment, when the building module is configured to determine, according to the feature point pair and the first essential matrix, whether the image pair satisfies the initialization condition, specifically

Determining at least one group of displacement variables according to the first essential matrix of the image pair, triangulating the characteristic points of the pair of characteristic points for each group of displacement variables, respectively, to obtain three-dimensional points corresponding to the displacement parameters of each group forming, and filtering the three-dimensional points according to a triangulation angle and a reprojection error of each group of three-dimensional points, wherein the displacement variable includes a rotation variable and a translation variable;

in response to the number of three-dimensional points in one group having the highest number being greater than a first preset number threshold, determine a corresponding displacement variable as the first displacement variable;

An essential matrix in which the number of internal points is greater than or equal to the point count threshold is selected from the essential matrix obtained by multiple calculations, and at least one group of displacement variables is determined according to each essential cell matrix, respectively, and a characteristic for each group of displacement variables Each of the feature points of the point pair is triangulated to form a three-dimensional point corresponding to the displacement variable of each group, and the three-dimensional point is filtered according to the re-projection error and triangulation angle of the three-dimensional point of each group, each essential a displacement variable corresponding to the three-dimensional point of the group having the largest number of mattresses;

and determine that the image pair satisfies the initialization condition when a difference between the displacement variable held by each essential matrix and the first displacement variable satisfies a preset range.

In one embodiment, the building module further comprises:

Optimize the camera pose and position of the initial three-dimensional point of each panoramic image, through which the initial three-dimensional point minimizes the re-projection error in the two panoramic images of the initial image pair;

after each time determining the camera pose of the registered image, the three-dimensional point optimizes the camera pose of the registered image through minimizing the re-projection error in the registered image;

After triangulating the feature points of the registered image each time to form a corresponding three-dimensional point, and performing triangulation on the third feature point of the registered image to form a corresponding three-dimensional point, each three-dimensional point is configured to at least one of optimizing a camera pose of each registered image and a position of each three-dimensional point through minimizing a re-projection error in each registered image.

In one embodiment, the matching module is also

determine a corresponding panoramic image to be matched according to a space corresponding to each panoramic image; or

and determine a panoramic image to be matched corresponding to each panoramic image according to a preset matching rule. With respect to the apparatus in the above embodiment, the specific form in which each module executes the operation has already been described in detail in the embodiment related to the point cloud model building method in the first aspect, and thus is not repeated here.

Referring to FIG. 8 , according to a third aspect of an embodiment of the present invention, there is provided an electronic device, wherein the electronic device includes a memory 801 and a processor 802 , wherein the memory 801 includes a processor 802 . for storing computer instructions operable on the above, and the processor is for building a point cloud model based on the method according to the first aspect when executing the computer instructions. The memory 801 may use a volatile or non-volatile storage medium.

According to a fourth aspect of an embodiment of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method according to the first aspect. The computer-readable storage medium may be a volatile storage medium or a non-volatile storage medium.

The present invention also provides a computer program product comprising computer readable code, wherein when the computer readable code is run in a device, a processor in the device implements the method for constructing a point cloud model provided in any one of the foregoing embodiments. Execute the command to

The computer program product may be specifically implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium, and in another optional example, the computer program product is specifically software such as a Software Development Kit (SDK) or the like. implemented as a product.

In the present invention, the terms “first” and “second” are for purposes of explanation only, and cannot be understood as indicating or implying relative importance. The term "plurality" means two or more than two, unless expressly limited otherwise.

Other embodiments of the present invention will readily occur to those skilled in the art from consideration of the specification and the invention disclosed herein. The present invention is intended to cover any modifications, uses or adaptations of the present invention, such modifications, uses or adaptations being in accordance with the general principles of the present disclosure, which are not disclosed in the present invention, but are not disclosed herein. Includes conventional technical means. The specification and examples are to be regarded as illustrative only, the true scope and spirit of the invention being pointed out by the claims below.

It should be understood that the present invention is not limited to the precise structure described above and shown in the drawings, and various modifications and changes may be made thereto without departing from the scope thereof. The scope of the invention is limited only by the appended claims.

An embodiment of the present invention provides a point cloud model building method, apparatus, apparatus and storage medium, wherein the point cloud model building method includes: acquiring a first characteristic of a panoramic image in a panoramic image set; determining, according to the first characteristic, at least one group of image pairs in the panoramic image set and a corresponding matching result, wherein the image pair includes two panoramic images to which the first characteristic is matched, the matching result is indicating the correspondence between the first features of the two panoramic images; and constructing a point cloud model according to the at least one group of image pairs and a corresponding matching result. Since the matching was performed using the panoramic image and a point cloud model was additionally built with the matched result, the number of images in the image set can be reduced, thereby improving the matching efficiency and modeling efficiency, and Since the range of the corresponding space is relatively large, the matching effect between the panoramic images can be increased, thereby improving the accuracy and quality of the point cloud model construction.

Claims (35)

A method for building a point cloud model, comprising:
Applied to electronic devices, the point cloud model building method comprises:
obtaining a first characteristic of a panoramic image in the panoramic image set;
determining, according to the first characteristic, at least one group of image pairs in the panoramic image set and a corresponding matching result, wherein the image pair includes two panoramic images to which the first characteristic is matched, the matching result is indicating the correspondence between the first features of the two panoramic images; and
and constructing a point cloud model according to the at least one group of image pairs and a corresponding matching result. According to claim 1,
Acquiring a first characteristic of a panoramic image in the panoramic image set comprises:
determining a plurality of perspective images corresponding to the panoramic image, wherein a set of spaces corresponding to the plurality of perspective images is a space corresponding to the panoramic image;
acquiring a second characteristic of at least one perspective image in the plurality of perspective images;
determining a first sub-feature of a corresponding position of the panoramic image according to a second characteristic of the perspective image, wherein the corresponding position of the perspective image and the panoramic image correspond to the same space; and
and determining a first characteristic of the panoramic image according to at least one of the first sub-features. 3. The method of claim 2,
The step of determining a plurality of perspective images corresponding to the panoramic image,
obtaining a unit sphere corresponding to the panoramic image, and determining a first mapping relationship between pixel point coordinates of the panoramic image and point coordinates of the unit sphere;
determining a plurality of perspective images according to the unit sphere, and determining a second mapping relationship between pixel point coordinates of the perspective image and point coordinates of the unit sphere - A set of spherical points corresponding to the plurality of perspective images is a unit sphere lim - ; and
A third mapping relationship between pixel point coordinates of a panoramic image and pixel point coordinates of a perspective image is determined according to the first mapping relationship and the second mapping relationship, and the pixel information of the pixel points of the panoramic image and the third mapping relationship A method for building a point cloud model, comprising the step of determining pixel information of a pixel point of a perspective image according to 4. The method of claim 3,
the second characteristic of the perspective image includes a second characteristic point and a corresponding second descriptor;
Determining the first sub-feature of the corresponding position of the panoramic image according to the second feature of the perspective image comprises:
determining the coordinates of the first feature point of the panoramic image according to the third mapping relationship and the coordinates of the second feature point of the perspective image; and
and determining a first descriptor corresponding to the first feature point of the panoramic image according to a second descriptor corresponding to the second feature point of the perspective image. 5. The method of claim 4,
the first characteristic includes a first characteristic point and a corresponding first descriptor;
The step of determining at least one group of image pairs and a corresponding matching result in the panoramic image set according to the first feature comprises:
determining a plurality of groups of image pairs according to a panoramic image to be matched corresponding to each panoramic image;
determining a plurality of groups of feature point pairs according to a first descriptor of the two panoramic images of the image pairs of each group, wherein the feature point pairs of each group correspond to two correspondingly matched first pairs belonging to the two panoramic images Includes feature points - ; and
A first essential matrix is determined according to the pair of feature points of the plurality of groups, and filtering is performed on the pair of feature points of the plurality of groups using the first essential matrix to obtain a matching result corresponding to the image pair. A method for building a point cloud model, comprising the step of obtaining. 6. The method of claim 5,
After determining a pair of feature points of a plurality of groups according to a first descriptor of the two panoramic images of the image pair of each group, the step comprises:
obtaining the number of feature point pairs of the two panoramic images of the image pair of each group; and
The method of constructing a point cloud model, characterized in that it further comprises the step of filtering the image pairs that satisfy the first condition in which the number of feature point pairs is preset. 7. The method of claim 5 or 6,
The step of determining the first essential matrix according to the pair of feature points of the plurality of groups,
determining the angular error of the feature point pair according to the angular error of the two first feature points in the feature point pair - The angular error of the first feature point is a spherical surface of a unit sphere corresponding to the first feature point The included angle between the line connecting the point and the optical center of the unit sphere and the outer polar plane - ;
determining an essential matrix multiple times according to a preset number of the feature point pairs by using the angular error of the corresponding feature point pair as a residual term; and
Determining the number of internal points corresponding to each of the essential matrices, and determining an essential matrix having the largest number of internal points as the first essential matrix. 8. The method of claim 7,
Determining the number of internal points corresponding to each of the essential matrix comprises:
determining an angular error of a feature point pair of each group of the image pair according to the essential matrix;
determining, as an internal point, a pair of feature points in which the angular error satisfies a predetermined second condition; and
Point cloud model construction method comprising the step of determining the number of internal points corresponding to the essential matrix according to all internal points. 9. The method of claim 7 or 8,
The step of filtering a plurality of groups of feature point pairs using the first essential matrix includes:
determining an angular error of a feature point pair of each group of the image pair according to the first essential matrix; and
and filtering a pair of feature points in which the angular error meets a preset third condition. 10. The method according to any one of claims 5 to 9,
The method of building the point cloud model,
obtaining coordinates in a panoramic image of two first feature points of a feature point pair of the image pair;
determining a perspective image associated with a feature point pair according to a third mapping relationship corresponding to the panoramic image and coordinates of a first feature point belonging to the feature point pair in the panoramic image - a perspective image associated with the feature point pair is a perspective image in which a second feature point corresponding to a first feature point belonging to a feature point pair exists; and
Point cloud model building method comprising the step of performing filtering on the image pair by using the perspective image related to the feature point pair. 11. The method of claim 10,
The step of filtering the image pair by using the perspective image related to the feature point pair,
In response to a perspective image associated with a feature point pair corresponding to at least one panoramic image of the image pair being a plurality of consecutive images, and the number of the plurality of consecutive images being smaller than a preset filtering threshold, the image pair Point cloud model building method comprising the step of filtering. 12. The method according to any one of claims 5 to 11,
The step of constructing a point cloud model according to the at least one group of image pairs and corresponding matching results,
Determine that a group of image pairs is an initial image pair according to a preset initialization condition and a feature point pair and a first essential matrix of each group of image pairs, and determine a camera pose of each panoramic image of the initial image pair, the initial performing triangulation on the first feature point pair of the image pair to form an initial three-dimensional point;
According to the matching relationship between the first feature point corresponding to the first three-dimensional point and the first feature point of each unregistered image, one unregistered image is registered until each panoramic image in the panoramic image set becomes a registered image. determining as an image multiple times - the unregistered image is a panoramic image in which all first feature points are not triangulated, the registered image is a panoramic image in which the first feature point is triangulated, and the first three-dimensional point is the initial including a three-dimensional point, or including a three-dimensional point formed by triangulating the initial three-dimensional point and a first feature point of the registered image; and
After each registration image is determined, the camera pose of the registered image is determined, triangulation is performed on the first feature point of the registered image to form a corresponding three-dimensional point, and a third feature point of the registered image performing triangulation to form a corresponding three-dimensional point, wherein the third feature point is a first feature point matching the first feature point of the registered image in the registered image. How to build a point cloud model. 13. The method of claim 12,
The step of determining that the image pair of a group is the initial image pair according to the preset initialization condition and the feature point pair and the first essential matrix of the image pair of each group,
Select image pairs sequentially according to the descending order of the number of feature point pairs, and after each image pair selection, all according to the feature point pair and the first essential matrix, until the selected image pair satisfies the initialization condition, the and determining whether the image pair satisfies the initialization condition, and determining the selected image pair as the initial image pair. 14. The method of claim 13,
Determining whether the image pair satisfies the initialization condition according to the feature point pair and the first essential matrix comprises:
Determining at least one group of displacement variables according to the first essential matrix of the image pair, triangulating the characteristic points of the pair of characteristic points for each group of displacement variables, respectively, to obtain three-dimensional points corresponding to the displacement parameters of each group forming, and filtering the three-dimensional points according to a triangulation angle and a reprojection error of each group of three-dimensional points, wherein the displacement variable includes a rotation variable and a translational variable;
determining a corresponding displacement variable as a first displacement variable in response to the number of three-dimensional points in a group having the largest number being greater than a predetermined first number threshold;
An essential matrix in which the number of internal points is greater than or equal to the point count threshold is selected from the essential matrix obtained by multiple calculations, and at least one group of displacement variables is determined according to each essential cell matrix, respectively, and a characteristic for each group of displacement variables Each of the feature points of the point pair is triangulated to form a three-dimensional point corresponding to the displacement variable of each group, and the three-dimensional point is filtered according to the re-projection error and triangulation angle of the three-dimensional point of each group, each essential retaining a displacement variable corresponding to the three-dimensional point of the group having the largest number of mattresses; and
and determining that the image pair satisfies the initialization condition when a difference between the displacement variable held by each essential matrix and the first displacement variable satisfies a preset range. 13. The method of claim 12,
The method of building the point cloud model,
optimizing the camera pose and the position of the initial three-dimensional point of each panoramic image, such that the initial three-dimensional point minimizes the re-projection error in the two panoramic images of the initial image pair;
after determining the camera pose of the registered image each time, optimizing the camera pose of the registered image through the three-dimensional point minimizing the re-projection error in the registered image; and
After triangulating the feature points of the registered image each time to form a corresponding three-dimensional point, and performing triangulation on the third feature point of the registered image to form a corresponding three-dimensional point, each three-dimensional point and optimizing the camera pose of each registered image and the position of each three-dimensional point through minimizing a re-projection error in each registered image. 6. The method of claim 5,
The method of building the point cloud model,
determining a panoramic image to be matched according to a space corresponding to each panoramic image; or
and determining a panoramic image to be matched corresponding to each panoramic image according to a preset matching rule. A point cloud model building device, comprising:
The point cloud model building device,
an acquiring module, configured to acquire a first characteristic of a panoramic image in the panoramic image set;
determine, according to the first characteristic, at least one group of image pairs and a corresponding matching result in the panoramic image set, wherein the image pair includes two panoramic images to which the first characteristic is matched, and wherein the matching result is two a matching module configured to indicate a correspondence between the first features of the panoramic image; and
and a building module configured to build a point cloud model according to the at least one group of image pairs and a corresponding matching result. 18. The method of claim 17,
an acquiring module, configured to acquire a first characteristic of a panoramic image in the panoramic image set,
determine a plurality of perspective images corresponding to the panoramic image, wherein a set of spaces corresponding to the plurality of perspective images is a space corresponding to the panoramic image;
obtain a second characteristic of at least one perspective image in the plurality of perspective images;
determine a first sub-feature of a corresponding position of the panoramic image according to a second characteristic of the perspective image, wherein the corresponding position of the perspective image and the panoramic image correspond to the same space; and
and determine a first characteristic of the panoramic image according to at least one of the first sub-features. 19. The method of claim 18,
The acquisition module, configured to determine a plurality of perspective images corresponding to the panoramic image,
acquiring a unit sphere corresponding to the panoramic image, and determining a first mapping relationship between pixel point coordinates of the panoramic image and point coordinates of the unit sphere;
determine a plurality of perspective images according to the unit sphere, and determine a second mapping relationship between pixel point coordinates of the perspective image and point coordinates of the unit sphere; and a set of spherical points corresponding to the plurality of perspective images is a unit sphere - ; and
A third mapping relationship between pixel point coordinates of a panoramic image and pixel point coordinates of a perspective image is determined according to the first mapping relationship and the second mapping relationship, and the pixel information of the pixel points of the panoramic image and the third mapping relationship and determine pixel information of pixel points of the perspective image according to the point cloud model building apparatus. 20. The method of claim 19,
the second characteristic of the perspective image includes a second characteristic point and a corresponding second descriptor;
the acquiring module, configured to determine a first sub-feature of a corresponding position of the panoramic image according to a second feature of the perspective image,
determine the coordinates of the first feature point of the panoramic image according to the third mapping relationship and the coordinates of the second feature point of the perspective image; and
and determine a first descriptor corresponding to the first feature point of the panoramic image according to a second descriptor corresponding to the second feature point of the perspective image. 21. The method of claim 20,
the first characteristic includes a first characteristic point and a corresponding first descriptor;
the matching module, configured to determine at least one group of image pairs and a corresponding matching result in the panoramic image set according to the first feature,
determining a plurality of groups of image pairs according to the panoramic images to be matched corresponding to each panoramic image;
determine a plurality of groups of pairs of feature points according to a first descriptor of the two panoramic images of the pairs of images of each group, wherein the pairs of feature points in each group are two correspondingly matched first features belonging to the two panoramic images including points - ; and
A first essential matrix is determined according to the pair of feature points of the plurality of groups, and filtering is performed on the pair of feature points of the plurality of groups using the first essential matrix to obtain a matching result corresponding to the image pair. Point cloud model building device, characterized in that configured to obtain. 22. The method of claim 21,
After the matching module is configured to determine a plurality of groups of feature point pairs according to a first descriptor of the two panoramic images of the image pairs of each group, the matching module is configured to:
obtaining the number of feature point pairs of the two panoramic images of the image pairs of each group; and
Point cloud model building apparatus, characterized in that the number of feature point pairs is configured to filter image pairs that meet a first preset condition. 23. The method of claim 21 or 22,
The matching module, configured to determine a first essential matrix according to the pair of feature points in the plurality of groups,
determine the angular error of the feature point pair according to the angular error of the two first feature points in the feature point pair, wherein the angular error of the first feature point is a spherical point of a unit sphere corresponding to the first feature point - is the included angle between the line connecting the optical center of the unit sphere and the outer polar plane;
determining the essential matrix multiple times according to a preset number of the feature point pairs by using the angular error of the corresponding feature point pair as a residual term; and
Point cloud model building apparatus, characterized in that configured to determine the number of internal points corresponding to each of the essential matrix, and to determine the essential matrix having the largest number of internal points as the first essential matrix. 24. The method of claim 23,
The matching module, configured to determine the number of internal points corresponding to each of the essential matrices,
determine an angular error of a feature point pair of each group of the image pair according to the essential matrix;
determining a feature point pair in which the angular error satisfies a preset second condition as an internal point; and
Point cloud model building apparatus, characterized in that configured to determine the number of internal points corresponding to the essential matrix according to all internal points. 25. The method of claim 23 or 24,
Filtering a plurality of groups of feature point pairs using the first essential matrix is,
determining an angular error of a feature point pair of each group of the image pair according to the first essential matrix; and
Point cloud model building apparatus, characterized in that the angular error filters the pair of feature points that meet a preset third condition. 26. The method according to any one of claims 21 to 25,
The point cloud model building device,
two first feature points of the feature point pair of the image pair acquire coordinates in the panoramic image;
determine a perspective image associated with a feature point pair according to a third mapping relationship corresponding to the panoramic image and coordinates of a first feature point belonging to the feature point pair in the panoramic image, wherein the perspective image associated with the feature point pair comprises: a perspective image in which a second feature point corresponding to a first feature point belonging to a feature point pair exists; and
Point cloud model building apparatus, characterized in that configured to perform filtering on the image pair by using the perspective image associated with the feature point pair. 27. The method of claim 26,
Filtering the image pair using the perspective image related to the feature point pair comprises:
In response to a perspective image associated with a feature point pair corresponding to at least one panoramic image of the image pair being a plurality of consecutive images, and the number of the plurality of consecutive images being smaller than a preset filtering threshold, the image pair Point cloud model building apparatus comprising filtering. 28. The method according to any one of claims 21 to 27,
Building a point cloud model according to the at least one group of image pairs and corresponding matching results,
Determine that a group of image pairs is an initial image pair according to a preset initialization condition and a feature point pair and a first essential matrix of each group of image pairs, and determine a camera pose of each panoramic image of the initial image pair, the initial performing triangulation on the first pair of feature points of the image pair to form an initial three-dimensional point;
According to the matching relationship between the first feature point corresponding to the first three-dimensional point and the first feature point of each unregistered image, one unregistered image is registered until each panoramic image in the panoramic image set becomes a registered image. determining multiple times as an image, wherein the unregistered image is a panoramic image in which all first feature points are not triangulated, the registered image is a panoramic image in which the first feature points are triangulated, and wherein the first three-dimensional point is the initial including a three-dimensional point, or including a three-dimensional point formed by triangulating the initial three-dimensional point and a first feature point of the registered image; and
After each registration image is determined, the camera pose of the registered image is determined, triangulation is performed on the first feature point of the registered image to form a corresponding three-dimensional point, and a third feature point of the registered image performing triangulation to form a corresponding three-dimensional point, wherein the third feature point is a first feature point matching the first feature point of the registered image in the registered image. A point cloud model building device. 29. The method of claim 28,
Determining that the image pair of a group is the initial image pair according to the preset initialization condition and the feature point pair and the first essential matrix of the image pair of each group,
Select image pairs sequentially according to the descending order of the number of feature point pairs, and after each image pair selection, all according to the feature point pair and the first essential matrix, until the selected image pair satisfies the initialization condition, the and determining whether the image pair satisfies the initialization condition, and determining the selected image pair as the initial image pair. 30. The method of claim 29,
According to the feature point pair and the first essential matrix, determining whether the image pair satisfies the initialization condition comprises:
Determining at least one group of displacement variables according to the first essential matrix of the image pair, triangulating the characteristic points of the pair of characteristic points for each group of displacement variables, respectively, to obtain three-dimensional points corresponding to the displacement parameters of each group forming, and filtering the three-dimensional points according to triangulation angles and reprojection errors of each group of three-dimensional points, wherein the displacement variables include rotation variables and translation variables;
determining a corresponding displacement variable as a first displacement variable in response to the number of three-dimensional points in a group having the highest number being greater than a predetermined first number threshold;
An essential matrix in which the number of internal points is greater than or equal to the point count threshold is selected from the essential matrix obtained by multiple calculations, and at least one group of displacement variables is determined according to each essential cell matrix, respectively, and a characteristic for each group of displacement variables Each of the feature points of the point pair is triangulated to form a three-dimensional point corresponding to the displacement variable of each group, and the three-dimensional point is filtered according to the re-projection error and triangulation angle of the three-dimensional point of each group, each essential retaining the displacement variable corresponding to the three-dimensional point of the group having the largest number of mattresses; and
and determining that the image pair satisfies the initialization condition when the difference between the displacement variable held by each essential matrix and the first displacement variable satisfies a preset range. 29. The method of claim 28,
The point cloud model building device,
optimizing the camera pose and position of the initial three-dimensional point of each panoramic image, through which the initial three-dimensional point minimizes the re-projection error in the two panoramic images of the initial image pair;
optimizing the camera pose of the registered image through a three-dimensional point minimizing the re-projection error in the registered image after each time determining the camera pose of the registered image; and
After triangulating the feature points of the registered image each time to form a corresponding three-dimensional point, and performing triangulation on the third feature point of the registered image to form a corresponding three-dimensional point, each three-dimensional point is configured to perform at least one of optimizing a camera pose of each registered image and a position of each three-dimensional point through minimizing a re-projection error in each registered image. 22. The method of claim 21,
The point cloud model building device,
determining a corresponding panoramic image to be matched according to a space corresponding to each panoramic image; or
Point cloud model building apparatus, characterized in that it is configured to determine a panoramic image to be matched corresponding to each panoramic image according to a preset matching rule. As an electronic device,
17. A method comprising: a memory and a processor, wherein the memory is for storing computer instructions operable on the processor, wherein the processor implements the method for building a point cloud model according to any one of claims 1 to 16 when executing the computer instructions. An electronic device, characterized in that for A computer readable storage medium comprising:
17. A computer-readable storage medium having a computer program stored thereon, which when the program is executed by a processor implements the method for constructing a point cloud model according to any one of claims 1 to 16. A computer program comprising:
17. A method comprising computer readable code, wherein when the computer readable code is operated in an electronic device, the processor in the electronic device implements the method for building a point cloud model according to any one of claims 1 to 16. Characterized computer program.

Images