WO2022183657A1

WO2022183657A1 - Point cloud model construction method and apparatus, electronic device, storage medium, and program

Info

Publication number: WO2022183657A1
Application number: PCT/CN2021/105574
Authority: WO
Inventors: 冯友计; 江明轩; 周立阳; 姜翰青; 章国锋
Original assignee: 浙江商汤科技开发有限公司
Priority date: 2021-03-04
Filing date: 2021-07-09
Publication date: 2022-09-09
Also published as: JP2023519466A; CN112837419B; CN112837419A; KR20220125714A; KR102638632B1

Abstract

Embodiments of the present disclosure relate to a point cloud model construction method and apparatus, a device, and a storage medium. The method comprises: obtaining first features of panoramic images in a panoramic image set; determining at least one group of image pairs in the panoramic image set and a corresponding matching result according to the first features, wherein the image pairs comprise two panoramic images subjected to first feature matching, and the matching result indicates the correspondence between the first features of the two panoramic images; and constructing a point cloud model according to the at least one group of image pairs and the corresponding matching result. The panoramic images are adopted for matching, and the point cloud model is further constructed according to the matching result, such that the number of images in the image set can be decreased, and the matching efficiency and the modeling efficiency are improved; and the range of a space corresponding to the panoramic images is relatively large, such that the matching effect between the panoramic images can be improved, and the precision and quality of constructing the point cloud model is improved.

Description

Point cloud model construction method, device, electronic device, storage medium and program

CROSS-REFERENCE TO RELATED APPLICATIONS

The present disclosure is based on the Chinese patent application with the application number of 202110240320.4, the application date of March 04, 2021, and the patent name of "point cloud model construction method, device, equipment and storage medium", and claims the priority of the Chinese patent application , the entire content of the Chinese patent application is incorporated herein by reference.

technical field

The present disclosure relates to the technical field of three-dimensional modeling, and relates to, but is not limited to, a method, apparatus, electronic device, computer storage medium and computer program for constructing a point cloud model.

Background technique

With the development of artificial intelligence technology, the spatial modeling technology has become richer and more accurate. The motion recovery structure technology can construct a 3D point cloud model from multiple images, which can be widely used in the fields of physical space digitization, high-precision map construction, and augmented reality. Therefore, how to improve the accuracy and quality of point cloud model construction is of great significance in the field of 3D modeling technology.

SUMMARY OF THE INVENTION

Embodiments of the present disclosure provide a method, apparatus, electronic device, computer storage medium, and computer program for constructing a point cloud model.

According to a first aspect of the embodiments of the present disclosure, a method for constructing a point cloud model is provided, which is applied to an electronic device, including:

obtaining the first feature of the panoramic image in the panoramic image set;

At least one set of image pairs in the panoramic image set and corresponding matching results are determined according to the first feature, wherein the image pair includes two panoramic images matched by the first feature, and the matching results indicate two panoramic images the correspondence between the first features of the image;

A point cloud model is constructed according to the at least one set of image pairs and the corresponding matching results.

In one embodiment, the obtaining the first feature of the panoramic image in the panoramic image set includes:

determining multiple perspective images corresponding to the panoramic image, wherein the set of spaces corresponding to the multiple perspective images is the space corresponding to the panoramic image;

acquiring a second feature of at least one fluoroscopic image in the plurality of fluoroscopic images;

Determine the first sub-feature of the corresponding position of the panoramic image according to the second feature of the fluoroscopic image, wherein the fluoroscopic image corresponds to the same space as the corresponding position of the panoramic image;

A first feature of the panoramic image is determined based on at least one of the first sub-features.

In one embodiment, the determining a plurality of perspective images corresponding to the panoramic image includes:

acquiring the unit sphere corresponding to the panoramic image, and determining the first mapping relationship between the pixel point coordinates of the panoramic image and the point coordinates of the unit sphere;

Determine a plurality of perspective images according to the unit sphere, and determine the second mapping relationship between the pixel point coordinates of the perspective image and the point coordinates of the unit sphere, wherein the set of spherical points corresponding to the plurality of perspective images is the unit sphere;

The third mapping relationship between the pixel point coordinates of the panoramic image and the pixel point coordinates of the fluoroscopic image is determined according to the first mapping relationship and the second mapping relationship, and the third mapping relationship between the pixel point coordinates of the panoramic image and the third mapping relationship is determined according to the first mapping relationship and the second mapping relationship. The mapping relationship determines the pixel information of the pixels of the fluoroscopic image.

In one embodiment, the second feature of the fluoroscopic image includes a second feature point and a corresponding second descriptor;

The determining of the first sub-feature of the corresponding position of the panoramic image according to the second feature of the fluoroscopic image includes:

Determine the coordinates of the first feature point of the panoramic image according to the coordinates of the second feature point of the fluoroscopic image and the third mapping relationship;

The first descriptor corresponding to the first feature point of the panoramic image is determined according to the second descriptor corresponding to the second feature point of the fluoroscopic image.

In one embodiment, the first feature includes a first feature point and a corresponding first descriptor;

The determining of at least one group of image pairs in the panoramic image set and the corresponding matching results according to the first feature includes:

Determine multiple groups of image pairs according to each panoramic image and the corresponding panoramic image to be matched;

Determine a plurality of sets of feature point pairs according to the first descriptors of the two panoramic images of the image pair, wherein each group of the feature point pairs includes two correspondingly matched first feature points belonging to the two panoramic images;

A first essential matrix is determined according to the multiple sets of feature point pairs, and the multiple sets of feature point pairs are filtered by using the first essential matrix to obtain matching results corresponding to the image pairs.

In one embodiment, after determining multiple groups of feature point pairs according to the first descriptors of the two panoramic images of each group of the image pairs, the method further includes:

obtaining the number of feature point pairs of the two panoramic images of the image pair;

The image pairs whose number of feature point pairs meet the preset first condition are filtered.

In one embodiment, the first essential matrix is determined according to the multiple sets of feature point pairs, including:

The angle error of the pair of feature points is determined according to the angle errors of the two first feature points in the pair of feature points, wherein the angle errors of the first feature points are the unit spheres corresponding to the first feature points The angle between the line connecting the spherical point of the unit sphere and the optical center of the unit sphere and the outer polar plane;

Taking the angular error of the corresponding pair of feature points as the residual term, determining the essential matrix according to the preset number of pairs of the feature points for many times;

The number of interior points of each of the essential matrices is determined, and the essential matrix with the largest number of interior points is determined as the first essential matrix.

In one embodiment, the determining the number of interior points corresponding to each of the essential matrices includes:

Determine the angle error of each group of feature point pairs of the image pair according to the essential matrix;

It is determined that the feature point pair whose angle error meets the preset second condition is an interior point;

The number of interior points corresponding to the essential matrix is determined according to all interior points.

In one embodiment, the filtering of multiple sets of feature point pairs by using the first essential matrix includes:

Determine the angle error of each group of feature point pairs of the image pair according to the first essential matrix;

The feature point pairs whose angle errors meet the preset third condition are filtered.

In one embodiment, it also includes:

Obtaining the coordinates of the two first feature points of the feature point pair of the image pair in the panorama image to which they belong;

A perspective image related to the feature point pair is determined according to the third mapping relationship corresponding to the panoramic image and the coordinates of the first feature point belonging to the feature point pair on the panoramic image, wherein the feature point pair-related The fluoroscopic image is a fluoroscopic image in which there is a second feature point corresponding to the first feature point belonging to the feature point pair;

The image pairs are filtered using the fluoroscopic images associated with the feature point pairs.

In one embodiment, the filtering of the image pair using the fluoroscopic image that is related to the feature point pair includes:

In response to the fluoroscopic images related to the feature point pair corresponding to at least one panoramic image of the image pair being consecutive multiple images, and the number of the consecutive multiple images is less than a preset filtering threshold, filtering the image pair .

In one embodiment, the constructing a point cloud model according to the at least one set of image pairs and corresponding matching results includes:

Determine a group of image pairs as initial image pairs according to preset initialization conditions, feature point pairs of each group of image pairs and the first essential matrix, and determine the camera pose of each panoramic image of the initial image pair, and triangulating the first feature point pair of the initial image pair to form an initial three-dimensional point;

According to the matching relationship between the first feature point corresponding to the first three-dimensional point and the first feature point of each unregistered image, an unregistered image is determined as a registered image, until each panoramic image in the panoramic image set is a registered image. A registered image, wherein the unregistered image is a panoramic image in which all the first feature points are not triangulated, the registered image is a panoramic image with triangulated first feature points, and the first three-dimensional point includes all the initial three-dimensional point, or a three-dimensional point formed by triangulating the initial three-dimensional point and the first feature point of the registered image;

After each registration image is determined, the camera pose of the registration image is determined, the first feature point of the registration image is triangulated to form a corresponding three-dimensional point, and the third feature point of the registered image is triangulated. to form a corresponding three-dimensional point, and the third feature point is a first feature point in the registered image that matches the first feature point of the registered image.

In one embodiment, determining a group of image pairs as initial image pairs according to preset initialization conditions, feature point pairs of each group of image pairs and the first essential matrix, including:

Select image pairs in descending order of the number of feature point pairs. After each image pair is selected, it is determined whether the image pair satisfies the initialization condition according to the feature point pair and the first essential matrix. Until the selected image pair satisfies the initialization condition, it is determined that the selected image pair is an initial image pair.

In one embodiment, determining whether the image pair satisfies the initialization condition according to the feature point pair and the first essential matrix includes:

Determine at least one group of displacement variables according to the first essential matrix of the image pair, and triangulate the feature points of the feature point pair for each group of displacement variables to form three-dimensional points corresponding to each group of displacement variables, and according to each group of three-dimensional points The reprojection error and triangulation angle filter the three-dimensional point, wherein the displacement variable includes a rotation variable and a translation variable;

In response to the number of the largest group of three-dimensional points being greater than the preset first number threshold, determining the corresponding displacement variable as the first displacement variable;

Select an essential matrix with the number of inner points greater than or equal to the threshold of the number of points from the essential matrices obtained by multiple calculations, determine at least one set of displacement variables according to each essential matrix, and triangulate the feature points of the feature point pair for each set of displacement variables respectively, To form three-dimensional points corresponding to each group of displacement variables, and filter the three-dimensional points according to the reprojection error and triangulation angle of each group of three-dimensional points, and retain the displacement variables corresponding to a group of three-dimensional points with the largest number of each essential matrix;

In the case that the difference between the displacement variable retained by each essential matrix and the first displacement variable satisfies a preset range, it is determined that the image pair satisfies the initialization condition.

In one embodiment, it also includes:

optimize the camera pose of each panorama image and the position of the initial 3D point by minimizing the reprojection error of the initial 3D point on the two panorama images of the initial image pair; and/or,

After each determination of the camera pose of the registration image, optimize the camera pose of the registration image by minimizing the reprojection error of 3D points on the registration image; and/or,

After each time the feature points of the registered image are triangulated to form corresponding three-dimensional points, and the third feature points of the registered images are triangulated to form corresponding three-dimensional points, by minimizing each three-dimensional point in each The reprojection error on each registered image, optimizes the camera pose for each registered image and the position of each 3D point.

In one embodiment, it also includes:

Determine the corresponding panoramic image to be matched according to the space corresponding to each panoramic image; or,

The panorama image to be matched corresponding to each panorama image is determined according to a preset collocation rule.

According to a second aspect of the embodiments of the present disclosure, there is provided an apparatus for constructing a point cloud model, including:

an acquisition module, configured to acquire the first feature of the panoramic image in the panoramic image set;

a matching module, configured to determine at least one set of image pairs in the panoramic image set and corresponding matching results according to the first feature, wherein the image pairs include two panoramic images matched by the first feature, and the matching the result indicates the correspondence between the first features of the two panoramic images;

The building module is configured to build a point cloud model according to the at least one set of image pairs and corresponding matching results.

In one embodiment, the obtaining module is specifically configured as:

In one embodiment, when the acquisition module is configured to determine multiple perspective images corresponding to the panoramic image, the specific configuration is:

When the acquisition module is configured to determine the first sub-feature of the corresponding position of the panoramic image according to the second feature of the fluoroscopic image, the specific configuration is as follows:

The matching module is specifically configured as:

A plurality of sets of feature point pairs are determined according to the first descriptors of the two panoramic images of each set of the image pairs, wherein each set of the feature point pairs includes two correspondingly matched first feature points belonging to the two panoramic images ;

In one embodiment, after the matching module is configured to determine multiple sets of feature point pairs according to the first descriptors of the two panoramic images of each set of the image pairs, it is further configured to:

Obtain the number of feature point pairs of the two panoramic images of each group of the image pairs;

In one embodiment, when the matching module is configured to determine the first essential matrix according to the multiple sets of feature point pairs, the specific configuration is:

The angle error of the feature point pair is determined according to the angle error of the two first feature points in the feature point pair, wherein the angle error of the first feature point is the angular error of the unit sphere corresponding to the first feature point The angle between the line connecting the spherical point and the optical center of the unit sphere and the outer polar plane;

The number of interior points corresponding to each of the essential matrices is determined, and the essential matrix with the largest number of interior points is determined as the first essential matrix.

In one embodiment, when the matching module is configured to determine the number of interior points corresponding to each of the essential matrices, the specific configuration is:

In one embodiment, when the matching module is configured to use the first essential matrix to filter multiple sets of feature point pairs, the specific configuration is:

In one embodiment, the matching module is further configured to:

In one embodiment, when the matching module is configured to filter the image pair by using the perspective image related to the feature point pair, the specific configuration is:

In one embodiment, the building module is specifically configured as:

In one embodiment, when the building module is configured to determine a group of image pairs as initial image pairs according to preset initialization conditions, feature point pairs of each group of image pairs, and the first essential matrix, the specific configuration is:

In one embodiment, when the building module is configured to determine whether the image pair satisfies the initialization condition according to the feature point pair and the first essential matrix, the specific configuration is:

In one embodiment, the building block is further configured to:

In one embodiment, the matching module is further configured to:

According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, the device includes a memory and a processor, the memory is used for storing computer instructions executable on the processor, the processor is used for executing the A point cloud model is constructed based on the method described in the first aspect when instructed by the computer.

According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the method of the first aspect.

According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer program, comprising computer-readable codes, when the computer-readable codes are executed in an electronic device, a processor in the electronic device executes any of the above a way.

According to the above embodiment, a panoramic image set is formed by panoramic images, the panoramic images in the panoramic image set are further matched according to the first feature, and the two panoramic images that have completed the first feature matching are regarded as a set of image pairs, so as to determine at least one group of image pairs and corresponding matching results, and finally construct a point cloud model according to the determined at least one group of image pairs and corresponding matching results. Since the panoramic image is used for matching and the point cloud model is further constructed according to the matching result, the number of images in the image set can be reduced, thereby improving the matching efficiency and modeling efficiency, and the spatial range corresponding to the panoramic image is large, so the panoramic image can be improved. The matching effect between images, thereby improving the accuracy and quality of the point cloud model.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure.

1 is a flowchart of a method for constructing a point cloud model according to an embodiment of the present disclosure;

2 is a schematic diagram of a panoramic image according to an embodiment of the present disclosure;

3 is a schematic diagram of a manner of acquiring a first feature of a panoramic image according to an embodiment of the present disclosure;

4 is a schematic diagram of a manner for determining an image pair and a corresponding matching result according to an embodiment of the present disclosure;

5 is a schematic diagram of an angle error shown in an embodiment of the present disclosure;

6 is a schematic diagram of a method for comparing initialization conditions, feature point pairs and a first essential matrix according to an embodiment of the present disclosure;

7 is a structural diagram of an apparatus for constructing a point cloud model according to an embodiment of the present disclosure;

FIG. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.

Detailed ways

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. Where the following description refers to the drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as recited in the appended claims.

The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to limit the present disclosure. As used in this disclosure and the appended claims, the singular forms "a," "the," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various pieces of information, such information should not be limited by these terms. These terms are only used to distinguish the same type of information from each other. For example, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information, without departing from the scope of the present disclosure. Depending on the context, the word "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determining."

With the development of artificial intelligence technology, the spatial modeling technology has become richer and more accurate. The motion recovery structure technology can construct a 3D point cloud model from multiple images, which can be widely used in the fields of physical space digitization, high-precision map construction, and augmented reality. In the related art, the matching effect between multiple images is not good, which leads to low precision and poor quality of the point cloud models constructed sequentially.

Based on this, in the first aspect, at least one embodiment of the present disclosure provides a method for constructing a point cloud model. Please refer to FIG. 1 , which shows a flow of the method, including steps S101 to S103 .

Wherein, the point cloud model is a three-dimensional model corresponding to the space, and the space may refer to the real world. Each object in the real world is represented by a corresponding point cloud in the model, and a point cloud is a collection of points composed of three-dimensional points.

In addition, the method can be executed by electronic equipment such as terminal equipment or server, and the terminal equipment can be user equipment (User Equipment, UE), mobile equipment, user terminal, terminal, cellular phone, cordless phone, Personal Digital Assistant (Personal Digital Assistant, PDA) handheld device, computing device, vehicle-mounted device, wearable device, etc., the method can be implemented by the processor calling the computer-readable instructions stored in the memory. Alternatively, the method may be performed by a server, and the server may be a local server, a cloud server, or the like.

In step S101, the first feature of the panoramic image in the panoramic image set is acquired.

The panoramic image set includes multiple panoramic images, and the panoramic images may be panoramic images from various angles, such as 360° panoramic images; the panoramic images may be images obtained by spherical cameras, or stitched images obtained by multiple fisheye cameras The image obtained by the equidistant rectangular projection method, please refer to FIG. 2, which exemplarily shows a panoramic image. Each panorama image corresponds to a local subspace of the modeling space, and the size of the local subspace corresponding to the panorama image is larger than the size of the local subspace corresponding to the ordinary image under the same parameters, and all the panorama images of the panorama image set correspond to Local subspaces can form the entire modeling space; local subspaces corresponding to different panoramic images can have overlapping regions.

In this step, a pre-trained neural network may be used to obtain the first feature of the panoramic image, or other methods may be used to obtain the first feature of the panoramic image, which is not intended to be specifically limited in the present disclosure. The first feature of each panorama image in the panorama image set may be acquired.

In step S102, at least one set of image pairs in the panoramic image set and corresponding matching results are determined according to the first feature, wherein the image pair includes two panoramic images matched by the first feature, and the matching The result indicates the correspondence between the first features of the two panoramic images.

The first feature matching refers to the existence of a first feature point corresponding to the same space in the two panoramic images, that is, at least one first feature point of one panoramic image and at least one first feature point of another panoramic image, corresponding to reality the same space in the world.

In this step, the panorama images in the panorama image set can be matched by means of brute force matching, that is, the way of traversal is used for matching. Matching: when two panoramic images are matched, the first feature of the two panoramic images is used for matching, that is, the matching result of the panoramic image is determined according to the matching result of the first feature of the two panoramic images. The first features of the two panoramic images are matched, that is, the feature matching of the two panoramic images is completed, then the two images can be determined as a set of image pairs, and the corresponding relationship between the first features of the two panoramic images is determined at the same time. for matching results.

Among them, each panoramic image can form an image pair with another panoramic image, and can also form multiple sets of image pairs with other panoramic images respectively. That is to say, after each panoramic image forms an image pair, it is not locked. You can also continue to compose new image pairs with other panoramic images.

In step S103, a point cloud model is constructed according to the at least one set of image pairs and the corresponding matching results.

In this step, the point cloud model is constructed by using the matching result in the above step S102. The modeling process is also the process of motion recovery structure, including camera registration and point cloud reconstruction. Camera registration is to restore the camera motion parameters of each panoramic image in the panoramic image set (for example, it can be represented by camera pose), point cloud reconstruction It is to restore the point cloud of the three-dimensional structure of the corresponding local subspace (that is, the local subspace mentioned in step S102).

In one example, the point cloud model can be constructed by means of incremental reconstruction.

In the embodiment of the present disclosure, a panoramic image set is composed of panoramic images, the panoramic images in the panoramic image set are further matched according to the first feature, and the two panoramic images whose features have been matched are regarded as a set of image pairs, thereby determining at least one set of at least one set of panoramic images. Image pairs and corresponding matching results, and finally construct a point cloud model according to the determined at least one set of image pairs and corresponding matching results. Since the panoramic image is used for matching and the point cloud model is further constructed according to the matching result, the number of images in the image set can be reduced, thereby improving the matching efficiency and modeling efficiency, and the spatial range corresponding to the panoramic image is large, so the panoramic image can be improved. The matching effect between images, thereby improving the accuracy and quality of the point cloud model.

In some embodiments of the present disclosure, the first feature of the panoramic image of the panoramic image set can be obtained in the following manner, please refer to FIG.

In step S301, multiple perspective images corresponding to the panoramic image are determined, wherein a set of spaces corresponding to the multiple perspective images is a space corresponding to the panoramic image.

In an example, multiple perspective images may be determined in the following manner: first, acquiring the unit sphere corresponding to the panoramic image, and determining a first mapping relationship between the pixel point coordinates of the panoramic image and the point coordinates of the unit sphere; then Next, determine a plurality of perspective images according to the unit sphere, and determine the second mapping relationship between the pixel point coordinates of the perspective image and the point coordinates of the unit sphere, wherein the set of spherical points corresponding to the plurality of perspective images is a unit spherical surface; finally, a third mapping relationship between the pixel coordinates of the panoramic image and the pixel coordinates of the perspective image is determined according to the first mapping relationship and the second mapping relationship, and the pixel information of the pixels of the panoramic image and The third mapping relationship determines pixel information of pixel points of the fluoroscopic image.

Wherein, acquiring the unit sphere corresponding to the panoramic image may be by back-projecting the panoramic image back to the unit sphere. To determine multiple perspective images according to the unit sphere, a virtual perspective camera with the center of the unit sphere as the optical center can shoot the unit sphere, and the obtained image can be determined as a perspective image; when shooting the unit sphere, the virtual camera can be made according to a certain The angle is rotated at equal intervals, and a perspective image is taken for each rotation, until the entire spherical surface is captured, and multiple perspective images covering all viewing angles are obtained. For example, every 60° rotation to take a perspective image requires six perspective images. Only the image can capture the entire spherical surface; the field of view (FOV) and focal length of the virtual camera can be set to the parameters of commonly used image acquisition devices (such as mobile phones, digital cameras, etc.) image.

The first mapping relationship between the pixel point coordinates of the panoramic image and the point coordinates of the unit sphere can be expressed by the following formula (1):

Among them, (s _x , s _y , s _z ) are the point coordinates of the unit sphere, (u, v) are the pixel coordinates of the panoramic image, w is the width of the panoramic image, and h is the height of the panoramic image.

The second mapping relationship between the pixel point coordinates of the perspective image obtained by the virtual camera with the same coordinate system as the unit sphere and the point coordinates of the unit sphere can be expressed by the following formula (2):

Among them, (s _x , s _y , s _z ) are the point coordinates of the unit sphere, (x, y) are the pixel coordinates of the perspective image,

is the width of the perspective image,

is the height of the perspective image, and f is the focal length of the virtual camera.

The second mapping relationship between the pixel point coordinates of the perspective image obtained by the virtual camera rotated by a certain angle with respect to the coordinate system of the unit sphere and the point coordinates of the unit sphere can be expressed by the following formula (3):

Among them, (s' _x , s' _y , s' _z )T=R(s _x ,s _y ,s _z ) ^T is the inverse of the coordinate point (s _x ,s _y ,s _z ) of the unit sphere through the virtual camera The coordinates transformed by the rotation matrix R, (x, y) are the coordinates on the perspective image,

is the width of the perspective image,

In the embodiment of the present disclosure, one of the multiple shooting angles of the virtual camera is the same as the coordinate system of the unit sphere. Therefore, at this angle, the third mapping relationship between the pixel coordinates of the panoramic image and the pixel coordinates of the perspective image is It can be obtained by combining the above formula (1) and formula (2); under other angles, the third mapping relationship between the pixel coordinates of the panoramic image and the pixel coordinates of the perspective image can be obtained by combining the above formula (1) and formula (3) get.

Wherein, determining the pixel information of the pixel point of the fluoroscopic image according to the pixel information of the pixel point of the panoramic image and the third mapping relationship may be to directly determine the pixel information of the pixel point of the panoramic image as the pixel of the corresponding pixel point of the fluoroscopic image The pixel information of the perspective image can also be obtained by sampling and/or bilinear interpolation of the pixel points of the panoramic image. The pixel information may be the luminance value of the pixel, or the value of each color channel (eg, three channels of red, green, and blue) of the pixel.

In step S302, a second feature of at least one fluoroscopic image in the plurality of fluoroscopic images is acquired.

In this step, a pre-trained neural network can be used to extract the second feature of the fluoroscopic image, or other methods can be used to extract the second feature of the fluoroscopic image, and the present disclosure does not intend to make specific restrictions on the extraction method. The second feature of each perspective image corresponding to the panoramic image may be acquired.

In one example, the second feature is the second feature point and the corresponding second descriptor, that is, all the second feature points and the corresponding second descriptor in the fluoroscopic image constitute the second feature of the fluoroscopic image.

In step S303, the first sub-feature of the corresponding position of the panoramic image is determined according to the second feature of the fluoroscopic image, wherein the fluoroscopic image and the corresponding position of the panoramic image correspond to the same space.

Wherein, the corresponding positions of the perspective image and the panoramic image correspond to the same space, that is, the corresponding positions of the perspective image and the panoramic image correspond to the same spherical point set on a single spherical surface.

The first sub-feature may include all the first feature points and corresponding first descriptors in the corresponding positions of the panoramic image.

In an example corresponding to the example of step S302, the first sub-feature of the corresponding position of the panoramic image may be determined in the following manner: First, determine according to the coordinates of the second feature point of the perspective image and the third mapping relationship The coordinates of the first feature point of the panoramic image; next, the first descriptor corresponding to the first feature point of the panoramic image is determined according to the second descriptor corresponding to the second feature point of the fluoroscopic image.

The point on the panoramic image corresponding to the second feature point is the first feature point, that is, the first feature point corresponds to the second feature point, or the spherical point of the unit sphere corresponding to the first feature point is the same as the The spherical points of the unit sphere corresponding to the second feature point are the same. The second descriptor corresponding to the second feature point may be directly used as the first descriptor of the corresponding first feature point.

In step S304, a first feature of the panoramic image is determined according to at least one of the first sub-features.

Wherein, the first feature of the panoramic image includes all the first feature points in the panoramic image and the corresponding first descriptors.

In the embodiment of the present disclosure, the mapping relationship between the panoramic image and the perspective image is determined through the mapping relationship between the panoramic image and the unit sphere and the mapping relationship between the unit sphere and the perspective image, that is, the unit sphere is used as the medium to map the panoramic image. The image is divided into a plurality of perspective images, and the extraction of the first feature of the panoramic image is further achieved by extracting the second feature of the perspective image and inversely mapping the second feature point back to the first feature point of the panoramic image.

In some embodiments of the present disclosure, the first feature includes a first feature point and a corresponding first descriptor. Correspondingly, at least one of the panoramic image sets may be determined according to the first feature in the following manner. For a set of image pairs and corresponding matching results, please refer to FIG. 4 , which shows the flow of the above determination method, including steps S401 to S403 .

In step S401, multiple groups of image pairs are determined according to each panoramic image and the corresponding panoramic image to be matched.

In this step, the two panoramic images form a set of image pairs. The corresponding panorama image to be matched may be determined according to the space corresponding to each panorama image; or the panorama image to be matched corresponding to each panorama image may be determined according to a preset collocation rule. That is to say, when determining a panoramic image of a panoramic image to be matched, the panoramic image that overlaps with the corresponding space of the panoramic image may be used as the panoramic image to be matched, and may also be determined according to a preset matching rule. The collocation rules can be determined according to the above principles, for example, the panoramic images are numbered according to the corresponding spatial order, and then a preset number (eg, 10) of panoramic images after each panoramic image are used as the panoramic images to be matched. All other panoramic images other than one panoramic image may also be used as the panoramic image to be matched for the panoramic image.

In step S402, a plurality of sets of feature point pairs are determined according to the first descriptors of the two panoramic images of the image pair, wherein each set of the feature point pairs includes two correspondingly matched first descriptors belonging to the two panoramic images. a feature point.

In this step, the first descriptor with the closest Euclidean distance can be searched for each first descriptor in the first panoramic image of the image pair in the second panoramic image, and then reversely, for the first descriptor of the image pair For each first descriptor in the second panoramic image, find the first descriptor with the closest Euclidean distance in the first panoramic image, if a certain first descriptor in the first panoramic image and the second panoramic image A certain first descriptor in the image is the first descriptor with the closest Euclidean distance in another panoramic image, then the above two first descriptors are considered to match, and then the two corresponding first descriptors are determined. One first feature point matching, that is, two first feature points complete feature matching to form a feature point pair.

In this step, after all feature point pairs between the two panoramic images of the image pair are determined in the above manner, the number of feature point pairs can also be counted, the first condition is preset, and the first condition is used to filter the data determined in step S401. Multiple sets of image pairs, that is, some image pairs can be removed by using the first condition. In one example, the first condition may be less than the second number threshold, that is, filter image pairs whose number of feature point pairs is less than the second number threshold, that is, remove image pairs whose number of feature point pairs is less than the second number threshold, for example , the second quantity threshold may be set to 5 or 10, etc. The embodiments of the present disclosure do not intend to impose specific limitations on the specific value of the second quantity threshold. By filtering out some image pairs, subsequent operations on image pairs with low matching degrees can be reduced, thereby reducing the complexity of operations and improving processing efficiency.

Among them, each group of feature point pairs represents the corresponding relationship between the two first feature points, and multiple groups of feature point pairs constitute the matching result of the image pairs.

In step S403, a first essential matrix is determined according to the multiple sets of feature point pairs, and the multiple sets of feature point pairs are filtered by using the first essential matrix to obtain a matching result corresponding to the image pair.

In this step, the first essential matrix may be determined in the following manner: first, the angle error of the pair of feature points is determined according to the angle error of the two first feature points in the pair of feature points, wherein the first The angle error of the feature point is the angle between the line connecting the spherical point of the unit sphere corresponding to the first feature point and the optical center of the unit sphere and the outer polar plane; next, the angle error of the corresponding pair of feature points is the residual item, and calculates the essential matrix according to the preset number of the feature point pairs for many times; finally, the number of interior points corresponding to each essential matrix is determined, and the essential matrix with the largest number of interior points is determined as the first essential matrix .

Wherein, the point coordinates of the unit sphere corresponding to the two first feature points of each group of feature point pairs, and the essential matrix satisfy the relationship shown in the following formula (4):

(s _x ′,s _y ′,s _z ′)E(s _x ,s _y ,s _z ) ^T =0 (4)

In the above formula (4), E is the essential matrix, S'=(s _x ', s _y ', s _z ') is the point coordinate of the unit sphere corresponding to the first feature point in the first panoramic image, S = (s _x , s _y , s _z ) are the point coordinates of the unit sphere corresponding to the first feature point in the second panoramic image.

Among them, the larger of the angle errors of the two first feature points is determined as the angle error of the feature point pair, that is, the angle error of the feature point pair is determined according to the following formula (5):

In the above formula (5),

That is, the angle error of the first feature point of the second panoramic image, that is, the difference between the connection line between the spherical point S' corresponding to the first feature point on the second panoramic image and the optical center O ₂ and the corresponding outer polar plane. The included angle, wherein the corresponding outer polar plane is formed by the connection line between the spherical point S corresponding to the first feature point on the first panoramic image and the optical center O ₁ and the connection line between the two optical centers Q ₁ and Q ₂ flat. Correspondingly, please refer to Figure 5,

is the angle error of the first feature point of the first panoramic image, that is, the angle between the connection line between the spherical point and the optical center corresponding to the first feature point on the first panoramic image and the corresponding epipolar plane, where , the corresponding outer polar plane is the plane formed by the connection line between the spherical point corresponding to the first feature point on the second panoramic image and the optical center and the connection line between the two optical centers. In the embodiment of the present disclosure, the angle error and the spherical error can be used to better adapt to the camera model of the panoramic image.

Among them, multiple essential matrices can be calculated by RANSAC (Random Sample Consensus) and the 5-point algorithm with the angle error of the feature point pair as the residual item, and an essential matrix can be calculated for every 5 sets of feature point pairs. Therefore, the above method is used. Multiple essential matrices can be derived.

Wherein, the number of interior points corresponding to the essential matrix can be determined in the following manner: first, the angle error of each group of feature point pairs of the image pair is calculated according to the essential matrix; next, it is determined that the angle error conforms to the preset first The feature point pairs of the two conditions are interior points; finally, the number of interior points corresponding to the essential matrix is determined according to all interior points. That is, the above formula (5) and the essential matrix are used to determine the angle error of each group of feature point pairs; the second condition is preset, and the second condition is used to filter the interior points. In an example, the second condition may be that the angle error is less than the first An angle threshold, that is, the feature point pair corresponding to the angle error smaller than the first angle threshold is determined as an interior point.

In this step, the first essential matrix may be used to filter the multiple sets of feature point pairs: first, the angle error of each set of feature point pairs of the image pair is determined according to the first essential matrix ; Next, filter the feature point pairs whose angle error meets the preset third condition.

Wherein, the above-mentioned formula (5) and the first essential matrix can be used to calculate the angle error of each group of feature point pairs; a third condition is preset, and the feature point pair is screened by the third condition. In an example, the third condition can be The angle error is greater than or equal to the second angle threshold (such as greater than or equal to 0.4 degrees), that is, filtering the feature point pairs corresponding to the angle error greater than or equal to the second angle threshold, that is, removing the angle error greater than or equal to the second angle threshold corresponding to The feature point pair corresponding to the angle error smaller than the second angle threshold is reserved.

In the embodiment of the present disclosure, the feature point pairs between the image pairs are determined by performing feature matching on the image pairs, and the first essential matrix is further determined according to the feature point pairs, and finally the first essential matrix is used to filter the feature point pairs, and in the The angle error is used when determining the first essential matrix and filtering the feature point pairs. Therefore, compared with other essential matrices, the feature point pairs consistent with the first essential matrix are the most, and the filtering step will be the feature points that are inconsistent with the first essential matrix. Pair removal not only improves the accuracy of the first essential matrix, but also maximizes the number of feature point pairs on the premise of removing wrong feature point pairs, thereby improving the matching accuracy of the two panoramic images of the image pair. and accuracy. Moreover, the 360-degree viewing angle coverage of the panoramic image is used to increase the number of feature matches between images and reduce the probability of camera registration failure in weak texture areas.

In some embodiments of the present disclosure, after completing the calculation of the first essential matrix of each group of image pairs and the filtering of the feature point pairs, the distribution of the first feature points can also be used to determine whether the matching of the two panoramic images is determined by repeated texture, and further filter multiple groups of image pairs, specifically, the following methods can be used: first, obtain the coordinates of the two first feature points of the feature point pair of the image pair in the panorama image to which they belong; The third mapping relationship corresponding to the panoramic image and the coordinates of the first feature point belonging to the feature point pair on the panoramic image determine a perspective image related to the feature point pair, wherein the perspective image related to the feature point pair The image is a fluoroscopic image in which a second feature point corresponding to the first feature point belonging to the feature point pair exists; finally, the image pair is filtered using the fluoroscopic image associated with the feature point pair.

It is also possible to preset a third quantity threshold, and determine the quantity of the second feature points corresponding to the pair of feature points included in the fluoroscopic image, and only when the quantity is greater than or equal to the above-mentioned third quantity threshold, the fluoroscopic image is determined to be the same as the feature point. For point-to-correlation fluoroscopic images, for example, the above-mentioned third number threshold can be set to 5, which can avoid erroneous statistics caused by a small amount of noise matching.

Wherein, in response to the perspective images related to the feature point pair corresponding to at least one panoramic image of the image pair being consecutive multiple images, and the number of the consecutive multiple images is less than a preset filtering threshold, filtering the image pair. That is to say, when the second feature points corresponding to the first feature points related to the feature point pair are all concentrated in the partial fluoroscopic image, and the number of the partial fluoroscopic images is less than the preset filtering threshold, it is considered that the two panoramic images are A match is a false match caused by a repeating texture, so the image pair is filtered out, ie, the image pair is removed. The filtering threshold can be determined according to the total amount of fluoroscopic images and a preset first ratio. For example, if the total amount of fluoroscopic images is 6 and the preset first ratio is 0.5, it is considered that the number of fluoroscopic images related to feature point pairs is less than 3. , the image pair is filtered out.

In the embodiment of the present disclosure, the first feature points for which feature matching is completed all correspond to the second feature points of the fluoroscopic image. Therefore, by determining the distribution of the second feature points, it can be determined whether the matching of the two panoramic images is caused by repeated textures, and By removing noise matching, the accuracy of the above judgment can be further improved, and erroneously matched image pairs can be excluded. Most of the repeated textures are local, and the global matching between panoramic images is used to eliminate false matching as much as possible to avoid the error rate of camera registration based on this.

In some embodiments of the present disclosure, a point cloud model may be constructed according to the at least one group of image pairs and the corresponding matching results in the following manner: First, according to preset initialization conditions and feature points of each group of image pairs and the first essential matrix to determine a set of image pairs as initial image pairs, and determine the camera pose of each panoramic image of the initial image pair, and triangulate the first feature point pair of the initial image pair to form The initial three-dimensional point; next, according to the matching relationship between the first feature point corresponding to the first three-dimensional point and the first feature point of each unregistered image, an unregistered image is determined as a registered image, until the Each panoramic image is a registered image, wherein the unregistered image is a panoramic image in which all the first feature points are not triangulated, and the registered image is a panoramic image with triangulated first feature points, so The first three-dimensional point includes the initial three-dimensional point, or includes the three-dimensional point formed by triangulation of the initial three-dimensional point and the first feature point of the registered image; and, after each time the registered image is determined, the registering the camera pose of the image, triangulating the first feature point of the registered image to form a corresponding three-dimensional point, and triangulating the third feature point of the registered image to form a corresponding three-dimensional point, the The third feature point is the first feature point in the registered image that matches the first feature point of the registered image.

Wherein, when determining the initial image pairs, the image pairs may be selected in descending order of the number of feature point pairs, and after each image pair is selected, the Whether the image pair satisfies the initialization condition, until the selected image pair satisfies the initialization condition, it is determined that the selected image pair is the initial image pair.

Moreover, the following method can be used to determine whether the image pair satisfies the initialization condition according to the feature point pair and the first essential matrix. Please refer to FIG. 6 , which shows the flow of the above method, including step S601 Go to step S604.

In step S601, at least one group (for example, four groups) of displacement variables is determined according to the first essential matrix of the image pair, and the feature points of the pair of feature points are triangulated for each group of displacement variables to form the corresponding displacement variables of each group and filtering the three-dimensional points according to the reprojection error and triangulation angle of each group of three-dimensional points, wherein the displacement variables include rotation variables and translation variables.

Among them, the rotation variable can be represented by a 3*3 matrix R, and the translation variable can be represented by a 3-dimensional vector T. When calculating the reprojection error of each group of three-dimensional points X in the ith panoramic image (ie, the first or second panoramic image), the following formula can be used:

In the above formula: S=(s _x , s _y , s _z ) is the point coordinate of the unit sphere corresponding to the first feature point, X is the three-dimensional point coordinate vector, f is the focal length of the virtual perspective camera, let P=[R _i |T _i ] is the camera matrix of the ith panoramic image. During initialization, R ₁ =I is a unit matrix, T ₁ =0 is a zero vector, R ₂ =R, T ₂ =T, and R and T are respectively The rotation variables and translation variables corresponding to the set of three-dimensional points.

Wherein, when filtering the three-dimensional points according to the re-projection error and triangulation angle of each group of three-dimensional points, a third angle threshold and a fourth angle threshold may be set, and then the re-projection errors retained in the two panoramic images are both smaller than the third angle threshold. Three-dimensional points with an angle threshold and a triangulation angle greater than a fourth angle threshold.

In step S602, in response to the number of the largest group of three-dimensional points being greater than the preset first number threshold, determine the corresponding displacement variable as the first displacement variable.

In step S603, select an essential matrix whose inner number of points is greater than or equal to the threshold of the number of points from the essential matrices obtained by multiple calculations, and determine at least one group (for example, four groups) of displacement variables according to each essential matrix, and for each group of displacement variables Triangulate the feature points of the feature point pairs respectively to form the three-dimensional points corresponding to each group of displacement variables, and filter the three-dimensional points according to the reprojection error and triangulation angle of each group of three-dimensional points, and retain the one with the largest number of each essential matrix. Displacement variables corresponding to a set of 3D points.

Wherein, after calculating and obtaining a plurality of essential matrices in step S403 and determining the first essential matrix, the remaining essential matrices can be reserved for use in this step; Essential Matrix.

Wherein, a second ratio can be preset, and then the threshold of points can be determined by using the number of inner points of the first essential matrix and the above-mentioned second ratio; the product of the number of inner points of the first essential matrix and the second ratio can be used as the threshold of points. The second ratio is preset to be 0.6, but the present disclosure does not intend to limit the specific value of the second ratio.

The operations performed on the selected essential matrix in this step are the same as the operations in steps S601 to S602, and a displacement variable is reserved for each essential matrix.

In step S604, if the difference between the displacement variable retained by each essential matrix and the first displacement variable satisfies a preset range, it is determined that the image pair satisfies the initialization condition.

Wherein, the difference between the displacement variable retained by the essential matrix and the first displacement variable can be represented by the angle between the directions of the two displacement variables, and the angle between the directions is obtained by multiplying the rotation matrices of the two displacement variables; the preset range can be expressed as A preset fifth angle threshold value indicates that the value less than the fifth angle threshold value satisfies the preset range; therefore, when the included angle between the displacement variable retained by each essential matrix and the direction of the first displacement variable is smaller than the fifth angle threshold value, determine The image pair satisfies the initialization condition.

In the case that the image pair does not satisfy the initialization condition, steps S601 to S604 are used to continue to judge whether other image pairs satisfy the initialization condition.

In the above process of judging the initialization conditions, the judgment of multiple solutions of each essential matrix can make the judgment result of the initialization conditions more stable.

In addition, the camera pose of each panoramic image and the position of the initial 3D point can also be optimized by minimizing the reprojection error of the initial 3D point on the two panoramic images of the initial image pair;

In addition, after each time the camera pose of the registration image is determined, the camera pose of the registration image can also be optimized by minimizing the reprojection error of the three-dimensional point on the registration image;

In addition, each time the feature points of the registered image are triangulated to form corresponding three-dimensional points, and the third feature points of the registered images are triangulated to form corresponding three-dimensional points, each time can also be minimized by minimizing each The reprojection error of 3D points on each registered image, optimizing the camera pose for each registered image and the position of each 3D point.

The above optimization can be optimized using the following loss function:

In the above formula (7), S=(s _x , s _y , s _z ) is the point coordinates of the unit sphere corresponding to the first feature point, X _i is the coordinate vector of the i-th three-dimensional point, and f is the virtual perspective camera. Focal length, let P=[R _i |T _i ] be the camera matrix of the ith panoramic image, during initialization, R ₁ =I is the identity matrix, T ₁ =0 is a zero vector, R ₂ =R,T ₂ = T, and R and T are the rotation variables and translation variables corresponding to the three-dimensional point, respectively, m is the number of panoramic images, and n is the number of three-dimensional points.

The above-mentioned determination of the camera pose of the image can be performed using RANSAC (Random Sample Consensus) and P3P algorithms.

Based on the above description of the point cloud model construction method, it can be seen that this method can use panoramic images for camera registration and point cloud reconstruction, thereby completing the construction of point cloud models, and the point cloud model constructed based on panoramic images is better than the traditional one based on ordinary perspective images. The constructed point cloud model has higher accuracy, better robustness to repeated textures, and more comprehensive scene reconstruction. This method can be used to build a high-precision visual map, providing visual features and 3D landmark points for positioning for autonomous driving and AR, and can also use this method to build a 3D model of a specific scene for scene display and VR applications, such as tourist attractions, museums , AR/VR tour of the exhibition hall, or build a 3D model of a building, a block or a city, AR special effects, etc.

According to a second aspect of the embodiments of the present disclosure, an apparatus for constructing a point cloud model is provided. Please refer to FIG. 7 , which shows the structure of the apparatus, including:

The obtaining module 701 is configured to obtain the first feature of the panoramic image in the panoramic image set;

A matching module 702, configured to determine at least one set of image pairs in the panoramic image set and corresponding matching results according to the first feature, wherein the image pair includes two panoramic images matched by the first feature, the The matching result indicates the correspondence between the first features of the two panoramic images;

The construction module 703 is configured to construct a point cloud model according to the at least one set of image pairs and corresponding matching results.

In one embodiment, the obtaining module is specifically configured as:

Determine the first sub-feature of the corresponding position of the panoramic image according to the second feature of the fluoroscopic image, wherein the fluoroscopic image and the corresponding position of the panoramic image correspond to the same space;

Determine multiple perspective images according to the unit sphere, and determine the second mapping relationship between the pixel point coordinates of the perspective image and the point coordinates of the unit sphere, wherein, the set of spherical points corresponding to the multiple perspective images is the unit sphere;

The matching module is specifically configured as:

In one embodiment, the matching module is further configured to:

In one embodiment, the building module is specifically configured as:

In one embodiment, the building block is further configured to:

In one embodiment, the matching module is further configured to:

The panorama image to be matched corresponding to each panorama image is determined according to a preset collocation rule. Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment of the method related to the first aspect, and will not be described in detail here.

Referring to FIG. 8 , according to a third aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device includes a memory 801 and a processor 802 , and the memory 801 is used to store a computer that can run on the processor 802 . Instructions, the processor 802 is configured to construct a point cloud model based on the method described in the first aspect when executing the computer instructions. The memory 801 may use a volatile or non-volatile storage medium.

According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the method of the first aspect. Computer-readable storage media can be volatile or non-volatile storage media.

The present disclosure also provides a computer program product, including computer-readable code, when the computer-readable code is run on a device, a processor in the device executes instructions for implementing the method for constructing a point cloud model provided in any of the above embodiments .

The computer program product can be specifically implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.

In the present disclosure, the terms "first" and "second" are used for descriptive purposes only, and should not be construed as indicating or implying relative importance. The term "plurality" refers to two or more, unless expressly limited otherwise.

Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common general knowledge or techniques in the technical field not disclosed by this disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Industrial Applicability

Embodiments of the present disclosure provide a point cloud model construction method, device, device, and storage medium, wherein the method includes: acquiring a first feature of a panoramic image in a panoramic image set; determining the panoramic image according to the first feature At least one group of image pairs in the set and corresponding matching results, wherein the image pairs include two panoramic images matched by a first feature, and the matching results indicate the correspondence between the first features of the two panoramic images; according to The at least one set of image pairs and the corresponding matching results construct a point cloud model. Since the panoramic image is used for matching and the point cloud model is further constructed according to the matching result, the number of images in the image set can be reduced, thereby improving the matching efficiency and modeling efficiency, and the spatial range corresponding to the panoramic image is large, so the panoramic image can be improved. The matching effect between images, thereby improving the accuracy and quality of the point cloud model.

Claims

A method for constructing a point cloud model, which is applied to electronic equipment, including:

obtaining the first feature of the panoramic image in the panoramic image set;

At least one set of image pairs in the panoramic image set and corresponding matching results are determined according to the first feature, wherein the image pair includes two panoramic images matched by the first feature, and the matching results indicate two panoramic images the correspondence between the first features of the image;

A point cloud model is constructed according to the at least one set of image pairs and the corresponding matching results.
The method for constructing a point cloud model according to claim 1, wherein the acquiring the first feature of the panoramic image in the panoramic image set comprises:

determining multiple perspective images corresponding to the panoramic image, wherein the set of spaces corresponding to the multiple perspective images is the space corresponding to the panoramic image;

acquiring a second feature of at least one fluoroscopic image in the plurality of fluoroscopic images;

Determine the first sub-feature of the corresponding position of the panoramic image according to the second feature of the fluoroscopic image, wherein the fluoroscopic image corresponds to the same space as the corresponding position of the panoramic image;

A first feature of the panoramic image is determined based on at least one of the first sub-features.
The method for constructing a point cloud model according to claim 2, wherein the determining a plurality of perspective images corresponding to the panoramic image comprises:

acquiring the unit sphere corresponding to the panoramic image, and determining the first mapping relationship between the pixel point coordinates of the panoramic image and the point coordinates of the unit sphere;

Determine a plurality of perspective images according to the unit sphere, and determine the second mapping relationship between the pixel point coordinates of the perspective image and the point coordinates of the unit sphere, wherein the set of spherical points corresponding to the plurality of perspective images is the unit sphere;

The third mapping relationship between the pixel point coordinates of the panoramic image and the pixel point coordinates of the fluoroscopic image is determined according to the first mapping relationship and the second mapping relationship, and the third mapping relationship between the pixel point coordinates of the panoramic image and the third mapping relationship is determined according to the first mapping relationship and the second mapping relationship. The mapping relationship determines the pixel information of the pixels of the fluoroscopic image.
The method for constructing a point cloud model according to claim 3, wherein the second feature of the perspective image comprises a second feature point and a corresponding second descriptor;

The determining of the first sub-feature of the corresponding position of the panoramic image according to the second feature of the fluoroscopic image includes:

Determine the coordinates of the first feature point of the panoramic image according to the coordinates of the second feature point of the fluoroscopic image and the third mapping relationship;

The first descriptor corresponding to the first feature point of the panoramic image is determined according to the second descriptor corresponding to the second feature point of the fluoroscopic image.
The method for constructing a point cloud model according to claim 4, wherein the first feature comprises a first feature point and a corresponding first descriptor;

The determining of at least one group of image pairs in the panoramic image set and the corresponding matching results according to the first feature includes:

Determine multiple groups of image pairs according to each panoramic image and the corresponding panoramic image to be matched;

A plurality of sets of feature point pairs are determined according to the first descriptors of the two panoramic images of each set of the image pairs, wherein each set of the feature point pairs includes two correspondingly matched first feature points belonging to the two panoramic images ;

A first essential matrix is determined according to the multiple sets of feature point pairs, and the multiple sets of feature point pairs are filtered by using the first essential matrix to obtain matching results corresponding to the image pairs.
The method for constructing a point cloud model according to claim 5, wherein after determining multiple groups of feature point pairs according to the first descriptors of the two panoramic images of each group of the image pairs, the method further comprises:

Obtain the number of feature point pairs of the two panoramic images of each group of the image pairs;

The image pairs whose number of feature point pairs meet the preset first condition are filtered.
The method for constructing a point cloud model according to claim 5 or 6, wherein determining the first essential matrix according to the multiple sets of feature point pairs, comprising:

The angle error of the feature point pair is determined according to the angle error of the two first feature points in the feature point pair, wherein the angle error of the first feature point is the angular error of the unit sphere corresponding to the first feature point The angle between the line connecting the spherical point and the optical center of the unit sphere and the outer polar plane;

Taking the angular error of the corresponding pair of feature points as the residual term, determining the essential matrix according to the preset number of pairs of the feature points for many times;

The number of interior points corresponding to each of the essential matrices is determined, and the essential matrix with the largest number of interior points is determined as the first essential matrix.
The method for constructing a point cloud model according to claim 7, wherein said determining the number of interior points corresponding to each said essential matrix comprises:

Determine the angle error of each group of feature point pairs of the image pair according to the essential matrix;

It is determined that the feature point pair whose angle error meets the preset second condition is an interior point;

The number of interior points corresponding to the essential matrix is determined according to all interior points.
The method for constructing a point cloud model according to claim 7 or 8, wherein the filtering of multiple sets of feature point pairs by using the first essential matrix comprises:

Determine the angle error of each group of feature point pairs of the image pair according to the first essential matrix;

The feature point pairs whose angle errors meet the preset third condition are filtered.
The point cloud model construction method according to any one of claims 5-9, wherein, further comprising:

Obtaining the coordinates of the two first feature points of the feature point pair of the image pair in the panorama image to which they belong;

A perspective image related to the feature point pair is determined according to the third mapping relationship corresponding to the panoramic image and the coordinates of the first feature point belonging to the feature point pair on the panoramic image, wherein the feature point pair-related The fluoroscopic image is a fluoroscopic image in which there is a second feature point corresponding to the first feature point belonging to the feature point pair;

The image pairs are filtered using the fluoroscopic images associated with the feature point pairs.
The method for constructing a point cloud model according to claim 10, wherein the filtering the image pair by using the perspective image related to the feature point pair comprises:

In response to the fluoroscopic images related to the feature point pair corresponding to at least one panoramic image of the image pair being consecutive multiple images, and the number of the consecutive multiple images is less than a preset filtering threshold, filtering the image pair .
The method for constructing a point cloud model according to any one of claims 5-11, wherein the constructing a point cloud model according to the at least one group of image pairs and corresponding matching results, comprising:

Determine a group of image pairs as initial image pairs according to preset initialization conditions, feature point pairs of each group of image pairs and the first essential matrix, and determine the camera pose of each panoramic image of the initial image pair, and triangulating the first feature point pair of the initial image pair to form an initial three-dimensional point;

According to the matching relationship between the first feature point corresponding to the first three-dimensional point and the first feature point of each unregistered image, an unregistered image is determined as a registered image, until each panoramic image in the panoramic image set is a registered image. A registered image, wherein the unregistered image is a panoramic image in which all the first feature points are not triangulated, the registered image is a panoramic image with triangulated first feature points, and the first three-dimensional point includes all the initial three-dimensional point, or a three-dimensional point formed by triangulating the initial three-dimensional point and the first feature point of the registered image;

After each registration image is determined, the camera pose of the registration image is determined, the first feature point of the registration image is triangulated to form a corresponding three-dimensional point, and the third feature point of the registered image is triangulated. to form a corresponding three-dimensional point, and the third feature point is a first feature point in the registered image that matches the first feature point of the registered image.
The method for constructing a point cloud model according to claim 12, wherein determining a group of image pairs as initial image pairs according to preset initialization conditions, feature point pairs of each group of image pairs and the first essential matrix, comprising: :

Select image pairs in descending order of the number of feature point pairs. After each image pair is selected, it is determined whether the image pair satisfies the initialization condition according to the feature point pair and the first essential matrix. Until the selected image pair satisfies the initialization condition, it is determined that the selected image pair is an initial image pair.
The method for constructing a point cloud model according to claim 13, wherein the determining whether the image pair satisfies the initialization condition according to the feature point pair and the first essential matrix comprises:

Determine at least one group of displacement variables according to the first essential matrix of the image pair, and triangulate the feature points of the feature point pair for each group of displacement variables to form three-dimensional points corresponding to each group of displacement variables, and according to each group of three-dimensional points The reprojection error and triangulation angle filter the three-dimensional point, wherein the displacement variable includes a rotation variable and a translation variable;

In response to the number of the largest group of three-dimensional points being greater than the preset first number threshold, determining the corresponding displacement variable as the first displacement variable;

Select an essential matrix with the number of inner points greater than or equal to the threshold of the number of points from the essential matrices obtained by multiple calculations, determine at least one set of displacement variables according to each essential matrix, and triangulate the feature points of the feature point pair for each set of displacement variables respectively, To form three-dimensional points corresponding to each group of displacement variables, and filter the three-dimensional points according to the reprojection error and triangulation angle of each group of three-dimensional points, and retain the displacement variables corresponding to a group of three-dimensional points with the largest number of each essential matrix;

In the case that the difference between the displacement variable retained by each essential matrix and the first displacement variable satisfies a preset range, it is determined that the image pair satisfies the initialization condition.
The method for constructing a point cloud model according to claim 12, further comprising:

optimize the camera pose of each panorama image and the position of the initial 3D point by minimizing the reprojection error of the initial 3D point on the two panorama images of the initial image pair; and/or,

After each determination of the camera pose of the registration image, optimize the camera pose of the registration image by minimizing the reprojection error of 3D points on the registration image; and/or,

After each time the feature points of the registered image are triangulated to form corresponding three-dimensional points, and the third feature points of the registered images are triangulated to form corresponding three-dimensional points, by minimizing each three-dimensional point in each The reprojection error on each registered image, optimizes the camera pose for each registered image and the position of each 3D point.
The method for constructing a point cloud model according to claim 5, further comprising:

Determine the corresponding panoramic image to be matched according to the space corresponding to each panoramic image; or,

The panorama image to be matched corresponding to each panorama image is determined according to a preset collocation rule.
A point cloud model construction device, comprising:

an acquisition module, configured to acquire the first feature of the panoramic image in the panoramic image set;

a matching module, configured to determine at least one set of image pairs in the panoramic image set and corresponding matching results according to the first feature, wherein the image pairs include two panoramic images matched by the first feature, and the matching the result indicates the correspondence between the first features of the two panoramic images;

The building module is configured to build a point cloud model according to the at least one set of image pairs and corresponding matching results.
The device for constructing a point cloud model according to claim 17, wherein the obtaining module, configured to obtain the first feature of the panoramic image in the panoramic image set, comprises:

determining multiple perspective images corresponding to the panoramic image, wherein the set of spaces corresponding to the multiple perspective images is the space corresponding to the panoramic image;

acquiring a second feature of at least one fluoroscopic image in the plurality of fluoroscopic images;

Determine the first sub-feature of the corresponding position of the panoramic image according to the second feature of the fluoroscopic image, wherein the fluoroscopic image corresponds to the same space as the corresponding position of the panoramic image;

A first feature of the panoramic image is determined based on at least one of the first sub-features.
The device for constructing a point cloud model according to claim 18, wherein the acquisition module is configured to determine a plurality of perspective images corresponding to the panoramic image, comprising:

acquiring the unit sphere corresponding to the panoramic image, and determining the first mapping relationship between the pixel point coordinates of the panoramic image and the point coordinates of the unit sphere;

Determine a plurality of perspective images according to the unit sphere, and determine the second mapping relationship between the pixel point coordinates of the perspective image and the point coordinates of the unit sphere, wherein the set of spherical points corresponding to the plurality of perspective images is the unit sphere;

The third mapping relationship between the pixel point coordinates of the panoramic image and the pixel point coordinates of the fluoroscopic image is determined according to the first mapping relationship and the second mapping relationship, and the third mapping relationship between the pixel point coordinates of the panoramic image and the third mapping relationship is determined according to the first mapping relationship and the second mapping relationship. The mapping relationship determines the pixel information of the pixels of the fluoroscopic image.
The device for constructing a point cloud model according to claim 19, wherein the second feature of the perspective image comprises a second feature point and a corresponding second descriptor;

The acquisition module, configured to determine the first sub-feature of the corresponding position of the panoramic image according to the second feature of the fluoroscopic image, includes:

Determine the coordinates of the first feature point of the panoramic image according to the coordinates of the second feature point of the fluoroscopic image and the third mapping relationship;

The first descriptor corresponding to the first feature point of the panoramic image is determined according to the second descriptor corresponding to the second feature point of the fluoroscopic image.
The device for constructing a point cloud model according to claim 20, wherein the first feature comprises a first feature point and a corresponding first descriptor;

The matching module, configured to determine at least one group of image pairs in the panoramic image set and corresponding matching results according to the first feature, includes:

Determine multiple groups of image pairs according to each panoramic image and the corresponding panoramic image to be matched;

A plurality of sets of feature point pairs are determined according to the first descriptors of the two panoramic images of each set of the image pairs, wherein each set of the feature point pairs includes two correspondingly matched first feature points belonging to the two panoramic images ;

A first essential matrix is determined according to the multiple sets of feature point pairs, and the multiple sets of feature point pairs are filtered by using the first essential matrix to obtain matching results corresponding to the image pairs.
The device for constructing a point cloud model according to claim 21, wherein the matching module, after being configured to determine multiple sets of feature point pairs according to the first descriptors of the two panoramic images of each set of the image pairs, further comprises:

Obtain the number of feature point pairs of the two panoramic images of each group of the image pairs;

The image pairs whose number of feature point pairs meet the preset first condition are filtered.
The device for constructing a point cloud model according to claim 21 or 22, wherein the matching module, configured to determine the first essential matrix according to the multiple sets of feature point pairs, comprises:

The angle error of the feature point pair is determined according to the angle error of the two first feature points in the feature point pair, wherein the angle error of the first feature point is the angular error of the unit sphere corresponding to the first feature point The angle between the line connecting the spherical point and the optical center of the unit sphere and the outer polar plane;

Taking the angular error of the corresponding pair of feature points as the residual term, determining the essential matrix according to the preset number of pairs of the feature points for many times;

The number of interior points corresponding to each of the essential matrices is determined, and the essential matrix with the largest number of interior points is determined as the first essential matrix.
The device for constructing a point cloud model according to claim 23, wherein the matching module, configured to determine the number of interior points corresponding to each of the essential matrices, comprises:

Determine the angle error of each group of feature point pairs of the image pair according to the essential matrix;

It is determined that the feature point pair whose angle error meets the preset second condition is an interior point;

The number of interior points corresponding to the essential matrix is determined according to all interior points.
The device for constructing a point cloud model according to claim 23 or 24, wherein the filtering of multiple sets of feature point pairs by using the first essential matrix comprises:

Determine the angle error of each group of feature point pairs of the image pair according to the first essential matrix;

The feature point pairs whose angle errors meet the preset third condition are filtered.
The device for constructing a point cloud model according to any one of claims 21-25, further comprising:

Obtaining the coordinates of the two first feature points of the feature point pair of the image pair in the panorama image to which they belong;

A perspective image related to the feature point pair is determined according to the third mapping relationship corresponding to the panoramic image and the coordinates of the first feature point belonging to the feature point pair on the panoramic image, wherein the feature point pair-related The fluoroscopic image is a fluoroscopic image in which there is a second feature point corresponding to the first feature point belonging to the feature point pair;

The image pairs are filtered using the fluoroscopic images associated with the feature point pairs.
The device for constructing a point cloud model according to claim 26, wherein the filtering of the image pair by using the perspective image related to the feature point pair comprises:

In response to the fluoroscopic images related to the feature point pair corresponding to at least one panoramic image of the image pair being consecutive multiple images, and the number of the consecutive multiple images is less than a preset filtering threshold, filtering the image pair .
The device for constructing a point cloud model according to any one of claims 21-27, wherein the constructing the point cloud model according to the at least one set of image pairs and corresponding matching results comprises:

Determine a group of image pairs as initial image pairs according to preset initialization conditions, feature point pairs of each group of image pairs and the first essential matrix, and determine the camera pose of each panoramic image of the initial image pair, and triangulating the first feature point pair of the initial image pair to form an initial three-dimensional point;

According to the matching relationship between the first feature point corresponding to the first three-dimensional point and the first feature point of each unregistered image, an unregistered image is determined as a registered image, until each panoramic image in the panoramic image set is a registered image. A registered image, wherein the unregistered image is a panoramic image in which all the first feature points are not triangulated, the registered image is a panoramic image with triangulated first feature points, and the first three-dimensional point includes all the initial three-dimensional point, or a three-dimensional point formed by triangulating the initial three-dimensional point and the first feature point of the registered image;

After each registration image is determined, the camera pose of the registration image is determined, the first feature point of the registration image is triangulated to form a corresponding three-dimensional point, and the third feature point of the registered image is triangulated. to form a corresponding three-dimensional point, and the third feature point is a first feature point in the registered image that matches the first feature point of the registered image.
The device for constructing a point cloud model according to claim 28, wherein determining a group of image pairs as initial image pairs according to preset initialization conditions, feature point pairs of each group of image pairs and the first essential matrix, comprising: :

Select image pairs in descending order of the number of feature point pairs. After each image pair is selected, it is determined whether the image pair satisfies the initialization condition according to the feature point pair and the first essential matrix. Until the selected image pair satisfies the initialization condition, it is determined that the selected image pair is an initial image pair.
The device for constructing a point cloud model according to claim 29, wherein determining whether the image pair satisfies the initialization condition according to the feature point pair and the first essential matrix comprises:

Determine at least one group of displacement variables according to the first essential matrix of the image pair, and triangulate the feature points of the feature point pair for each group of displacement variables to form three-dimensional points corresponding to each group of displacement variables, and according to each group of three-dimensional points The reprojection error and triangulation angle filter the three-dimensional point, wherein the displacement variable includes a rotation variable and a translation variable;

In response to the number of the largest group of three-dimensional points being greater than the preset first number threshold, determining the corresponding displacement variable as the first displacement variable;

Select an essential matrix with the number of inner points greater than or equal to the threshold of the number of points from the essential matrices obtained by multiple calculations, determine at least one set of displacement variables according to each essential matrix, and triangulate the feature points of the feature point pair for each set of displacement variables respectively, To form three-dimensional points corresponding to each group of displacement variables, and filter the three-dimensional points according to the reprojection error and triangulation angle of each group of three-dimensional points, and retain the displacement variables corresponding to a group of three-dimensional points with the largest number of each essential matrix;

In the case that the difference between the displacement variable retained by each essential matrix and the first displacement variable satisfies a preset range, it is determined that the image pair satisfies the initialization condition.
The device for constructing a point cloud model according to claim 28, further comprising:

optimize the camera pose of each panorama image and the position of the initial 3D point by minimizing the reprojection error of the initial 3D point on the two panorama images of the initial image pair; and/or,

After each determination of the camera pose of the registration image, optimize the camera pose of the registration image by minimizing the reprojection error of 3D points on the registration image; and/or,

After each time the feature points of the registered image are triangulated to form corresponding three-dimensional points, and the third feature points of the registered images are triangulated to form corresponding three-dimensional points, by minimizing each three-dimensional point in each The reprojection error on each registered image, optimizes the camera pose for each registered image and the position of each 3D point.
The device for constructing a point cloud model according to claim 21, further comprising:

Determine the corresponding panoramic image to be matched according to the space corresponding to each panoramic image; or,

The panorama image to be matched corresponding to each panorama image is determined according to a preset collocation rule.
An electronic device comprising a memory and a processor, wherein the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement any one of claims 1 to 16 when executing the computer instructions method described in item.
A computer-readable storage medium on which a computer program is stored, which implements the method of any one of claims 1 to 16 when the program is executed by a processor.
A computer program comprising computer readable code which, when executed in an electronic device, a processor in the electronic device executes for implementing the method of any one of claims 1 to 16.