WO2023093085A1 - Method and apparatus for reconstructing surface of object, and computer storage medium and computer program product - Google Patents

Abstract

Disclosed are a method and apparatus for reconstructing a surface of an object, and a computer storage medium and a computer program product. The method comprises: generating a corresponding dense point cloud by using a plurality of images of a scene, the plurality of images being obtained by photographing the scene from a plurality of photographing position points; generating, on the basis of the dense point cloud, a tetrahedral mesh corresponding to the dense point cloud, vertices of each tetrahedron in the tetrahedral mesh being coordinate points in the dense point cloud; determining a binary tag of each tetrahedron on the basis of energy function minimization, the binary tag being used for characterizing that the tetrahedron is located inside or outside the surface of the object; and extracting a common surface of tetrahedrons having different binary tags to reconstruct the surface of the object on the basis of the common surface. The energy function is determined on the basis of a first penalty term corresponding to each tetrahedron and a second penalty term corresponding to each common surface, and a mesh density weight in the second penalty item is used for representing the number of photographing position points corresponding to the common surface. By means of the present invention, the accuracy of reconstructing the surface of the object is improved.

Description

Method, device, computer storage medium and computer program product for reconstructing object surface

Cross References to Related Applications

The embodiment of the present disclosure is based on the Chinese patent application with the application number 202111433346.7, the application date is November 29, 2021, and the application name is "Method, device and computer storage medium for reconstructing the surface of an object", and the priority of the Chinese patent application is required. Right, the entire content of this Chinese patent application is hereby incorporated into this disclosure as a reference.

technical field

The present disclosure relates to the field of image processing, and in particular to a method, device, computer storage medium and computer program product for reconstructing an object surface.

Background technique

3D reconstruction techniques are used to reconstruct the 3D surfaces of the entire scene or objects in the scene from multiple images of a scene. With the development of technology, the application of 3D reconstruction technology is becoming more and more extensive. Such as virtual reality, augmented reality, digital twins, etc. have put forward higher requirements for the accuracy and detail restoration ability of 3D reconstruction.

In the process of reconstructing the surface of an object, it is necessary to generate a dense point cloud of the scene; and in the process of generating a dense point cloud, noise points or abnormal points may appear. These noise points or abnormal points will negatively affect the details of the reconstructed object surface, thereby affecting the accuracy of the reconstructed object surface.

Contents of the invention

Embodiments of the present disclosure provide a method, an apparatus, a computer storage medium and a computer program product for reconstructing an object surface, so as to solve the problem of reducing the influence of noise and retaining more details in the prior art.

In order to solve the above technical problems, a technical solution adopted by the embodiments of the present disclosure is to provide a method for reconstructing the surface of an object, the method includes: using multiple images to generate corresponding dense point clouds, the multiple images are obtained from different shooting positions The point is obtained by shooting the scene; based on the dense point cloud, a tetrahedral grid corresponding to the dense point cloud is generated, wherein the vertex of each tetrahedron in the tetrahedral grid is the coordinate point in the dense point cloud; based on the energy function Minimize, generate a binary label for each tetrahedron, where the binary label is used to characterize whether the tetrahedron is inside or outside the object surface; extract the common faces between tetrahedra with different binary labels to reconstruct the object surface. Wherein, the energy function includes the sum of the first penalty term corresponding to the tetrahedron and the sum of the second penalty term corresponding to the common face, and the second penalty term includes the grid density weight.

In order to solve the above technical problem, another technical solution adopted by the embodiments of the present disclosure is to provide an apparatus for reconstructing an object surface from multiple images of a scene. The device includes: a dense point cloud generation module configured to generate a corresponding dense point cloud using multiple images obtained by shooting the scene from different shooting positions; a grid generation module configured to generate a corresponding dense point cloud based on the dense point cloud , to generate a tetrahedral mesh corresponding to the dense point cloud, wherein the vertices of each tetrahedron in the tetrahedral mesh are coordinate points in the dense point cloud; the tetrahedron labeling module is configured to minimize based on the energy function, as Each tetrahedron generates a binary label, wherein the binary label is used to characterize that the tetrahedron is located inside or outside the surface of the object; the surface extraction module is configured to extract the common face between the tetrahedrons with different binary labels to reconstruct the object surface . Wherein, the energy function includes the sum of the first penalty term corresponding to the tetrahedron and the sum of the second penalty term corresponding to the common face, and the second penalty term includes the grid density weight.

In order to solve the above technical problem, another technical solution adopted by the embodiments of the present disclosure is to provide an apparatus for reconstructing an object surface from multiple images of a scene. The device includes a processor and memory. A computer program is stored in the memory. The processor is used to execute the computer program to realize the steps of the above method.

In order to solve the above technical problems, another technical solution adopted by the embodiments of the present disclosure is to provide a computer storage medium. The computer storage medium stores a computer program. When the computer program is executed by the processor, the steps of the above method are realized.

In order to solve the above technical problems, another technical solution adopted by the embodiments of the present disclosure is to provide a computer program product, including computer readable codes, where the computer program product includes computer programs or instructions, where the computer program or instructions are in In the case of running on an electronic device, the electronic device is made to execute the above business state analysis method.

The embodiment of the present disclosure constructs the energy function to include the sum of the first penalty term corresponding to the tetrahedron and the sum of the second penalty term corresponding to the common face, wherein the second penalty term includes the mesh density weight, due to the noise Points or abnormal points correspond to fewer shooting positions, and through the grid density weight, a greater penalty can be imposed on the sparse grid; that is, through the grid density weight, a greater penalty can be imposed on the noise point or abnormal point Penalty, so as to reduce the negative impact of noise points or abnormal points on details, retain more details, and improve the accuracy of object reconstruction surface. .

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present disclosure. Those of ordinary skill in the art can also obtain other drawings based on these drawings without any creative effort.

FIG. 1 is a schematic flowchart of an optional method for reconstructing an object surface provided by an embodiment of the present disclosure;

Fig. 2 is a schematic flow chart of an embodiment of the method for generating a dense point cloud in the present disclosure;

Fig. 3 is a schematic flow chart of another embodiment of the method for generating a dense point cloud in the present disclosure;

Fig. 4 is a schematic diagram of another embodiment of the generated dense point cloud provided by the embodiment of the present disclosure;

Figure 5 is a schematic diagram of one embodiment of a tetrahedral mesh corresponding to the dense point cloud in Figure 4;

FIG. 6A shows a schematic diagram of a line of sight passing through a common plane;

FIG. 6B shows a schematic diagram of an optional public surface weight;

FIG. 6C shows a schematic diagram of an optional public surface weight;

Figure 7 shows a schematic view of the included angle between adjacent tetrahedrons;

FIG. 8 shows a schematic diagram of an embodiment of a method for solving an energy function using a graph cut method;

Figure 9 shows a schematic diagram of a directed graph corresponding to the tetrahedral grid in Figure 4;

Fig. 10 is a schematic structural diagram of an embodiment of the device for reconstructing the surface of an object according to the present disclosure;

Fig. 11 is a schematic structural diagram of another embodiment of the device for reconstructing the surface of an object according to the present disclosure;

Fig. 12 is a schematic structural diagram of an embodiment of a computer storage medium of the present disclosure.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are only a part of the embodiments of the present disclosure, not all of them. Based on the embodiments in the present disclosure, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present disclosure.

Currently, there are various limitations in the application of reconstruction techniques for object surfaces. For example, the Poisson method is suitable for reconstructing watertight point clouds. The truncated signed distance function (TSDF) algorithm based on truncated signed distance function (TSDF) is not accurate enough in the object surface reconstruction of large scenes. Object surface reconstruction methods are very sensitive to noise. The loss of details and insufficient precision generally exist in related technologies.

An embodiment of the present disclosure provides a method for reconstructing an object surface, and the method may be executed by a processor of an apparatus for reconstructing an object surface. Wherein, the device for reconstructing the surface of an object refers to a server, a notebook computer, a tablet computer, a desktop computer, a smart TV, a set-top box, a mobile device (such as a mobile phone, a portable video player, a personal digital assistant, a dedicated message device, a portable game device, etc.) ) and other equipment with data processing capabilities. Referring to FIG. 1 , FIG. 1 is a schematic flowchart of a first embodiment of a method for reconstructing an object surface according to the present disclosure. The method for reconstructing an object surface according to an embodiment of the present disclosure includes the following steps.

S11. Generate a corresponding dense point cloud P by using multiple images I _i of the scene (wherein, i=1, ... N, where N is the number of multiple images). Among them, the multiple images of the scene are obtained by shooting the scene from different shooting positions; the dense point cloud P is also called dense point cloud P.

In the embodiment of the present disclosure, the above-mentioned scenes may be large-area scenes with various terrain changes, buildings and objects, such as university campuses, scenic spots, mountains, commercial areas, and residential areas; the scenes may also be indoor small-area scenes . In some embodiments, the scene may also be a single object such as a statue, an industrial product (for example, a single airplane). Embodiments of the present disclosure do not limit this. In some embodiments, the aforementioned scene may be a static or substantially static scene.

In the embodiment of the present disclosure, different shooting locations may include shooting locations such as land, air, and water. In this way, the acquisition device can shoot the same scene from different shooting positions to obtain multiple images I _i ; the device for reconstructing the object surface can obtain multiple images I _i from the acquisition device; wherein, the acquisition device can be a device for reconstructing the object surface The built-in device may also be a device other than the device for reconstructing the surface of the object, which is not limited by the embodiments of the present disclosure.

In some embodiments of the present disclosure, the collection device may include at least one of an unmanned aerial vehicle and a ground-based camera; the collection device may include at least one of a collection platform or a collection device; for the collection device, it may be set as required, Embodiments of the present disclosure are not limited.

In the embodiment of the present disclosure, the multiple images I _i captured by the acquisition device may be depth images or RGB (Red-Green-Blue) images. The multiple images I _i each include at least a part of the same scene. At least two of the plurality of images I _i contain scene overlapping parts. For example, in multiple images I _i of a certain residential area, both images _Im and _In include the same signal tower, then the area of the signal tower may be the scene overlapping part of images _Im and _In . The more scene overlapping parts among the multiple images I _i , the easier the surface reconstruction process in the embodiments of the present disclosure will be.

In the embodiment of the present disclosure, the acquisition device can store the camera pose corresponding to each image while taking multiple images I _i , so that the device for reconstructing the object surface can obtain the camera pose corresponding to each image from the acquisition device; The apparatus for reconstructing the surface of an object may also calculate the camera pose corresponding to each image based on the multiple images I _i after acquiring the multiple images I _i ; the embodiment of the present disclosure does not limit the acquisition method of the camera pose.

In some embodiments, the camera pose may include a camera shooting position c _i and a camera shooting angle.

In the embodiment of the present disclosure, the camera shooting position c _i may also be referred to as shooting position point c _i or camera position point information c _i . The shooting position point c _i can be the relative information of the camera relative to the scene or feature in the image when shooting the corresponding image I _i , or the space of the camera in the three-dimensional space (for example, the three-dimensional space corresponding to the physical reality space of the scene). coordinate.

In some embodiments, the above camera pose can also be calculated from multiple images I _i . The embodiment of the present disclosure does not limit the manner of acquiring the pose of the camera.

In the embodiment of the present disclosure, after acquiring multiple images I _i , the apparatus for reconstructing the object surface may generate corresponding dense point clouds P from the multiple images I _i . The dense point cloud P includes the first type of points p, and the first type of points can also be called coordinate points p; each coordinate point p is a point from at least one image among the above-mentioned plurality of images I _i . The point can be a sampling point or a point obtained by methods such as interpolation. In some embodiments, the coordinate point p includes spatial coordinate information of the point.

In some embodiments, the dense point cloud P also includes the second type of point c _i or the shooting location point c _i , and the first type of point can also be called the shooting location point c _i ; that is to say, the dense point cloud P also includes the same Each of the plurality of images I _i corresponds to a shooting location point c _i .

In some embodiments, the device for reconstructing the object surface can perform feature matching on multiple images I _i to obtain the common view points of the shooting cameras of at least two images in the multiple images, and then obtain the dense point cloud P based on these common view points. The coordinate point p. Wherein, the common view point of at least two images is a point that can be seen by the shooting cameras of at least two images. Correspondingly, the coordinate point p obtained from the common view point is the coordinate point p derived from the two images.

In the embodiment of the present disclosure, the source image of the coordinate point in the dense point cloud P is queried, and the shooting location point corresponding to the source image is determined according to the source image. The shooting location point is added to the dense point cloud to obtain an updated dense point cloud. The updated dense point cloud P will include the coordinate point p and the shooting location point from the image. In some embodiments, the shooting location point may be the optical center point of the shooting camera.

In some embodiments of the present disclosure, in S11, multiple images I _i of the scene are used to generate a corresponding dense point cloud P. Referring to FIG. 2, FIG. 2 shows an embodiment of the method for generating a dense point cloud in the present disclosure. Flow chart, the method includes: S21-S23.

S21. Shoot the scene from different shooting positions c _i to obtain multiple images I _i . The plurality of images I _i are color maps.

In the embodiment of the present disclosure, the color map may be an RGB image or other types of color map, which is not limited in the embodiment of the present disclosure.

S22. Estimate a dense depth map for each of the plurality of images I _i .

In an embodiment of the present disclosure, after obtaining multiple images, the device for reconstructing the object surface may perform depth estimation for each of the multiple images to obtain a dense depth map corresponding to each image; wherein, the depth estimation Methods can include:

In some embodiments, the implementation of estimating a dense depth map for each of the multiple images I _i in S22 may include: using an incremental SFM (structure-from-motion) method for the input multiple images I _i Rebuild a sparse map.

For example, extract the SIFT (Scale- _Invariant Feature Transform) feature of each image, match the SIFT features in multiple images I _i , and reconstruct the sparse map. The sparse map is a set of key points including at least a part or all of the SIFT features.

In some embodiments, SIFT is used to describe scale invariance in the field of image processing, and key points can be detected in an image through SIFT. The SIFT feature is obtained by performing SIFT transformation on some key points of local appearance on the object. Since the SIFT feature has nothing to do with the size and rotation of the image, and has a high tolerance for light, noise, and micro-angle changes, the SIFT feature is highly significant, easy to identify and relatively easy to extract, and is not easy to be misidentified. In some embodiments, the device for reconstructing the object surface only needs more than three SIFT features to calculate the position and orientation of key points.

In some embodiments, the implementation of estimating a dense depth map for each of the multiple images I _i in S22 may further include: based on the matching relationship between SIFT features of at least some of the multiple images I _i , Divide images I _i into clusters and estimate a dense depth map for each image I _i in each cluster.

In the embodiment of the present disclosure, the apparatus for reconstructing the object surface may classify images including as many identical SIFT features as possible into one cluster, thereby obtaining multiple clusters. The number of multiple clusters can be set according to specific needs, which is not limited in this embodiment of the present disclosure.

In the embodiment of the present disclosure, since the sparse map is a set of key points including at least a part or all of the SIFT features, and more than three SIFT features are enough to calculate the position and orientation of the key points. By performing SIFT feature matching between images in each cluster, a dense depth map can be estimated for each image in each cluster. Among them, feature matching is used to establish the correspondence between features, and feature matching can correspond to the images of the same feature in different views in space. The generated dense depth map includes sample points from the corresponding image. In some embodiments, the dense depth map includes depth information of sampling points. In some embodiments, the dense depth map includes three-dimensional coordinate information of sampling points.

It should be noted that the density (also known as dense) in a dense depth map is compared with the sparseness in a sparse map. In the embodiment of the present disclosure, the sparse map may contain key points corresponding to SIFT features. Here, the key points corresponding to SIFT features are relatively few and sparse, while the points in the dense depth map are denser, so more details can be preserved. information.

S23, merging the dense depth map corresponding to each image into a dense point cloud P.

In the embodiment of the present disclosure, after obtaining the dense depth map corresponding to each image, the device for reconstructing the object surface may perform fusion processing on multiple dense depth maps corresponding to multiple images to obtain a dense point cloud P. In some embodiments, the device for reconstructing the surface of an object may fuse the dense depth maps into a dense point cloud P according to the geometric shape and depth continuity relationship of images corresponding to different dense depth maps.

For example, for a point A in the dense depth map corresponding to an image I _l and a point B in the dense depth map corresponding to an image I _m , both point A and point B correspond to the same spatial coordinate position in the scene , or when the spatial distance between point A and point B in the scene is less than the spatial distance threshold, point A and point B can be fused into the same point _ph in the dense point cloud P, and point _ph comes from image I _l at the same time and image _Im . In other words, the point _ph can be seen simultaneously from the shooting location point c _l of the image I _l and the shooting location point _{c m} of the image Im (the subscripts l and m indicate the one-to-one correspondence between the image and the shooting location point), or point _Ph can see the shooting position point c _l of the image I _l and the shooting position point c _m of the image _Im . The point _ph corresponds to the shooting location point c _l of the image I _l and the shooting location point c _m of the image I _m ; when the spatial distance between point A and point B in the scene is less than the spatial distance threshold, the point _ph can be A point obtained by interpolation from point A and point B; wherein, the spatial distance threshold can be set as required; this is not limited in this embodiment of the present disclosure.

In some embodiments of the present disclosure, some coordinate points p in the dense point cloud P can be seen from more than two shooting location points corresponding to more than two images. In some embodiments of the present disclosure, some coordinate points p in the dense point cloud P can only be seen from one shooting location point corresponding to one image.

In some embodiments, the dense point cloud P is formed through at least two fusion steps.

In the embodiment of the present disclosure, the device for reconstructing the surface of an object may first fuse multiple dense depth maps of each cluster in multiple clusters into a cluster dense point cloud, and then combine multiple clusters corresponding to multiple clusters The dense point cloud is fused into a dense point cloud P for the entire scene.

It is understandable that by merging multiple dense depth maps in each cluster into a corresponding clustered dense point cloud, and then merging all clustered dense point clouds into a dense point cloud P of the entire scene; so , the device for reconstructing the surface of an object can only load dense depth maps in one or more clusters in memory for fusion at a time, instead of loading all dense depth maps at the same time, thereby reducing memory consumption during the generation of dense point clouds P, Increased applicability to large-scale high-precision scene reconstruction, thereby improving the robustness of reconstructed object surfaces.

In the dense point cloud P of the entire scene, the sampling points in multiple images _Ii can be expressed in the form of coordinate points p of the 3D point cloud transmitted into the 3D space, thus showing the 3D geometric relationship between the sampling points . In other words, the dense point cloud P is a depth point cloud or a 3D point cloud corresponding to multiple images I _i .

In some embodiments of the present disclosure, the dense point cloud P also includes the shooting location point c _i of each image I _i ; thus, the dense point cloud P can reflect the spatial position relationship between the shooting location point c _i and the sampling points .

Referring to FIG. 3 , FIG. 3 shows a schematic flow chart of another embodiment of a method for generating a dense point cloud; in FIG. 3 , the device for reconstructing the surface of an object can be based on multiple images obtained by shooting the scene from different shooting positions c _i Images _Ii generate corresponding dense point clouds P. As shown in Fig. 3, the method may include: S31-S32.

S31. Shoot the scene from different shooting position points c _i to obtain multiple depth maps.

In the embodiments of the present disclosure, the depth map can be obtained by image acquisition devices such as binocular cameras, multiple monocular or binocular cameras; the depth map can also be obtained by image acquisition devices such as ultrasound and laser, and the embodiments of the present disclosure do not make any reference to this. limit.

Since the depth map is directly used, this embodiment can omit the step of generating a dense depth map from the color map in FIG. 2 , and directly use the depth map as the dense depth map. In some embodiments, the apparatus for reconstructing the surface of an object may also perform preprocessing on the depth map, for example, changing the angle of the depth map, adjusting its size, and so on.

S32, fusing multiple depth maps to obtain a 3D dense point cloud P corresponding to the scene. This step is similar to the above step S23, and will not be repeated here.

In addition to the above method, the apparatus for reconstructing the surface of an object may also combine the depth map and the color map to generate a dense point cloud P, which is not limited in the embodiments of the present disclosure.

Referring to FIG. 4 , FIG. 4 shows a schematic flowchart of still another embodiment of the method for generating a dense point cloud. As shown in FIG. 4 , the dense point cloud P includes coordinate points p ₁ , p ₂ , p ₃ and p ₄ corresponding to sampling points in multiple images I _i . The dense point cloud P also includes shooting position points c ₁ and c ₂ . The dense point cloud P takes the form of a 3D point cloud showing the spatial relationship between the coordinate point p and the shooting location point c. Fig. 4 only schematically shows the dense point cloud p, and the embodiment of the present disclosure does not limit the number and/or geometric positional relationship of points in the dense point cloud p.

S12. Based on the dense point cloud P, generate a tetrahedral mesh corresponding to the dense point cloud P. Wherein, the vertex of each tetrahedron τ in the tetrahedral grid is the coordinate point p and/or the shooting location point c _i in the dense point cloud P.

In some embodiments, the apparatus for reconstructing the surface of an object may generate a tetrahedral mesh based on the shooting location point c _i and the coordinate point p. In some embodiments, the apparatus for reconstructing the object surface may generate a tetrahedral mesh based on each point in the updated dense point cloud obtained in step S11. That is to say, in the embodiment of the present disclosure, the apparatus for reconstructing the object surface may use the coordinate point p and the shooting position point from the image to construct the tetrahedral mesh. In some embodiments, for the coordinate point p derived from the image, the device for reconstructing the object surface can perform back projection based on the pixels in the collected image to obtain a three-dimensional point in space, and use the three-dimensional point as a dense point cloud P The coordinate point p. In some embodiments, the shooting position point may be the position of the optical center point of the camera. Each shooting location point _ci and coordinate point p is at least a vertex of a tetrahedron. In some embodiments, the tetrahedra are Delaunay tetrahedra. In some embodiments, the device for reconstructing the object surface may perform Delaunay triangulation on the dense point cloud P to construct a three-dimensional Delaunay triangular mesh. In the Delaunay triangular mesh, all faces are triangular faces, and the three vertices of all triangular faces are the coordinate point p and/or the shooting location point c _i in the dense point cloud P, and the circumscribed boundary of all triangular faces The coordinate point p in the dense point cloud P and the shooting location point _ci are not included in the circle. Delaunay triangulation can maximize the minimum interior angle in the formed triangle, thereby improving the uniformity and regularity of the triangular mesh.

It should be noted that for each dense point cloud P, its three-dimensional Delaunay triangular mesh is unique. In some embodiments, the method of implementing the Delaunay triangulation may be the Lawson method of pointwise interpolation.

In the embodiment of the present disclosure, the apparatus for reconstructing the surface of an object can obtain a set of tetrahedrons T={τ} and a set of triangular faces of tetrahedrons F={f}. A common triangular facet may be included between two adjacent tetrahedrons τ, also referred to as the common face f hereinafter.

Referring to FIG. 5 , FIG. 5 shows a schematic diagram of a tetrahedral mesh corresponding to the dense point cloud P in FIG. 4 . As shown in FIG. 5 , the tetrahedral grid includes tetrahedra τ ₁ , τ ₂ and τ ₃ . Among them, tetrahedron τ ₁ has coordinate points p ₁ , p ₂ , p ₃ and shooting position point c ₁ as vertices. Tetrahedron τ ₂ has coordinate points p ₂ and p ₃ and shooting position points c ₁ and c ₂ as vertices. Tetrahedron τ ₃ has coordinate points p ₂ , p ₃ , p ₄ and shooting position point c ₂ as vertices. Tetrahedron τ ₁ and tetrahedron τ ₂ have a common face f ₁ . Tetrahedron τ ₂ and tetrahedron τ ₃ have a common face f ₂ . FIG. 5 exemplarily shows an optional tetrahedral grid structure, and the embodiment of the present disclosure does not limit the tetrahedral quantity and positional relationship of the tetrahedral grid.

S13, based on the minimization of the energy function, determine the binary label λ of each tetrahedron τ, where the binary label λ is used to indicate that the tetrahedron τ is located inside or outside the surface of the object or scene.

In the embodiment of the present disclosure, the device for reconstructing the surface of an object can generate a binary label λ∈{inner, outer} for each tetrahedron τ in the above-mentioned tetrahedron set T, and the binary label λ of all tetrahedrons τ constitutes the The binary label set L={λ _w , w=1, . . . M} of the tetrahedral grid, where M is the number of tetrahedrons τ. The label λ of the tetrahedron τ is "inner" to indicate that the tetrahedron τ is located inside the surface of the scene or inside the surface of an object in the scene. The label λ of the tetrahedron τ is "outside", which means that the tetrahedron τ is located outside the surface of the scene or outside the surface of objects in the scene. For two adjacent tetrahedrons τ _x and τ _y , if the label λ _x of one tetrahedron τ _x is "inside" and the label λ _y of the other tetrahedron τ _y is "outside", then the tetrahedron τ _x and The common face f of τ _y will be the surface of the scene or the surface of an object in the scene.

In the disclosed embodiment, the binary label λ may have any suitable label, such as a binary value, a text label, a label number, an integer value, a real value, etc., and this disclosed embodiment is not limited. In one example, the binary label λ may be a set of either 0 or 1. In one example, a value of 0 may indicate that label λ is "in", a value of 1 may indicate that label λ is "out", and vice versa.

In the embodiment of the present disclosure, the device for reconstructing the surface of an object can generate a binary label set L for each tetrahedron τ by constructing an energy function (also called a cost function) and solving a binary label set L that minimizes the energy function. label lambda.

energy function

In one embodiment, energy function E _energy can refer to formula (1):

Among them, λ(τ)≠λ(τ'), T is the set of tetrahedron τ of the above tetrahedral grid, F is the set of triangular facet f, λ(τ) is the binary label λ of tetrahedron τ, And the triangular facet f is the common face f of tetrahedron τ and tetrahedron τ'.

It can be seen from the above formula that the energy function E _energy includes _{the sum} of the first penalty term E1 corresponding to the tetrahedron τ and the sum of the second penalty term _E2 corresponding to the common surface f.

Among them, the _first penalty item E1 is set based on whether the vertex of the tetrahedron τ is the shooting location point c _i or a coordinate point p other than the shooting location point c _i , and the second penalty item _E2 is based on the line of sight and the public The intersecting relationship of the surface f is set; where, the line of sight refers to the connection line from the coordinate point p to the shooting position point c _i in the above-mentioned 3D dense point cloud P.

Calculation of Coefficients in Energy Function

Refer to the line of sight shown in Figure 6A. In FIG. 6A , black dots represent vertices of tetrahedron τ in the tetrahedral mesh, and thick solid lines represent triangular facets f of tetrahedron τ. Where f includes f ₁ , f ₂ and f ₃ . A vertex of the tetrahedron corresponds to a coordinate point p in the above-mentioned 3D dense point cloud P. When it can see the shooting position point c, the connection line from the coordinate point p to the shooting position point c is called the line of sight p→ c. Here, the coordinate point p is called the starting point of the line of sight p→c, and the shooting position point c is called the end point of the line of sight p→c. The distance from the starting point of the sight line p→c to the end point of the sight line p→c can be called the length of the sight line p→c. A line of sight p→c may intersect multiple common faces f. As shown in Fig. 6, the line of sight p→c intersects common faces _f1 , _f2 and _f3 in this tetrahedral mesh.

It should be noted that the device for reconstructing the surface of an object may construct at least one line of sight for each vertex in the tetrahedron grid, as long as the vertex is not the shooting location point c. In the case that a coordinate point p corresponding to a vertex can be seen from multiple shooting position points c, multiple sight lines corresponding to multiple shooting position points c may be constructed for the coordinate point p.

In the embodiment of the present disclosure, each coordinate point p in the 3D dense point cloud P is a sampling point on the surface of the scene or object, and the line of sight of each vertex will not pass through the surface of the reconstructed object, thus, the reconstructed object surface The device can construct the first penalty term _Efirst in the energy function E _energy according to the relationship between the line of sight and the tetrahedron τ, and can construct _the second penalty term E2 according to the relationship between the line of sight and the public surface f.

In some embodiments, the device for reconstructing the object surface may initialize the first penalty term _E1 and the _second penalty term E2 to zero, and then calculate the corresponding first penalty term for each line of sight in turn E _first and second penalty term E _second .

Calculation of the first penalty term E _first

As shown in FIG. 6A , for a line of sight from the coordinate point p to the shooting position point c, τ _p is the tetrahedron τ where the vertex p is located, and τ _c is the tetrahedron τ where the shooting position point c is located.

In some embodiments, the coordinate point p may correspond to multiple tetrahedrons τ. Exemplarily, τ _p is a tetrahedron τ through which the reverse extension line of the line of sight passes.

In some embodiments, the shooting location point c may correspond to multiple tetrahedrons τ. Exemplarily, τ _c is a tetrahedron τ through which the line of sight passes.

In the embodiment of the present disclosure, τ _p is located behind the line of sight of vertex p, and is more likely to belong to the inside of the scene. As shown in formula (2), the device for reconstructing the object surface can add a binary label λ to τ _p as "outside ”The preset penalty coefficient α _v .

E _first (τ _p , outer) + = α _v formula (2)

Wherein, α _v is a positive number, which can be set by the user according to needs. For example, _αv can be set to 1. By setting α _v , the possibility of τ _p being marked out can be reduced.

In the embodiment of the present disclosure, the tetrahedron τ _c where the shooting position point c is located is more likely to belong to the outside of the scene. As shown in formula (3), the device for reconstructing the surface of an object can add a preset penalty coefficient α _v when the label is “inside” to τ _c .

E _first (τ _c , inner) + = α _v formula (3)

By setting α _v , the possibility of marking τ _c as inside can be reduced. The penalty coefficient α _v may also be replaced by other preset values, which is not limited in this embodiment of the present disclosure.

Calculation of the _second penalty term ESecond

Referring to FIG. 6A , the line of sight from the vertex p to the shooting location point c may pass through one or more common planes f. It can be seen from the above that the line of sight of the vertex p does not pass through the reconstructed surface, so the second penalty value _Esecond (f) is added to the one or more passed common faces f.

In some embodiments, as shown in formula (4), for the common plane f _i , the second penalty value _Esecond (f _i ) is grid density weight ω _d (f _i ), distance weight ω _v (f _i ) and the product of the grid quality weight ω _q (f _i ) and the preset penalty coefficient α _v .

E _second (f _i )+=ω _d (f _i )ω _v (f _i )ω _q (f _i )α _v formula (4)

In some embodiments, the second penalty value _Esecond (f _i ) may not include the distance weight ω _v (f _i ) and/or the mesh quality weight ω _q (f _i ). In some embodiments, the preset penalty coefficient α _v is a positive number and can be set as required. For example, _αv can be set to 1.

Grid density weight ω _d (f _i )

In the embodiment of the present disclosure, the sum of the number of shooting location points c _i that can be seen by each of the three vertices of the common surface f _i is the number of shooting location points corresponding to the public surface f _i ; The larger the number of shooting position points corresponding to the surface, the smaller the grid density weight. For example, the three vertices of the public surface f can respectively see 2, 1 and 3 shooting location points _ci , then the number of shooting location points _ci that can be seen by the three vertices of the public surface f The sum is 2+1+3=6. The larger the sum of the numbers is, the smaller the grid density weight ω _d (f _i ) is. The larger the sum of the numbers, the more vertices of the common face f _i appear in more images, and the greater the possibility that the common face f _i belongs to the scene or object surface.

In some embodiments, the larger the sum of the side lengths of the common plane f _i is, the larger the grid density weight ω _d (f _i ) is. The larger the sum of the side lengths of the common face f _i is, the sparser the grid near the common face f _i is, and the less likely the common face f _i belongs to the scene or object surface. The grid density weight ω _d (f _i ) indicates that the common face of the denser part of the grid is closer to the real object surface.

In some embodiments, the grid density weight can be calculated by formula (5).

Among them, V(f _i ) represents the value obtained by dividing the total side length of the public face f _i by the number of shooting positions corresponding to the public face f _i ; the public face f _i includes three vertices, and the shooting position points corresponding to the public face f _i The number of location points includes the sum of the numbers of shooting location points _ci corresponding to each vertex. η _d is a grid density control factor, which is used to control the influence of the grid density weight on the second penalty value. The size of _ηd can be set as required, which is not limited by the embodiments of the present disclosure. For example, _ηd can be set to 0.8. In the grid density weight formula, σ _d is the scale control quantity, which can be used to make ω _d (f) dimensionless. For example, σ _d may be a quarter of the smallest value among V(f) of all common planes f _i , which is not limited in this embodiment of the present disclosure.

It is understandable that due to noise points or abnormal points seeing less shooting location point c, through the grid density weight ω _d (f _i ), a larger penalty can be imposed on the sparse grid, that is, by The grid density weight ω _d (f _i ) can impose greater penalties on noise points or abnormal points, thereby reducing the negative impact of noise points or abnormal points on details, retaining more details, and improving the accuracy of reconstructing the object surface.

Distance weight ω _v (f _i )

In the embodiment of the present disclosure, the second penalty item E2 _further includes a distance weight ω _v (f _i ). The farther the intersection point of the line of sight and the public plane f _i is from the starting point of the line of sight, the greater the distance weight ω _v (f _i ).

As shown in FIG. 6 , there is an intersection point between any public plane f _i and the line of sight. The farther the distance between the intersection point and the starting point p of the line of sight is, the greater the distance weight ω _v (f _i ) of the common plane corresponding to the intersection point is. Grid density weight ω _v (f _i ) can be calculated by formula (6).

Wherein, D(f _i ) represents the distance from the intersection of the common plane f _i and the line of sight to the starting point p of the line of sight. _σv is a grid complexity constant, and _σv can be set according to actual needs, which is not limited in the embodiments of the present disclosure.

In some embodiments, considering that noise points or outliers may cause wrong line of sight to pass through fine structures, embodiments of the present disclosure introduce a truncated distance coefficient in the distance weight ω _v (f _i ). The device for reconstructing the surface of an object is at distance D(f _i ) and line-of-sight length

When the ratio between them is greater than the first threshold (that is, the truncated distance coefficient), it is determined that the distance weight is zero.

In some embodiments, the device for reconstructing the surface of the object can be between the distance D(f _i ) and the line-of-sight length

When the ratio between is greater than the first threshold, and the ratio V(f) between the perimeter of the public plane _fi and the number of shooting position points (image acquisition points) corresponding to the public plane _fi is greater than the second threshold, Make sure the distance weight is zero. That is to say, considering the truncation distance coefficient while also considering the grid density can reduce the probability of imposing unnecessary penalties on high-density grid areas and improve the accuracy of imposing penalties.

In some embodiments, the cutoff distance factor may be 1-S(P). Among them, S(P) is used to represent the uncertainty of the line of sight origin, and S(P) can be calculated according to related methods.

In the embodiment of the present disclosure, the truncation distance coefficient may be a constant set as required, which is not limited in the embodiment of the present disclosure.

In some embodiments, the second threshold may be the scale control amount σ _d , or may be other values set according to needs, which is not limited in this embodiment of the present disclosure.

Exemplarily, based on FIG. 6A , FIG. 6B shows a schematic diagram of visualization of weights in the related art, and FIG. 6C shows an optional schematic diagram of visualization of weights provided by an embodiment of the present disclosure. It can be seen that the grid density at f ₃ is high, and it is far away from the coordinate point P, which is likely to be a small structure in the space. In the related art, due to the accumulation of large weights to it, the details will be lost; however, in the embodiment of the present disclosure, the device for reconstructing the surface of the object resets the weight here to 0, and the weight farther away from the coordinate point P Set it to 0, so as to reduce the probability of applying unnecessary penalties to high-density grid areas and improve the accuracy of applying penalties.

It can be understood that by introducing the truncated distance coefficient, the influence of the false line of sight caused by noise is reduced, the robustness of the surface reconstruction process is increased, and the detailed reconstruction ability of the model is effectively improved.

Grid quality weight ω _q (f _i )

In the disclosed embodiments, the grid quality weight ω _q (f _i ) is used to consider the influence of the local grid shape. Generally speaking, the better the grid shape and the higher the grid quality, the more reliable the results obtained by using the grid, and the smaller the grid quality weight ω _q (f _i ).

Referring to FIG. 7, the common plane f is the common plane between the tetrahedron τ ₁ and the tetrahedron τ ₂ . ω _q (f) can be calculated according to formula (7).

Among them, θ is the angle between the circumscribed sphere of the tetrahedron _τ1 and the common surface f,

is the angle between the circumsphere of the tetrahedron τ ₂ and the public surface f. The included angle between the circumscribing sphere and the public surface f can be defined as the line-surface angle between the line between the center of the circumscribing sphere and any vertex of the public surface and the public surface f.

The mesh quality weight _ωq characterizes the effect of the relative angle of the two tetrahedrons _τ1 and _τ2 . The smaller relative angle between the two tetrahedra τ proves the better shape of the local mesh.

In some embodiments, for the public surface f _i with the second penalty term _Esecond (f _i ), before starting to calculate the coefficient of the energy function E _energy , the second penalty term _Esecond (f _i ) can be initialized to 0 . Accumulate a _{corresponding} value ω _d (f _{i )} ω _v (f _i ) _{ω q} (f _i )α _v .

In some embodiments, for each vertex p in the tetrahedral mesh, there is a _first penalty term E(τ,inner) labeled "inner" and another first penalty term labeled "outer". _Efirst (τ,outer). The first penalty term _Efirst (τ,inner) when the label is "inside" means that the tetrahedron τ is marked as penalized inside the scene. The first penalty term _Efirst (τ,outside) when the label is “outside” indicates the penalty for tetrahedron τ being marked as outside the scene.

In some embodiments, the first penalty term _Efirst (τ) may be initialized to 0 before starting to calculate the coefficients of the energy function _Eenergy . For each line of sight, in the case that the tetrahedron τ is the tetrahedron τ _c where the shooting position point c of the line of sight is located, a corresponding preset penalty coefficient can be accumulated for the first penalty item _Efirst (τ, inner) α _v . For each line of sight, when the tetrahedron τ is the tetrahedron τ _p where the starting point p of the line of sight is located, a corresponding preset penalty coefficient α _v can be accumulated for the first penalty item _Efirst (τ, outer).

In the embodiment of the present disclosure, after obtaining the formula (1), the apparatus for reconstructing the object surface can minimize the energy of the energy function E, so as to obtain a binary label set L. It can be seen from the formula (1) that the energy function E _energy (T, F, L) includes the sum of the first penalty term E _first (τ, λ(τ)) corresponding to each tetrahedron τ and the sum with The sum of the second penalty term _E2 (f, λ(τ), λ(τ′)) corresponding to the common face f of adjacent tetrahedra τ with different labels λ. Wherein, for any common surface f _i , the value of _Esecond (f _i ,λ(τ),λ(τ′)) is _Esecond (f _i ) mentioned above.

Obviously, for different binary label sets, the label λ of the tetrahedron τ in the tetrahedral grid may be different, and the _second penalty term E2 (f,λ(τ),λ(τ′)) corresponds to the public The surface f may be different, the sum of the _second penalty item E (f, λ(τ), λ(τ′)) may be different, and the value of the energy function E _energy (T, F, L) may be different. In this way, the device for reconstructing the object surface can solve a binary label set L to minimize the energy function E _energy (T, F, L).

In some embodiments, the problem of minimizing the energy function E _energy (T, F, L) can be solved using the st graph cut method. Referring to FIG. 8 , the method for minimizing the energy function E _energy (T, F, L) using the st graph cut method includes the following steps.

S41, mapping the tetrahedron grid into a directed graph G.

In the embodiment of the present disclosure, each tetrahedron τ in the tetrahedral grid can be mapped to a point ν in the directed graph G, and the common face f is used as the connection between the points ν of the directed graph G Line ζ.

As shown in Fig. 9, the graph points ν ₁ , ν ₂ and ν ₃ in the directed graph G correspond to the tetrahedrons τ ₁ , τ ₂ and τ ₃ in Fig. 5, respectively. The connections ζ ₁ and ζ ₂ in the directed graph G correspond to the common planes f ₁ and f ₂ in Fig. 5, respectively.

S42. Add a virtual start point s and a virtual end point t to the directed graph G, and map the first penalty item _E1 and the second penalty item _E2 to the flow on the link ζ in the directed graph G.

In the embodiment of the present disclosure, the first penalty item _Efirst (τ, outer) of the tetrahedron is the flow capacity of the line between the virtual starting point and the corresponding graph point of the tetrahedron. The _first (τ, inner) of the tetrahedron is the flow capacity of the line between the corresponding graph point and the virtual end point of the tetrahedron. Exemplarily, as shown in FIG. 9 , the virtual source point s is connected to all graph points ν _i via a connecting line. The virtual sink t is connected with all graph points ν _i by connecting lines. The first penalty term _E1 (τ _i , outer) of the tetrahedron τ _i is the flow capacity of the connection line flowing from the virtual starting point s to the graph point ν _i . The first penalty term _Efirst (τ _i ,inner) of the tetrahedron τ _i is the flow capacity of the line flowing from the graph point ν _i to the virtual endpoint t.

In the present disclosure embodiment, the second penalty term E2( _{f i} ) of the common face f _i of two different tetrahedron τ and tetrahedron τ 4 acts as a connection between tetrahedron τ and tetrahedron τ 4 The flow capacity of the line.

Exemplarily, as shown in Figure 9, the second penalty term E2( _{f i} ) of the common surface f ₁ between the tetrahedron τ ₁ and τ ₂ is the flow capacity of the line between ν ₁ and ν ₂ .

It should be noted that, in the directed graph G, the flow between tetrahedron τ and tetrahedron τ4 is bidirectional, and can be from τ to τ, or from τ to τ. In the embodiment of the present disclosure, the flow capacities from τ to τ, and from τ to τ are the same, which are equal to the second penalty term E2( _{f i} ) of the common plane f _i .

In the embodiments of the present disclosure, the traffic capacity of a connection refers to the maximum value of traffic allowed by the connection, which is also referred to as the weight of the connection.

In the embodiment of the present disclosure, the directed graph G after adding the virtual start point s and the virtual end point t can be regarded as a network flow graph. In this network flow, only the virtual origin s will generate traffic, and the virtual endpoint t will receive traffic. The flow flows from the virtual starting point s to the virtual end point t through the graph point ν. For the graph point ν in the directed graph G except the virtual start point s and the virtual end point t, its net flow must be 0. That is to say, the flow of the virtual starting point s will eventually reach the virtual endpoint t through the connection in the directed graph G.

S43, with the goal of maximizing the total flow from the virtual starting point s to the virtual end point t, calculate the binary label L, wherein, in the calculation process of the total flow, only two map points ν corresponding to the inside and outside of the object respectively The flows between them are summed.

In the embodiment of the present disclosure, for each binary label set, all graph points ν in the directed graph G are divided into two categories: the first category, the graph points whose labels are outside, including the virtual starting point s; the second category, Labeled plot points within, including the virtual endpoint t. The process of dividing all graph points ν into two classes including virtual start point s and virtual end point t is called s-t graph cut.

Since only the virtual starting point s will generate traffic, and the virtual endpoint t will receive traffic, the net flow of the graph point ν must be zero, so the net flow of the network flow from the virtual starting point s to the virtual endpoint t is equal to the adjacent connection of the two types of graph point ν Net flow on line ζ. That is, the total flow is the sum of the flows between two map points corresponding to the interior and exterior of the object. By solving the maximum flow in the network flow, the above s-t graph cut can be realized, and then the binary label set L we need can be obtained.

S14, extracting a common face f between tetrahedrons τ with different binary labels λ, and reconstructing the object surface based on the common face f.

In the embodiment of the present disclosure, the common face f between two adjacent tetrahedrons τ whose labels λ are respectively "inner" and "outer" in the tetrahedral grid is extracted. These common faces f are fused together as the reconstructed object surface.

In some embodiments of the present disclosure, the apparatus for reconstructing object surfaces may perform enhancement processing on these extracted common surfaces f. Here, the enhancement processing may include color rendering for the reconstructed surface, etc., which is not limited in this embodiment of the present disclosure. In some embodiments of the present disclosure, the apparatus for reconstructing the object surface may also perform smoothing processing on the extracted common plane f.

Through the above steps, the device for reconstructing the object surface can complete the reconstruction of the object in the scene or the surface of the entire scene.

As shown in FIG. 10 , the embodiment of the present disclosure also provides a device 100 for reconstructing the surface of an object. The device 100 for reconstructing the surface of an object includes: a dense point cloud generation module 110, a mesh generation module 120, a tetrahedron marking module 130, and surface extraction Module 140.

Wherein, the dense point cloud generating module 110 is configured to generate a corresponding dense point cloud using multiple images obtained by shooting the scene from different shooting positions. The mesh generation module 120 is configured to generate a tetrahedral mesh corresponding to the dense point cloud based on the dense point cloud. The tetrahedron labeling module 130 is configured to determine the binary label of each tetrahedron based on the minimization of the energy function, wherein the binary label is used to indicate that the tetrahedron is located inside or outside the surface of the object; the surface extraction module 140 is configured To extract the common face between tetrahedrons with different binary labels, and reconstruct the surface of the object based on the common face; wherein, the energy function includes the sum of the first penalty term corresponding to each tetrahedron and the sum of the The sum of the second penalty term corresponding to each public face; the first penalty term is determined based on the binary label of the corresponding tetrahedron; the second penalty term includes a grid density weight; the grid density The weight is used to characterize the number of shooting location points corresponding to the common surface.

In some embodiments, the tetrahedron labeling module 130 is further configured to generate a binary label for each tetrahedron based on energy function minimization, before each coordinate point in the dense point cloud When the line of sight to the shooting location point intersects with the common surface, set the second penalty item for the common surface; wherein, the second penalty item is the grid density weight and the preset Set the product of penalty coefficients.

In some embodiments, the number of shooting location points corresponding to the common surface includes a sum of the numbers of shooting location points corresponding to each of the three vertices of the common surface.

In some embodiments, the greater the sum of the side lengths of the common surfaces is, the greater the grid density weight is.

In some embodiments, the dense point cloud generation module 110 is further configured to perform feature point matching based on the multiple images to obtain multiple common view points; the common view points are used to represent the corresponding points of the multiple images Points in the scene captured by multiple shooting position points; coordinate points in the dense point cloud are obtained based on the common view points, thereby obtaining the dense point cloud.

In some embodiments, the dense point cloud generating module 110 is further configured to query the coordinate points of the dense point cloud before generating the tetrahedral mesh corresponding to the dense point cloud based on the dense point cloud The source image of the source image; determine the shooting location point corresponding to the source image according to the source image; add the shooting location point to the dense point cloud to obtain an updated dense point cloud; the grid generation module 120 , is further configured to generate a tetrahedral mesh corresponding to the dense point cloud based on the updated dense point cloud.

In some embodiments, the grid generation module 120 is further configured to generate the tetrahedron grid based on the shooting location point and the coordinate point; wherein, for each point cloud from the dense The line of sight from coordinate points to the shooting position point is determined based on the type of vertices of the tetrahedron; the vertices of the tetrahedron include: the shooting position point of the line of sight or the coordinate point of the line of sight.

In some embodiments, the second penalty item further includes a distance weight; the farther the intersection point of the line of sight and the common plane is from the starting point of the line of sight, the greater the distance weight is.

In some embodiments, the ratio between the distance and the length of the line of sight is greater than a first threshold, and the sum of the side lengths of the common surface is equal to that of each of the three vertices of the common surface In a case where the ratio of the sum of the numbers of corresponding shooting location points is greater than the second threshold, the distance weight is zero.

In some embodiments, the tetrahedron labeling module 130 is further configured to map each tetrahedron into a graph point in a directed graph, and use the common face as a graph of the directed graph A connection line between points; a virtual start point and a virtual end point are set in the directed graph, wherein the first penalty item is converted into the flow between the graph point and the virtual start point or virtual end point, and the second The two penalty items are converted into the flow between the graph points; with the goal of maximizing the total flow from the virtual start point to the virtual end point, the binary label is calculated, wherein, in the calculation process of the total flow, only for The flow is summed between two plot points corresponding to the interior and exterior of the object, respectively.

In some embodiments, the tetrahedron is a Delaunay tetrahedron.

In some embodiments, the dense point cloud generating module 110 is further configured to generate depth point clouds corresponding to the multiple images as the dense point cloud.

The above-mentioned method for reconstructing an object surface is generally implemented by an apparatus for reconstructing an object surface from multiple images of a scene. Therefore, an embodiment of the present disclosure also proposes an apparatus for reconstructing an object surface from multiple images of a scene. Please refer to FIG. 11 . FIG. 11 is a schematic structural view of an embodiment of an apparatus 200 for reconstructing an object surface provided by an embodiment of the present disclosure. The apparatus 200 for reconstructing an object surface includes: a processor 210 and a memory 220 . A computer program is stored in the memory 220, and the processor 210 is used to execute the computer program to realize the steps of the method for reconstructing the object surface as described above.

The logical process of the method for reconstructing the object surface from multiple images of the scene is presented as a computer program. In terms of the computer program, if it is sold or used as an independent software product, it can be stored in a computer storage medium. Therefore, the present disclosure implements An example presents a computer storage medium. Please refer to FIG. 12. FIG. 12 is a schematic structural diagram of an optional computer storage medium provided by an embodiment of the present disclosure. A computer program 310 is stored in the computer storage medium 300 provided by an embodiment of the present disclosure, and the computer program 310 is executed by a processor. Realize the method for reconstructing the surface of an object as described above.

The computer storage medium 300 can be a medium that can store computer programs such as a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, or It may also be a server storing the computer program, and the server may send the stored computer program to other devices for execution, or may run the stored computer program itself. From a physical point of view, the computer storage medium 300 may be a combination of multiple entities, such as multiple servers, servers plus storage, or storage plus a mobile hard disk.

In summary, the device for reconstructing the surface of an object constructs the energy function to include the sum of the first penalty term corresponding to the tetrahedron and the sum of the second penalty term corresponding to the common face, since the second penalty term includes the grid Density weight, thereby reducing the negative impact of noise points or abnormal points on the details of the reconstructed object surface, and improving the reconstruction accuracy.

The above is only the implementation of the present disclosure, and does not limit the patent scope of the present disclosure. Any equivalent structure or equivalent process conversion made by using the contents of this disclosure and the accompanying drawings, or directly or indirectly used in other related technologies fields, are equally included in the patent protection scope of the present disclosure.

Industrial Applicability

In the embodiment of the present disclosure, since there are fewer shooting locations corresponding to noise points or abnormal points, a larger penalty can be imposed on sparse grids through the grid density weight, that is, the noise can be reduced through the grid density weight. Points or outliers impose greater penalties, thereby reducing the negative impact of noise points or outliers on details, retaining more details, and improving the accuracy of object reconstruction surfaces.

Claims (18)

1. A method of reconstructing a surface of an object, the method comprising:

   Generate a corresponding dense point cloud by using multiple images of the scene, wherein the multiple images are images obtained by shooting the scene through multiple shooting position points;

   Based on the dense point cloud, generating a tetrahedral mesh corresponding to the dense point cloud;

   Based on the minimization of the energy function, determining a binary label of each tetrahedron, wherein the binary label is used to characterize that the tetrahedron is located inside or outside the surface of the object;

   extracting common faces between tetrahedrons with different binary labels, and reconstructing the object surface based on the common faces,

   Wherein, the energy function includes the sum of the first penalty term corresponding to each tetrahedron and the sum of the second penalty term corresponding to each common face; the first penalty term is based on the corresponding The binary label of the tetrahedron is determined; the second penalty item includes a grid density weight; the grid density weight is used to characterize the number of shooting location points corresponding to the common surface.
2. The method according to claim 1, wherein, prior to generating a binary label for each tetrahedron based on the minimization of the energy function, the method further comprises:

   In the case that the line of sight from each coordinate point in the dense point cloud to the shooting location point intersects with the common surface, the second penalty item is set for the common surface; wherein, the The second penalty item is the product of the grid density weight and a preset penalty coefficient.
3. The method according to claim 2, wherein the number of shooting location points corresponding to the common surface comprises a sum of the number of shooting location points corresponding to each of the three vertices of the common surface.
4. The method according to any one of claims 1-3, wherein the greater the number of shooting location points corresponding to the common surface, the smaller the weight of the grid density.
5. The method according to any one of claims 1-3, wherein the greater the sum of the side lengths of the common surfaces is, the greater the grid density weight is.
6. The method according to any one of claims 1-5, wherein said generating a corresponding dense point cloud using multiple images of the scene comprises:

   Perform feature point matching based on the plurality of images to obtain a plurality of common view points; the common view points are used to represent points in the scene captured by the plurality of shooting position points corresponding to the plurality of images;

   The coordinate points in the dense point cloud are obtained based on the common view points, so as to obtain the dense point cloud.
7. The method according to any one of claims 1-6, wherein, before generating a tetrahedral mesh corresponding to the dense point cloud based on the dense point cloud, the method further comprises:

   Query the source image of the coordinate points of the dense point cloud;

   determining the shooting location point corresponding to the source image according to the source image;

   Adding the shooting location point to the dense point cloud to obtain an updated dense point cloud;

   The generating a tetrahedron grid corresponding to the dense point cloud based on the dense point cloud includes:

   Based on the updated dense point cloud, a tetrahedral mesh corresponding to the dense point cloud is generated.
8. The method according to any one of claims 1-7, wherein said generating a tetrahedral mesh corresponding to said dense point cloud based on said dense point cloud comprises:

   generating the tetrahedron grid based on the shooting location point and the coordinate point;

   Wherein, the line of sight from each coordinate point in the dense point cloud to the shooting position point is determined based on the type of vertices of the tetrahedron; the vertices of the tetrahedron include: the line of sight The shooting position point or the coordinate point of the line of sight.
9. The method according to any one of claims 2-8, wherein the second penalty item further comprises a distance weight; the method further comprises:

   The farther the distance between the intersection point of the line of sight and the common plane is from the starting point of the line of sight, the greater the weight of the distance.
10. The method according to claim 9, wherein the method further comprises:

    The ratio between the distance and the length of the line of sight is greater than a first threshold, and the sum of the side lengths of the common surface corresponds to the number of shooting position points corresponding to each of the three vertices of the common surface When the ratio of the sum is greater than the second threshold, the distance weight is zero.
11. The method according to claim 1, wherein said minimization based on energy function generates a binary label for each tetrahedron, comprising:

    mapping each of the tetrahedrons into a point in a directed graph, and using the common face as a connection between the points in the directed graph;

    A virtual start point and a virtual end point are set in the directed graph, wherein the first penalty item is converted to the flow between the graph point and the virtual start point or virtual end point, and the second penalty item is converted to the flow between graph points;

    The binary label is calculated with the goal of maximizing the total flow from the virtual start point to the virtual end point, wherein, during the calculation of the total flow, only two map points corresponding to the inside and outside of the object The flows between them are summed.
12. The method of claim 1, wherein the tetrahedron is a Delaunay tetrahedron.
13. The method of claim 1, wherein the plurality of images are color images of at least a portion of the scene.
14. The method according to claim 1, wherein said generating a corresponding dense point cloud using said plurality of images comprises:

    Generate a depth point cloud corresponding to the plurality of images as the dense point cloud.
15. A device for reconstructing the surface of an object, the device comprising:

    The dense point cloud generation module is configured to generate a corresponding dense point cloud using multiple images of the scene, wherein the multiple images are images obtained by shooting the scene through multiple shooting position points;

    A mesh generation module configured to generate a tetrahedral mesh corresponding to the dense point cloud based on the dense point cloud, wherein the vertex of each tetrahedron in the tetrahedral mesh is the dense point Coordinate points in the cloud;

    A tetrahedron labeling module configured to generate a binary label for each of the tetrahedrons based on the minimization of an energy function, wherein the binary label is used to characterize that the tetrahedron is located inside or outside a surface of an object;

    a surface extraction module configured to extract common faces between tetrahedrons with different binary labels, and reconstruct said object surface based on said common faces,

    Wherein, the energy function includes the sum of the first penalty term corresponding to each tetrahedron and the sum of the second penalty term corresponding to each common face; the first penalty term is based on the corresponding The binary label of the tetrahedron is determined; the second penalty item includes a grid density weight; the grid density weight is used to characterize the number of shooting location points corresponding to the common surface.
16. A device for reconstructing the surface of an object, the device comprising a processor and a memory; a computer program is stored in the memory, and the processor is used to execute the computer program to realize any one of claims 1-14 method steps.
17. A computer storage medium, the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1-14 are realized.
18. A computer program product comprising computer readable code, said computer program product comprising a computer program or instructions which, when run on an electronic device, causes said electronic device to perform claims 1-14 any one of the methods described.

Images