TW202309834A - Model reconstruction method, electronic device and computer-readable storage medium - Google Patents

Abstract

The disclosure provides a model reconstruction method, electronic device and computer-readable storage medium, wherein the model reconstruction method comprises: fusing a depth image and a reference voxel model of a target object to obtain a first voxel model of the target object; wherein adjacent first voxels in the first voxel model are separated by a first voxel distance, and at least part of the first voxels have first voxel information; resampling the first voxel model to obtain a second voxel model, wherein adjacent second voxels in the second voxel model are separated by a second voxel distance, the second voxel distance is greater than the first voxel distance, and at least part of the second voxel has second voxel information.

Description

Model reconstruction method, electronic device, and computer-readable storage medium

The invention relates to the technical field of computer vision, and relates to a model reconstruction method, electronic equipment and a computer-readable storage medium.

In the field of 3D reconstruction, voxel is a common method to realize model modeling. Voxel discretizes the 3D space into densely arranged voxels, and stores related voxel information on the voxels to model objects or scenes. For example, in Truncated Signed Distance Function (TSDF), the truncated signed distance to the nearest surface of the object or scene is stored on the voxel, so as to realize the implicit expression of the object or scene.

However, with limited computational memory, if the object or scene exceeds the representation range of the reconstruction model, it is usually discarded, or try to choose a larger voxel distance and start reconstruction from scratch, the above method, or Is it sacrificing reconstruction integrity, or sacrificing time efficiency, and these costs are unbearable in practical applications. In view of this, how to improve the time efficiency of reconstruction on the basis of ensuring the integrity of reconstruction has become an urgent technical problem to be solved.

Embodiments of the present invention provide a model reconstruction method, electronic equipment, and a computer-readable storage medium.

The first aspect of the embodiment of the present invention provides a model reconstruction method, the method is executed by an electronic device, and the method includes: fusing the depth image of the target object with a reference voxel model to obtain the first voxel of the target object model; wherein, in the first voxel model, adjacent first voxels are separated by a first voxel distance, and at least part of the first voxels have first voxel information; the first voxel model is resampled to obtain the first voxel Two-voxel model; wherein, in the second voxel model, adjacent second voxels are separated by a second voxel distance, the second voxel distance is greater than the first voxel distance, and at least some of the second voxels have a second voxel Information.

Therefore, the depth image of the target object is fused with the reference voxel model to obtain the first voxel model of the target object, and the first voxel model is adjacent to the first voxel with a first voxel distance, and at least part of the first voxel model One voxel has the first voxel information, and then the first voxel model is resampled to obtain the second voxel model, and the second voxel distance between adjacent second voxels in the second voxel model is the second voxel distance. The second voxel distance is greater than the first voxel distance, and at least part of the second voxel has second voxel information. Since the second voxel distance is greater than the first voxel distance, on the one hand, the representation of the model can not be lost after resampling range, and on the other hand, because the second voxel model can be obtained directly by resampling on the basis of the first voxel model, there is no need to rebuild from scratch, and the time efficiency of model reconstruction can also be improved, so it is possible to ensure complete reconstruction On the basis of reliability, the reconstruction time efficiency is improved. In addition, because the second voxel distance is greater than the first voxel distance, it can also reduce the number of voxels required to represent the target object, which can help reduce the computational memory required for model reconstruction. body while ensuring reconstruction integrity and improving time efficiency.

The second aspect of the embodiment of the present invention provides a model reconstruction device, including: a depth fusion part and a resampling part, the depth fusion part is used to fuse the depth image of the target object with the reference voxel model to obtain the first object of the target object A voxel model; wherein, in the first voxel model, adjacent first voxels are separated by a first voxel distance, and at least some of the first voxels have first voxel information; the resampling part is used for the first voxel The voxel model is resampled to obtain a second voxel model; wherein, in the second voxel model, adjacent second voxels are separated by a second voxel distance, and the second voxel distance is greater than the first voxel distance, and at least partially The second voxel has second voxel information.

The third aspect of the embodiment of the present invention provides an electronic device, including a memory and a processor coupled to each other, and the processor is used to execute program instructions stored in the memory to implement the model reconstruction method in the first aspect above.

The fourth aspect of the embodiment of the present invention provides a computer-readable storage medium, on which program instructions are stored, and when the program instructions are executed by a processor, the model reconstruction method in the above-mentioned first aspect is implemented.

The fifth aspect of the embodiment of the present invention provides a computer program product, including computer programs or instructions, which enable the computer to execute the model reconstruction method in the first aspect above when the computer programs or instructions are run on the computer.

In the above solution, the depth image of the target object is fused with the reference voxel model to obtain the first voxel model of the target object, and the first voxel model is adjacent to the first voxel with a first voxel distance, and at least part of The first voxel has first voxel information, and then the first voxel model is resampled to obtain a second voxel model, and in the second voxel model, adjacent second voxels are separated by a second voxel distance, The second voxel distance is greater than the first voxel distance, and at least some of the second voxels have second voxel information. Since the second voxel distance is greater than the first voxel distance, on the one hand, the model can not be lost after resampling represents the range, and on the other hand, the second voxel model can be obtained by resampling directly on the basis of the first voxel model, without having to rebuild from scratch, and can also improve the time efficiency of model reconstruction, so it is possible to ensure that the reconstruction On the basis of completeness, the reconstruction time efficiency is improved. In addition, because the second voxel distance is greater than the first voxel distance, it can also reduce the number of voxels required to represent the target object, which can help reduce the computational memory required for model reconstruction. body while ensuring reconstruction integrity and improving time efficiency.

The solutions of the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings.

In the following description, for the purpose of illustration rather than limitation, details such as specific system structures, interfaces, and techniques are presented for a thorough understanding of the embodiments of the present invention.

The terms "system" and "network" are often used interchangeably herein. The term "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist at the same time, and B exists alone. three conditions. In addition, the character "/" in this article generally indicates that the contextual objects are an "or" relationship. In addition, "many" herein means two or more than two.

Please refer to FIG. 1 . FIG. 1 is a schematic flowchart of an embodiment of a model reconstruction method according to an embodiment of the present invention, which may include the following steps.

Step S11: Fusion the depth image of the target object with the reference voxel model to obtain a first voxel model of the target object.

In some embodiments, the target object may include but not limited to: single objects such as statues and chairs, or the target object may also include: scenes containing multiple objects such as a sand table and living room, which are not limited here.

In some embodiments, the depth camera can be used to shoot the target object to obtain the depth image of the target object; or, a mobile terminal (such as a mobile phone, a tablet computer, etc.) integrated with an RGB camera and a depth camera can also be used to image the target object Shoot to obtain the color image and depth image of the target object; or, use the mobile terminal integrated as an RGB camera to shoot the target object, and perform depth restoration based on multi-view stereo matching algorithm, neural network algorithm, etc. , to obtain the depth image of the target object, which is not limited here.

In some embodiments, when the model reconstruction operation is performed for the first time, an initial model can be initialized as a reference voxel model of the target object, and adjacent voxels in the reference voxel model are separated by a first voxel distance, and the reference voxel model Each voxel in has no voxel information. On this basis, the depth image and the reference voxel model are fused to obtain the first voxel model. For the fusion process, please refer to the related description below.

In some embodiments, when the model reconstruction operation is performed for the ith (i>1) time, the reconstruction model obtained by performing the model reconstruction operation for the i-1 time can be used as a reference voxel model, and the depth image and The reference voxel model is fused to obtain the first voxel model. For the fusion process, please refer to the related description below.

In an embodiment of the present invention, adjacent first voxels in the first voxel model are separated by a first voxel distance, and at least some of the first voxels have first voxel information. As mentioned above, the interval between adjacent voxels in the reference voxel model may be the same as that in the first voxel model, and the first voxel interval may be denoted as δ for expressing description.

In some embodiments, in order to facilitate fusion calculation, both the first voxel model and the reference voxel model can be located in the world coordinate system, and the first voxel information includes the first truncated signed distance and the first voxel voxel weight. Similarly, if a voxel in the reference voxel model has voxel information, the voxel information can also include the truncated sign distance and voxel weight of the voxel. It should be noted that the truncated signed distance represents the directed distance from the voxel to the surface of the target object. Please refer to FIG. 2 , which is a schematic diagram of an embodiment of the truncated symbol distance. As shown in Figure 2, the grid intersection represents a voxel, the value above the voxel represents the truncated sign distance of the voxel, and the thickened curve represents the surface of the target object. When the truncated sign distance of the voxel is a positive number, it represents the voxel Located inside the target object, when the truncation sign distance of the voxel is a negative number, it means that the voxel is located outside the target object; when the truncation sign distance of the voxel is 0, it means that the voxel is located on the surface of the target object, so by truncation Signed distances enable the construction of implicit representations of arbitrary surface indirections.

In some embodiments, based on the back-projection of the primitive points in the depth image to the projection points of the world coordinate system, select voxels in the reference voxel model as the voxels to be fused, and based on the camera internal reference and the captured depth image The camera pose at the time, obtain the truncated symbol distance to be fused, on this basis, based on the voxel weight of the voxel to be fused in the reference voxel model, the truncated symbol distance to be fused and the truncated symbol distance to be fused The voxel is fused with reference to the truncated signed distance in the reference voxel model, and the first truncated signed distance of the first voxel corresponding to the position of the voxel to be fused is obtained, and the weight of the voxel to be fused is calculated in the reference voxel model. update to obtain the first voxel weight of the first voxel corresponding to the position of the voxel to be fused. The above method can continuously incorporate in-depth information during the model reconstruction process, which is conducive to continuously improving the accuracy and integrity of the model.

……（1） In some embodiments, for the convenience of description, the depth value of the primitive point (u, v) in the depth image can be recorded as d, then the image The element point (u, v) is back-projected to the world coordinate system to obtain the projection point P, as shown in formula (1):

……(1)

表示拍攝深度圖像時的相機位姿，

表示反投影函數。反投影函數可以表示為公式（2）：

……（2） In the above formula (1),

Indicates the camera pose when capturing the depth image,

Represents the backprojection function. The backprojection function can be expressed as formula (2):

……(2)

均為相機內參中的參數，分別表示相機在u，v方向上的焦距，

也均為相機內參中的參數，分別表示相機在u，v方向上的光心位置。在此基礎上，可以將深度圖像中各個圖元點（或者，屬於目標對象的圖元點）反投影至世界座標系，得到各個圖元點（或者，屬於目標對象的圖元點）在世界座標系中的投影點，則可以將參考體素模型中位於這些投影點預設截斷範圍

內的體素，作為待融合體素。 In the above formula (2),

Both are the parameters in the internal reference of the camera, respectively representing the focal length of the camera in the u and v directions,

They are also parameters in the internal reference of the camera, respectively representing the position of the optical center of the camera in the u and v directions. On this basis, each primitive point (or, the primitive point belonging to the target object) in the depth image can be back-projected to the world coordinate system, and each primitive point (or, the primitive point belonging to the target object) can be obtained in The projected points in the world coordinate system, you can preset the truncated range of these projected points in the reference voxel model

The voxels within are the voxels to be fused.

，其中，

表示相機位姿，

表示投影函數，投影函數可以表示為公式（3）：

……（3） In an implementation scenario, the voxels to be fused can be reprojected based on the internal camera parameters and the camera pose to obtain the first depth of the voxels to be fused in the depth image and the second depth in the camera coordinate system. For the convenience of description, the position coordinate of the voxel to be fused in the world coordinate system can be marked as V, then the first depth can be expressed as

,in,

represents the camera pose,

Represents the projection function, which can be expressed as formula (3):

... (3)

表示待融合體素重投影到相機座標系中的位置座標。需要說明的是，通過相機位姿

可以將待融合體素在世界座標系中位置座標V轉換至相機座標系，得到待融合體素在相機座標系中位置座標

，在此基礎上，可以進一步利用上述投影函數將待融合體素在相機座標系中的位置座標轉換至圖像座標系，得到待融合體素在圖像座標系中的位置座標，從而可以根據該位置座標得到待融合體素在深度圖像中的第一深度

。為了簡化表述，對於待融合體素集合內任一體素可以記為v，其重投影至深度圖像的位置座標可以記為u（v），深度圖像在該位置座標處的第一深度可以記為D（u（v））。此外，如前所述，通過相機位姿

可以將待融合體素在世界座標系中位置座標V轉換至相機座標系，得到待融合體素在相機座標系中的位置座標

，則待融合體素在相機座標系的第二深度可以記為

，其中，

表示取深度值，可以取其在相機座標系z軸上的座標值作為第二深度。為了簡化表述，對於待融合體素集合內任一體素可以記為v，其重投影至相機座標系的第二深度可以記為z（v）。在一些實施例中，在獲取第一深度D（u（v））和第二深度z（v）之後，即可基於兩者之間的偏差，得到待融合體素v的待融合截斷符號距離

，如公式（4）所示：

……（4） In the above formula (3),

Indicates the position coordinates of the reprojected voxels to be fused into the camera coordinate system. It should be noted that, through the camera pose

The position coordinate V of the voxel to be fused in the world coordinate system can be converted to the camera coordinate system to obtain the position coordinate of the voxel to be fused in the camera coordinate system

, on this basis, the position coordinates of the voxels to be fused in the camera coordinate system can be further converted to the image coordinate system by using the above projection function, and the position coordinates of the voxels to be fused in the image coordinate system can be obtained, so that according to The position coordinates get the first depth of the voxel to be fused in the depth image

. In order to simplify the expression, any voxel in the voxel set to be fused can be marked as v, and the position coordinate of its reprojection to the depth image can be marked as u(v), and the first depth of the depth image at this position coordinate can be Denote it as D(u(v)). Furthermore, as mentioned earlier, through the camera pose

The position coordinate V of the voxel to be fused in the world coordinate system can be converted to the camera coordinate system to obtain the position coordinate of the voxel to be fused in the camera coordinate system

, then the second depth of the voxel to be fused in the camera coordinate system can be recorded as

,in,

Indicates to take the depth value, and its coordinate value on the z-axis of the camera coordinate system can be taken as the second depth. In order to simplify the expression, any voxel in the voxel set to be fused can be denoted as v, and its second depth reprojected to the camera coordinate system can be denoted as z(v). In some embodiments, after obtaining the first depth D(u(v)) and the second depth z(v), the truncated symbol distance to be fused can be obtained based on the deviation between the two

, as shown in formula (4):

... (4)

表示前述位置座標u（v）在水平軸x方向上的座標值，

表示前述位置座標u（v）在垂直軸y方向上的座標值，

表示篩選待融合體素時所採用的截斷距離。在計算得到

的基礎上，再將其截斷至[-1，1]區間，即可得到待融合體素v的待融合截斷符號距離

，可以表示為

。 In the above formula (4),

Indicates the coordinate value of the aforementioned position coordinate u(v) in the direction of the horizontal axis x,

Indicates the coordinate value of the aforementioned position coordinate u(v) in the direction of the vertical axis y,

Indicates the cut-off distance used when screening voxels to be fused. in the calculated

On the basis of , then truncate it to the [-1, 1] interval, you can get the truncated symbol distance to be fused for the voxel v to be fused

,It can be expressed as

.

，在參考體素模型中參考截斷符號距離記為

，則在待融合體素v每次融合時的權重設置為1的情況下，與待融合體素位置對應的第一體素的第一截斷符號距離

可以表示為公式（5）：

……（5） Since the truncated symbol distance to be fused is obtained according to the deviation between the first depth and the second depth, it can accurately represent the directional distance from the voxel to be fused to the surface of the target object according to the depth information provided by the depth image, which is beneficial To improve the accuracy of the truncated symbol distance to be fused. In some embodiments, when fusing the truncated signed distance to be fused and the reference truncated signed distance of the voxel to be fused in the reference voxel model based on the voxel weight of the voxel to be fused in the reference voxel model, the voxel to be fused can be obtained The voxel weight in the reference voxel model, and the weight of the voxel to be fused in this fusion, and use the above two weights to treat the reference truncation sign distance of the fused voxel in the reference model, and the to-be-fused truncation The signed distances are weighted and averaged to obtain the first truncated signed distance of the first voxel corresponding to the position of the voxel to be fused. For the convenience of description, the voxel weight of the voxel to be fused in the reference voxel model can be recorded as

, and the reference truncated sign distance in the reference voxel model is denoted as

, then when the weight of the voxel v to be fused is set to 1 for each fusion, the first truncated sign distance of the first voxel corresponding to the voxel position to be fused

It can be expressed as formula (5):

... (5)

It should be noted that the voxel to be fused and the first voxel corresponding to the position of the voxel to be fused have the same position coordinates in the world coordinate system, that is, they are the same point in space.

、

可以視為0。這樣能夠在待融合體素在參考體素模型中不具有參考截斷符號距離的情況下，大大簡化第一截斷符號距離的獲取流程，有利於減少模型重建所需的計算記憶體。 In some embodiments, in the case that the voxel to be fused does not have a reference truncated signed distance in the reference voxel model, the truncated signed distance to be fused of the voxel to be fused can be used as the second corresponding to the position of the voxel to be fused The first truncated sign distance in voxels. That is, in this case, in formula (5)

,

Can be regarded as 0. In this way, when the voxel to be fused does not have a reference truncated signed distance in the reference voxel model, the process of obtaining the first truncated signed distance can be greatly simplified, which is beneficial to reduce the computational memory required for model reconstruction.

表示為公式（6）：

……（6） In some embodiments, after the voxel weight is updated, the first voxel weight of the first voxel corresponding to the position of the voxel to be fused is greater than the voxel weight of the voxel to be fused in the reference voxel model. For example, as mentioned above, the weight of the voxel to be fused v can be set to 1 each time it is fused, and the voxel weight of the voxel to be fused can be added to the reference voxel model by 1, as the voxel corresponding to the position of the voxel to be fused The first voxel weight of the first voxel, in this case, for the convenience of description, the first voxel weight of the first voxel corresponding to the position of the voxel to be fused can be

Expressed as formula (6):

... (6)

，則上述公式（5）所表示的第一截斷符號距離

可以相應地表示為：

，在此情況下，上述公式（6）所表示的第一體素權重

可以相應地表示為：

，也就是說，與待融合體素v位置對應的第一體素的第一體素權重為待融合體素v為在參考體素模型中體素權重與待融合體素v每次融合時的權重之和。需要說明的是，待融合體素v每次融合時的權重

可以根據待融合體素v的待融合截斷符號距離

確定，例如，若待融合體素v的待融合截斷符號距離

表示待融合體素至目標對象的表面距離越近，則待融合體素v每次融合時的權重

可以越大，且越趨近於1；或者，待融合體素v每次融合時的權重

也可以根據待融合體素v重投影至深度圖像的局部方差確定，例如，若待融合體素v重投影至深度圖像的局部方差越小，則待融合體素v每次融合時的權重

可以越大，且越趨近於1，在此不做限定。 It should be noted that, more generally, the weight of the voxel v to be fused each time it is fused can also be set to other values. For the convenience of description, it can be recorded as

, then the first truncated sign distance represented by the above formula (5)

can be expressed accordingly as:

, in this case, the first voxel weight expressed by the above formula (6)

can be expressed accordingly as:

, that is to say, the first voxel weight of the first voxel corresponding to the position of the voxel to be fused is Voxel to be fused v is the weight of the voxel in the reference voxel model and the voxel to be fused v Each fusion The sum of the weights of . It should be noted that the weight of voxels to be fused each time they are fused

The symbol distance to be fused can be truncated according to the voxel v to be fused

Determine, for example, the truncated sign distance to be fused if voxel to be fused

Indicates that the closer the surface distance between the voxel to be fused and the target object is, the weight of the voxel to be fused v for each fusion

Can be larger and closer to 1; or, the weight of voxels to be fused each time they are fused

It can also be determined according to the local variance of the reprojection of the voxel v to be fused to the depth image. For example, if the local variance of the reprojection of the voxel v to be fused to the depth image is smaller, the voxel v to be fused each time is fused. Weights

It can be larger and closer to 1, which is not limited here.

Therefore, setting the first voxel weight of the first voxel corresponding to the position of the voxel to be fused to be greater than the voxel weight of the voxel to be fused in the reference voxel model can gradually focus on the reference voxel during the model reconstruction process. The voxels to be fused refer to the truncated signed distance in the reference voxel model, which is beneficial to improve the accuracy of model reconstruction.

Step S12: resampling the first voxel model to obtain a second voxel model. Fusion the depth image of the target object with the reference voxel model to obtain the first voxel model of the target object

In the embodiment of the present invention, adjacent second voxels in the second voxel model are separated by a second voxel distance, the second voxel distance is greater than the first voxel distance, and at least some of the second voxels have second voxel information . In some embodiments, the second voxel information is calculated according to the first voxel information during the process of resampling the first voxel model.

In some embodiments, the second voxel distance may be a preset multiple of the first voxel distance, for example, 1.5 times, 2 times, etc., which is not limited herein. Please refer to FIG. 3 , which is a schematic diagram of an embodiment of the first voxel model and the second voxel model. It should be noted that Fig. 3 only schematically draws part of the first voxel model and part of the second voxel model. As shown in Fig. 3, the circles filled with oblique lines represent the first voxel, and the circles filled with black shadows represent The second voxel, and the distance of the second voxel in Fig. 3 is 1.5 times of the distance of the first voxel. Other situations can be deduced by analogy, and no more examples will be given here.

In some embodiments, after obtaining the first voxel model each time, based on the first voxel with the first voxel information, it is possible to detect whether the first voxel model meets the resampling condition, and respond to the first voxel The model meets the resampling conditions, execute the step of resampling the first voxel model to obtain the second voxel model, that is, check whether the first voxel model meets the resampling conditions before resampling, and can adjust the voxel by itself spacing. In addition, in order to continuously incorporate depth information to improve the model reconstruction effect, after resampling, the second voxel model can be further used as a new reference voxel model to obtain a new depth image, and re-execute the depth of the target object The image and reference are fused to obtain the first voxel model of the target object and the subsequent steps, so that in the process of model reconstruction, depth information can be continuously integrated to improve the effect of model reconstruction. It should be noted that the new depth image may be obtained by shooting the target object with different camera poses. You can refer to the related description below.

In some embodiments, the resampling condition can be set as the number of first voxels with the first voxel information is more than a preset value. It should be noted that if the first voxel has the first voxel information, then in the reconstruction process, it is necessary to allocate computing memory for the first voxel, so when the number of first voxels with the first voxel information is relatively small If there are too many, the computing memory required for model reconstruction is also large. In this case, it can be considered that the first voxel model meets the resampling conditions, and the first voxel model is resampled to reduce the cost of model reconstruction. required computing memory.

個第一體素，k的數值在此不做限定，如可以設置為15、16、17等等。在此情況下，重採樣條件可以設置為包括：具有參考體素的第一區域多於預設閾值，參考體素為具有第一體素資訊的第一體素。也就是說，可以統計第一體素模型中具有參考體素的第一區域的數量，該數量越多，說明模型重建需要分配的計算記憶體也越多，則為了減少計算記憶體，可以對第一體素模型進行重採樣。 In some embodiments, the first voxel model may contain several first regions, and the first regions include a preset number of first voxels, such as each first region may contain

The first voxel, the value of k is not limited here, for example, it can be set to 15, 16, 17 and so on. In this case, the resampling condition may be set to include: the first region with reference voxels is more than a preset threshold, and the reference voxels are the first voxels with the first voxel information. That is to say, the number of the first regions with reference voxels in the first voxel model can be counted. The larger the number, the more computing memory needs to be allocated for model reconstruction. In order to reduce the computing memory, the The first voxel model is resampled.

In some embodiments, as mentioned above, if the first voxel has the first voxel information, it is necessary to allocate computing memory for the first voxel during the reconstruction process, and the first voxel model can also be obtained Afterwards, the currently consumed computing memory is counted, and the resampling condition may be set to include: the currently consumed computing memory is greater than a preset threshold. In this case, if the first voxel model satisfies the resampling condition, in order to reduce the probability of occurrence of subsequent freezes, etc., the first voxel model may be resampled to reduce computing memory.

In some embodiments, in response to the fact that the first voxel model does not meet the resampling conditions, the first voxel model can be used as a new reference voxel model, and a new depth image is acquired, and the depth image of the target object is re-executed. The image and the reference voxel model are fused to obtain the first voxel model of the target object and subsequent steps, so that in the process of model reconstruction, depth information can be continuously integrated to improve the effect of model reconstruction. It should be noted that the new depth image may be obtained by shooting the target object with different camera poses. You can refer to the related description below.

In some embodiments, each time after the first voxel model is obtained, the first voxel model may be directly resampled without checking whether the first voxel model meets the resampling condition. It should be noted that, in this case, in order to reduce as much as possible the loss of accuracy caused by resampling because the first voxel model does not meet the resampling conditions, the preset multiple can be set appropriately smaller. For example, in the case of detecting whether the first voxel model meets the resampling condition, the preset multiple can be set to 1.5, 2, etc., then in the case of not detecting whether the first voxel model meets the resampling condition, the preset The multiple can be set to 1.1, 1.2, etc., which is not limited here.

In some embodiments, resampling can be implemented by using but not limited to interpolation algorithms such as trilinear interpolation, and reference can be made to the following disclosed embodiments.

It should be noted that the embodiments of the present invention and the following disclosed embodiments can be applied to model reconstruction during real-time scanning, and can also be applied to model reconstruction after scanning, and are not limited here.

In some embodiments, taking model reconstruction during real-time scanning as an example, a mobile terminal integrated with an RGB camera and a depth camera can be used to photograph the target object to obtain the first frame of image information, and the first frame of image information Including color images and depth images, and obtaining the camera pose when shooting the first frame of image information, and using the first frame of image information as key image information, on this basis, an initial model can be initialized, and Using the model reconstruction method in the embodiment of the present invention, a reconstruction model is obtained by reconstructing the first frame of depth image and the initial model as a reference voxel model for the next model reconstruction; at the same time, the second frame of image is obtained after shooting After information, the camera position of the second frame of image information can be obtained by combining the pose tracking algorithm (such as the nearest neighbor iterative algorithm, etc.), the second frame of image information and the camera pose of the first frame of image information. pose, and when the difference between the camera pose of the second frame of image information and the camera pose of the first frame of image information satisfies a preset condition (for example, the difference is greater than a threshold), the second frame of image Information is the key image information. On this basis, a reconstruction model can be reconstructed by using the second frame depth image and the latest reference voxel model, which will be used as the reference voxel model for the next model reconstruction. If the condition is set, continue to track the camera pose of the third frame of image information, and check whether the third frame of image information can be used as the key image information, and if so, use the third frame of depth image and the latest reference Voxel model reconstruction obtains a reconstructed model as the reference voxel model for the next model reconstruction. If not possible, continue to perform pose tracking on the fourth frame of image information, and so on until the scanning is completed, so that the scanning process can In-depth information is continuously integrated to improve the model reconstruction effect. It should be noted that for each process of model reconstruction using the reference voxel model and the depth image, reference may be made to the foregoing related descriptions.

In some embodiments, take model reconstruction after scanning as an example. As mentioned above, during the scanning process, pose tracking can be performed continuously to obtain key image information. After scanning, the scanning time can be obtained. There are several key image information and their camera poses sorted from early to late. On this basis, a key image information can be selected sequentially, and a reconstruction model can be obtained by using the depth image in the key image information and the latest reference voxel model, which can be used as the reference voxel for the next model reconstruction model, and so on, until key image information is selected. It should be noted that, when the model is reconstructed for the first time, an initial model can be initialized as a reference voxel model. In addition, for each process of model reconstruction using the reference voxel model and the depth image, reference may be made to the aforementioned related descriptions.

In the above solution, the depth image of the target object is fused with the reference voxel model to obtain the first voxel model of the target object, and the first voxel model is adjacent to the first voxel with a first voxel distance, and at least part of The first voxel has first voxel information, and then the first voxel model is resampled to obtain a second voxel model, and in the second voxel model, adjacent second voxels are separated by a second voxel distance, The second voxel distance is greater than the first voxel distance, and at least some of the second voxels have second voxel information. Since the second voxel distance is greater than the first voxel distance, on the one hand, the model can not be lost after resampling represents the range, and on the other hand, the second voxel model can be obtained by resampling directly on the basis of the first voxel model, without having to rebuild from scratch, and can also improve the time efficiency of model reconstruction, so it is possible to ensure that the reconstruction On the basis of completeness, the reconstruction time efficiency is improved. In addition, because the second voxel distance is greater than the first voxel distance, it can also reduce the number of voxels required to represent the target object, which can help reduce the computational memory required for model reconstruction. body while ensuring reconstruction integrity and improving time efficiency.

Please refer to FIG. 4 , which is a schematic flowchart of an embodiment of step S12 in FIG. 1 . In the embodiment of the present invention, both the first voxel model and the second voxel model are located in the world coordinate system, which may include the following steps.

Step S41: Screen the first voxel as a candidate voxel for the second voxel.

，k的數值可以設置為15、16、17等等，在此不做限定。在此基礎上，可以在若干第一區域中篩選候選區域，且候選區域中至少一個第一體素具有第一體素資訊，從而選擇與候選區域至少部分重合的第二區域，作為與候選區域對應的目標區域，再選擇目標區域內預設數值個第二體素，以分別為各個選擇的第二體素篩選第一體素，得到候選體素，也就是說，可以將各個選擇的第二體素，作為即將確定第二體素資訊的第二體素。上述方式，能夠在將第一體素模型和第二體素模型劃分區域的基礎上，根據區域劃分針對性地選擇第二區域，以將其內預設閾值個第二體素作為後續即將確定第二體素資訊的第二體素，能夠排除其他無關的第二體素對重採樣的影響，有利於加速模型重建。 In some embodiments, as mentioned above, the first voxel model may contain several first regions, the second voxel model may contain several second regions, and the first region contains a preset number of first voxels, and the second voxel model may contain several second regions. The second area contains a preset number of second voxels, and the preset value can be set to

, the value of k can be set to 15, 16, 17, etc., which is not limited here. On this basis, the candidate area can be screened in several first areas, and at least one first voxel in the candidate area has the first voxel information, so as to select the second area that at least partially overlaps with the candidate area as the candidate area. The corresponding target area, and then select a preset number of second voxels in the target area to screen the first voxels for each selected second voxel to obtain candidate voxels, that is to say, each selected second voxel can be The second voxel is used as the second voxel to determine the second voxel information. In the above method, on the basis of dividing the first voxel model and the second voxel model into regions, the second region can be targetedly selected according to the region division, so that the preset threshold second voxels in it can be used as subsequent determinations. The second voxel of the second voxel information can eliminate the influence of other irrelevant second voxels on resampling, which is beneficial to speed up model reconstruction.

In some embodiments, please refer to FIG. 3 , taking k=2 as an example, the first voxel in the first area of 2*2*2 on the left side (that is, the 2*2*2 circles filled with slash shadows on the left side shape) has the first voxel information, then the first region can be used as a candidate region, which is the same as the second region of 2*2*2 shown in Figure 3 (that is, the 2*2*2 in Figure 3 composed of The circle filled with black shadow) partially overlaps, then the second area can be used as the target area, and the 2*2*2 second voxels in the target area can be used as the second voxels to determine the second voxel information . Other situations can be deduced by analogy, and no more examples will be given here.

，各個候選區域對應的目標區域可以記為

，在此基礎上，對於上述目標區域集合

每一個目標區域

內各個第二體素

均可以執行下述插值計算過程。 In some embodiments, for the convenience of description, the candidate region in the first voxel model can be recorded as

, the target area corresponding to each candidate area can be recorded as

, on this basis, for the above target area set

each target area

Each second voxel in

Both can perform the following interpolation calculation process.

之後，可以根據第二體素

在世界座標系中的位置座標

，在第一體素模型中搜索與該第二體素

最接近的八個第一體素，作為其候選體素。請結合參閱圖3，以圖3中實線箭頭所指第二體素為例，可以先在z座標軸方向低於該第二體素的第一體素中搜索一個與該第二體素最接近的第一體素，如可以搜索到虛線箭頭所指第一體素，在此基礎上，可以將該第一體素右側相鄰的第一體素、後方相鄰的第一體素以及上方相鄰的第一體素均作為該第二體素的候選體素，以及這些候選體素作為頂點的正方體上其他第一體素也作為該第二體素的候選體素，即可以獲取到該第二體素的八個最接近的候選體素。其他情況可以以此類推，在此不再一一舉例。 In some embodiments, after obtaining the second voxel that is about to determine the second voxel information

Then, according to the second voxel

Position coordinates in the world coordinate system

, in the first voxel model to search for the second voxel

The eight closest first voxels are taken as its candidate voxels. Please refer to FIG. 3. Taking the second voxel indicated by the solid arrow in FIG. 3 as an example, you can first search for a voxel that is closest to the second voxel in the z-axis direction of the first voxel that is lower than the second voxel. The closest first voxel, if the first voxel indicated by the dotted arrow can be searched, on this basis, the first voxel adjacent to the right side of the first voxel, the first voxel adjacent to the rear and The first voxels adjacent above are all candidates for the second voxel, and the other first voxels on the cube whose vertices are the vertices of these candidate voxels are also candidates for the second voxel, that is, you can get The eight closest candidate voxels to this second voxel. Other situations can be deduced by analogy, and no more examples will be given here.

Step S42: Obtain the reference weight of the candidate voxel based on the deviation distance between the second voxel and the candidate voxel in the world coordinate system.

In some embodiments, the deviation distance is negatively correlated with the reference weight, that is, the greater the deviation distance, the smaller the reference weight, and the smaller the deviation distance, the greater the reference weight, so as much as possible can be referenced with the second volume The first voxel information of the candidate voxels that are closer to the second voxel, and the first voxel information of the candidate voxels that are farther away from the second voxel are referred to as little as possible, which is beneficial to improve the accuracy of the second voxel information.

In some embodiments, the world coordinate system is composed of the first coordinate axis, the second coordinate axis and the third coordinate axis, then the candidate voxel can be obtained based on the first deviation distance between the second voxel and the candidate voxel on the first coordinate axis The first weight, and based on the second deviation distance between the second voxel and the candidate voxel on the second coordinate axis, the second weight of the candidate voxel is obtained, and based on the distance between the second voxel and the candidate voxel on the third coordinate axis The third deviation distance obtains the third weight of the candidate voxel, and on this basis, the reference weight of the candidate voxel can be obtained based on the first weight, the second weight and the third weight. The above method can measure the weights in each direction of the world coordinate system, which is beneficial to improve the accuracy of the reference weights.

In some embodiments, the first weight is negatively correlated with the first offset distance, the second weight is negatively correlated with the second offset distance, and the third weight is negatively correlated with the third offset distance.

In some embodiments, after the first weight, the second weight and the third weight are obtained, the first weight, the second weight and the third weight may be multiplied to obtain the reference weight of the candidate voxel.

而言，其候選體素v的第一權重

、第二權重

、第三權重

可以分別表示為公式（7）、公式（8）、公式（9）：

……（7）

……（8）

……（9） In some embodiments, for ease of description, the first coordinate axis can be marked as x-axis, the second coordinate axis can be marked as y-axis, and the third coordinate axis can be marked as z-axis, then for the second voxel

In terms of, the first weight of its candidate voxel v

, the second weight

, the third weight

It can be expressed as formula (7), formula (8), formula (9):

... (7)

……(8)

……(9)

、

、

分別表示第二體素

分別在x軸、y軸、z軸上的座標值，

、

、

分別表示候選體素分別在在x軸、y軸、z軸上的座標值。此外，

表示第一體素距離。 In the above formulas (7), (8) and (9),

,

,

respectively represent the second voxel

The coordinate values on the x-axis, y-axis, and z-axis respectively,

,

,

represent the coordinate values of the candidate voxels on the x-axis, y-axis, and z-axis respectively. also,

Indicates the first voxel distance.

Step S43: Obtain second voxel information of the second voxel based on the first voxel information of the candidate voxel and the reference weight.

In some embodiments, the reference weights of each candidate voxel can be used to weight the first voxel information of the corresponding candidate voxel to obtain the second voxel information of the second voxel, which can be obtained through simple weighting calculation. Obtaining the second voxel information is beneficial to reduce the computational memory required for model reconstruction.

的第二截斷符號距離

可以表示為公式（10）：

……（10） In some embodiments, the second voxel information includes a second truncated signed distance, the first voxel information includes a first truncated signed distance, then the second voxel

The second truncated signed distance of

can be expressed as formula (10):

... (10)

表示第二體素

的候選體素集合，

表示候選體素的第一截斷符號距離。 In the above formula (10),

represents the second voxel

The set of candidate voxels,

Indicates the first truncated sign distance of candidate voxels.

的第二體素權重

可以表示為公式（11）：

……（11） In some embodiments, the second voxel information further includes a second voxel weight, the first voxel information further includes a first voxel weight, then the second voxel

The second voxel weight of

can be expressed as formula (11):

... (11)

表示第二體素

的候選體素集合，

表示候選體素的第一體素權重。需要說明的是，在採用第二體素的八個最接近的候選體素的情況下，可以採用上述公式（7）至公式（11），來計算該第二體素的第二體素資訊。其他情況可以以此類推，在此不再一一舉例。 In the above formula (11),

represents the second voxel

The set of candidate voxels,

Indicates the first voxel weight of the candidate voxel. It should be noted that, in the case of using the eight closest candidate voxels of the second voxel, the above formula (7) to formula (11) can be used to calculate the second voxel information of the second voxel . Other situations can be deduced by analogy, and no more examples will be given here.

In the above scheme, both the first voxel model and the second voxel model are located in the world coordinate system, by screening the first voxel as the candidate voxel of the second voxel, and based on the second voxel and the candidate voxel in the world coordinate system The reference weight of the candidate voxel is obtained based on the deviation distance in , and on this basis, based on the first voxel information and reference weight of the candidate voxel, the second voxel information of the second voxel is obtained, so that each candidate voxel can be obtained according to The deviation distances between the voxels and the second voxels refer to the first voxel information of each candidate voxel to varying degrees, and on the one hand, it can improve the accuracy of the second voxel information, which is conducive to improving the accuracy of the second voxel model. Accuracy, on the other hand, because the second voxel information can be obtained only by simple calculations, it is beneficial to reduce the computational memory for model reconstruction.

Please refer to FIG. 5 . FIG. 5 is a schematic flowchart of another embodiment of a model reconstruction method according to an embodiment of the present invention. The following steps may be included.

Step S51: Fusion the depth image of the target object with the reference voxel model to obtain a first voxel model of the target object.

In an embodiment of the present invention, adjacent first voxels in the first voxel model are separated by a first voxel distance, and at least some of the first voxels have first voxel information. Reference may be made to relevant descriptions in the aforementioned disclosed embodiments.

Step S52: Based on the first voxel with the first voxel information, detect whether the first voxel model meets the resampling condition, if yes, perform step S53, otherwise, perform step S56.

In some embodiments, the first voxel model includes a plurality of first regions, the first regions include a preset number of first voxels, and the resampling condition includes: the first region with reference voxels is more than a first threshold, and The reference voxel is the first voxel with the first voxel information, and reference can be made to related descriptions in the aforementioned disclosed embodiments.

In some embodiments, the resampling condition can be set to include: the first voxels with the first voxel information are more than the second threshold, since the first voxels with the first voxel information will occupy the computing memory, so Setting the resampling condition by measuring the first voxel with the first voxel information is beneficial to realize model reconstruction under the condition of limited computing memory. Reference may be made to relevant descriptions in the aforementioned disclosed embodiments.

Step S53: Resampling the first voxel model to obtain a second voxel model.

In the embodiment of the present invention, adjacent second voxels in the second voxel model are separated by a second voxel distance, the second voxel distance is greater than the first voxel distance, and at least some of the second voxels have second voxel information . You can refer to the relevant descriptions in the aforementioned disclosed embodiments

Step S54: using the second voxel model as a new reference voxel model, and acquiring a new depth image.

After resampling the first voxel model and obtaining the second voxel model, the second voxel model can be used as a new reference part, and a new depth image can be obtained for the next model reconstruction.

Step S55: Re-execute step S51 and subsequent steps.

After the second voxel model is used as a new reference voxel model and a new depth image is obtained, the above step S51 and subsequent steps can be re-executed to incorporate new depth information again.

Step S56: using the first voxel model as a new reference voxel model, and acquiring a new depth image.

If the first voxel model does not meet the resampling conditions, the first voxel model can be used as a new reference voxel model, and a new depth image can be obtained for the next model reconstruction.

Step S57: Re-execute step S51 and subsequent steps.

After using the first voxel model as a new reference voxel model and obtaining a new depth image, the above step S51 and subsequent steps can be re-executed to integrate new depth information again, so that depth information can be continuously integrated, Improve the accuracy and completeness of model reconstructions.

It should be noted that the embodiments of the present invention can be applied to model reconstruction during real-time scanning, and can also be applied to model reconstruction after scanning, which is not limited here, and reference can be made to relevant descriptions in the aforementioned disclosed embodiments.

Please refer to Fig. 6a-Fig. 6c in combination. Fig. 6a is a schematic diagram of the effect of the first voxel model in the model reconstruction method of the embodiment of the present invention, and Fig. 6b is a schematic diagram of the effect of the second voxel model in the model reconstruction method of the embodiment of the present invention, Fig. 6c is a schematic diagram of the final model effect in the model reconstruction method of the embodiment of the present invention, as shown in Figure 6a-Figure 6c, after the reconstruction model of each stage is obtained based on the truncated symbol distance, such as moving cube (Marching Cubes, MC) can be used to extract From the grid model, it can be seen that in the three stages of the first voxel model, the second voxel model after resampling, and the final model after reconstruction, the geometric shape of the grid model is basically the same, but compared with the first voxel model For the voxel model, the grid density of the second voxel model and the final model is reduced, so it is beneficial to ensure the integrity of the reconstruction and improve the time efficiency under the limited computing memory, so that the model reconstruction can be suitable for Mobile terminal applications such as mobile phones and tablets. In addition, after the final model is obtained, texture mapping and other processing can be performed on the final model to obtain a rendering model of the target object, which can be applied to augmented reality (Augmented Reality, AR), virtual reality (Virtual Reality) Reality, VR) and other scenarios, there is no limitation here.

In the above solution, the depth image of the target object is fused with the reference voxel model to obtain the first voxel model of the target object, and based on the first voxel with the first voxel information, it is detected whether the first voxel model conforms to the Sampling conditions, if so, resample the first voxel model to obtain the second voxel model, use the second voxel model as a new reference voxel model, and obtain a new depth image and re-execute the above fusion process , otherwise, the first voxel model is used as a new reference voxel model, and a new depth image is obtained and the above fusion process is performed again, so the voxel spacing can be self-adjusted during the reconstruction process, which is beneficial in the limited calculation memory while ensuring reconstruction integrity and improving time efficiency.

Please refer to FIG. 7 . FIG. 7 is a schematic block diagram of an embodiment of a model reconstruction device 70 according to an embodiment of the present invention. The model reconstruction device 70 includes: a depth fusion part 71 and a resampling part 72, the depth fusion part 71 is configured to fuse the depth image of the target object with the reference voxel model to obtain the first voxel model of the target object; , adjacent first voxels in the first voxel model are separated by a first voxel distance, and at least some of the first voxels have first voxel information; the resampling part 72 is configured to re-sample the first voxel model Sampling to obtain a second voxel model; wherein, adjacent second voxels in the second voxel model are separated by a second voxel distance, the second voxel distance is greater than the first voxel distance, and at least part of the second voxel Has second voxel information.

In the above solution, the depth image of the target object is fused with the reference voxel model to obtain the first voxel model of the target object, and the first voxel model is adjacent to the first voxel with a first voxel distance, and at least part of The first voxel has first voxel information, and then the first voxel model is resampled to obtain a second voxel model, and in the second voxel model, adjacent second voxels are separated by a second voxel distance, The second voxel distance is greater than the first voxel distance, and at least some of the second voxels have second voxel information. Since the second voxel distance is greater than the first voxel distance, on the one hand, the model can not be lost after resampling represents the range, and on the other hand, the second voxel model can be obtained by resampling directly on the basis of the first voxel model, without having to rebuild from scratch, and can also improve the time efficiency of model reconstruction, so it is possible to ensure that the reconstruction On the basis of completeness, the reconstruction time efficiency is improved. In addition, because the second voxel distance is greater than the first voxel distance, it can also reduce the number of voxels required to represent the target object, which can help reduce the computational memory required for model reconstruction. body while ensuring reconstruction integrity and improving time efficiency.

In some disclosed embodiments, both the first voxel model and the second voxel model are located in the world coordinate system, and the resampling part 72 includes a candidate voxel screening subsection configured to screen the first voxel as the second voxel Candidate voxel; the resampling part 72 includes a reference weight calculation subsection, configured to obtain the reference weight of the candidate voxel based on the deviation distance between the second voxel and the candidate voxel in the world coordinate system; the resampling part 72 includes a voxel The voxel information calculation subpart is configured to obtain second voxel information of the second voxel based on the first voxel information of the candidate voxel and the reference weight.

Therefore, on the one hand, the accuracy of the second voxel information can be improved, which is conducive to improving the accuracy of the second voxel model; Computational memory.

In some disclosed embodiments, the world coordinate system is composed of a first coordinate axis, a second coordinate axis and a third coordinate axis, and the reference weight calculation subpart includes a first weight calculation unit configured to The first deviation distance on the coordinate axis obtains the first weight of the candidate voxel, and the reference weight calculation subpart includes a second weight calculation unit configured to be based on the second voxel and the second weight of the candidate voxel on the second coordinate axis The deviation distance is used to obtain the second weight of the candidate voxel, and the reference weight calculation subpart includes a third weight calculation unit configured to obtain the candidate voxel based on the third deviation distance between the second voxel and the candidate voxel on the third coordinate axis The third weight of the voxel, the reference weight calculation subpart includes a reference weight calculation unit configured to obtain the reference weight of the candidate voxel based on the first weight, the second weight and the third weight.

Therefore, being able to measure the weights in each direction of the world coordinate system is beneficial to improve the accuracy of the reference weights.

In some disclosed embodiments, the deviation distance is negatively correlated with the reference weight; and/or, the second voxel information is obtained by weighting the reference weight of each candidate voxel to the first voxel information of the candidate voxel.

Therefore, the deviation distance is set to be negatively correlated with the reference weight, that is, the larger the deviation distance is, the smaller the reference weight is, and the smaller the deviation distance is, the greater the reference weight is, so it is possible to refer to as many candidates as possible that are closer to the second voxel. The first voxel information of the voxel, and refer to the first voxel information of the candidate voxel farther away from the second voxel as little as possible, which is beneficial to improve the accuracy of the second voxel information; and because the second voxel The voxel information is obtained by weighting the first voxel information of the candidate voxels with the reference weight of each candidate voxel, so the second voxel information can be obtained through simple weighting calculation, which is beneficial to reduce the computational memory required for model reconstruction body.

In some disclosed embodiments, the first voxel model includes a plurality of first regions, the second voxel model includes a plurality of second regions, and the first region contains a preset number of first voxels, and the second region contains a preset value a second voxel, the resampling part 72 includes a candidate area screening subsection configured to screen candidate areas in several first areas; wherein, at least one first voxel in the candidate area has first voxel information; resampling Part 72 includes a target area selection subsection configured to select a second area that at least partially overlaps with the candidate area as a target area corresponding to the candidate area; the resampling section 72 includes a second voxel selection subsection configured to select There are a preset number of second voxels in the target area, and the voxel screening subpart is configured to screen the first voxels for each selected second voxels to obtain candidate voxels.

Therefore, on the basis of dividing the first voxel model and the second voxel model into regions, the second region can be targetedly selected according to the region division, so that the second voxels within the preset threshold can be used as the second voxels to be determined later. The second voxel of the two-voxel information can eliminate the influence of other irrelevant second voxels on resampling, which is beneficial to speed up model reconstruction.

In some disclosed embodiments, the model reconstruction device 70 further includes a model detection part configured to detect whether the first voxel model meets the resampling condition based on the first voxel having the first voxel information; the model reconstruction device 70 also Including a first branch part and a first acquisition part, the first branch part is configured to perform resampling on the first voxel model in combination with the resampling part 72 in response to the first voxel model meeting the resampling condition, so as to obtain the second Steps for voxel models.

Therefore, through the first voxel having the first voxel information, it is detected whether the first voxel model meets the resampling condition, and in response to the first voxel model meeting the resampling condition, resampling the first voxel model is performed The step of obtaining the second voxel model is to detect whether the first voxel model meets the resampling condition before resampling, and the voxel spacing can be adjusted by itself.

In some disclosed embodiments, the first voxel model includes a plurality of first regions, the first regions include a preset number of first voxels, and the resampling condition includes: the number of first regions with reference voxels is greater than a first threshold, And the reference voxel is the first voxel with the first voxel information; or, the resampling condition includes: the first voxel with the first voxel information is more than the second threshold.

Therefore, the first voxel model includes several first regions, the first regions contain a preset number of first voxels, the resampling condition is set to have more first regions with reference voxels than the first threshold, and the reference voxels are the first voxel with the first voxel information, or the resampling condition is set to include more than the second threshold of the first voxel with the first voxel information, since the first voxel with the first voxel information will Occupies computing memory, so the resampling condition is set by measuring the first voxel with information of the first voxel, which is beneficial to realize model reconstruction under the condition of limited computing memory.

In some disclosed embodiments, the first branch part is further configured to use the second voxel model as a new reference voxel model in combination with the first acquisition part, and acquire a new depth image, and re-execute in combination with the depth fusion part 71 The step of fusing the depth image of the target object with the reference voxel model to obtain the first voxel model of the target object; and/or, the model reconstruction device 70 also includes a second branch part and a second acquisition part, the second branch The part is configured to combine the second acquisition part in response to the first voxel model not meeting the resampling condition, use the first voxel model as a new reference voxel model, and acquire a new depth image, and combine the depth fusion part 71 Re-executing the step of fusing the depth image of the target object with the reference voxel model to obtain the first voxel model of the target object.

Therefore, the second voxel model is used as a new reference voxel model, and a new depth image is obtained, and the depth image of the target object is fused with the reference voxel model to obtain the first voxel of the target object The steps of the model and subsequent steps can continuously incorporate in-depth information to improve the accuracy and completeness of model reconstruction. In response to the fact that the first voxel model does not meet the resampling conditions, the first voxel model is used as a new reference voxel model , and obtain a new depth image, and re-execute the steps of fusing the depth image of the target object with the reference voxel model to obtain the first voxel model of the target object and the follow-up, so it can continuously integrate depth information and improve the model Accuracy and completeness of reconstruction.

In some disclosed embodiments, both the first voxel model and the reference voxel model are located in the world coordinate system, adjacent voxels in the reference voxel model are separated by a first voxel distance, and the first voxel information includes the first voxel The first truncated symbol distance and the first voxel weight, the depth fusion part 71 includes a voxel screening subsection to be fused, which is configured to back-project the primitive point in the depth image to the projection point of the world coordinate system, in the reference Select voxel in the voxel model as the voxel to be fused; the depth fusion part 71 includes a truncated sign distance acquisition subsection to be fused, which is configured to obtain the voxel to be fused based on the internal reference of the camera and the camera pose when shooting the depth image. The truncated symbol distance to be fused; the depth fusion part 71 includes a first truncated symbol distance acquisition subsection, which is configured to combine the truncated symbol distance to be fused and the voxel to be fused in the reference voxel model based on the weight of the voxel to be fused. In the reference voxel model, the reference truncated symbol distance is fused to obtain the first truncated symbol distance of the first voxel corresponding to the position of the voxel to be fused; the depth fusion part 71 includes a first voxel weight acquisition subsection, which is configured to The voxel weight of the voxel to be fused is updated in the reference voxel model to obtain the first voxel weight of the first voxel corresponding to the position of the voxel to be fused.

Therefore, both the first voxel model and the reference voxel model are located in the world coordinate system, adjacent voxels in the reference voxel model are separated by the first voxel distance, and the first voxel information includes the first truncated sign of the first voxel The distance and the weight of the first voxel are based on the back-projection of the primitive points in the depth image to the projection points of the world coordinate system, and the voxels are selected in the reference voxel model as the voxels to be fused, and then based on the camera internal reference and the shooting depth map The camera pose at the time of imaging, obtain the truncated sign distance of the voxel to be fused, and based on the voxel weight of the voxel to be fused in the reference voxel model, the truncated sign distance to be fused and the voxel to be fused in the reference volume In the voxel model, the reference truncated symbol distance is fused to obtain the first truncated symbol distance of the first voxel corresponding to the position of the voxel to be fused, and the voxel weight of the voxel to be fused is updated in the reference voxel model to obtain the same as The first voxel weight of the first voxel corresponding to the voxel position to be fused, so in the process of model reconstruction, depth information can be continuously integrated, which is conducive to continuously improving the accuracy and integrity of the model.

In some disclosed embodiments, the acquisition subsection of the truncated symbol distance to be fused includes a reprojection unit configured to reproject the voxels to be fused based on the camera internal reference and the camera pose, to obtain the voxels to be fused in the depth image The first depth and the second depth in the camera coordinate system; the truncated symbol distance to be fused acquisition subpart includes a truncated symbol distance calculation unit configured to obtain the truncated symbol distance to be fused based on the deviation between the first depth and the second depth .

Therefore, it can accurately represent the directional distance from the voxel to be fused to the surface of the target object according to the depth information provided by the depth image, which is beneficial to improve the accuracy of the truncated symbol distance to be fused.

In some disclosed embodiments, the first voxel weight of the first voxel corresponding to the position of the voxel to be fused is greater than the voxel weight of the voxel to be fused in the reference voxel model; and/or, when the voxel to be fused is in If there is no reference truncated signed distance in the reference voxel model, the to-be-fused truncated signed distance of the to-be-fused voxel is used as the first truncated signed distance of the first voxel corresponding to the position of the to-be-fused voxel.

Therefore, setting the first voxel weight of the first voxel corresponding to the position of the voxel to be fused to be greater than the voxel weight of the voxel to be fused in the reference voxel model can gradually focus on the reference voxel during the model reconstruction process. The voxels to be fused refer to the truncated signed distance in the reference voxel model, which is beneficial to improve the accuracy of model reconstruction; and when the voxels to be fused do not have the reference truncated signed distance in the reference voxel model, the fused voxel The truncated signed distance of the voxel to be fused, as the first truncated signed distance of the first voxel corresponding to the position of the voxel to be fused, can be in the case that the voxel to be fused does not have a reference truncated signed distance in the reference voxel model, The acquisition process of the first truncated sign distance is greatly simplified, which is beneficial to reduce the computational memory required for model reconstruction.

Please refer to FIG. 8 . FIG. 8 is a schematic block diagram of an embodiment of an electronic device 80 according to an embodiment of the present invention. The electronic device 80 includes a memory 81 and a processor 82 coupled to each other, and the processor 82 is configured to execute the program instructions stored in the memory 81 to implement the steps of any one of the model reconstruction method embodiments above. In some embodiments, the electronic device 80 may include, but is not limited to: a microcomputer, a server. In addition, the electronic device 80 may also include mobile devices such as notebook computers and tablet computers, which are not limited here. It should be noted that since the technical solution provided by any of the above-mentioned embodiments of the model reconstruction method can ensure the integrity of the reconstruction and improve the time efficiency under the condition of limited computing memory, the electronic device 80 can include mobile phones, tablet Computers and other mobile terminals.

The processor 82 is configured to control itself and the memory 81 to implement the steps in any of the above embodiments of the model reconstruction method. The processor 82 may also be called a central processing unit (Central Processing Unit, CPU). The processor 82 may be an integrated circuit chip with signal processing capability. The processor 82 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) Or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like. In addition, the processor 82 may be collectively realized by an integrated circuit chip.

The above solution, on the one hand, can not lose the representation range of the model after resampling, and on the other hand, the second voxel model can be obtained by directly resampling on the basis of the first voxel model, without having to rebuild from scratch , can also improve the time efficiency of model reconstruction, so it can improve the time efficiency of reconstruction on the basis of ensuring the integrity of reconstruction. In addition, because the second voxel distance is greater than the first voxel distance, it can also reduce the number of voxels required to represent the target object, which can help reduce the computational memory required for model reconstruction. body while ensuring reconstruction integrity and improving time efficiency.

Please refer to FIG. 9 . FIG. 9 is a schematic block diagram of an embodiment of a computer-readable storage medium 90 according to an embodiment of the present invention. The computer-readable storage medium 90 stores program instructions 91 that can be executed by the processor, and the program instructions 91 are used to implement the steps of any one of the above-mentioned model reconstruction method embodiments.

The embodiment of the present invention also provides a computer program product, the computer program product includes a computer program or instruction, and when the computer program or instruction is run on the computer, the computer executes any model reconstruction in the above method embodiment Steps of a method embodiment.

The above solution, on the one hand, can not lose the representation range of the model after resampling, and on the other hand, the second voxel model can be obtained by directly resampling on the basis of the first voxel model, without having to rebuild from scratch , can also improve the time efficiency of model reconstruction, so it can improve the time efficiency of reconstruction on the basis of ensuring the integrity of reconstruction. In addition, because the second voxel distance is greater than the first voxel distance, it can also reduce the number of voxels required to represent the target object, which can help reduce the computational memory required for model reconstruction. body while ensuring reconstruction integrity and improving time efficiency.

The present invention relates to the field of augmented reality, by acquiring the image information of the target object in the real environment, and then using various visual correlation algorithms to detect or identify the relevant characteristics, states and attributes of the target object, so as to obtain the Apply matching virtual and reality combined AR effects. Exemplarily, the target object may involve faces, limbs, gestures, actions, etc. related to the human body, or markers and markers related to objects, or sand tables, display areas or display items related to venues or places. Vision-related algorithms may involve visual positioning, simultaneous localization and mapping (SLAM), 3D reconstruction, image registration, background segmentation, object key point extraction and tracking, object pose or depth detection, etc. Applications can not only involve interactive scenes such as tours, navigation, explanations, reconstructions, virtual effect overlays and display related to real scenes or objects, but can also involve special effects processing related to people, such as makeup beautification, body beautification, special effect display, virtual model Display and other interactive scenes.

The relevant characteristics, states and attributes of the target object can be detected or identified through the convolutional neural network. The above-mentioned convolutional neural network is a network model obtained through model training based on deep learning blocks.

In the several embodiments provided by the present invention, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementations described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, units or components can be combined or integrated. to another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

A unit described as a separate component may or may not be physically separated, and a component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or may also be distributed to network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the embodiments of the present invention may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented not only in the form of hardware, but also in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of software products, and the computer software products are stored in a storage medium Among them, several instructions are included to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) execute all or part of the steps of the methods in various embodiments of the present invention.

The aforementioned storage medium may be a tangible device capable of holding and storing instructions used by the instruction execution device, and may be a volatile storage medium or a non-volatile storage medium. The computer-readable storage medium may be, for example but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, Random Access Memory (RAM), Read Only Memory (RAM), ROM), Erasable Programmable Read Only Memory (Erasable Programmable Read Only Memory, EPROM or flash memory), Static Random-Access Memory (Static Random-Access Memory, SRAM), portable compressed disk Compact Disk Read Only Memory (CD-ROM), Digital versatile Disc (DVD), memory sticks, floppy disks, mechanically encoded devices such as punched cards or embossed cards with instructions stored on them The protrusion structure in the groove, and any suitable combination of the above. A computer-readable storage medium as used herein is not to be construed as a transient signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other Electrical signals transmitted by wires.

Industrial Applicability In the embodiment of the present invention, the depth image of the target object is fused with the reference voxel model to obtain the first voxel model of the target object, and the first voxel model is adjacent to the first voxel with a first voxel distance, and at least Part of the first voxels have the first voxel information, and then resample the first voxel model to obtain the second voxel model, and the second voxel distance between adjacent second voxels in the second voxel model , the second voxel distance is greater than the first voxel distance, and at least some of the second voxels have second voxel information. Since the second voxel distance is greater than the first voxel distance, on the one hand, the model can not be lost after resampling On the other hand, the second voxel model can be obtained by resampling directly on the basis of the first voxel model without rebuilding from scratch, and the time efficiency of model reconstruction can also be improved, so it can be ensured On the basis of rebuilding integrity, the efficiency of rebuilding time is improved. In addition, because the second voxel distance is greater than the first voxel distance, it can also reduce the number of voxels required to represent the target object, which can help reduce the computational memory required for model reconstruction. body while ensuring reconstruction integrity and improving time efficiency.

70:Model reconstruction device 71: Deep fusion part 72: Resampling part 80: Electronic equipment 81: memory 82: Processor 90: computer readable storage medium 91: Program instruction S11~S12, S41~S43, S51~S57: steps

Fig. 1 is a schematic flow chart of an embodiment of a model reconstruction method according to an embodiment of the present invention; Fig. 2 is a schematic diagram of an embodiment of truncated symbol distance; Fig. 3 is a schematic diagram of an embodiment of a first voxel model and a second voxel model; Fig. 4 is a schematic flow chart of an embodiment of step S12 in Fig. 1; Fig. 5 is a schematic flowchart of another embodiment of the model reconstruction method of the embodiment of the present invention; Fig. 6a is a schematic diagram of the effect of the first voxel model in the model reconstruction method of the embodiment of the present invention; Fig. 6b is a schematic diagram of the effect of the second voxel model in the model reconstruction method of the embodiment of the present invention; Fig. 6c is a schematic diagram of the final model effect in the model reconstruction method of the embodiment of the present invention; 7 is a schematic block diagram of an embodiment of a model reconstruction device according to an embodiment of the present invention; FIG. 8 is a schematic block diagram of an embodiment of an electronic device according to an embodiment of the present invention; FIG. 9 is a schematic block diagram of an embodiment of a computer-readable storage medium according to an embodiment of the present invention.

S11~S12: Steps

Claims (15)

A method for model reconstruction, comprising: Fusing the depth image of the target object with the reference voxel model to obtain a first voxel model of the target object; wherein, in the first voxel model, adjacent first voxels are separated by a first voxel distance, and at least some of the first voxels have first voxel information; Resampling the first voxel model to obtain a second voxel model; wherein, in the second voxel model, adjacent second voxels are separated by a second voxel distance, and the second voxel distance is greater than The first voxel distance, and at least some of the second voxels have second voxel information. According to the method described in claim 1, wherein, both the first voxel model and the second voxel model are located in the world coordinate system, and the second voxel model is obtained by resampling the first voxel model prime models, including: screening the first voxel as a candidate voxel for the second voxel; Obtaining a reference weight of the candidate voxel based on the deviation distance between the second voxel and the candidate voxel in the world coordinate system; Based on the first voxel information and the reference weight of the candidate voxels, second voxel information of the second voxels is obtained. The method according to claim 2, wherein the world coordinate system is composed of a first coordinate axis, a second coordinate axis and a third coordinate axis, and the world coordinate system based on the second voxel and the candidate voxel The deviation distance in the system is obtained to obtain the reference weight of the candidate voxel, including: Based on the first offset distance between the second voxel and the candidate voxel on the first coordinate axis, obtain the first weight of the candidate voxel, and based on the second voxel and the candidate voxel The second deviation distance of the voxel on the second coordinate axis to obtain the second weight of the candidate voxel, and the third deviation on the third coordinate axis based on the second voxel and the candidate voxel distance to obtain the third weight of the candidate voxel; The reference weight of the candidate voxel is obtained based on the first weight, the second weight, and the third weight. The method according to claim 2, wherein the deviation distance is negatively correlated with the reference weight. The method according to claim 2 or 4, wherein the second voxel information is obtained by weighting the first voxel information of the candidate voxels based on the reference weights of each of the candidate voxels. According to the method described in claim 2, wherein, the first voxel model includes several first regions, the second voxel model includes several second regions, and the first region contains a preset number of the The first voxel, the second region contains the preset number of the second voxels, and before the screening of the first guaranteed voxels as the candidate voxels of the second voxels, the Methods also include: Screening candidate regions in the plurality of first regions; wherein at least one of the first voxels in the candidate regions has the first voxel information; selecting a second area that at least partially overlaps with the candidate area as a target area corresponding to the candidate area; selecting the preset number of second voxels within the target area; The screening of the first voxel as the candidate voxel of the second voxel includes: Screening the first voxels for each of the selected second voxels as the candidate voxels respectively. The method according to any one of claims 1 to 4, wherein the method further comprises: Detecting whether the first voxel model meets a resampling condition based on the first voxel having the first voxel information; In response to the first voxel model meeting the resampling condition, the step of resampling the first voxel model to obtain a second voxel model is performed. According to the method described in claim 7, wherein, the first voxel model includes several first regions, and the first regions include a preset number of the first voxels, and the resampling condition includes: having a reference a number of first regions of voxels is greater than a first threshold, and the reference voxel is a first voxel having the first voxel information; Alternatively, the resampling condition includes: the number of first voxels with the first voxel information is greater than a second threshold. The method according to claim 7, wherein the method further includes at least one of the following: Using the second voxel model as a new reference voxel model, and acquiring a new depth image, and re-executing the fusion of the depth image of the target object and the reference voxel model to obtain the target object's Steps for the first voxel model and subsequent steps; Responding to the fact that the first voxel model does not meet the resampling condition, using the first voxel model as a new reference voxel model, and acquiring a new depth image, and re-executing the resampling of the target object. The step of fusing the depth image and the reference voxel model to obtain the first voxel model of the target object and subsequent steps. According to the method described in any one of claim items 1 to 4, wherein, the first voxel model and the reference voxel model are both located in the world coordinate system, and the interval between adjacent voxels in the reference voxel model is The first voxel distance, and the first voxel information includes the first truncated symbol distance and the first voxel weight of the first voxel, and the depth image of the target object is fused with the reference voxel model to obtain The first voxel model of the target object, comprising: Selecting the voxel in the reference voxel model as the voxel to be fused based on the back-projection of the primitive point in the depth image to the projection point of the world coordinate system; Obtaining the truncated sign distance to be fused of the voxel to be fused based on the camera internal reference and the camera pose when the depth image is taken; Based on the voxel weight of the voxel to be fused in the reference voxel model, the truncated signed distance to be fused and the reference truncated signed distance of the voxel to be fused in the reference voxel model are fused to obtain a first truncated signed distance of a first voxel corresponding to the voxel position to be fused; and, Updating the voxel weight of the voxel to be fused in the reference voxel model to obtain the first voxel weight of the first voxel corresponding to the position of the voxel to be fused. According to the method described in claim 10, wherein the acquisition of the truncated sign distance to be fused of the voxels to be fused based on the camera internal reference and the camera pose when the depth image is taken includes: Based on the camera internal reference and the camera pose, reproject the voxel to be fused to obtain a first depth of the voxel to be fused in the depth image and a second depth in the camera coordinate system ; Based on the deviation between the first depth and the second depth, the truncated symbol distance to be fused is obtained. The method according to claim 10, wherein the first voxel weight of the first voxel corresponding to the position of the voxel to be fused is greater than the voxel weight of the voxel to be fused in the reference voxel model. According to the method described in claim 10, wherein, in the case that the voxel to be fused does not have the reference truncated sign distance in the reference voxel model, the truncated sign to be fused of the voxel to be fused is distance, as the first truncated signed distance of the first voxel corresponding to the position of the voxel to be fused. An electronic device, comprising a memory and a processor coupled to each other, the processor is used to execute program instructions stored in the memory to implement the model reconstruction method described in any one of claims 1 to 13. A computer-readable storage medium, on which program instructions are stored, and when the program instructions are executed by a processor, the model reconstruction method described in any one of claims 1 to 13 is realized.

Images