WO2024002064A1 - Method and apparatus for constructing three-dimensional model, and electronic device and storage medium - Google Patents

Abstract

The present application belongs to the technical field of images. Disclosed are a method and apparatus for constructing a three-dimensional model, and an electronic device and a storage medium. The method comprises: determining a mutation area of a target image; performing non-uniform sampling on the target image on the basis of the mutation area of the target image and a sampling step length updating strategy corresponding to the mutation area so as to obtain a set of non-uniform sampling vertexes; and constructing a three-dimensional model on the basis of the set of non-uniform sampling vertexes.

Description

Three-dimensional model construction method, device, electronic equipment and storage medium

Cross-references to related applications

This application claims priority to the Chinese patent with application number 202210774950.4 filed in China on July 1, 2022, the entire content of which is incorporated herein by reference.

Technical field

This application belongs to the field of image technology, and specifically relates to a three-dimensional model construction method, device, electronic equipment and storage medium.

Background technique

In short video services and 3D scene mobile games, in order to present the 3D parallax effect of a scene or object during movement, 3D reconstruction technology needs to be applied to restore the 2D image to a 3D image with 3D physical structure and texture information in the 3D space. The scene is then rendered in 3D. Currently, the existing method is to use uniform sampling of the depth map to reconstruct the 3D model structure and texture, that is, setting a fixed sampling step so that the sampling points (i.e., grid vertices) are evenly distributed on the depth map.

However, in this method, since a fixed sampling step is set, during sampling, if the sampling step is small, the grid vertices will be dense, and the number of model vertices and faces will be large, which will affect the rendering of electronic devices. Performance, if the sampling step size is large, the number of mesh vertices and faces will be small, making it difficult to reflect the detailed structure of the model, and the rendering effect will be difficult to guarantee. In this way, the 3D reconstruction method using uniform sampling of the depth map cannot guarantee the efficiency of 3D model reconstruction and the rendering effect of the 3D model.

Contents of the invention

The purpose of the embodiments of the present application is to provide a three-dimensional model construction method, device, electronic equipment and storage medium that can solve the problem that the three-dimensional reconstruction method using uniform sampling of the depth map cannot guarantee the efficiency of the three-dimensional model reconstruction and the rendering effect of the three-dimensional model. question.

In order to solve the above technical problems, this application is implemented as follows:

In the first aspect, embodiments of the present application provide a three-dimensional model construction method, which method includes: determining a mutation area of the target image; based on the mutation area of the target image and the sampling step update strategy corresponding to the mutation area, the target image is Non-uniform sampling is performed to obtain a non-uniform sampling vertex set; based on the non-uniform sampling vertex set, a three-dimensional model is constructed.

In the second aspect, embodiments of the present application provide a three-dimensional model construction device, which includes: a determination module, a sampling module, and a construction module. The determination module is used to determine the mutation area of the target image. The sampling module is used to non-uniformly sample the target image based on the mutation area of the target image determined by the determination module and the sampling step update strategy corresponding to the mutation area, and obtain a non-uniform sampling vertex set. Building blocks for procurement-based The non-uniform sampling vertex set obtained by sampling module is used to construct a three-dimensional model.

In a third aspect, embodiments of the present application provide an electronic device. The electronic device includes a processor and a memory. The memory stores programs or instructions that can be run on the processor. The programs or instructions are processed by the processor. When the processor is executed, the steps of the method described in the first aspect are implemented.

In a fourth aspect, embodiments of the present application provide a readable storage medium. Programs or instructions are stored on the readable storage medium. When the programs or instructions are executed by a processor, the steps of the method described in the first aspect are implemented. .

In a fifth aspect, embodiments of the present application provide a chip. The chip includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the first aspect. the method described.

In a sixth aspect, embodiments of the present application provide a computer program product, the program product is stored in a storage medium, and the program product is executed by at least one processor to implement the method as described in the first aspect.

In the embodiment of the present application, the electronic device can use the sampling step update strategy to perform non-uniform sampling based on the mutation area of the target image to obtain a non-uniform sampling vertex set, thereby constructing a three-dimensional model based on the set. In this solution, the electronic device can determine the corresponding sampling step update strategy based on the mutation area of the target image. That is, during the sampling process, based on the mutation area of the target image, the distribution characteristics of the three-dimensional space points are calculated to update the sampling step size. Adaptively samples key vertices in the depth map to construct a non-uniform vertex space topology, so that the 3D model generates a denser grid structure in the edge area of the object, ensuring the authenticity of the model rendering effect, while generating flat areas in space A sparser mesh structure is used to reduce the number of model vertices and sides, improve the rendering performance of the mesh model in electronic devices, and ultimately obtain a 3D model that can achieve both the authenticity of the 3D model rendering effect and the efficiency of the rendering performance of the electronic device. .

Description of drawings

Figure 1 is a schematic diagram of an example of depth map vertex sampling provided by an embodiment of the present application;

Figure 2 is one of the flow charts of a three-dimensional model construction method provided by an embodiment of the present application;

Figure 3 is the second flow chart of a three-dimensional model construction method provided by an embodiment of the present application;

Figure 4 is the third flow chart of a three-dimensional model construction method provided by an embodiment of the present application;

Figure 5 is one of the schematic diagrams of an example of a mutation mask value provided by the embodiment of the present application;

Figure 6 is a second schematic diagram of an example of a mutation mask value provided by an embodiment of the present application;

Figure 7 is a schematic structural diagram of a three-dimensional model building device provided by an embodiment of the present application;

Figure 8 is one of the schematic diagrams of the hardware structure of an electronic device provided by an embodiment of the present application;

FIG. 9 is a second schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts, All fall within the scope of protection of this application.

The terms "first", "second", etc. in the description and claims of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. It is to be understood that the figures so used are interchangeable under appropriate circumstances so that the embodiments of the present application can be practiced in orders other than those illustrated or described herein, and that "first," "second," etc. are distinguished Objects are usually of one type, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "and/or" in the description and claims indicates at least one of the connected objects, and the character "/" generally indicates that the related objects are in an "or" relationship.

The three-dimensional model construction method provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings through specific embodiments and application scenarios.

With the development of electronic devices such as mobile phones and personal computers, scenes or objects with three-dimensional visual effects appearing in videos or games are increasingly used. Three-dimensional reconstruction technology can restore two-dimensional images to scenes with three-dimensional physical structure and texture information in three-dimensional space, and then perform three-dimensional rendering. In the existing technology, it is often difficult to balance the rendering performance of the mobile device and the sophistication of the reconstruction effect with the realism of the rendering result by using uniform sampling of the depth map (setting a fixed sampling step) for 3D model structure and texture reconstruction. .

When using uniform sampling of the depth map to reconstruct the 3D model structure and texture, a fixed sampling step is usually set with reference to the direction of pixel arrangement so that the sampling points (i.e., grid vertices) are evenly distributed on the depth map. When the sampling step size is large, as shown in (A) in Figure 1, the number of vertices and sides of the mesh model can be controlled to a smaller order of magnitude, and the device rendering performance is good, but it is difficult to reflect the model with fewer vertices and sides. The detailed structure and rendering effect are difficult to guarantee, and the edges of the character's arms appear distorted and meshed. On the contrary, when the sampling step size is small, as shown in (B) in Figure 1, the grid vertices are denser, the surface is more refined, the grid model rendering effect is better, and the character texture details are clear and realistic, but the number of model vertices and A large number of partials will seriously affect the rendering performance of mobile devices and make it difficult to meet the needs of real-time rendering.

This application proposes a three-dimensional model construction method. As shown in (C) in Figure 1, a denser grid structure is generated in the edge area of the object to ensure the authenticity of the model rendering effect, while the flat area in the space is generated more densely. Sparse mesh structure to reduce the number of model vertices and sides and improve the rendering performance of mesh models on mobile devices.

In the embodiment of the present application, the electronic device can use the sampling step update strategy to perform non-uniform sampling based on the mutation area of the target image to obtain a non-uniform sampling vertex set, thereby constructing a three-dimensional model based on the set. In this solution, the electronic device can determine the corresponding sampling step update strategy based on the mutation area of the target image. That is, during the sampling process, based on the mutation area of the target image, the distribution characteristics of the three-dimensional space points are calculated to update the sampling step size. Adaptively samples key vertices in the depth map to construct a non-uniform vertex space topology, so that the 3D model generates a denser grid structure in the edge area of the object, ensuring the authenticity of the model rendering effect, while generating flat areas in space A sparser mesh structure is used to reduce the number of model vertices and sides, improve the rendering performance of the mesh model in electronic devices, and ultimately obtain a 3D model that can achieve both the authenticity of the 3D model rendering effect and the efficiency of the rendering performance of the electronic device. .

It should be noted that the depth map can be understood as: each pixel value in the image represents the distance between the corresponding spatial point in the scene and the subject.

Three-dimensional reconstruction can be understood as: reconstructing three-dimensional images through discrete data such as multi-view images, point cloud data, or depth maps. An object model or scene made up of grids in space.

An embodiment of the present application provides a three-dimensional model construction method. FIG. 2 shows a flow chart of a three-dimensional model construction method provided by an embodiment of the present application. As shown in Figure 2, the three-dimensional model construction method provided by the embodiment of the present application may include the following steps 201 to 203.

Step 201: The electronic device determines the mutation area of the target image.

In the embodiment of the present application, the electronic device can acquire the target image and perform image recognition processing on the target image to determine the mutation area of the target image. The mutation area can be used to indicate the pixels in the target image (ie, the depth map of the target image). distribution of spatial points), so that the electronic device can perform non-uniform sampling of the depth map based on the mutation area.

In the embodiment of the present application, the mutation area of the target image can be represented by a mutation mask value, that is, the mutation mask value is used to indicate the mutation area of the depth map of the target image (which can also be called a mutation zone area).

It should be noted that the above mutation zone area can be understood as: if the difference in depth information between adjacent pixels in the depth map is greater than or equal to a threshold, then the image area where these adjacent pixels are located is called a mutation zone. area.

Optionally, in this embodiment of the present application, the above-mentioned target image may be an image that the user can select from an electronic device (such as a photo album application of the electronic device); or the above-mentioned target image may be captured by the electronic device through a camera of the electronic device. image; alternatively, the above-mentioned target image may be an image received by the electronic device from other electronic devices.

Optionally, in this embodiment of the present application, the electronic device can determine the depth map of the target image by performing image recognition processing on the target image, and determine the mutation mask value area (ie, mutation area) of the depth map. Specifically, with reference to Figure 2, as shown in Figure 3, the above step 201 can be implemented through the following steps 201a to 201c.

Step 201a: The electronic device uses a single-frame depth estimation network to determine the depth map of the target image.

Optionally, in the embodiment of the present application, the electronic device performs image segmentation on the target image through a single-frame depth estimation network to obtain multiple image blocks; the electronic device selects absolute depth features and relative depth features respectively, and estimates each corresponding depth feature. The absolute depth of each image block and the estimated relative depth of adjacent blocks (ie, the depth difference); the electronic device builds a back-end solution model, and establishes the correlation between local features and depth through the back-end model, as well as the relationship between different image blocks. Correlation between depths; then the electronic device trains multiple image blocks through the back-end solution model to obtain the model predicted depths of multiple image blocks, thereby obtaining the depth map of the target image.

Optionally, in this embodiment of the present application, the electronic device may also use other image depth estimation methods such as single-eye or multi-eye image depth estimation to determine the depth map of the target image.

Step 201b: The electronic device obtains the depth information of each pixel of the depth map and determines at least one depth difference value.

In the embodiment of the present application, each of the above-mentioned at least one depth difference value is the difference between the depth information of a pixel in the depth map and the depth information of a pixel in a pixel area, and the pixel area includes All pixels adjacent to a pixel in the depth map.

In the embodiment of the present application, the electronic device can, for each pixel of the depth map, calculate the difference between the depth information of each pixel and the depth information of each pixel in each pixel area, that is, calculate each pixel. The difference between the depth information of each pixel and the depth information of all pixels adjacent to each pixel.

Specifically, for any pixel point p _i,j in the depth map D, where i is the number of rows where the pixel point is located, and j is the number of rows where the pixel point is located. In the number of columns, the pixel area B(p _i,j ) corresponding to any pixel point p _i ,j = {q _mn |i-1≤m≤i+1,j-1≤n≤j+1}; The electronic device separately calculates the depth difference (ie, the difference between depth information) of the pixel point p _i,j and each pixel point in the pixel area B(pi _,j ).

It should be noted that all pixels adjacent to a pixel in the depth map may include: adjacent pixels in the same row or column as the pixel, and pixels on a diagonal as the pixel. adjacent pixels.

For example, assume that the depth map includes 8*8 pixels. The electronic device can calculate the difference between the depth information of the first pixel (i.e. pixel p _1,1 ) and the depth information of each pixel in the pixel area B (p _1,1 ) corresponding to the pixel p _1,1 , the pixels included in the pixel area B(p _1,1 ) are pixel point p _1,2 , pixel point p _2,1 , pixel point p _2,2 , that is, the electronic device can calculate the depth of pixel point p _1,1 The difference between the information and the depth information of pixel point p _1,2 , the difference between the depth information of pixel point p _1,1 and the depth information of pixel point p _2,1 , the difference between the depth information of pixel point p 1,1 and the depth information of pixel point p _1,1 The difference between the depth information of p _2,2 ; the electronic device can calculate the depth information of the second pixel point (i.e. pixel point p _1,2 ) and the pixel area B(p _1,2 ) corresponding to the pixel point p _{1,2 The} difference _between the depth information _{of each} pixel in p _2,2 , pixel point p _2,3 , that is, the electronic device can calculate the difference between the depth information of pixel point p _1,2 and the depth information of pixel point p _1,1 , and the difference between the depth information of pixel point p _1,2 and The difference between the depth information of pixel point p _1,3 , the difference between the depth information of pixel point p _1,2 and the depth information of pixel point p _2,1 , the difference between the depth information of pixel point p _1,2 and pixel point p _{2 , the difference between the depth information of 2} , the difference between the depth information of pixel point p _1,2 and the depth information of pixel point p _2,3 ; and so on, until the depth information and pixel area of all pixels in the depth map are calculated The difference between the depth information of the pixels.

Step 201c: The electronic device determines the mutation area of the target image based on at least one depth difference value.

In the embodiment of the present application, each pixel in the depth map corresponds to one or more depth difference values, that is, each depth difference value can represent the depth between each pixel and the pixels adjacent to each pixel. The electronic device can determine the mutation mask value of each pixel (i.e., the following first value, second value) based on the depth difference between each pixel and all pixels adjacent to the pixel. value), thereby determining the mutation area of the target image based on the mutation mask value of each pixel.

Optionally, in this embodiment of the present application, the above step 201c can be specifically implemented through the following steps 201c11 and 201c12.

Step 201c11: The electronic device determines the depth mutation mask value of a pixel corresponding to a depth difference value greater than or equal to the first threshold in at least one depth difference value as a first value.

It can be understood that the electronic device can determine the size relationship between each depth difference value in the at least one depth difference value and the first threshold, and determine the pixel corresponding to each depth difference value based on the size relationship between each depth difference value and the first threshold value. The depth mutation mask value of the point.

Step 201c12: The electronic device determines the depth mutation mask value of a pixel corresponding to a depth difference value smaller than the first threshold in at least one depth difference value as a second value.

In the embodiment of the present application, the second numerical value is smaller than the first numerical value. Wherein, the mutation area of the target image is an image area composed of pixels corresponding to the first numerical value.

Specifically, with reference to Figure 3, as shown in Figure 4, before the above step 201c, the three-dimensional model provided by the embodiment of the present application The model construction method also includes the following step 204, and the above step 201c can be specifically implemented through the following step 201c1 or step 201c2.

Step 204: For each pixel of the depth map, the electronic device determines whether each of the depth differences corresponding to a pixel is greater than or equal to the first threshold.

In the embodiment of the present application, the above-mentioned one pixel is any pixel in the depth map.

Step 201c1: When at least one of the depth differences corresponding to a pixel is greater than or equal to the first threshold, the electronic device determines that the mutation mask value of the one pixel is the first value.

In the embodiment of the present application, if the depth difference of the pixel point p _i,j in any direction is greater than or equal to the first threshold, the mutation mask value M(p _i,j ) of the pixel point p _i ,j is the first A numerical value (e.g. 1). Specifically, the calculation formula is as shown in Formula 1:

It should be noted that the depth difference of pixel point p _i,j in any direction can be understood as: pixel point p _i,j and all pixels in the pixel area B(p _i,j ) corresponding to pixel point p _i,j Any depth difference among the depth differences of pixels.

Optionally, in the embodiment of the present application, the above-mentioned first threshold may be the default of the electronic device system or preset by the user. For example, the first threshold may be 0.05; the above-mentioned first value may be the default of the electronic device system or preset by the user. , for example, the first value can be expressed in the form of 1 or 01.

For example, assuming that the difference between the depth information of pixel point p _1,1 and the depth information of pixel point p _1,2 is a, the difference between the depth information of pixel point p _1,1 and the depth information of pixel point p _2,1 The difference is b, and the difference between the depth information of pixel point p _1,1 and the depth information of pixel point p _2,2 is c. If the difference a is less than the first threshold, the difference b is greater than the first threshold, and the difference c is greater than the first threshold, then the electronic device can determine that the mutation mask value of the pixel point p _1,1 is the first value.

Step 201c2: When all the depth differences in the depth differences corresponding to one pixel are less than the first threshold, the electronic device determines the mutation mask value of the one pixel as the second value to obtain the mutation of the target image. area.

Optionally, in this embodiment of the present application, the above-mentioned second numerical value may be defaulted by the electronic device system or preset by the user. For example, the second numerical value may be expressed in the form of 0 or 00.

It should be noted that for each pixel of the depth map, the electronic device can determine the mutation mask value of each pixel by performing the above steps 204, 201c1, and 201c2. That is, the electronic device can traverse the depth map. For all pixels, the mutation area of the target image is obtained.

For example, as shown in Figure 5, the electronic device can traverse all pixels of the depth map of the target image to determine the mutation mask values of all pixels, thereby obtaining the mutation mask value of the target image. In Figure 5, The black line illustrates the pixels whose mutation mask value is the first value, and the mutation mask value of the pixels in other image areas (ie, the white filled area) is the second value.

In the embodiment of the present application, the electronic device can determine the mutation mask value of all pixels of the depth map, thereby obtaining the mutation mask value of the target image. That is, the electronic device traverses all pixels to more accurately determine the mutation mask value of the target image. The mutation mask value of the target image is determined, so that the electronic device can update the sampling step using an appropriate sampling step update strategy to adaptively sample key vertices in the depth map and construct a non-uniform vertex space topology.

Step 202: The electronic device updates the strategy based on the mutation area of the target image and the sampling step size corresponding to the mutation area, Perform non-uniform sampling on the target image to obtain a non-uniform sampling vertex set.

In the embodiment of the present application, the electronic device can perform non-uniform sampling of the depth map based on the mutation mask value and the sampling step update strategy corresponding to the mutation mask value, and obtain a non-uniform sampling vertex set (which can also be called non-uniform depth map). sample vertex collection).

In the embodiment of the present application, after determining the mutation mask value, the electronic device can determine the sampling step update strategy corresponding to the mutation mask value according to the mutation mask value of the target image, that is, according to the pixel points in the target image (i.e., the target The distribution of spatial points of the depth map of the image) to determine the way to update the sampling step, that is, to determine at least one sampling step that needs to be used in the adoption process, so as to perform non-uniform sampling of the depth map based on the at least one sampling step.

It should be noted that the above-mentioned non-uniform sampling of the depth map can be understood as: when the electronic device performs vertex sampling (which can also be understood as pixel sampling) on the depth map, after each vertex is sampled, the electronic device performs sampling of the next vertex. Before sampling, the electronic device will determine whether the sampling step size needs to be updated based on the mutation mask value of the target image (that is, the sum of the depth mutation mask values and the sum of the vertex sampling mask values described in the following embodiments), and then When it is determined that the sampling step needs to be updated, the updated sampling step is used to sample the next vertex until the vertex sampling of the depth map is completed, and a non-uniform sampling vertex set of the depth map (that is, including the vertex sampling of the depth map) is obtained. all sampled vertices).

Optionally, in this embodiment of the present application, the above step 202 can be specifically implemented through the following steps 202a to 202d.

Step 202a: For each pixel in the depth map of the target image, the electronic device determines a third value based on the initialization sampling step of any pixel and the mutation mask value of all pixels in the depth map, and based on the arbitrary The initial sampling step size of a pixel and the sampling mask of all sampled vertices determine the fourth value.

In the embodiment of the present application, the above-mentioned third value is the sum of the depth mutation mask values in the sampling step window corresponding to any pixel point, and the fourth value is the vertex sampling mask value in the sampling step window corresponding to any pixel point. The sum of code values.

In the embodiment of the present application, for each pixel in the depth map, the electronic device can determine the sampling step window corresponding to any pixel according to the initial sampling step of any pixel, and determine the sampling step window. The sum of depth mutation mask values and the sum of vertex sampling mask values within .

Specifically, for any pixel point p _i,j in the depth map D, set the sampling step to s, let the initial value (ie, initialization sampling step) s=10 (it can also be other preset values), and Set the vertex sampling mask to R(p _i,j ), where R(p _i,j )=0 indicates that the pixel point p _i,j is not sampled, and R(p _i,j )=1 indicates that the pixel point p _{i , j} has been sampled, let the initial value (that is, initialize the vertex sampling mask) R (p _{i, j} ) = 0.

The electronic device uses the current sampling step size s to obtain the sampling pixel point set P _a (p _{i, j} ) = {q _mn | is ≤ m ≤ i + s, js ≤ n ≤ j + s}, and calculates the pixel point set P The sum of the depth mutation mask values _{S a} (p _i,j ) of all pixels within a (p _i,j ). Specifically, the calculation formula is as shown in Formula 2:

The electronic device uses the current sampling step size s to obtain the sampling vertex pixel point set P _r (p _{i, j} ) = {q _mn |i≤m≤i+s-1,j≤n≤j+s-1}, and Calculate the sum of the vertex sampling mask values S _r (p _i,j ) of all pixels in the sampled vertex pixel set P _r (p _i,j ). Specifically, the calculation formula is as shown in Formula 3:

Step 202b: If the third value or the fourth value is greater than the second threshold, the electronic device updates the initialization sampling step until the first condition is met.

In the embodiment of this application, the above-mentioned first condition is that the updated sampling step is equal to the third threshold, or the sum of the depth mutation mask values and the sum of the vertex sampling mask values within the sampling step window corresponding to any pixel point equal to the second threshold.

In the embodiment of the present application, for each pixel in the depth map, the electronic device can determine whether the third value or the fourth value of each pixel is greater than the second threshold, thereby determining whether to update the initial sampling step of each pixel. long. Moreover, when the electronic device updates the initial sampling step of any pixel point and the sampling step of the electronic device for vertex sampling of the arbitrary pixel changes, then the sampling step of the electronic device for vertex sampling of the arbitrary pixel changes. The long window will also change accordingly. The electronic device can determine the sum of the depth mutation mask values and the sum of the vertex sampling mask values within the sampling step window of the arbitrary pixel in real time to determine the sampling step of the arbitrary pixel. Whether the sum of depth mutation mask values and the sum of vertex sampling mask values within the long window is equal to the second threshold.

Optionally, in the embodiment of the present application, the electronic device can update the initial sampling step size of the pixel point to the target sampling step size, and the target sampling step size is smaller than the initial sampling step size, that is, the electronic device can reduce the initial sampling step size of the pixel point. is the target sampling step size. Specifically, the electronic device can calculate the difference between the initial sampling step size of the pixel and a preset value to determine the difference as the target sampling step size. For example, if Sa>0 or Sr>0, then the target sampling step size s'=s-1, and s is the initial sampling step size of the pixel.

It can be understood that after the electronic device updates the initial sampling step size of any one pixel, the electronic device can continue to repeatedly perform the above steps 202a and 202b to continue to determine the third value and the fourth value of any one pixel, thereby determining Whether to continue updating the sampling step size of any pixel point is to determine whether the first condition is met after updating the sampling step size of any pixel point, thereby determining whether to stop updating the sampling step size of any pixel point. For example, if Sa=0 and Sr=0, or s=1, stop updating the sampling step size of any pixel.

Optionally, in this embodiment of the present application, when the third value is less than or equal to the second threshold, or the fourth value is less than or equal to the second threshold, the electronic device does not update the initialization sampling step size of a pixel, that is, Use an initial sampling step of one pixel for vertex sampling.

Step 202c: The electronic device determines the target sampling vertex from the sampling step window corresponding to any pixel point according to the updated sampling step size, and updates the vertex sampling mask value in the sampling step window corresponding to any pixel point. , obtain a depth map sampling vertex set.

In the embodiment of the present application, the electronic device determines the sampling step window corresponding to any pixel point based on the updated sampling step size, and then determines the pixel point within the sampling step window as the target sampling vertex. Specifically, the electronic device can determine the target sampling vertex for the pixel point p _i,j as {p _i,j ,pi _+s,j , _{pi ,j+s} ,p _i+s,j+s }.

In the embodiment of the present application, after the electronic device updates the sampling step size of any pixel point, the sampling step size window of any pixel point will change, then the vertex sampling within the sampling step size window corresponding to any pixel point will be cover The code value will also change, that is, the electronic device will update the vertex sampling mask value within the sampling step window corresponding to any pixel point. Specifically, the electronic device updates the vertex sampling mask value R(q _mn ∈P _r (p _i,j ))=1, P within the sampling step window corresponding to a pixel point ( _{P r} (p _i,j )) _r (p _i,j ) is the collection of sampling vertex pixel points.

In the embodiment of the present application, for any pixel point, the depth map sampling vertex set of the arbitrary pixel point includes the target sampling vertex of the arbitrary pixel point and the vertex sampling within the sampling step window corresponding to the arbitrary pixel point is updated. Mask value.

Step 202d: The electronic device traverses all pixels of the depth map of the target image to obtain a non-uniform sampling vertex set.

It can be understood that for each pixel in the depth map of the target image, one pixel corresponds to a depth map sampling vertex set, and the depth map can be obtained from the depth map sampling vertex set corresponding to all pixels in the depth map of the target image. Non-uniformly sampled vertex collections.

It should be noted that for each pixel in the depth map of the target image, the electronic device can obtain the depth map sampling vertex set corresponding to each pixel by performing the above steps 202a to 202d, thereby obtaining the depth map non-uniformity. Sample a collection of vertices.

In the embodiment of the present application, when performing vertex sampling, the electronic device can determine whether to update the sampling step for each pixel point, and after determining to update the sampling step, use the updated sampling step as a reference to perform vertex sampling and vertex sampling. The sampling mask value is updated to obtain the depth map sampling vertex set of each pixel, thereby obtaining the depth map non-uniform sampling vertex set; in this way, it can be ensured that the sampling step size can adapt to the spatial distribution of the scene (i.e., the depth map of the target image (distribution of spatial points), using a larger step size in the flat area to reduce the number of vertices of the model, and using a smaller step size at the edge of the object to improve the fineness of model details. In addition, it is ensured that there is no overlap between the sides of the model and redundant vertices and sides are avoided.

Step 203: The electronic device constructs a three-dimensional model based on the non-uniform sampling vertex set.

In the embodiment of the present application, the electronic device can perform three-dimensional mapping on each sampling vertex in the non-uniform sampling vertex set to obtain the three-dimensional space point of each sampling vertex; and, the electronic device can perform three-dimensional mapping on each sampling vertex in the non-uniform sampling vertex set. Vertices are connected by vertices to obtain a three-dimensional slice of the non-uniformly sampled vertex set; thus, the electronic device constructs a three-dimensional model based on the three-dimensional space point of each sampled vertex and the three-dimensional slice of the non-uniformly sampled vertex set.

Optionally, in this embodiment of the present application, the above step 203 can be specifically implemented through the following steps 203a to 203c.

Step 203a: The electronic device maps the non-uniformly sampled vertex set into a three-dimensional space point set to construct three-dimensional vertices.

In the embodiment of the present application, the electronic device can separately map each sampling vertex in the non-uniform sampling vertex set, that is, map each sampling vertex into a three-dimensional space point to obtain all sampling vertices in the non-uniform sampling vertex set. The three-dimensional space points, that is, the three-dimensional space point set, thereby obtaining the three-dimensional vertices of the non-uniformly sampled vertex set.

Specifically, for any sampling vertex p _i,j in the non-uniform sampling vertex set, it is assumed that the coordinates of the three-dimensional space point corresponding to the sampling vertex p _i,j are P _i,j = {x, y, z}; the electronic device can Use calculation formula 4 to map the sampling vertices p _i,j :

Among them, W is the width of the depth map of the target image, H is the height of the depth map of the target image, D is the depth map of the target image, and f is the mapping hyperparameter, for example, set f=500.

Step 203b: The electronic device constructs a connection relationship between each vertex in the non-uniformly sampled vertex set to construct a three-dimensional surface.

Optionally, in the embodiment of the present application, the electronic device can connect all sampling vertices in the non-uniform sampling vertex set, that is, connect any sampling vertex in the non-uniform sampling vertex set with other sampling vertices to obtain a three-dimensional one-sided .

Optionally, in the embodiment of the present application, for the depth map sampling vertex set of each pixel point, the electronic device can perform the sampling on all sampling vertices in the first non-uniform sampling vertex set in the depth map sampling vertex set of one pixel point. Connect to obtain a space triangle slice, and connect all the sampling vertices in the second non-uniform sampling vertex set of the depth map sampling vertex set of a pixel point to obtain another space triangle slice; the electronic device traverses all pixels By using the depth map sampling vertex set, the spatial triangular slice corresponding to each pixel point can be obtained, thereby obtaining the three-dimensional slice of the depth map non-uniformly sampling vertex set. Wherein, the electronic device uses a diagonal line of the depth map sampling vertex set of a pixel as a dividing line, and divides the depth map sampling vertex set of a pixel into a first non-uniform sampling vertex set and a second non-uniform sampling vertex set. .

For example, assuming that the depth map sampling vertex set of a pixel is {pi _,j ,pi _+s,j ,pi _,j+s , _pi+s,j+s }, the electronic device can be The first non-uniform sampling vertex set {p _i,j , p _i+s,j , p _i+s,j+s } and the second non-uniform sampling vertex set {p _i,j ,p _i,j+s , p _i+s,j+s } Construct two spatial triangle faces, that is, all samples in the first non-uniform sampling vertex set {p _i,j ,p _i+s,j ,p _i+s,j+s } The vertices are connected to obtain a space triangle surface, and all the sampling vertices in the second non-uniform sampling vertex set {p _i,j , p _i,j+s , p _i+s,j+s } are connected to obtain Another space triangle.

Step 203c: The electronic device obtains a three-dimensional model based on the three-dimensional vertices and three-dimensional faces.

In the embodiments of the present application, the electronic device can construct a model based on the three-dimensional vertices and three-dimensional faces to obtain a three-dimensional model. Specifically, electronic equipment can connect three-dimensional vertices and three-dimensional faces to obtain a three-dimensional model.

Optionally, in the embodiment of the present application, for the two spatial triangles corresponding to each pixel point, the electronic device can compare the three-dimensional vertex corresponding to one vertex in the depth map non-uniform sampling vertex set and the pixel point corresponding to the three-dimensional vertex. The corresponding two space triangles are connected, and the electronic device traverses all pixels to connect all pixels with the two corresponding space triangles to obtain a three-dimensional model.

Embodiments of the present application provide a three-dimensional model construction method. The electronic device can use the sampling step update strategy to perform non-uniform sampling based on the mutation area of the target image to obtain a non-uniform sampling vertex set, and then perform a three-dimensional model based on the set. Construct. In this solution, the electronic device can determine the corresponding sampling step update strategy based on the mutation area of the target image. That is, during the sampling process, the distribution characteristics of the three-dimensional space points are calculated based on the mutation area of the target image. feature to update the sampling step size, adaptively sample key vertices in the depth map, and construct a non-uniform vertex space topology, so that the 3D model generates a denser grid structure in the edge area of the object, ensuring the model rendering effect. Authenticity, while the flat area of space generates a sparser grid structure to reduce the number of model vertices and sides, improve the rendering performance of the grid model in electronic devices, and ultimately achieve the authenticity of the three-dimensional model rendering effect and the rendering performance of electronic devices A highly efficient 3D model.

Optionally, in this embodiment of the present application, the electronic device can perform image rendering on the three-dimensional model to obtain a rendered image. Specifically, the electronic device can use a graphics rendering engine such as OpenGL to load the vertices and sides of the three-dimensional model, as well as the corresponding textures, and adjust the corresponding rendering parameters to obtain a rendered image of the three-dimensional scene.

Optionally, in this embodiment of the present application, the above step 201 can be specifically implemented through the following step 301.

Step 301: The electronic device performs median filtering on the depth map of the target image, and corrects the edge abnormal depth information in the median-filtered depth map to obtain a corrected mutation area.

In the embodiment of the present application, the electronic device can first perform median filtering on the depth map of the target image, and then correct abnormal depth information in the depth map after the median filtering. The electronic device can then perform median filtering and corrected depth information based on the median filtering and the corrected depth information. Mutation mask values are non-uniformly sampled.

Optionally, in the embodiment of the present application, due to the limited depth of the single-frame depth estimation network, there are a large number of depth outliers in the object edge area, that is, the depth mutation zone area, which will distort the rendering effect of the object edge, so the electronic device can Correct abnormal depth values at the edges of objects.

For example, in conjunction with Figure 5, as shown in Figure 6, after the electronic device performs median filtering on the depth map of the target image, the edge abnormal depth information in the depth map of the target image is corrected, as shown in black lines in Figure 6 The corrected mutation mask value is illustrated, that is, the mutation mask value of the pixels with edge abnormal depth information in Figure 5 is changed from the first value to the second value.

In the embodiment of the present application, the electronic device can first perform median filtering on the depth map of the target image, and then correct the edge abnormal depth information in the depth map after the median filtering, so as to pass the two methods of median filtering and correction. The process processes edge abnormal depth information in the depth map, thereby improving the accuracy of the depth information in the depth map.

Optionally, in this embodiment of the present application, the above step 301 can be specifically implemented through the following steps 301a to 301c.

Step 301a: For each pixel in the depth map of the target image, the electronic device uses a preset filter window radius to determine the filtered pixel set corresponding to any pixel, and determine the weight values of all pixels in the filtered pixel set.

In the embodiment of this application, for any pixel point p _i,j in the depth map D of the target image, the filter window radius is r (that is, the preset filter window radius is r), and the filter pixel corresponding to the arbitrary pixel point p _i,j Set F(p _i,j )={q _mn |ir≤e≤i+r,jr≤n≤j+r}, and define the weight function of pixel point q∈R(p _i,j ) as shown in Formula 5 :

Among them, M is the mutation mask value of the target image, and D(q) is the depth map feature value of pixel point q.

Step 301b: The electronic device updates the depth information of any pixel to the target depth information based on the weight values of all pixels in the filtered pixel set.

In the embodiment of the present application, the above-mentioned target depth information is the median value of the depth information of adjacent pixels of any pixel.

In the embodiment of the present application, the electronic device can sort the pixel set R(pi _,j ) according to the weight value w _pq (for example, in ascending order), and update the depth value D(p) of the pixel point p to the depth value D(p) of the pixel p ^* . Depth value D(p ^* ) (that is, target depth information), where the calculation formula of pixel p ^* is as shown in Formula 6:

Step 301c: The electronic device traverses all pixels in the depth map of the target image to obtain the corrected mutation area.

In the embodiment of the present application, the electronic device corrects the edge abnormal depth information in the depth map of the target image to obtain the corrected mutation mask value (that is, the corrected mutation area).

It can be understood that for each pixel in the depth map of the target image, the electronic device can correct the edge abnormal depth information in the depth map of the target image by performing the above steps 301a and 301b.

Optionally, in this embodiment of the present application, "the electronic device performs median filtering on the depth map of the target image" in the above-mentioned step 301c can be specifically implemented through the following steps 301d and 301e.

Step 301d: The electronic device performs the first median filtering on the depth map of the target image based on the target filter window radius, and updates the target filter window radius.

In this embodiment of the present application, the electronic device can preset the number of iterations of the median filter, and the electronic device can repeatedly perform the above steps 301a to 301c in a loop, wherein during the iteration process, the radius of the median filter window is successively reduced.

Step 301e: The electronic device performs the next median filtering on the depth map after the first median filtering based on the updated target filtering window radius, and continues to update the target filtering window radius until the preset number of median filterings is completed. , to perform median filtering on the depth map of the target image.

In the embodiment of the present application, during each iteration process, the electronic device can update the mutation mask value M and the depth map D in real time. After a preset number of iteration operations (for example, 5 times), the above step 201 and related steps can be repeated again. scheme to obtain the final mutation mask value of the target image.

In the embodiment of the present application, the electronic device undergoes multiple median filtering operations based on the mutation mask value, which greatly reduces the width of the depth mutation band, effectively corrects the depth outliers at the edge of the object, and improves the accuracy of the depth map.

It should be noted that, for the three-dimensional model construction method provided by the embodiments of the present application, the execution subject may also be a three-dimensional model construction device. In the embodiment of the present application, a three-dimensional model construction device executing a three-dimensional model construction method is used as an example to illustrate the three-dimensional model construction device provided by the embodiment of the present application.

Figure 7 shows a possible structural schematic diagram of the three-dimensional model building device involved in the embodiment of the present application. As shown in FIG. 7 , the three-dimensional model construction device 70 may include: a determination module 71 , a sampling module 72 and a construction module 73 .

Among them, the determination module 71 is used to determine the mutation area of the target image. The sampling module 72 is configured to non-uniformly sample the target image based on the mutation area of the target image determined by the determination module 71 and the sampling step update strategy corresponding to the mutation area, to obtain a non-uniform sampling vertex set. The construction module 73 is used to construct a three-dimensional model based on the non-uniform sampling vertex set sampled by the sampling module 72 .

In a possible implementation, the above-mentioned determination module 71 is specifically used to determine the depth map of the target image using a single-frame depth estimation network; and obtain the depth information of each pixel of the depth map, and determine at least one depth difference. value, Each depth difference value is the difference between the depth information of a pixel in the depth map and the depth information of a pixel in a pixel area. The pixel area includes all pixels adjacent to a pixel in the depth map; and determining a mutation area of the target image based on at least one depth difference value.

In a possible implementation, the above-mentioned determination module 71 is specifically configured to determine the depth mutation mask value of a pixel corresponding to a depth difference value greater than or equal to the first threshold in at least one depth difference value as the first value; And determine the depth mutation mask value of the pixel corresponding to the depth difference value of at least one depth difference value that is smaller than the first threshold as a second value, and the second value is smaller than the first value; wherein, the mutation area of the target image is An image area composed of pixels corresponding to the first value.

In a possible implementation, the above-mentioned sampling module 72 is specifically used to:

For each pixel in the depth map of the target image, determine a third value based on the initialization sampling step of any pixel and the mutation mask value of all pixels in the depth map, and based on the initialization of any pixel The sampling step and the sampling mask of all the vertices that have been sampled determine the fourth value. The third value is the sum of the depth mutation mask values in the sampling step window corresponding to any pixel point. The fourth value is The sum of the vertex sampling mask values within the sampling step window corresponding to any pixel;

When the third value or the fourth value is greater than the second threshold, the initial sampling step is updated until the first condition is met. The first condition is that the updated sampling step is equal to the third threshold, or any pixel corresponds to The sum of the depth mutation mask values and the sum of the vertex sampling mask values within the sampling step window are both equal to the second threshold;

According to the updated sampling step, determine the target sampling vertex from the sampling step window corresponding to any pixel point, and update the vertex sampling mask value in the sampling step window corresponding to any pixel point to obtain a depth Graph sampling vertex collection;

Traverse all pixels of the depth map of the target image to obtain a set of non-uniformly sampled vertices.

In a possible implementation, the above-mentioned building module 73 is specifically used to map the non-uniformly sampled vertex set into a three-dimensional space point set to construct three-dimensional vertices; and construct connections between each vertex in the non-uniformly sampled vertex set. relationship to construct a three-dimensional side; and based on the three-dimensional vertices and the three-dimensional side, obtain a three-dimensional model.

In a possible implementation, the above-mentioned determination module 71 is specifically used to perform median filtering on the depth map of the target image, and correct the edge abnormal depth information in the median-filtered depth map to obtain the corrected mutation region.

In a possible implementation, the above-mentioned determination module 71 is specifically configured to use a preset filter window radius for each pixel in the depth map of the target image to determine the set of filtered pixels corresponding to any pixel, and determine Filter the weight values of all pixels in the pixel set; and update the depth information of any pixel to the target depth information based on the weight value of all pixels in the filtered pixel set. The target depth information is any pixel. the median value of the depth information of adjacent pixels; and traverse all pixels in the depth map of the target image to obtain the corrected mutation area.

In a possible implementation, the above-mentioned determination module 71 is specifically used to perform the first median filtering on the depth map of the target image based on the target filter window radius, and update the target filter window radius; and based on the updated Target filter window radius, perform the next median filter on the depth map after the first median filter, and continue to update the target filter window radius until the preset number of median filters is completed to perform the next median filter on the depth map of the target image. Median filtering.

Embodiments of the present application provide a three-dimensional model construction device. The three-dimensional model construction device can determine the corresponding sampling step update strategy according to the mutation area of the target image, that is, during the sampling process, calculate the three-dimensional space points according to the mutation area of the target image. distribution characteristics to update the sampling step size, adaptively sample key vertices in the depth map, and construct a non-uniform vertex space topology, so that the 3D model generates a denser grid structure in the edge area of the object to ensure model rendering. The authenticity of the effect, while the flat area of the space generates a sparser grid structure to reduce the number of model vertices and faces, improve the rendering performance of the grid model in electronic devices, and finally obtain the authenticity of the three-dimensional model rendering effect and electronic devices Render 3D models with high efficiency and performance.

The three-dimensional model building device in the embodiment of the present application may be a device, or may be a component, integrated circuit, or chip in an electronic device. The device may be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device may be a mobile phone, a tablet computer, a notebook computer, a handheld computer, a vehicle-mounted electronic device, a mobile Internet device (MID), or an augmented reality (AR)/virtual reality (VR). VR) equipment, robots, wearable devices, ultra-mobile personal computers (UMPC), netbooks or personal digital assistants (personal digital assistants, PDA), etc., and can also be servers, network attached storage (Network Attached Storage) , NAS), personal computer (personal computer, PC), television (television, TV), teller machine or self-service machine, etc., the embodiments of this application are not specifically limited.

The three-dimensional model building device in the embodiment of the present application may be a device with an operating system. The operating system can be an Android operating system, an ios operating system, or other possible operating systems, which are not specifically limited in the embodiments of this application.

The three-dimensional model construction device provided by the embodiments of the present application can implement various processes implemented by the above method embodiments. To avoid repetition, they will not be described again here.

Optionally, as shown in Figure 8, this embodiment of the present application also provides an electronic device 90, including a processor 91 and a memory 92. The memory 92 stores programs or instructions that can be run on the processor 91. When the program or instruction is executed by the processor 91, each step of the above three-dimensional model construction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, the details will not be described here.

It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

FIG. 9 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

The electronic device 100 includes but is not limited to: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, processor 110, etc. part.

Those skilled in the art can understand that the electronic device 100 may also include a power supply (such as a battery) that supplies power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby managing charging, discharging, and function through the power management system. Consumption management and other functions. The structure of the electronic device shown in Figure 9 does not constitute a limitation on the electronic device. The electronic device may include more or less components than shown in the figure, or combine certain components, or arrange different components, which will not be described again here. .

Among them, the processor 110 is used to determine the mutation area of the target image; and based on the mutation area of the target image and the sampling step update strategy corresponding to the mutation area, perform non-uniform sampling on the target image to obtain non-uniform sampling vertices. a set; and constructing a three-dimensional model based on the non-uniformly sampled vertex set.

Embodiments of the present application provide an electronic device. The electronic device can determine a corresponding sampling step update strategy according to the mutation area of the target image. That is, during the sampling process, the distribution characteristics of the three-dimensional space points are calculated according to the mutation area of the target image. By updating the sampling step size, the key vertices in the depth map are adaptively sampled to construct a non-uniform vertex space topology, thereby allowing the 3D model to generate a denser grid structure in the edge area of the object, ensuring the authenticity of the model rendering effect. , and a sparser grid structure is generated in the flat area of the space to reduce the number of model vertices and sides, improve the rendering performance of the grid model in electronic devices, and ultimately obtain an efficient and effective solution that can achieve the authenticity of the three-dimensional model rendering effect and the rendering performance of electronic devices. Excellent 3D model.

Optionally, the processor 110 is specifically configured to use a single-frame depth estimation network to determine the depth map of the target image; and obtain the depth information of each pixel of the depth map, and determine at least one depth difference value, each depth difference value The value is the difference between the depth information of a pixel in the depth map and the depth information of a pixel in a pixel area, which includes all pixels adjacent to a pixel in the depth map; and according to at least one Depth difference determines the mutation area of the target image.

Optionally, the processor 110 is specifically configured to determine the depth mutation mask value of a pixel corresponding to a depth difference value greater than or equal to the first threshold in at least one depth difference value as a first value; and The depth mutation mask value of the pixel corresponding to the depth difference value smaller than the first threshold value is determined as a second value, and the second value is smaller than the first value; wherein, the mutation area of the target image is corresponding to the first value An image area composed of pixels.

Optionally, the processor 110 is specifically configured to, for each pixel in the depth map of the target image, determine the third step based on the initialization sampling step size of any pixel and the mutation mask values of all pixels in the depth map. value, and determine the fourth value based on the initial sampling step of any pixel point and the sampling mask of all sampled vertices. The third value is the depth mutation mask within the sampling step window corresponding to any pixel point. The sum of code values, the fourth value is the sum of the vertex sampling mask values in the sampling step window corresponding to any pixel point; when the third value or the fourth value is greater than the second threshold, the initialization sampling is updated step size until the first condition is met, which is that the updated sampling step size is equal to the third threshold, or the sum of the depth mutation mask value and the vertex sampling mask value in the sampling step size window corresponding to any pixel point The sum of code values is equal to the second threshold; according to the updated sampling step, determine the target sampling vertex from the sampling step window corresponding to any pixel point, and update the sampling step window corresponding to any pixel point The vertex sampling mask value is used to obtain a depth map sampling vertex set; traverse all pixels of the depth map of the target image to obtain a non-uniform sampling vertex set.

Optionally, the processor 110 is specifically configured to map the non-uniformly sampled vertex set into a three-dimensional space point set to construct three-dimensional vertices; and construct a connection relationship between each vertex in the non-uniformly sampled vertex set to construct a three-dimensional one-sided ; and obtain a three-dimensional model based on three-dimensional vertices and three-dimensional faces.

Optionally, the processor 110 is specifically configured to perform median filtering on the depth map of the target image, and correct the edge abnormal depth information in the median filtered depth map to obtain a corrected mutation area.

Optionally, the processor 110 is specifically configured to use a preset filter window radius for each pixel in the depth map of the target image, determine the filtered pixel set corresponding to any pixel, and determine all the filtered pixels in the filtered pixel set. The weight value of the pixel; according to the weight value of all pixels in the filtered pixel set, the depth information of any pixel is updated to the target depth information, and the target depth information is the depth information of the adjacent pixels of any pixel. Depth information The median value of; traverse all pixels in the depth map of the target image to obtain the corrected mutation area.

Optionally, the processor 110 is specifically configured to perform a first median filter on the depth map of the target image based on the target filter window radius, and update the target filter window radius; and based on the updated target filter window radius, perform The depth map after the first median filtering is subjected to the next median filtering, and the target filter window radius is continued to be updated until the preset number of median filtering is completed to perform median filtering on the depth map of the target image.

The electronic device provided by the embodiments of the present application can implement each process implemented by the above method embodiments, and can achieve the same technical effect. To avoid duplication, the details will not be described here.

For specific beneficial effects of various implementation methods in this embodiment, please refer to the beneficial effects of corresponding implementation methods in the above method embodiments. To avoid duplication, they will not be described again here.

It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042. The graphics processor 1041 is responsible for the image capture device (GPU) in the video capture mode or the image capture mode. Process the image data of still pictures or videos obtained by cameras (such as cameras). The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072 . Touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, and joysticks, which will not be described again here.

Memory 109 may be used to store software programs as well as various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, Image playback function, etc.) etc. Additionally, memory 109 may include volatile memory or nonvolatile memory, or memory 109 may include both volatile and nonvolatile memory. Among them, non-volatile memory can be read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), electrically removable memory. Erase programmable read-only memory (Electrically EPROM, EEPROM) or flash memory. Volatile memory can be random access memory (Random Access Memory, RAM), static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDRSDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Synch link DRAM) , SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DRRAM). Memory 109 in embodiments of the present application includes, but is not limited to, these and any other suitable types of memory.

The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, where the application processor mainly handles operations related to the operating system, user interface, application programs, etc., Modem processors mainly process wireless communication signals, such as baseband processors. It can be understood that the above modem processor may not be integrated into the processor 110 .

Embodiments of the present application also provide a readable storage medium. Programs or instructions are stored on the readable storage medium. When the program or instructions are executed by a processor, each process of the above method embodiments is implemented and the same technology can be achieved. Effect, To avoid repetition, they will not be repeated here.

Wherein, the processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes computer readable storage media, such as computer read-only memory ROM, random access memory RAM, magnetic disk or optical disk, etc.

An embodiment of the present application further provides a chip. The chip includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement various processes of the above method embodiments. , and can achieve the same technical effect, so to avoid repetition, they will not be described again here.

It should be understood that the chips mentioned in the embodiments of this application may also be called system-on-chip, system-on-a-chip, system-on-a-chip or system-on-chip, etc.

Embodiments of the present application provide a computer program product. The program product is stored in a storage medium. The program product is executed by at least one processor to implement each process of the above three-dimensional model construction method embodiment, and can achieve the same technology. The effect will not be described here to avoid repetition.

It should be noted that, in this document, the terms "comprising", "comprises" or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device that includes a series of elements not only includes those elements, It also includes other elements not expressly listed or inherent in the process, method, article or apparatus. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article or apparatus that includes that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, but may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved. Functions may be performed, for example, the methods described may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better. implementation. Based on this understanding, the technical solution of the present application can be embodied in the form of a computer software product that is essentially or contributes to the existing technology. The computer software product is stored in a storage medium (such as ROM/RAM, disk , optical disk), including several instructions to cause a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of this application.

The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above-mentioned specific implementations. The above-mentioned specific implementations are only illustrative and not restrictive. Those of ordinary skill in the art will Inspired by this application, many forms can be made without departing from the purpose of this application and the scope protected by the claims, all of which fall within the protection of this application.

Claims (21)

1. A three-dimensional model construction method, the method includes:

   Determine the mutation area of the target image;

   Based on the mutation area of the target image and the sampling step update strategy corresponding to the mutation area, perform non-uniform sampling on the target image to obtain a non-uniform sampling vertex set;

   Based on the non-uniform sampling vertex set, a three-dimensional model is constructed.
2. The method according to claim 1, wherein determining the mutation area of the target image includes:

   Use a single-frame depth estimation network to determine the depth map of the target image;

   Obtain the depth information of each pixel of the depth map and determine at least one depth difference. Each depth difference is the depth information of one pixel of the depth map and the depth of one pixel in a pixel area. The difference value of information, the one pixel area includes all pixels adjacent to one pixel of the depth map;

   Based on the at least one depth difference value, a mutation area of the target image is determined.
3. The method according to claim 2, wherein determining the mutation area of the target image according to the at least one depth difference value includes:

   Determine the depth mutation mask value of the pixel corresponding to the depth difference value greater than or equal to the first threshold in the at least one depth difference value as the first value;

   Determine the depth mutation mask value of the pixel corresponding to the depth difference value of the at least one depth difference value that is smaller than the first threshold as a second value, where the second value is smaller than the first value;

   Wherein, the mutation area of the target image is an image area composed of pixels corresponding to the first numerical value.
4. The method according to any one of claims 1 to 3, wherein the target image is non-uniformly sampled based on a mutation area of the target image and a sampling step update strategy corresponding to the mutation area. , obtain a non-uniformly sampled vertex set, including:

   For each pixel in the depth map of the target image, a third value is determined based on the initialization sampling step of any pixel and the mutation mask value of all pixels in the depth map, and based on the arbitrary The initial sampling step of a pixel and the sampling mask of all sampled vertices determine the fourth value, and the third value is the sum of the depth mutation mask values within the sampling step window corresponding to any pixel. , the fourth value is the sum of the vertex sampling mask values within the sampling step window corresponding to any pixel point;

   When the third value or the fourth value is greater than the second threshold, the initialization sampling step is updated until the first condition is met, and the first condition is that the updated sampling step is equal to the third threshold. , or the sum of the depth mutation mask values within the sampling step window corresponding to any pixel point and the sum of the vertex sampling mask values and are both equal to the second threshold;

   According to the updated sampling step, determine the target sampling vertex from the sampling step window corresponding to any pixel point, and update the vertex sampling mask value in the sampling step window corresponding to any pixel point, and obtain A depth map sampling vertex collection;

   Traverse all pixels of the depth map of the target image to obtain the non-uniform sampling vertex set.
5. The method according to claim 1, wherein said constructing a three-dimensional model based on the non-uniformly sampled vertex set includes:

   Map the non-uniformly sampled vertex set into a three-dimensional space point set to construct three-dimensional vertices;

   Construct a connection relationship between each vertex in the non-uniform sampling vertex set to construct a three-dimensional side;

   Based on the three-dimensional vertex and the three-dimensional facet, the three-dimensional model is obtained.
6. The method according to claim 1, wherein determining the mutation area of the target image includes:

   Perform median filtering on the depth map of the target image, and correct the edge abnormal depth information in the depth map after median filtering to obtain the corrected mutation area.
7. The method according to claim 6, wherein the depth map of the target image is subjected to median filtering, and the edge abnormal depth information in the depth map after median filtering is corrected to obtain the corrected The mutated regions include:

   For each pixel in the depth map of the target image, use a preset filter window radius to determine a filtered pixel set corresponding to any pixel, and determine the weight values of all pixels in the filtered pixel set;

   According to the weight values of all pixels in the filtered pixel set, the depth information of any one pixel is updated to the target depth information, and the target depth information is the depth of the adjacent pixels of any one pixel. median of information;

   Traverse all pixels in the depth map of the target image to obtain the corrected mutation area.
8. The method according to claim 6 or 7, wherein performing median filtering on the depth map of the target image includes:

   Based on the target filter window radius, perform the first median filtering on the depth map of the target image, and update the target filter window radius;

   Based on the updated target filter window radius, perform the next median filter on the depth map after the first median filter, and continue to update the target filter window radius until the median value of the preset number of times is completed. Filter to perform median filtering on the depth map of the target image.
9. A three-dimensional model construction device, the device includes: a determination module, a sampling module and a construction module;

   The determination module is used to determine the mutation area of the target image;

   The sampling module is configured to non-uniformly sample the target image based on the mutation area of the target image determined by the determination module and the sampling step update strategy corresponding to the mutation area, and obtain non-uniform sampling vertices. gather;

   The building module is used to build a three-dimensional model based on the non-uniform sampling vertex set sampled by the sampling module.
10. The device according to claim 9, wherein the determination module is specifically configured to use a single-frame depth estimation network to determine the depth map of the target image; and obtain the depth information of each pixel of the depth map, And determine at least one depth difference value, each depth difference value is the difference between the depth information of a pixel point in the depth map and the depth information of a pixel point in a pixel field, the one pixel field includes and the All pixels adjacent to one pixel in the depth map; and determining the mutation area of the target image based on the at least one depth difference value.
11. The device according to claim 10, wherein the determining module is specifically configured to determine the depth mutation mask value of a pixel corresponding to a depth difference value greater than or equal to a first threshold in the at least one depth difference value as a first numerical value; and determine the depth mutation mask value of a pixel corresponding to a depth difference value smaller than the first threshold in the at least one depth difference value as a second numerical value, where the second numerical value is smaller than the first numerical value;

    Wherein, the mutation area of the target image is an image area composed of pixels corresponding to the first numerical value.
12. The device according to any one of claims 9 to 11, wherein the sampling module is specifically used for:

    For each pixel in the depth map of the target image, a third value is determined based on the initialization sampling step of any pixel and the mutation mask value of all pixels in the depth map, and based on the arbitrary The initial sampling step of a pixel and the sampling mask of all sampled vertices determine the fourth value, and the third value is the sum of the depth mutation mask values within the sampling step window corresponding to any pixel. , the fourth value is the sum of the vertex sampling mask values within the sampling step window corresponding to any pixel point;

    When the third value or the fourth value is greater than the second threshold, the initialization sampling step is updated until the first condition is met, and the first condition is that the updated sampling step is equal to the third threshold. , or the sum of the depth mutation mask values and the sum of the vertex sampling mask values within the sampling step window corresponding to any one pixel point are both equal to the second threshold;

    According to the updated sampling step, determine the target sampling vertex from the sampling step window corresponding to any pixel point, and update the vertex sampling mask value in the sampling step window corresponding to any pixel point, and obtain A depth map sampling vertex collection;

    Traverse all pixels of the depth map of the target image to obtain the non-uniform sampling vertex set.
13. The device according to claim 9, wherein the building module is specifically used to convert the non-uniform The sampling vertex set is mapped to a three-dimensional space point set to construct a three-dimensional vertex; and a connection relationship between each vertex in the non-uniform sampling vertex set is constructed to construct a three-dimensional side; and based on the three-dimensional vertex and the three-dimensional side , to obtain the three-dimensional model.
14. The device according to claim 9, wherein the determination module is specifically configured to perform median filtering on the depth map of the target image, and perform a median filtering on the edge abnormality depth information in the depth map after median filtering. Correction to obtain the corrected mutation region.
15. The device according to claim 14, wherein the determination module is specifically configured to use a preset filter window radius for each pixel in the depth map of the target image to determine the filter pixel corresponding to any pixel. Set, and determine the weight values of all pixel points in the filtered pixel set; and update the depth information of any one pixel point to the target depth information based on the weight values of all pixel points in the filtered pixel set, The target depth information is the median value of the depth information of adjacent pixels of any one pixel; and all pixels in the depth map of the target image are traversed to obtain the corrected mutation area.
16. The device according to claim 14 or 15, wherein the determination module is specifically configured to perform a first median filter on the depth map of the target image based on the target filter window radius, and update the target filter window radius; and based on the updated target filter window radius, perform the next median filter on the depth map after the first median filter, and continue to update the target filter window radius until the preset is completed times of median filtering to perform median filtering on the depth map of the target image.
17. An electronic device, including a processor, a memory and a program or instructions stored on the memory and executable on the processor. When the program or instructions are executed by the processor, the implementation of claims 1 to 8 is achieved. The steps of the three-dimensional model construction method described in any one of the above.
18. A readable storage medium on which programs or instructions are stored. When the programs or instructions are executed by a processor, the steps of the three-dimensional model construction method according to any one of claims 1 to 8 are implemented.
19. A chip. The chip includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the three-dimensional process as claimed in any one of claims 1 to 8. Model building methods.
20. A computer program product, the program product is stored in a storage medium, and the program product is executed by at least one processor to implement the three-dimensional model building method according to any one of claims 1 to 8.
21. An electronic device, characterized by including the electronic device for executing the three-dimensional model building method according to any one of claims 1 to 8.