CN113160296B - Three-dimensional reconstruction method and device for vibration liquid drop based on micro-rendering - Google Patents
Three-dimensional reconstruction method and device for vibration liquid drop based on micro-rendering Download PDFInfo
- Publication number
- CN113160296B CN113160296B CN202110348718.XA CN202110348718A CN113160296B CN 113160296 B CN113160296 B CN 113160296B CN 202110348718 A CN202110348718 A CN 202110348718A CN 113160296 B CN113160296 B CN 113160296B
- Authority
- CN
- China
- Prior art keywords
- model
- rendering
- micro
- image
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000009877 rendering Methods 0.000 title claims abstract description 106
- 239000007788 liquid Substances 0.000 title claims abstract description 78
- 238000000034 method Methods 0.000 title claims abstract description 65
- 230000006870 function Effects 0.000 claims description 44
- 230000008569 process Effects 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 9
- 238000006073 displacement reaction Methods 0.000 claims description 8
- 238000003860 storage Methods 0.000 claims description 8
- 238000010586 diagram Methods 0.000 claims description 7
- 238000009826 distribution Methods 0.000 claims description 7
- 230000002776 aggregation Effects 0.000 claims description 5
- 238000004220 aggregation Methods 0.000 claims description 5
- 238000010276 construction Methods 0.000 claims description 4
- 238000009499 grossing Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 4
- 239000007787 solid Substances 0.000 claims description 4
- 230000000007 visual effect Effects 0.000 abstract description 2
- 238000005457 optimization Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 6
- 238000000605 extraction Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 239000012530 fluid Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 230000000284 resting effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/005—General purpose rendering architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/08—Indexing scheme for image data processing or generation, in general involving all processing steps from image acquisition to 3D model generation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T90/00—Enabling technologies or technologies with a potential or indirect contribution to GHG emissions mitigation
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Graphics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Image Processing (AREA)
Abstract
The invention provides a three-dimensional reconstruction method and device for vibration liquid drops based on micro-rendering, wherein the method comprises the following steps: establishing a micro rendering model according to the input camera parameters; converting an input multi-view liquid drop vibration video into a multi-view image at the same moment, and extracting edges of the multi-view image; initializing a three-dimensional model of the liquid drop, and obtaining a corresponding rendering image through the micro-rendering model; and calculating a loss function value between the rendering image and the multi-view image, and reversely transmitting the loss function value to three-dimensional model parameters of the liquid drop to update the model parameters. The method considers the characteristics of unobvious characteristic points and rapid vibration of the liquid drops, and can make the three-dimensional model of the liquid drops consistent with the photographed image under a plurality of visual angles by micro-rendering combined with physical model description, thereby achieving faster reconstruction speed and accurate reconstruction precision.
Description
Technical Field
The invention relates to the technical field of computer graphics, in particular to a method and a device for reconstructing three-dimensional vibration liquid drops based on micro rendering.
Background
Traditional three-dimensional reconstruction technology for objects mainly relies on the multi-view geometric principle to calculate based on parallax. The motion restoration structure method (Structure from Motion) mainly utilizes images obtained by a plurality of cameras to restore three dimensions, firstly extracts characteristic points of multi-view images and matches the characteristic points, restores spatial correspondence among different view angles according to a geometric principle, firstly reconstructs camera parameters, and then combines unified characteristic points of different images to calculate point cloud information of a target object.
The method has low robustness to dynamic objects, has poor effect to objects such as liquid drops with few textures and undefined characteristic points, consumes a large amount of time, has poor instantaneity, can only recover the point cloud of the target object, and needs a post-processing process if triangular grid representation is needed.
Disclosure of Invention
Aiming at the problem that the real-time reconstruction of high-speed vibration liquid drops with few characteristic points cannot be effectively realized in the prior art, the invention provides a three-dimensional reconstruction method and device for vibration liquid drops based on micro-rendering.
The invention provides a three-dimensional reconstruction method of vibration liquid drops based on micro-rendering, which comprises the following steps: establishing a micro rendering model according to the input camera parameters; converting an input multi-view liquid drop vibration video into a multi-view image at the same moment, and extracting edges of the multi-view image; initializing a three-dimensional model of the liquid drop, and obtaining a corresponding rendering image through the micro-rendering model; and calculating a loss function value between the rendering image and the multi-view image, and reversely transmitting the loss function value to three-dimensional model parameters of the liquid drop to update the model parameters.
According to one embodiment of the invention, the method for reconstructing the three-dimensional liquid drop based on micro-rendering vibration further comprises the following steps: repeating the process of updating the three-dimensional model parameters of the previous frame as an initial model, obtaining a corresponding rendering image through the micro-rendering model, calculating a loss function value between the rendering image and the multi-view image of the current frame, and updating the model parameters until a preset condition is met.
According to one embodiment of the invention, the method for reconstructing a vibro-droplet three-dimensional model based on micro-renderable includes: and (3) establishing a rendering model by using a soft rasterization renderer, replacing the solid triangular patches by using probability distribution of pixel positions, and replacing front-rear shielding relation of the triangular patches by using a distance-related aggregation function.
According to a method for reconstructing a three-dimensional model of a liquid drop based on micro-renderable vibration liquid drop, the method comprises the following steps: constructing a triangular mesh model of the unit sphere, calculating the displacement sum of all vertex vibration modes, radially moving the vertices of the triangular mesh, and generating a triangular mesh model of the liquid drop.
According to one embodiment of the invention, the three-dimensional reconstruction method of the vibro-droplet based on micro-rendering comprises the following steps:
wherein I is s ,
Masks for capturing and rendering images, respectively.
According to one embodiment of the invention, the three-dimensional reconstruction method of the vibro-droplet based on micro-rendering is characterized in that the three-dimensional model parameters of the droplet are back propagated to perform model parameter updating, and the method comprises the following steps: and determining an objective function according to the Laplace regularization term, the loss function and the smoothing regularization term, and updating parameters according to the objective function.
The invention also provides a device for reconstructing the three-dimensional vibration liquid drop based on micro-rendering, which comprises the following components: the model construction module is used for establishing a micro rendering model according to the input camera parameters; the image processing module is used for converting the input multi-view liquid drop vibration video into multi-view images at the same moment and extracting edges of the multi-view images; the rendering diagram generation module is used for initializing a three-dimensional model of the liquid drop and obtaining a corresponding rendering image through the micro-rendering model; and the model reconstruction module is used for calculating a loss function value between the rendering image and the multi-view image, and reversely transmitting the loss function value to three-dimensional model parameters of the liquid drops to update the model parameters.
According to one embodiment of the invention, the model reconstruction module is further used for: repeating the process of updating the three-dimensional model parameters of the previous frame as an initial model, obtaining a corresponding rendering image through the micro-rendering model, calculating a loss function value between the rendering image and the multi-view image of the current frame, and updating the model parameters until a preset condition is met.
The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to realize the steps of the micro-renderable vibration liquid drop three-dimensional reconstruction method.
The invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the vibro-droplet three-dimensional reconstruction method based on micro-renderable as defined in any one of the above.
According to the vibration liquid drop three-dimensional reconstruction method and device based on micro-rendering, provided by the invention, the characteristics of unobvious liquid drop characteristic points and rapid vibration are considered, and the three-dimensional model of the liquid drop can be consistent with a shot image under a plurality of visual angles by combining micro-rendering with physical model description, so that the reconstruction speed is high and the accurate reconstruction precision is achieved.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a three-dimensional reconstruction method of vibro-droplet based on micro-renderable;
FIG. 2 is a graphical illustration of output results provided by the present invention;
FIG. 3 is a schematic diagram of a three-dimensional reconstruction device based on vibro-droplets capable of micro-rendering according to the present invention;
fig. 4 is a schematic structural diagram of an electronic device provided by the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Rendering is the opposite process to three-dimensional reconstruction, which requires the input of a two-dimensional image, reasoning about a three-dimensional structure, while rendering is the three-dimensional information of a given object or scene, simulating the shooting of a two-dimensional image by a camera. Traditional rendering comprises a rasterization method and a ray tracing method, wherein the rasterization method is good in real-time performance, and the ray tracing method can generate pictures with strong authenticity. While the rapid development of deep learning has led to a great deal of attention to gradient-based optimization techniques. Traditional rasterization has an uncomfortable calculation process, soft rasterization replaces these uncomfortable parts with micro-operations through probability distribution approximation and aggregation functions, which makes the whole rendering pipeline completely micro, i.e. the gradient of the output image relative to the input parameters can be calculated, and further parameter optimization is performed through gradient descent, so that the rendering pipeline is reversed, and the process of reconstructing three-dimensional information through two-dimensional images is realized.
Aiming at the problems that the traditional reconstruction method cannot process the characteristics of liquid drops, and the vibration is rapid, the invention provides a three-dimensional reconstruction method of the vibration liquid drops based on micro-rendering. According to the method, a physical model of vibration of the liquid drop in a gravity-free state is considered, a parameterized liquid drop three-dimensional model is built on the basis of spherical harmonics, the whole rendering process is calculated in a micro-rendering frame, in order to optimize consistency of a rendered image and a photographed image, a loss function between the rendering image and the photographed image is calculated, and the parameterized model is adjusted through back propagation and gradient optimization so that the two are as close as possible, so that reconstruction of the vibration liquid drop is completed.
The method and apparatus for three-dimensional reconstruction of vibro-droplet based on micro-renderable according to the present invention are described below with reference to fig. 1 to 4. Fig. 1 is a schematic flow chart of a three-dimensional reconstruction method of a vibro-droplet based on micro-rendering according to the present invention, as shown in fig. 1, the three-dimensional reconstruction method of a vibro-droplet based on micro-rendering according to the present invention includes:
101. and establishing a micro rendering model according to the input camera parameters.
Parameters of a camera are required to be known, camera internal parameters are obtained after the camera is calibrated, and the pose of the camera relative to a shot object is fixed, wherein the pose is represented by a triplet of distance, azimuth angle and pitch angle. The micro-renderable model is built with a soft rasterized renderer that describes the rendering as a micro-aggregate process.
102. And converting the input multi-view liquid drop vibration video into a multi-view image at the same moment, and performing edge extraction on the multi-view image.
Firstly, video streams of different cameras are aligned in time, and images of different view angles at the same time are combined to obtain multi-view images in time sequence for the follow-up step to reconstruct frame by frame. And (3) extracting the outline of the liquid drop by applying an edge extraction operator to the single image, and dividing the liquid drop from the background by combining brightness information. The image edge lifting can be carried out by using the Laplacian operator, the image is divided into an object and liquid drops by combining brightness information, the positions of the liquid drops are represented by using a mask, the liquid pixels are 1, and the background pixels are 0.
103. Initializing a three-dimensional model of the liquid drop, and obtaining a corresponding rendering image through the micro-renderable model.
The vibration of the liquid drop can be described through spherical harmonic function to initialize the three-dimensional model, and the rendered three-dimensional model can obtain corresponding rendered images in any direction.
104. And calculating a loss function value between the rendering image and the multi-view image, and reversely transmitting the loss function value to three-dimensional model parameters of the liquid drop to update the model parameters.
The image in the multi-view image is the real state of the liquid drop, and the information in the multi-view image is learned into the three-dimensional model in the form of three-dimensional model parameters, so that an accurate liquid drop three-dimensional model is obtained.
The positions where the droplets appear in the rendered image and the photographed image can be described by using a mask, the intersection ratio of the two is calculated as a loss function, and the Laplace smoothing term is used as a regular. Gradient descent optimization is performed by calculating the gradient of the loss function relative to the input through the micro-renderable pipeline.
According to the vibration liquid drop three-dimensional reconstruction method based on micro-rendering, provided by the invention, the characteristics of unobvious liquid drop characteristic points and rapid vibration are considered, and the three-dimensional model of the liquid drop can be consistent with a shot image under a plurality of view angles by combining micro-rendering with physical model description, so that the reconstruction speed is high and the accurate reconstruction precision is achieved.
In one embodiment, after the updating of the model parameters, the method further includes: repeating the process of updating the three-dimensional model parameters of the previous frame as an initial model, obtaining a corresponding rendering image through the micro-rendering model, calculating a loss function value between the rendering image and the multi-view image of the current frame, and updating the model parameters until a preset condition is met.
Steps 103-104 can be repeatedly performed to iteratively update the parameters of the droplet to obtain a more accurate three-dimensional model of the droplet. The vibration parameters of the drops under the initial frame may be quite different from the actual vibration parameters, thus requiring multiple iterative updates. The preset condition may be a set number of times, or other termination condition. For the (i+1) th frame, the optimization result of the (n) th frame is used for initializing parameters, and as the vibration parameters have continuity in time, the iteration times and time required by each optimization are reduced, and the reconstruction precision and speed are improved.
The specific method comprises the following steps: and fixing the learning rate, and carrying out gradient descent optimization parameters for a plurality of times to obtain an accurate reconstruction result of a single frame. For dynamic reconstruction, the physical parameters of each frame are initialized by the result of the previous frame in consideration of the time continuity, so that the reconstruction time is shortened, and the reconstruction robustness is improved.
In one embodiment, the establishing a differentiable rendering model according to the input camera parameters includes: and (3) establishing a rendering model by using a soft rasterization renderer, replacing the solid triangular patches by using probability distribution of pixel positions, and replacing front-rear shielding relation of the triangular patches by using a distance-related aggregation function.
The color contributions of the different triangular patches to each pixel location are described in terms of probability distribution and fused together according to distance-dependent weights. The probability map used on screen space can describe the position of the triangular patches slightly, enabling gradients to flow over all mesh triangles, allowing supervisory signals to control different parameters in the model.
First of all, it is required thatInternal parameters of the known cameras are that the pose of the camera relative to the photographed liquid drop is fixed and expressed by a triplet distance, azimuth angle and pitch angle, and the pose of the kth camera is recorded as
The micro-renderable model receives the input of camera pose and triangle surface element, and the pose input of a model is set as +.>
The trigonometric surface element input is f j Transforming the bin into the screen space of the camera according to pose, wherein T represents a series of rotation and translation transformations, and the transformed bin is recorded as
Describing the contribution of the bin to each pixel by probability distribution, assuming the contribution of bin j to pixel i as
Where d (i, j) is the minimum distance of the pixel from the edge of the bin;
the value is 1 or-1, if the pixel is 1 in the bin, the value is-1 otherwise; σ is a parameter that controls rendering sharpness. The method only requires the rendering of a mask of the object, which mask is +.>
In one embodiment, the initializing a three-dimensional model of a droplet includes: constructing a triangular mesh model of the unit sphere, calculating the displacement sum of all vertex vibration modes, radially moving the vertices of the triangular mesh, and generating a triangular mesh model of the liquid drop.
The infinitely small vibrations of incompressible viscosity-free fluid in the gravity-free state mathematically have a closed form of analytical solution. The radial displacement of the liquid drop is superposition of different eigenmodes i, m radial displacement:
wherein Y is lm X is real spherical harmonic lm 、V lm To correspond to the initial displacement and velocity of the vibration mode omega lm The vibration frequency, θ, φ is the angular parameter of the sphere in real spherical harmonics. The vibration frequency satisfies:
where α is the surface tension coefficient, ρ is the liquid density, R is the radius of the droplet in the resting state, and l is the order of the spherical harmonics.
It can be seen that the coefficient of each vibration mode is only related to the coefficient of the same mode, and the vibration coefficients of different modes are completely decoupled, so that the vibration coefficients of different modes can be used as physical parameters for constructing a three-dimensional model of the liquid drop. In actual rendering, a triangular mesh is required to be used for representing the shape, and since the shape of the liquid drop is close to a sphere, a triangular mesh model of a unit sphere can be constructed first, and the displacement of each vertex is calculated, and the vertices of the triangular mesh are moved radially, so that the triangular mesh model of the liquid drop is generated.
In one embodiment, the loss function includes:
wherein I is s ,
Masks for taking and rendering images, respectively, < >>
Respectively element-wise additions and multiplications.
In one embodiment, the back-propagating into the three-dimensional model parameters of the droplet, performing model parameter updating includes: and determining an objective function according to the Laplace regularization term, the loss function and the smoothing regularization term, and updating parameters according to the objective function.
The Laplace regularization term is recorded as
For preventing sphere selfing, the smooth regularization term is +.>
Increasing the smoothness of the surface of the droplet. The optimization objective may be:
wherein,,
is Laplacian regular term for preventing sphere selfing, +.>
To smooth regular terms, lambda is used to increase the smoothness of the drop surface lap ,λ f Respectively for balancing the corresponding regularization terms.
Fig. 2 is a schematic diagram of an output result provided by the present invention, and as shown in fig. 2, the first row and the second row are respectively a mask of an input image and a mask of a rendered image during current optimization, and the third row is a droplet model reconstructed from a certain frame.
The three-dimensional reconstruction device based on the micro-renderable vibration liquid drop provided by the invention is described below, and the three-dimensional reconstruction device based on the micro-renderable vibration liquid drop described below and the three-dimensional reconstruction method based on the micro-renderable vibration liquid drop described above can be correspondingly referred to each other.
Fig. 3 is a schematic structural diagram of a three-dimensional reconstruction device for vibro-droplet based on micro-rendering according to the present invention, as shown in fig. 3, the three-dimensional reconstruction device for vibro-droplet based on micro-rendering includes: a model construction module 301, an image processing module 302, a rendering map generation module 303, and a model reconstruction module 304. The model building module 301 is configured to build a micro rendering model according to the input camera parameters; the image processing module 302 is configured to convert an input multi-view droplet vibration video into a multi-view image at the same time, and perform edge extraction on the multi-view image; the rendering graph generating module 303 is configured to initialize a three-dimensional model of the droplet, and obtain a corresponding rendering image through the micro-renderable model; the model reconstruction module 304 is configured to calculate a loss function value between the rendered image and the multi-view image, and back-propagate the loss function value to three-dimensional model parameters of the droplet, and update the model parameters.
In one embodiment, the model reconstruction module is further configured to: repeating the process of updating the three-dimensional model parameters of the previous frame as an initial model, obtaining a corresponding rendering image through the micro-rendering model, calculating a loss function value between the rendering image and the multi-view image of the current frame, and updating the model parameters until a preset condition is met.
The embodiment of the device provided by the embodiment of the present invention is for implementing the above embodiments of the method, and specific flow and details refer to the above embodiments of the method, which are not repeated herein.
According to the vibration liquid drop three-dimensional reconstruction device based on micro-rendering, provided by the embodiment of the invention, the characteristics of unobvious liquid drop characteristic points and rapid vibration are considered, and the three-dimensional model of the liquid drop can be consistent with a shot image under a plurality of view angles by micro-rendering combined with physical model description, so that the reconstruction speed is high and the accurate reconstruction precision is achieved.
Fig. 4 is a schematic structural diagram of an electronic device according to the present invention, as shown in fig. 4, the electronic device may include: a processor (processor) 401, a communication interface (Communications Interface) 402, a memory (memory) 403 and a communication bus 404, wherein the processor 401, the communication interface 402 and the memory 403 complete communication with each other through the communication bus 404. The processor 401 may call logic instructions in the memory 403 to perform a micro-renderable vibro-droplet based three-dimensional reconstruction method comprising: establishing a micro rendering model according to the input camera parameters; converting an input multi-view liquid drop vibration video into a multi-view image at the same moment, and extracting edges of the multi-view image; initializing a three-dimensional model of the liquid drop, and obtaining a corresponding rendering image through the micro-rendering model; and calculating a loss function value between the rendering image and the multi-view image, and reversely transmitting the loss function value to three-dimensional model parameters of the liquid drop to update the model parameters.
Further, the logic instructions in the memory 403 may be implemented in the form of software functional units and stored in a computer readable storage medium when sold or used as a stand alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the method of three-dimensional reconstruction of vibro-droplets based on micro-renderable as provided by the above methods, the method comprising: establishing a micro rendering model according to the input camera parameters; converting an input multi-view liquid drop vibration video into a multi-view image at the same moment, and extracting edges of the multi-view image; initializing a three-dimensional model of the liquid drop, and obtaining a corresponding rendering image through the micro-rendering model; and calculating a loss function value between the rendering image and the multi-view image, and reversely transmitting the loss function value to three-dimensional model parameters of the liquid drop to update the model parameters.
In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the vibro-droplet three-dimensional reconstruction method based on micro-renderable provided by the above embodiments, the method comprising: establishing a micro rendering model according to the input camera parameters; converting an input multi-view liquid drop vibration video into a multi-view image at the same moment, and extracting edges of the multi-view image; initializing a three-dimensional model of the liquid drop, and obtaining a corresponding rendering image through the micro-rendering model; and calculating a loss function value between the rendering image and the multi-view image, and reversely transmitting the loss function value to three-dimensional model parameters of the liquid drop to update the model parameters.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (8)
1. The three-dimensional reconstruction method of the vibrating liquid drop based on micro-rendering is characterized by comprising the following steps of:
establishing a micro rendering model according to the input camera parameters;
converting an input multi-view liquid drop vibration video into a multi-view image at the same moment, and extracting edges of the multi-view image;
initializing a three-dimensional model of the liquid drop, and obtaining a corresponding rendering image through the micro-rendering model;
calculating a loss function value between the rendering image and the multi-view image, and reversely transmitting the loss function value to three-dimensional model parameters of the liquid drops to update the model parameters;
the establishing a micro rendering model according to the input camera parameters comprises the following steps:
a rendering model is built by a soft grating renderer, the probability distribution of pixel positions is used for replacing a solid triangular patch, and a distance-related aggregation function is used for replacing the front-back shielding relation of the triangular patch;
the initializing a three-dimensional model of a droplet includes:
constructing a triangular mesh model of the unit sphere, calculating the displacement sum of all vertex vibration modes, radially moving the vertices of the triangular mesh, and generating a triangular mesh model of the liquid drop.
2. The method for three-dimensional reconstruction of vibro-droplet based on micro-renderable according to claim 1, further comprising, after the model parameter updating:
repeating the process of updating the three-dimensional model parameters of the previous frame as an initial model, obtaining a corresponding rendering image through the micro-rendering model, calculating a loss function value between the rendering image and the multi-view image of the current frame, and updating the model parameters until a preset condition is met.
3. The micro-renderable vibrating droplet three-dimensional reconstruction method according to claim 1, wherein the loss function comprises:
wherein I is s ,
Masks for capturing and rendering images, respectively.
4. The method for three-dimensional reconstruction of vibro-droplet based on micro-rendering according to claim 1, wherein the back-propagating to three-dimensional model parameters of droplet, updating model parameters, comprises:
and determining an objective function according to the Laplace regularization term, the loss function and the smoothing regularization term, and updating parameters according to the objective function.
5. A micro-renderable vibrating droplet three-dimensional reconstruction device, comprising:
the model construction module is used for establishing a micro rendering model according to the input camera parameters;
the image processing module is used for converting the input multi-view liquid drop vibration video into multi-view images at the same moment and extracting edges of the multi-view images;
the rendering diagram generation module is used for initializing a three-dimensional model of the liquid drop and obtaining a corresponding rendering image through the micro-rendering model;
the model reconstruction module is used for calculating a loss function value between the rendering image and the multi-view image, and reversely transmitting the loss function value to three-dimensional model parameters of the liquid drops to update the model parameters;
the model construction module is specifically configured to:
a rendering model is built by a soft grating renderer, the probability distribution of pixel positions is used for replacing a solid triangular patch, and a distance-related aggregation function is used for replacing the front-back shielding relation of the triangular patch;
the rendering graph generation module is specifically configured to:
constructing a triangular mesh model of the unit sphere, calculating the displacement sum of all vertex vibration modes, radially moving the vertices of the triangular mesh, and generating a triangular mesh model of the liquid drop.
6. The micro-renderable vibrating droplet three-dimensional reconstruction device according to claim 5, the model reconstruction module further configured to:
repeating the process of updating the three-dimensional model parameters of the previous frame as an initial model, obtaining a corresponding rendering image through the micro-rendering model, calculating a loss function value between the rendering image and the multi-view image of the current frame, and updating the model parameters until a preset condition is met.
7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the micro-renderable vibrating droplet three-dimensional reconstruction method according to any one of claims 1 to 4 when the program is executed.
8. A non-transitory computer readable storage medium having stored thereon a computer program, characterized in that the computer program when executed by a processor implements the steps of the micro-renderable-based vibro-droplet three-dimensional reconstruction method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110348718.XA CN113160296B (en) | 2021-03-31 | 2021-03-31 | Three-dimensional reconstruction method and device for vibration liquid drop based on micro-rendering |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110348718.XA CN113160296B (en) | 2021-03-31 | 2021-03-31 | Three-dimensional reconstruction method and device for vibration liquid drop based on micro-rendering |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113160296A CN113160296A (en) | 2021-07-23 |
| CN113160296B true CN113160296B (en) | 2023-06-06 |
Family
ID=76885775
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110348718.XA Active CN113160296B (en) | 2021-03-31 | 2021-03-31 | Three-dimensional reconstruction method and device for vibration liquid drop based on micro-rendering |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113160296B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114120062B (en) * | 2021-11-26 | 2023-07-28 | 北京百度网讯科技有限公司 | Sample generation method and device |
| CN114140510A (en) * | 2021-12-03 | 2022-03-04 | 北京影谱科技股份有限公司 | Incremental three-dimensional reconstruction method and device and computer equipment |
| CN115115780B (en) * | 2022-06-29 | 2024-07-09 | 聚好看科技股份有限公司 | Three-dimensional reconstruction method and system based on multi-view RGBD camera |
| CN116206035B (en) * | 2023-01-12 | 2023-12-01 | 北京百度网讯科技有限公司 | Face reconstruction method, device, electronic equipment and storage medium |
| CN116824026B (en) * | 2023-08-28 | 2024-01-09 | 华东交通大学 | Three-dimensional reconstruction method, device, system and storage medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111243071A (en) * | 2020-01-08 | 2020-06-05 | 叠境数字科技（上海）有限公司 | Texture rendering method, system, chip, device and medium for real-time three-dimensional human body reconstruction |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7948485B1 (en) * | 2005-12-12 | 2011-05-24 | Sony Computer Entertainment Inc. | Real-time computer simulation of water surfaces |
| CN102930588B (en) * | 2012-09-20 | 2015-05-20 | 四川川大智胜软件股份有限公司 | Real-time rendering method for water drops at screen space lens |
| CN102930583B (en) * | 2012-10-17 | 2015-02-11 | 中国科学院自动化研究所 | Method for interactively generating droplet effect |
-
2021
- 2021-03-31 CN CN202110348718.XA patent/CN113160296B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111243071A (en) * | 2020-01-08 | 2020-06-05 | 叠境数字科技（上海）有限公司 | Texture rendering method, system, chip, device and medium for real-time three-dimensional human body reconstruction |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113160296A (en) | 2021-07-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113160296B (en) | Three-dimensional reconstruction method and device for vibration liquid drop based on micro-rendering | |
| Liu et al. | Infinite nature: Perpetual view generation of natural scenes from a single image | |
| Yang et al. | High-resolution image inpainting using multi-scale neural patch synthesis | |
| Liu et al. | Soft rasterizer: Differentiable rendering for unsupervised single-view mesh reconstruction | |
| US10740897B2 (en) | Method and device for three-dimensional feature-embedded image object component-level semantic segmentation | |
| DE102020000810A1 (en) | 3D object reconstruction using a photometric network representation | |
| CN111784821B (en) | Three-dimensional model generation method and device, computer equipment and storage medium | |
| Weng et al. | Vid2actor: Free-viewpoint animatable person synthesis from video in the wild | |
| CN114863038A (en) | Real-time dynamic free visual angle synthesis method and device based on explicit geometric deformation | |
| CN112991537B (en) | City scene reconstruction method and device, computer equipment and storage medium | |
| Ji et al. | Geometry-aware single-image full-body human relighting | |
| CN117413300A (en) | Method and system for training quantized nerve radiation field | |
| CN113781621A (en) | Three-dimensional reconstruction processing method, device, equipment and storage medium | |
| Liu et al. | Real-time neural rasterization for large scenes | |
| CN113593033A (en) | Three-dimensional model feature extraction method based on grid subdivision structure | |
| CN112509144A (en) | Face image processing method and device, electronic equipment and storage medium | |
| CN115115805A (en) | Training method, device and equipment for three-dimensional reconstruction model and storage medium | |
| CN116543086A (en) | Nerve radiation field processing method and device and electronic equipment | |
| CN114170290A (en) | Image processing method and related equipment | |
| Caruso et al. | 3d reconstruction of non-cooperative resident space objects using instant ngp-accelerated nerf and d-nerf | |
| US20110012910A1 (en) | Motion field texture synthesis | |
| Ramwala et al. | Reminiscent net: conditional GAN-based old image de-creasing | |
| Kubade et al. | Afn: Attentional feedback network based 3d terrain super-resolution | |
| CN114118367B (en) | Method and equipment for constructing incremental nerve radiation field | |
| CN115375816A (en) | Micro-rendering method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |