CN117095129A - Object three-dimensional reconstruction method and device, electronic equipment and storage medium - Google Patents
Object three-dimensional reconstruction method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN117095129A CN117095129A CN202311141384.4A CN202311141384A CN117095129A CN 117095129 A CN117095129 A CN 117095129A CN 202311141384 A CN202311141384 A CN 202311141384A CN 117095129 A CN117095129 A CN 117095129A
- Authority
- CN
- China
- Prior art keywords
- color
- reflection color
- dimensional
- pixel point
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000012545 processing Methods 0.000 claims abstract description 24
- 230000006870 function Effects 0.000 claims description 65
- 238000004590 computer program Methods 0.000 claims description 17
- 238000012549 training Methods 0.000 claims description 16
- 238000000605 extraction Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 description 9
- 239000003086 colorant Substances 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 230000003068 static effect Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004040 coloring Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/50—Lighting effects
- G06T15/55—Radiosity
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Graphics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the disclosure discloses a three-dimensional reconstruction method and device of an object, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring a three-dimensional coordinate set of a pixel point set on the surface of a target object under a preset three-dimensional coordinate system, and acquiring camera sight line sets of virtual cameras respectively facing all pixel points in the pixel point set under the preset coordinate system; processing the three-dimensional coordinate set by using a diffuse reflection color prediction network in the color prediction model to obtain a diffuse reflection color value set of the pixel point set, and processing the three-dimensional coordinate set and the camera sight set by using a specular reflection color prediction network in the color prediction model to obtain a specular reflection color value set of the pixel point set; and determining the three-dimensional structure color model of the target object based on the three-dimensional structure model, the diffuse reflection color value set and the specular reflection color value set of the target object. The embodiment of the disclosure can obtain the three-dimensional structure color model of the target object with accurate color information.
Description
Technical Field
The disclosure relates to the technical field of three-dimensional reconstruction, in particular to a three-dimensional reconstruction method and device for an object, electronic equipment and a storage medium.
Background
Multi-view three-dimensional reconstruction methods, such as depth based on prediction/laser/structured light collection, point cloud, poisson reconstruction, and mapping methods, are more difficult to model for objects with weak textures/reflections/complex structures, and are limited by expensive shooting equipment and shooting environments. The hidden three-dimensional reconstruction method based on the neural network can lead out the white model from the hidden field, however, the corresponding color information on the model cannot be directly led out, and the method can only render and obtain the final colored picture, but cannot obtain the colored three-dimensional model.
When reconstructing an object in three dimensions, how to accurately obtain a three-dimensional model with color is a problem to be solved.
Disclosure of Invention
The embodiment of the disclosure provides a three-dimensional object reconstruction method and device, electronic equipment and a storage medium, so as to solve the problems.
In a first aspect of an embodiment of the present disclosure, there is provided a three-dimensional reconstruction method of an object, including:
acquiring a three-dimensional coordinate set of a pixel point set on the surface of a target object under a preset three-dimensional coordinate system, and acquiring camera sight line sets of virtual cameras respectively facing all pixel points in the pixel point set under the preset coordinate system;
processing the three-dimensional coordinate set by using a diffuse reflection color prediction network in a color prediction model to obtain a diffuse reflection color value set of the pixel point set, and processing the three-dimensional coordinate set and the camera sight set by using a specular reflection color prediction network in the color prediction model to obtain a specular reflection color value set of the pixel point set;
and determining a three-dimensional structure color model of the target object based on the three-dimensional structure model of the target object, the diffuse reflection color value set and the specular reflection color value set.
In one embodiment of the disclosure, the acquiring a camera line-of-sight set of the virtual camera towards all pixels in the set of pixels respectively in the preset coordinate system includes:
acquiring a camera three-dimensional coordinate of the virtual camera under the preset coordinate system;
determining the camera gaze set based on the camera three-dimensional coordinates and the three-dimensional coordinate set;
the camera sight of the virtual camera towards any pixel point in the pixel point set is obtained by the following steps: and determining the camera sight of the virtual camera towards any pixel point based on the camera three-dimensional coordinates and the pixel three-dimensional coordinates of any pixel point.
In one embodiment of the present disclosure, before the processing the three-dimensional coordinate set by using the diffuse reflection color prediction network in the color prediction model to obtain a diffuse reflection color value set of the pixel point set, and processing the three-dimensional coordinate set and the camera line-of-sight set by using the specular reflection color prediction network in the color prediction model to obtain a specular reflection color value set of the pixel point set, the method further includes:
determining a color supervision loss function of the color prediction model based on the diffuse reflection color parameter and the specular reflection color parameter;
obtaining a constraint cost function of the color prediction model;
determining a final loss function of the color prediction model based on the color supervised loss function and the constrained cost function;
and training the color prediction model based on the three-dimensional coordinates of the sample pixel points, and the diffuse reflection color labels of the sample pixel points under the constraint of the final loss function.
In one embodiment of the present disclosure, the obtaining the constraint cost function of the color prediction model includes:
the constraint cost function is determined based on a difference between the diffuse reflection color and the true diffuse reflection color.
In one embodiment of the present disclosure, the obtaining the constraint cost function of the color prediction model includes:
the constrained cost function is determined based on the specular reflection color parameter.
In one embodiment of the present disclosure, the obtaining the constraint cost function of the color prediction model includes:
and determining the constraint cost function based on the diffuse reflection color difference value of the same position point on the surface of the target object under different viewing angles.
In one embodiment of the disclosure, the three-dimensional structural model of the target object is obtained by three-dimensionally reconstructing the target object by using an isosurface extraction method.
In a second aspect of embodiments of the present disclosure, there is provided an object three-dimensional reconstruction apparatus including:
the information acquisition module is used for acquiring a three-dimensional coordinate set of a pixel point set on the surface of a target object under a preset three-dimensional coordinate system and acquiring camera sight line sets of virtual cameras respectively facing all pixel points in the pixel point set under the preset coordinate system;
the model processing module is used for processing the three-dimensional coordinate set by using a diffuse reflection color prediction network in a color prediction model to obtain a diffuse reflection color value set of the pixel point set, and processing the three-dimensional coordinate set and the camera sight set by using a specular reflection color prediction network in the color prediction model to obtain a specular reflection color value set of the pixel point set;
and the color model determining module is used for determining a three-dimensional structure color model of the target object based on the three-dimensional structure model of the target object, the diffuse reflection color value set and the specular reflection color value set.
In one embodiment of the disclosure, the information obtaining module is configured to obtain a camera three-dimensional coordinate of a virtual camera under the preset coordinate system, and determine the camera sight line set based on the camera three-dimensional coordinate and the three-dimensional coordinate set; the camera sight of the virtual camera towards any pixel point in the pixel point set is obtained by the following steps: and determining the camera sight of the virtual camera towards any pixel point based on the camera three-dimensional coordinates and the pixel three-dimensional coordinates of any pixel point.
In one embodiment of the present disclosure, the object three-dimensional reconstruction apparatus further includes:
the model training module is used for determining a color supervision loss function of the color prediction model based on diffuse reflection color parameters and specular reflection color parameters, determining a final loss function of the color prediction model based on the color supervision loss function and the constraint cost function, and training the color prediction model based on three-dimensional coordinates of sample pixel points, diffuse reflection color labels and diffuse reflection color labels of the sample pixel points under the constraint of the final loss function.
In one embodiment of the disclosure, the model training module is configured to determine the constraint cost function based on a difference between a diffuse reflectance color and a true diffuse reflectance color.
In one embodiment of the disclosure, the model training module is to determine the constrained cost function based on the specular reflection color parameters.
In one embodiment of the disclosure, the model training module is configured to determine the constraint cost function based on diffuse reflection color differences of the same location point on the surface of the target object observed at different viewing angles.
In one embodiment of the disclosure, the three-dimensional structural model of the target object is obtained by three-dimensionally reconstructing the target object by using an isosurface extraction method.
In a third aspect of the disclosed embodiments, there is provided an electronic device, including:
a memory for storing a computer program product;
a processor for executing the computer program product stored in the memory, and when the computer program product is executed, implementing the method according to the first aspect.
A fourth aspect of an embodiment of the present disclosure provides a computer readable storage medium having stored thereon computer program instructions, wherein the computer program instructions, when executed by a processor, implement the method according to the first aspect.
According to the object three-dimensional reconstruction method and device, the electronic equipment and the storage medium, a diffuse reflection prediction network for predicting static colors of the surface of a target object is utilized to process a three-dimensional coordinate set of a pixel point set on the surface of the target object to obtain a diffuse reflection color value set of the pixel point set, a specular reflection color prediction network for predicting dynamic colors of the surface of the target object is utilized to process the pixel point set and a camera sight line of the surface of the target object to obtain a specular reflection color value set of the pixel point set, and then a three-dimensional structure model of the target object is colored according to the diffuse reflection color value set and the specular reflection color value set to obtain the three-dimensional structure color model of the target object with accurate color information.
The technical scheme of the present disclosure is described in further detail below through the accompanying drawings and examples.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description, serve to explain the principles of the disclosure.
The disclosure may be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings in which:
FIG. 1 is a flow chart of a method of three-dimensional reconstruction of an object in one embodiment of the present disclosure;
FIG. 2 is a block diagram of a three-dimensional reconstruction apparatus for an object in one embodiment of the present disclosure;
fig. 3 is a block diagram of an electronic device in one embodiment of the present disclosure.
Detailed Description
Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless it is specifically stated otherwise.
It will be appreciated by those of skill in the art that the terms "first," "second," etc. in embodiments of the present disclosure are used merely to distinguish between different steps, devices or modules, etc., and do not represent any particular technical meaning nor necessarily logical order between them.
It should also be understood that in embodiments of the present disclosure, "plurality" may refer to two or more, and "at least one" may refer to one, two or more.
It should also be appreciated that any component, data, or structure referred to in the presently disclosed embodiments may be generally understood as one or more without explicit limitation or the contrary in the context.
In addition, the term "and/or" in this disclosure is merely an association relationship describing an association object, and indicates that three relationships may exist, for example, a and/or B may indicate: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" in the present disclosure generally indicates that the front and rear association objects are an or relationship.
It should also be understood that the description of the various embodiments of the present disclosure emphasizes the differences between the various embodiments, and that the same or similar features may be referred to each other, and for brevity, will not be described in detail.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses.
Techniques, methods, and apparatus known to one of ordinary skill in the relevant art may not be discussed in detail, but are intended to be part of the specification where appropriate.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further discussion thereof is necessary in subsequent figures.
Embodiments of the present disclosure may be applicable to electronic devices such as terminal devices, computer systems, servers, etc., which may operate with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known terminal devices, computing systems, environments, and/or configurations that may be suitable for use with the terminal device, computer system, server, or other electronic device include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, mainframe computer systems, and distributed cloud computing technology environments that include any of the foregoing, and the like.
Electronic devices such as terminal devices, computer systems, servers, etc. may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform particular tasks or implement particular abstract data types. The computer system/server may be implemented in a distributed cloud computing environment in which tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computing system storage media including memory storage devices.
FIG. 1 is a flow chart of a method of three-dimensional reconstruction of an object in one embodiment of the present disclosure. As shown in fig. 1, the three-dimensional reconstruction method of an object includes the following steps:
s1: and acquiring a three-dimensional coordinate set of the pixel point set on the surface of the target object under a preset three-dimensional coordinate system, and acquiring camera sight line sets of the virtual camera facing all the pixel points in the pixel point set under the preset coordinate system.
The preset three-dimensional coordinate system may be a three-dimensional coordinate system established with the preset position as the origin of the coordinate system, for example, a three-dimensional coordinate system established with the position of the virtual camera as the origin of the coordinate system, or a three-dimensional coordinate system established with the position of the center point of the target object as the origin of the coordinate system.
All the pixels on the surface of the target object form a pixel point set. The position of each pixel point in the pixel point set has a three-dimensional coordinate under a preset coordinate system, and the three-dimensional coordinates of all the pixel points in the pixel point set under the preset coordinate system form a three-dimensional coordinate set.
The virtual camera faces each pixel point in the pixel point set respectively, and a plurality of camera sights are obtained. Wherein each pixel point in the pixel point set corresponds to a camera line of sight. The start point of each camera view may be the position of the virtual camera and the end point of each camera view may be the position of one pixel in the set of pixels. A camera view set is composed of a plurality of camera views.
The three-dimensional coordinate set and the camera view line set are given before step S1, and the three-dimensional coordinate set and the camera view line set can be acquired when step S1 is executed.
S2: and processing the three-dimensional coordinate set by using a diffuse reflection color prediction network in the color prediction model to obtain a diffuse reflection color value set of the pixel point set, and processing the three-dimensional coordinate set and the camera sight set by using a specular reflection color prediction network in the color prediction model to obtain a specular reflection color value set of the pixel point set.
Prior to step S2, a color prediction model is trained in advance. Wherein the color prediction model comprises: a diffuse reflection prediction network for predicting the static color of the surface of the target object and a specular reflection color prediction network for predicting the dynamic color of the surface of the target object. The diffuse reflection prediction network inputs three-dimensional coordinates of a given pixel and outputs diffuse reflection color values of the given pixel. The inputs to the specular reflection prediction network are the three-dimensional coordinates of the given pixel and the camera line of sight, and the inputs are the specular reflection color values of the given pixel.
And when the step S2 is executed, the diffuse reflection color prediction network is utilized to process the three-dimensional coordinate set, so that a diffuse reflection color value set corresponding to the pixel point set can be obtained. Wherein each pixel in the set of pixels corresponds to one diffuse reflection color value in the set of diffuse reflection color values.
And processing the three-dimensional coordinate set and the camera sight set by using a specular reflection color prediction network, so that a specular reflection color value set corresponding to the pixel point set can be obtained. Wherein each pixel in the set of pixels corresponds to one specular reflection color value in the set of specular reflection color values.
S3: and determining the three-dimensional structure color model of the target object based on the three-dimensional structure model, the diffuse reflection color value set and the specular reflection color value set of the target object.
A three-dimensional structural model of the target object is acquired. The three-dimensional structure model of the target object can be obtained by three-dimensional reconstruction according to a plurality of images comprising the target object.
And coloring the three-dimensional structure model of the target object according to the diffuse reflection color value set and the specular reflection color value set, so as to obtain the three-dimensional structure color model of the target object with color information.
In this embodiment, a diffuse reflection prediction network for predicting a static color of a surface of a target object is used to process a three-dimensional coordinate set of a pixel point set on the surface of the target object to obtain a diffuse reflection color value set of the pixel point set, and a specular reflection color prediction network for predicting a dynamic color of the surface of the target object is used to process the pixel point set and a camera sight line on the surface of the target object to obtain a specular reflection color value set of the pixel point set, and further, a three-dimensional structure model of the target object is colored according to the diffuse reflection color value set and the specular reflection color value set to obtain a three-dimensional structure color model of the target object with accurate color information.
In one embodiment of the present disclosure, in step S1, obtaining a camera line-of-sight set of a virtual camera toward all pixels in a set of pixels respectively in a preset coordinate system includes:
s1-1: and acquiring the three-dimensional coordinates of the camera of the virtual camera under a preset coordinate system.
And determining the three-dimensional coordinates of the camera based on the spatial position relation between the position of the virtual camera and a preset coordinate system.
S1-2: a set of camera views is determined based on the camera three-dimensional coordinates and the set of three-dimensional coordinates.
The camera sight of the virtual camera towards any pixel point in the pixel point set is obtained by the following steps: and determining the camera sight of the virtual camera towards any pixel point based on the camera three-dimensional coordinates and the pixel three-dimensional coordinates of any pixel point.
In this embodiment, under a preset coordinate system, according to the three-dimensional coordinates of the camera of the virtual camera and the three-dimensional coordinates of the pixel point sets, the camera sight sets of the virtual camera, which respectively face all the pixel points in the pixel point sets, can be rapidly determined.
Before step S2, the method further includes:
s0-1: a color monitor loss function of the color prediction model is determined based on the diffuse reflectance color parameter and the specular reflectance color parameter.
And setting a diffuse reflection color parameter d to correspond to diffuse reflection colors of three channels output by the diffuse reflection color prediction network. And setting a specular reflection color parameter s to correspond to the specular reflection color of the three channels output by the specular reflection color prediction network. Color supervision Loss function Loss of color prediction model in training process c The following formula may be used:
where c represents the color output by the color prediction model for one pixel in the set of pixels.
S0-2: and obtaining a constraint cost function of the color prediction model. Wherein the constraint cost function of the color prediction model can be a constraint cost function Loss based on diffuse reflection assumption reg 。
The assumption based on diffuse reflection may include one of the following:
1. the diffuse reflection color is as close to the true color as possible;
2. specular reflection is as small as possible;
3. and diffuse reflection across viewing angles.
For the assumption in 3 above, a corresponding constraint cost function L can be setoss reg 。
S0-3: and determining a final Loss function Loss of the color prediction model based on the color supervision Loss function and the constraint cost function.
Loss＝Loss c +Loss reg 。
S0-4: and training a color prediction model based on the three-dimensional coordinates of the sample pixel points, the diffuse reflection color labels of the sample pixel points and the diffuse reflection color labels under the constraint of a final Loss function Loss.
In this embodiment, a color supervision loss function during training of the color prediction model is set according to the diffuse reflection color parameter and the specular reflection color parameter, and a constraint cost function during training of the color prediction model can be set based on diffuse reflection assumption, so that the color prediction model including the diffuse reflection color prediction network and the specular reflection color prediction network can be quickly and reasonably trained under the common constraint of the color supervision loss function and the constraint cost function, and a diffuse reflection color value set and a specular reflection color value set corresponding to a pixel point set on the surface of a target object can be accurately obtained by using the color prediction model, thereby being beneficial to obtaining a three-dimensional structure color model of the target object with accurate color information.
In one embodiment of the present disclosure, S0-2 comprises: a constraint cost function is determined based on a difference between the diffuse reflectance color and the true diffuse reflectance color.
Hypothesis 1 corresponding to S0-2 constrains the cost function Loss reg The following are provided:
Loss reg ＝|c-d|。
in this embodiment, the constraint cost function set based on the assumption that specular reflection is as small as possible is helpful to train the diffuse reflection color prediction network to predict that the diffuse reflection color of the pixel points in the pixel point set approximates to the true color.
In another embodiment of the present disclosure, S0-2 comprises: a constrained cost function is determined based on the specular reflection color parameters.
Hypothesis 2 corresponding to S0-2 constrains the cost function Loss reg May be a two-range version of the specular color parameter,the method comprises the following steps:
Loss reg ＝||s|| 2
in this embodiment, the constraint cost function set based on the assumption that the diffuse reflection color approaches the true color as much as possible helps to train the specular reflection color prediction network to predict that the specular reflection of the pixel points in the pixel point set is as small as possible.
In yet another embodiment of the present disclosure, S0-2 comprises: and determining a constraint cost function based on the diffuse reflection color difference value of the same position point on the surface of the observed target object under different visual angles.
Hypothesis 3 corresponding to S0-2 constrains the cost function Loss reg May be determined based on the view index, as follows:
Loss reg ＝∑ j∈i |d i -d j |。
wherein i, j represents view index, namely different view index numbers of the same pixel point on the surface of the target object which can be watched for different view angles, d i And d j Representing the diffuse reflection colors of view indices i and j, respectively.
Based on constraint cost function set by the assumption of consistent diffuse reflection across viewing angles, the diffuse reflection color prediction network is trained to predict that diffuse reflection pixel values of the same pixel point in the pixel point set are as equal as possible under different viewing angles.
In one embodiment of the present disclosure, the three-dimensional structure model of the target object is obtained by three-dimensionally reconstructing the target object by using a contour extraction (MC) method. In the MC method, it is assumed that the original data is a discrete three-dimensional spatially regular data field. The MC method processes cubes in the data field one by one, finds cubes intersected with the isosurface, and calculates the intersection point of the isosurface and the cube edge by adopting linear interpolation. And connecting the intersection points of the isosurfaces and the edges of the cube according to the relative positions of each vertex and the isosurfaces of the cube to generate the isosurfaces as an approximation representation of the isosurfaces in the cube.
In this embodiment, the three-dimensional reconstruction is performed on the target object by adopting the iso-surface extraction method, so that a three-dimensional structure model of the target object can be accurately obtained.
Fig. 2 is a block diagram of a three-dimensional reconstruction apparatus for an object in one embodiment of the present disclosure. As shown in fig. 2, the object three-dimensional reconstruction apparatus includes:
the information acquisition module 100 is configured to acquire a three-dimensional coordinate set of a pixel point set on a surface of a target object under a preset three-dimensional coordinate system, and acquire camera line-of-sight sets of virtual cameras respectively facing all pixel points in the pixel point set under the preset coordinate system;
the model processing module 200 is configured to process the three-dimensional coordinate set by using a diffuse reflection color prediction network in the color prediction model to obtain a diffuse reflection color value set of the pixel point set, and process the three-dimensional coordinate set and the camera sight line set by using a specular reflection color prediction network in the color prediction model to obtain a specular reflection color value set of the pixel point set;
the color model determining module 300 is configured to determine a three-dimensional structural color model of the target object based on the three-dimensional structural model, the diffuse reflection color value set, and the specular reflection color value set of the target object.
In one embodiment of the present disclosure, the information obtaining module 100 is configured to obtain a camera three-dimensional coordinate of the virtual camera under a preset coordinate system, and determine a camera line-of-sight set based on the camera three-dimensional coordinate and the three-dimensional coordinate set; the camera sight of the virtual camera towards any pixel point in the pixel point set is obtained by the following steps: and determining the camera sight of the virtual camera towards any pixel point based on the camera three-dimensional coordinates and the pixel three-dimensional coordinates of any pixel point.
In one embodiment of the present disclosure, the object three-dimensional reconstruction apparatus further includes:
the model training module is used for determining a color supervision loss function of the color prediction model based on the diffuse reflection color parameter and the specular reflection color parameter, determining a final loss function of the color prediction model based on the color supervision loss function and the constraint cost function, and training the color prediction model based on the three-dimensional coordinates of the sample pixel points, the diffuse reflection color labels of the sample pixel points and the diffuse reflection color labels under the constraint of the final loss function.
In one embodiment of the present disclosure, the model training module is to determine the constraint cost function based on a difference between the diffuse reflectance color and the true diffuse reflectance color.
In one embodiment of the present disclosure, the model training module is configured to determine a constraint cost function based on specular reflection color parameters.
In one embodiment of the present disclosure, the model training module is configured to determine a constraint cost function based on differences in diffuse reflection colors of the same location point on the surface of the object under different viewing angles.
In one embodiment of the present disclosure, the three-dimensional structural model of the target object is obtained by three-dimensionally reconstructing the target object using an isosurface extraction method.
It should be noted that, a specific implementation manner of the object three-dimensional reconstruction device according to the embodiment of the present disclosure is similar to a specific implementation manner of the object three-dimensional reconstruction method according to the embodiment of the present disclosure, and specific reference is made to a description of a part of the object three-dimensional reconstruction method, so that redundancy is reduced and a detailed description is omitted.
In addition, the embodiment of the disclosure also provides an electronic device, which comprises:
a memory for storing a computer program;
and a processor, configured to execute the computer program stored in the memory, where the computer program is executed to implement the method for three-dimensional reconstruction of an object according to any one of the embodiments of the present disclosure.
Next, an electronic device according to an embodiment of the present disclosure is described with reference to fig. 3. As shown in fig. 3, the electronic device includes one or more processors and memory.
The processor may be a Central Processing Unit (CPU) or other form of processing unit having data processing and/or instruction execution capabilities, and may control other components in the electronic device to perform the desired functions.
The memory may store one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or nonvolatile memory. The volatile memory may include, for example, random Access Memory (RAM) and/or cache memory (cache), and the like. The non-volatile memory may include, for example, read Only Memory (ROM), hard disk, flash memory, and the like. One or more computer program products may be stored on the computer readable storage medium that can be run by a processor to implement the object three-dimensional reconstruction methods of the various embodiments of the present disclosure and/or other desired functions as described above.
In one example, the electronic device may further include: input devices and output devices, which are interconnected by a bus system and/or other forms of connection mechanisms (not shown).
In addition, the input device may include, for example, a keyboard, a mouse, and the like.
The output device may output various information including the determined distance information, direction information, etc., to the outside. The output device may include, for example, a display, speakers, a printer, and a communication network and remote output devices connected thereto, etc.
Of course, only some of the components of the electronic device relevant to the present disclosure are shown in fig. 3 for simplicity, components such as buses, input/output interfaces, etc. being omitted. In addition, the electronic device may include any other suitable components depending on the particular application.
In addition to the methods and apparatus described above, embodiments of the present disclosure may also be a computer program product comprising computer program instructions which, when executed by a processor, cause the processor to perform the steps in the method of three-dimensional reconstruction of an object according to the various embodiments of the present disclosure described in the above section of the specification.
The computer program product may write program code for performing the operations of embodiments of the present disclosure in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present disclosure may also be a computer-readable storage medium, having stored thereon computer program instructions, which when executed by a processor, cause the processor to perform the steps in the method of three-dimensional reconstruction of an object according to various embodiments of the present disclosure described in the above section of the present disclosure.
The computer readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The basic principles of the present disclosure have been described above in connection with specific embodiments, however, it should be noted that the advantages, benefits, effects, etc. mentioned in the present disclosure are merely examples and not limiting, and these advantages, benefits, effects, etc. are not to be considered as necessarily possessed by the various embodiments of the present disclosure. Furthermore, the specific details disclosed herein are for purposes of illustration and understanding only, and are not intended to be limiting, since the disclosure is not necessarily limited to practice with the specific details described.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different manner from other embodiments, so that the same or similar parts between the embodiments are mutually referred to. For system embodiments, the description is relatively simple as it essentially corresponds to method embodiments, and reference should be made to the description of method embodiments for relevant points.
The block diagrams of the devices, apparatuses, devices, systems referred to in this disclosure are merely illustrative examples and are not intended to require or imply that the connections, arrangements, configurations must be made in the manner shown in the block diagrams. As will be appreciated by one of skill in the art, the devices, apparatuses, devices, systems may be connected, arranged, configured in any manner. Words such as "including," "comprising," "having," and the like are words of openness and mean "including but not limited to," and are used interchangeably therewith. The terms "or" and "as used herein refer to and are used interchangeably with the term" and/or "unless the context clearly indicates otherwise. The term "such as" as used herein refers to, and is used interchangeably with, the phrase "such as, but not limited to.
The methods and apparatus of the present disclosure may be implemented in a number of ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, firmware. The above-described sequence of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the sequence specifically described above unless specifically stated otherwise. Furthermore, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, the programs including machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.
It is also noted that in the apparatus, devices and methods of the present disclosure, components or steps may be disassembled and/or assembled. Such decomposition and/or recombination should be considered equivalent to the present disclosure.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the disclosure to the form disclosed herein. Although a number of example aspects and embodiments have been discussed above, a person of ordinary skill in the art will recognize certain variations, modifications, alterations, additions, and subcombinations thereof.
Claims (10)
1. A method for three-dimensional reconstruction of an object, comprising:
acquiring a three-dimensional coordinate set of a pixel point set on the surface of a target object under a preset three-dimensional coordinate system, and acquiring camera sight line sets of virtual cameras respectively facing all pixel points in the pixel point set under the preset coordinate system;
processing the three-dimensional coordinate set by using a diffuse reflection color prediction network in a color prediction model to obtain a diffuse reflection color value set of the pixel point set, and processing the three-dimensional coordinate set and the camera sight set by using a specular reflection color prediction network in the color prediction model to obtain a specular reflection color value set of the pixel point set;
and determining a three-dimensional structure color model of the target object based on the three-dimensional structure model of the target object, the diffuse reflection color value set and the specular reflection color value set.
2. The method according to claim 1, wherein the acquiring a set of camera lines of sight of the virtual camera towards all pixels in the set of pixels respectively in the preset coordinate system comprises:
acquiring a camera three-dimensional coordinate of the virtual camera under the preset coordinate system;
determining the camera gaze set based on the camera three-dimensional coordinates and the three-dimensional coordinate set;
the camera sight of the virtual camera towards any pixel point in the pixel point set is obtained by the following steps: and determining the camera sight of the virtual camera towards any pixel point based on the camera three-dimensional coordinates and the pixel three-dimensional coordinates of any pixel point.
3. The method of claim 1, wherein prior to processing the three-dimensional coordinate set with the diffuse reflection color prediction network in the color prediction model to obtain a diffuse reflection color value set for the set of pixels, and processing the three-dimensional coordinate set and the camera view set with the specular reflection color prediction network in the color prediction model to obtain a specular reflection color value set for the set of pixels, further comprising:
determining a color supervision loss function of the color prediction model based on the diffuse reflection color parameter and the specular reflection color parameter;
obtaining a constraint cost function of the color prediction model;
determining a final loss function of the color prediction model based on the color supervised loss function and the constrained cost function;
and training the color prediction model based on the three-dimensional coordinates of the sample pixel points, and the diffuse reflection color labels of the sample pixel points under the constraint of the final loss function.
4. A method according to claim 3, wherein said obtaining a constrained cost function of said color prediction model comprises:
the constraint cost function is determined based on a difference between the diffuse reflection color and the true diffuse reflection color.
5. A method according to claim 3, wherein said obtaining a constrained cost function of said color prediction model comprises:
the constrained cost function is determined based on the specular reflection color parameter.
6. A method according to claim 3, wherein said obtaining a constrained cost function of said color prediction model comprises:
and determining the constraint cost function based on the diffuse reflection color difference value of the same position point on the surface of the target object under different viewing angles.
7. The method according to any one of claims 1 to 6, wherein the three-dimensional structural model of the target object is obtained by three-dimensionally reconstructing the target object by using an isosurface extraction method.
8. A three-dimensional reconstruction apparatus for an object, comprising:
the information acquisition module is used for acquiring a three-dimensional coordinate set of a pixel point set on the surface of a target object under a preset three-dimensional coordinate system and acquiring camera sight line sets of virtual cameras respectively facing all pixel points in the pixel point set under the preset coordinate system;
the model processing module is used for processing the three-dimensional coordinate set by using a diffuse reflection color prediction network in a color prediction model to obtain a diffuse reflection color value set of the pixel point set, and processing the three-dimensional coordinate set and the camera sight set by using a specular reflection color prediction network in the color prediction model to obtain a specular reflection color value set of the pixel point set;
and the color model determining module is used for determining a three-dimensional structure color model of the target object based on the three-dimensional structure model of the target object, the diffuse reflection color value set and the specular reflection color value set.
9. An electronic device, comprising:
a memory for storing a computer program product;
a processor for executing a computer program product stored in said memory, which, when executed, implements the method of any of the preceding claims 1-7.
10. A computer readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the method of any of the preceding claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311141384.4A CN117095129A (en) | 2023-09-05 | 2023-09-05 | Object three-dimensional reconstruction method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311141384.4A CN117095129A (en) | 2023-09-05 | 2023-09-05 | Object three-dimensional reconstruction method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117095129A true CN117095129A (en) | 2023-11-21 |
Family
ID=88769684
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311141384.4A Pending CN117095129A (en) | 2023-09-05 | 2023-09-05 | Object three-dimensional reconstruction method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117095129A (en) |
-
2023
- 2023-09-05 CN CN202311141384.4A patent/CN117095129A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11842438B2 (en) | Method and terminal device for determining occluded area of virtual object | |
| US10854007B2 (en) | Space models for mixed reality | |
| US11222471B2 (en) | Implementing three-dimensional augmented reality in smart glasses based on two-dimensional data | |
| US20210241495A1 (en) | Method and system for reconstructing colour and depth information of a scene | |
| CN106846497B (en) | Method and device for presenting three-dimensional map applied to terminal | |
| CA3138907A1 (en) | System and method for interactive projection | |
| US11526272B2 (en) | Systems and methods for interactive image caricaturing by an electronic device | |
| US9171393B2 (en) | Three-dimensional texture reprojection | |
| CN113379885B (en) | Virtual hair processing method and device, readable storage medium and electronic equipment | |
| US20220358735A1 (en) | Method for processing image, device and storage medium | |
| CN113689508B (en) | Point cloud labeling method and device, storage medium and electronic equipment | |
| Canessa et al. | A dataset of stereoscopic images and ground-truth disparity mimicking human fixations in peripersonal space | |
| CN114047823A (en) | Three-dimensional model display method, computer-readable storage medium and electronic device | |
| CN111161398B (en) | Image generation method, device, equipment and storage medium | |
| CN113129362B (en) | Method and device for acquiring three-dimensional coordinate data | |
| CN115512046B (en) | Panorama display method and device for points outside model, equipment and medium | |
| CN112950759A (en) | Three-dimensional house model construction method and device based on house panoramic image | |
| US10754498B2 (en) | Hybrid image rendering system | |
| CN117095129A (en) | Object three-dimensional reconstruction method and device, electronic equipment and storage medium | |
| CN113762173B (en) | Training method and device for face optical flow estimation and optical flow value prediction model | |
| US11281935B2 (en) | 3D object detection from calibrated 2D images | |
| US20210097731A1 (en) | Presenting environment based on physical dimension | |
| US20240161391A1 (en) | Relightable neural radiance field model | |
| CN117979070A (en) | Method and device for sharing display picture of head-mounted display device | |
| Beňo et al. | RGBD mapping solution for low-cost robot |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |