CN110390717B - 3D model reconstruction method and device and electronic equipment - Google Patents
3D model reconstruction method and device and electronic equipment Download PDFInfo
- Publication number
- CN110390717B CN110390717B CN201910591658.7A CN201910591658A CN110390717B CN 110390717 B CN110390717 B CN 110390717B CN 201910591658 A CN201910591658 A CN 201910591658A CN 110390717 B CN110390717 B CN 110390717B
- Authority
- CN
- China
- Prior art keywords
- model
- target object
- contour
- image
- patch
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30008—Bone
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Processing Or Creating Images (AREA)
- Image Processing (AREA)
Abstract
The embodiment of the disclosure provides a 3D model reconstruction method, a device and an electronic device, belonging to the technical field of data processing, wherein the method comprises the following steps: performing a segmentation operation on a first image selected from the image set to obtain a target object contained in the first image; performing bone detection on the target object to form a 3D bone model of the target object; determining a mapping corresponding to a triangular patch in the 3D skeleton model based on all pictures in the image set; determining a 3D reconstructed model of the target object based on the map and the 3D bone model. Through the scheme disclosed by the invention, the texture of the map can be generated based on a plurality of images, and the accuracy of reconstructing the texture of the 3D model is improved.
Description
Technical Field
The present disclosure relates to the field of 3D model reconstruction technologies, and in particular, to a 3D model reconstruction method and apparatus, and an electronic device.
Background
The 3D modeling based on the character is to set up a three-dimensional model of the character through a certain technical means, and a common 3D model reconstruction method comprises the steps of scanning the whole body based on high-precision hardware and establishing a human body 3D model according to scanning data. The 3D model can also be adjusted by professional artists according to the standard human body 3D model and the appearance of the target human body (recorded appearance of photos, videos and the like) to be as close to the 3D appearance of the target human body as possible. Or a GAN network is used for realizing similar functions, a human body 3D model is created, and a common use scene is to make the model perform some specified actions (such as dancing). Similar effects can be achieved using a competing network in deep learning (GAN). The method is to learn a GAN model of a character A through a plurality of image data of a target character (character A), and then generate the dancing action of the character A according to a 2D skeleton of the dancing action.
The above implementation has the following disadvantages: both modeling and mapping will have a distorted appearance. Since a single picture of the human body needs to be a full-body picture (or at least a picture of the exposed half of the body), the quality of the map is not high. Because the visual angle of a single photo is limited, and a part of human body cannot collect information due to some shelters, a large amount of GAN networks are used in the method to generate sheltered parts (including 3D structures and maps). Distortion is inevitably caused in this process.
Disclosure of Invention
In view of this, embodiments of the present disclosure provide a method, an apparatus, and an electronic device for reconstructing a 3D model, which at least partially solve the problems in the prior art.
In a first aspect, an embodiment of the present disclosure provides a 3D model reconstruction method, including:
performing a segmentation operation on a first image selected from the set of images to obtain a target object contained in the first image;
performing bone detection on the target object to form a 3D bone model of the target object;
determining a chartlet corresponding to a triangular patch in the 3D skeleton model based on all pictures in the image set;
determining a 3D reconstructed model of the target object based on the map and the 3D bone model.
According to a specific implementation manner of the embodiment of the present disclosure, the determining a map corresponding to a triangular patch in the 3D bone model based on all pictures in the image set includes:
searching a plurality of chartlet textures corresponding to the triangular patch in all pictures in the image set;
determining different weights for a plurality of chartlet textures based on different perspectives of the triangular patch;
determining a final map texture of the triangular patch based on different weights of the map texture.
According to a specific implementation manner of the embodiment of the present disclosure, determining the final mapping texture of the triangular patch based on different weights of the mapping texture includes:
each angle i capable of seeing the triangular patch is used for calculating the visible proportion ri of the triangular patch S;
all ri's are normalized so that ∑ k × r i =1, where k is the coefficient to be determined;
using k r i The weight of the triangular patch as the angle i when the texture of the patch is fused is recorded as wi;
all the maps of all the angles i of the triangular patch and the corresponding weights wi are used as input and input into the GAN network, and the GAN network outputs the maps as final map textures.
According to a specific implementation manner of the embodiment of the present disclosure, before all the maps of the angles i of the triangular patch and the weights wi corresponding to the maps are input to the GAN network, the method further includes:
setting an input layer GAN network capable of receiving a plurality of picture element progenitors, wherein each picture element progenitor comprises a picture and the weight of the picture;
overlapping all the picture element progenitors to form a high-dimensional picture tensor;
convolving the picture tensors and performing output concatenation according to weights, wherein the output concatenation is used as the input of the first pReLU layer in the GAN.
According to a specific implementation manner of the embodiment of the present disclosure, the method further includes:
acquiring any high-definition picture, randomly converting the high-definition picture in a 3D space, and calculating projection on a 2D picture after conversion;
the projection of a high-definition picture on a 2D picture is converted into the high-definition picture which is opposite to a camera, so that a low-definition picture is formed;
the GAN network is trained using the plurality of low-definition pictures and the pictures that generate their high definition as a training set.
According to a specific implementation manner of the embodiment of the present disclosure, the performing bone detection on the target object to form a 3D bone model of the target object includes:
projecting a preset 3D initial model onto a 2D plane to form a model outline;
forming an input contour of the target object based on the segmentation mask;
respectively setting a first key point set and a second key point set on the input contour and the model contour, wherein the number of key points in the first key point set is the same as that of the key points in the second key point set, and the first key point set and the second key point set are in one-to-one correspondence;
calculating the corresponding plane coordinates (xi, yi) of any key point pi in the model contour in the input contour;
transforming the space coordinate zi of any key point pi in the 3D initial model according to the change proportion of the input contour and the model contour on the length and the width;
determining a final shape of the 3D bone model based on the planar coordinates and the spatial coordinates.
According to a specific implementation manner of the embodiment of the present disclosure, the determining a 3D reconstructed model of the target object based on the map and the 3D bone model includes:
confirming camera coordinates in a 3D space coordinate system, so that a contour formed by projection of the 3D bone model on a display picture is completely coincided with the input contour in the camera coordinates;
determining 3 2D corresponding points of three vertexes of any triangular patch forming the 3D skeleton model on the original image corresponding to the input contour;
determining a mapping patch on the first image by using the corresponding point;
taking the mapping patch as a mapping of a 3D triangular patch corresponding to the mapping patch, and mapping the mapping patch on the first image on the 3D triangular patch;
a 3D bone model comprising a mapped patch on a first image is used as a first 3D reconstructed model of the target object.
According to a specific implementation manner of the embodiment of the present disclosure, the performing bone detection on the target object to form a 3D bone model of the target object includes:
projecting a preset 3D initial model onto a 2D plane to form a model outline;
forming an input contour of the target object based on the segmentation mask of the target object;
forming a 3D bone model of the target object based on the input contour and the model contour.
In a second aspect, an embodiment of the present disclosure provides a 3D model reconstruction apparatus, including:
a segmentation module, configured to perform a segmentation operation on a first image selected from the image set to obtain a target object included in the first image;
a forming module for performing bone detection on the target object, forming a 3D bone model of the target object;
a first determining module, configured to determine, based on all pictures in the image set, a map corresponding to a triangular patch in the 3D bone model;
a second determination module for determining a 3D reconstructed model of the target object based on the map and the 3D bone model.
In a third aspect, an embodiment of the present disclosure further provides an electronic device, where the electronic device includes:
at least one processor; and (c) a second step of,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of reconstructing a 3D model of any one of the preceding first aspects or any implementation manner of the first aspect.
In a fourth aspect, the disclosed embodiments also provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the 3D model reconstruction method of the first aspect or any implementation manner of the first aspect.
In a fifth aspect, the embodiments of the present disclosure further provide a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the 3D model reconstruction method in the foregoing first aspect or any implementation manner of the first aspect.
The 3D model reconstruction scheme in the embodiment of the present disclosure includes performing a segmentation operation on a first image selected from an image set to obtain a target object included in the first image; performing bone detection on the target object to form a 3D bone model of the target object; determining a mapping corresponding to a triangular patch in the 3D skeleton model based on all pictures in the image set; determining a 3D reconstructed model of the target object based on the map and the 3D bone model. According to the scheme, the texture of the map can be generated based on a plurality of images, and the accuracy of reconstructing the texture of the 3D model is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings needed to be used in the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present disclosure, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic view of a 3D model reconstruction process provided in an embodiment of the present disclosure;
fig. 2 is a schematic view of another 3D model reconstruction process provided in an embodiment of the present disclosure;
fig. 3 is a schematic view of another 3D model reconstruction process provided in the embodiment of the present disclosure;
fig. 4 is a schematic view of another 3D model reconstruction process provided in an embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of a 3D model reconstruction apparatus provided in an embodiment of the present disclosure;
fig. 6 is a schematic diagram of an electronic device provided in an embodiment of the present disclosure.
Detailed Description
The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.
The embodiments of the present disclosure are described below with specific examples, and other advantages and effects of the present disclosure will be readily apparent to those skilled in the art from the disclosure in the specification. It is to be understood that the described embodiments are merely illustrative of some, and not restrictive, of the embodiments of the disclosure. The disclosure may be embodied or carried out in various other specific embodiments, and various modifications and changes may be made in the details within the description without departing from the spirit of the disclosure. It should be noted that the features in the following embodiments and examples may be combined with each other without conflict. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.
It is noted that various aspects of the embodiments are described below within the scope of the appended claims. It should be apparent that the aspects described herein may be embodied in a wide variety of forms and that any specific structure and/or function described herein is merely illustrative. Based on the disclosure, one skilled in the art should appreciate that one aspect described herein may be implemented independently of any other aspects and that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented and/or a method practiced using any number of the aspects set forth herein. In addition, such an apparatus may be implemented and/or such a method may be practiced using other structure and/or functionality in addition to or other than one or more of the aspects set forth herein.
It should be further noted that the drawings provided in the following embodiments are only schematic illustrations of the basic concepts of the present disclosure, and the drawings only show the components related to the present disclosure rather than the numbers, shapes and dimensions of the components in actual implementation, and the types, the numbers and the proportions of the components in actual implementation may be arbitrarily changed, and the layout of the components may be more complicated.
In addition, in the following description, specific details are provided to provide a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects may be practiced without these specific details.
The embodiment of the disclosure provides a 3D model reconstruction method. The 3D model reconstruction method provided by the present embodiment may be executed by a computing device, which may be implemented as software, or implemented as a combination of software and hardware, and may be integrally provided in a server, a terminal device, or the like.
Referring to fig. 1 and fig. 2, a 3D model reconstruction method provided in an embodiment of the present disclosure includes the following steps:
s101, segmentation operation is carried out on a first image selected from the image set so as to obtain a target object contained in the first image.
The image set is a set including a plurality of images, each of which includes a target object, for example, the image may be a photograph including a person or a video frame image extracted from a video including a person, the image is a static image and can be subjected to image processing, the first image is any one of the image sets, and the first image includes the target object.
The target object is an object existing in the image, and the target object may be various objects that can be described by a 3D model, such as a person, an animal, and an automobile, and the target object exists in the image in a 2D form.
The target object usually occupies a certain area in the image, and for this reason, a segmentation operation needs to be performed on the image, and by the segmentation operation, the contour of the target object on the image can be acquired, and by the contour, the existence region of the target object can be determined, so as to further determine the segmentation mask of the target object. The segmentation operation on the target object can adopt various segmentation algorithms existing in the prior art, and the manner of the segmentation operation is not limited herein.
S102, carrying out bone detection on the target object to form a 3D bone model of the target object.
After the contour of the target object is obtained, 2D bone detection may be performed on the target object based on the contour of the target object, for example, the body posture of the target object may be analyzed in a deep learning manner, and texture information and spatial information may be expressed by a convolution layer. The network structure of deep learning is divided into a plurality of stages, wherein the first stage can generate a preliminary key point detection effect, and the next stages take the prediction output of the previous stage and the characteristics extracted from the original image as input, so that the key point detection effect is further improved. In addition to the above 2D bone detection method, other 2D bone detection methods may be used, which are not limited herein.
Next, the 3D model (initial model) that has been created is projected onto the 2D plane, thus forming a model outline silhouette in the 2D plane. And respectively carrying out projection operation on the input contour silhouette and the model contour silhouette of the target object in the input image to form an input contour and a model contour.
And taking the same number of points on the input contour and the model contour, ensuring that the points are in one-to-one correspondence, forming a point set P by the points on the model contour, and converting the point set into a point set on the input contour through a conversion formula. In this way, the corresponding coordinates of any point in the model contour in the input contour can be determined.
The x and y coordinates of the vertex coordinates of all triangular patches in the 3D model are processed so that the silhouette of the new 3D model can be completely coincident with the input silhouette. And transforming the z coordinate in the 3D model according to the change proportion of the input contour and the model contour in length and width.
Through the above steps, the physique of the changed 3D model is consistent with the physique of the human body in the picture. The 3D human body reconstruction can be more vivid.
S103, determining a mapping corresponding to a triangular patch in the 3D skeleton model based on all pictures in the image set.
The 3D skeleton model consists of a plurality of triangular patches, in the solution of the application, a visual scale is calculated for each triangular patch. The view scale of each triangle patch is defined as r = SV/ST, where SV is the area occupied by the projection of the triangle patch on the 2D picture at the current view angle, and ST is the real area of the triangle patch. While all the triangular patches are mapped, the visual scale of each triangular patch needs to be recorded.
For the same triangular patch S, there may be multiple frames that can see the triangular patch S, and therefore, these frames also all have the texture of the S. For each angle i at which a triangular patch S can be seen, the visible scale of S is calculated as ri, and all ri are first normalized so that ∑ k × r i =1, where k is the coefficient to be determined. Then k r is used i The weight of the triangle patch at the angle i when the patch textures are merged is denoted as wi.
And inputting all the maps of the angles i of the triangular patch S and the weights wi corresponding to the angles i into the GAN network, outputting a high-definition map of the triangular patch S, and determining the map as the map corresponding to the triangular patch in the 3D skeleton model.
S104, determining a 3D reconstruction model of the target object based on the map and the 3D bone model.
And matching the calculated mapping with the corresponding triangular patch in the 3D skeleton model until all the triangular patches in the 3D skeleton model are matched with the corresponding patches, so as to form a 3D reconstruction model corresponding to the target object.
Specifically, suitable camera coordinates in a 3D space coordinate system are found, so that in the camera coordinates, the contour formed by projection of the 3D bone model on the visualization picture and the input contour are completely overlapped.
For each triangular patch on the 3D bone model, three vertices are used for representation, specifically (xi, yi, zi) where i =1, \ 8230;, 3. Since the contours are completely coincident, these three points can find the corresponding 3 2D points on the input image framed by the contours, i.e. (xi, yi) where i =1, \ 8230;, 3.
The three 2D points on the original image (input image) can extract a map from the input image, the map is used as the map of the corresponding 3D triangular patch, and the triangular patch in the original image can be pasted on the 3D triangular patch by calculating the difference value during pasting.
Since a single picture has only one view angle, not all triangle patches in the 3D skeleton model can find the corresponding triangle patch in the 2D input image (because the model is self-occluded and is on the back of the model), which is called an invisible patch. At this time, since the model of the human body is a 3D single communication domain, for a triangle patch that is occluded and located on the back of the model, a corresponding triangle patch that is located on the front and is not occluded can be always found, and this triangle patch is called a visible patch. In this modeling approach, the invisible patches are mapped using the maps on the corresponding visible patches.
By the method, the following beneficial effects can be achieved: 1. the target object (such as a person) to be modeled only needs simple cooperation, a plurality of pictures are taken for the whole body, or a video is recorded for the whole body, and the sample collection mode is simple. 2. Low cost and no need of special hardware. 3. The method is fully automatic, and the whole modeling process does not need manual intervention. 4. The obtained human body 3D model can be applied to various scenes, and the use aspect is not limited. 5. The human body 3D model with higher trueness and more accurate precision is obtained, which is superior to the human body 3D reconstruction only using a single photo. Meanwhile, by a novel texture conflict problem solving method, a texture map with higher precision can be obtained, and a clearer human body modeling effect is achieved.
Referring to fig. 3, according to a specific implementation manner of the embodiment of the present disclosure, determining a map corresponding to a triangular patch in the 3D bone model based on all pictures in the image set may include the following steps:
s301, searching a plurality of chartlet textures corresponding to the triangular patch in all pictures in the image set.
The 3D skeleton model consists of a plurality of triangular patches, for each of which a visual scale is calculated. The view scale of each triangular patch is defined as r = SV/ST, where SV is the area occupied by the projection of the triangular patch on the 2D picture at the current view angle, and ST is the real area of the triangular patch. While all the triangular patches are mapped, the visual scale of each triangular patch needs to be recorded. For the same triangular patch S, there may be multiple frames that can see the triangular patch S, and therefore, these frames also have the mapping texture of S, and for this reason, all the mapping textures corresponding to the triangular patch can be searched in the image set.
S302, determining different weights of a plurality of chartlet textures based on different visual angles of the triangular patch.
For each angle i at which a triangular patch S can be seen, compute the visible scale of S as ri, first normalize all ri so that Σ k × r i =1, where k is the coefficient to be determined. Then k r is used i Let wi be the weight of the triangular patch at angle i when merging the patch textures.
S303, determining the final mapping texture of the triangular patch based on different weights of the mapping texture.
And inputting all the maps of the angles i of the triangular patch S and the weights wi corresponding to the angles i into the GAN network, outputting a high-definition map of the triangular patch S, and determining the map as the map corresponding to the triangular patch in the 3D skeleton model.
According to a specific implementation manner of the embodiment of the present disclosure, referring to fig. 4, before all the maps of the angles i of the triangular patch and the corresponding weights wi are input to the GAN network, the method further includes:
s401, an input layer GAN network capable of receiving a plurality of primitive progenitors is configured, wherein each primitive progenitor includes a picture and a weight of the picture.
S402, overlapping all the picture element progenitors to form a high-dimensional picture tensor.
And S403, convolving the picture tensors and performing output concatenation according to weights, wherein the output concatenation is used as the input of the first pReLU layer in the GAN.
Before the prediction of the GAN network, a training data set of the GAN network needs to be obtained, so that any high-definition picture can be obtained, the high-definition picture is randomly transformed in a 3D space, and the projection on a 2D picture after transformation is calculated; transforming the projection of the high-definition picture on the 2D picture into the high-definition picture which is just opposite to the camera to form a low-definition picture; the GAN network is trained using the plurality of low-resolution pictures and the high-resolution pictures from which they were generated as a training set.
According to a specific implementation manner of the embodiment of the present disclosure, the performing bone detection on the target object to form a 3D bone model of the target object includes: projecting a preset 3D initial model onto a 2D plane to form a model outline; forming an input contour of the target object based on the segmentation mask; respectively setting a first key point set and a second key point set on the input contour and the model contour, wherein the number of key points in the first key point set is the same as that in the second key point set, and the first key point set and the second key point set correspond to each other one by one; calculating the corresponding plane coordinates (xi, yi) of any key point pi in the model contour in the input contour; according to the change proportion of the input contour and the model contour on the length and the width, converting the space coordinate zi of any key point pi in the 3D initial model; determining a final shape of the 3D bone model based on the planar coordinates and the spatial coordinates.
According to a specific implementation manner of the embodiment of the present disclosure, the determining a 3D reconstructed model of the target object based on the map and the 3D bone model includes: confirming camera coordinates in a 3D space coordinate system, so that a contour formed by projection of the 3D bone model on a display picture is completely coincided with the input contour in the camera coordinates; determining 3 2D corresponding points of three vertexes of any triangular patch forming the 3D skeleton model on the original graph corresponding to the input contour; determining a mapping patch on the first image by using the corresponding point; taking the mapping patch as a mapping of a 3D triangular patch corresponding to the mapping patch, and mapping the mapping patch on the first image on the 3D triangular patch; a 3D bone model comprising a mapped patch on a first image is used as a first 3D reconstructed model of the target object.
According to a specific implementation manner of the embodiment of the present disclosure, the performing bone detection on the target object to form a 3D bone model of the target object includes: projecting a preset 3D initial model onto a 2D plane to form a model outline; forming an input contour of the target object based on the segmentation mask of the target object; forming a 3D bone model of the target object based on the input contour and the model contour.
Corresponding to the above method embodiment, referring to fig. 5, an embodiment of the present disclosure provides a 3D model reconstruction apparatus 50, including:
a segmentation module 501, configured to perform a segmentation operation on a first image selected from the image set to obtain a target object included in the first image;
a forming module 502 for performing bone detection on the target object, forming a 3D bone model of the target object;
a first determining module 503, configured to determine, based on all pictures in the image set, a map corresponding to a triangular patch in the 3D bone model;
a second determination module 504 for determining a 3D reconstructed model of the target object based on the map and the 3D bone model.
The apparatus shown in fig. 5 may correspondingly execute the contents in the foregoing method embodiment, and details of parts not described in detail in this embodiment refer to the contents described in the foregoing method embodiment, which are not repeated herein.
Referring to fig. 6, an embodiment of the present disclosure also provides an electronic device 60, which includes:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of 3D model reconstruction in the above method embodiments.
The disclosed embodiments also provide a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the foregoing method embodiments.
The disclosed embodiments also provide a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, cause the computer to perform the 3D model reconstruction method of the aforementioned method embodiments.
Referring now to FIG. 6, a block diagram of an electronic device 60 suitable for use in implementing embodiments of the present disclosure is shown. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., car navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 6, the electronic device 60 may include a processing means (e.g., central processing unit, graphics processor, etc.) 601 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage means 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the electronic apparatus 60 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
Generally, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touch pad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, or the like; output devices 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 608 including, for example, magnetic tape, hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 60 to communicate with other devices wirelessly or by wire to exchange data. While the figures illustrate an electronic device 60 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may be alternatively implemented or provided.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer-readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication means 609, or installed from the storage means 608, or installed from the ROM 602. The computer program, when executed by the processing device 601, performs the above-described functions defined in the methods of the embodiments of the present disclosure.
It should be noted that the computer readable medium of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring at least two internet protocol addresses; sending a node evaluation request comprising the at least two internet protocol addresses to node evaluation equipment, wherein the node evaluation equipment selects the internet protocol addresses from the at least two internet protocol addresses and returns the internet protocol addresses; receiving an internet protocol address returned by the node evaluation equipment; wherein the obtained internet protocol address indicates an edge node in the content distribution network.
Alternatively, the computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: receiving a node evaluation request comprising at least two internet protocol addresses; selecting an internet protocol address from the at least two internet protocol addresses; returning the selected internet protocol address; wherein the received internet protocol address indicates an edge node in the content distribution network.
Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. Where the name of a unit does not in some cases constitute a limitation of the unit itself, for example, the first obtaining unit may also be described as a "unit obtaining at least two internet protocol addresses".
It should be understood that portions of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof.
The above description is only for the specific embodiments of the present disclosure, but the scope of the present disclosure is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present disclosure should be covered within the scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.
Claims (9)
1. A method of 3D model reconstruction, comprising:
performing a segmentation operation on a first image selected from the set of images to obtain a target object contained in the first image;
performing bone detection on the target object to form a 3D bone model of the target object;
determining a mapping corresponding to a triangular patch in the 3D skeleton model based on all pictures in the image set;
determining a 3D reconstructed model of the target object based on the map and the 3D bone model;
performing bone detection on the target object to form a 3D bone model of the target object, including:
projecting a preset 3D initial model onto a 2D plane to form a model outline;
forming an input contour of the target object based on a segmentation mask;
respectively setting a first key point set and a second key point set on the input contour and the model contour, wherein the number of key points in the first key point set is the same as that of the key points in the second key point set, and the first key point set and the second key point set are in one-to-one correspondence;
calculating the corresponding plane coordinates (xi, yi) of any key point pi in the model contour in the input contour;
transforming the space coordinate zi of any key point pi in the 3D initial model according to the change proportion of the input contour and the model contour on the length and the width;
determining a final shape of the 3D bone model based on the planar coordinates and the spatial coordinates.
2. The method of claim 1, wherein determining the map corresponding to the triangular patch in the 3D bone model based on all the pictures in the image set comprises:
searching a plurality of chartlet textures corresponding to the triangular patch in all pictures in the image set;
determining different weights for a plurality of chartlet textures based on different perspectives of the triangular patch;
determining a final map texture of the triangular patch based on different weights of the map texture.
3. The method of claim 2, wherein determining the final texture of the triangular patch based on different weights of the texture of the patch comprises:
each angle i capable of seeing the triangular patch is used for calculating the visible proportion ri of the triangular patch S;
all ri's are normalized so that ∑ k × r i =1, where k is the coefficient to be determined;
using k r i The weight of the triangular patch serving as the angle i when the textures of the patches are fused is marked as wi;
all the maps of all the angles i of the triangular patch and the corresponding weights wi are used as input and input into the GAN network, and the output maps of the GAN network are used as final map textures.
4. The method of claim 3, wherein before inputting all the maps of the angles i of the triangular patches and their corresponding weights wi into the GAN network, the method further comprises:
setting an input layer GAN network capable of receiving a plurality of picture element progenitors, wherein each picture element progenitor comprises a picture and the weight of the picture;
overlapping all the picture element progenitors to form a high-dimensional picture tensor;
convolving the picture tensors and performing output concatenation according to weights, wherein the output concatenation is used as the input of the first pReLU layer in the GAN.
5. The method of claim 4, further comprising:
acquiring any high-definition picture, randomly converting the high-definition picture in a 3D space, and calculating projection on a 2D picture after conversion;
the projection of a high-definition picture on a 2D picture is converted into the high-definition picture which is opposite to a camera, so that a low-definition picture is formed;
the GAN network is trained using the plurality of low-resolution pictures and the high-resolution pictures from which they were generated as a training set.
6. The method of claim 1, wherein determining a 3D reconstructed model of the target object based on the map and the 3D bone model comprises:
confirming camera coordinates in a 3D space coordinate system, so that a contour formed by projection of the 3D bone model on a display picture is completely overlapped with an input contour in the camera coordinates;
determining 3 2D corresponding points of three vertexes of any triangular patch forming the 3D skeleton model on the original graph corresponding to the input contour;
determining a mapping patch on the first image by using the corresponding point;
taking the mapping patch as a mapping of a 3D triangular patch corresponding to the mapping patch, and mapping the mapping patch on the first image on the 3D triangular patch;
a 3D bone model comprising a mapped patch on a first image is used as a first 3D reconstructed model of the target object.
7. A 3D model reconstruction apparatus, comprising:
a segmentation module, configured to perform a segmentation operation on a first image selected from the image set to obtain a target object included in the first image;
the forming module is used for carrying out bone detection on the target object to form a 3D bone model of the target object, and specifically used for projecting a preset 3D initial model onto a 2D plane to form a model outline; forming an input contour of the target object based on a segmentation mask; respectively setting a first key point set and a second key point set on the input contour and the model contour, wherein the number of key points in the first key point set is the same as that of the key points in the second key point set, and the first key point set and the second key point set are in one-to-one correspondence; calculating the corresponding plane coordinates (xi, yi) of any key point pi in the model contour in the input contour; transforming the space coordinate zi of any key point pi in the 3D initial model according to the change proportion of the input contour and the model contour on the length and the width; determining a final shape of the 3D bone model based on the planar coordinates and the spatial coordinates;
a first determining module, configured to determine, based on all pictures in the image set, a map corresponding to a triangular patch in the 3D bone model;
a second determination module for determining a 3D reconstructed model of the target object based on the map and the 3D bone model.
8. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the 3D model reconstruction method of any one of the preceding claims 1-6.
9. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the 3D model reconstruction method of any one of the preceding claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910591658.7A CN110390717B (en) | 2019-07-02 | 2019-07-02 | 3D model reconstruction method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910591658.7A CN110390717B (en) | 2019-07-02 | 2019-07-02 | 3D model reconstruction method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110390717A CN110390717A (en) | 2019-10-29 |
| CN110390717B true CN110390717B (en) | 2023-03-31 |
Family
ID=68286036
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910591658.7A Active CN110390717B (en) | 2019-07-02 | 2019-07-02 | 3D model reconstruction method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110390717B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112819928B (en) * | 2021-01-27 | 2022-10-28 | 成都数字天空科技有限公司 | Model reconstruction method and device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101593205A (en) * | 2009-06-24 | 2009-12-02 | 清华大学 | Method for searching three-dimension model based on video |
| CN106447725A (en) * | 2016-06-29 | 2017-02-22 | 北京航空航天大学 | Spatial target attitude estimation method based on contour point mixed feature matching |
| CN107689079A (en) * | 2017-08-28 | 2018-02-13 | 北京航空航天大学 | The cloudland method for reconstructing that a kind of satellite cloud picture is combined with natural image |
| CN109142374A (en) * | 2018-08-15 | 2019-01-04 | 广州市心鉴智控科技有限公司 | Method and system based on the efficient Checking model of extra small sample training |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190189840A1 (en) * | 2017-12-18 | 2019-06-20 | National Cheng Kung University | Method of transferring nanostructures and device having the nanostructures |
-
2019
- 2019-07-02 CN CN201910591658.7A patent/CN110390717B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101593205A (en) * | 2009-06-24 | 2009-12-02 | 清华大学 | Method for searching three-dimension model based on video |
| CN106447725A (en) * | 2016-06-29 | 2017-02-22 | 北京航空航天大学 | Spatial target attitude estimation method based on contour point mixed feature matching |
| CN107689079A (en) * | 2017-08-28 | 2018-02-13 | 北京航空航天大学 | The cloudland method for reconstructing that a kind of satellite cloud picture is combined with natural image |
| CN109142374A (en) * | 2018-08-15 | 2019-01-04 | 广州市心鉴智控科技有限公司 | Method and system based on the efficient Checking model of extra small sample training |
Non-Patent Citations (3)
| Title |
|---|
| 基于骨架的单幅图像三维建模;张淑军等;《计算机与数字工程》;20180520(第05期);全文 * |
| 生成式对抗网络及其计算机视觉应用研究综述;曹仰杰等;《中国图象图形学报》;20181016(第10期);全文 * |
| 自由多视角恢复表面纹理的三维重建;崔桂涣等;《微计算机应用》;20090115(第01期);标题3内容 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110390717A (en) | 2019-10-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110058685B (en) | Virtual object display method and device, electronic equipment and computer-readable storage medium | |
| CN110378947B (en) | 3D model reconstruction method and device and electronic equipment | |
| CN106846497B (en) | Method and device for presenting three-dimensional map applied to terminal | |
| CN109754464B (en) | Method and apparatus for generating information | |
| CN112073748A (en) | Panoramic video processing method and device and storage medium | |
| WO2023029893A1 (en) | Texture mapping method and apparatus, device and storage medium | |
| CN115810101A (en) | Three-dimensional model stylizing method and device, electronic equipment and storage medium | |
| WO2022166868A1 (en) | Walkthrough view generation method, apparatus and device, and storage medium | |
| CN116310036A (en) | Scene rendering method, device, equipment, computer readable storage medium and product | |
| CN110378948B (en) | 3D model reconstruction method and device and electronic equipment | |
| CN111862349A (en) | Virtual brush implementation method and device and computer readable storage medium | |
| CN110390717B (en) | 3D model reconstruction method and device and electronic equipment | |
| CN111862342A (en) | Texture processing method and device for augmented reality, electronic equipment and storage medium | |
| CN109816791B (en) | Method and apparatus for generating information | |
| CN110264430B (en) | Video beautifying method and device and electronic equipment | |
| WO2020077912A1 (en) | Image processing method, device, and hardware device | |
| CN115082636B (en) | Single image three-dimensional reconstruction method and device based on mixed Gaussian network | |
| CN110363860B (en) | 3D model reconstruction method and device and electronic equipment | |
| CN114049403A (en) | Multi-angle three-dimensional face reconstruction method and device and storage medium | |
| CN111383337B (en) | Method and device for identifying objects | |
| KR102534449B1 (en) | Image processing method, device, electronic device and computer readable storage medium | |
| CN111354070A (en) | Three-dimensional graph generation method and device, electronic equipment and storage medium | |
| CN110288554B (en) | Video beautifying method and device and electronic equipment | |
| CN111626919B (en) | Image synthesis method and device, electronic equipment and computer readable storage medium | |
| CN111489428B (en) | Image generation method, device, electronic equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |