CN108921908B - Surface light field acquisition method and device and electronic equipment - Google Patents

Surface light field acquisition method and device and electronic equipment Download PDF

Info

Publication number
CN108921908B
CN108921908B CN201810720340.XA CN201810720340A CN108921908B CN 108921908 B CN108921908 B CN 108921908B CN 201810720340 A CN201810720340 A CN 201810720340A CN 108921908 B CN108921908 B CN 108921908B
Authority
CN
China
Prior art keywords
patch
image
images
mesh
grid
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201810720340.XA
Other languages
Chinese (zh)
Other versions
CN108921908A (en
Inventor
李伟
乔慧
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Baidu Netcom Science and Technology Co Ltd
Original Assignee
Beijing Baidu Netcom Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Baidu Netcom Science and Technology Co Ltd filed Critical Beijing Baidu Netcom Science and Technology Co Ltd
Priority to CN201810720340.XA priority Critical patent/CN108921908B/en
Publication of CN108921908A publication Critical patent/CN108921908A/en
Application granted granted Critical
Publication of CN108921908B publication Critical patent/CN108921908B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Image Processing (AREA)

Abstract

The embodiment of the invention discloses a method and a device for acquiring a surface light field and electronic equipment, wherein the method comprises the following steps: acquiring a plurality of images of each patch grid on the surface of the object model at different visual angles; for each patch grid, acquiring an image from a plurality of images of the patch grid as a target image, and aligning the other images except the target image in the patch grid with the target image to generate an image set of the patch grid; sampling the images in each image set to ensure that the pixels of the images in each image set are the same and the visual angle intervals are equal; and decomposing and compressing the sampled image sets corresponding to all the surface patch grids to obtain the surface light field data of the object model. According to the method and the device, before the surface light field is compressed, the sampled images are aligned, and then the influence of inaccurate geometric shapes is eliminated, so that the dependency of surface light field sampling and processing on accurate geometric models is reduced.

Description

Surface light field acquisition method and device and electronic equipment
Technical Field
The embodiment of the invention relates to the technical field of image processing, in particular to a method and a device for acquiring a surface light field and electronic equipment.
Background
The light field rendering technology is generally used to capture, record, compress and store all light rays (so-called light field) around a target object, and when rendering, restore all light ray information according to a viewing angle, thereby accurately restoring the appearance of the target object in the real world. Because the light field rendering technology avoids a plurality of difficult problems of material estimation of a target object, environmental illumination estimation and the like, the light field can effectively reconstruct the appearance of a real world object, photo-level realistic rendering is realized, and the light field can play an important role in a plurality of VR/AR applications.
The surface light field is a branch of the light field, the traditional light field rendering technology is further promoted by utilizing the geometric information of the target object, and the light field can be recorded in a distortion-free mode within a larger visual angle (360 degrees). Thus, the surface light field is used to recover some objects with more complex appearance and geometric models. However, surface light fields rely on an accurate geometric model to sample and process the light field. This limits its widespread use because many target object geometric surfaces are not easily accurately recovered using conventional precision reconstruction equipment.
Disclosure of Invention
The embodiment of the invention provides a method and a device for collecting a surface light field and electronic equipment.
In a first aspect, an embodiment of the present invention provides a method for acquiring a surface light field, including:
acquiring a plurality of images of each patch grid on the surface of an object model at different visual angles, wherein the object model is formed by meshing the surface of an object;
for each patch grid, acquiring an image from a plurality of images of the patch grid as a target image, and aligning each of the other images except the target image in the patch grid with the target image to generate an image set of the patch grid;
sampling the images in each image set to ensure that the pixels of the images in each image set are the same and the viewing angle intervals are equal;
and decomposing and compressing the sampled image set corresponding to all the surface patch grids to determine the surface light field data of the object model.
In a possible implementation manner of the first aspect, the acquiring, for each patch mesh, one image from a plurality of images of the patch mesh as a target image includes:
and for each patch grid, taking the image with the maximum diffusion color value in the images corresponding to the patch grid as a target image of the patch grid.
In another possible implementation manner of the first aspect, the acquiring, for each patch mesh, one image from the multiple images of the patch mesh as a target image includes:
determining the energy sum of all images corresponding to all patch grids;
and taking the image corresponding to each patch grid when the sum of the energies is minimum as a target image of each patch grid.
In another possible implementation manner of the first aspect, the determining a sum of energies of the images corresponding to the patch meshes includes:
according to the formula
Figure BDA0001718506280000021
Determining the energy sum E (P) of each image of each patch grid;
wherein, the
Figure BDA0001718506280000022
Color values of the ith image for patch mesh f
Figure BDA0001718506280000023
Is that it is
Figure BDA0001718506280000024
Corresponding luminance value, said
Figure BDA0001718506280000025
Is that it is
Figure BDA0001718506280000026
Corresponding sample masses, f' being a neighboring patch mesh of the patch mesh f, f
Figure BDA0001718506280000027
Color values of the jth image of the patch mesh f
Figure BDA0001718506280000028
Color differences on the edges are shared for the patch mesh f and the patch mesh f'.
In another possible implementation manner of the first aspect, the aligning, with the target image, the remaining images of the patch mesh except for the target image includes:
determining similar energy values of other images of the patch grid and the target image;
when the similar energy value is maximum, determining that the other images of the patch grid are aligned with the target image.
In another possible implementation manner of the first aspect, the determining the similar energy values of the remaining images of the patch mesh and the target image includes:
according to the formula
Figure BDA0001718506280000029
Determining the similar energy value E of the other images of the patch grid and the target imagef(Df,t)；
Wherein, D isfIs a target image of the patch mesh f, the
Figure BDA0001718506280000031
Color value of ith image of patch mesh f, tiThe translation amount of the ith image of the patch grid f.
In another possible implementation manner of the first aspect, the determining the similar energy values of the remaining images of the patch mesh and the target image includes:
according to the formula
Figure BDA0001718506280000032
Determining the similar energy value E of the other images of the patch grid and the target imagef(Df,t)；
Wherein, D isfIs a target image of the patch mesh f, the
Figure BDA0001718506280000033
Color value of ith image of patch mesh f, tiThe translation amount of the ith image of the patch grid f, t0Is a preset value.
In another possible implementation manner of the first aspect, the patch mesh is a triangular patch mesh.
In a second aspect, an embodiment of the present invention provides an apparatus for acquiring a surface light field, including:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring a plurality of images of each patch grid on the surface of an object model at different visual angles, and the object model is formed by meshing the surface of an object;
the second acquisition module is used for acquiring one image from a plurality of images of each patch grid as a target image;
an alignment module, configured to align each of the other images in the patch mesh except for the target image with the target image, and generate an image set of the patch mesh;
the sampling module is used for sampling the images in each image set, so that the pixels of the images in each image set are the same, and the viewing angle intervals are equal;
and the decompression and decomposition module is used for decomposing and compressing the sampled image set corresponding to all the surface patch grids and determining the surface light field data of the object model.
In a possible implementation manner of the second aspect, the second obtaining module is configured to, for each patch grid, use an image with a maximum diffusion color value in each image corresponding to the patch grid as a target image of the patch grid.
In another possible implementation manner of the second aspect, the second obtaining module includes:
the first calculation unit is used for determining the sum of the energies of all images corresponding to all patch grids;
and the second determining unit is used for taking the image corresponding to each patch grid when the sum of the energies is minimum as the target image of each patch grid.
In another possible implementation manner of the second aspect, the first calculating unit is specifically configured to calculate the first value according to a formula
Figure BDA0001718506280000041
Determining the energy sum E (P) of each image of each patch grid;
wherein, the
Figure BDA0001718506280000042
Color values of the ith image for patch mesh f
Figure BDA0001718506280000043
Is that it is
Figure BDA0001718506280000044
Corresponding luminance value, said
Figure BDA0001718506280000045
Is that it is
Figure BDA0001718506280000046
Corresponding sample masses, f' being a neighboring patch mesh of the patch mesh f, f
Figure BDA0001718506280000047
Color values of the jth image of the patch mesh f
Figure BDA0001718506280000048
Color differences on the edges are shared for the patch mesh f and the patch mesh f'.
In another possible implementation manner of the second aspect, the alignment module includes:
the second calculation unit is used for determining the similar energy values of the rest images of the patch grid and the target image;
a second determining unit, configured to determine that the remaining images of the patch mesh are aligned with the target image when the similar energy value is maximum.
In another possible implementation manner of the second aspect, the second calculating unit is specifically configured to calculate the second value according to a formula
Figure BDA0001718506280000049
Determining the similar energy value E of the other images of the patch grid and the target imagef(Df,t)；
Wherein, D isfIs a target image of the patch mesh f, the
Figure BDA00017185062800000410
Color value of ith image of patch mesh f, tiThe translation amount of the ith image of the patch grid f.
In another possible implementation manner of the second aspect, the second calculating unit is further specifically configured to calculate the second value according to a formula
Figure BDA00017185062800000411
Determining the similar energy value E of the other images of the patch grid and the target imagef(Df,t)；
Wherein, D isfIs a target image of the patch mesh f, the
Figure BDA00017185062800000412
Color value of ith image of patch mesh f, tiThe translation amount of the ith image of the patch grid f, t0Is a preset value.
In another possible implementation manner of the second aspect, the patch mesh is a triangular patch mesh.
In a third aspect, an embodiment of the present invention provides an electronic device, including:
a memory for storing a computer program;
a processor for executing the computer program to implement the method for acquiring a surface light field according to the first aspect.
In a fourth aspect, an embodiment of the present invention provides a computer storage medium, where a computer program is stored, where the computer program is used to execute the method for acquiring a surface light field according to the first aspect.
According to the method, the device and the electronic equipment for acquiring the surface light field, provided by the embodiment of the invention, a plurality of images of each patch grid on the surface of an object model at different view angles are acquired, wherein the object model is a model formed by meshing the surface of an object; for each patch grid, acquiring an image from a plurality of images of the patch grid as a target image, and aligning each of the other images except the target image in the patch grid with the target image to generate an image set of the patch grid; sampling the images in each image set to ensure that the pixels of the images in each image set are the same and the viewing angle intervals are equal; and decomposing and compressing the sampled image set corresponding to all the surface patch grids to determine the surface light field data of the object model. That is, in this embodiment, before the surface light field is compressed, the sampled images are aligned, so as to eliminate the influence of an inaccurate geometric shape, thereby reducing the dependency of surface light field sampling and processing on an accurate geometric model.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a schematic flow chart of a method for acquiring a surface light field according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of partial surface light field data according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of selecting a target image from unaligned images according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of images after alignment according to an embodiment of the present invention;
FIG. 5 is a schematic illustration of surface light field data optimized for the surface light field data shown in FIG. 2;
fig. 6 is a flowchart illustrating a method for acquiring a surface light field according to a second embodiment of the present invention;
fig. 7 is a flowchart illustrating a method for acquiring a surface light field according to a third embodiment of the present invention;
fig. 8 is a schematic structural diagram of a surface light field acquisition device according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of a surface light field acquisition device according to a second embodiment of the present invention;
fig. 10 is a schematic structural diagram of a surface light field acquisition device according to a third embodiment of the present invention;
fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The surface light field relies on an accurate geometric model to sample and process the light field. This limits its widespread use because many geometric surfaces of the target object are not easily accurately recovered using conventional precision reconstruction equipment
In order to solve the technical problem, the application provides a method for acquiring a surface light field, which is not used for directly optimizing a geometric structure so as to realize stable and high-fidelity modeling and rendering, but is used for aligning a plurality of images of each surface patch grid on the surface of an object model at different view angles so as to optimize sampling points, and further eliminating the influence of inaccurate geometric shapes. Then, the aligned images are compressed to generate surface light field data of the object model.
That is, the embodiment aligns the sampled images before the decomposition and compression process of the surface light field, thereby eliminating the influence of an inaccurate geometric shape, and reducing the dependency of the surface light field sampling and processing on an accurate geometric model.
The technical solution of the present invention will be described in detail below with specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.
Fig. 1 is a schematic flow chart of a method for acquiring a surface light field according to an embodiment of the present invention. As shown in fig. 1, the method of this embodiment may include:
s101, obtaining a plurality of images of each patch grid on the surface grid of the object model at different visual angles, wherein the object model is formed by performing patch grid division on the surface of the object.
The executing entity of this embodiment may be an acquisition apparatus of a surface light field having a function of determining surface light field data, and the acquisition apparatus of the surface light field of this embodiment may be a part of an electronic device, for example, a processor of the electronic device. The acquisition of the surface light field of the alternative embodiment may also be a separate electronic device.
The electronic device of this embodiment may be an electronic device such as a smart phone, a desktop computer, a notebook computer, an intelligent bracelet, an AR device, a VA device, and the like.
The embodiment takes the execution subject as an electronic device as an example for explanation.
Surface light fields are an image-based rendering technique for visualizing results. A new representation is created on the polygonal surface for the actual data sampling. The spatial samples are placed on the vertices and interpolated over the triangles. While the directional samples are subdivided from the polygons in screen space. Since the sampled data in the representation is stored as an image array, compression is performed using block coding techniques.
Before the surface light field is collected, the surface of the object needs to be gridded to generate an object module.
Optionally, the present embodiment may use a grid of any shape to divide the object, for example, a polygon grid such as a quadrangle, a pentagon, and a hexagon to divide the surface of the object.
In one example, to reduce the difficulty of partitioning, the present embodiment may use a triangular patch mesh to mesh the surface of the object.
In this way, the surface of the object is made up of a mesh of patches.
The sizes of the meshes of the patches can be the same or different, the meshes of the same shape can be used for dividing the same object, and the meshes of different shapes can also be used for dividing the same object.
And for each patch grid on the object model, photographing the patch grid at different visual angles to obtain a plurality of images of the patch grid at different visual angles.
For example, the patch mesh a is photographed from 3 different viewing angles, and 3 images of the patch mesh a at 3 viewing angles are obtained.
In practical application, assuming that an object is captured by a surface mesh M, the object is subjected to meshing to form an object model, wherein the surface of the object model is defined by NfEach patch mesh, i.e. set of patch meshes F ═ Fi，…fNfAnd (9) composition.
Next, the object model is placed at the center of the calibration plate, and a plurality of images I ═ I are taken at different viewing angles1，....IN}. At the same time, corresponding camera extrinsic parameters are extracted.
Optionally, since only a partial image is visible to the target face, partial patches are extracted from the image of each patch mesh F ∈ F, a visibility check is first performed to find a subset of the image visible to the face, and the face is backprojected into image space using the corresponding camera parameters to generate a set of images Pf＝{Pf 1,…Pf n}. Here, n is equal to the number of visible images of the human face.
S102, for each patch grid, acquiring an image from a plurality of images of the patch grid as a target image, aligning the other images except the target image in the patch grid with the target image, and generating an image set of the patch grid.
As shown in fig. 2, conventional direct compression may cause artifacts when the geometry is significantly inaccurate.
To solve this technical problem, the present embodiment optimizes the patch mesh prior to the light field decomposition to eliminate the negative effects of inaccurate geometry.
Specifically, according to the above steps, after obtaining a plurality of images of each patch mesh, aligning the plurality of images corresponding to each patch mesh, for example, aligning 3 images of patch mesh a.
In the alignment process, taking the patch mesh a as an example, first, one of the multiple images corresponding to the patch mesh a is selected as a target image of the patch mesh a. For example, as shown in fig. 3, the dark image is the target image of patch mesh a.
Then, the rest of the images except the target image in the patch mesh a are aligned with the target image, for example, introducing a 2D translation t in spaceiCalculating the translation t between the rest of each image and the target image in the patch mesh AiTo align the remaining images with the target image. The remaining images of patch mesh a are aligned with the target image of the dark image, as shown in fig. 4, for example.
In this way, a set of aligned patch mesh a images is defined as a patch mesh a image set.
Referring to the above steps, an image set of each patch mesh may be obtained.
In a possible implementation manner, for each patch mesh, the above-mentioned obtaining an image from multiple images of the patch mesh as a target image may be:
and for each patch grid, taking the image with the maximum diffusion color value in the images corresponding to the patch grid as a target image of the patch grid.
For example, for patch mesh a, the diffusion color value of each image is determined, and the image with the maximum diffusion color value is taken as the target image of the patch mesh a.
In this way, a target image corresponding to each patch mesh can be obtained.
S103, sampling the images in each image set to enable the pixels of the images in each image set to be the same and enable the viewing angle intervals to be equal.
Since the pixels (i.e., sizes) of the aligned images may be different, and the angles of the cameras are not equally spaced when the images are captured, the aligned images need to be sampled before decomposition, specifically, the pixels of the images in the same image set are the same, and the viewing angle intervals are equal.
Therefore, the pixels of the images in the same image set are the same, and the visual angle intervals are equal, so that the decomposition and compression accuracy can be improved when subsequent decomposition and compression are carried out, and the finally generated surface light field data is more accurate.
And S104, decomposing and compressing the sampled image set corresponding to all the surface patch grids, and determining the surface light field data of the object model.
Specifically, the images after the mesh alignment of the patches are sampled according to the above steps, so that the pixels of the images are the same and the viewing angle intervals are equal. And then, decomposing the sampled image sets corresponding to all the surface patch grids, compressing the decomposed images to generate high-precision surface light field data of the object model, and storing the surface light field data.
Fig. 5 is a schematic diagram of compressing the images corresponding to fig. 2 after alignment, and as shown in fig. 5, the images are compressed after alignment, so that ghost images can be effectively reduced, and the precision of surface light field data can be improved.
In VR (Virtual Reality) or AR (Augmented Reality) technology, the above determined surface light field data may be used to render an object, so as to present a light field effect of the object in a real environment in a Virtual environment, so as to improve user experience.
The method for decomposing and compressing the image set in the present embodiment is not limited, and any existing method may be used specifically.
In one example, the surface light field may be represented as a four-dimensional function L (u, v, s, t), where (u, v) is the position on the surface and (s, t) is the view direction.
The light field function can be further decomposed into a sum of a small number of products of the low dimensional functions:
L(u,v,s,t)≈∑S(u,v)V(s,t) (1)
wherein S (u, V) is a curved surface function and V (S, t) is a view function.
This decomposition attempts to separate the changes in surface texture from the changes in illumination. These functions can be constructed by using PCA (Principal Component Analysis) or nonlinear optimization. The functional parameters may be stored in a texture map and rendered in real-time.
To facilitate the implementation of the curved lightfield in the rendering pipeline, L (u, v, s, t) is made to span small surface primitives and approximate each portion independentlyx(u,v,s,t)。
In an implementation, the vertex light field function Lx(u, v, s, t) may be represented as a matrix Lx[u,v,s,t]∈Rm×nThe columns n of the matrix represent camera views and the rows m represent surface locations storage matrix LxIs impractical, requires decomposition and compression of the light field data, and, in addition, requires a secondary matrix L according to the theory of a two-color reflection modelxMiddle separation of the diffuse component Dx[s,t]And LxIs the residual component Gx[s,t,u,v]. The remaining portion is conventionally compressed as follows:
Lx[u,v,s,t]＝Dx[u,v]+Gx[u,v,s,t]＝Dx[u,v]+∑Sx[u,v]Vx[s,t](2)
wherein S isx[u,v]Is the surface mapping matrix of vertex x, Vx[s,t]Is the view mapping matrix for vertex x, which can be discretized from the surface and view functions in equation (1). (u, v) are the spatial coordinates in the vertex-by-vertex patch synthesized by its one circular triangular neighbor. [ s, t ]]Are the view coordinates in the hemisphere harmonic.
In one example, G of SVD (Singular Value Decomposition) is usedx＝Sx.VxResidual color G to be resampledx(i.e., the remaining components of L x above) into k-term surface maps and viewsxM × k matrix which is left singular vector, multiplying diagonal matrix of singular values ordered in decreasing order, VxIs a k × n matrix of right singular vectors, k<n。
It follows that this embodiment does not perform SVD completely, but iteratively computes the first k terms using a power iteration method.
The method for acquiring the surface light field comprises the steps of acquiring a plurality of images of each patch grid on the surface of an object model at different view angles, wherein the object model is formed by meshing the surface of an object; for each patch grid, acquiring an image from a plurality of images of the patch grid as a target image, and aligning each of the other images except the target image in the patch grid with the target image to generate an image set of the patch grid; sampling the images in each image set to ensure that the pixels of the images in each image set are the same and the viewing angle intervals are equal; and decomposing and compressing the sampled image set corresponding to all the surface patch grids to determine the surface light field data of the object model. That is, in this embodiment, before the surface light field is compressed, the sampled images are aligned, so as to eliminate the influence of an inaccurate geometric shape, thereby reducing the dependency of surface light field sampling and processing on an accurate geometric model.
Fig. 6 is a flowchart illustrating a method for acquiring a surface light field according to a second embodiment of the present invention. On the basis of the above embodiment, the present embodiment relates to a specific process of acquiring, for each patch mesh, one image from a plurality of images of the patch mesh as a target image. As shown in fig. 6, the S102 may specifically include:
s201, determining the sum of the energies of all images corresponding to all patch grids.
In practical application, the target image may not be found by using the existing method due to factors such as large brightness change. This embodiment solves this problem by finding the best image among all the images. Specifically, color values of all images corresponding to all patch grids on the surface of the object model are determined
Figure BDA0001718506280000111
And determining the sum of the energies of all images corresponding to all patch meshes as an energy function.
In one example, the sum of the intensities of all images corresponding to all patch meshes is calculated as the sum of the energies of all images corresponding to all patch meshes.
In another example, the sum of the quality of all images corresponding to all patch meshes is calculated and used as the sum of the energy of all images corresponding to all patch meshes.
In another example, the sum of the quality of all images corresponding to all patch meshes is calculated and used as the sum of the energy of all images corresponding to all patch meshes.
In another example, the sum of the brightness and the sum of the quality of all images corresponding to all patch meshes are calculated, and the sum of the brightness and the sum of the quality is taken as the sum of the energy of the images.
In a possible implementation manner of this embodiment, the sum e (p) of energies of all images corresponding to all patch meshes may also be determined according to formula (3):
Figure BDA0001718506280000112
wherein, the Pi fIs the first of a patch mesh fColor values of i images, said El(Pi f) Is said Pi fCorresponding brightness value, said Eq(Pi f) Is said Pi fCorresponding sample masses, f' being a neighboring patch mesh of the patch mesh f, f
Figure BDA0001718506280000113
Color values of the jth image of the patch mesh f
Figure BDA0001718506280000114
Color differences on the edges are shared for the patch mesh f and the patch mesh f'.
Above, El(Pi f) The above-mentioned Eq(Pi f)、
Figure BDA0001718506280000115
The determination may be performed according to an existing manner, and the description of the embodiment is omitted here.
Alternatively to this, the first and second parts may,
Figure BDA0001718506280000116
to this end, the mean luminance value of each patch mesh is calculated, assuming that diffusion should be captured under good luminance conditions without specular highlights
Figure BDA0001718506280000117
Sum variance
Figure BDA0001718506280000118
The 5% samples with the lowest average brightness are discarded, which may be captured without enough light.
Then use
Figure BDA0001718506280000119
And
Figure BDA00017185062800001110
the most probable luminance mean and variance are extracted to define the luminance El(Pi f) The energy of (a).
Due to specular highlight
Figure BDA00017185062800001111
And
Figure BDA00017185062800001112
compared with the obvious difference, therefore ElAn illuminated patch grid without highlights is favored.
Alternatively to this, the first and second parts may,
Figure BDA0001718506280000121
Figure BDA0001718506280000122
to represent
Figure BDA0001718506280000123
The original projection size of (a). In this embodiment, the projected size of the quality item can be chosen because it is a combined indicator of the camera position distance and the angular distance between the camera view and the triangle normal.
Alternatively to this, the first and second parts may,
Figure BDA0001718506280000124
in this embodiment, P is the shared edge of the patch mesh f and the patch mesh f'. While
Figure BDA0001718506280000125
Is that
Figure BDA0001718506280000126
RGB information of the middle p point. In short, Es calculates the color difference on the shared side of two adjacent triangles.
S202, taking the image corresponding to each patch mesh when the sum of the energies is minimum as a target image of each patch mesh.
Specifically, according to the above manner, the sum of the energies of all images corresponding to all patch meshes can be determined. Then, by minimizing the sum of energies, a set of images can be determined, and the set of images are used as target images of the patch mesh in a one-to-one correspondence.
For example, the above equation (3) is minimized, and the image corresponding to each patch mesh in the equation (3) at this time is taken as the target image of each patch mesh.
According to the method, when the target image is determined, the influence of the color difference of the adjacent patches of the patch grid is considered, so that the determined target image is more in line with the actual requirement.
Fig. 7 is a flowchart illustrating an acquisition method of a surface light field according to a third embodiment of the present invention. On the basis of the above embodiments, the present embodiment is directed to aligning each of the images other than the target image in the patch mesh with the target image. As shown in fig. 7, the S102 may specifically include:
s301, determining the similar energy values of the other images of the patch grid and the target image.
Specifically, in this embodiment, when the other images of the patch grid are aligned with the target image, the alignment between the other images and the target image can be ensured by determining the similar energy values of the other images of the patch grid and the target image, and maximizing the similar energy values.
The embodiment does not limit the specific manner of determining the similar energy values of the rest images of the patch grid and the target image.
In one example, the above S301 may be to determine the similar energy value E of each of the rest images of the patch mesh and the target image according to formula (4)f(Df,t)
Figure BDA0001718506280000127
Wherein, D isfIs a target image of the patch mesh f, Pi fColor value of ith image of patch mesh fT is describediThe translation amount of the ith image of the patch grid f.
Figure BDA0001718506280000131
Is to have a 2D shift t ═ t (t) in the original image spacex，ty) The resampling model of (1). Since the similarity comparison needs to be performed bypassing the mirror information, the present embodiment can use the mutual information metric as MI. An appropriate D can be calculated by alternately searching for t according to the above equation (4)f。
In another example, the step S301 may be determining the similar energy value E of each of the rest images of the patch mesh and the target image according to formula (5)f(Df,t)
Figure BDA0001718506280000132
Wherein, D isfIs a target image of the patch mesh f, Pi fColor value of ith image of patch mesh f, tiThe translation amount of the ith image of the patch grid f, t0Is a preset value.
t0For avoiding zero offset and adjusting tiBy a weight at a distance limit tmaxSearch t greedily insideiTo solve this problem.
Optionally, t0May be (15, 15), tmaxSet to 3 pixels.
S302, when the similar energy value is maximum, determining that the other images of the patch grid are aligned with the target image.
According to the steps, the similar energy value E of the other images of the patch grid and the target image is determinedf(DfT). When the similar energy value is maximum, it may be determined that the remaining images of the patch mesh are aligned with the target image.
In the embodiment, the other images of the patch grid are aligned with the target image by determining the similar energy values of the other images of the patch grid and the target image, so that the alignment reliability and efficiency are improved.
Fig. 8 is a schematic structural diagram of a surface light field acquisition device according to an embodiment of the present invention. As shown in fig. 8, the surface light field collecting apparatus 100 of the present embodiment may include:
a first obtaining module 110, configured to obtain multiple images of each patch mesh on a surface of an object model at different viewing angles, where the object model is a model formed by meshing the surface of an object;
a second obtaining module 120, configured to obtain, for each patch mesh, one image from multiple images of the patch mesh as a target image;
an alignment module 130, configured to align each of the other images in the patch mesh except for the target image with the target image, and generate an image set of the patch mesh;
a sampling module 140, configured to sample the images in each image set, so that the pixels of the images in each image set are the same and the viewing angle intervals are equal;
and a decompression module 150, configured to decompose and compress the sampled image set corresponding to all patch grids, and determine surface light field data of the object model.
The surface light field acquisition device of the embodiment of the present invention may be used to implement the technical solutions of the above-described method embodiments, and the implementation principles and technical effects thereof are similar, and are not described herein again.
In a possible implementation manner of this embodiment, the second obtaining module 120 is configured to, for each patch grid, use an image with a maximum diffusion color value in each image corresponding to the patch grid as a target image of the patch grid.
Fig. 9 is a schematic structural diagram of a surface light field acquisition device according to a second embodiment of the present invention. As shown in fig. 9, the second obtaining module 120 includes:
a first calculating unit 121, configured to determine a sum of energies of all images corresponding to all patch meshes;
and a second determining unit 122, configured to use an image corresponding to each patch mesh when the sum of the energies is minimum as a target image of each patch mesh.
In a possible implementation manner of this embodiment, the first calculating unit 121 is specifically configured to calculate the first value according to a formula
Figure BDA0001718506280000141
Determining the energy sum E (P) of each image of each patch grid;
wherein, the Pi fColor values of the ith image for patch mesh f, El(Pi f) Is said Pi fCorresponding brightness value, said Eq(Pi f) Is said Pi fCorresponding sample masses, f' being a neighboring patch mesh of the patch mesh f, f
Figure BDA0001718506280000142
Color values of the jth image of the patch mesh f
Figure BDA0001718506280000143
Color differences on the edges are shared for the patch mesh f and the patch mesh f'.
The surface light field acquisition device of the embodiment of the present invention may be used to implement the technical solutions of the above-described method embodiments, and the implementation principles and technical effects thereof are similar, and are not described herein again.
Fig. 10 is a schematic structural diagram of a surface light field acquisition device according to a third embodiment of the present invention. On the basis of the above embodiment, as shown in fig. 10, the alignment module 130 further includes:
a second calculating unit 131, configured to determine similar energy values of the remaining images of the patch mesh and the target image;
a second determining unit 132, configured to determine that the remaining images of the patch mesh are aligned with the target image when the similar energy value is maximum.
In a possible implementation manner of this embodiment, the second calculating unit 131 is specifically configured to calculate according to a formula
Figure BDA0001718506280000151
Determining the similar energy value E of the other images of the patch grid and the target imagef(Df,t)；
Wherein, D isfIs a target image of the patch mesh f, Pi fColor value of ith image of patch mesh f, tiThe translation amount of the ith image of the patch grid f.
In another possible implementation manner of this embodiment, the second calculating unit 131 is further specifically configured to calculate the second value according to a formula
Figure BDA0001718506280000152
Determining the similar energy value E of the other images of the patch grid and the target imagef(Df,t)；
Wherein, D isfIs a target image of the patch mesh f, Pi fColor value of ith image of patch mesh f, tiThe translation amount of the ith image of the patch grid f, t0Is a preset value.
Optionally, the patch mesh is a triangular patch mesh.
The surface light field acquisition device of the embodiment of the present invention may be used to implement the technical solutions of the above-described method embodiments, and the implementation principles and technical effects thereof are similar, and are not described herein again.
Fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 11, the electronic device 200 according to the embodiment includes:
a memory 220 for storing a computer program;
the processor 230 is configured to execute the computer program to implement the above-mentioned method for acquiring a surface light field, which has similar implementation principles and technical effects, and is not described herein again.
Further, when at least a part of the functions of the method for acquiring the surface light field in the embodiment of the present invention are implemented by software, the embodiment of the present invention further provides a computer storage medium, which is used to store computer software instructions for acquiring the surface light field described above, and when the computer storage medium is run on a computer, the computer storage medium enables the computer to execute various possible methods for acquiring the surface light field in the above method embodiments. The processes or functions described in accordance with the embodiments of the present invention may be generated in whole or in part when the computer-executable instructions are loaded and executed on a computer. The computer instructions may be stored on a computer storage medium or transmitted from one computer storage medium to another via wireless (e.g., cellular, infrared, short-range wireless, microwave, etc.) to another website site, computer, server, or data center. The computer storage media may be any available media that can be accessed by a computer or a data storage device, such as a server, data center, etc., that incorporates one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD), among others.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims (16)

1. A method for acquiring a surface light field, comprising:
acquiring a plurality of images of each patch grid on the surface of an object model at different visual angles, wherein the object model is formed by meshing the surface of an object;
for each patch grid, acquiring an image from a plurality of images of the patch grid as a target image, and aligning each of the other images except the target image in the patch grid with the target image to generate an image set of the patch grid; wherein the aligning the images except the target image in the patch mesh with the target image comprises: determining similar energy values of other images of the patch grid and the target image; aligning each of the remaining images of the patch mesh with the target image by maximizing the similar energy value;
sampling the images in each image set to ensure that the pixels of the images in each image set are the same and the viewing angle intervals are equal;
and decomposing and compressing the sampled image set corresponding to all the surface patch grids to determine the surface light field data of the object model.
2. The method of claim 1, wherein for each patch mesh, obtaining an image from the plurality of images of the patch mesh as the target image comprises:
and for each patch grid, taking the image with the maximum diffusion color value in the images corresponding to the patch grid as a target image of the patch grid.
3. The method of claim 1, wherein for each patch mesh, obtaining an image from the plurality of images of the patch mesh as the target image comprises:
determining the energy sum of all images corresponding to all patch grids;
and taking the image corresponding to each patch grid when the sum of the energies is minimum as a target image of each patch grid.
4. The method of claim 3, wherein determining the sum of the energies of the respective images corresponding to the respective patch meshes comprises:
according to the formula
Figure FDA0002530990730000011
Determining the energy sum E (P) of each image of each patch grid;
wherein, the
Figure FDA0002530990730000012
Color values of the ith image for patch mesh f
Figure FDA0002530990730000013
Is that it is
Figure FDA0002530990730000014
Corresponding luminance value, said
Figure FDA0002530990730000015
Is that it is
Figure FDA0002530990730000016
Corresponding sample masses, f' being a neighboring patch mesh of the patch mesh f, f
Figure FDA0002530990730000017
Color values of the jth image of the patch mesh f
Figure FDA0002530990730000021
Color differences on the edges are shared for the patch mesh f and the patch mesh f'.
5. The method of claim 1, wherein determining the similarity energy values of the remaining images of the patch mesh to the target image comprises:
according to the formula
Figure FDA0002530990730000022
Determining the similar energy value E of the other images of the patch grid and the target imagef(Df,t)；
Wherein, D isfIs a target image of the patch mesh f, the
Figure FDA0002530990730000023
Color value of ith image of patch mesh f, tiThe amount of translation of the ith image of the patch mesh f
Figure FDA0002530990730000024
Is to have a 2D shift t ═ t (t) in the original image spacex，ty) The MI is a mutual information metric.
6. The method of claim 1, wherein determining the similarity energy values of the remaining images of the patch mesh to the target image comprises:
according to the formula
Figure FDA0002530990730000025
Determining the similar energy value E of the other images of the patch grid and the target imagef(Df,t)；
Wherein, D isfIs a target image of the patch mesh f, the
Figure FDA0002530990730000026
Color value of ith image of patch mesh f, tiThe translation amount of the ith image of the patch grid f, t0Is a preset value, the
Figure FDA0002530990730000027
Is provided in the original image spaceWith 2D shift t ═ t (t)x，ty) The MI is a mutual information metric.
7. The method of any of claims 1-4, wherein the patch mesh is a triangular patch mesh.
8. An apparatus for acquiring a surface light field, comprising:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring a plurality of images of each patch grid on the surface of an object model at different visual angles, and the object model is formed by meshing the surface of an object;
the second acquisition module is used for acquiring one image from a plurality of images of each patch grid as a target image;
an alignment module, configured to align each of the other images in the patch mesh except for the target image with the target image, and generate an image set of the patch mesh; wherein the alignment module comprises: the second calculation unit is used for determining the similar energy values of the rest images of the patch grid and the target image; a second determining unit, configured to implement alignment between each of the other images of the patch mesh and the target image by maximizing the similar energy value;
the sampling module is used for sampling the images in each image set, so that the pixels of the images in each image set are the same, and the viewing angle intervals are equal;
and the decompression and decomposition module is used for decomposing and compressing the sampled image set corresponding to all the surface patch grids and determining the surface light field data of the object model.
9. The apparatus of claim 8,
and the second obtaining module is configured to, for each patch grid, use an image with the largest diffusion color value in each image corresponding to the patch grid as a target image of the patch grid.
10. The apparatus of claim 8, wherein the second obtaining module comprises:
the first calculation unit is used for determining the sum of the energies of all images corresponding to all patch grids;
and the second determining unit is used for taking the image corresponding to each patch grid when the sum of the energies is minimum as the target image of each patch grid.
11. Device according to claim 10, characterized in that said first calculation unit is in particular adapted to calculate according to a formula
Figure FDA0002530990730000031
Determining the energy sum E (P) of each image of each patch grid;
wherein, the
Figure FDA0002530990730000032
Color values of the ith image for patch mesh f
Figure FDA0002530990730000033
Is that it is
Figure FDA0002530990730000034
Corresponding luminance value, said
Figure FDA0002530990730000035
Is that it is
Figure FDA0002530990730000036
Corresponding sample masses, f' being a neighboring patch mesh of the patch mesh f, f
Figure FDA0002530990730000037
The color value of the jth image of the patch grid f',the above-mentioned
Figure FDA0002530990730000038
Color differences on the edges are shared for the patch mesh f and the patch mesh f'.
12. The apparatus of claim 8,
the second calculation unit is specifically configured to calculate the second calculation value according to a formula
Figure FDA0002530990730000039
Determining the similar energy value E of the other images of the patch grid and the target imagef(Df,t)；
Wherein, D isfIs a target image of the patch mesh f, the
Figure FDA00025309907300000310
Color value of ith image of patch mesh f, tiThe amount of translation of the ith image of the patch mesh f
Figure FDA00025309907300000311
Is to have a 2D shift t ═ t (t) in the original image spacex，ty) The MI is a mutual information metric.
13. The apparatus according to claim 8, wherein the second calculation unit is further configured to calculate the second value according to a formula
Figure FDA00025309907300000312
Determining the similar energy value E of the other images of the patch grid and the target imagef(Df,t)；
Wherein, D isfIs a target image of the patch mesh f, the
Figure FDA00025309907300000313
Color value of ith image of patch mesh f, tiThe translation amount of the ith image of the patch grid f, t0Is a preset value, the
Figure FDA0002530990730000041
Is to have a 2D shift t ═ t (t) in the original image spacex，ty) The MI is a mutual information metric.
14. The apparatus of any of claims 8-11, wherein the patch mesh is a triangular patch mesh.
15. An electronic device, comprising:
a memory for storing a computer program;
a processor for executing the computer program for implementing the method of acquiring a surface light field according to any of claims 1-7.
16. A computer storage medium, characterized in that the storage medium has stored therein a computer program which, when executed, implements the method of acquisition of a surface light field according to any one of claims 1 to 7.
CN201810720340.XA 2018-07-03 2018-07-03 Surface light field acquisition method and device and electronic equipment Active CN108921908B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810720340.XA CN108921908B (en) 2018-07-03 2018-07-03 Surface light field acquisition method and device and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810720340.XA CN108921908B (en) 2018-07-03 2018-07-03 Surface light field acquisition method and device and electronic equipment

Publications (2)

Publication Number Publication Date
CN108921908A CN108921908A (en) 2018-11-30
CN108921908B true CN108921908B (en) 2020-07-28

Family

ID=64425343

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810720340.XA Active CN108921908B (en) 2018-07-03 2018-07-03 Surface light field acquisition method and device and electronic equipment

Country Status (1)

Country Link
CN (1) CN108921908B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113570620B (en) * 2021-07-14 2024-03-05 北京达佳互联信息技术有限公司 Light field data acquisition method and device, electronic equipment and storage medium

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101706830A (en) * 2009-11-12 2010-05-12 中国人民解放军国防科学技术大学 Method for reconstructing model after drilling surface grid model of rigid object
CN102306396A (en) * 2011-09-15 2012-01-04 山东大学 Three-dimensional entity model surface finite element mesh automatic generation method
CN104392477A (en) * 2014-12-11 2015-03-04 大连大学 Triangular mesh curve surface recovery post-treatment method based on removal of overlapped patches with included angles
CN104778755A (en) * 2015-03-27 2015-07-15 浙江理工大学 Region-division-based three-dimensional reconstruction method for texture image
CN106373179A (en) * 2016-08-29 2017-02-01 北京像素软件科技股份有限公司 Illumination scene rendering method
CN106683068A (en) * 2015-11-04 2017-05-17 北京文博远大数字技术有限公司 Three-dimensional digital image acquisition method and equipment thereof
CN107918926A (en) * 2017-11-22 2018-04-17 徐李添 A kind of digital image magnification method based on quadrangle and triangle surface

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8754829B2 (en) * 2012-08-04 2014-06-17 Paul Lapstun Scanning light field camera and display
US10887581B2 (en) * 2016-11-11 2021-01-05 Disney Enterprises, Inc. Object reconstruction from dense light fields via depth from gradients

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101706830A (en) * 2009-11-12 2010-05-12 中国人民解放军国防科学技术大学 Method for reconstructing model after drilling surface grid model of rigid object
CN102306396A (en) * 2011-09-15 2012-01-04 山东大学 Three-dimensional entity model surface finite element mesh automatic generation method
CN104392477A (en) * 2014-12-11 2015-03-04 大连大学 Triangular mesh curve surface recovery post-treatment method based on removal of overlapped patches with included angles
CN104778755A (en) * 2015-03-27 2015-07-15 浙江理工大学 Region-division-based three-dimensional reconstruction method for texture image
CN106683068A (en) * 2015-11-04 2017-05-17 北京文博远大数字技术有限公司 Three-dimensional digital image acquisition method and equipment thereof
CN106373179A (en) * 2016-08-29 2017-02-01 北京像素软件科技股份有限公司 Illumination scene rendering method
CN107918926A (en) * 2017-11-22 2018-04-17 徐李添 A kind of digital image magnification method based on quadrangle and triangle surface

Also Published As

Publication number Publication date
CN108921908A (en) 2018-11-30

Similar Documents

Publication Publication Date Title
CN111598998B (en) Three-dimensional virtual model reconstruction method, three-dimensional virtual model reconstruction device, computer equipment and storage medium
CN109859296B (en) Training method of SMPL parameter prediction model, server and storage medium
CN109118582B (en) Commodity three-dimensional reconstruction system and reconstruction method
Zhang et al. A UAV-based panoramic oblique photogrammetry (POP) approach using spherical projection
CN106228507B (en) A kind of depth image processing method based on light field
KR100443552B1 (en) System and method for embodying virtual reality
Li et al. Detail-preserving and content-aware variational multi-view stereo reconstruction
US20170330375A1 (en) Data Processing Method and Apparatus
EP3756163B1 (en) Methods, devices, and computer program products for gradient based depth reconstructions with robust statistics
Mousavi et al. The performance evaluation of multi-image 3D reconstruction software with different sensors
JP2001067463A (en) Device and method for generating facial picture from new viewpoint based on plural facial pictures different in viewpoint, its application device and recording medium
WO2018133119A1 (en) Method and system for three-dimensional reconstruction of complete indoor scene based on depth camera
CN116051747A (en) House three-dimensional model reconstruction method, device and medium based on missing point cloud data
CN113781621A (en) Three-dimensional reconstruction processing method, device, equipment and storage medium
CN113220251A (en) Object display method, device, electronic equipment and storage medium
CN116563493A (en) Model training method based on three-dimensional reconstruction, three-dimensional reconstruction method and device
CN114863061A (en) Three-dimensional reconstruction method and system for remote monitoring medical image processing
CN118247429A (en) Air-ground cooperative rapid three-dimensional modeling method and system
CN109166176B (en) Three-dimensional face image generation method and device
Bullinger et al. 3D Surface Reconstruction From Multi-Date Satellite Images
Frommholz et al. Reconstructing buildings with discontinuities and roof overhangs from oblique aerial imagery
CN108921908B (en) Surface light field acquisition method and device and electronic equipment
CN114913287B (en) Three-dimensional human body model reconstruction method and system
CN108151712B (en) Human body three-dimensional modeling and measuring method and system
Hao et al. Image completion with perspective constraint based on a single image

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant