CN110599578A - Realistic three-dimensional color texture reconstruction method - Google Patents
Realistic three-dimensional color texture reconstruction method Download PDFInfo
- Publication number
- CN110599578A CN110599578A CN201910687176.1A CN201910687176A CN110599578A CN 110599578 A CN110599578 A CN 110599578A CN 201910687176 A CN201910687176 A CN 201910687176A CN 110599578 A CN110599578 A CN 110599578A
- Authority
- CN
- China
- Prior art keywords
- image
- dimensional
- texture
- color texture
- color
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 65
- 230000004927 fusion Effects 0.000 claims abstract description 30
- 238000013507 mapping Methods 0.000 claims abstract description 25
- 230000000007 visual effect Effects 0.000 claims abstract description 12
- 238000005457 optimization Methods 0.000 claims description 17
- 230000008569 process Effects 0.000 claims description 16
- 230000003287 optical effect Effects 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000006073 displacement reaction Methods 0.000 claims description 3
- 239000002131 composite material Substances 0.000 abstract description 8
- 230000002457 bidirectional effect Effects 0.000 abstract description 7
- 238000012935 Averaging Methods 0.000 abstract description 2
- 238000003384 imaging method Methods 0.000 description 15
- 239000011159 matrix material Substances 0.000 description 9
- 238000005259 measurement Methods 0.000 description 9
- 230000014509 gene expression Effects 0.000 description 5
- 239000013598 vector Substances 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000013519 translation Methods 0.000 description 4
- 230000005603 centrifugal distortion Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000009826 distribution Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 238000013178 mathematical model Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000011514 reflex Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000002411 adverse Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000002238 attenuated effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000002146 bilateral effect Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000000691 measurement method Methods 0.000 description 1
- 238000012634 optical imaging Methods 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000002310 reflectometry Methods 0.000 description 1
- 239000004576 sand Substances 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000003860 storage Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/04—Texture mapping
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/40—Analysis of texture
- G06T7/41—Analysis of texture based on statistical description of texture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Probability & Statistics with Applications (AREA)
- Computer Graphics (AREA)
- Image Generation (AREA)
- Image Processing (AREA)
Abstract
The invention provides a realistic three-dimensional color texture reconstruction method, which comprises the following steps: pre-calibrating system parameters of a three-dimensional sensor and a color texture camera; acquiring a multi-view three-dimensional image and a two-dimensional color texture image; generating a three-dimensional mesh model using the multi-view three-dimensional image; establishing a mapping relation between the two-dimensional color texture image and the three-dimensional grid model under each visual angle according to the system parameters; performing texture fusion based on the mapping relation to obtain a fused image so as to realize color texture reconstruction of the integral three-dimensional model; and generating a corresponding texture mapping according to the mapping relation. The confidence of the texture color is evaluated by introducing the composite weight parameter. By weighted averaging the projected texture image, texture discontinuities can be eliminated. For inaccurate geometric figures, a bidirectional similarity function representing the structural similarity of two images is introduced to correct the inconsistency and generate a realistic texture.
Description
Technical Field
The invention belongs to the technical field of electronics, and particularly relates to a realistic three-dimensional color texture reconstruction method.
Background
The three-dimensional measurement technology is widely applied to multiple industries and disciplines such as urban measurement, human body measurement, prototype manufacture and the like, and the optical three-dimensional measurement technology provides a flexible method for obtaining three-dimensional images. With the development of high-performance photoelectric devices such as Charge Coupled Devices (CCDs), Digital Light Processing (DLP) projectors and the like, optical three-dimensional measurement can obtain data with high sensitivity and high speed. Structured light three-dimensional measurement techniques illuminated by dynamically spatially varying patterns, which may be periodic stripes, two-dimensional grids, or random spots, are widely used in three-dimensional measurement systems. The geometry of the object is encoded by the distorted structured-light pattern in order to be accurately demodulated from the captured image.
However, three-dimensional geometric measurements do not solve the color problem. Typically, multi-view images are captured for mapping on a geometric surface to generate color information independent of the geometric reconstruction. Any errors in geometry or camera pose may result in difficult alignment of the mapping, and inconsistent illumination between different views may result in unrealistic colors, which may lead to texture artifacts such as blurring, ghosting, and color discontinuities.
In order to solve the above problems, various texture reconstruction methods have been proposed. Color consistency can be improved by image fusion in the spatial or frequency domain, but these methods are achieved at the expense of image sharpness. Image stitching using a markov random field optimization method is another method to avoid image degradation, however visible seams are not completely eliminated. Post-processing is commonly used to adjust the color of the texture patches at the seams, such as poisson fusion, thermal diffusion, and other color adjustment methods. By optimizing camera pose, it can be used to correct misalignment phenomena such as manual camera calibration, mutual information based methods, and methods that maximize color consistency. Some methods deal with the offset by correcting the input image using non-rigid calibration techniques. For example, optical flow methods for image warping have been introduced to address the image bias problem. In addition, super-resolution methods have been proposed to overcome the blur problem. One recent approach proposes a patch-based optimization method for texture mapping of multiple images. In the above method, although various means are employed to remove various artifacts, manual involvement is required to reduce efficiency.
To address at least one of the above problems, a realistic three-dimensional color texture reconstruction method is proposed herein.
Disclosure of Invention
In order to solve the above problems, the present invention provides a method for reconstructing a realistic three-dimensional color texture, comprising: pre-calibrating system parameters of a three-dimensional sensor and a color texture camera; acquiring a multi-view three-dimensional image and a two-dimensional color texture image; generating a three-dimensional mesh model using the multi-view three-dimensional image; establishing a mapping relation between the two-dimensional color texture image and the three-dimensional grid model under each visual angle according to the system parameters; performing texture fusion based on the mapping relation to obtain a fused image so as to realize color texture reconstruction of the integral three-dimensional model; and generating a corresponding texture mapping according to the mapping relation.
In some embodiments, the texture fusion is to evaluate the confidence of each texture pixel by a method of defining complex weight by depth data, and perform weighted average according to the confidence of each view to calculate the fusion result. The complex weight is calculated by:
f(xk)＝fnorm(xk)·fdepth(xk)·fedge(xk)
wherein the normal weightDepth weightingEdge weightsAnd each weight is normalized to have a value range of [0, 1%]. The coefficients in the normal weight take values as: a is 0.1, b is 50 °, and coefficients in the depth weight take values of: a is 0.4, b is 50mm, d0The coefficient value in the edge weight is 55 mm: a is-0.08 and b is 50 mm.
In some embodiments, a target image is introduced between the original two-dimensional color texture image and the fused image, and a bidirectional similarity function E is adopted according to the displacement condition of the fused image at each view angleBDSAnd (S, T) carrying out reconstruction calculation on the original two-dimensional color texture image to generate an energy function, and reducing image blurring of integral model texture fusion by minimizing the energy function so as to enable the overall target image to be displaced, thereby finally obtaining a new high-resolution target image.
In some embodiments, the energy function further comprises an optical measure consistency function EC：
Wherein M isiRepresenting said fused image at the ith viewing angle, xkRepresenting the pixel position of the image, P (-) represents the projection function, and N represents the number of views. w is ajRepresenting the complex weight of the image at the jth view.
In some embodiments, the energy function E is ultimately configured as:
E＝E1+λE2
where λ is two energy functions E1And E2A scaling factor in between.
In some embodiments, the objective function is solved by a two-step alternating optimization strategy, i.e. the following two steps are solved iteratively: s1, fixing the fusion image MiOptimizing said target image Ti(ii) a S2: fixing the target image TiOptimizing said fused image Mi。
In step S1, the target image Ti is expressed by the following equation:
in step S2, the objective function Mi is represented by the following formula:
in some embodiments, a multi-scale optimization method is adopted in the iterative operation process, that is, in a low-scale stage, all images are down-sampled to a low resolution and the iterative operation is performed. After the energy function E converges, the target image and the fused image are up-sampled to a larger scale, and the original two-dimensional color texture image is still down-sampled, in order to inject the high-frequency information of the original two-dimensional color texture image into the target image and the fused image.
The invention has the beneficial effects that: a realistic three-dimensional color texture reconstruction method is provided, and confidence of texture color is evaluated by introducing a composite weight parameter. By weighted averaging the projected texture image, texture discontinuities can be eliminated. For imprecise geometry, a bilateral similarity (BDS) function representing the structural similarity of the two images is introduced to correct the disparity, generating a realistic texture.
Drawings
Fig. 1 is a schematic diagram of a realistic three-dimensional color texture reconstruction method according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating various types of weight curves according to an embodiment of the present invention.
Fig. 3 is a flow chart of a BSF-based color texture fusion algorithm according to one embodiment of the present invention.
Detailed Description
The present invention is described in further detail below with reference to specific embodiments and with reference to the attached drawings, it should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.
Fig. 1 is a schematic diagram illustrating a realistic three-dimensional color texture reconstruction method according to an embodiment of the present invention.
In step 101, system parameters of the three-dimensional sensor and the color texture camera are pre-calibrated. The three-dimensional sensor is used for acquiring a collected depth image, namely a three-dimensional image, and the three-dimensional sensor can be a binocular vision three-dimensional sensor based on structured light technology, such as a binocular vision three-dimensional sensor consisting of a digital stripe projector and a bicolor camera. Of course, any three-dimensional sensor capable of acquiring three-dimensional images can be adopted, such as a monocular structured light three-dimensional sensor, a time-of-flight three-dimensional sensor, and the like. The color texture camera is used for acquiring high-resolution object texture information, for example, a high-resolution single lens reflex camera or the like can be used. The calibration is to calibrate the internal parameters of the three-dimensional sensor and the color texture camera and to calibrate the relative external parameters of the three-dimensional sensor and the color texture camera.
In one embodiment, the three-dimensional sensor contains three cameras, two of which are black and white industrial cameras for generating depth data and the third is a color camera for acquiring texture pictures. The calibration of the three-dimensional sensor is a precondition for subsequent generation of depth data and texture fusion. The calibration of a binocular sensor composed of black and white cameras, and the principle of color texture camera calibration will be described below, starting from a mathematical model of a single camera.
A camera model.
If the diffraction effect of an imaging system is ignored and the camera lens strictly meets paraxial conditions, the camera imaging can be equivalent to small-hole imaging, the imaging process meets perspective projection transformation, and the object point is marked as X under the world coordinate systemw＝(Xw,Yw,Zw)TThe ideal image point in the image coordinate system is mc＝(u,v)TThen the imaging process is represented as
Wherein, the upper labelRepresenting homogeneous coordinates; xcRepresenting the coordinates of the object point in a camera coordinate system;is XcProjection onto the camera image plane; rc、tcRespectively a rotation matrix and a translation vector from a world coordinate system to a camera coordinate system, which are called external parameters of the camera; kcIs a camera intrinsic parameter matrix including equivalent focal lengths (f) along the image coordinate axesu,fv)TProjection of the optical center on the image plane, i.e. principal point (u) of the image plane0,v0)TAnd a tilt factor γ of the image; mc＝Kc[Rc|tc]The projection matrix is called as a projection matrix and contains internal and external parameters of the camera; λ ═ ZcIs a scale factor.
The real optical imaging system inevitably causes deviation between an actual imaging plane and the ideal imaging plane due to the influence of factors such as the processing technology, the structural assembly and the like of the imaging lens, and is called as camera lens distortion. The classical Brown-Conrady model is the most widely used lens distortion model at present, and the distortion can be expressed as:
x′c＝xc+Δ(xc),
wherein, x'c＝(x′c,y′c)TRepresenting a distorted image point; delta (x)c) Representing distortion terms including radial distortion and centrifugal distortion;is the distance of the undistorted image point to the principal point; (k)1,k2,k3…) and (p)1,p2,p3…) are the radial distortion and centrifugal distortion parameters, respectively. Usually three radial distortions and two centrifugal distortions already meet the accuracy requirement. Let k be (k)1,k2,k3,p1,p2)TRepresenting distortion parameter vectors, the camera model containing lens distortion is represented as
In the nonlinear camera model represented by the formula (3), K and KcDenotes an internal reference of the camera, RcAnd tcRepresenting the camera's external parameters. The camera calibration process is typically to minimize the reprojection error of the target reference point to the actual image point, i.e., Xc→mcTherefore by parsing expression x'c＝xc+Δ(xc(ii) a k) The ideal image point can be distorted, i.e. from an undistorted image point xcCalculating the distorted image point x'c. In the three-dimensional reconstruction process, an accurate object point, namely X, is constructed through actual image pointsc→mc. The three-dimensional reconstruction process is then to reconstruct the spatial coordinates, i.e. m, of the object points from the actual image pointsc→XcThis process requires the distortion to be removed, i.e. from the distorted actual image point x'cCalculating an undistorted image point xc. Since the formula (2) is a complex nonlinear function, the analytical expression of the inverse function thereof cannot be obtained, and considering that the distortion term is relatively small, the method can be used for obtaining the distortion term by a recursive approximation methodNumerical solution of undistorted image points:
the distorted image point is approximate to the initial undistorted image point, and the accurate undistorted image point can be obtained by controlling the iteration times.
And calibrating the binocular sensor.
A three-dimensional sensor based on binocular stereo vision can be composed of left and right cameras, and a world coordinate system is usually established on the cameras. Taking the left camera as an example, the mathematical model of the binocular sensor is expressed as
Xl＝RlXw+tl (5)
Wherein I is an identity matrix and R issAnd tsIs the rotation matrix and translation vector of the left camera coordinate system to the right camera coordinate system, [ R ]s|ts]Represents a structural parameter of the sensor, satisfies
The target of binocular calibration is to determine the internal parameters of the two cameras and the structural parameters between the two cameras.
In the camera calibration process, the reprojection error between the projection image point of the target reference point and the actually measured image point is generally used as an optimization objective function, and the optimal estimated values of the internal and external parameters are obtained from the target image data. The target function calibrated by the binocular sensor is expressed as
Wherein m'lAnd m'rIs the coordinates of the real image or images,andare the reprojected image coordinates of the reference points calculated from the model. Optimization algorithms by Gauss-Newton or Levenberg-Marquardt may be employed[207]And (5) carrying out optimization solution on the above formula to finally obtain system parameters.
And calibrating a color texture camera.
The color texture camera is used for acquiring color two-dimensional image information of an object, acquiring color information of a three-dimensional grid by establishing a mapping relation from a three-dimensional geometric model to a two-dimensional color image, and finally realizing color three-dimensional imaging. In order to realize the mapping from the three-dimensional geometric model to the two-dimensional color image, the internal parameters and the structural parameters of the color texture camera need to be solved in advance, namely the color texture camera is calibrated. Generally, there are two ways for a color three-dimensional sensor to acquire a color image of an object: firstly, two cameras of a binocular sensor are color cameras, and can acquire depth information and color texture information at the same time, and the inside and outside parameters of the cameras are determined through calibration of the binocular sensor under the condition, so that extra calibration operation is not needed; and the binocular camera is separated from the color camera, and the color camera needs to be calibrated additionally. In the practical application process, the two modes have advantages and disadvantages. The first method has simple structure and low cost, but because the color camera is influenced by the Bayer filter imaging principle, the gray scale precision of the gray scale image obtained by color image conversion is lower than that obtained by a black and white camera directly, thereby influencing the precision of depth data. In addition, in consideration of the three-dimensional measurement speed, the resolution of an industrial camera used for the binocular sensor cannot be too high, thereby limiting the resolution of the color texture image. The second mode is relatively complex in structure and high in cost, but a special color camera such as a professional single lens reflex camera can be selected according to requirements due to the fact that the second mode is not limited by depth data acquisition, and therefore extremely high resolution and color rendition are achieved.
If the left camera coordinate system is a three-dimensional sensor coordinate system, the structural parameters of the three cameras are as follows:
wherein R istAnd ttRespectively rotation matrix and translation vector from world coordinate system to color camera coordinate system, RpAnd tpThe rotation matrix and translation vector between the left camera and the color texture camera. In order to obtain a structure parameter with higher precision, a transformation matrix is added into a nonlinear objective function of a three-phase machine, and the objective function is minimized by a Gauss-Newton or Levenberg-Marquardt method to realize camera parameter estimation:
where τ ═ { K ═l,Kr,Kc,kl,kr,kc,Rs,ts,Rp,tp},Kc、kcWhich are the internal parameters of the color camera, respectively.
General procedure for three-dimensional sensor calibration
Taking a plane target of a circular datum point as an example, the specific calibration process is as follows:
(1) target image reference point extraction: shooting a plurality of groups of target images, extracting circle center coordinates on the target images, corresponding to known three-dimensional coordinates of the reference points, and solving and optimizing system parameters of the sensor by taking the three-dimensional coordinates and the image coordinates of the reference points as input parameters;
(2) acquiring initial values of camera parameters: firstly, the distortion of a lens is not considered, and the internal and external parameters of the camera are estimated by adopting a linear camera model; in order to prevent the target function from being over-fitted in the next step and accelerate the convergence speed of the target function, the estimated parameters are further optimized by adopting a least square algorithm, and the obtained result is used as initial values of camera parameters and sensor structure parameters;
(3) nonlinear optimization of sensor parameters: lens distortion is added into a camera model, an optimization objective function is constructed by adopting a nonlinear camera model and sensor structure parameters, and optimal estimation of sensor parameters is realized by minimizing the objective function.
Referring back to fig. 1, in step 102, a three-dimensional sensor and a color texture camera are used to perform multi-view acquisition on an object to obtain a multi-view three-dimensional image and a two-dimensional color texture image. In one embodiment, the object to be measured may be placed on a rotating table, and the object to be measured may be collected using a three-dimensional sensor and a color texture camera while the rotating table is rotated, so that three-dimensional images and two-dimensional color texture images at a plurality of viewing angles including 360-degree information of the object to be measured may be collected.
Step 103 generates a three-dimensional mesh model using the multi-view three-dimensional image.
And 104, establishing a mapping relation between the two-dimensional color texture image and the three-dimensional grid model at each visual angle according to the system parameters so as to realize grid parameterization.
And 105, performing texture fusion based on the mapping relation to obtain a fused image so as to realize color texture reconstruction of the integral three-dimensional model. In one embodiment, the multi-view two-dimensional color texture image may be projected onto the three-dimensional network model according to the mapping relationship, and when the two-dimensional color texture images at all views are projected, the color three-dimensional texture model may be obtained, that is, the color texture reconstruction of the entire three-dimensional model is achieved.
And 106, generating a corresponding texture map according to the mapping relation. In order to facilitate storage, a plurality of texture maps are generated according to the mapping relation, and the three-dimensional grid model and the texture maps are stored in the formats of obj, ply, wrl and the like. It will be appreciated that in practical applications, new mesh parameterizations of the model are often required and new texture maps are generated.
The mean value operation of all texture images establishing the mapping relation is a direct and simple global texture fusion method, but actually, due to the relations of uneven illumination, object morphology change and the like, the brightness of the object surface images acquired by a color camera is inconsistent, so that the fusion result has obvious color jump. In order to realize the reconstruction of the three-dimensional color texture with the sense of reality, the patent proposes that the texture fusion based on the composite weight is an effective method for solving the problem of texture color jump and realizing the natural transition of texture boundaries with different visual angles. Namely, the confidence coefficient of each texture pixel is evaluated by a depth data definition complex weight method, and the fusion result is calculated by weighted average according to the confidence coefficient under each view angle.
Here, a Sigmoid kernel function (also called a logistic function) is introduced to perform the distribution of the composite weight. The Sigmoid kernel is defined as follows:
wherein f (·) epsilon (0,1), coefficients a and b are real numbers, and control the distribution of the Sigmoid curve. The Sigmoid kernel function is introduced into the complex weight to flexibly control the weight curve through the adjustment of the coefficient so as to meet the actual requirement. The complex weights include normal weights, depth weights, and edge weights.
The normal weight is assigned according to the angle between the normal direction of the object surface and the sight line direction of the camera. According to a classical Bidirectional Reflectance (BRDF) model, a camera acquires the brightness of the surface of an object and has a relation with parameters such as a light source incidence angle, the normal direction of the surface of the object, the sight line direction of the camera, the surface reflectance and the like. However, it is difficult to obtain accurate values of these parameters in practical applications, such as the specific spatial position of the light source, the true reflectivity of the object surface, etc. Therefore, we approximate the normal weights as Sigmoid functions. Setting the included angle between the normal direction of the object surface in a certain effective area of the image and the sight line direction of the camera as delta thetakThen the normal weight satisfies:
wherein xkThe larger the angle, the lower the weight for a pixel in the image. The normal weighting curve is shown in fig. 2(a), where the coefficient a is 0.1 and the coefficient b is 50 °.
The depth weight is assigned according to the distance from the object surface to the camera imaging plane. Since the camera imaging model is limited by the depth of field (DOF), when some regions of the object surface exceed the depth of field range of the lens, they become blurred due to defocus, thereby adversely affecting the texture fusion quality. The depth weight takes the optimal imaging distance as a reference, the larger the deviation degree is, the smaller the weight is, the weight is slowly attenuated in the depth of field range, the attenuation is fast at the position close to the depth of field boundary, and the fast cutoff is realized when the depth of field range is exceeded. The depth weights are defined as follows:
wherein D (-) represents the object surface point D to the optimal imaging reference plane D0The shortest distance of (c). The depth weight curve is shown in fig. 2(b), in which the coefficient a is 0.4, b is 50mm, and d is0＝55mm。
And the edge weight is distributed according to the shortest Euclidean distance from the target pixel point in the image to the edge contour line of the effective region. Because texture fusion under different viewing angles easily causes brightness jump at the edge, the closer the target pixel point is to the edge contour line of the effective area, the lower the weight is. Edge weights herein are defined as follows:
wherein D (-) represents the point xkThe shortest distance to the edge contour. The edge weight can significantly reduce color jump at the boundary of the texture, and is a very important weight function in texture fusion. The edge weight curve is shown in fig. 2(c), and the coefficients a ═ 0.08 and b ═ 50 mm.
The complex weight is constructed by multiplying the above three weights:
f(xk)＝fnorm(xk)·fdepth(xk)·fedge(xk) (15)
each weight is normalized, and the value range is between [0,1 ].
If the texture mapping relation under each view angle is accurate enough and the reconstructed geometric model is fine enough, the texture fusion based on the composite weight can realize good texture reconstruction effect. However, due to the comprehensive influence of each link of three-dimensional imaging such as system calibration, single-view depth data reconstruction, ICP matching and the like, it is difficult to actually satisfy these assumed conditions, which results in texture dislocation and blurring, and reduces texture reconstruction quality. Therefore, the patent also provides a texture fusion algorithm combining the composite weight and a Bidirectional similarity (BDS) function. The main idea of the algorithm is to introduce a target image between an original image and a fused image, reconstruct and calculate the original image by adopting a bidirectional similarity function according to the displacement condition of the fused image under each visual angle to generate an energy function, and reduce the image blur of the integral model texture fusion by displacing the global target image through minimizing the energy function so as to obtain a new target image. The bidirectional similarity function is introduced to deform according to the fused image in the target image reconstruction process, and simultaneously contain original image information as much as possible.
In 2008, Simakov et al defined the two-way similarity function as:
where S denotes an original image, T denotes a target image, S, T denote blocks of the original image and the target image, respectively, D (S, T) denotes a sum of squared differences of the blocks S and T in an RGB color space, α is a scaling parameter of two terms, L denotes the number of pixels in each block, for example, a block of 7 × 7 size, and L is 49. The first item on the right side of the formula (16) is a complete item (complete), which represents the integrity of information in the original image contained in the target image, and the lower the value is, the more complete the value is; the second term is a correlation term (Coherence) that indicates that new visual structures (e.g., caused by artifacts) appear in the target image relative to the original image, with lower values for fewer new visual structures. By minimizing this function, the target image is made to contain the original image information to the maximum extent under the visual correlation constraint.
However, the pure dependence on the bidirectional similarity function does not improve the texture fusion quality well, and it is also required to keep the optical measurement consistency between the target image and the fused image under multiple viewing angles (photomtrically Consistent). To this end, a further energy function is introduced:
wherein M isiRepresenting the fused image at the ith viewing angle, xkRepresenting pixel positions of the image, P (-) representing a projection function, e.g. Pi(Tj) And projecting the target image representing the jth view angle to the target image representing the ith view angle, wherein N represents the number of view angles. w is ajAnd (5) representing the composite weight of the image at the jth view angle, wherein the weight value is generated according to the formula (5). Extending equations (6) and (7) from a single view to a global view and constructing a final energy function:
where λ is two energy functions E1And E2A scaling factor in between. By minimizing the energy function E, a target image T at each view angle can be generatediThe target image is made to satisfy two constraints: similarity constraint, i.e. containing as much information as possible of the original image (corresponding to the energy function E)1) (ii) a Consistency (Consistency) constraint, i.e. maintaining Consistency with the fused image (corresponding to the energy function E)2)。
Texture alignment and fusion
As can be seen from the energy function (8), the objective function T1,...,TNAnd fusing the images M1,...,MNAre all variables. In order to find the optimal solution for the energy function (8), a two-step alternative optimization strategy is used herein, the basic idea of which is that when optimizing the target images, all fused images remain unchanged, whereas when generating the fused images, all target images remain unchanged. First, an initialization image, i.e., T, is initialized with an original image as a target image and a fused imagei＝Si,Mi＝SiThe two-step alternate optimization method comprises the following steps:
step S1: fixed fusion image MiOptimizing the target image Ti. At this stage, the image M is fused1,...,MNAnd treated as a known item. According to equation (18), the target image and the energy function E1And E2Are all related, so the target image T is solved separatelyi. For equation (16), according to the method of Simakov, a block search is performed by minimizing D (s, t), and the correspondence between all blocks in the target image and the block with the smallest error in the original image is determined. For a clearer description of the solution process, equation (16) is rewritten:
wherein E1(i,xk) Is the pixel x of the target image under the ith visual anglekEnergy function of(s)uAnd svCovering target image pixel x in complete term and related term of BSF function respectivelykThe block of the original image corresponding to the block of (a) is determined by block search, yuAnd yvAre respectively a block suAnd svAnd corresponds to x in the target image blockkU and V correspond to the number of tiles in the complete entry and the related entry, respectively, e.g., the tile size is 7 × 7, U and V are less than or equal to 49. From equation (19) one can see the energy function E1(i,xk) Is about Ti(xk) Is derived from (19) and the derivative is made 0 to obtain
Thereby obtaining an expression of the target image:
it can be seen from equation (21) that the target image of the first term is reconstructed from the information of the original image. Note that 1/L is reserved in the formula for the purpose of merging with formula (14).
Energy function E2Similar to the minimization solution method of (1), consider Pj(Pi(Tj))＝TjRewriting the energy function E2Expression (c):
the formula (22) is derived and the derivative is made 0 to obtain
To with E1The expressions of (a) and (b) are kept consistent, and the symbols i and j are exchanged to obtain TiExpression (c):
likewise, w is noti(xk) The approximation is for the combination with formula (21). It can be seen from equation (24) that the target image is solved by performing weighted average on the texture images at all the relevant current viewing angles, and this constraint shows that the target image is aligned according to the result of the fused image.
Finally, the energy function E is derived and the derivative is made 0, i.e.And combining equation (21) and equation (24) may obtain an expression of the target image:
step S2: fixed texture image TiOptimizing the fused image Mi. At this stage, the image M is fused1,...,MNTo optimize the parameters. According to formula (18), fusingImage only with energy function E2There is a relationship, so a similar method can be used to obtain a generation formula of a texture image:
as can be seen from equation (26), the fused image is obtained by performing weighted average on the target images at the respective relevant viewing angles. At the initial stage of iterative operation, if the target images at all the viewing angles are dislocated, the fused images can generate ghosting and blurring, and in the iterative operation process, the target images at all the viewing angles can be aligned according to the fused images, so that the ghosting and blurring of the images are continuously reduced in the reconstruction process of the fused images until the energy function E is smaller than the set critical value c, and then convergence is judged.
In the iterative operation process, in order to avoid trapping in local optimization and accelerate convergence speed, a multi-scale optimization method is adopted. In the low-scale phase, all images are down-sampled to a low resolution and subjected to the iterative operation described above. After the energy function E converges, the target image and the fused image are up-sampled to a larger scale, and the original image is still down-sampled, in order to inject the high-frequency information of the original image into the target image and the fused image. In some embodiments, the fused image is blurred in the initial stage, the gray curve has obvious ripples, and the image becomes clearer and the contrast of the gray curve is enhanced along with the iterative operation of different scales. In the iterative operation of the highest scale, the resolution of all images is adjusted to the initial resolution of the original image, and simultaneously the target images T under all visual angles are obtained1,...,TNAnd fusing the images M1,...,MN. In one embodiment, 10-level scale is used for multi-scale optimization, wherein the image size l of any one dimension of the image of the ith leveliThe calculation formula of (2) is as follows:
li＝(l0/8)·8(i-1)/9 (27)
l0for the original image size, e.g. of the original imageThe resolution is 5520 pixels × 3680 pixels, then the image resolution at level 1 scale is 690 pixels × 460 pixels.
And finally obtaining new target images with the highest resolution under all the visual angles through iterative operation of the two-step method, performing alignment optimization on the target images, and fusing all the target images by adopting a composite weight fusion algorithm to obtain the final texture mapping of the three-dimensional model. The overall flow of the BSF-based color texture fusion algorithm is shown in fig. 3.
The foregoing is a more detailed description of the invention in connection with specific/preferred embodiments and is not intended to limit the practice of the invention to those descriptions. It will be apparent to those skilled in the art that various substitutions and modifications can be made to the described embodiments without departing from the spirit of the invention, and these substitutions and modifications should be considered to fall within the scope of the invention.
Claims (10)
1. A method for reconstructing a realistic three-dimensional color texture, comprising:
pre-calibrating system parameters of a three-dimensional sensor and a color texture camera;
acquiring a multi-view three-dimensional image and a two-dimensional color texture image;
generating a three-dimensional mesh model using the multi-view three-dimensional image;
establishing a mapping relation between the two-dimensional color texture image and the three-dimensional grid model under each visual angle according to the system parameters;
performing texture fusion based on the mapping relation to obtain a fused image so as to realize color texture reconstruction of the integral three-dimensional model;
and generating a corresponding texture mapping according to the mapping relation.
2. The method according to claim 1, wherein the texture fusion is to evaluate the confidence of each texel by a depth data-defined complex weight method, and perform weighted average according to the confidence at each view angle to calculate the fusion result.
3. The method according to claim 2, wherein the complex weight is calculated by the following formula:
f(xk)＝fnorm(xk)·fdepth(xk)·fedge(xk)
wherein the normal weightDepth weightingEdge weightsAnd each weight is normalized to have a value range of [0, 1%]。
4. The method according to claim 3, wherein the coefficients in the normal weight take the values of: a is 0.1, b is 50 °, and coefficients in the depth weight take values of: a is 0.4, b is 50mm, d0The coefficient value in the edge weight is 55 mm: a is-0.08 and b is 50 mm.
5. The method according to claim 1, wherein a target image is introduced between the original two-dimensional color texture image and the fused image, and a two-way similarity function E is used according to the displacement of the fused image at each view angleBDSAnd (S, T) carrying out reconstruction calculation on the original two-dimensional color texture image to generate an energy function, and reducing image blurring of integral model texture fusion by minimizing the energy function so as to enable the overall target image to be displaced, thereby finally obtaining a new high-resolution target image.
6. The method of claim 5, wherein the energy function further comprises an optical measure consistency function EC：
Wherein M isiRepresenting said fused image at the ith viewing angle, xkRepresenting the pixel position of the image, P (-) represents the projection function, and N represents the number of views. w is ajRepresenting the complex weight of the image at the jth view.
7. The method of realistic three-dimensional color texture reconstruction according to claim 6, characterized in that the energy function E is finally configured to:
E＝E1+λE2
where λ is two energy functions E1And E2A scaling factor in between.
8. The method according to claim 7, wherein the objective function is solved by a two-step alternative optimization strategy, that is, the following two steps are solved iteratively:
s1, fixing the fusion image MiOptimizing said target image Ti；
S2: fixing the target image TiOptimizing said fused image Mi。
9. The method for reconstructing a realistic three-dimensional color texture according to claim 8, wherein in the step S1, the target image Ti is expressed by the following formula:
in step S2, the objective function Mi is represented by the following formula:
10. the method according to claim 9, wherein a multi-scale optimization method is adopted in the iterative operation process, that is, in a low-scale stage, all images are down-sampled to a low resolution and the iterative operation is performed. After the energy function E converges, the target image and the fused image are up-sampled to a larger scale, and the original two-dimensional color texture image is still down-sampled, in order to inject the high-frequency information of the original two-dimensional color texture image into the target image and the fused image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910687176.1A CN110599578A (en) | 2019-07-29 | 2019-07-29 | Realistic three-dimensional color texture reconstruction method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910687176.1A CN110599578A (en) | 2019-07-29 | 2019-07-29 | Realistic three-dimensional color texture reconstruction method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110599578A true CN110599578A (en) | 2019-12-20 |
Family
ID=68852938
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910687176.1A Pending CN110599578A (en) | 2019-07-29 | 2019-07-29 | Realistic three-dimensional color texture reconstruction method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110599578A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111539251A (en) * | 2020-03-16 | 2020-08-14 | 重庆特斯联智慧科技股份有限公司 | Security check article identification method and system based on deep learning |
| CN113487729A (en) * | 2021-07-30 | 2021-10-08 | 上海联泰科技股份有限公司 | Surface data processing method and system of three-dimensional model and storage medium |
| CN113538649A (en) * | 2021-07-14 | 2021-10-22 | 深圳信息职业技术学院 | Super-resolution three-dimensional texture reconstruction method, device and equipment |
| CN114049423A (en) * | 2021-10-13 | 2022-02-15 | 北京师范大学 | Automatic realistic three-dimensional model texture mapping method |
| CN115546379A (en) * | 2022-11-29 | 2022-12-30 | 思看科技(杭州)股份有限公司 | Data processing method and device and computer equipment |
| CN115601490A (en) * | 2022-11-29 | 2023-01-13 | 思看科技（杭州）股份有限公司（Cn） | Texture image pre-replacement method and device based on texture mapping and storage medium |
| CN116758205A (en) * | 2023-08-24 | 2023-09-15 | 先临三维科技股份有限公司 | Data processing method, device, equipment and medium |
| CN117830392A (en) * | 2024-03-05 | 2024-04-05 | 季华实验室 | Environmental object identification method and imaging system |
| CN113592698B (en) * | 2021-08-16 | 2024-04-26 | 齐鲁工业大学 | Sixteen-element moment-based multi-view color image zero watermark processing method and system |
-
2019
- 2019-07-29 CN CN201910687176.1A patent/CN110599578A/en active Pending
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111539251A (en) * | 2020-03-16 | 2020-08-14 | 重庆特斯联智慧科技股份有限公司 | Security check article identification method and system based on deep learning |
| CN113538649A (en) * | 2021-07-14 | 2021-10-22 | 深圳信息职业技术学院 | Super-resolution three-dimensional texture reconstruction method, device and equipment |
| CN113487729A (en) * | 2021-07-30 | 2021-10-08 | 上海联泰科技股份有限公司 | Surface data processing method and system of three-dimensional model and storage medium |
| CN113592698B (en) * | 2021-08-16 | 2024-04-26 | 齐鲁工业大学 | Sixteen-element moment-based multi-view color image zero watermark processing method and system |
| CN114049423A (en) * | 2021-10-13 | 2022-02-15 | 北京师范大学 | Automatic realistic three-dimensional model texture mapping method |
| CN115546379A (en) * | 2022-11-29 | 2022-12-30 | 思看科技(杭州)股份有限公司 | Data processing method and device and computer equipment |
| CN115601490A (en) * | 2022-11-29 | 2023-01-13 | 思看科技（杭州）股份有限公司（Cn） | Texture image pre-replacement method and device based on texture mapping and storage medium |
| CN116758205A (en) * | 2023-08-24 | 2023-09-15 | 先临三维科技股份有限公司 | Data processing method, device, equipment and medium |
| CN116758205B (en) * | 2023-08-24 | 2024-01-26 | 先临三维科技股份有限公司 | Data processing method, device, equipment and medium |
| CN117830392A (en) * | 2024-03-05 | 2024-04-05 | 季华实验室 | Environmental object identification method and imaging system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110599578A (en) | Realistic three-dimensional color texture reconstruction method | |
| Maier et al. | Intrinsic3D: High-quality 3D reconstruction by joint appearance and geometry optimization with spatially-varying lighting | |
| CN110288642B (en) | Three-dimensional object rapid reconstruction method based on camera array | |
| KR100681320B1 (en) | Method for modelling three dimensional shape of objects using level set solutions on partial difference equation derived from helmholtz reciprocity condition | |
| JP5285619B2 (en) | Camera system calibration | |
| JP4324238B2 (en) | Image processing system, method and apparatus | |
| CN107155341B (en) | Three-dimensional scanning system and frame | |
| CN110458932B (en) | Image processing method, device, system, storage medium and image scanning apparatus | |
| Bimber et al. | Enabling view-dependent stereoscopic projection in real environments | |
| CN106875339A (en) | A kind of fish eye images joining method based on strip scaling board | |
| Douxchamps et al. | High-accuracy and robust localization of large control markers for geometric camera calibration | |
| CN103503025A (en) | Determining model parameters based on transforming a model of an object | |
| Aliaga et al. | A self-calibrating method for photogeometric acquisition of 3D objects | |
| CN115830103A (en) | Monocular color-based transparent object positioning method and device and storage medium | |
| WO2013076605A1 (en) | Method and system for alignment of a pattern on a spatial coded slide image | |
| CN110738608B (en) | Plane image correction method and system | |
| CN109945841B (en) | Industrial photogrammetry method without coding points | |
| CN112308963A (en) | Non-inductive three-dimensional face reconstruction method and acquisition reconstruction system | |
| Von Gioi et al. | Towards high-precision lens distortion correction | |
| CN110349257B (en) | Phase pseudo mapping-based binocular measurement missing point cloud interpolation method | |
| CN113360964A (en) | Convergence type binocular vision guided robot positioning method under high dynamic range | |
| Huang et al. | End-to-end full projector compensation | |
| Huang et al. | Deprocams: Simultaneous relighting, compensation and shape reconstruction for projector-camera systems | |
| Forster et al. | Towards 3D reconstruction of endoscope images using shape from shading | |
| CN109859137A (en) | A kind of irregular distortion universe bearing calibration of wide angle camera |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |