CN110599578A - Realistic three-dimensional color texture reconstruction method - Google Patents

Abstract

The present invention provides a realistic three-dimensional color texture reconstruction method, comprising: pre-calibrating the system parameters of a three-dimensional sensor and a color texture camera; acquiring a multi-view three-dimensional image and a two-dimensional color texture image; using the multi-view three-dimensional image to generate a three-dimensional grid Model; establish the mapping relationship between the two-dimensional color texture image and the three-dimensional mesh model under each viewing angle by the system parameters; perform texture fusion based on the mapping relationship to obtain a fusion image to realize the color texture of the overall three-dimensional model Reconstruction; generate a corresponding texture map according to the mapping relationship. The present invention evaluates the confidence of texture color by introducing compound weight parameters. Texture discontinuities can be removed by weighted averaging of projected texture images. For imprecise geometries, a bidirectional similarity function representing the structural similarity of two images is introduced to correct inconsistencies and generate realistic textures.

Description

A Realistic 3D Color Texture Reconstruction Method

technical field

The invention belongs to the field of electronic technology, and more specifically relates to a realistic three-dimensional color texture reconstruction method.

Background technique

3D measurement technology has been widely used in many industries and disciplines such as urban surveying, anthropometry, prototyping, etc. Optical 3D measurement technology provides a flexible method for acquiring 3D images. With the development of high-performance optoelectronic devices such as charge-coupled devices (CCD) and digital light processing (DLP) projectors, optical three-dimensional measurement can obtain high-sensitivity, high-speed data. Structured light 3D measurement technology illuminated by dynamic spatially varying patterns has been widely used in 3D measurement systems, and the patterns can be periodic stripes, 2D grids, or random spots. The geometry of the object is encoded by the distorted structured light pattern in order to be accurately decoded from the captured image.

However, 3D geometric measurements cannot resolve color issues. In general, multi-view images are captured for mapping on geometric surfaces to generate color information independent of geometric reconstruction. Any errors in geometry or camera pose can make the mapping difficult to align, inconsistent lighting between different views can lead to unrealistic colors, and the above problems will lead to texture artifacts such as blurring, ghosting and color discontinuity.

In order to solve the above problems, a variety of texture reconstruction methods have been proposed. Color consistency can be improved by image fusion in the spatial or frequency domain, but these methods do so at the expense of image clarity. Image stitching using Markov random field optimization method is another way to avoid image degradation, however visible seams cannot be completely eliminated. Post-processing is usually used to adjust the color of texture blocks at seams, such as Poisson fusion, thermal diffusion, and other color adjustment methods. The misalignment can be corrected by optimizing the camera pose, such as manual camera calibration, methods based on mutual information, and methods for maximizing color consistency. Some methods deal with bias by correcting the input image using a non-rigid calibration technique. For example, the optical flow method for image warping is introduced to solve the image deviation problem. Additionally, super-resolution methods have been proposed to overcome the blurring problem. A recent approach proposes a patch-based optimization method for texture mapping of multiple images. In the above methods, although various means are used to eliminate various artifacts, all of them require manual participation, thus reducing the efficiency.

To solve at least one of the above problems, this paper proposes a photorealistic 3D color texture reconstruction method.

Contents of the invention

In order to solve the above problems, the present invention provides a realistic three-dimensional color texture reconstruction method, which is characterized in that it includes: pre-calibrating the system parameters of the three-dimensional sensor and the color texture camera; acquiring multi-view three-dimensional images and two-dimensional color texture images; using the The multi-view 3D image generates a 3D mesh model; the mapping relationship between the 2D color texture image and the 3D mesh model under each viewing angle is established by the system parameters; texture fusion is performed based on the mapping relationship to obtain fusion Image to realize the color texture reconstruction of the overall three-dimensional model; generate the corresponding texture map according to the mapping relationship.

In some embodiments, the texture fusion is to evaluate the confidence of each texel by defining composite weights through depth data, and perform a weighted average according to the confidence of each viewing angle to calculate the fusion result. The composite weight is calculated by the following formula:

f(x _k )＝f _norm (x _k )·f _depth (x _k )·f _edge (x _k )

Among them, the normal weight depth weight edge weight And each weight is normalized, and the value range is [0,1]. The value of the coefficient in the normal weight is: a=0.1, b=50°, the value of the coefficient in the depth weight is: a=0.4, b=50mm, d ₀ =55mm, the edge weight The values of the coefficients are: a=-0.08, b=50mm.

In some embodiments, a target image is introduced between the original two-dimensional color texture image and the fused image, and according to the displacement of the fused image at each viewing angle, a bidirectional similarity function E _BDS (S, T) is used The original two-dimensional color texture image is reconstructed and calculated to generate an energy function, and the image blur of the overall model texture fusion is reduced by minimizing the energy function so that the global target image is displaced, and finally a new high-resolution image is obtained. target image.

In some embodiments, the energy function also includes an optical measure consistency function E _C :

Where M _i represents the fused image under the i-th viewing angle, x _k represents the pixel position of the image, P(·) represents a projection function, and N represents the number of viewing angles. _wj denotes the composite weight of the image at the jth viewing angle.

In some embodiments, the energy function E is finally constructed as:

E＝E ₁ +λE ₂

where λ is the scaling factor between the _two energy functions _E1 and E2.

In some embodiments, the objective function is solved by a two-step alternate optimization strategy, that is, the following two steps are iteratively solved: S1: fix the fused image M _i , optimize the target image T _i ; S2: fix the The target image T _i is used to optimize the fused image M _i .

In the step S1, the target image Ti is expressed by the following formula:

In the step S2, the objective function Mi is represented by the following formula:

In some embodiments, a multi-scale optimization method is adopted in the iterative operation process, that is, in the low-scale stage, all images are down-sampled to a low resolution and the above-mentioned iterative operation is performed. After the energy function E converges, the target image and the fused image are up-sampled to a larger scale, while the original two-dimensional color texture image is still down-sampled, in order to reduce the high Injecting video information into the target image and the fused image.

Beneficial effects of the present invention: A realistic three-dimensional color texture reconstruction method is proposed, which evaluates the confidence of texture color by introducing composite weight parameters. Texture discontinuities can be removed by weighted averaging of projected texture images. For imprecise geometries, a Bidirectional Similarity (BDS) function representing the structural similarity of two images is introduced to correct inconsistencies and generate realistic textures.

Description of drawings

Fig. 1 is a schematic diagram of a realistic three-dimensional color texture reconstruction method according to an embodiment of the present invention.

Fig. 2 is a schematic diagram of various weight curves according to an embodiment of the present invention.

Fig. 3 is a flowchart of a BSF-based color texture fusion algorithm according to an embodiment of the present invention.

Detailed ways

The present invention will be described in further detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be emphasized that the following descriptions are only exemplary and not intended to limit the scope of the present invention and its application.

FIG. 1 is a schematic diagram of a realistic three-dimensional color texture reconstruction method according to an embodiment of the present invention.

In step 101, the system parameters of the 3D sensor and the color texture camera are pre-calibrated. The 3D sensor here is used to acquire depth images, that is, 3D images. The 3D sensor can be a binocular vision 3D sensor based on structured light technology, such as a binocular vision 3D sensor composed of a digital stripe projector and a dual-color camera. Of course, any 3D sensor that can collect 3D images can be used, such as a monocular structured light 3D sensor, a time-of-flight 3D sensor, and the like. The color texture camera is used to collect high-resolution object texture information, for example, a high-resolution SLR camera can be used. Calibration means to calibrate the internal parameters of the 3D sensor and the color texture camera, and also to calibrate the relative external parameters of the two.

In one embodiment, the 3D sensor contains three cameras, two of which are black and white industrial cameras used to generate depth data, and the third is a color camera used to capture texture pictures. The calibration of the 3D sensor is the prerequisite for the subsequent generation of depth data and texture fusion. The following will start from the mathematical model of a single camera to describe the calibration of the binocular sensor composed of black and white cameras and the principle of color texture camera calibration.

camera model.

If the diffraction effect of the imaging system is ignored, and assuming that the camera lens strictly satisfies the paraxial condition, the camera imaging can be equivalent to the pinhole imaging, and the imaging process satisfies the perspective projection transformation, and the object point is marked as X _w = in the world coordinate system (X _w ,Y _w ,Z _w ) ^T , the ideal pixel in the image coordinate system is m _c =(u,v) ^T , then the imaging process is expressed as

where superscript Indicates homogeneous coordinates; X _c indicates the coordinates of the object point in the camera coordinate system; is the projection of X _c on the camera image plane; R _c and t _c are the rotation matrix and translation vector from the world coordinate system to the camera coordinate system respectively, which are called the external parameters of the camera; K _c is the internal parameter matrix of the camera, including The equivalent focal length of the coordinate axis (fu , f _v ) ^T , the projection of the optical center on the image plane, that is, the principal point ( _{u 0} , v ₀ ) ^T of the image plane, and the tilt factor γ of the image; M _c ＝K _c [R _c |t _c ] is called the projection matrix, which also includes the internal and external parameters of the camera; λ=Z _c is the scale factor.

In a real optical imaging system, due to the influence of factors such as the processing technology and structural assembly of the imaging lens, there will inevitably be a deviation between the actual imaging surface and the above-mentioned ideal imaging surface, which is called camera lens distortion. The classic Brown-Conrady model is currently the most widely used lens distortion model, and its distortion can be expressed as:

x′ _c ＝x _c +Δ(x _c ),

Among them, x' _c = (x' _c , y' _c ) ^T represents a distorted image point; Δ(x _c ) represents a distortion item, including radial distortion (radial distortion) and centrifugal distortion (decentering distortion); is the distance from the undistorted image point to the principal point; (k ₁ ,k ₂ ,k ₃ ,…) and (p ₁ ,p ₂ ,p ₃ ,…) are the radial and centrifugal distortion parameters, respectively. Usually three radial distortions and two centrifugal distortions have met the accuracy requirements. Let k=(k ₁ ,k ₂ ,k ₃ ,p ₁ ,p ₂ ) ^T represent the distortion parameter vector, then the camera model including lens distortion is expressed as

In the nonlinear camera model represented by formula (3), k and K _c represent the internal parameters of the camera, and R _c and t _c represent the external parameters of the camera. The camera calibration process is usually to minimize the reprojection error from the target reference point to the actual image point, that is, X _c →m _c , so the ideal image can be calculated by the analytical expression x′ _c =x _c +Δ(x _c ; Point plus distortion, that is, calculate the distorted image point x′ _c from the undistorted image point x _c . The 3D reconstruction process is to construct an accurate object point through actual image points, that is, X _c → m _c . The 3D reconstruction process is to reconstruct the spatial coordinates of the object point through the actual image point, that is, m _c → X _c . This process needs to remove the distortion, that is, calculate the undistorted image point x _c from the actual image point x′ _c with distortion. Since formula (2) is a complex nonlinear function, the analytical expression of its inverse function cannot be obtained. Considering that the distortion term is relatively small, the numerical solution of the undistorted image point can be obtained by recursive approximation:

Using the distorted image point to approximate the initial undistorted image point, a more accurate undistorted image point can be obtained by controlling the number of iterations.

Binocular sensor calibration.

Two left and right cameras can form a three-dimensional sensor based on binocular stereo vision, and the world coordinate system is usually set up on the camera. Taking the left camera as an example, the mathematical model of the binocular sensor is expressed as

X _l ＝R _l X _w +t _l (5)

Among them, I is the identity matrix, R _s and t _s are the rotation matrix and translation vector from the left camera coordinate system to the right camera coordinate system, [R _s |t _s ] represents the structural parameters of the sensor, satisfying

The goal of binocular setting is to determine the internal parameters of the two cameras and the structural parameters between the two cameras.

The process of camera calibration usually takes the reprojection error between the projected image point of the target reference point and the actual measured image point as the optimization objective function, and obtains the optimal estimated value of the internal and external parameters from the target image data. The objective function of binocular sensor calibration is expressed as

Among them, m′ _l and m′ _r are the real image coordinates, and are the reprojected image coordinates of the fiducial points calculated from the model. Gauss-Newton or Levenberg-Marquardt optimization algorithm ^[207] can be used to optimize and solve the above formula, and finally obtain the system parameters.

Color Texture Camera Calibration.

The function of the color texture camera is to obtain the color two-dimensional image information of the object, and obtain the color information of the three-dimensional mesh by establishing the mapping relationship between the three-dimensional geometric model and the two-dimensional color image, and finally realize the color three-dimensional imaging. In order to realize the mapping from the 3D geometric model to the 2D color image, it is necessary to solve the internal parameters and structural parameters of the color texture camera in advance, that is, to calibrate the color texture camera. Usually, there are two ways for color 3D sensors to acquire color images of objects: ① The two cameras of the binocular sensor are color cameras, which can simultaneously acquire depth information and color texture information. The internal and external parameters of the camera are determined, and no additional calibration operation is required; ②The binocular camera is separated from the color camera, and an additional calibration of the color camera is required. In practical application, both methods have advantages and disadvantages. The first method is simple in structure and low in cost, but because the color camera is affected by the Bayer filter imaging principle, the grayscale accuracy of the grayscale image converted from the color image is lower than that obtained directly from the black and white camera, thus affecting the depth The precision of the data. In addition, considering the three-dimensional measurement speed, the resolution of the industrial camera used by the binocular sensor cannot be too high, thus limiting the resolution of the color texture image. The second method has a relatively complex structure and high cost, but since it is not limited by depth data collection, a dedicated color camera can be selected according to the needs, such as a professional SLR camera, etc., so as to achieve very high resolution and color reproduction.

Let the left camera coordinate system be the 3D sensor coordinate system, then the structural parameters of the three cameras are:

Among them, R _t and t _t are the rotation matrix and translation vector from the world coordinate system to the color camera coordinate system, respectively, and R _p and t _p are the rotation matrix and translation vector between the left camera and the color texture camera. In order to obtain higher-precision structural parameters, we add the transformation matrix to the nonlinear objective function of the three-camera, and minimize the objective function through the Gauss-Newton or Levenberg-Marquardt method to achieve camera parameter estimation:

Where τ={K _l , K _r , K _c , k _l , k _r , k _c , R _s , t _s , R _p , t _p }, K _c and k _c are the internal references of the color camera respectively.

General process of 3D sensor calibration

Taking the plane target with a circular reference point as an example, the specific calibration process is as follows:

(1) Target image fiducial point extraction: take multiple groups of target images, extract the coordinates of the center of the target image and correspond to the known three-dimensional coordinates of the fiducial point, and use the three-dimensional coordinates of the fiducial point and the image coordinates as input parameters to solve the problem and optimize the system parameters of the sensor;

(2) Obtain the initial value of the camera parameters: ignore the lens distortion first, and use the linear camera model to estimate the internal and external parameters of the camera; The parameters are further optimized, and the obtained results are used as the initial values of the camera parameters and sensor structure parameters;

(3) Nonlinear optimization of sensor parameters: Lens distortion is added to the camera model, the optimization objective function is constructed by using the nonlinear camera model and sensor structure parameters, and the optimal estimation of sensor parameters is realized by minimizing the objective function.

Returning to FIG. 1 , in step 102 , the three-dimensional sensor and the color texture camera are used to collect multi-view images of the object to obtain multi-view three-dimensional images and two-dimensional color texture images. In one embodiment, the object to be measured can be placed on a rotating platform. When the rotating table rotates, the three-dimensional sensor and the color texture camera are used to collect the object to be measured, so that the image containing the 360-degree information of the object to be measured can be collected. 3D images from multiple viewing angles and 2D color texture images.

Step 103 uses the multi-view 3D images to generate a 3D mesh model.

Step 104 establishes a mapping relationship between the two-dimensional color texture image and the three-dimensional mesh model at each viewing angle from the system parameters to realize mesh parameterization.

Step 105 performs texture fusion based on the mapping relationship to obtain a fused image to realize color texture reconstruction of the overall 3D model. In one embodiment, the multi-view two-dimensional color texture images can be projected onto the three-dimensional network model according to the mapping relationship. After the two-dimensional color texture images under all viewing angles are projected, the color three-dimensional texture model can be obtained, namely The color texture reconstruction of the overall 3D model is realized.

Step 106 generates a corresponding texture map according to the mapping relationship. For the convenience of storage, multiple texture maps are generated according to the mapping relationship, and the 3D mesh model and texture maps are saved in obj, ply, wrl and other formats. Understandably, in practice, new mesh parameterizations and new texture maps for this model need to be frequently generated.

It is a direct and simple global texture fusion method to perform mean calculation on all texture images that establish a mapping relationship, but in fact, due to uneven illumination and changes in object shape, the brightness of object surface images collected by color cameras is also inconsistent. This leads to a more obvious color jump in the fusion result. In order to achieve realistic 3D color texture reconstruction, this patent proposes that texture fusion based on compound weights is an effective method to solve texture color jumps and realize natural transition of texture boundaries from different perspectives. That is, the confidence of each texel is evaluated by defining the composite weight through the depth data, and the fusion result is calculated by weighted average according to the confidence of each viewing angle.

Here, the Sigmoid kernel function (also called logistic function) is introduced to assign compound weights. The Sigmoid kernel function is defined as follows:

Among them, f(·)∈(0,1), the coefficients a and b are real numbers, which control the distribution of the Sigmoid curve. The purpose of introducing the Sigmoid kernel function in the composite weight is to flexibly control the weight curve through the adjustment of coefficients to meet actual needs. Composite weights include normal weights, depth weights, and edge weights.

The normal weight is assigned based on the angle between the object's surface normal and the camera's line of sight. According to the classic bidirectional reflectance (BRDF) model, the brightness of the surface of the object collected by the camera is related to the incident angle of the light source, the normal direction of the object surface, the direction of the camera's line of sight, and the surface reflectance. However, it is difficult to obtain accurate values of these parameters in practical applications, such as the specific spatial position of the light source, the real reflectivity of the object surface, and so on. Therefore, we approximate the normal weights as a sigmoid function. Assuming that the angle between the normal of the object surface and the direction of the camera's line of sight in a certain effective area of the image is Δθ _k , the weight of the normal direction satisfies:

Among them, x _k is the pixel in the image, and the larger the included angle, the lower the weight. The normal weight curve is shown in Fig. 2(a), the coefficients in the curve are a=0.1, b=50°.

The depth weight is assigned according to the distance from the object surface to the camera imaging plane. Because the camera imaging model is limited by the depth of field (DOF), when some areas of the object surface exceed the depth of field range of the lens, they will become blurred due to defocus, which will adversely affect the quality of texture fusion. The depth weight is based on the optimal imaging distance, the greater the deviation, the smaller the weight, and the weight decays slowly within the depth of field range, and the decay is fast near the depth of field boundary, and it cuts off quickly beyond the depth of field range. The depth weights are defined as follows:

where D( ) represents the shortest distance from the object surface point d to the best imaging datum plane d ₀ . The depth weight curve is shown in Fig. 2(b), and the coefficients in the curve are a=0.4, b=50mm, and d ₀ =55mm.

The edge weight is assigned according to the shortest Euclidean distance from the target pixel point in the image to the edge contour line of the effective area. Since texture fusion under different viewing angles is likely to cause brightness jumps at the edge, the closer the target pixel is to the edge contour of the effective area, the lower the weight. The edge weights in this paper are defined as follows:

where D( ) represents the shortest distance from point x _k to the edge contour. Edge weight can significantly reduce the color jump at the texture boundary, and is a very important weight function in texture fusion. The edge weight curve is shown in Fig. 2(c), coefficient a=-0.08, b=50mm.

Composite weights are constructed by multiplying the above three weights:

f(x _k )＝f _norm (x _k )·f _depth (x _k )·f _edge (x _k ) (15)

Each weight is normalized, and the value range is between [0,1].

If the texture mapping relationship in each viewing angle is accurate enough and the reconstructed geometric model is fine enough, texture fusion based on compound weights can achieve good texture reconstruction effect. However, due to the combined effects of system calibration, single-view depth data reconstruction, and ICP matching and other 3D imaging links, it is actually difficult to meet these assumptions, resulting in texture misalignment and blurring, and reducing the quality of texture reconstruction. For this reason, this patent also proposes a texture fusion algorithm combining compound weights and a bidirectional similarity (BDS) function. The main idea of the algorithm is to introduce the target image between the original image and the fused image, and according to the displacement of the fused image at each viewing angle, use the bidirectional similarity function to reconstruct the original image and generate an energy function. By minimizing the energy function, The global target image is displaced to reduce the image blurring of the overall model texture fusion to obtain a new target image. The bidirectional similarity function is introduced in order to deform according to the fused image during the reconstruction of the target image while containing the original image information as much as possible.

In 2008, Simakov et al. defined the bidirectional similarity function as:

Where S represents the original image, T represents the target image, s, t represent the blocks of the original image and the target image respectively, D(s, t) represents the sum of the square differences between blocks s and t in the RGB color space, and α is The ratio parameter of two items, L represents the number of pixels in each block, for example, for a block with a size of 7×7, L=49. The first item on the right side of formula (16) is Completeness, which means that the target image contains the completeness of the information in the original image, and the lower the value, the more complete; the second item is Coherence, which means that in the target image The new visible structure that appears relative to the original image (eg, caused by artifacts), the lower the value, the less the new visible structure. By minimizing this function, the target image can contain the original image information to the greatest extent under the constraint of visual correlation.

However, relying solely on the bidirectional similarity function cannot improve the quality of texture fusion very well, and it is also necessary to keep the target image under multiple views and the fusion image photometrically consistent. To this end, another energy function is introduced:

Where M _i represents the fused image at the i-th viewing angle, x _k represents the pixel position of the image, P(·) represents the projection function, for example, P _i (T _j ) represents the target image at the j-th viewing angle to the i-th viewing angle The target image is projected, and N represents the number of viewing angles. w _j represents the composite weight of the image at the jth viewing angle, and the weight value is generated according to formula (5). Extend formulas (6) and (7) from single perspective to global perspective, and construct the final energy function:

where λ is the scaling factor between the _two energy functions _E1 and E2. By minimizing the energy function E, the target image T _i under each viewing angle can be generated, so that the target image satisfies two constraints: the similarity (Similarity) constraint, that is, contains the information of the original image as much as possible (corresponding to the energy function E ₁ ) ; Consistency constraint, that is, to maintain consistency with the fused image (corresponding to the energy function E ₂ ).

Texture Alignment and Blending

It can be seen from the energy function formula (8) that the target functions T ₁ ,...,T _N and the fused images M ₁ ,...,M _N are all variables. In order to obtain the optimal solution of the energy function (8), this paper uses a two-step alternate optimization strategy. The basic idea is that when the target image is optimized, all fused images remain unchanged, and when the fused image is generated, all target images remain unchanged. Change. First initialize the initial image with the original image as the target image and the fused image, that is, T _i =S _i , M _i =S _i , the two-step alternate optimization method is as follows:

Step S1: Fix the fused image M _i and optimize the target image T _i . At this stage, the fused images M ₁ ,...,M _N are taken as known items. According to formula (18), the target image is related to both energy functions E ₁ and E ₂ , so the target image T _i is solved separately. For formula (16), according to Simakov's method, block search is performed by minimizing D(s, t), and the corresponding relationship between all blocks in the target image and the block with the smallest error in the original image is determined. In order to describe the solution process more clearly, rewrite equation (16):

Where E ₁ (i,x _k ) is the energy function at the target image pixel x _k at the i-th viewing angle, s _u and s _v are the complete term of the BSF function and the block covering the target image pixel x _k in the related term, respectively The block corresponding to the original image is determined by block search, y _u and y _v are the pixels in blocks s _u and s _v respectively, and correspond to the pixel position of x _k in the target image block, U and V are respectively Corresponding to the number of blocks in the complete item and the related item, for example, if the size of the block is 7×7, then U and V are less than or equal to 49. It can be seen from formula (19) that the energy function E ₁ (i, x _k ) is a quadratic equation about T _i (x _k ), and taking the derivative of (19) and setting the derivative to 0, we get

Thus, the expression of the target image is obtained:

From formula (21), it can be seen that the target image of the first item is reconstructed according to the information of the original image. Note that 1/L is retained in Eq. (14) for merging.

The solution to the minimization of the energy function E ₂ is similar. Considering P _j (P _i (T _j ))=T _j , rewrite the expression of the energy function E ₂ as follows:

Deriving formula (22) and setting the derivative to 0, we get

In order to be consistent with the expression of E ₁ , the symbols i and j are exchanged to obtain the expression of T _i :

Similarly, _wi (x _k ) is not reduced for the purpose of merging with formula (21). From formula (24), it can be seen that the target image is solved by weighted average of the texture images under all current relevant viewing angles. This constraint reflects that the target image will be aligned according to the result of the fusion image.

Finally, take the derivative of the energy function E and set the derivative to be 0, that is And combining formula (21) and formula (24), the expression of the target image can be obtained:

Step S2: Fix the texture image T _i and optimize the fusion image M _i . In this stage, the fused images M ₁ ,...,M _N are optimized parameters. According to formula (18), the fused image is only related to the energy function _E2 , so a similar method can be used to obtain the generation formula of the texture image:

It can be seen from formula (26) that the fused image is obtained by weighting and averaging the target images under each relevant viewing angle. At the beginning of the iterative operation, if the target images in each viewing angle are misaligned, the fused image will produce ghosting and blurring. During the iterative operation, the target images in each viewing angle will be aligned according to the fused image. During the image reconstruction process, the ghosting and blurring of the image are continuously reduced until the energy function E is less than the set critical value c, then it is judged as convergent.

In the iterative operation process, in order to avoid falling into local optimum and accelerate the convergence speed, we adopt a multi-scale optimization method. In the low-scale stage, all images are down-sampled to a low resolution and the above-mentioned iterative operation is performed. After the energy function E converges, the target image and the fusion image are up-sampled to a larger scale, while the original image is still down-sampled, in order to inject the high-frequency information of the original image into the target image and the fusion image. In some embodiments, the fused image is blurred at the initial stage, and the grayscale curve has obvious ripples. With the iterative operation of different scales, the image becomes clearer and the contrast of the grayscale curve is also enhanced. In the highest-scale iterative operation, the resolution of all images is adjusted to the initial resolution of the original image, and the target images T ₁ ,...,T _N and fused images M ₁ ,..., M _N . In one embodiment, 10-level scales are used for multi-scale optimization, wherein the formula for calculating the image size l _i of any one-dimensional image of the i-th level is:

l _i =(l ₀ /8)·8 ^(i-1)/9 (27)

l ₀ is the initial image size of the original image. For example, the initial resolution of the original image is 5520pixel×3680pixel, and the image resolution at the first level of scale is 690pixel×460pixel.

After the iterative operation of the above two-step method, the new target image with the highest resolution under all viewing angles is finally obtained, and the target image has been aligned and optimized. At this time, the composite weight fusion algorithm is used to fuse all the target images. Get the final texture map of the 3D model. The overall flow of the BSF-based color texture fusion algorithm is shown in Figure 3.

The above content is a further detailed description of the present invention in conjunction with specific/preferred embodiments, and it cannot be assumed that the specific implementation of the present invention is limited to these descriptions. For those of ordinary skill in the technical field to which the present invention belongs, without departing from the concept of the present invention, they can also make some substitutions or modifications to the described embodiments, and these substitutions or modifications should be regarded as Belong to the protection scope of the present invention.

Claims (10)

1. A realistic three-dimensional color texture reconstruction method, characterized in that, comprising: System parameters of pre-calibrated 3D sensor and color texture camera; Obtain multi-view 3D images and 2D color texture images; generating a three-dimensional mesh model by using the multi-view three-dimensional image; Establishing a mapping relationship between the two-dimensional color texture image and the three-dimensional mesh model under each viewing angle by the system parameters; Performing texture fusion based on the mapping relationship to obtain a fusion image to realize color texture reconstruction of the overall three-dimensional model; Generate a corresponding texture map according to the mapping relationship. 2. The realistic three-dimensional color texture reconstruction method according to claim 1, wherein the texture fusion is to evaluate the confidence of each texel by defining composite weights through depth data, and according to the confidence of each viewing angle Degrees are weighted average to calculate the fusion result. 3. The realistic three-dimensional color texture reconstruction method according to claim 2, wherein the composite weight is calculated by the following formula: f(x _k )＝f _norm (x _k )·f _depth (x _k )·f _edge (x _k ) Among them, the normal weight depth weight edge weight And each weight is normalized, and the value range is [0,1]. 4. The realistic three-dimensional color texture reconstruction method according to claim 3, wherein the values of the coefficients in the normal weight are: a=0.1, b=50°, and the coefficients in the depth weight are The values are: a=0.4, b=50mm, d ₀ =55mm, and the coefficients in the edge weight are: a=-0.08, b=50mm. 5. The realistic three-dimensional color texture reconstruction method according to claim 1, wherein a target image is introduced between the original two-dimensional color texture image and the fused image, and according to the In the case of displacement, the original two-dimensional color texture image is reconstructed and calculated using the bidirectional similarity function E _BDS (S, T) to generate an energy function, and the overall target image is displaced by minimizing the energy function to reduce the overall The image of the model texture fusion is blurred, and finally a new high-resolution target image is obtained. 6. The realistic three-dimensional color texture reconstruction method according to claim 5, wherein the energy function further comprises an optical measure consistency function E _C : Where M _i represents the fused image under the i-th viewing angle, x _k represents the pixel position of the image, P(·) represents a projection function, and N represents the number of viewing angles. _wj denotes the composite weight of the image at the jth viewing angle. 7. The realistic three-dimensional color texture reconstruction method according to claim 6, wherein the energy function E is finally constructed as: E＝E ₁ +λE ₂ where λ is the scaling factor between the _two energy functions _E1 and E2. 8. The realistic three-dimensional color texture reconstruction method according to claim 7, wherein the objective function is solved through a two-step alternate optimization strategy, that is, the following two steps are iteratively solved: S1: fixing the fused image M _i , optimizing the target image T _i ; S2: Fix the target image T _i and optimize the fused image M _i . 9. The realistic three-dimensional color texture reconstruction method according to claim 8, characterized in that, in the step S1, the target image Ti is expressed by the following formula: In the step S2, the objective function Mi is represented by the following formula: 10. The realistic three-dimensional color texture reconstruction method according to claim 9, characterized in that, a multi-scale optimization method is adopted in the iterative operation process, that is, in the low-scale stage, all images are down-sampled to a low resolution and performed The above iterative operation. After the energy function E converges, the target image and the fused image are up-sampled to a larger scale, while the original two-dimensional color texture image is still down-sampled, in order to reduce the high Injecting video information into the target image and the fused image.