CN107657585A - High magnification super-resolution method based on double transform domains - Google Patents

Abstract

The invention belongs to computer vision field, to improve the qualitatively and quantitatively result of depth map high magnification super-resolution on the basis of double transform domains.Therefore, the technical solution adopted by the present invention is, the depth map super-resolution method of the multilayer cascade based on double transform domains, step is：One powerful problem is become the cascade problem of multistage small multiplying power, for the small multiplying power super-resolution problem of every one-level, be modeled using formula (1)： $\begin{array}{c}\underset{x}{\arg \min }F\left(\mathrm{\α}\right)\\ =\underset{x}{\arg \min }{E}_D(x,y)+{\mathrm{\λ}}_1{E}_{TR}(x,z)+{\mathrm{\λ}}_2{E}_{SR}(x,z)\\ =\underset{x}{\arg \min }\left|\right|y-Hx|{|}_2^2+{\mathrm{\λ}}_1{\mathrm{\Σ}}_i||{\mathrm{\α}}_i-{\mathrm{\Σ}}_{q\∈{\mathrm{\Ω}}_i}{\mathrm{\ζ}}_{i,q}{\mathrm{\α}}_q|{|}_1+{\mathrm{\λ}}_2\underset{(m,l)\∈C}{\mathrm{\Σ}}\left|\right|P_{m,l}(x-S_1^l{S}_2^{m}x)|{|}_1\end{array}$ $P_{m,l}=\frac{1}{\mathrm{\Θ}}{P}_{m,l}^Q{P}_{m,l}^C{P}_{m,l}^D{P}_{m,l}^T---\left(1\right)$ Solution formula (1) obtains the result of the super-resolution of final small multiplying power.Present invention is mainly applied to computer vision to handle occasion.

Description

High-magnification super-resolution method based on double transformation domains

Technical Field

The invention belongs to the field of computer vision. The invention makes innovative improvement to the problem that the result of the latest level can be obtained for the super-resolution with small magnification based on the regularization terms of the frequency domain and the spatial domain, but the satisfactory result can not be obtained for the super-resolution with high magnification.

Background

Depth information has been widely used in the fields of virtual reality, augmented reality, and 3D reconstruction. Despite the rapid development of depth cameras in recent years, the resolution of depth maps obtained by current mainstream depth sensors (such as Kinect and TOF) is still much lower than that of color maps. Therefore, the effective depth map super-resolution method has practical significance. At present, a plurality of excellent methods obtain better results on the problem of small-magnification super-resolution of the depth map, but the problem of high-magnification super-resolution is still troublesome.

The super-resolution problem is usually changed into a proper problem by introducing a regular term, and the existing super-resolution methods can be divided into three categories according to the difference of the regular term: a space domain regular term method, a transform domain regular term method and a double transform domain regular term method.

The central idea behind the transform domain class approach is to design a dictionary that can provide efficient representation while being suitable for a particular application. The main advantage of this type of approach is that the extracted depth features can be compactly represented by a small number of basis functions or dictionary atoms. High-order associations between chunks may be efficiently utilized. A disadvantage of this type of approach is the lack of pixel-level adaptability.

The spatial domain method can make up the problem of the lack of adaptability of the transform domain method, and the method mainly utilizes the strong connection of color depth image pairs. Among them, total variation (total variation) reconstruction models are widely used because they can describe sparse gradients well.

The dual transform domain class approach combines the advantages of both the transform domain class and the spatial domain class approaches. Much recent work has shown that the use of the dual transform domain approach is a trend. However, although the dual transform domain method can obtain excellent results on the small-magnification super-resolution problem, the high-magnification reconstruction result is still unsatisfactory. The invention provides a method for improving the super-resolution performance of high magnification by utilizing a multi-level cascade method on the basis of double transform domains.

Disclosure of Invention

In order to overcome the defects of the prior art, the invention aims to improve the quantitative and qualitative results of the high-magnification super-resolution of the depth map on the basis of the double transform domains. Therefore, the technical scheme adopted by the invention is that the double transform domain-based multi-layer cascaded depth map super-resolution method comprises the following steps:

1) Changing a high-magnification problem into a multi-level small-magnification cascade problem, and modeling the small-magnification super-resolution problem of each level by using an equation (1):

wherein, let F (alpha) = E _D (x,y)+λ ₁ E _TR (x,z)+λ ₂ E _SR (x,z),x∈R ⁿ ，y∈R ^m ,z∈R ⁿ Vectorized recovered depth map, vectorized low resolution depth map and vectorized high resolution color map, E, respectively _D (x, y) is a data item, E _TR (x, z) is a transform domain regularization term, E _SR (x, z) is a spatial regularization term, λ ₁ And λ ₂ Is the weight parameter for balancing each item, min represents the minimum value,represents the value of the variable x when the following equation is minimized; h denotes the synthesis operator of the blur and sample, α _i Representing a block x _i On-site dictionaryThe coefficients of (a) and (b) are,representing a block x _i Dictionary, x, corresponding to the class of block in which it is located _i Is an image block with the center located at i in the depth map x; alpha is alpha _q Is a block x _i Q-th similar block x of (1) _i,q Corresponding sparse coefficient, q belongs to a set omega composed of similar blocks _i ，ζ _i,q Are the respective weighted weights; nucleus P _m,l The four contained terms are spatial terms respectivelyColour itemDepth termAnd the structure tensor term Andrepresenting shifting the image x by l and m pixels in the horizontal and vertical directions, respectively, C being a selected set of (m, l), in different (m, l) combinationsThe total variation operator representing different directions, e, represents belonging. I | · | live through ₂ Is a ₂ Norm, | · | luminance ₁ Is a ₁ Norm, Σ, represents a summation operator; Θ is the normalization factor; item of structure tensorComprises the following steps:

r _m,l (u)＝ψ{|cos(v _m,l ,v ₁ (u))|},(m,l)∈C (2)

where max (-) is a function of the maximum value,is an exponential function with the base of a constant e, ψ {. Is a sorting function in descending order, and returns a sorted numerical result r _m,l (u), cos (. Cndot.) is a cosine function, v _m,l Is the total variation operatorDirection of (v) ₁ (u) the first bit of the structure tensor at pixel uThe eigenvector, | · | represents an absolute value. a, b and c are constants;

and solving the formula (1) to obtain a final small-magnification super-resolution result.

The specific construction method of the super-resolution method of multi-layer cascade, which changes the problem of high magnification into the problem of cascade of multi-level and small magnification, is as follows: constructing an L-layer cascade network, wherein the sampling multiplying power of each layer is 2, namely, each layer only solves the problem of 2 times of up-sampling, and the output of the k-1 layer is set asFirstly, 2 times of upsampling is carried out on the bicubicThis up-sampled depth map is used as input for the k-th layer, i.e.For super-resolution magnifications that cannot be expressed as powers of 2, up-sampling is performed at each layer using fractional magnifications.

The concrete solving steps for the formula (1) are as follows:

31 The equation (1) is transformed using the weighted least squares algorithm IRLS:

whereinW _m,l Is a diagonal element ofA diagonal matrix of (a);

matrix ofWhere I is the unit array. Matrix ofWhere 0 is a vector with all zero elements, denotes all alpha _i Is connected in series with the dictionaryA multiplication and addition operation of the series phi is specificallyR _i Representing the extraction of a block x from a graph x _i The linear operator of (a) is determined,representing a block x _i Dictionary corresponding to the class of block in which it is located, α _i Representing a block x _i In a dictionaryA coefficient of M representing the number of blocks extracted from the depth map, (. C) ^-1 Is an inversion operation, (.) ^T Is a transposition operation in which the first and second operations are performed,is an open root number operation;

32 Solving the optimization equation:

the main idea of the PG algorithm is to make one function Q (α, ρ) iteratively approach the original problem F (α) near the point ρ, and then minimize the original problem F (α) is replaced by minimize Q (α, ρ), defining functions h (α) and g (α) as follows:

g(α)＝∑ _i ||α _i -β _i || ₁ (4)

then the function Q (α, ρ) is defined according to the PG algorithm as follows:

whereinDenotes the gradient of h (-), L _f Is a constant greater than the spectral norm of the hessian matrix of h (alpha),<·,·&gt represents an inner product operator, and the coefficient alpha is obtained by the (l + 1) th iteration ^(l+1) The updating is as follows:

wherein

ρ ^(l) Is ρ, Φ at the l-th iteration ^T Denotes the transposition of phi, B ^T Which represents the transpose of B and,a multiply-add operation after the representation is concatenated is finally obtained

Where soft (-) is a soft threshold function,d ₀ 1, beta is all beta _i A series connection of ^(l) Is alpha at the time of the l iteration, and updated alpha is obtained ^(l+1) Then, the first +1 iteration is obtainedUpdated depth map x ^(l+1) ：

The invention has the technical characteristics and effects that:

the method is a troublesome problem aiming at the high-magnification super-resolution of the depth camera, and recovers the depth map after high-quality high-magnification super-resolution by introducing a weight item capable of reflecting a space geometric structure and applying a multi-layer cascade structure. The invention has the following characteristics:

1. and a structure tensor is introduced into a space domain regular term, so that the gradient characteristic of rapid change in the depth map is better described.

2. A multi-layer cascade method is provided to solve the problem of high-magnification super-resolution.

3. The sub-problems are solved by using algorithms such as PG, IRLS and the like, and the advantages of the existing algorithms are integrated.

Drawings

Fig. 1 is a structural block diagram corresponding to equation (6), namely a structural block diagram corresponding to handling a small-magnification super-resolution (or per-level super-resolution) problem.

FIG. 2 is the inputs and outputs of the present invention: the input is (a) a low resolution depth map, (b) a high resolution color map, and the output is (c) a high resolution depth map.

Fig. 3 is a schematic diagram of a super-resolution method of multi-layer cascade.

Fig. 4 is a comparison of the results of the respective methods for 16 times super resolution. (first, second row is the correlation results for reinneer, left to right graphs are (a) bicubic interpolation, RMSE:8.78 (b) GF, RMSE:6.40 (c) SDF, RMSE:5.55 (d) AE, RMSE:6.42 (e) TGV, RMSE:8.99 (f) NCSR, RMSE:5.20 (g) JID, RMSE:5.13 (h) our results without cascade, RMSE:3.81 (i) our results with cascade, RMSE:3.12 (j) GT.

Detailed Description

1) Changing a high-magnification problem into a multi-level small-magnification cascade problem, and modeling the small-magnification super-resolution problem of each level by using an equation (1):

wherein let F (alpha) = E _D (x,y)+λ ₁ E _TR (x,z)+λ ₂ E _SR (x,z),x∈R ⁿ ，y∈R ^m ,z∈R ⁿ Respectively vectorized recovered depth map, vectorized low resolution depth map and vectorized high resolution color map, E _D (x, y) is a data item, E _TR (x, z) is a transform domain regularization term, E _SR (x, z) is a spatial regularization term, λ ₁ And λ ₂ Is the weight parameter for balancing each item, min represents the minimum value,represents the value of the variable x when the following equation is minimized; h denotes the synthesis operator of the blur and sample, α _i Representing a block x _i In a dictionaryThe coefficients of (a) and (b) are,representing a block x _i Dictionary, x, corresponding to the class of block in which it is located _i Is an image block with the center located at i in the depth map x; alpha is alpha _q Is a block x _i Q-th similar block x of _i,q Corresponding sparse coefficient, q belongs to a set omega composed of similar blocks _i ，ζ _i,q Are the corresponding weighted weights; nucleus P _m,l The four contained terms are spatial terms respectivelyColour itemDepth termAnd the structure tensor termWherein the structural tensor terms are the weight terms proposed by the invention;andrepresenting shifting the image x by l and m pixels in the horizontal and vertical directions, respectively, C being a selected set of (m, l), in different (m, l) combinationsThe total variation operator representing different directions, e, represents belonging. I | · | purple wind ₂ Is a 1 ₂ Norm, | · | luminance ₁ Is a ₁ Norm, sigma, represents the sum operator; Θ is the normalization factor;

since the ordinary spatial distance point by point does not reflect the image structure. A better method for reflecting the spatial structure is to distinguish the total variation operators in different directions, namely, different weights are given to different operators according to the distances between the tangential direction of the current pixel and the direction of the variation operator. Therefore, the present invention is based on spatial weighting P _m,l Further introduces a structural tension termItem of structure tensorComprises the following steps:

r _m,l (u)＝ψ{|cos(v _m,l ,v ₁ (u))|},(m,l)∈C (2)

where max (-) is a function of the maximum value,is an exponential function with the base of a constant e, ψ {. Is a sorting function in descending order, and returns a sorted numerical result r _m,l (u), cos (. Cndot.) is a cosine function, v _m,l Is the total variation operatorDirection of (v) ₁ (u) represents the first eigenvector of the structure tensor at pixel u, | · | represents the absolute value. a, b and c are constants;

and solving the formula (1) to obtain a final small-magnification super-resolution result.

2) Step 1) of the present invention describes the method for solving the small-magnification super-resolution under the double transform domain, and although the method has excellent performance on the small-magnification problem, the reconstruction result of the high-magnification is still not satisfactory. Since for high-power super-resolution, e.g. 8,16 times, it is very difficult to get details of the high-resolution map directly from the low-resolution depth map, the main problem is that the original high-resolution map from the low-resolution map is too smooth and contains little detail information. In order to solve the problem, the invention provides a multi-layer cascaded super-resolution method, which is qualitatively understood that the problem of high magnification is changed into the problem of multi-stage small magnification cascade, and the small magnification super-resolution problem of each stage can be modeled by using an equation (1).

The specific construction method of the super-resolution method of multi-layer cascade, which changes the problem of high magnification into the problem of cascade of multi-level and small magnification, is as follows: constructing a cascade network of L layers, wherein the sampling multiplying power of each layer is 2, namely each layer only solvesTo solve the problem of 2 times up-sampling, the output of the k-1 layer is set asFirstly, the conventional bicubic method is adopted to carry out 2 times of upsampling on the bicubicThis up-sampled depth map is used as input for the k-th layer, i.e.For super-resolution magnifications that cannot be expressed as powers of 2, up-sampling can be performed at each layer using fractional magnifications.

3) We solve equation (1) using the near-end gradient (PG) algorithm. For the super-resolution problem with small magnification, the result obtained by directly solving the formula (1) is the super-resolution result, and for the super-resolution problem with high magnification, the method of the step 2) is adopted to convert the problem into a multi-stage super-resolution problem with small magnification, and each stage needs to solve the formula (1).

The concrete solving steps for the formula (1) are as follows:

31 The original problem (equation (1)) is transformed using the weighted least squares algorithm IRLS:

whereinW _m,l Is a diagonal element ofThe diagonal matrix of (a).

Matrix arrayWhere I is the unit array. Matrix arrayWhere 0 is a vector with all zero elements, denotes all alpha _i Is connected in series with the dictionaryA multiplication and addition operation of the series phi is specificallyR _i Representing the extraction of a block x from a graph x _i The linear operator of (a) is determined,representing a block x _i Dictionary corresponding to the class of block in which it is located, α _i Representing a block x _i On-site dictionaryA coefficient, M represents the number of blocks extracted from the depth map, (-) ¹ Is an inversion operation (·) ^T Is a transposition operation in which the phase of the input signal is inverted,is an open root number operation;

32 Solving the optimization equation:

the main idea of the PG algorithm is to make one function Q (α, ρ) iteratively approach the original problem F (α) near the point ρ, and then minimizing the original problem F (α) can be replaced by minimizing Q (α, ρ), defining functions h (α) and g (α) as follows:

g(α)＝∑ _i ||α _i -β _i || ₁ (4)

then the function Q (α, ρ) can be defined according to the PG algorithm as follows:

whereinDenotes the gradient of h (-), L _f Is a constant greater than the spectral norm of the hessian matrix of h (alpha),<·,·&gt, representing inner product operator, coefficient alpha obtained by the (l + 1) th iteration ^(l+1) The updating is as follows:

wherein

ρ ^(l) Is ρ, Φ at the l-th iteration ^T Denotes the transposition of phi, B ^T The transpose of B is represented by,a multiply-add operation after the representation is concatenated is finally obtained

Where soft (-) is a soft threshold function,d ₀ =1, β is all β _i A series connection of ^(l) Is alpha at the time of the l iteration, and updated alpha is obtained ^(l+1) Then, the l +1 th iteration update can be further obtainedDepth map x of ^(l+1) ：

The invention is described in detail below with reference to the figures and examples.

1) A high-magnification problem is changed into a multi-level small-magnification cascade problem, modeling is carried out on the small-magnification super-resolution problem of each level by using an equation (1), and a corresponding structural block diagram is shown in figure 1:

wherein let F (alpha) = E _D (x,y)+λ ₁ E _TR (x,z)+λ ₂ E _SR (x,z),x∈R ⁿ ，y∈R ^m ,z∈R ⁿ The vectorised restored depth map (as in fig. 2 (c)), the vectorised low resolution depth map (fig. 2 (a)) and the vectorised high resolution color map (fig. 2 (b)), E respectively _D (x, y) are data items, E _TR (x, z) is a transform domain regularization term, E _SR (x, z) is a spatial regularization term, λ ₁ And λ ₂ Is the weight parameter for balancing each item, min represents the minimum value,represents the value of the variable x when the following equation is minimized; h denotes the synthesis operator of the blur and sample, alpha _i Representing a block x _i In a dictionaryThe coefficients of (a) and (b) are,representing a block x _i Dictionary, x, corresponding to the class of block in which it is located _i Is an image block centered at i in the depth map x; alpha is alpha _q Is a block x _i Q-th similar block x of (1) _i,q Corresponding sparse coefficient, q belongs to a set omega composed of similar blocks _i ，ζ _i,q Are the corresponding weighted weights; nucleus P _m,l The four contained terms are spatial terms respectivelyColour itemDepth termAnd the structure tensor termWherein, the structural tensor term is a weight term provided by the invention;andrepresenting the movement of the image x by l and m pixels in the horizontal and vertical directions, respectively, C being a selected set of (m, l), different (m, l) combinationsRepresents the total variation operator in different directions, and epsilon represents belonging. I | · | purple wind ₂ Is a ₂ Norm, | | · | non-conducting phosphor ₁ Is a ₁ Norm, Σ, represents a summation operator; Θ is the normalization factor;

since the ordinary spatial distance point by point does not reflect the image structure. A better method for reflecting the spatial domain structure is to distinguish the total variation operators to be treated in different directions, namely, different weights are given to different operators according to the distance between the tangential direction of the current pixel and the direction of the variation operator. Therefore, the present invention is based on spatial weighting P _m,l Further introduces a structural tension termItem of structure tensorComprises the following steps:

r _m,l (u)＝ψ{|cos(v _m,l ,v ₁ (u))|},(m,l)∈C (2)

where max (-) is a function of the maximum value,is an exponential function with the base of a constant e, ψ {. Is a sorting function in descending order, and returns a sorted numerical result r _m,l (u), cos (. Cndot.) is a cosine function, v _m,l Is total variation operatorDirection of (v) ₁ (u) represents the first eigenvector of the structure tensor at pixel u, | · | indicates the absolute value. a, b and c are constants;

and solving the formula (1) to obtain a final small-magnification super-resolution result.

2) Step 1) of the present invention describes the method for solving the small-magnification super-resolution under the double-transform domain, and although this method has an excellent performance on the small-magnification problem, the reconstruction result of the high-magnification still cannot be satisfied. Since for high-power super-resolution, e.g. 8,16 times, it is very difficult to get details of the high-resolution map directly from the low-resolution depth map, the main problem is that the original high-resolution map from the low-resolution map is too smooth and contains little detail information. In order to solve the problem, the invention provides a multi-layer cascade super-resolution method (fig. 3), which qualitatively understands that a high-magnification problem is changed into a multi-stage small-magnification cascade problem, and the small-magnification super-resolution problem of each stage can be modeled by using an equation (1).

The specific construction method of the super-resolution method of multi-layer cascade, which changes the problem of high magnification into the problem of cascade of multi-level and small magnification, is as follows: constructing a cascade network of L layers, wherein the multiplying power of the up-sampling of each layer is 2, namely, each layer only solves the problem of up-sampling of 2 times, and the output of the k-1 layer is set asFirstly, the conventional bicubic method is adopted to carry out 2 times of upsampling on the bicubicThis up-sampled depth map is used as input for the k-th layer, i.e.For super-resolution magnification that cannot be expressed as a power of 2, up-sampling can be performed using fractional magnification at each layer.

Because each layer of the multilayer cascade method only needs to process a 2-time small-magnification up-sampling problem, each layer can obtain a relatively ideal reconstruction result. As shown in fig. 4 and corresponding RMSE values, it is shown that the performance of high-magnification super-resolution can be effectively improved by using the multilayer cascade method.

3) We solve equation (1) using the near-end gradient (PG) algorithm. For the super-resolution problem with small magnification, the result obtained by directly solving the formula (1) is the result of the super-resolution, and for the super-resolution problem with high magnification, the method in the step 2) is adopted to convert the problem into a multi-stage super-resolution problem with small magnification, and the formula (1) needs to be solved for each stage.

The concrete solving steps for the formula (1) are as follows:

31 The original problem (equation (1)) is transformed using the reweighted least squares algorithm IRLS algorithm:

whereinW _m,l Is a diagonal element ofThe diagonal matrix of (a).

Matrix ofWhere I is the unit array. Matrix arrayWhere 0 is a vector with all zero elements, denotes all alpha _i Is connected in series with the dictionaryA multiplication and addition operation of the series phi is specificallyR _i Representing the extraction of a block x from a graph x _i The linear operator of (a) is determined,representing a block x _i Dictionary corresponding to the class of block in which it is located, α _i Representing a block x _i On-site dictionaryThe coefficient of the following, M represents the number of blocks extracted from the depth map, (. Cndot.) ^-1 Is an inversion operation (·) ^T Is a transposition operation in which the phase of the input signal is inverted,is an open root number operation;

32 Solving the optimization equation:

the main idea of the PG algorithm is to make one function Q (α, ρ) iteratively approach the original problem F (α) near the point ρ, and then minimizing the original problem F (α) can be replaced by minimizing Q (α, ρ), defining functions h (α) and g (α) as follows:

g(α)＝∑ _i ||α _i -β _i || ₁ (4)

then the function Q (α, ρ) can be defined according to the PG algorithm as follows:

whereinDenotes the gradient of h (-), L _f Is a constant greater than the spectral norm of the hessian matrix of h (alpha),<·,·&gt, representing inner product operator, coefficient alpha obtained by the (l + 1) th iteration ^(l+1) The updating is as follows:

wherein

ρ ^(l) Is ρ, Φ at the l-th iteration ^T Denotes the transposition of phi, B ^T The transpose of B is represented by,representing stringsA subsequent multiply-add operation is performed to obtain the final result

Where soft (-) is a soft threshold function,d ₀ =1, β is all β _i A series connection of ^(l) Is alpha at the time of the l iteration, and updated alpha is obtained ^(l+1) Then, the depth map x updated by the (l + 1) th iteration can be obtained ^(l+1) ：

Claims (3)

1. A multilayer cascade depth map super-resolution method based on double transform domains is characterized by comprising the following steps: changing a high-magnification problem into a multi-level small-magnification cascade problem, and modeling the small-magnification super-resolution problem of each level by using an equation (1):

wherein, let F (alpha) = E _D (x,y)+λ ₁ E _TR (x,z)+λ ₂ E _SR (x,z),x∈R ⁿ ，y∈R ^m ,z∈R ⁿ Vectorized recovered depth map, vectorized low resolution depth map and vectorized high resolution color map, E, respectively _D (x, y) are data items, E _TR (x, z) is a transform domain regularization term, E _SR (x, z) is a spatial regularization term，λ ₁ And λ ₂ Is the weight parameter for balancing each item, min represents the minimum value,represents the value of the variable x when the following equation is minimized; h denotes the synthesis operator of the blur and sample, alpha _i Representing a block x _i In a dictionaryThe coefficients of (a) and (b) are,representing a block x _i Dictionary, x, corresponding to the class of block in which it is located _i Is an image block centered at i in the depth map x; alpha is alpha _q Is a block x _i Q-th similar block x of _i,q Corresponding sparse coefficient, q belongs to a set omega composed of similar blocks _i ，ζ _i,q Are the corresponding weighted weights; nucleus P _m,l The four contained terms are respectively spatial domain termsColour itemDepth termAnd the structure tensor term Andrepresenting shifting the image x by l and m pixels in the horizontal and vertical directions, respectively, C being a selected set of (m, l) s, under different (m, l) combinationsIs/are as followsRepresents the total variation operator in different directions, and epsilon represents belonging. I | · | purple wind ₂ Is a 1 ₂ Norm, | · | luminance ₁ Is a ₁ Norm, Σ, represents a summation operator; Θ is the normalization factor; item of structure tensorComprises the following steps:

r _m,l (u)＝ψ{|cos(v _m,l ,v ₁ (u))|},(m,l)∈C (2)

where max (-) is a function of the maximum value,is an exponential function based on a constant e, ψ {. Is a sorting function sorted in descending order, and returns a sorted numerical result r _m,l (u), cos (. Cndot.) is a cosine function, v _m,l Is total variation operatorDirection of (v) ₁ (u) represents the first eigenvector of the structure tensor at pixel u, |, represents the absolute value. a, b and c are constants;

and solving the formula (1) to obtain a final small-magnification super-resolution result.

2. The method for super-resolution of depth maps based on multi-layer cascade of double transform domains as claimed in claim 1, wherein a high-magnification problem is changed into a multi-level small-magnification cascade problem, namely a super-resolution method of multi-layer cascade, and the specific construction method is as follows: constructing an L-layer cascade network, wherein the sampling multiplying power of each layer is 2, namely, each layer only solves the problem of 2 times of upsampling, and the first layer is providedThe output of the k-1 layer isFirstly, 2 times of upsampling is carried out on the bicubicThis up-sampled depth map is used as input for the k-th layer, i.e.For super-resolution magnification that cannot be expressed as a power of 2, upsampling is performed using fractional magnification at each layer.

3. The method for super-resolution of the depth map based on multi-layer cascade of the double transform domains as claimed in claim 1, wherein the specific solving step for equation (6) is: the concrete solving steps of the formula (1) are as follows:

31 ) the formula (1) is transformed using the weighted least squares algorithm IRLS:

whereinW _m,l Is a diagonal element ofA diagonal matrix of (a);

matrix arrayWhere I is the unit array. Matrix arrayWhere 0 is a vector with all zero elements, denotes all alpha _i Is connected in series with the dictionaryA multiplication and addition operation of the series phi is specificallyR _i Representing the extraction of a block x from a graph x _i The linear operator of (a) is used,representing a block x _i Dictionary corresponding to the class of block in which it is located, alpha _i Representing a block x _i In a dictionaryA coefficient of M representing the number of blocks extracted from the depth map, (. C) ^-1 Is an inversion operation, (.) ^T Is a transposition operation in which the phase of the input signal is inverted,is an open root number operation;

32 Solving an optimization equation: the main idea of the PG algorithm is to make one function Q (α, ρ) iteratively approach the original problem F (α) near the point ρ, and then the minimized original problem F (α) is replaced by the minimized Q (α, ρ), defining functions h (α) and g (α) as follows:

g(α)＝∑ _i ||α _i -β _i || ₁ (4)

then the function Q (α, ρ) is defined according to the PG algorithm as follows:

whereinDenotes the gradient of h (-), L _f Is a constant greater than the spectral norm of the hessian matrix of h (alpha),<·,·&gt, representing inner product operator, coefficient alpha obtained by the (l + 1) th iteration ^(l+1) The updating is as follows:

wherein:

ρ ^(l) is ρ, Φ at the l-th iteration ^T Denotes the transposition of phi, B ^T Which represents the transpose of B and,a multiply-add operation after the representation is concatenated is finally obtained

Where soft (-) is a soft threshold function,d ₀ =1, β is all β _i Series connection of (a) ^(l) Is alpha at the time of the l iteration, and updated alpha is obtained ^(l+1) Then, a depth map x updated by the (l + 1) th iteration is further obtained ^(l+1) ：