CN112884645B - Tensor sparse constraint-based light field filling method and device - Google Patents
Tensor sparse constraint-based light field filling method and device Download PDFInfo
- Publication number
- CN112884645B CN112884645B CN202110063571.XA CN202110063571A CN112884645B CN 112884645 B CN112884645 B CN 112884645B CN 202110063571 A CN202110063571 A CN 202110063571A CN 112884645 B CN112884645 B CN 112884645B
- Authority
- CN
- China
- Prior art keywords
- light field
- tensor
- blocks
- resolution
- original
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000005070 sampling Methods 0.000 claims abstract description 55
- 239000011159 matrix material Substances 0.000 claims description 20
- 238000000354 decomposition reaction Methods 0.000 claims description 19
- 230000003321 amplification Effects 0.000 claims description 4
- 230000000295 complement effect Effects 0.000 claims description 4
- 238000003199 nucleic acid amplification method Methods 0.000 claims description 4
- 230000009191 jumping Effects 0.000 claims description 3
- 230000003287 optical effect Effects 0.000 claims description 3
- 238000012163 sequencing technique Methods 0.000 claims description 2
- 230000000007 visual effect Effects 0.000 abstract description 4
- 238000005516 engineering process Methods 0.000 abstract description 2
- 239000000203 mixture Substances 0.000 description 5
- 238000002474 experimental method Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000000513 principal component analysis Methods 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T3/4076—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution using the original low-resolution images to iteratively correct the high-resolution images
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The tensor sparse constraint-based light field filling method and device avoid vectorization and introduction of super-resolution reconstruction technology, so that the correlation characteristics in the light field are utilized to the greatest extent, and the filled light field with good objective performance is obtained, thereby giving a user better visual experience. The method comprises the following steps: (I) Sliding window sampling is carried out on the light field losing part of information, similar blocks are searched and stacked, a five-dimensional tensor obtained by filling is filled by a light field filling model based on tensor sparse constraint, and then image blocks in the five-dimensional tensor are respectively placed back to the original index of the light field, so that a filled light field is obtained; (II) sliding window sampling is carried out on the light field obtained in the step (I) and the high-resolution light field obtained by carrying out super-resolution reconstruction on the light field and the high-resolution light field; and projecting the image blocks obtained by the latter into blocks with the same size, sorting the similarity to obtain five-dimensional tensors consisting of the two blocks, filling the five-dimensional tensors by using a tensor sparse constraint-based light field filling model, and placing the image blocks with the original sizes back to the original indexes to obtain a filled light field.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a light field filling method based on tensor sparse constraint and a light field filling device based on tensor sparse constraint, which are mainly used for filling high-dimensional images.
Background
The light field contains both spatial and angular information, which can more accurately represent the light field. In the problems of detection, identification, computer image and the like, the light field occupies very important positions, has very wide application, and can be applied to various fields of edge detection, material identification, image refocusing, virtual reality and the like. Image data acquired by hardware requires a higher dimension to more accurately represent the scene, while a high-dimensional image is desired as opposed to taking a series of two-dimensional images directly of the scene. Due to hardware limitations, it is still very difficult to obtain a high resolution light field. And due to the high sensitivity of the sensor and the high processing speed that is required by people, the acquired light field is usually low resolution and even loss of part of the data occurs, which limits the development of subsequent processing techniques. Therefore, the filling of the light field has become an important research direction.
The most straightforward approach to light field filling is to treat each view (view) of the light field as an independent two-dimensional image, and then fill each image with a matrix filling model, which mainly has matrix singular value decomposition (singular value decomposition, SVD) and matrix kernel norm minimization (nuclear norm minimization, NNM). While these algorithms are the simplest method of restoring a light field, they treat the light field as a stack of a series of two-dimensional images, breaking the inter-relationship in each view of the light field image.
The tensor can better represent the relation inside the light field, so that researchers can also use the tensor to represent the light field, and the filling problem of the light field is the filling problem of the tensor, and the filling algorithm of the tensor can be applied to the light field filling problem. Such filling methods can be divided into two classes, one class is a tensor rank-based minimization algorithm and one class is a tensor decomposition-based algorithm. The former is populated with a minimum tensor rank, and typical algorithms are HaLRTC [1], [2] and [3]. This method can be regarded as a generalized form of a two-dimensional matrix low-rank filling method, which takes advantage of the characteristics of ranks and performs filling by minimizing ranks. However, unlike two-dimensional matrices, the tensor rank is not well defined, and minimizing the tensor rank directly is also an NP-hard problem. The partial tensor fill method also solves this problem with a low rank tensor that approximates the original tensor. HaLRTC [1] proposes the definition of tensor trace norms, which is a generalized matrix trace norm, and finally solved by the ADMM framework. HaLRTC [2] proposes that several theoretical boundary measures (theoretical bounds) of the random Gaussian measure (random Gaussian measurements) handle the problem of recovery of low rank tensors. HaLRTC [3] uses the order n of tensors as sparseness, and solves the tensor filling problem by solving the lowest n-order tensor which satisfies the linear constraint. Tensor filling methods based on low ranks typically require a large amount of computation and cost quite a lot of resources. The latter tensor population is based on tensor decomposition, typically using CP (CANDECOMP/PARAFAC) decomposition or turner decomposition. Among tensor-fill algorithms based on CP decomposition, the most representative algorithms are CPWOPT, FBCP and NCCP. CPWOPT is a weighted optimization algorithm for tensor population. FBCP automatically find the proper tensor rank by using a hierarchical probability model through full Bayesian processing, and fill the tensor under the constraint of the rank. NCCP is a tensor-filling algorithm based on CP decomposition that uses factor matrix kernel norm minimization for filling. Representative tensor fill algorithms based on Tucker decomposition are Tucker [1] and STDC. Tucker [1] creates an optimization problem and solves with a nonlinear conjugate gradient (Nonlinear Conjugated Gradient). STDC combining rank minimization with a Tucker decomposition model to construct a new model and solve the filling problem. While these methods can all fill the light field, they default that all modes of the light field share the same weight, ignoring the distinction of tensors along the modes.
There has also been proposed a multispectral image filling method [9] based on Kronecker-basic-presentation (KBR), which fills multispectral images by utilizing the characteristics of both CP decomposition and Tucker decomposition. However, the light field image is a high-dimensional data comprising an angular dimension and a spatial dimension, and if a tensor sparse filling method (KBR) based on the Kernel representation is applied to the light field, the angular dimension of the light field is vectorized, so that the characteristics inside the light field are broken.
Disclosure of Invention
In order to overcome the defects of the prior art, the technical problem to be solved by the invention is to provide a light field filling method based on tensor sparse constraint, which can avoid vectorization of any dimension in the reconstruction process, furthest save the characteristics in the light field, and the filled light field has good objectively expression and better visual experience for users.
The technical scheme of the invention is as follows: the light field filling method based on tensor sparse constraint comprises the following steps:
(I) For a light field losing part of information, firstly, sliding window sampling is carried out on the light field to obtain a series of tensor image blocks, searching is carried out on the tensor image blocks, the searched similar blocks are stacked to a fifth dimension, the five-dimension tensor is filled by using a light field filling model based on tensor sparse constraint, and tensor image blocks in the filled five-dimension tensor are respectively placed back to an original index of a light field to obtain a filled light field image;
(II) carrying out super-resolution reconstruction on the filled light field image obtained in the step (I) to obtain a high-resolution light field; sliding window sampling is carried out on the light field with the original resolution, and sliding window sampling is carried out on the light field with the high resolution; searching similar blocks from the light field with original resolution, searching similar blocks from tensor blocks with corresponding integer times of the size in the high-resolution light field, projecting the latter back to the original size, and then carrying out similarity sorting on the two to obtain a five-dimensional tensor consisting of two groups of data, filling the five-dimensional tensor by using a light field filling model based on tensor sparse constraint, and respectively placing image blocks with the original size in the five-dimensional tensor back to the original index to obtain an information-completed light field;
the light field filling model based on tensor sparse constraint is as follows:
On the upper part Satisfy Tucker decomposition
Wherein,For the sought, filled light field,/>For the light field of information absence,/>For/>A high resolution light field reconstructed with super resolution; /(I)Is a five-dimensional tensor obtained by stacking similar blocks,/>For/>Tensor after filling,/>For/>Five-dimensional tensor obtained by stacking similar blocks searched in (1)/>Is/>Is a projection result of (2); /(I)Is/>Is X (j) isAn expansion matrix along the modulus j; u j is/>A factor matrix along each mode; rank (X (j)) is the rank of X (j); gamma is a regularization parameter and,
T is a compromise parameter, α and b are fidelity parameters, and Ω is a known element subscript.
The method comprises the steps of firstly carrying out sliding window sampling on a light field losing part of information to obtain a series of tensor blocks, carrying out searching on similar blocks of all tensor blocks, stacking the searched similar blocks to a fifth dimension to obtain a five-dimensional tensor, filling the five-dimensional tensor of the missing information by using a light field filling model based on tensor sparse constraint, and placing filled tensor image blocks in the five-dimensional tensor back to an original index to obtain a filled light field image; and performing super-resolution reconstruction on the filled light field to obtain a high-resolution light field, performing sliding window sampling, similar block searching and stacking on the light field with the original resolution, and performing sliding window sampling, similar block searching, stacking and projection on the light field with the high resolution to obtain the block size of the sliding window sampling of the light field with the original resolution. And performing similarity sorting on the two groups of similar blocks obtained by the first two groups of similar blocks to obtain a five-dimensional tensor, wherein the obtained similar blocks are the mixture of the two groups of data. The method is used for filling the light field filling model based on tensor sparse constraint, and the filled image blocks in the five-dimensional tensor are put back to the original index to obtain an information-complement light field, so that vectorization of any dimension can be avoided in the reconstruction process, the characteristics in the light field are stored to the greatest extent, the light field filled by the method has good performance in objective aspect, and better visual experience is provided for users.
Still another method for filling a light field based on tensor sparsity constraint is provided, the method comprising the steps of:
(1) Inputting a light field image in a data set, randomly selecting pixels with fixed sampling rate as known elements, taking the rest elements as unknowns, and inputting the unknown elements into the step (2);
(2) For the input light field with missing information, sliding window sampling is carried out on views with all angle dimensions according to the input block size and step length to obtain a series of tensor blocks;
(3) Searching similar blocks of all tensor blocks, calculating the similarity between all pixels of each tensor block by taking Euclidean distance as a measure, sorting the searched similar blocks by distance, selecting the first n similar blocks, stacking the searched similar blocks on a fifth dimension to obtain a five-dimensional tensor, and recording the index of the space dimension of all similar blocks in an original light field;
(4) Filling the five-dimensional tensor with a sparse tensor model, and respectively calculating The value of the constant variable is used for calculating a five-dimensional tensor after filling;
(5) Setting the tensor image blocks in the five-dimensional tensor back to the original light field index according to the indexes in the steps (2) and (3); for one tensor block to be put back to the original index, n tensor image blocks similar to the tensor image block are provided, all similar blocks are averaged to be used as a result, and the tensor image blocks are put back to the original index; and (3) taking the result of the step (5) as the missing light field input in the step (2), jumping to the step (2), and carrying out a certain number of iterations. Inputting the iterative result into the step (6);
(6) Sliding window sampling is carried out on the input filled light field to obtain a series of tensors, and indexes of pixels at the upper left corner of all tensor space dimensions in the space dimensions of the original light field are recorded;
(7) Performing super-resolution reconstruction on the filled light field input in the step (6), wherein the magnification is 2 times, and interpolating between every two pixels to obtain a high-resolution light field;
(8) Sliding window sampling is carried out on the light field after super-resolution reconstruction to obtain a series of tensor blocks (the block size in the block size step (1) is multiplied by the amplification factor), and similar blocks are searched in the set;
(9) Searching similar blocks for a series of tensors obtained in the step (6) and obtained in the step (8), wherein the tensors are projected to be the same as the tensors in the former, and selecting similar blocks with the front N of similarity; the similar blocks obtained at this time include: the similar blocks of the filled light field obtained in the step (6) and the similar blocks of the projection after the super-resolution reconstruction of the light field in the step (6);
(10) Filling the five-dimensional tensor obtained in the step (9) by using a sparse tensor model to obtain a filled five-dimensional tensor;
(11) The tensor image blocks in the five-dimensional tensor are placed back to the original light field index according to the index; for Zhang Liangkuai needing to be placed back to the original index in the five-dimensional tensor, N tensor image blocks similar to the tensor image blocks are provided, at the moment, the average value of all similar blocks is taken, and the tensor image blocks are placed back to the original index of the light field; taking the result obtained in the step as input in the step (6), iterating for a certain number of times, and inputting the iterated result into the step (12);
(12) And (3) calculating a signal to noise ratio (PSNR) of the input filled light field and the original light field in the step (1) to obtain a comparison result of the filling algorithm.
Also provided is a light field filling device based on tensor sparsity constraint, the device comprising:
the preliminary filling module is used for carrying out sliding window sampling on a light field losing part of information to obtain a series of tensor image blocks, searching the similar blocks, stacking the searched similar blocks to a fifth dimension, filling tensors of the missing information by using a light field filling model based on tensor sparse constraint, and respectively placing the image blocks in the filled tensors back
Obtaining a filled light field image at the original index of the light field;
The information complement module is used for carrying out super-resolution reconstruction on the filled light field image obtained by the preliminary filling module to obtain a high-resolution light field; performing sliding window sampling and similar block searching on the light field with the original resolution, and performing sliding window sampling, similar block searching and projection on the light field with the high resolution; sorting the similarity of the two data to obtain a similar block consisting of two groups of data, obtaining a five-dimensional tensor, filling the five-dimensional tensor by using a light field filling model based on tensor sparse constraint, and respectively placing the image blocks with the original sizes in the tensor back to the original index to obtain an information-completed light field;
wherein the tensor sparsity constraint-based light field filling model is:
On the upper part Satisfy Tucker decomposition
Wherein,For the sought, filled light field,/>For the light field of information absence,/>For/>A high resolution light field reconstructed with super resolution; /(I)Is a five-dimensional tensor obtained by stacking similar blocks,/>For/>Tensor after filling,/>For/>Five-dimensional tensor obtained by stacking similar blocks searched in (1)/>Is/>Is a projection result of (2); /(I)Is/>Is X (j) isAn expansion matrix along the modulus j; u j is/>A factor matrix along each mode; rank (X (j)) is the rank of X (j); gamma is the regularization parameter, t is a compromise parameter, a and b are fidelity parameters, and Ω is a known element subscript.
Drawings
FIG. 1 illustrates a flow chart of one embodiment of a tensor sparsity constraint based light field filling method in accordance with the present invention.
Detailed Description
The light field filling method based on tensor sparse constraint comprises the following steps:
(I) For a light field losing part of information, firstly, sliding window sampling is carried out on the light field to obtain a series of tensor image blocks, searching is carried out on the tensor image blocks, the searched similar blocks are stacked to a fifth dimension, the five-dimension tensor is filled by using a light field filling model based on tensor sparse constraint, and tensor image blocks in the filled five-dimension tensor are respectively placed back to an original index of a light field to obtain a filled light field image;
(II) carrying out super-resolution reconstruction on the filled light field image obtained in the step (I) to obtain a high-resolution light field; sliding window sampling is carried out on the light field with the original resolution, and sliding window sampling is carried out on the light field with the high resolution; searching similar blocks from the light field with original resolution, searching similar blocks from tensor blocks with corresponding integer times of the size in the high-resolution light field, projecting the latter back to the original size, sequencing the similarity of the two to obtain a five-dimensional tensor consisting of two groups of data, filling the five-dimensional tensor by using a light field filling model based on tensor sparse constraint, and respectively placing image blocks with the original size in the five-dimensional tensor back to the original index to obtain an information-complement light field;
the light field filling model based on tensor sparse constraint is as follows:
On the upper part Satisfy Tucker decomposition
Wherein,For the sought, filled light field,/>For the light field of information absence,/>For/>A high resolution light field reconstructed with super resolution; /(I)Is a five-dimensional tensor obtained by stacking similar blocks,/>For/>Tensor after filling,/>For/>Five-dimensional tensor obtained by stacking similar blocks searched in (1)/>Is/>Is a projection result of (2); /(I)Is/>Is X (j) is/>An expansion matrix along the modulus j; u j is/>A factor matrix along each mode; rank (X (j)) is the rank of X (j); gamma is the regularization parameter, t (0 < t < 1) is a compromise parameter, a and b are fidelity parameters, and Ω is a known element subscript.
The method comprises the steps of firstly carrying out sliding window sampling on a light field losing part of information to obtain a series of tensor blocks, carrying out searching on similar blocks of all tensor blocks, stacking the searched similar blocks to a fifth dimension to obtain a five-dimensional tensor, filling the five-dimensional tensor of the missing information by using a light field filling model based on tensor sparse constraint, and placing filled blocks in the five-dimensional tensor back to an original index to obtain a filled light field image; performing super-resolution reconstruction on the filled light field to obtain a high-resolution light field, performing sliding window sampling, similar block searching and stacking on the light field with original resolution, performing sliding window sampling, similar block searching, stacking and projection on the light field with high resolution to obtain the block size of the sliding window sampling with original resolution, and performing similarity sorting on two groups of similar blocks obtained by the former two groups to obtain a five-dimensional tensor, wherein the obtained similar blocks are the mixture of two groups of data. The method is used for filling the light field filling model based on tensor sparse constraint, and the filled image blocks in the five-dimensional tensor are put back to the original index to obtain an information-complement light field, so that vectorization of any dimension can be avoided in the reconstruction process, the characteristics in the light field are stored to the greatest extent, the light field filled by the method has good performance in objective aspect, and better visual experience is provided for users.
Preferably, in the step (I), one four-dimensional light field is expressed asWherein W and H represent spatial resolution, u and v represent angular resolution, dividing the whole light field image into a series of tensor blocks based on the relation of the angular dimensions of the light field and the non-local self-similarity, each tensor block maintaining the position and angular information of the original light field, searching for similar blocks, gathering all the searched similar blocks together, stacking in a fifth dimension to obtain a five-dimensional tensor block/>N is the number of similar blocks and i is the index of the reference block in the original light field.
Preferably, in the step (II), super-resolution reconstruction is performed on the original light field, so as to obtain a light field with high resolution. Sliding window sampling, similar block searching and stacking are carried out on the original light field, and the obtained tensor isSliding window sampling, similar block searching and stacking are carried out on a high-resolution light field, and the obtained tensor is/>Pair/>Projecting to make the size of the block consistent with that of the original light field sliding window sampling, wherein the obtained similar block is/>Is a projection operation. At this time, the similar blocks in the obtained tensor are/>And (3) withAnd performs similarity ordering on them together, selecting the top N most similar blocks as the final similar blocks. The resulting five-dimensional tensor is a mixture of similar blocks of the original resolution light field and the high resolution light field.
Preferably, the light field filling model based on tensor sparsity constraint is:
Wherein,
Preferably, when a=1 and b=0, the light field filling model based on tensor sparsity constraint is:
As shown in fig. 1, there is further provided a specific tensor sparsity constraint-based light field filling method, including the steps of:
(1) Inputting a light field image in a data set, randomly selecting pixels with fixed sampling rate as known elements, and inputting the rest pixels as unknowns in the step (2);
(2) Inputting a light field with missing information, and carrying out sliding window sampling on views of all angular dimensions according to the size and the step length of the input block to obtain a series of tensor blocks;
(3) Searching similar blocks for all tensor blocks, calculating the similarity between all pixels of each tensor block by taking Euclidean distance as a measure, sorting the searched similar blocks by distance, selecting the first n similar blocks, stacking the searched similar blocks on a fifth dimension to obtain a five-dimensional tensor, and recording the indexes of the spatial dimensions of all similar blocks in an original light field;
(4) Filling the five-dimensional tensor with a sparse tensor model, and respectively calculating The value of the constant variable is used for calculating a five-dimensional tensor after filling;
(5) Placing the five-dimensional tensor back to the original light field index according to the indexes in the steps (2) and (3); for one tensor block to be put back to the original index, n tensor image blocks similar to the tensor image block are provided, and at the moment, the average value of all similar blocks is taken, and the tensor image blocks are put back to the original index; jumping the missing light field input in the step (2) of the step (5) to the step (2) and carrying out iteration for a certain number of times; inputting the iterative result into the step (6);
(6) Sliding window sampling is carried out on the input filled light field to obtain a series of tensors, and indexes of pixels at the upper left corner of all tensor space dimensions in the space dimensions of the original light field are recorded;
(7) Performing super-resolution reconstruction on the filled light field input in the step (6), wherein the amplification factor is 2 times, interpolating between every two pixels, and estimating the pixel value between the two amplified pixels;
(8) Sliding window sampling is carried out on the light field after super-resolution reconstruction to obtain a series of tensor blocks (the size of the blocks is the resolution multiplied by the amplification factor in the step (1)), and similar blocks are searched in the set;
(9) Searching similar blocks for a series of tensors obtained in the step (6) and obtained in the step (8), wherein the tensors are projected to be the same as the tensors in the former, and selecting similar blocks with the front N of similarity; the similar blocks obtained at this time include: the similar blocks of the filled light field obtained in the step (6) and the similar blocks of the projection after the super-resolution reconstruction of the light field in the step (6);
(10) Filling the five-dimensional tensor obtained in the step (9) by using a sparse tensor model to obtain a filled five-dimensional tensor;
(11) The tensor image blocks in the five-dimensional tensor are placed back to the original light field index according to the index; for tensor blocks needing to be set back to the original index in the five-dimensional tensor, N tensors similar to the tensor blocks are provided, at the moment, the average value of all similar blocks is taken, and the tensor blocks are set back to the original index of the optical field; taking the result obtained in the step as input in the step (6), iterating for a certain number of times, and inputting the iterated result into the step (12);
(12) And (3) calculating a signal to noise ratio (PSNR) of the input filled light field and the original light field in the step (1), and obtaining a filled comparison result.
It will be understood by those skilled in the art that all or part of the steps in implementing the above embodiment method may be implemented by a program to instruct related hardware, where the program may be stored in a computer readable storage medium, where the program when executed includes the steps of the above embodiment method, and the storage medium may be: ROM/RAM, magnetic disks, optical disks, memory cards, etc. Thus, corresponding to the method of the invention, the invention also comprises a light field filling device based on tensor sparsity constraint, which is generally represented in the form of functional modules corresponding to the steps of the method. The device comprises:
The primary filling module is used for carrying out sliding window sampling on a light field losing part of information to obtain a series of tensor image blocks, searching the similar blocks, stacking the searched similar blocks to a fifth dimension, filling tensors of the missing information by using a light field filling model based on tensor sparse constraint, and respectively placing the image blocks in the filled tensors back to original indexes of the light field to obtain a filled light field image;
The information complement module is used for carrying out super-resolution reconstruction on the filled light field image obtained by the preliminary filling module to obtain a high-resolution light field; sliding window sampling is carried out on the light field with the original resolution ratio, and then searching of similar blocks is carried out; sliding window sampling is carried out on the high-resolution light field, then similar block searching is carried out, and the searched similar block is projected into a block with the same size as the original resolution sliding window sampling; sorting the two data to obtain a five-dimensional tensor composed of two groups of data, filling the five-dimensional tensor by using a light field filling model based on tensor sparse constraint, and respectively placing image blocks with original sizes in the tensor back to the original index to obtain an information-complement light field;
wherein the tensor sparsity constraint-based light field filling model is:
On the upper part Satisfy Tucker decomposition
Wherein,For the sought, filled light field,/>For the light field of information absence,/>For/>A high resolution light field reconstructed with super resolution; /(I)Is a five-dimensional tensor obtained by stacking similar blocks,/>For/>Tensor after filling,/>For/>Five-dimensional tensor obtained by stacking similar blocks searched in (1)/>Is/>Is a projection result of (2); /(I)Is/>Is X (j) is/>An expansion matrix along the modulus j; u j is/>A factor matrix along each mode; rank (X (j)) is the rank of X (j); gamma is the regularization parameter, t (0 < t < 1) is a compromise parameter, a and b are fidelity parameters, and Ω is a known element subscript.
The present invention is described in more detail below.
Based on tensor expression, sparse reconstruction and KBR model, the invention provides a light field filling algorithm. Representing a four-dimensional light field image asWhere W and H represent spatial resolution and u and v represent angular resolution. The goal of light field filling is to fill the light field/>, of the original, non-missing informationLight field/>, from observed, missing part of the informationAnd is reconstructed. Based on the relation of the angular dimensions and the non-local self-similarity of the light field, sliding window sampling is carried out on the whole light field image to obtain a series of tensor blocks, each tensor block stores the position and the angular information of the original light field, so that similar blocks are searched, all the searched similar blocks are gathered together, and stacking is carried out on a fifth dimension to obtain the tensor blocks of five dimensionsWhere w and h are block sizes, n is the number of similar blocks, and i is the index of the reference block in the original light field.
The light field image may be regarded approximately as an array of images of different angular views. Unlike multispectral images, there is a link between each angular view. If the angular dimension of the light field is simply vectorized and is approximately regarded as a multispectral image to be filled, the image features in the angular dimension are lost, and the filling effect is poor. If the light field image is considered as a series of linked images, each view will have subtle differences and links in terms of object occlusion, angle, etc. This association exists not only on integer pixels but also on sub-pixels. When the similar blocks are matched, the similar blocks existing on the sub-pixels cannot be searched, and then errors of the similar block matching are caused. In order to eliminate the problem of sub-pixel similar block mismatching of each angular view in the light field, a light field super-resolution reconstruction technology is introduced, and an original light field image is reconstructed into a light field with higher resolution. At this time, similar blocks existing on the sub-pixels can be searched, so that the problem of sub-pixel mismatching in the angle dimension is solved to a certain extent, and the filling quality is improved. A super-resolution reconstruction method (Block matching, PRINCIPAL COMPONENT ANALYSIS AND multivariate ridge regression, bm+pca+rr) based on Block matching, principal component analysis, and multiple ridge regression estimation is used herein to obtain high-resolution light field images.
Original light fieldThe super-resolution reconstruction result of (a) is/>Similar/>Sampling by sliding window to obtain a series of tensor blocks, searching similar blocks, and stacking on the fifth dimension to obtain five-dimensional tensorN H is the number of similar blocks, α is the magnification factor, and w and h are the block sizes for sliding window sampling of the original resolution light field. In order to obtain the optimal effect, similar blocks are searched for in blocks obtained in the original light field and the light field after super-resolution reconstruction. The similar block searched for the original light field is/>The similar blocks searched for the high-resolution light field arePair/>Projection is performed to make its block size and/>In agreement, the tensor obtained at this time is/>Is a projection operation. At this time, the obtained similarity block is/>And/>And (3) sorting the similarity, and selecting the first N similar blocks as final similar blocks. In this step, the final five-dimensional tensor is a mixture of similar blocks acquired in the original resolution light field and the high resolution light field.
The algorithm comprises the following steps:
For a light field with lost information, sliding window sampling is firstly carried out on the light field to obtain a series of tensor image blocks, and searching for similar blocks is carried out in the tensor image blocks. The search in this step is a similar block search over integer pixels. And stacking the searched similar blocks to a fifth dimension to obtain a five-dimensional tensor. Tensor sparse filling model (KBR-TC) based on Cronecker basis representation is a model for filling multispectral images (three-dimensional tensors) with missing information, and is expanded to a high dimension to obtain a light field filling model based on tensor sparse constraint. Filling the obtained five-dimensional tensor by using a tensor sparse constraint-based light field filling model, and placing filled tensor image blocks in the five-dimensional tensor back to the original index to obtain a filled light field, which is a preliminary result of the algorithm. The above-mentioned flow is the first step of the method.
Then, super-resolution reconstruction is carried out on the light field obtained in the last step, so that the problem of mismatching of similar blocks is solved. Carrying out sliding window sampling, similar block searching and stacking on the light field image obtained in the previous step; and carrying out sliding window sampling, similar block searching and stacking on the reconstructed high-resolution light field, and projecting the light field to the original size. There are two sets of data available, the "original size similar block searched by the original resolution light field" and the "integer multiple size block searched by the high resolution light field, and then projected back to the original size similar block", where the latter is the searched similar block present on the sub-pixels. The similarity ordering is performed on the two groups of data, and the obtained similarity block is the mixture of the two groups of data. Filling the image blocks by using a light field filling model based on tensor sparse constraint, and placing tensor image blocks in the five-dimensional tensor back to the original index to obtain an information complement light field. This is the end result of the present process. The above procedure is the second step of the method.
For the two-step filling mentioned above, a tensor sparsity constraint model is required.
On the upper partSatisfy Tucker decomposition
Wherein,For the sought, filled light field,/>For the light field of information absence,/>For/>A high resolution light field reconstructed with super resolution; /(I)Is a five-dimensional tensor obtained by stacking similar blocks,/>For/>Tensor after filling,/>For/>Five-dimensional tensor obtained by stacking similar blocks searched in (1)/>Is/>Is a projection result of (2); /(I)Is/>Is X (j) is/>An expansion matrix along the modulus j; u j is/>A factor matrix along each mode; rank (X (j)) is the rank of X (j); gamma is the regularization parameter, t (0 < t < 1) is a compromise parameter, a and b are fidelity parameters, and Ω is a known element subscript.
The above formula can be equivalently written as follows:
Wherein,
In particular, when a=1, b=0, the above formula can be written as follows:
namely the generalized form of the KBR model.
In order to show the effect of the method, experiments are carried out on a light field data set HCI old data set, and the method comprises six color light field images. And (3) carrying out averaging on the band (RGB) dimensions of all the color light fields to obtain a gray light field, and then cutting the gray light field from the center to obtain a gray light field image with the resolution of 256 multiplied by 9. All light fields are normalized.
During the experiment, a part of pixels are randomly selected as a known part, and the rest is a lost part. The sample rate of the experiment was 10%. The block size of the sliding window samples is 6 x 6. In the first step, a=1, b=0, and the number of iterations is 5. In the second step, the number of iterations is 3.
For more objective comparison, the experiment was compared with the other seven filling algorithms, with the signal to noise ratio (PSNR) as the criterion for the determination.
The algorithms for comparison include MC-ALM, haLRTC, trace-TV, t-SVD, mcpTC, scadTC, and KBR-TC.
TABLE 1
/>
From the comparison of the results in the table, it can be seen that at a sampling rate of 10%, the result PSNR of the algorithm of the present invention exceeds all other algorithms, and the result of the second step is an optimization of the result of the first step.
The present invention is not limited to the preferred embodiments, but can be modified in any way according to the technical principles of the present invention, and all such modifications, equivalent variations and modifications are included in the scope of the present invention.
Claims (7)
1. The light field filling method based on tensor sparse constraint is characterized by comprising the following steps of: the method comprises the following steps:
(I) For a light field losing part of information, firstly, sliding window sampling is carried out on the light field to obtain a series of tensor image blocks, searching for similar blocks is carried out on the tensor image blocks, the searched similar blocks are stacked to a fifth dimension, the five-dimension tensor is filled by a light field filling model based on tensor sparse constraint, and then the image blocks with the original size in the filled five-dimension tensor are respectively placed back to the original index of the light field to obtain a filled light field image;
(II) carrying out super-resolution reconstruction on the filled light field image obtained in the step (I) to obtain a high-resolution light field; sliding window sampling is carried out on the light field with the original resolution, and sliding window sampling is carried out on the light field with the high resolution; searching similar blocks from the light field with original resolution, searching similar blocks from tensor blocks with corresponding integer times of the size in the high-resolution light field, projecting the latter back to the original size, sequencing the similarity of the two to obtain a five-dimensional tensor consisting of two groups of data, filling the five-dimensional tensor by using a light field filling model based on tensor sparse constraint, and respectively placing image blocks with the original size in the five-dimensional tensor back to the original index to obtain an information-complement light field;
the light field filling model based on tensor sparse constraint is as follows:
On the upper part Satisfy Tucker decomposition
Wherein,For the sought, filled light field,/>For the light field of information absence,/>For/>A high resolution light field reconstructed with super resolution; /(I)Is a five-dimensional tensor obtained by stacking similar blocks,/>For/>Tensor after filling,/>For/>Five-dimensional tensor obtained by stacking similar blocks searched in (1)/>Is/>Is a projection result of (2); /(I)Is/>Is X (j) is/>An expansion matrix along the modulus j; u j is/>A factor matrix along modulo j; rank (X (j)) is the rank of X (j); gamma is the regularization parameter, t is a compromise parameter, a and b are fidelity parameters, Ω is a known element subscript, and i is the index of the reference block in the original light field.
2. The tensor sparsity constraint-based light field filling method of claim 1, wherein: in the step (I), a four-dimensional light field image is expressed asWherein W and H represent spatial resolution, u and v represent angular resolution, sliding window sampling is performed on the whole light field image based on the relation of the angular dimension of the light field and the non-local self-similarity to obtain a series of tensor image blocks, each tensor block stores the position and the angular information of the original light field, each tensor block is used as a reference block to search for similar blocks, all the searched similar blocks are gathered together and stacked on a fifth dimension to obtain a five-dimensional tensor block/>Where w and h are block sizes, n is the number of similar blocks, and i is the index of the reference block in the original light field.
3. The tensor sparsity constraint-based light field filling method of claim 2, wherein: in the step (II), sliding window sampling, similar block searching and stacking are carried out on the light field obtained in the step (I), and the obtained tensor isPerforming super-resolution reconstruction on the light field obtained in the step (I), performing sliding window sampling, similar block searching and stacking on the obtained high-resolution light field, wherein the obtained tensor is/>Pair/>Projecting to obtain tensors/>, which are consistent with the block sizes in the step (I)P (.) is the projection operation, where the resulting tensor is/>And/>And (3) carrying out similarity sorting on the set of the five-dimensional tensor, and selecting the first N most similar blocks, wherein the finally obtained five-dimensional tensor is the similar block searched in the original resolution light field and the high resolution light field.
4. A tensor sparsity constraint based light field filling method as claimed in claim 3, wherein: the light field filling model based on tensor sparse constraint is as follows:
Wherein,
5. The tensor sparsity constraint based light field filling method of claim 4, wherein: when a=1 and b=0, the light field filling model based on tensor sparsity constraint is:
6. the light field filling method based on tensor sparse constraint is characterized by comprising the following steps of: the method comprises the following steps:
(1) Inputting a light field image in a data set, randomly selecting pixels with fixed sampling rate as known elements, taking the rest elements as unknowns, and inputting the unknown elements into the step (2);
(2) For the input light field with missing information, sliding window sampling is carried out on views with all angle dimensions according to the input block size and step length to obtain a series of tensor blocks;
(3) Searching similar blocks of all tensor blocks, calculating the similarity between all pixels of each tensor block by taking Euclidean distance as a measure, sorting the searched similar blocks by distance, selecting the first n similar blocks, stacking the searched similar blocks on a fifth dimension to obtain a five-dimensional tensor, and recording the index of the space dimension of all similar blocks in an original light field;
(4) Filling the five-dimensional tensor with a sparse tensor model, respectively calculating the values of the kernel tensors, and calculating the filled five-dimensional tensor;
(5) Setting the tensor image blocks in the five-dimensional tensor back to the original light field index according to the indexes in the steps (2) and (3); for one tensor block to be put back to the original index, n tensor image blocks similar to the tensor image block are provided, all similar blocks are averaged to be used as a result, and the tensor image blocks are put back to the original index; taking the result of the step (5) as the missing light field input in the step (2), jumping to the step (2), and iterating; inputting the iterative result into the step (6);
(6) Sliding window sampling is carried out on the input filled light field to obtain a series of tensors, and indexes of pixels at the upper left corner of all tensor space dimensions in the space dimensions of the original light field are recorded;
(7) Performing super-resolution reconstruction on the filled light field input in the step (6), wherein the magnification is 2 times, and interpolating between every two pixels to obtain a high-resolution light field;
(8) Sliding window sampling is carried out on the light field reconstructed by super resolution to obtain a series of tensor blocks, wherein the size of each block is the size multiplied by the amplification factor of the block in the step (1);
(9) Searching similar blocks for a series of tensors obtained in the step (6) and obtained in the step (8), wherein the tensors are projected to be the same as the tensors in the former, and selecting similar blocks with the front N of similarity; the similar blocks obtained at this time include: the similar blocks of the filled light field obtained in the step (6) and the similar blocks of the light field super-resolution reconstructed projection obtained in the step (6);
(10) Filling the five-dimensional tensor obtained in the step (9) by using a sparse tensor model to obtain a filled five-dimensional tensor;
(11) The tensor image blocks in the five-dimensional tensor are placed back to the original light field index according to the index; for tensor blocks needing to be set back to the original index in the five-dimensional tensor, N tensors similar to the tensor blocks are provided, at the moment, the average value of all similar blocks is taken, and the tensor blocks are set back to the original index of the optical field; taking the result obtained in the step as input in the step (6), iterating, and inputting the iterated result into the step (12);
(12) And (3) calculating the peak signal-to-noise ratio (PSNR) of the input filled light field and the original light field in the step (1) to obtain a filled comparison result.
7. Light field filling device based on tensor sparse constraint, its characterized in that: it comprises the following steps:
The primary filling module is used for carrying out sliding window sampling on a light field losing part of information to obtain a series of tensor image blocks, searching the similar blocks, stacking the searched similar blocks to a fifth dimension, filling tensors of the missing information by using a light field filling model based on tensor sparse constraint, and respectively placing the image blocks in the filled tensors back to original indexes of the light field to obtain a filled light field image;
The information complement module is used for carrying out super-resolution reconstruction on the filled light field image obtained by the preliminary filling module to obtain a high-resolution light field; sliding window sampling is carried out on the light field with the original resolution ratio, and then searching of similar blocks is carried out; sliding window sampling is carried out on the high-resolution light field, then similar block searching is carried out, and the searched similar block is projected into a block with the same size as the original resolution sliding window sampling; similarity sorting is carried out on similar blocks searched from an original resolution light field and similar blocks searched from a high resolution light field and obtained through re-projection, so that similar blocks formed by two groups of data are obtained, a five-dimensional tensor is obtained, a light field filling model based on tensor sparse constraint is used for filling the five-dimensional tensor, and then image blocks with the original size in the tensor are respectively placed back to an original index to obtain an information-completed light field;
wherein the tensor sparsity constraint-based light field filling model is:
On the upper part Satisfy Tucker decomposition
Wherein,For the sought, filled light field,/>For the light field of information absence,/>For/>A high resolution light field reconstructed with super resolution; /(I)Is a five-dimensional tensor obtained by stacking similar blocks,/>For/>Tensor after filling,/>For/>Five-dimensional tensor obtained by stacking similar blocks searched in (1)/>Is/>Is a projection result of (2); /(I)Is/>Is X (j) is/>An expansion matrix along the modulus j; u j is/>A factor matrix along each mode; rank (X (j)) is the rank of X (j); gamma is the regularization parameter, t is a compromise parameter, a and b are fidelity parameters, Ω is a known element subscript, and i is the index of the reference block in the original light field.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110063571.XA CN112884645B (en) | 2021-01-18 | 2021-01-18 | Tensor sparse constraint-based light field filling method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110063571.XA CN112884645B (en) | 2021-01-18 | 2021-01-18 | Tensor sparse constraint-based light field filling method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112884645A CN112884645A (en) | 2021-06-01 |
| CN112884645B true CN112884645B (en) | 2024-05-03 |
Family
ID=76049106
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110063571.XA Active CN112884645B (en) | 2021-01-18 | 2021-01-18 | Tensor sparse constraint-based light field filling method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112884645B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2011004378A1 (en) * | 2009-07-08 | 2011-01-13 | Technion Research And Development Foundation Ltd. | Method and system for super-resolution signal reconstruction |
| CN104933684A (en) * | 2015-06-12 | 2015-09-23 | 北京工业大学 | Light field reconstruction method |
| JP2017033172A (en) * | 2015-07-30 | 2017-02-09 | 日本電信電話株式会社 | Image processing method, image processing device, and image processing program |
| CN107038719A (en) * | 2017-03-22 | 2017-08-11 | 清华大学深圳研究生院 | Depth estimation method and system based on light field image angle domain pixel |
| CN109064505A (en) * | 2018-07-26 | 2018-12-21 | 清华大学深圳研究生院 | A kind of depth estimation method extracted based on sliding window tensor |
-
2021
- 2021-01-18 CN CN202110063571.XA patent/CN112884645B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2011004378A1 (en) * | 2009-07-08 | 2011-01-13 | Technion Research And Development Foundation Ltd. | Method and system for super-resolution signal reconstruction |
| CN104933684A (en) * | 2015-06-12 | 2015-09-23 | 北京工业大学 | Light field reconstruction method |
| JP2017033172A (en) * | 2015-07-30 | 2017-02-09 | 日本電信電話株式会社 | Image processing method, image processing device, and image processing program |
| CN107038719A (en) * | 2017-03-22 | 2017-08-11 | 清华大学深圳研究生院 | Depth estimation method and system based on light field image angle domain pixel |
| CN109064505A (en) * | 2018-07-26 | 2018-12-21 | 清华大学深圳研究生院 | A kind of depth estimation method extracted based on sliding window tensor |
Non-Patent Citations (4)
| Title |
|---|
| Light Field Image Processing: An Overview;Gaochang Wu 等;IEEE Journal of Selected Topics in Signal Processing;第11卷(第7期);第926-954页 * |
| Reconstruction of compressively sampled light field by using tensor dictionaries;Yuping Wang 等;Multimedia Tools and Applications;第20449-20460页 * |
| 光场的稀疏编码模型及重构算法的研究;张晓帅;中国优秀硕士学位论文全文数据库（信息科技辑）;第I138-2086页 * |
| 多维稀疏表示模型及其应用研究;齐娜;中国博士学位论文全文数据库（信息科技辑）;第I138-16页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112884645A (en) | 2021-06-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Xie et al. | Hyperspectral image super-resolution using deep feature matrix factorization | |
| Liu et al. | A spectral grouping and attention-driven residual dense network for hyperspectral image super-resolution | |
| Du et al. | Spatial and spectral unmixing using the beta compositional model | |
| Bodrito et al. | A trainable spectral-spatial sparse coding model for hyperspectral image restoration | |
| Qi et al. | Using the kernel trick in compressive sensing: Accurate signal recovery from fewer measurements | |
| CN108520495B (en) | Hyperspectral image super-resolution reconstruction method based on clustering manifold prior | |
| Zheng et al. | Separable-spectral convolution and inception network for hyperspectral image super-resolution | |
| CN108805061A (en) | Hyperspectral image classification method based on local auto-adaptive discriminant analysis | |
| Tang et al. | Hyperspectral Band Selection via Spatial-Spectral Weighted Region-wise Multiple Graph Fusion-Based Spectral Clustering. | |
| JP4682670B2 (en) | Similar image search method and similar image search device | |
| Bai et al. | A hypergraph kernel from isomorphism tests | |
| CN111680579B (en) | Remote sensing image classification method for self-adaptive weight multi-view measurement learning | |
| CN108470192B (en) | Hyperspectral classification method and device | |
| Balestriero et al. | A data-augmentation is worth a thousand samples: Exact quantification from analytical augmented sample moments | |
| AU2015218184B2 (en) | Processing hyperspectral or multispectral image data | |
| CN115496950A (en) | Neighborhood information embedded semi-supervised discrimination dictionary pair learning image classification method | |
| CN115861076A (en) | Unsupervised hyperspectral image super-resolution method based on matrix decomposition network | |
| Dyer et al. | Self-expressive decompositions for matrix approximation and clustering | |
| Zhang et al. | 3D geometrical total variation regularized low-rank matrix factorization for hyperspectral image denoising | |
| Yang et al. | Learning degradation-aware deep prior for hyperspectral image reconstruction | |
| CN113486876A (en) | Hyperspectral image band selection method, device and system | |
| Song et al. | Nonlocal patch based t-svd for image inpainting: Algorithm and error analysis | |
| CN112884645B (en) | Tensor sparse constraint-based light field filling method and device | |
| CN111667495A (en) | Image scene analysis method and device | |
| Ferreira et al. | Geometry-aware neighborhood search for learning local models for image superresolution |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |