CN110458802A - Stereoscopic image quality evaluation method based on normalization of projection weights - Google Patents

Abstract

The invention belongs to the field of image processing, in order to propose a new image quality evaluation method, which can keep consistency with the subjective evaluation of human eyes, and solve the morbid problem in the network training process. It provides a research idea for the deep learning method of stereo image quality evaluation, and promotes the development of stereo imaging technology on a certain basis. To this end, the present invention, based on a stereoscopic image quality evaluation method based on the normalization of projection weights, fuses the left and right viewpoint maps of the stereoscopic image to obtain a single fused image, and then preprocesses the single image: dicing and normalizing Build a deep convolutional neural network model, use the preprocessed image slicing as the input of the deep convolutional neural network, and use the projection weight normalization and data batch normalization for the deep convolutional neural network. The structure is optimized, and the quality evaluation result of the stereo image is obtained through the output of the deep convolutional neural network. The present invention is mainly applied to image processing occasions.

Description

Stereoscopic image quality evaluation method based on normalization of projection weights

technical field

The invention belongs to the field of image processing, and relates to the application and optimization of image fusion and deep learning in stereoscopic image quality evaluation.

Background technique

Stereoscopic imaging technology can bring people a better visual experience, but from the collection of stereoscopic images to the display, there will be degradation problems ^[1-2] , and degraded images will affect people's perception of stereoscopic content. Reasonable and efficient assessment of image quality has become one of the hotspots in the field of stereoscopic information. Stereoscopic image quality evaluation methods are mainly divided into subjective evaluation and objective evaluation. However, the subjective evaluation experiment is time-consuming and labor-intensive, and the cost is high. The objective evaluation has strong operability. Therefore, it is of great practical significance to establish a reasonable and efficient objective evaluation mechanism for stereoscopic image quality.

Up to now, researchers have proposed a variety of stereo image quality evaluation methods, which can be roughly divided into traditional methods and artificial neural network methods. Most of the traditional methods perform feature extraction on the left and right views respectively, and then weight the quality scores of the left and right views to obtain the final objective evaluation value ^[3-7] . However, the features extracted by traditional methods may not truly reflect the essential features of the image. In order to better simulate the mechanism of feature extraction by the human eye, researchers apply artificial neural network to stereo image quality evaluation, such as [8-10], etc. The structure is relatively simple, and it cannot more accurately simulate the process of hierarchical processing of information by the human visual system. Compared with shallow neural networks, deep learning can better simulate the way the human brain processes information, and features can be extracted layer by layer through deep networks. Convolutional Neural Network (CNN) is a classic network in deep learning, suitable for computer vision, natural language processing and other fields. Zhang Wei et al. applied convolutional neural network to stereo image quality evaluation, used 2 convolutional layers and 2 pooling layers for feature extraction, and introduced a Multi-layer Perception (MLP) at the end of the network. , the learned features are fully connected to obtain the quality score ^[11] ; Chen Hui et al. adopted a convolutional neural network model with 12 convolutional layers ^[12] , Ding et al. adopted a convolutional neural network model with 5 convolutional layers. Convolutional neural network model, the obtained objective evaluation scores and human subjective evaluation scores have high consistency ^[13] . The structure of the deep neural network currently used in the field of stereo image quality evaluation has certain limitations: on the one hand, the arrangement of the convolution kernels in the network is relatively simple, they are all connected in sequence, and the extracted features are relatively single; On the one hand, the layers that make up the network are the most basic convolutional layers, pooling layers, and fully-connected layers, with fewer functions and no normalization, which makes the network unable to deal with the gradient dispersion problem.

In addition, it is found in practical research that when the human brain perceives stereo images, it first fuses the left and right views, and then processes the fused images hierarchically ^[14] . Lin et al. used traditional methods to evaluate the quality of the fused stereo images, but only fused the phase map and the magnitude map ^[15] . In order to better simulate this feature, the literature that uses deep learning to evaluate the quality of stereo images (such as [16]) also begins to use fusion images for processing, but the fusion method of this literature does not consider the threshold for gain enhancement and gain control ^{[ 17]} .

In view of the above problems, the present invention proposes a stereo image quality evaluation model based on a deep convolutional neural network. The preprocessed fusion image is used as the input of the network, so that the learning process of the network is more in line with the visual characteristics of the human eye. The model introduces a data batch normalization layer (Batch Normalization, BN) to ensure that the network output data and the input data are equally distributed to avoid the disappearance of gradients; the projection based weight normalization layer (PBWN) is introduced to normalize different orders of magnitude. The parameters of , alleviate the ill-conditioned phenomenon of the Hessian matrix, thereby improving the learning ability of the network. The first stage of the model is the convolution kernel parallel connection module, the second stage is the convolution kernel connecting the modules in sequence, and the residual unit is introduced to avoid the degradation of the network, and finally the fully connected layer is introduced to complete the quality evaluation of the stereo image.

SUMMARY OF THE INVENTION

In order to overcome the deficiencies of the prior art, the present invention aims to propose a stereo image quality evaluation method based on the normalization of projection weights based on a deep convolutional neural network. This method has better performance and is consistent with the subjective evaluation of the human eye. It also introduces batch normalization of data and normalization of projection weights to solve the ill-posed problem in the process of network training. This method provides a research idea for the deep learning method of stereo image quality evaluation, and promotes the development of stereo imaging technology on a certain basis. To this end, the technical solution adopted by the present invention is to fuse the left and right viewpoint images of the stereoscopic image based on the stereoscopic image quality evaluation method normalized by the projection weight to obtain a single fused image, and then preprocess the single image: Slicing and normalization; build a deep convolutional neural network model, use the preprocessed image slicing as the input of the deep convolutional neural network, and use projection weight normalization and data batch normalization to analyze the depth The structure of the hierarchical convolutional neural network is optimized, and the quality evaluation result of the stereo image is obtained through the output of the deep convolutional neural network.

The specific steps of obtaining the fusion image

Using a Gabor filter with 6 scales f _s ∈ {1.5, 2.5, 3.5, 5, 7, 10 }} and 8 directions θ ∈ {kπ/8|k=0,1…7}, the The left and right views after Gabor filtering are fused into one image according to formula (1).

Among them, I _l (x, y) and I _r (x, y) represent the pixel value at position (x, y) in the left and right views respectively, C(x, y) represents the pixel value of the fused image, and TCE represents For the enhancement component of this viewpoint, TCE ^* represents the suppression component of another viewpoint, and the calculation method is shown in formulas (2) and (3):

Among them, t represents the left view or right view, gc represents the enhancement threshold, ge represents the control threshold, and 48 images are obtained after Gabor filtering, represents the frequency information of the nth image of the t viewpoint filtered by the contrast sensitivity function, Represents the weight of the n-th image of the t viewpoint, i, j represent the Gabor filtering 6 scales f _s ∈ {1.5, 2.5, 3.5, 5, 7, 10} (cycles/degree) and 8 directions θ ∈ { kπ/8|k=0,1...7};

image preprocessing

The normalization calculation process is shown in formula (5):

Among them, I(x,y) represents the pixel value at the (x,y) coordinate point, μ(x,y) is the average value of the pixel value, σ(x,y) is the standard deviation of the pixel value, and ε is infinite Any positive number approaching 0.

Convolutional Neural Network Model

Based on the multi-scale extraction feature Inception structure and the residual network structure block Block, a deep convolutional neural network model with two convolution kernel arrangements is built. The input of the model is the cut block, and the model contains 1 Inception Structure, 1 convolution layer, 3 Block structures, 1 pooling layer and 1 fully connected layer, in the same layer inside the network Inception structure, through parallel operations of different sizes of convolution kernels, images of different scales are extracted features, and introduce a 1×1 convolution kernel to reduce network parameters and reduce computational complexity.

(1) Normalization of projection weights

In the planning problem that the network seeks the optimal solution, constraints on the weight matrix W of each layer are added:

min l(y,f(x;W))

where W={ _wi ,i=1,2...L} represents the set of network weight matrices, and the elements in the set are the weight matrices of each layer from the 1st layer to the Lth layer, l(y,f(x; W)) represents the loss function, with y as the expected output and f(x; W) as the actual output. Represents the retention matrix the main diagonal elements of and make the matrix All off-diagonal elements become 0.

This constraint defines the weight matrix of each layer in a subspace of the manifold space, that is, the weight matrix w of each layer satisfies the

ddiag(ww ^T )=E (7)

Using the Riemannian optimization theory to solve this constraint, the Riemann gradient in the manifold space can be obtained as

in, is the gradient obtained in the unconstrained case. When the weight matrix ω of each neuron satisfies the unit normalization, that is, ωω ^T =1, the Riemann gradient can be obtained based on formula (8) as:

The Riemann gradient is reduced by one term compared to the original gradient Analyze the norm of this reduced term:

The original gradient is used for calculation to reduce the amount of calculation, and formula (11) is used to update the weights:

(2) Batch normalization of data

The method of data batch normalization is shown in formula (12). During the training process, the mean μ and variance σ ² are calculated for the data of each batch, and each feature _xi is processed to obtain the batch normalized data. The activation after transformation is _yi .

During testing, E[x] is represented by the mean of all training batches, and var[x] is represented by an unbiased estimate of the variance of all training batches, as shown in formulas (13) and (14), where m is the size of each batch.

E[x]=E _B [μ _B ] (13)

Therefore, in the test phase, the formula for batch normalization of data is shown in formula (15). The functions of parameters γ and β are to zoom and translate, restore the expressive ability of the model, and improve the generalization performance of the network:

The characteristics and beneficial effects of the present invention are:

Based on the deep convolutional neural network, the present invention proposes a stereoscopic image quality evaluation method by introducing the normalization of projection weights, and the recognition rate of the stereoscopic image quality evaluation is high. The CNN model extracts the features of the preprocessed fused stereo images through two modules of convolution kernels, anterograde and parallel, so that the network can learn more fully from the images. Compared with the existing deep learning evaluation algorithm, the present invention introduces BN and PBWN for network optimization, solves the ill-posed problem in the network training process, and effectively improves the accuracy of network evaluation.

The stereoscopic image quality evaluation method of the present invention considers the human visual mechanism, takes the preprocessed fused image as the input of the network, and introduces the optimization of the structure of the deep convolutional neural network, which effectively improves the performance of the network. Experiments show that the evaluation results of the present invention have good consistency with subjective quality.

Description of drawings:

Figure 1 is a specific flow chart of the method.

Detailed ways

The steps of the technical solution adopted in the present invention are as follows: firstly, the left and right viewpoint maps of the stereoscopic image are fused to obtain a single fused image, and then the single image is sliced and normalized. Build a deep convolutional neural network model, take the preprocessed image small blocks as the input of the deep convolutional neural network, and use the projection weight normalization and data batch normalization to carry out the deep convolutional neural network structure. Optimization, the quality of the stereo image is obtained through the output of the deep convolutional neural network.

1. Fusion images

Inspired by the binocular competition phenomenon of human eyes in the biological science human visual system (Human Visual System, HVS), the present invention adopts a fusion image method and uses a Gabor filter to filter the image. The Gabor filter has six scales f _s ∈ {1.5, 2.5, 3.5, 5, 7, 10} (cycles/degree)} and eight directions θ ∈ {kπ/8|k=0,1,…7}. After filtering, 48 feature maps of each channel of each viewpoint can be obtained. According to the binocular competition mechanism, the final fusion image is calculated by combining the enhancement component of this viewpoint and the suppression component of another viewpoint.

2. Deep Learning

The convolutional neural network algorithm that started earlier and developed more maturely in deep learning is selected. Based on the Inception structure and the Block structure ^[18-19] , the present invention builds a deep convolutional neural network model with both the above two convolution kernel arrangements.

3. Optimization of network structure

Introduce a data batch normalization layer (Batch Normalization, BN) into the network to ensure that the network output data and the input data are equally distributed to avoid gradient disappearance; introduce a projection weight normalization layer (Projection Based Weight Normalization, PBWN) to normalize different quantities It can alleviate the ill-conditioned phenomenon of the Hessian matrix and improve the learning ability of the network.

The purpose of normalization of projection weights is to solve the ill-conditioned network training problem caused by the spatial symmetry of scaling weights in deep learning nonlinear networks ^[20] . The symmetry of the scaling weight space makes the Hessian matrix fall into an ill-conditioned state, which makes the network easy to fall into the local optimal value during training, which is not conducive to the network seeking the global optimal solution ^[21] . To alleviate this problem, the weights are unit-normalized to ensure that the magnitudes of the weights at each layer are the same.

Batch normalization of data can avoid the gradual shift of the data distribution, and effectively solve the problem of inconsistency between the original space and the target space ^[22] . Elements are activated to avoid gradient dispersion and gradient explosion. And the learnable reconstruction parameters are introduced to improve the network learning ability and generalization ability.

The present invention is tested on the public stereo image libraries LIVE I and LIVE II. LIVE-3DⅠ has 20 pairs of original images, 365 symmetrically distorted images, including 5 kinds of distortions (Gblur, WN, JPEG, JP2K and FF), LIVE-3DⅡ has 8 pairs of original images, 360 symmetrical and asymmetrical distorted images, including 5 distortion types (Gluur, WN, JPEG, JP2K and FF).

The method is described in detail below in conjunction with specific examples.

Based on the deep convolutional neural network, the present invention proposes a stereoscopic image quality evaluation method by introducing the normalization of projection weights, and the recognition rate of the stereoscopic image quality evaluation is high. The CNN model extracts the features of the preprocessed fused stereo images through two modules of convolution kernels, anterograde and parallel, so that the network can learn more fully from the images. Compared with the existing deep learning evaluation algorithm, the present invention introduces BN and PBWN for network optimization, solves the ill-posed problem in the network training process, and effectively improves the accuracy of network evaluation. The specific flow chart of the method proposed by the present invention is shown in FIG. 1 .

Specific steps are as follows:

1. Acquisition of fusion images

A Gabor filter is adopted, which has 6 scales f _s ∈ {1.5, 2.5, 3.5, 5, 7, 10} (cycles/degree)} and 8 directions θ ∈ {kπ/8|k=0,1 …7}. The left and right views after Gabor filtering are fused into one image according to formula (16).

Among them, I _l (x, y) and I _r (x, y) represent the pixel value at position (x, y) in the left and right views, respectively, and C(x, y) represent the pixel value of the fused image. TCE represents the enhancement component for this viewpoint, and TCE ^* represents the suppression component for another viewpoint. The calculation methods are shown in formulas (17) and (18):

Among them, t represents the left view or right view, gc represents the enhancement threshold, and ge represents the control threshold. After Gabor filtering, 48 images are obtained, represents the frequency information of the nth image of the t viewpoint filtered by the contrast sensitivity function, Represents the weight of the n-th image of the t viewpoint, i, j represent the Gabor filtering 6 scales f _s ∈ {1.5, 2.5, 3.5, 5, 7, 10} (cycles/degree) and 8 directions θ ∈ { kπ/8|k=0,1...7}.

2. Image preprocessing

The size of the fused single image is large, so the original image is cut into 32 × 32 image blocks to reduce the amount of network operations, and then the normalization calculation is performed. The normalization calculation process is shown in formula (20):

Among them, I(x,y) represents the pixel value at the (x,y) coordinate point, μ(x,y) is the average value of the pixel value, σ(x,y) is the standard deviation of the pixel value, and ε is infinite Any positive number approaching 0.

3. Convolutional Neural Network Model

Based on the Inception structure and the Block structure, the present invention builds a deep-level convolutional neural network model with two convolution kernel arrangement modes at the same time, and the input of the model is the cut block. The model contains 1 Inception structure, 1 convolution layer, 3 Block structures, 1 pooling layer and 1 fully connected layer, as shown in Figure 1.

In the same layer inside the network Inception structure, through the parallel operation of convolution kernels of different sizes, the features of different scales of the image can be extracted to make the extraction process more comprehensive and sufficient, and a 1×1 size convolution kernel is introduced to reduce the network size. parameters to reduce computational complexity.

Table 1 Network model parameter settings

The block structure introduces the idea of "residual error". By adding a channel, the input of the previous layer is directly connected to the output to solve the problem of network degradation.

4. Optimization of network structure

In the CNN network adopted in the present invention, a projection weight normalization layer (PBWN) pair and a data batch normalization layer (BN) are introduced after each convolution layer to normalize the weight parameters of each layer and the input data respectively. unify.

(1) Normalization of projection weights

In the planning problem that the network seeks the optimal solution, constraints on the weight matrix W of each layer are added:

min l(y,f(x;W))

where W={ _wi ,i=1,2...L} represents the set of network weight matrices, and the elements in the set are the weight matrices of each layer from the 1st layer to the Lth layer, l(y,f(x; W)) represents the loss function, with y as the expected output and f(x; W) as the actual output. Represents the retention matrix the main diagonal elements of and make the matrix All off-diagonal elements become 0.

This constraint defines the weight matrix of each layer in a subspace of the manifold space, that is, the weight matrix w of each layer satisfies the

ddiag(ww ^T )=E (22)

Using the Riemann optimization theory to solve this constraint, the Riemann gradient in the manifold space can be obtained as

in, is the gradient obtained in the unconstrained case. When the weight matrix ω of each neuron satisfies the unit normalization, that is, ωω ^T =1, the Riemann gradient can be obtained based on formula (8) as:

The Riemann gradient is reduced by one term compared to the original gradient Analyze the norm of this reduced term:

It shows that this term is not the dominant term in formula (24), and it is proved by experiments that the effect of the Riemann gradient is almost the same as that of the original gradient. Therefore, the present invention adopts the original gradient for calculation to reduce the amount of calculation.

Therefore, the present invention adopts formula (26) to update the weights.

(2) Batch normalization of data

The method of data batch normalization is shown in formula (27). During the training process, the mean μ and variance σ ² are calculated for each batch, and each feature _xi is processed to obtain the batch normalized data. The activation is y _i .

During testing, E[x] is represented by the mean of all training batches, and var[x] is represented by an unbiased estimate of the variance of all training batches, as shown in formulas (28) and (29), where m is the size of each batch.

E[x]=E _B [μ _B ] (28)

Therefore, in the test phase, the formula for batch normalization of data is shown in formula (30). The functions of parameters γ and β are to zoom and translate, restore the expressive ability of the model, and improve the generalization performance of the network.

5. Stereoscopic image quality evaluation results and analysis

The experiments of the present invention were carried out in two public stereo image databases, LIVE I and LIVE II databases, respectively. The evaluation indicators selected in the present invention are Pearson linear correlation coefficient (PLCC), Spearman rank order correlation coefficient (SROCC) and root mean square error (Root mean square error, RMSE). The larger the value of PLCC and SROCC, the smaller the value of RMSE, which means that the consistency between the model evaluation results and the subjective results is stronger, and the effect is better.

Table 2 is the performance comparison between the algorithm of the present invention and other methods on the LIVE-I and LIVE-II databases.

Table 2 Overall performance comparison of each evaluation method

Chen[12] did not give the overall evaluation index value of LIVE-II database, but only gave the index value of sub-distortion type, so the comparison with Chen[12] is carried out in Table 3 and Table 4. Table 2 shows that the performance of the algorithm of the present invention is obviously better than that of Heeseok [16]. This is because the present invention fully considers both linearity and nonlinearity in the process of fusing images. The received stimuli are linearly weighted, and when the stimuli reach the threshold of gain enhancement and gain control, nonlinear weighting is used. Compared with Lin [15], the present invention fuses the original image when fusing images, and [15] only fuses the underlying features of the image, so the index obtained by the present invention is better than the index [15]. Compared with other deep learning methods [11, 13] and traditional methods [5-7] that do not fuse images, the PLCC and SROCC obtained by the present invention are significantly improved. On LIVE-II, the PLCC obtained by the present invention ranks second, which is 0.0122% lower than that of Ding [13]. Compared with other algorithms, the RMSE calculated by the model of the present invention on the LIVE-I and LIVE-II databases is smaller. Considering the three indicators, the algorithm of the present invention has better performance in the quality evaluation of stereo images of symmetrical distortion and asymmetrical distortion. good performance.

The evaluation effect of the algorithm of the present invention on different distortion types is analyzed, as shown in Table 3 and Table 4.

Table 3 Performance comparison of each evaluation method for quality evaluation of stereo images with different distortion types in the LIVE-3DI database

Table 4. Performance comparison of each evaluation method for quality evaluation of stereo images with different distortion types in the LIVE-3D II database

When the network is tested, the PLCC and SROCC indicators in Table 3 and Table 4 are generally lower than the existing algorithms. This is because the experiments done by the present invention are classified into two categories, so even if only one picture is wrongly judged in the test, it will also be correct. PLCC has a great impact. Experiments show that the algorithm of the present invention has a better overall evaluation effect on the five distortion types. For the FF distortion types in the LIVE-I database and the FF and BLUR distortion types in the LIVE-II database, since the recognition rate reaches 100%, the PLCC, The value of SROCC also reaches 1, and the value of RMSE is 0.

Table 5 shows the effect of adding a PBWN layer after each convolutional layer and not adding a PBWN layer on the performance of the model. The results showed that the experimental results were significantly improved after adding PBWN. The recognition rate for LIVE-I image quality evaluation increased by 2.833% to 98.113%, and the recognition rate for LIVE-II image quality evaluation increased by 5.88% to 96.47%.

Table 5. Recognition rate of this algorithm for stereo image quality evaluation

Table 6 Time required for this algorithm test (unit: seconds)

Table 6 shows the effect of comparing with and without PBWN on the test time. PBWN makes the magnitudes of the weight parameters of each layer the same, and the weight parameters are normalized by units, which effectively avoids the ill-conditioned phenomenon of the Hessian matrix during the training process, improves the learning ability and generalization ability of the network, and accelerates the network convergence. Reduced time required for network testing.

references

[1] Zilly F, Kluger J, Kauff P. Production rules for stereo acquisition [J]. Proceedings of the IEEE, 2011, 99(4): 590-606.

[2] Urey H, Chellappan K V, Erden E, et al. State of the art instereoscopic and autostereoscoic displays[J]. Proceedings of the IEEE, 2011, 99(4): 540-555.

[3] Mao Xiangying, Yu Mei, Jiang Gangyi, et al. An Objective Evaluation Model of Stereo Image Quality Based on Structural Distortion Analysis [J]. Journal of Computer Aided Design and Graphics, 2012, 24(8): 1047-1056.

[4] Xu Shuning, Li Sumei. Stereoscopic image quality evaluation method based on visual saliency [J]. Information Technology, 2016, 2016(10): 91-93.

[5] Bensalma Rafik, Larabi Mohamed-Chaker. A perceptual metric forstereoscopic image quality assessment based on the binocular energy [J]. Multidimensional Systems and Signal Processing, 2013, 24(2): 281-316.

[6] Shao Feng, Jiang Gangyi, Yu Mei, et al.Binocular energy responsebased quality assessment of stereoscopic images[J].Digital Signal Processing, 2014, 29:45-53.

[7] Shao Feng, Lin Weisi, Wang Shanshan, et al. Learning Receptive Fields and Quality Lookups for Blind Quality Assessment of Stereoscopic Images [J]. IEEE Transactions on Cybernetics, 2016, 46(3): 730-743.

[8] Wang Guanghua, Li Sumei, Zhu Dan, et al. Application of extreme learning machine in objective evaluation of stereoscopic image quality [J]. Optoelectronics Laser, 2014, 2014(9): 1837-1842.

[9] Gu Shanbo, Shao Feng, Jiang Gangyi, et al. Objective Quality Evaluation Model of Stereo Image Based on Support Vector Regression [J]. Journal of Electronics and Information, 2012, 34(2): 368-374.

[10] Wu Xingguang, Li Sumei, Cheng Jincui. Objective Evaluation of Stereo Image Based on Genetic Neural Network [J]. Information Technology, 2013, 2013(5): 148-153.

[11] Zhang Wei, Qu Cchenfei, Lin Ma, et al. Learning structure of stereoscopic image for no-reference quality assessment with convolutionalneural network [J]. Pattern Recognition, 2016, 59(C): 176-187.

[12] Chen Hui, Li Chaofeng. Stereo color image quality evaluation of deep convolutional neural network [J]. Computer Science and Exploration, 2018, 12(08): 1315-1322

[13] Ding Yong, Deng Ruizhe, Xie Xin, et al. Reference stereoscopic imagequality assessment using convolutional neural network for adaptive featureextraction[J]. IEEE Access , 2018, 2018(6): 37595-37603.

[14] Hubel D.H., Wiesel T.N. Receptive fields of singleneurones in the cat’s striate cortex [J]. The Journal of Physiology, 1959, 148(3): 574-591.

[15] Lin Yancong, Yang Jiachen, Lu Wen, et al. Quality index forstereoscopic images by jointly evaluating cyclopean amplitude and cyclopean phase [J]. IEEE Journal of Selected Topics in Signal Processing, 2017, 11(11): 89-101.

[16] Oh Heeseok, Ahn Sewoong, Kim Jongyoo, et al. Blind deep S3D imagequality evaluation via local to global feature aggregation [J]. IEEE Transactions on Image Processing, 2017, 26(10): 4923-4936.

[17] Ding Jian, Klein S.A., Levi D.M.Binocular combination of phaseandcontrast explained by a gain-control and gain-enhancement model[J].Journal of Vision, 2013, 13(2): 13.

[18] Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions [J]. 2014: 1-9.

[19] He K, Zhang X, Ren S, et al. Deep Residual Learning for ImageRecognition [J]. 2015:770-778.

[20] L. Huang, X. Liu, B. Lang, and B. Li. Projection based weightnormalization for deep neural networks. CoRR, abs/1710.02338, 2017.

[21] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. DeepLearning. MIT Press, 2016.

[22] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015.

Claims (3)

1. A three-dimensional image quality evaluation method based on projection weight normalization is characterized in that left and right viewpoint images of a three-dimensional image are fused to obtain a single fused image, and then the single image is preprocessed: cutting and normalizing; and constructing a deep convolutional neural network model, taking the preprocessed image blocks as the input of the deep convolutional neural network, optimizing the structure of the deep convolutional neural network by adopting projection weight normalization and data batch normalization, and obtaining the quality evaluation result of the stereo image through the output of the deep convolutional neural network.

2. The method of claim 1, wherein the projection weight normalization-based stereo image quality evaluation method,

the specific steps of obtaining the fused image

Using a Gabor filter having 6 dimensions f_sE {1.5,2.5,3.5,5,7,10} } and 8 directions theta e { k pi/8 | k ═ 0,1 … 7}, and fusing the Gabor-filtered left and right views into an image according to formula (1).

Wherein, I_l(x, y) and I_r(x, y) denotes a pixel value at a position (x, y) in the left and right views, respectively, C (x, y) denotes a pixel value of the fused image, TCE denotes an enhancement component to the present viewpoint, TCE denotes^*Representing the suppressed component for the other viewpoint, the calculation is as shown in equations (2) and (3):

wherein t represents a left viewpoint or a right viewpoint, gc represents an enhancement threshold, ge represents a control threshold, 48 images are obtained after Gabor filtering,frequency information of the nth image representing the t viewpoint filtered by the contrast sensitivity function,weights of the n-th image representing t-viewpoint, i, j representing 6 scales f of Gabor filtering, respectively_sE {1.5,2.5,3.5,5,7,10} (cycles/degree) and 8 directions θ e { k pi/8 | k ═ 0,1 … 7 };

image pre-processing

The normalization calculation process is shown in equation (5):

wherein I (x, y) represents a pixel value at the (x, y) coordinate point, μ (x, y) is an average value of the pixel values, σ (x, y) is a standard deviation of the pixel values, and ∈ is an arbitrary positive number approaching 0 infinitely;

convolutional neural network model

Based on multi-scale extraction of feature inclusion structure and residual network structure Block, build a deep convolutional neural network model with two convolutional kernel arrangement modes, the input of the model is a small Block after cutting, the model comprises 1 inclusion structure, 1 convolutional layer, 3 Block structures, 1 pooling layer and 1 full-connection layer, in the same layer in the network inclusion structure, through convolutional kernel parallel operation of different sizes, the features of different scales of the image are extracted, and the convolutional kernel of 1 × 1 size is introduced to reduce network parameters, so that the computational complexity is reduced.

3. The method of claim 1, wherein the projection weight normalization-based stereo image quality evaluation method,

(1) projection weight normalization

In the planning problem of seeking the optimal solution by the network, adding the constraint on the weight matrix W of each layer:

min l(y,f(x；W))

wherein W ═ { W ═ W_iI-1, 2 … L represents a set of network weight matrices, the elements in the set being layers 1 to LL (y, f (x; W)) represents the loss function, with y being the desired output and f (x; W) being the actual output.Representation reservation matrixAnd the main diagonal elements of the matrixAll off-diagonal elements become 0.

The constraint defines the weight matrix of each layer in a subspace of the manifold space, namely, the weight matrix w of each layer satisfies

ddiag(ww^T)＝E (7)

Solving the constraint by using Riemann Riemannian optimization theory to obtain Riemannian gradient in manifold space

Wherein,the gradient is obtained without constraint. When the weight matrix omega of each neuron meets the unit normalization, namely omega^TThe riemann gradient is obtained based on equation (8) as 1:

riemann gradient is reduced by one term compared with original gradientThe norm of this reduced term is analyzed:

the original gradient is adopted for calculation to reduce the calculation amount, and the formula (11) is adopted for weight updating:

(2) batch normalization of data

The batch normalization method of data is shown in formula (12), and during the training process, the mean value mu and the variance sigma are calculated for the data of each batch²For each feature x_iProcessing to obtain the activation y after data batch normalization_i。

During the test, the mean value of all training batchs is used to represent E [ x ], the unbiased estimation of the variance of all training batchs is used to represent var [ x ], and m is the size of each batch as shown in the formulas (13) and (14).

E[x]＝E_B[μ_B] (13)

Therefore, in the testing stage, the formula of data batch normalization is shown in formula (15), the function of the parameter gamma and beta is zooming and translation, the expression capability of the model is restored, and the network generalization performance is improved: