CN105160678A - Convolutional-neural-network-based reference-free three-dimensional image quality evaluation method - Google Patents

Abstract

The invention discloses a convolutional-neural-network-based reference-free three-dimensional image quality evaluation method. The method comprises the following steps: carrying out pretreatment on a 2D image; to be specific, inputting an image block obtained by pretreatment into a deep convolutional neural network, carrying out convolutional pooling processing to obtain an advanced feature of the image block, and then carrying out BP training by using the neural network based on a mass fraction of an original image so as to obtain a parameter of a base model; obtaining a corresponding basic model according to the parameter of the base model, carrying out centering on a three-dimensional image and carrying out pretreatment identical with that of the 2D image on the left image and the right image, and inputting the image blocks at the same position into the basic module simultaneously to obtain a corresponding advanced feature vector; and carrying out testing under a LIVE 3D database. On the basis of the reference-free evaluation algorithm, a good result better that the existing quality evaluation result can be obtained.

Description

Based on the nothing reference stereo image quality appraisal procedure of convolutional neural networks

Technical field

The present invention relates to a kind of nothing based on convolutional neural networks with reference to stereo image quality appraisal procedure.

Background technology

Developing rapidly in recent years along with multimedia technology, 3D stereo-picture enters the life of people in a variety of manners, and the visual effect experienced when participating in the cintest and shake true to nature that 3D stereo-picture brings is that 2D image is incomparable.It not only brings visual enjoyment on the spot in person, has also evoked the people's interest of things and cognition to external world.Meanwhile, widely using of stereo-picture also proposes higher requirement to the algorithm of stereoscopic image quality evaluation.

Stereo image quality evaluation occupies very consequence at stereo-picture, and it not only can pass judgment on the quality of Processing Algorithm in stereo-picture, can also optimize and algorithm for design, improves the efficiency of stereo-picture disposal system.Stereo image quality evaluation method is divided into subjective quality assessment and evaluating objective quality.Subjective evaluation method is exactly that the scoring of several observer to stereo image quality to be evaluated is weighted average comprehensive evaluation, its result meets human visual system's characteristic, but be subject to calculating that inconvenience, speed are slow, the restriction of high in cost of production factors, embedded system is difficult, thus cannot be used widely in actual applications; Method for evaluating objective quality then have simple to operate, cost is low, be easy to realize and can the feature such as real-time optimization algorithm parameter, is the emphasis of stereo image quality evaluation study.

Can be divided three classes the need of with reference to original reference picture according to evaluation procedure in evaluating objective quality algorithm: full reference model, half reference model and no reference model.

(1) full reference model: full reference algorithm, when calculating, must provide original undistorted reference picture.Its thinking is comparatively simple, but in actual applications, reference picture original in a lot of situation is difficult to obtain, and therefore its application scenario 55 receives restriction to a certain degree.

(2) half reference models: half reference model is not high to the quantitative requirement of reference picture, only need to provide the Partial Feature information that can represent original reference image, but are still limited to the reference information obtained by reference picture.

(3) no reference model: do not rely on original reference image without with reference to algorithm more realistic application scenarios, but to realize difficulty also maximum for it.Under normal circumstances, need to carry out modeling to natural image without with reference to algorithm, obtained the massfraction of entire image by some statistical information or 60 local features.

Therefore, there are problems in the existing assessment for image, can not meet existing demand, needs a kind of new algorithm badly to realize the assessment of stereo image quality.

Summary of the invention

For solving the deficiency that prior art exists, the invention discloses a kind of nothing based on convolutional neural networks with reference to stereo image quality appraisal procedure, the present invention utilizes 2D pictorial information, learn picture feature by convolutional neural networks, from the 2D picture of large amount of complex, extract key message be applied in three-dimensional image quality evaluation.Processed respectively by stereoscopic image centering left images, and regretional analysis is carried out to the advanced features obtained in the respective training process of left images, and use backpropagation to carry out overall situation fine setting at this, eventually through the unceasing study of degree of depth network, obtain the assessment of accurate stereoscopic image quality.

For achieving the above object, concrete scheme of the present invention is as follows:

Based on a nothing reference stereo image quality appraisal procedure for convolutional neural networks, comprise the following steps:

Select picture in LIVE2D database as training sample;

Pre-service is carried out to 2D image: by picture block identical sized by picture segmentation, and processed by local norms method;

Image block pre-service obtained is input to degree of depth convolutional neural networks, processed by convolution pondization, obtain the advanced features of image block thus, utilize original image DMOS mark to carry out BP training to neural network subsequently, obtain the network layer weight of basic model and biased optimum solution;

To select in LIVE3D database picture as new training sample;

Train the network layer weight obtained with biased according to basic model, build corresponding dual input basic model, by stereo image pair left images respectively through after the pre-service identical with 2D image, the image block of same position is input in basic model simultaneously, obtains corresponding advanced features vector;

Utilize multi-layer perception（MLP） training study two groups of advanced features vectors, obtain the image block massfraction belonging to this stereo-picture, eventually through the quality evaluation weighted mean of all image block massfractions of image pair being obtained to general image.

Further, to 2D image carry out pre-service time, adopt non-overlapped formula dividing method to be 32 × 32 image blocks by each Iamge Segmentation.

Further, adopt five layers of single input convolutional neural networks model of when utilizing 2D image to train, wherein output layer neuron number is 1, the image quality score namely predicted, asks for error train by this predicted value and given DMOS.

Further, five layers of single input convolutional neural networks model its be input as image block through pretreated 32 × 32, convolutional neural networks is by carrying out the computing of convolution pond to input, proper vector is obtained by feature map, pass through neural metwork training subsequently, random initializtion network weight, adopts stochastic gradient descent algorithm and back-propagation algorithm, the adjustment positioning system parameter of the overall situation.

During the process of convolution pondization, Convolution Formula is: wherein M _jrepresent convolved image size, for convolution kernel size, represent that convolutional layer is biased, f () represents activation function in convolution algorithm.Pond formula is: with represent that multiplicative is biased and an additivity is biased respectively, f () represents activation function in the algorithm of pond.

The acquisition of image block massfraction, with this formula obtain the image block massfraction belonging to this stereo-picture, O in formula _jn () represents that node exports, w _ji(n) and b _jrepresent corresponding network layer weight respectively and be biased.

When multi-layer perception（MLP） training positioning system parameter, have employed dropout method and momentum method, the probability wherein arranging the random zero setting of dropout is that 0.5, momentum formula is as follows:

${\mathrm{\&Delta;w}}^t=r^t{\mathrm{\&Delta;w}}^{t-1}-(1-r^t){\mathrm{\&epsiv;}}^t<{\&dtri;}_{w}L>;w^t=w^{t-1}+{\mathrm{\&Delta;w}}^t;{\mathrm{\&epsiv;}}^t={\mathrm{\&epsiv;}}_0{\left(d\right)}^t;r^t=\left\{\begin{array}{cc}\frac{t}{T}{r}_e+(1-\frac{t}{T})r_s,& t<T\\ r_e,& t\&GreaterEqual;T\end{array}\right.$

Wherein w ^trepresent network weight in t iterative process, ε ₀for learning rate, d represents the pad value of learning rate, r _sand r _esubmeter represents initial and final momentum (momentums) value, and T is the threshold value that Schistosomiasis control rate changes in different iterations.

For stereo-picture, extended network structure, increase by one group and there is identical network weight and biased single input convolutional network, two groups of single input convolutional network are connected in parallel, reach stereoscopic image to process left images simultaneously, multi-layer perception（MLP） middle layer is connected simultaneously, obtain the comprehensive characteristics that left and right multi-view image is obtained.It is specifically intended that this structure makes to belong to the right left images correspondence position image block of same image penetration depth network simultaneously, utilize the network trained before obtaining respective massfraction respectively, obtain corresponding advanced features vector.

When training, guarantee to enter the image block of network as input from the same position of the right left images of same image.

Beneficial effect of the present invention:

In order to carry out quality evaluation to non-reference picture better, a kind of convolutional neural networks (ConvolutionalNeuralNetwork, CNN) of the study stereo-picture partial structurtes information based on non-reference picture is newly proposed.Choose stereo image pair image block as input, make network can effectively learn to be present in perception information more responsive to the mankind in Local Structure of Image, and carry out picture quality estimation.Utilize many group convolution ponds layer, obtain the high-grade character representation to structure.Finally utilize multi-layer perception（MLP） (Multi-layerPerceptron, MLP) to the high-level characteristic that learns carry out study further and finally obtain the assessment mark of stereoscopic image quality.For different In-put design two class networks, utilize the single input network of 2D image to obtain network initial value, be applied to the dual input network (left and right multi-view image) for stereo-picture.Test under LIVE3D database, this nothing obtains compared to the better result of existing quality evaluation with reference to assessment algorithm.

Accompanying drawing explanation

An image in Fig. 1 aLIVE2D image library;

Another image in Fig. 1 bLIVE2D image library;

The left view of an image in Fig. 2 aLIVE3D image library;

The right view of an image in Fig. 2 bLIVE3D image library;

Fig. 3 single input convolutional network schematic diagram;

Fig. 4 dual input convolutional network schematic diagram.

Embodiment:

Below in conjunction with accompanying drawing, the present invention is described in detail:

Convolutional neural networks designs:

Task is herein exactly by the internal relation between 2D image searching picture quality and characteristics of image, and is applied to stereo image pair, utilizes regression algorithm to obtain the assessment of stereoscopic image quality by the advanced features of the left images obtained respectively.

(1) pre-service of 2D image

In the process that training network and carrying out is tested, the picture size majority in the LIVE2D database selected is 768 × 512, and picture entirety enters network affects network speed and accuracy rate by high degree.Consider that in picture, distortion level is identical everywhere, by 32 × 32 picture block identical sized by former picture segmentation, and processed by local norms method, this will weaken the impact that picture distortion brings and promote network to the robustness under contrast situation of change.

(2) single input convolutional network design

Image block pre-service obtained is input to degree of depth convolutional neural networks, by the process of convolution pondization, obtains the advanced features of image block, utilizes original image sole mass mark to carry out the parameter of BP training acquisition basic model subsequently by neural network.

(3) dual input convolutional network design

Basic model is obtained by first stage training, by stereo image pair left images respectively through after the pre-service identical with 2D image, the image block of same position is input in basic model simultaneously, Recurrent networks is utilized to learn the advanced features of the image block obtained in last two groups of basic models, to seek new network layer weight and biased optimum solution, finally obtain the optimization model of dual input convolutional neural networks.

Algorithm realization:

The pre-service of image: Fig. 1 a-Fig. 1 b is the LIVE2D image library image for first stage training, and its picture size is not of uniform size, mostly is 768 × 512.For solving the impact having dimension of picture problem to bring subsequent experimental, we adopt non-overlapped formula dividing method to be divided into 32 × 32 image blocks, and process image block by gray processing and local norms (LocalNormalization), local norms formula is as follows:

$\hat{I}(i,j)=\frac{I(j,j)-\mathrm{\&mu;}(j,j)}{\mathrm{\&sigma;}(i,j)+C}$

$\mathrm{\&mu;}(i,j)=\underset{p=-P}{\overset{p=P}{\&Sigma;}}\underset{q=-Q}{\overset{q=Q}{\&Sigma;}}I(i+p,j+q)$

$\mathrm{\&sigma;}(i,j)=\sqrt{\underset{p=-P}{\overset{p=P}{\&Sigma;}}\underset{q=-Q}{\overset{q=Q}{\&Sigma;}}{\left(I\right(i+p,j+q)-\mathrm{\&mu;}(i,j\left)\right)}^2}$

Wherein, I (i, j) represents the pixel value of pixel position (i, j), P, Q represent standardization computing time window size, C be greater than 0 constant.

To take the local norms process of image based on following two factors: first, the data volume that standardization computing will reduce in calculating process greatly, thus the impact that the error produced under the data environment on a large scale avoided is brought, and improve arithmetic speed.Secondly, the distortion level due to image is identical within the scope of full figure, and this also determines and local norms means can be used to process image block.

Single input convolutional neural networks: as shown in Figure 3, adopt five layers of single input convolutional neural networks model of a 32 × 32-7 × 7 × 50-50 × 2-800-800-1 when utilizing 2D image to train, wherein output layer neuron number is 1, i.e. the massfraction of image.It is input as the image block through pretreated 32 × 32.Convolutional neural networks is by carrying out the computing of convolution pond to input, proper vector is obtained by feature map, pass through neural metwork training subsequently, random initializtion network weight, adopt stochastic gradient descent (Stochasticgradientdescent, SGD) algorithm and backpropagation (backpropagation, BP) algorithm, the adjustment positioning system parameter of the overall situation.In order to more effective training positioning system parameter, we have employed dropout method and momentum method, and the probability of the wherein random zero setting of dropout is 0.5.To effectively improve the problem being absorbed in Local Minimum occurred in network training process like this.

Dual input convolutional neural networks: by using after 2D image trains for the first time to network, network has certain cognition degree to 2D picture quality, namely carries out reliable Score on Prediction to inputted 2D image.On this basis, the design feature of left and right dual-view is there is as illustrated in figures 2 a-2b for stereo pairs, extended network structure as shown in Figure 4, make to belong to the right left images correspondence position image block of same image penetration depth network simultaneously, the network utilizing the first stage to train obtained corresponding advanced features vector before obtaining respective massfraction respectively.Utilize multi-layer perception（MLP） (Multi-layerPerceptron, MLP) training study two groups of advanced features vectors, the image block massfraction belonging to this stereo-picture is obtained, eventually through the quality evaluation weighted mean of all image block massfractions of image pair being obtained to general image with this.The training of this part is the network weight obtained based on single input network, adopts a small amount of stereo pairs to train.

Experimental result: be checking algorithm and model herein, the database that we select in training and testing process is LIVE2DDATABASE and LIVE3DDATABASE, as Fig. 2 a-Fig. 2 b.Wherein first stage training is that training picture is all from 2DDATABASE, select 5 class distorted images as training sample, comprise JPEG compression (group), JPEG2000 compresses (group), GBLUR Gaussian Blur (group), white Gaussian noise (group), fast-fading transmits the distortion (group) that JPEG2000 code stream produces in rapid fading Rayleigh channel, and undistorted reference picture is not counted in training sample.Each time in iterative process, we are according to the ratio random selecting training group image of 4:1 and checking group image, ensure that the study of network can not be absorbed in local minimum and produce error result with this.In dual input network training, we, for 5 kinds of dissimilar distortions in LIVE3D, have chosen the picture training accounting for respective quantity 20%, and residue picture is for verifying the accuracy of network.For ensureing the accuracy of network and confidence level, we guarantee to enter the image block of network as input from the same position of the right left images of same image.

All give mean subjective mark difference (DifferenceMeanOpinionScore, DMOS) often organizing image in two benches training process, the subjective quality of DMOS value less expression volume image is poorer, otherwise subjective quality is then better.The vertical Spearman of employing herein related coefficient (SpearmanRank-orderCorrelationCoefficient, and linear parameter of consistency (LinearCorrelationCoefficient SROCC), LCC) as the index of model accuracy rate prediction, the value of SROCC and LCC, more close to 1, illustrates that the mark correlativity of difference of method for objectively evaluating and mean subjective is better.

By table 1 and table 2, we can see, the proposed evaluate parameter obtained under LIVE3D public data storehouse based on the dual input convolutional neural networks of non-reference picture is totally better than other quality evaluation algorithms.Wherein, the evaluation index for overall data storehouse image all promotes comparatively large, and the independent assessment simultaneously for all kinds of type of distortion picture is also improved.Which illustrate present networks, to dissimilar type of distortion, there is higher adaptability.

The comparison of all kinds of stereo-picture assessment algorithms of table 1 under SROCC index

The comparison of all kinds of stereo-picture assessment algorithms of table 2 under LCC index

There is shown herein a kind of convolutional neural networks image quality measure algorithm of non-reference picture.Basic ideas are, utilize convolutional network to the cognitive ability of image, by obtaining advanced features to the study of 2D characteristics of image, are applied in the assessment of 3D stereo image quality.Use a series of convolution, Chi Hua, regression algorithm, analyze eventually through the comprehensive of stereoscopic image centering left images the massfraction obtaining 3D rendering.By showing in the test in public data storehouse, this algorithm all can obtain desirable effect in the quality evaluation of overall data storehouse and each class type of distortion image.Research direction from now on mainly comprises improves network structure, optimized network parameter, obtains the quality evaluation algorithm of better adaptability.

By reference to the accompanying drawings the specific embodiment of the present invention is described although above-mentioned; but not limiting the scope of the invention; one of ordinary skill in the art should be understood that; on the basis of technical scheme of the present invention, those skilled in the art do not need to pay various amendment or distortion that creative work can make still within protection scope of the present invention.

Claims (10)

1., based on a nothing reference stereo image quality appraisal procedure for convolutional neural networks, it is characterized in that, comprise the following steps:

Select picture in LIVE2D database as training sample;

Pre-service is carried out to 2D image: by picture block identical sized by picture segmentation, and processed by local norms method;

Image block pre-service obtained is input to degree of depth convolutional neural networks, processed by convolution pondization, obtain the advanced features of image block thus, utilize original image DMOS mark to carry out BP training to neural network subsequently, obtain the network layer weight of basic model and biased optimum solution;

To select in LIVE3D database picture as new training sample;

Train the network layer weight obtained with biased according to basic model, build corresponding dual input basic model, by stereo image pair left images respectively through after the pre-service identical with 2D image, the image block of same position is input in basic model simultaneously, obtains corresponding advanced features vector;

Utilize multi-layer perception（MLP） training study two groups of advanced features vectors, obtain the image block massfraction belonging to this stereo-picture, eventually through the quality evaluation weighted mean of all image block massfractions of image pair being obtained to general image.

2. as claimed in claim 1 a kind of based on convolutional neural networks without with reference to stereo image quality appraisal procedure, it is characterized in that, to 2D image carry out pre-service time, adopt non-overlapped formula dividing method to be 32 × 32 image blocks by each Iamge Segmentation.

3. as claimed in claim 1 a kind of based on convolutional neural networks without with reference to stereo image quality appraisal procedure, it is characterized in that, adopt five layers of single input convolutional neural networks model of when utilizing 2D image to train, wherein output layer neuron number is 1, i.e. the massfraction of image.

4. as claimed in claim 1 a kind of based on convolutional neural networks without with reference to stereo image quality appraisal procedure, it is characterized in that, five layers of single input convolutional neural networks model its be input as image block through pretreated 32 × 32, convolutional neural networks is by carrying out the computing of convolution pond to input, proper vector is obtained by feature map, pass through neural metwork training subsequently, random initializtion network weight, adopt stochastic gradient descent algorithm and back-propagation algorithm, the adjustment positioning system parameter of the overall situation.

5. as claimed in claim 1 a kind of based on convolutional neural networks without with reference to stereo image quality appraisal procedure, it is characterized in that, when training positioning system parameter, have employed dropout method and momentum method, the probability wherein arranging random zero setting is 0.5.

6. as claimed in claim 1 a kind of based on convolutional neural networks without with reference to stereo image quality appraisal procedure, it is characterized in that, for stereo-picture, extended network structure, make to belong to the right left images correspondence position image block of same image penetration depth network simultaneously, utilize the network trained before obtaining respective massfraction respectively, obtain corresponding advanced features vector.

7. as claimed in claim 1 a kind of based on convolutional neural networks without with reference to stereo image quality appraisal procedure, it is characterized in that, when training, guaranteeing to enter the image block of network as input from the same position of the right left images of same image.

8. a kind of nothing based on convolutional neural networks, with reference to stereo image quality appraisal procedure, is characterized in that as claimed in claim 1, and during the process of convolution pondization, Convolution Formula is: wherein M _jrepresent convolved image size, for convolution kernel size, represent that convolutional layer is biased, f () represents activation function in convolution algorithm; Pond formula is: with represent that multiplicative is biased and an additivity is biased respectively, f () represents activation function in the algorithm of pond.

9. a kind of nothing based on convolutional neural networks, with reference to stereo image quality appraisal procedure, is characterized in that, the acquisition of image block massfraction, with this formula as claimed in claim 1 obtain the image block massfraction belonging to this stereo-picture, O in formula _jn () represents that node exports, w _ji(n) and b _jrepresent corresponding network layer weight respectively and be biased.

10. as claimed in claim 1 a kind of based on convolutional neural networks without with reference to stereo image quality appraisal procedure, it is characterized in that, when multi-layer perception（MLP） training positioning system parameter, have employed dropout method and momentum method, the probability wherein arranging the random zero setting of dropout is that 0.5, momentum formula is as follows:

Wherein w ^trepresent network weight in t iterative process, ε ₀for learning rate, d represents the pad value of learning rate, r _sand r _esubmeter represents initial and final momentum (momentums) value, and T is the threshold value that Schistosomiasis control rate changes in different iterations.