KR20210076660A - Method and Apparatus for Stereoscopic Image Quality Assessment Based on Convolutional Neural Network - Google Patents

Abstract

Provided, in the present embodiment, are a device and method that extracts, for a stereoscopic image, a regional feature using a patch-based neural network, and evaluates a quality of the stereoscopic image by grouping the regional feature into a plurality of feature groups to predict a global score. Therefore, the present invention is capable of having an effect that can evaluate the quality of the stereoscopic image.

Description

Method and Apparatus for Stereoscopic Image Quality Assessment Based on Convolutional Neural Network

The technical field to which the present invention pertains relates to a method and apparatus for evaluating the image quality of a stereoscopic image.

The content described in this section merely provides background information for the present embodiment and does not constitute the prior art.

A stereoscopic image is a stereoscopic image that utilizes binocular disparity so that the viewer can feel a three-dimensional image through the difference in angle between the two eyes.

There are three major problems in applying the existing two-dimensional image quality evaluation model to the stereoscopic image quality evaluation.

First, there is insufficient training data for stereoscopic images, secondly, there is no regional evaluation score of the image, and thirdly, a general image classification model cannot be used as it is when evaluating the quality of stereoscopic images.

Korean Patent Publication No. 10-1393621 (2014.05.02)

Embodiments of the present invention evaluate the picture quality of a stereoscopic image by extracting local features using a patch-based neural network for a stereoscopic image, and predicting a global score by grouping the local features into a plurality of feature groups. has the main purpose of

Other objects not specified in the present invention may be additionally considered within the scope that can be easily inferred from the following detailed description and effects thereof.

According to one aspect of this embodiment, in the method for evaluating the quality of a stereoscopic image by a computing device, the step of calculating a local correct score through a reference quality evaluation model for each patch for an original stereoscopic image and a distorted stereoscopic image , extracting regional features through a patch-based non-reference quality evaluation model for the distorted stereoscopic image and predicting a regional score based on the regional correct score, grouping the regional features into a plurality of feature groups, and the Provided is a method for evaluating the quality of a stereoscopic image, comprising predicting a global score for a plurality of feature groups.

In the step of calculating the local correct score, a stereoscopic image is converted into a left-right composite image (Cyclopean Image), and a reference left-right composite image obtained by converting the original stereoscopic image and the distorted stereoscopic image The transformed distorted left and right composite images may be divided into patches and applied to the reference quality evaluation model.

The calculating of the regional correct answer score may include calculating the local correct answer score for each patch by averaging the result scores calculated by applying the reference quality evaluation model to each pixel of the image.

Predicting the global score includes: (i) a first feature group in which an average is calculated for the regional feature, (ii) a second feature group in which a variance is calculated for the local feature, (iii) the local feature A global score can be predicted for an integrated feature group that combines all of the third feature group in which the average of the upper proportion is calculated in the histogram of have.

According to another aspect of this embodiment, in the apparatus for evaluating the image quality of a stereoscopic image including one or more processors and a memory for storing one or more programs executed by the one or more processors, the processor is configured to distort the original stereoscopic image For the stereoscopic image, a local correct score is calculated through a reference quality evaluation model for each patch, and the processor extracts a regional feature through a patch-based non-reference quality evaluation model for the distorted stereoscopic image, and the local correct answer Predicting a regional score based on the score, the processor groups the regional features into a plurality of feature groups, and provides an apparatus for evaluating the quality of stereoscopic images, characterized in that predicting global scores for the plurality of feature groups can do.

As described above, according to the embodiments of the present invention, for a stereoscopic image, a regional feature is extracted using a patch-based neural network, and a global score is predicted by grouping the local feature into a plurality of feature groups, It has the effect of evaluating the quality of a stereoscopic image.

Even if it is an effect not explicitly mentioned herein, the effects described in the following specification expected by the technical features of the present invention and their potential effects are treated as if they were described in the specification of the present invention.

1 is a block diagram illustrating an apparatus for evaluating the quality of a stereoscopic image according to an embodiment of the present invention.
2 is a flowchart illustrating a method for evaluating the quality of a stereoscopic image according to another embodiment of the present invention.
3 is a diagram illustrating a non-referenced image quality evaluation model applied to a method for evaluating the image quality of a stereoscopic image according to another embodiment of the present invention.
4 is a diagram illustrating a picture quality map calculated by the method for evaluating the picture quality of a stereoscopic image according to another embodiment of the present invention.
5 is a diagram illustrating a method for evaluating the quality of a stereoscopic image according to another embodiment of the present invention to extract regional features and predict a regional score.
6 is a diagram illustrating the prediction of a global score for a feature group by a method for evaluating the quality of a stereoscopic image according to another embodiment of the present invention.
7 shows simulation results performed according to embodiments of the present invention.

Hereinafter, in the description of the present invention, if it is determined that the subject matter of the present invention may be unnecessarily obscured as it is obvious to those skilled in the art with respect to related known functions, the detailed description thereof will be omitted, and some embodiments of the present invention will be described. It will be described in detail with reference to exemplary drawings.

Unlike the existing computer vision processing method, there are three problems when deep learning is applied to the stereoscopic image quality evaluation method.

First, there is a lack of training data for stereoscopic images.

The ImageNet image database used for image classification contains more than 15 million images matched with class labels. Since the labels stored in the existing image database are semantically clear, it is easy to obtain ground truth.

On the other hand, the database for image quality evaluation of stereoscopic images does not exceed 1000 images. The LIVE database only has 365 distorted images for the 5 distortion types, and the SA database only has 360 images for the 9 distortion types.

Second, there is no regional evaluation score of the image.

It is not easy to obtain verification data because a psychological test must be performed to obtain a subjective score. Even if the patch method that divides the image is applied, subjective scores are required for each patch, but there is no verified database. Only subjective scores for the entire image are secured, and there is no regional verification data. The average of the subjective scores for the whole image is different from the score of each patch. The score in each patch of the image is not uniform. For example, if a road region in the middle of a park photo is severely distorted, the regional image quality of this region may be lower than the image quality score of one image. Patch-based supervised learning cannot be performed without applying reliable local verification data.

Third, when evaluating the quality of stereoscopic images, a general image classification model cannot be used as it is.

Since the image recognition model must detect the target object even if distortion occurs in the image, it must learn a characteristic strong against distortion. On the other hand, since the image quality evaluation model needs to sensitively capture distortion information, it is necessary to extract distortion-sensitive features.

The apparatus for evaluating the image quality of a stereoscopic image according to the present embodiment evaluates the image quality of the stereoscopic image in a non-referential manner by applying deep learning.

The stereoscopic image quality evaluation device uses a patch-based approach to overcome the lack of training data. In order to solve the problem of not having a reliable regional subjective score, pseudo-verification data for regional patches are obtained through an evaluation model of patch-based learning and reference methods.

1 is a block diagram illustrating an apparatus for evaluating the quality of a stereoscopic image according to an embodiment of the present invention.

The stereoscopic image quality evaluation apparatus 110 includes at least one processor 120 , a computer-readable storage medium 130 , and a communication bus 170 .

The processor 120 may control to operate as the apparatus 110 for evaluating the quality of a stereoscopic image. For example, the processor 120 may execute one or more programs stored in the computer-readable storage medium 130 . The one or more programs may include one or more computer-executable instructions, which, when executed by the processor 120 , cause the stereoscopic image quality evaluation apparatus 110 to perform operations according to the exemplary embodiment. may be configured to perform.

Computer-readable storage medium 130 is configured to store computer-executable instructions or program code, program data, and/or other suitable form of information. The program 140 stored in the computer-readable storage medium 130 includes a set of instructions executable by the processor 120 . In one embodiment, the computer-readable storage medium 130 includes memory (volatile memory, such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, It may be flash memory devices, other types of storage media that can be accessed by the apparatus 110 for evaluating the image quality of a stereoscopic image and store desired information, or a suitable combination thereof.

The communication bus 170 interconnects various other components of the apparatus 110 for evaluating the quality of a stereoscopic image, including the processor 120 and the computer-readable storage medium 140 .

The stereoscopic image quality evaluation apparatus 110 may also include one or more input/output interfaces 150 and one or more communication interfaces 160 that provide interfaces for one or more input/output devices. The input/output interface 150 and the communication interface 160 are connected to the communication bus 170 . The input/output device may be connected to other components of the stereoscopic image quality evaluation device 110 through the input/output interface 150 .

The stereoscopic image quality evaluation apparatus 110 calculates regional correct scores for the original stereoscopic image and the distorted stereoscopic image through a reference quality evaluation model for each patch.

The stereoscopic image quality evaluation apparatus 110 extracts regional features from the distorted stereoscopic image through a patch-based non-referenced image quality evaluation model, and predicts a regional score based on a regional correct score.

The stereoscopic image quality evaluation apparatus 110 groups regional features into a plurality of feature groups and predicts global scores for the plurality of feature groups.

2 is a flowchart illustrating a method for evaluating the quality of a stereoscopic image according to another embodiment of the present invention. The method for evaluating the picture quality of a stereoscopic image may be performed by an apparatus or a computing device for evaluating the picture quality of a stereoscopic image.

In step S210, the processor calculates a local correct score for the original stereoscopic image and the distorted stereoscopic image through the reference quality evaluation model for each patch.

In step S220, the processor extracts regional features through a patch-based non-reference quality evaluation model for the distorted stereoscopic image, and predicts a regional score based on the local correct score.

In step S230, the processor groups local features into a plurality of feature groups and predicts global scores for the plurality of feature groups.

3 is a diagram illustrating a non-referenced image quality evaluation model applied to a method for evaluating the image quality of a stereoscopic image according to another embodiment of the present invention.

The image quality evaluation model applied to the method for evaluating the image quality of a stereoscopic image according to the present embodiment performs two training steps. The first step is to predict regional quality scores using the pseudo-validation data obtained through the reference quality evaluation model. The second step predicts the global subjective score using the model learned in the first step.

First, the stereoscopic image quality evaluation apparatus method generates pseudo-verification data through a reference image quality evaluation model.

4 is a diagram illustrating a picture quality map calculated by the method for evaluating the picture quality of a stereoscopic image according to another embodiment of the present invention.

The reference image and the distorted image are input to the reference quality evaluation model. The reference picture quality evaluation model creates a picture quality map. The quality map can be divided into 64x36 patches, and the average of each patch is taken to obtain a regional score. For each patch, pseudo-verification data is obtained, and a regional image quality score map is generated. A Structural Similarity Index (SSIM) may be applied to generate the image quality map.

In the step of the processor calculating the local correct score, the stereoscopic image is converted into a left-right composite image (Cyclopean Image), and the standard left-right composite image converted from the original stereoscopic image and the distorted stereoscopic image are converted. The distorted left and right composite images are divided into patches and applied to the reference quality evaluation model.

A method of acquiring the left and right composite images is expressed as in Equation (1).

d is the pixel parallax of the left _{and right images I l} and I _{r , and W l} and W _r represent the normalized weights by the Gabor filter response.

The left and right composite image pairs I ^' _c for the distorted stereoscopic image are synthesized in a mapping manner. Since there is no reliable disparity measurement from the distorted stereoscopic image pair, the disparity map of the reference image pair is used for the left and right composite image pairs.

A Structural Similarity Index (SSIM) can be applied to measure the image quality map between the reference left and right composite images I _c and the distorted left and right composite images I ^' _c. Structural Similarity Index (SSIM) is an evaluation technique that satisfies a representative human visual system that evaluates images using structural information using the fact that the human visual system is sensitive to structural information of images.

N은 패치의 전체 개수이다.It refers to a segmented set of patches obtained in the left and right composite images to the stereoscopic image distortion to P _C.

N is the total number of patches.

In the step of calculating the local correct score, a local correct score is calculated for each patch by averaging the result scores calculated by applying the reference quality evaluation model to each pixel of the image. The local correct score is expressed as Equation (2).

h _p and w _p are the vertical and horizontal patch sizes. For example, h _p may be set to 16, and w _p may be set to 18.

및

를 생성한다. M은 왜곡된 스테레오스코픽 이미지 쌍의 전체 개수이다. The processor may pre-process data input to the non-reference image quality evaluation model. The distorted left _{and right images I l} and I _r are normalized, and the processor sets the normalized image

and

create M is the total number of distorted stereoscopic image pairs.

가 비참조 화질 평가 모델에 입력된다.Each image is divided into non-overlapping patches. patch set

is input to the non-reference quality evaluation model.

5 is a diagram illustrating a method for evaluating the quality of a stereoscopic image according to another embodiment of the present invention to extract regional features and predict a regional score.

The first step performed by the patch-based non-reference quality evaluation model is the regression step of regional subjective scores. The non-reference quality evaluation model is a layered network and a model that learns weights and biases. The non-reference quality evaluation model may be implemented as a neural network such as a Convolutional Neural Network (CNN).

A CNN model is applied to train a local quality score for each patch. Two convolutional layers and four fully connected layers can be used. Using the extracted feature vectors with 100 dimensions, the local quality score can be regressed.

는 파라미터

에 의한 컨볼루션 레이어와 풀리 커넥티드 레이러를 갖는 특징 벡터 추출기이다.

는 파라미터

에 의한 회귀 함수이다. 100 개의 차원의 특징 벡터

는 n 번째 패치 쌍으로부터

에 의해 추출된다.

is the parameter

It is a feature vector extractor with a convolutional layer and a fully connected layer by

is the parameter

is a regression function by 100 dimensional feature vector

is from the nth patch pair

is extracted by

은 모델에 독립적으로 입력되고 다른 패치와 상관 관계가 없다.each patch pair

is input independently into the model and has no correlation with other patches.

를 최적화하기 위한 목적 함수는 슈도 검증자료와 예측된 점수 간의 평균 제곱 오차를 사용할 수 있다.network parameters

The objective function for optimizing α can use the mean square error between the pseudo-test data and the predicted score.

는 평균 제곱 오차로 정의된 손실 함수이다.

is the loss function defined as the mean squared error.

6 is a diagram illustrating the prediction of a global score for a feature group by a method for evaluating the quality of a stereoscopic image according to another embodiment of the present invention.

The second step performed by the patch-based non-reference quality evaluation model is the regression step of the global subjective score.

는

에 의해 스테레오스코픽 이미지 쌍으로부터 획득된다.Features vector set

is

is obtained from a stereoscopic image pair by

The step of predicting the global score includes (i) a first feature group in which the average of the regional features is calculated, (ii) a second feature group in which the variance is calculated for the regional features, and (iii) the top ratio in the histogram of the regional features. A global score is predicted with respect to the integrated feature group in which all of the third feature group for which the average is calculated and the fourth feature group for which the average of the lower proportions are calculated in the histogram of (iv) period regional features.

와 같이 4 가지의 풀링 함수를 결합한다. 각 통계적 풀링 함수는 패치 특징을 이미지 특징으로 변환(

)한다. The feature pooling layer is

Combine the four pooling functions as Each statistical pooling function converts patch features to image features (

)do.

는 각 풀링 함수에 의해 획득하고, 수학식 4 내지 수학식 7과 같이 표현된다.100-dimensional features are extracted for each patch using the trained model. If a given image consists of N patches, N x 100 feature vectors can be generated from one image. image feature vector

is obtained by each pooling function, and is expressed as in Equations 4 to 7.

Equation 4 represents average pooling, Equation 5 is variance pooling, Equation 6 represents high percentile pooling, and Equation 7 represents low percentile pooling, and an integrated feature vector is obtained using four types of pooling.

l is the feature index.

에서 p 번째 비율의 상한과 하한이다.

는 p 번째 비율의 품질이며, 예컨대

로 설정될 수 있다.n ^{p +} and n ^{p -} are histograms of the patch features

are the upper and lower bounds of the p-th ratio in .

is the quality of the p-th ratio, e.g.

can be set to

는 z()에 의해 통합된다. image feature vector

is integrated by z().

를 예측하기 위해 회귀한다. 풀링된 특징 벡터의 가중치를 미세 조정하기 위해 역 전파 프로세스에 의해 모델이 학습된다. The pooled feature vector is a global subjective score

regress to predict A model is trained by a back-propagation process to fine-tune the weights of the pooled feature vectors.

를 최적화하기 위한 목적 함수는 주관적 점수와 예측된 점수 간의 평균 제곱 오차를 사용할 수 있다.network parameters

The objective function for optimizing α can use the mean squared error between the subjective score and the predicted score.

는 평균 제곱 오차로 정의된 손실 함수이다.

is the loss function defined as the mean squared error.

7 shows simulation results performed according to embodiments of the present invention.

In FIG. 7 , the first row is a distorted image, the second row is a reference image quality map, and the third row is a predicted image quality map. It can be easily seen that the predicted regional quality map is almost similar to the actual SSIM map for the distorted image.

The stereoscopic image quality evaluation apparatus may be implemented in a logic circuit by hardware, firmware, software, or a combination thereof, or may be implemented using a general-purpose or special-purpose computer. The device may be implemented using a hardwired device, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like. In addition, the device may be implemented as a system on chip (SoC) including one or more processors and controllers.

The apparatus for evaluating the image quality of a stereoscopic image may be mounted in a form of software, hardware, or a combination thereof on a computing device or server provided with hardware elements. A computing device or server is all or part of a communication device such as a communication modem for performing communication with various devices or a wired/wireless communication network, a memory for storing data for executing a program, and a microprocessor for executing operations and commands by executing the program It can mean a variety of devices, including

Although it is described that each process is sequentially executed in FIG. 2, this is only an exemplary description, and those skilled in the art change the order described in FIG. 2 without departing from the essential characteristics of the embodiment of the present invention. Alternatively, various modifications and variations may be applied by executing one or more processes in parallel or adding other processes.

The operations according to the present embodiments may be implemented in the form of program instructions that can be performed through various computer means and recorded in a computer-readable medium. Computer-readable medium represents any medium that participates in providing instructions to a processor for execution. Computer-readable media may include program instructions, data files, data structures, or a combination thereof. For example, there may be a magnetic medium, an optical recording medium, a memory, and the like. A computer program may be distributed over a networked computer system so that computer readable code is stored and executed in a distributed manner. Functional programs, codes, and code segments for implementing the present embodiment may be easily inferred by programmers in the technical field to which the present embodiment pertains.

The present embodiments are for explaining the technical idea of the present embodiment, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the equivalent range should be interpreted as being included in the scope of the present embodiment.

Claims (5)

In the method for evaluating the image quality of a stereoscopic image by a computing device,
calculating a local correct score through a reference quality evaluation model for each patch for the original stereoscopic image and the distorted stereoscopic image;
extracting regional features from the distorted stereoscopic image through a patch-based non-referenced quality evaluation model and predicting a regional score based on the regional correct score;
grouping the local features into a plurality of feature groups and predicting a global score for the plurality of feature groups;
A method for evaluating the image quality of a stereoscopic image comprising a. According to claim 1,
The step of calculating the local correct score is,
A stereoscopic image is converted into a left-right composite image (Cyclopean Image), and a reference left-right composite image converted from the original stereoscopic image and a distorted left-right composite image converted from the distorted stereoscopic image are converted into a patch. A method for evaluating the quality of a stereoscopic image, characterized in that it is divided and applied to the reference image quality evaluation model. According to claim 1,
The step of calculating the local correct score is,
A method for evaluating the image quality of a stereoscopic image, characterized in that the local correct score is calculated for each patch by averaging the results obtained by applying the reference quality evaluation model to each pixel of the image. According to claim 1,
Predicting the global score comprises:
(i) a first feature group in which the average of the regional features is calculated, (ii) a second feature group in which the variance is computed with respect to the regional features, (iii) the average of the upper proportions in the histogram of the regional features is calculated Quality evaluation of a stereoscopic image, characterized in that the global score is predicted for the third feature group, and (iv) the integrated feature group that combines all the fourth feature groups that calculate the average of the lower proportions in the histogram of the regional features Way. An apparatus for evaluating the quality of a stereoscopic image comprising one or more processors and a memory for storing one or more programs executed by the one or more processors,
The processor calculates a local correct score through a reference quality evaluation model for each patch for the original stereoscopic image and the distorted stereoscopic image,
The processor extracts regional features through a patch-based non-reference quality evaluation model for the distorted stereoscopic image and predicts a regional score based on the regional correct score,
wherein the processor groups the regional features into a plurality of feature groups and predicts a global score for the plurality of feature groups.

Images