KR102284096B1 - System and method for estimating subject image quality using visual saliency and a recording medium having computer readable program for executing the method - Google Patents

Abstract

Disclosed are a subjective image quality evaluation system using an importance map, a method, and a recording medium recording a computer readable program for executing the method. The subjective image quality evaluation system includes an image acquisition unit, an image information calculation unit, an importance map calculation unit, an information integration unit, and an image quality evaluation unit. The image acquisition unit acquires a test image, the image information calculation unit acquires preset image information for the test image using machine learning, and the importance map calculation unit acquires an importance map for the test image using machine learning, information The integration unit assigns weights to the image information using the importance map, and the image quality evaluation unit calculates a quality evaluation result of the test image using the weighted image information. According to such a configuration, image quality evaluation is possible only with the test image without the reference image by acquiring the image information and the importance map using machine learning.

Description

An intelligent subjective image quality evaluation system using an importance map, a method, and a recording medium recording a computer readable program for executing the method METHOD}

The present invention relates to a quality evaluation system and method, and more particularly, to a system and method for evaluating image quality.

In general, a test image (T) and a reference image (R) are required for image quality evaluation, and the image quality score of the test image is obtained through a predetermined calculation model. In order to improve the accuracy in the quality evaluation, the degree of perceptual difference between the two images and the quality score can be derived by reflecting the visual saliency. There are two main methods for reflecting the importance map.

The first method calculates the perceptual difference between the two images in advance and then transforms the perceptual difference value according to the importance map to obtain a quality score ( FIG. 1 ). The second method first obtains an importance map, divides the salient region (S) and the background region (B), and then calculates the perceptual difference for each region to obtain a quality score ( FIG. 2 ). 1 and 2 are diagrams schematically illustrating examples of a method of reflecting an importance map for image quality evaluation.

As described above, since the conventional image quality evaluation is performed by comparing a test image and a reference image, there is a problem that a very clean reference image without distortion is absolutely necessary for image quality evaluation.

Also, in the prior art, in order to obtain a perceptual difference or image characteristics between two images in the image quality evaluation process, transformation (eg, discrete wavelet transform, discrete cosine transform) is performed or image information is directly modeled. However, there is a problem in that the performance of the used model or evaluation index varies depending on the type of distortion.

Also, in the image quality evaluation, the bottom-up method based on the stimulus is mainly used for the importance map, but the understanding of the process of feature extraction and importance map calculation of the method is still incomplete and meaningful (semantic). ), there is a problem that it works well only for images without information or objects.

US 101717399 B1

The present invention has been devised to solve the above-mentioned problems of the prior art, and it is possible to evaluate the image quality only with a test image without a reference image, and image quality information independent of the type and degree of distortion and a top-down queue An object of the present invention is to provide a quality evaluation system and method capable of acquiring an importance map that can include both (cue) and bottom-up cues.

In order to achieve the above object, a subjective image quality evaluation system according to the present invention includes an image acquisition unit, an image information calculation unit, an importance map calculation unit, an information integration unit, and an image quality evaluation unit.

The image acquisition unit acquires a test image, the image information calculation unit acquires preset image information for the test image using machine learning, and the importance map calculation unit acquires an importance map for the test image using machine learning, information The integration unit assigns weights to the image information using the importance map, and the image quality evaluation unit calculates a quality evaluation result of the test image using the weighted image information.

According to such a configuration, image quality evaluation is possible only with the test image without the reference image by acquiring the image information and the importance map using machine learning.

In this case, the machine learning may be deep neural network-based learning. According to such a configuration, it is possible to extract image information that can be used regardless of the type and degree of distortion using the deep neural network.

In addition, deep neural network-based learning for calculating the importance map may be learned using images and eye tracking data for images. According to such a configuration, it is possible to obtain a high-performance importance map that can include both the top-down cue and the bottom-up cue.

In addition, deep neural network-based learning for image information calculation can be learned using image information corresponding to an image displayed with image information, and neural network-based learning for image quality evaluation is image quality corresponding to an image to which a subjective image quality evaluation result is given. It can be learned using the evaluation results.

Also, the information integrator may match the size of the image information with the size of the importance map. According to this configuration, even when the size of the image information calculated based on the deep neural network and the importance map do not match, the image quality is evaluated by matching the size of the image information with the importance map.

In this case, the information integrator may divide the image information into a plurality of preset patches, calculate a representative value from the separated patches, and combine the calculated representative values.

Also, the information integrator may assign different weights to the image information according to the reliability of the preset importance map. According to such a configuration, it is possible to analyze the importance of the image information differently for each location according to the performance of the calculated importance map, and the degree of reflection of the importance map can be varied according to the image information.

According to the present invention, image quality evaluation is possible with only a test image without a reference image by acquiring image information and an importance map using machine learning.

Also, it is possible to extract image information that can be used irrespective of the type and degree of distortion by using a deep neural network.

In addition, it is possible to obtain a high-performance importance map that can include both top-down cues and bottom-up cues.

In addition, even when the size of the image information calculated based on the deep neural network and the importance map do not match, the image quality is evaluated by matching the size of the image information with the importance map.

In addition, according to the performance of the calculated importance map, it is possible to analyze by varying the importance of the image information for each location, and the degree of reflection of the importance map can be varied according to the image information.

1 and 2 are diagrams schematically illustrating examples of a method of reflecting an importance map for image quality evaluation.
3 is a schematic block diagram of a subjective image quality evaluation system according to an embodiment of the present invention;
4 is a diagram illustrating the concept of image information acquired using a test image and a deep neural network.
5 is a diagram illustrating an example of a method of resizing image information by dividing image information into overlapping patches and extracting a maximum value.
6 is a diagram illustrating an example of integrating image information and an importance map;
7 is a diagram illustrating an example in which different weights are applied according to types of image information;
8 is a diagram illustrating an example of evaluating image quality only for a part of a test image.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

3 is a schematic block diagram of a subjective image quality evaluation system according to an embodiment of the present invention. In FIG. 3 , the subjective image quality evaluation system 100 includes an image acquisition unit 110 , an image information calculation unit 120 , an importance map calculation unit 130 , an information integration unit 140 , and a picture quality evaluation unit 150 . include

3, the image acquisition unit 110 is an image acquisition device, the image information calculation unit 120 is a deep neural network-based image information acquisition unit, and the importance map calculation unit 130 is a deep neural network-based importance map acquisition unit, a quality evaluation unit ( 150) is implemented as a subjective image quality evaluation result derivation unit, respectively.

In FIG. 3 , each of the components of the subjective image quality evaluation system 100 may be implemented only by hardware, but in general, it will be implemented together with hardware and software operating on the hardware.

The image acquisition unit 110 acquires a test image. To acquire a test image for subjective image quality evaluation. In this case, the test image includes a still image stored in a storage device or a still image obtained from a moving picture being captured in real time.

The image information calculating unit 120 obtains preset image information for the test image by using machine learning. As image information required for image quality evaluation, the type is various, such as image color, brightness, contrast, edge information, and the like.

In this case, the machine learning may be deep neural network-based learning. Deep neural network-based learning can be used not only in the image information calculating unit 120 but also in the importance map calculating unit 130 and the image quality evaluation unit 150 . can be formed with

Deep neural network-based learning for calculating image information may be learned using image information corresponding to an image on which image information is displayed. To this end, the data set used to train the deep neural network must be directly or indirectly labeled for the corresponding information.

For example, a data set in which image quality evaluation results (scores or grades) are displayed, a data set in which subjective image information (eg, bright/not bright) is displayed, or a data set in which image classification (type of object) is displayed, etc. can be used for learning. Alternatively, a deep neural network may be trained through transfer learning.

The acquired image information may be the same as or smaller than the size of the test image, and in the case of a small size, one pixel of the acquired image information contains information about several to tens of pixels (>1 pixel) of the corresponding position in the test image. It can be considered that there is (FIG. 4). 4 is a diagram illustrating the concept of image information acquired using a test image and a deep neural network.

The importance map calculator 130 obtains an importance map for the test image by using machine learning. It is to acquire an importance map that can predict a salient region of an image based on a deep neural network.

Deep neural network-based learning for calculating the importance map can be learned using eye tracking data for images and images. can

The information integrator 140 assigns weights to the image information by using the importance map. Through this, the image information obtained by the deep neural network-based image information obtaining unit 120 and the importance map obtained by the deep neural network-based importance map calculating unit 130 are integrated.

In this case, the information integrator 140 may match the size of the image information with the size of the importance map. If the size of the image information before integration and the size of the importance map are not the same, the resizing method is used to adjust them to the same size. Similarly, the importance map obtained before integration may or may not be the same as the size of the test image, and if not the same, it may be changed to a desired size using the resizing method.

To this end, the information integrator 140 may divide the image information or the importance map into a plurality of preset patches, calculate a representative value from the separated patches, and combine the calculated representative values. More specifically, when the size of image information needs to be reduced, the size can be adjusted by dividing the information into overlapping or non-overlapping patches and selecting the maximum/average value from each patch as a representative value (FIG. 5). 5 is a diagram illustrating an example of a method of resizing image information by dividing image information into overlapping patches and extracting a maximum value.

When the size of the image information and the importance map becomes the same, the two information is integrated in the form of an element-wise product. In this case, the information integrator 140 may assign different weights to the image information according to the reliability of the importance map.

That is, image information may be given more weight according to the reliability of the importance map ( FIG. 6 ). 6 is a diagram illustrating an example of integrating image information and an importance map. In FIG. 6 , it can be seen that the image information values to which the weight w is assigned after the product of the image information and the importance map are summed.

This weight may be different depending on the type of image information (FIG. 7). In addition, weights can be optimized through learning. 7 is a diagram illustrating an example in which different weights are applied according to types of image information.

When the image quality is evaluated for only a part of the test image, image information can be extracted only for the corresponding region, and the importance map is obtained for the entire image, and then only the importance map of the corresponding region is taken and information is integrated (FIG. 8).

8 is a diagram illustrating an example of evaluating image quality only for a part of a test image. In FIG. 8 , an example of correcting the importance map calculated for the entire image using the location information used when image information is segmented and then integrating it with image information calculated from the segmented image is shown.

The image quality evaluation unit 150 calculates the image quality evaluation result of the test image by using the weighted image information. It is to derive the subjective image quality evaluation result using the integrated information.

In this case, neural network-based learning for image quality evaluation may be learned using an image to which a subjective image quality evaluation result is given and a corresponding image quality evaluation result. The integrated information becomes a feature vector and is applied to various machine learning techniques, and through this, a result in the form of a subjective image quality score or grade is derived.

The machine learning techniques used to derive the quality evaluation result may include Linear regression, Logistic regression, Discriminant analysis, Support vector machine, Neural network, Decision tree, Random forest, Clustering, Naive Bayesian classifier, Adaboost, and Ensemble of classifiers. .

In summary, the present invention provides a part for acquiring an image to be evaluated for image quality, a part for acquiring an importance map for the acquired image, and image quality information for the acquired image for subjective image quality evaluation of a still image. It relates to a system including a part, an information integration part for giving weight to image information using the obtained importance map, and a part for deriving a subjective image quality evaluation result using the integrated information.

The importance map acquisition unit and the image quality information acquisition unit are configured in the form of a deep neural network. The importance map acquisition unit is learned using an image and eye tracking data for the corresponding image, the image quality information acquisition unit is learned using the image and the corresponding information on which image quality or image information is labeled, and the subjective image quality evaluation result derivation unit The subjective image quality evaluation result is learned using the given image and the corresponding result.

The present invention is a system capable of subjective image quality evaluation using only a test image requiring evaluation, unlike the prior art in which a reference image is required. It is possible to evaluate the image quality by extracting the importance map and image quality information only from the test image.

Unlike the prior art that uses hand-crafted features to obtain the importance map and quality information, the importance map and features that can be used regardless of the type and degree of distortion are extracted using a deep neural network.

Also, in the prior art, a quality score could be derived only when there is a perceptual difference value between a reference image and a test image, but in the present invention, image quality evaluation can be performed by acquiring information on the test image itself.

In particular, the information integration unit can analyze the importance of the image information differently for each location according to the performance of the importance map, and the degree of reflection of the importance map can be varied according to the image information.

Although the present invention has been described with reference to some preferred embodiments, the scope of the present invention should not be limited thereto, but should also extend to modifications or improvements of the above embodiments supported by the claims.

100: subjective image quality evaluation system
110: image acquisition unit
120: image information calculating unit
130: importance map calculation unit
140: information integration unit
150: quality evaluation unit

Claims (17)

an image acquisition unit for acquiring a test image;
an image information calculator for obtaining preset image information for the test image by using machine learning;
an importance map calculation unit for obtaining an importance map for the test image by using machine learning;
an information integrator for assigning weights to the image information using the importance map; and
A subjective image quality evaluation system including a quality evaluation unit for calculating a quality evaluation result of the test image by using the weighted image information,
The information integrator matches the size of the image information with the size of the importance map, and adds the weighted image information to a result of multiplying the image information and the importance map in the form of an element-wise product,
The machine learning is deep neural network-based learning,
Neural network-based learning for calculating the importance map is learned using an image and eye tracking data for the image,
The neural network-based learning for the image quality evaluation is a subjective image quality evaluation system, characterized in that it is learned using the image to which the subjective image quality evaluation result is given and the image quality evaluation result corresponding to the image.
delete delete The method according to claim 1,
The neural network-based learning for calculating the image information is a subjective image quality evaluation system, characterized in that it is learned using image information corresponding to an image on which image information is displayed.
delete delete 5. The method according to claim 4,
and the information integrator divides the image information into a plurality of preset patches, calculates a representative value from the separated patches, and combines the calculated representative values.
8. The method of claim 7,
and the information integrator assigns different weights to the image information according to the reliability of the importance map.
A method for evaluating the image quality of a subjective image quality evaluation system, comprising:
an image acquisition step of acquiring a test image;
an image information calculation step of obtaining preset image information for the test image using machine learning;
an importance map calculation step of obtaining an importance map for the test image by using machine learning;
an information integration step of assigning weights to the image information using the importance map; and
A subjective image quality evaluation method comprising: an image quality evaluation step of calculating a quality evaluation result of the test image using the weighted image information;
The information integration step matches the size of the image information with the size of the importance map, and adds the weighted image information to a result of multiplying the image information and the importance map in the form of an element-wise product,
The machine learning is deep neural network-based learning,
Neural network-based learning for calculating the importance map is learned using an image and eye tracking data for the image,
The neural network-based learning for the image quality evaluation is a subjective image quality evaluation method, characterized in that it is learned using the image to which the subjective image quality evaluation result is given and the image quality evaluation result corresponding to the image.
delete delete 10. The method of claim 9,
The neural network-based learning for calculating the image information is a subjective image quality evaluation method, characterized in that learning is performed using image information corresponding to an image on which image information is displayed.
delete delete 13. The method of claim 12,
The information integration step divides the image information into a plurality of preset patches, calculates a representative value from the separated patches, and combines the calculated representative values.
16. The method of claim 15,
In the information integration step, different weights are given to the image information according to the reliability of the importance map.
Claim 9, claim 12, claim 15, any one of claims 16 for executing the method of any one of the computer-readable recording medium recorded.

Images