KR102295652B1 - Method and apparatus for measurement of image quality based on deep-learning - Google Patents

Abstract

According to an embodiment of the present invention, a method for measurement of image quality performed by an image quality measurement device comprises the following steps of: passing image data requiring quality measurement through a trained first convolutional neural network; passing visual silence information for the image data through a learned second convolutional neural network; merging an output value of a first convolutional neural network and an output value of the second convolutional neural network; and obtaining a quality measurement result for the image data through a regression operation on a combined output value.

Description

Apparatus and method for measuring video quality based on deep learning

The present invention relates to an apparatus for measuring video quality based on a deep learning technique and a method for measuring the video quality.

A moving picture (video) is generally transmitted after lossy compression due to the vastness of the original image. However, after undergoing such lossy compression, user satisfaction (quality of experience) tends to decrease in general. Therefore, a demand for an objective indicator for the degree of performance degradation is bound to arise, and various performance measurement techniques have been developed, including a scale used to measure the performance of a still image (photograph).

There are two major methods for measuring image quality: objective image quality measurement and subjective image quality measurement. Objective quality measurement is a technique for measuring the degree of image quality degradation without contrast or contrast of the original image based on data. One of the representative measures is PSNR (Peak Signal to Noise Ratio). PSNR is a method of deriving a value through contrast with the original image, but has a disadvantage in that it does not reflect the degree of deterioration subjectively felt by the user due to the characteristic calculated through a simple comparison of pixels between the original and the reconstructed copy. To overcome this point, various performance evaluation methods such as SSIM have been proposed.

Objective quality measurement methods can be roughly divided into three categories: a full reference method that compares the original image with the processed image, a reduced reference method that extracts and compares only some information about the reference image, not the reference image itself, It can be classified as a no-reference method that evaluates only with processed images.

On the other hand, subjective image quality measurement can be seen as the ultimate goal of image quality measurement because the final audience of a video is human, but measuring it requires considerable effort and time, and as the subjectivity of the tester is reflected, it should be used as a general indicator. in a difficult situation.

Korean Patent Laid-Open Publication No. 10-2019-0076288 (published on 02.07.2019)

According to an embodiment of the present invention, in a situation in which subjective image quality measurement is the ultimate goal of image quality measurement, but it is difficult to use it as a general indicator, objective image quality that is similar to the subjective image quality measurement result, that is, mathematically has a large correlation It can be said that the ultimate task is to make the measurement technique under no-standard conditions. Therefore, by learning the image data and visual saliency information in the convolutional neural network, respectively, and using it for video quality measurement, the image quality measurement is performed to know the degree of performance degradation after the video is compressed/restored without comparison with the original image. Provided are a method and apparatus for measuring video quality while still having a great similarity with subjective quality measurement results.

However, the problems to be solved of the present invention are not limited to those mentioned above, and other problems to be solved that are not mentioned can be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. will be.

The video quality measurement method performed by the video quality measurement apparatus according to the first aspect includes the steps of passing image data requiring image quality measurement through a learned first convolutional neural network, and passing visual silence information on the image data to the learned second 2 Passing through the convolutional neural network, merging the output value of the first convolutional neural network and the output value of the second convolutional neural network, and regression operation on the combined output value to the image data and obtaining a quality measurement result for the .

A video quality measuring apparatus according to a second aspect comprises: an input unit for receiving image data requiring image quality measurement and visual silence information for the image data; a processor unit measuring image quality with respect to the image data; an output unit for outputting a quality measurement result for the image data, wherein the processor unit passes the image data through a learned first convolutional neural network and passes the visual silence information through a learned second convolutional neural network, , sums the output value of the first convolutional neural network and the output value of the second convolutional neural network, and obtains the image quality measurement result for the image data through a regression operation on the combined output value.

According to the third aspect, a computer readable recording medium storing a computer program includes instructions for causing the processor to perform the video quality measurement method when the computer program is executed by a processor.

According to a fourth aspect, a computer program stored in a computer-readable recording medium includes instructions for causing the processor to perform the video quality measurement method when the computer program is executed by a processor.

According to an embodiment, the convolutional neural network learns image data and visual silence information, respectively, and uses them to measure video quality, so that after a video is compressed/restored, quality measurement is performed to determine the degree of performance degradation without comparison with the original image. However, to have a large similarity with the subjective image quality measurement result.

1 is a block diagram of an apparatus for measuring video quality according to an embodiment of the present invention.
2 is a flowchart illustrating a video quality measurement method performed by an apparatus for measuring video quality according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

In describing the embodiments of the present invention, if it is determined that a detailed description of a well-known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, the terms to be described later are terms defined in consideration of functions in an embodiment of the present invention, which may vary according to intentions or customs of users and operators. Therefore, the definition should be made based on the content throughout this specification.

In this specification, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as 'comprise' or 'comprise' are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification is present, and one or more other It should be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

In addition, in an embodiment of the present invention, when it is said that a part is connected to another part, this includes not only a direct connection but also an indirect connection through another medium. In addition, the meaning that a certain component includes a certain component does not exclude other components unless otherwise stated, but may further include other components.

1 is a block diagram of an apparatus for measuring video quality according to an embodiment of the present invention.

Referring to FIG. 1 , an apparatus 100 for measuring video quality according to an embodiment includes an input unit 110 , a processor unit 120 , and an output unit 130 , and may further include a storage unit 140 . .

The input unit 110 receives image data requiring image quality measurement and visual saliency information on the image data, and provides it to the processor unit 120 . For example, the input unit 110 may include a communication module capable of receiving image data and visual silence information through a communication network, or an interface capable of directly receiving image data and visual silence information.

The processor unit 120 obtains a quality measurement result for image data requiring quality measurement by using a learned convolutional neural network (CNN). The processor unit 120 passes the image data through the learned first convolutional neural network, passes the visual silence information through the learned second convolutional neural network, and the output value of the first convolutional neural network and the second convolutional neural network. The output values are summed, and a quality measurement result of the image data is obtained through a regression operation on the combined output values. Here, the processor 120 may perform linear convolution before performing a regression operation on the combined output values. In addition, the image data passed through the first convolutional neural network may include the entire pixel area of the image data requiring quality measurement.

The output unit 130 outputs the quality measurement result of the image data by the processor unit 120 . For example, the output unit 130 includes a communication module capable of transmitting image quality measurement result data as a result of processing by the processor unit 120, or includes an interface capable of transmitting image quality measurement result data to another electronic device. can do. In addition, the output unit 130 may include a display device or a printing device capable of outputting the image quality measurement result data by the processor unit 120 to be visually identified.

The storage unit 140 may store image quality measurement result data as a result of processing by the processor unit 120 under the control of the processor unit 120 . For example, the storage unit 140 includes magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and floppy disks. It may be a computer-readable recording medium, such as a magneto-optical medium, a hardware device specially configured to store and execute program instructions, such as a flash memory.

2 is a flowchart illustrating a video quality measurement method performed by an apparatus for measuring video quality according to an embodiment of the present invention.

Referring to FIG. 2 , the method for measuring video quality according to an embodiment includes passing image data requiring quality measurement through a first learned convolutional neural network ( S210 ).

And, the method of measuring video quality according to an embodiment further includes passing the visual silence information on the image data to the learned second convolutional neural network (S220).

In addition, the method for measuring video quality according to an embodiment further includes the step of merging the output value of the first convolutional neural network and the output value of the second convolutional neural network (S230).

In addition, the method of measuring video quality according to an embodiment further includes performing linear convolution on the combined output values ( S240 ).

In addition, the method of measuring video quality according to an embodiment further includes obtaining a quality measurement result for image data through a regression operation on a result of linear convolution ( S250 ).

Hereinafter, a video quality measuring method performed by the video quality measuring apparatus 100 according to an embodiment of the present invention will be described in detail with reference to FIGS. 1 and 2 .

The processor unit 120 of the video quality measuring apparatus 100 according to an embodiment of the present invention measures the image quality of image data requiring image quality measurement, for example, data after image compression/restore, in measuring the image quality of the first convolutional neural network. and a second convolutional neural network. To this end, it is necessary to perform a preceding procedure for learning the first convolutional neural network and the second convolutional neural network. For example, in the training data set for the first convolutional neural network, an input may be image data for training, and a label may be a quality value of the image data for training. In addition, as the training data set for the second convolutional neural network, visual silence information of image data for training may be used, and the label may be a feature value of the corresponding visual silence information. For example, the first convolutional neural network and the second convolutional neural network may be trained in advance by each training data set.

In a state in which pre-learning of the first convolutional neural network and the second convolutional neural network is performed, the input unit 110 of the video quality measuring apparatus 100 receives visual silence information for image data and image data requiring quality measurement. This is provided to the processor unit 120 .

Then, the processor unit 120 of the video quality measuring apparatus 100 passes the image data requiring quality measurement to the pre-trained first convolutional neural network (S210), and the image quality measurement is performed to the pre-trained second convolutional neural network. Visual silence information of the necessary image data is passed (S220). Here, the image data passed through the first convolutional neural network may include the entire pixel area of the image data requiring quality measurement.

Next, the processor unit 120 combines the quality value that is the output value of the first convolutional neural network and the characteristic value that is the output value of the second convolutional neural network ( S230 ).

In addition, the processor 120 may perform linear convolution on the output values combined in step S230 ( S240 ). However, the processor 120 may omit the linear convolution.

Next, the processor unit 120 obtains a quality measurement result of the image data through a regression operation on the result of the linear convolution in step S240 . If the execution of step S240 is omitted, the processor 120 may obtain a quality measurement result for the image data through a regression operation on the output values combined in step S230 ( S250 ).

Thereafter, the storage unit 140 may store the image quality measurement result of the image data by the processor unit 120 under the control of the processor unit 120 . Also, the output unit 130 may output the quality measurement result of the image data by the processor unit 120 under the control of the processor unit 120 .

As described so far, according to the embodiment of the present invention, image data and visual silence information are respectively learned in the convolutional neural network and used for video quality measurement. is performed without comparison with the original image, while having a large similarity to the subjective image quality measurement result.

Combinations of each step in each flowchart attached to the present invention may be performed by computer program instructions. These computer program instructions may be embodied in a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, such that the instructions performed by the processor of the computer or other programmable data processing equipment provide the functions described in each step of the flowchart. It creates a means to do these things. These computer program instructions may also be stored in a computer-usable or computer-readable medium that may direct a computer or other programmable data processing equipment to implement a function in a particular manner, and thus the computer-usable or computer-readable medium. The instructions stored in the recording medium are also possible to produce an article of manufacture including instruction means for performing the functions described in each step of the flowchart. The computer program instructions may also be mounted on a computer or other programmable data processing equipment, such that a series of operational steps are performed on the computer or other programmable data processing equipment to create a computer-executed process to create a computer or other programmable data processing equipment. It is also possible that instructions for performing the processing equipment provide steps for performing the functions described in each step of the flowchart.

Further, each step may represent a module, segment, or portion of code comprising one or more executable instructions for executing the specified logical function(s). It should also be noted that in some alternative embodiments it is also possible for the functions recited in the steps to occur out of order. For example, it is possible that two steps shown one after another may in fact be performed substantially simultaneously, or that the steps may sometimes be performed in the reverse order depending on the function in question.

The above description is merely illustrative of the technical spirit of the present invention, and various modifications and variations will be possible without departing from the essential characteristics of the present invention by those skilled in the art to which the present invention pertains. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

100: video quality measurement device
110: input unit
120: processor unit
130: output unit

Claims (8)

A video quality measurement method performed by a video quality measurement device, comprising:
Passing the image data, which is the object of quality measurement, through the learned first convolutional neural network;
passing visual saliency information for the image data through a learned second convolutional neural network;
merging the output value of the first convolutional neural network and the output value of the second convolutional neural network;
obtaining a quality measurement result for the image data through a regression operation on the combined output values;
In the training data set learned by the first convolutional neural network, an input is image data for training, and a label is a quality value of the image data for training,
In the training data set learned by the second convolutional neural network, the input is visual silence information of the image data for training, and the label is a feature value of the visual silence information of the image data for training.
How to measure video quality. The method of claim 1,
The image data passed through the first convolutional neural network includes the entire pixel area of the image data subject to the image quality measurement.
How to measure video quality. The method of claim 1,
performing linear convolution before performing a regression operation on the combined output values
How to measure video quality. An input unit for receiving image data, which is a target of image quality measurement, and visual saliency information on the image data;
a processor for measuring the image quality of the image data;
and an output unit for outputting a quality measurement result for the image data by the processor unit,
The processor unit,
Passing the image data through the learned first convolutional neural network, passing the visual silence information through the learned second convolutional neural network, combining the output value of the first convolutional neural network and the output value of the second convolutional neural network, , obtaining the image quality measurement result for the image data through a regression operation on the combined output value,
In the training data set learned by the first convolutional neural network, an input is image data for training, and a label is a quality value of the image data for training,
In the training data set learned by the second convolutional neural network, the input is visual silence information of the image data for training, and the label is a feature value of the visual silence information of the image data for training.
Video quality measurement device. 5. The method of claim 4,
The image data passed through the first convolutional neural network includes the entire pixel area of the image data subject to the image quality measurement.
Video quality measurement device. 5. The method of claim 4,
The processor unit,
performing linear convolution before performing a regression operation on the combined output values
Video quality measurement device. As a computer-readable recording medium storing a computer program,
The computer program, when executed by a processor,
Passing the image data, which is the object of quality measurement, through a learned first convolutional neural network, and passing visual saliency information on the image data to a second learned convolutional neural network; Comprising the steps of merging the output value of the convolutional neural network and the output value of the second convolutional neural network, and obtaining a quality measurement result for the image data through a regression operation on the combined output value, the first 1 In the training data set learned by the convolutional neural network, the input is image data for training, the label is the quality value of the image data for training, and the training data set learned by the second convolutional neural network is the input of the image data for training. A computer-readable recording medium comprising instructions for causing the processor to perform a method in which the visual silence information and the label are a characteristic value of the visual silence information of the image data for training. As a computer program stored in a computer-readable recording medium,
The computer program, when executed by a processor,
Passing the image data, which is the object of quality measurement, through a learned first convolutional neural network, and passing visual saliency information on the image data to a second learned convolutional neural network; Comprising the steps of merging the output value of the convolutional neural network and the output value of the second convolutional neural network, and obtaining a quality measurement result for the image data through a regression operation on the combined output value, the first 1 In the training data set learned by the convolutional neural network, the input is image data for training, the label is the quality value of the image data for training, and the training data set learned by the second convolutional neural network is the input of the image data for training. A computer program comprising instructions for causing the processor to perform a method in which the visual silence information and the label is a feature value of the visual silence information of the image data for training.

Images