KR20200064930A - Method and apparatus for measuring video quality based on detection of difference of perceptual sensitivity regions - Google Patents

Abstract

Disclosed are a method for measuring video quality based on cognition sensitive region and a device thereof. The video quality can be measured based on the cognition sensitive region and change in the cognition sensitive region. The cognition sensitive region includes a spatial cognition sensitive region, a temporal cognition sensitive region, and a spatio-temporal cognition sensitive region. A cognitive weighting can be applied to the detected cognition sensitive region and the detected change in the cognition sensitive region. Based on the cognition sensitive region and the change in the cognition sensitive region, distortion is calculated and a result of measuring the image quality of a video is generated in accordance with the calculated distortion.

Description

METHOD AND APPARATUS FOR MEASURING VIDEO QUALITY BASED ON DETECTION OF DIFFERENCE OF PERCEPTUAL SENSITIVITY REGIONS}

The following embodiments relate to a method and apparatus for video quality measurement, and more particularly, to a video quality measurement method and apparatus based on detection of a change in a cognitive sensitive region.

In the field of video processing or video compression, an indicator indicating the quality of a video is required for verification of performance of a technology to be developed or optimization within a technology.

The better the indicator reflects subjective image quality performance, the better the technology can provide.

Subjective video quality measurement is a method of obtaining quality measurement values for video through statistical processing of individual quality evaluation values obtained through quality evaluation experiments involving multiple evaluators. The visual and economic costs of these subjective video quality measurements are relatively high.

The automatic video quality measurement method can measure image quality at a relatively low visual cost and economic cost. The purpose of the automatic video quality measurement method is to replace subjective image quality evaluation.

For this replacement, the automatic video quality measurement method must provide high measurement reliability. Typically, measurement reliability may be evaluated through Pearson's Correlation Coefficient (PCC) or Spearman's Rank Correlation Coefficient (SRCC).

One embodiment may provide a method and apparatus for measuring the image quality of a video based on a cognitive sensitive area.

One embodiment may provide a method and apparatus for measuring a video quality based on a change in a cognitive sensitive region.

One embodiment may provide a method and apparatus for deriving feature information of an image using cognitive weights.

In one side, the communication unit for receiving the reference video and the comparison video; And a processing unit that generates a result of measuring image quality for the comparison video, wherein the processing unit is characterized by the reference video feature information, the comparison video characteristic information, the reference video recognition sensitive area, and the comparison video recognition sensitive area. An image processing apparatus is provided that calculates a distortion using a change between the cognitive sensitive region of the reference video and the cognitive sensitive region of the comparison video, and generates the result of the image quality measurement based on the distortion.

The feature information of the reference video may include one or more of spatial feature information of the reference video, temporal feature information of the reference video, and spatial and temporal feature information of the reference video.

The feature information of the comparison video may include at least one of spatial feature information of the comparison video, temporal feature information of the comparison video, and spatiotemporal feature information of the comparison video.

The processing unit may extract the spatiotemporal feature information from the spatiotemporal slice of the input video.

The input video may be one or more of the reference video and the comparison video.

The space-time slice may be a structure in which a group of pictures (GOP) of the input video is spatially divided.

The processing unit may detect horizontal and vertical features of the spatial feature information from an image of the input video.

The processing unit may detect a characteristic for a direction other than a horizontal direction and a vertical direction from the image.

The processing unit may derive a region having high cognitive sensitivity from the image using an edge detection method.

The edge detection method may be a Sobel operation.

The horizontal feature and the vertical feature may be horizontal-vertical edge maps.

The characteristics for the directions other than the horizontal direction and the vertical direction may be edge maps in which information on horizontal edges and vertical edges is excluded.

The processing unit may update the horizontal-vertical edge map using a first cognitive weight.

The processor may update an edge map in which information on the horizontal edge and the vertical edge is excluded using the second cognitive weight.

The first cognitive weight and the second cognitive weight may reflect changes in the spatial cognitive sensitive area.

The cognitive sensitive region of the reference video may include one or more of a spatial cognitive sensitive region of the reference video, a temporal cognitive sensitive region of the reference video, and a spatial and temporal cognitive sensitive region of the reference video.

The cognitive sensitive region of the comparison video may include one or more of a spatial cognitive sensitive region of the comparison video, a temporal cognitive sensitive region of the comparison video, and a spatiotemporal cognitive sensitive region of the comparison video.

The processor may generate a spatial complexity map by calculating spatial complexity of pixels or first blocks of an image of an input video.

The input video may be the reference video or the comparison video.

The processor may generate a flatness map by calculating background flatnesses of the second blocks of the image of the input video.

The change may be a difference between the cognitive sensitive region of the reference video and the cognitive sensitive region of the comparison video.

The distortion may include one or more of spatial distortion, temporal distortion, and spatial and temporal distortion.

The processing unit may extract representative spatial information of the reference video by using spatial feature information of the reference video, spatial cognitive sensitive areas and spatial cognitive sensitive areas of the reference video.

The processing unit may extract representative spatial information of the comparison video by using the spatial characteristic information of the comparison video, the spatial cognitive sensitive area and the spatial cognitive sensitive area of the comparison video.

The processing unit may calculate the spatial distortion.

The spatial distortion may be a difference between representative spatial information of the reference video and representative spatial information of the comparison video.

The change of the spatial cognitive sensitive region may be a change between the spatial cognitive sensitive region of the reference video and the spatial cognitive sensitive region of the comparison video.

The processing unit may generate combined spatial information of the reference video by combining the spatial characteristic information of the reference video, the spatial cognitive sensitive area of the reference video, and the changes of the spatial cognitive sensitive area.

The processor may extract representative spatial information of each image of the reference video from the combined spatial information.

The processing unit may extract representative spatial information of the reference video from representative spatial information of each image of the reference video.

The processing unit may extract representative temporal information of the reference video by using temporal characteristic information of the reference video, temporal cognitive sensitive areas and temporal cognitive sensitive areas of the reference video.

The processing unit may extract representative temporal information of the comparison video by using temporal characteristic information of the comparison video, temporal cognitive sensitive area and temporal cognitive sensitive area of the comparative video.

The processing unit may calculate the temporal distortion.

The temporal distortion may be a difference between representative temporal information of the reference video and representative temporal information of the comparison video.

The change in the temporal cognitive sensitive region may be a change between the temporal cognitive sensitive region of the reference video and the temporal cognitive sensitive region of the comparison video.

The processor may generate combined temporal information of the reference video by combining temporal feature information of the reference video, temporal cognitive sensitive area of the reference video, and changes in the temporal cognitive sensitive area.

The processing unit may extract representative temporal information of each image of the reference video from the combined temporal information.

The processing unit may extract representative temporal information of the reference video from representative temporal information of each image of the reference video.

In another aspect, extracting feature information of the reference video; Detecting a cognitive sensitive region of the reference video: extracting feature information of the comparison video; Detecting a cognitive sensitive region of the comparison video; Calculating a change between a cognitive sensitive region of the reference video and a cognitive sensitive region of the comparison video; Calculating distortion using feature information of the reference video, feature information of the comparison video, a cognitive sensitive region of the reference video, a cognitive sensitive region of the comparison video, and the change; And generating a result of image quality measurement based on the distortion.

In another aspect, a computer readable recording medium containing a program for processing the image processing method is provided.

In another aspect, the communication unit for receiving a reference video and a comparison video; And a processing unit for driving a deep neural network for measuring cognitive image quality, wherein the processing unit detects a cognitive sensitive region of the reference video and a cognitive sensitive region of the comparison video, and between a cognitive sensitive region of the reference video and a cognitive sensitive region of the comparative video. Calculate the change, the cognitive sensitive area of the reference video, the cognitive sensitive area of the comparison video and the change are input into the cognitive quality measurement deep neural network, and the cognitive quality measurement deep neural network is the cognitive sensitive area of the reference video, the An image processing apparatus for generating a result of image quality measurement using the cognitive sensitive area of the comparison video and the change is provided.

A method and apparatus for measuring a video quality based on a cognitive sensitive area is provided.

A method and apparatus for measuring a video quality based on a change in a cognitive sensitive region is provided.

A method and apparatus for deriving feature information of an image using cognitive weights are provided.

1 shows an image processing apparatus according to an embodiment.
2 shows a configuration for measuring image quality according to an embodiment.
3 is a flowchart of a method for measuring image quality according to an embodiment.
4 shows a structure of a spatial/temporal feature information extraction unit according to an example.
5 is a flowchart of a method for extracting spatial/temporal feature information according to an example.
6 shows a structure of a spatial/temporal cognitive sensitive region detection unit according to an example.
7 is a flowchart of a method for detecting a spatial/temporal cognitive sensitive area according to an example.
8 shows the structure of the spatial/temporal cognitive sensitive area change calculator according to an example.
9 is a flowchart of a method for calculating spatial/temporal cognitive sensitive area change according to an example.
10 shows a structure of a distortion calculation unit according to an example.
11 is a flowchart of a distortion calculation method according to an example.
12 shows a structure of a spatial distortion calculator according to an example.
13 is a flowchart of spatial distortion calculation according to an example.
14 shows a structure of a spatial feature information extracting unit according to an example.
15 is a flowchart of spatial feature information extraction according to an example.
16 shows a structure of a spatial cognitive sensitive region detection unit according to an example.
17 is a flowchart of spatial cognitive sensitive area detection according to an example.
18 shows a measurement of spatial complexity of an image according to an example.
19 shows the structure of a temporal distortion calculation unit according to an example.
20 is a flowchart of temporal distortion calculation according to an example.
21 shows a structure of a temporal feature information extracting unit according to an example.
22 is a flowchart of temporal feature information extraction according to an example.
23 shows a structure of a temporal cognitive sensitive region detection unit according to an example.
24 is a flowchart of temporal cognitive sensitive area detection according to an example.
25 illustrates a configuration for measuring image quality using a deep neural network for measuring cognitive image quality according to an embodiment.
26 is a flowchart of a method for measuring image quality using a deep neural network for measuring cognitive image quality according to an embodiment.

For detailed description of exemplary embodiments described below, reference is made to the accompanying drawings that illustrate specific embodiments as examples. These embodiments are described in detail enough to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments are different, but need not be mutually exclusive. For example, certain shapes, structures, and properties described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in relation to one embodiment. In addition, it should be understood that the location or placement of individual components within each disclosed embodiment can be changed without departing from the spirit and scope of the embodiment. Therefore, the following detailed description is not intended to be taken in a limiting sense, and the scope of exemplary embodiments, if appropriately described, is limited only by the appended claims, along with all ranges equivalent to those claimed.

In the drawings, similar reference numerals refer to the same or similar functions throughout several aspects. The shape and size of elements in the drawings may be exaggerated for a clearer explanation.

The terms used in the examples are for describing the examples and are not intended to limit the present invention. In an embodiment, the singular form also includes the plural form unless specifically stated in the text. As used herein, "comprises" and/or "comprising" refers to the components, steps, operations and/or elements mentioned above, the presence of one or more other components, steps, operations and/or elements. Or it does not exclude addition, it means that additional configurations may be included in the scope of the technical spirit of the exemplary embodiments or the exemplary embodiments. When a component is said to be "connected" or "connected" to another component, the two components may be directly connected to or connected to each other, but the above 2 It should be understood that other components may exist in the middle of the dog components.

Terms such as first and second may be used to describe various elements, but the above elements should not be limited by the above terms. The above terms are used to distinguish one component from another component. For example, the first component may be referred to as a second component without departing from the scope of rights, and similarly, the second component may also be referred to as a first component.

In addition, the components shown in the embodiments are shown independently to indicate different characteristic functions, and do not mean that each component is composed of only separate hardware or one software configuration unit. That is, each component is listed as each component for convenience of description. For example, at least two of the components may be combined into one component. Also, one component may be divided into a plurality of components. The consolidated and separate embodiments of each of these components are also included in the scope of the claims, without departing from the essence.

Also, some of the components are not essential components for performing essential functions, but may be optional components for improving performance. Embodiments may be implemented including only components necessary to implement the essence of the embodiments, and structures in which optional components are excluded, such as components used to improve performance, are also included in the scope of rights.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings in order to enable those skilled in the art to easily implement the embodiments. In describing the embodiments, when it is determined that a detailed description of a related known configuration or function may obscure the subject matter of the present specification, the detailed description will be omitted.

Existing Full Reference (FR) quality indicators such as Mean Squared Error (MSE), Peak Signal to Noise Ratio (PSNR) or Structural SIMilarity (SSIM) are standard As a method of measuring the difference between a reference video and a comparison video, it can have an advantage of being intuitive.

However, there is a case where the difference between the measured value and the perceptual quality by the full reference picture quality index is large, and thus there is a disadvantage that the measurement reliability of the full reference picture quality index is low in comparison between images of different characteristics. Can be.

Furthermore, when methods using the full reference quality index are applied to a video, a result value of the image is generated through measurement of each image of the video, and an average of the result values of the images can be a result value of the video. have. Therefore, when the full reference picture quality index is applied to a video, measurement reliability of the full reference picture quality index may be further deteriorated.

Several automatic video quality measurement methods have been proposed, and Video Quality Model-Variable Frame Delays (VQM-VFD) have also been adopted as international standards. However, these methods are also not widely used because they are not sufficiently reliable.

The automatic video quality measurement methods as described above do not effectively utilize visual cognitive characteristics. Therefore, the performance of the automatic video quality measurement method can be greatly improved by effectively utilizing the visual recognition characteristics.

1 shows an image processing apparatus according to an embodiment.

The video processing apparatus 100 may include at least a part of the processing unit 110, the communication unit 120, the memory 130, the storage 140, and the bus 190. Components of the video processing apparatus 100, such as the processing unit 110, the communication unit 120, the memory 130, and the storage 140, may communicate with each other through the bus 190.

The processing unit 110 may be a semiconductor device that executes processing instructions stored in the memory 130 or the storage 140. For example, the processing unit 110 may be at least one hardware processor.

The processing unit 110 may process a task required for the operation of the video processing apparatus 100. The processing unit 110 may execute (execute) the code of the operation or step of the processing unit 110 described in the embodiments.

The processor 110 may generate, store, and output information to be described in the embodiments to be described later, and may perform operations of steps performed in other video processing devices 100.

The communication unit 120 may be connected to the network 199. Data or information required for the operation of the video processing apparatus 100 may be received, and data or information required for the operation of the video processing apparatus 100 may be transmitted. The communication unit 120 may transmit data to other devices through the network 199 and receive data from other devices. For example, the communication unit 120 may be a network chip or a port.

The memory 130 and the storage 140 may be various types of volatile or nonvolatile storage media. For example, the memory 130 may include at least one of a ROM (ROM) 131 and a RAM (RAM) 132. The storage 140 may include a built-in storage medium such as RAM, flash memory, and hard disk, and may include a removable storage medium such as a memory card.

The function or operation of the video processing apparatus 100 may be performed as the processing unit 110 executes at least one program module. The memory 130 and/or the storage 140 may store at least one program module. At least one program module may be configured to be executed by the processing unit 110.

At least some of the components of the video processing apparatus 100 described above may be at least one program module.

For example, units and neural networks to be described later may be program modules executed by the processing unit 110.

The program modules may be included in the video processing apparatus 100 in the form of an operating system, an application module, a library, and other program modules, and may be physically stored on various known storage devices. . Also, at least some of these program modules may be stored in a remote storage device capable of communicating with the video processing device 100. On the other hand, these program modules are routines, subroutines, programs, objects that perform specific operations or tasks according to embodiments or execute specific abstract data types (object), component (component) and data structure (data structure), and the like, but may not be limited to these.

The video processing apparatus 100 may further include a user interface (UI) input device 150 and a UI output device 160. The UI input device 150 may receive a user input required for the operation of the video processing apparatus 100. The UI output device 160 may output information or data according to the operation of the video processing apparatus 100.

The video processing apparatus 100 may further include a sensor 170. The sensor 170 may generate a video by continuously capturing an image.

Hereinafter, the terms "image" and the term "frame" may be used in the same sense, and may be replaced with each other.

2 shows a configuration for measuring image quality according to an embodiment.

The processing unit 110 includes a spatial/temporal feature information extraction unit 210, a spatial/temporal cognitive sensitive area detection unit 220, a spatial/temporal cognitive sensitive area change calculation unit 230, a distortion calculation unit 240, and an index calculation unit 250.

The image processing apparatus 100 may automatically measure the perceived image quality of the video. The image processing apparatus 100 may detect a cognitive sensitivity region in a video in order to provide high measurement reliability in measuring the cognitive image quality of the video, and utilize the detected cognitive sensitivity region for image quality measurement Can be. Hereinafter, image quality may mean cognitive image quality.

In the subjective picture quality evaluation measurement, each evaluator can evaluate the picture quality of the reference video and the picture quality of the comparative video, respectively. The difference between the evaluation value given by each evaluator to the quality of the reference video and the evaluation value given to the quality of the comparison video may be derived. The mean opinion score (Mean Opinion Score; MOS) may be an average value of differences of evaluation values by evaluators.

At this time, each evaluator may consider the image quality of the cognitively sensitive region relatively in evaluating the reference video, and may further consider the image quality of the cognitively sensitive region in evaluating the comparative video.

Generally, the cognitively sensitive region in the reference video and the cognitively sensitive region in the comparison video may be different.

In addition, when evaluating image quality with respect to a pair of a reference video and an evaluation video, such as a double stimulus continuous quality scale (DSCQS), a change in a sensitive region in both videos may be further considered. In order to consider these cognitive characteristics, in an embodiment, the spatial/temporal cognitive sensitive region detection unit 220 and the spatial/temporal cognitive sensitive region change calculation unit 230 may be introduced. Higher measurement reliability can be provided through image quality measurement using the spatial/temporal cognitive sensitive region detection unit 220 and the spatial/temporal cognitive sensitive region change calculation unit 230.

The functions of the spatial/temporal feature information extraction unit 210, the spatial/temporal cognitive sensitive region detection unit 220, the spatial/temporal cognitive sensitive region change calculation unit 230, the distortion calculation unit 240, and the index calculation unit 250 And the operation is described in detail below.

In embodiments, the term "spatial/temporal" may be "spatial, temporal and spatio-temporal" or "spatial, temporal and spatio-temporal" At least one.

In addition, spatial and temporal features in embodiments may be combined as spatial and temporal features.

Further, in embodiments, "spatial information" may be for an image of a video. In other words, "spatial information" may be related to an image of a video rather than the entire video. In an embodiment, "video" may be replaced with "video". For example, "reference video" may be replaced with "reference video of reference video". "Comparative video" may be replaced with "Comparative video of comparative video".

3 is a flowchart of a method for measuring image quality according to an embodiment.

Prior to step 310, the communication unit 120 may receive a reference video and a comparison video. Alternatively, the communication unit 120 may receive one of the reference video and the comparison video, and the sensor 170 may photograph the other.

In step 310, the spatial/temporal feature information extractor 210 may extract spatial/temporal feature information of a reference video.

The spatial/temporal characteristic information may mean one or more of spatial characteristic information, temporal characteristic information, and spatio-temporal characteristic information. In addition, spatial/temporal feature information may be abbreviated as feature information.

In step 320, the spatial/temporal cognitive sensitive region detection unit 220 may detect the spatial/temporal cognitive sensitive region of the reference video.

The spatial/temporal cognitive sensitive region may mean one or more of the spatial cognitive sensitive region, the temporal cognitive sensitive region and the spatiotemporal cognitive sensitive region. In addition, the spatial/temporal cognitive sensitive area can be abbreviated as a cognitive sensitive area.

The spatial/temporal cognitive sensitive region detection unit 220 may distinguish a spatially/temporally less sensitive region and a spatially/temporally sensitive region from the reference video.

For example, a region that is less spatially and temporally sensitive may mean a region having a large spatial/temporal masking effect.

The spatial/temporal masking effect may mean one or more of a spatial masking effect, a temporal masking effect, and a spatiotemporal masking effect. In addition, the spatial/temporal masking effect can be abbreviated as the masking effect.

In other words, the spatial/temporal cognitive sensitive region detector 220 may measure the sensitivity of each region with respect to regions of the reference video.

In step 330, the spatial/temporal feature information extraction unit 210 may extract spatial/temporal feature information of the comparison video.

In operation 340, the spatial/temporal cognitive sensitive region detection unit 220 may detect the spatial/temporal cognitive sensitive region of the comparison video.

The spatial/temporal cognitive sensitive region detection unit 220 may distinguish between spatially/temporally less sensitive regions and spatial/temporally sensitive regions in the comparison video.

In other words, the spatial/temporal cognitive sensitive region detector 220 may measure the sensitivity of each region with respect to regions of the comparison video.

In step 350, the spatial/temporal cognitive sensitive region change calculator 230 may calculate a change between the spatial/temporal cognitive sensitive region of the reference video and the spatial/temporal cognitive sensitive region of the comparison video.

Here, the terms "change" and the term "difference" may be used interchangeably with each other and may be interchanged with each other. In other words, the change may mean a difference between the spatial/temporal cognitive sensitive region of the reference video and the spatial/temporal cognitive sensitive region of the comparison video. Also, the change may mean a difference between cognitive sensitive regions corresponding to each other of the reference video and the comparison video.

In addition, "the change between the spatial/temporal cognitive sensitive area of the reference video and the spatial/temporal cognitive sensitive area of the reference video" is "the change in the spatial/temporal cognitive sensitive area of the reference video and the spatial/temporal cognitive sensitive area of the reference video" It can also be interpreted as

The spatial/temporal cognitive sensitive region change calculator 230 may detect a region in which the sensitivity has changed.

The region where the sensitivity is changed may be 1) a region having high sensitivity in the reference video, a region having low sensitivity in the comparison video, and 2) a region having low sensitivity in the reference video, and a region having high sensitivity in the comparison video.

Alternatively, if the sensitivity of the region in the reference video and the sensitivity of the region in the comparison video are different, the sensitivity of the region may be considered to have changed.

Here, a region having high sensitivity in the reference video and a low sensitivity in the comparison video may be a region in which cognitively important information is lost.

In addition, a region having low sensitivity in the reference video and a high sensitivity in the comparison video may be a region in which cognitively important information is generated. A representative example of this area is blocking artifacts that occur in compressed video when the video is compressed at a high compression rate.

In other words, the spatial/temporal cognitive sensitive region change calculator 230 may detect a region in which cognitively important information is lost and a region in which cognitively important information is generated.

In step 360, the distortion calculation unit 240 results from the spatial/temporal feature information extraction unit 210, results from the spatial/temporal cognitive sensitive area detection unit 220, and spatial/temporal cognitive sensitive area change calculation unit Spatial/temporal distortion can be calculated using the results from (230).

Here, the result from the spatial/temporal feature information extraction unit 210 may be spatial/temporal feature information of the reference video and spatial/temporal feature information of the comparison video.

The result from the spatial/temporal cognitive sensitive region detection unit 220 may be a spatial/temporal cognitive sensitive region of the reference video and a spatial/temporal cognitive sensitive region of the comparison video.

The result from the spatial/temporal cognitive sensitive region change calculation unit 230 may be a change between the spatial/temporal cognitive sensitive region of the reference video and the spatial/temporal cognitive sensitive region of the comparison video.

Spatial/temporal distortion may mean one or more of spatial distortion, temporal distortion, and spatial and temporal distortion. Also, spatial/temporal distortion can be abbreviated as distortion.

In step 370, the index calculator 250 may generate a result of measuring the quality of the comparison video based on the distortion that is the result from the distortion calculator 240. For example, the index calculator 250 may convert the distortion resulting from the distortion calculator 240 into a result of image quality measurement.

Here, the result from the distortion calculator 240 may be spatial/temporal distortion.

4 shows a structure of a spatial/temporal feature information extraction unit according to an example.

The spatial/temporal feature information extractor 210 may include a spatial feature information extractor 410, a temporal feature information extractor 420 and a spatiotemporal feature information extractor 430.

2 and 3, it has been described that one spatial/temporal feature information extractor 210 extracts spatial/temporal feature information from a reference video and a comparative video. On the other hand, two spatial/temporal feature information extractors may be configured for the reference video and the comparison video, respectively.

In an embodiment, a case in which one spatial/temporal feature information extraction unit 210 is used for a reference video and a comparison video is described.

The functions and operations of the spatial feature information extractor 410, the temporal feature information extractor 420, and the spatiotemporal feature information extractor 430 will be described in detail below.

5 is a flowchart of a method for extracting spatial/temporal feature information according to an example.

The step 310 described above with reference to FIG. 3 may include steps 510, 520 and 530. At this time, the input video may be a reference video.

Step 330 described above with reference to FIG. 3 may include steps 510, 520 and 530. At this time, the input video may be a comparison video.

In step 510, the spatial feature information extractor 410 may extract spatial feature information from each image of the input video. The spatial feature information extractor 410 may extract information for each image of the input video.

In operation 520, the temporal characteristic information extraction unit 420 may extract temporal characteristic information from a plurality of images of the input video. The plurality of images may be a group of pictures (GOP) or a group of features. The spatial feature information extractor 410 may extract information in units of a plurality of images of the input video.

In step 530, the spatiotemporal feature information extractor 430 may extract the spatiotemporal feature information from the spatio-temporal slice of the input video. The space-time slice may be a structure in which the GOP is spatially divided. The spatiotemporal feature information extraction unit 430 may extract information in units of the spatiotemporal slice of the input video.

Alternatively, the spatiotemporal feature information extractor 430 may be configured through a combination of the spatial feature information extractor 410 and the temporal feature information extractor 420.

6 shows a structure of a spatial/temporal cognitive sensitive region detection unit according to an example.

The spatial/temporal cognitive sensitive region detector 220 may include a spatial cognitive sensitive region detector 610, a temporal cognitive sensitive region detector 620, and a spatiotemporal cognitive sensitive region detector 630.

2 and 3, it has been described that one spatial/temporal cognitive sensitive region detection unit 220 extracts spatial/temporal feature information from a reference video and a comparison video. On the other hand, two spatial/temporal cognitive sensitive region detection units may be configured for reference video and comparative video, respectively.

In an embodiment, a case in which one spatial/temporal cognitive sensitive region detection unit 220 is used for the reference video and the comparative video is described.

In the subjective image quality evaluation, each evaluator can evaluate the quality of the reference video and the quality of the comparison video, respectively. At this time, the evaluator may consider the cognitively sensitive region in the video relatively in evaluating the quality of each video. In order to use this characteristic, the spatial/temporal cognitive sensitive region detection unit 220 may detect cognitively sensitive regions in the reference video and the comparative video, respectively.

The functions and operations of the spatial cognitive sensitive region detector 610, the temporal cognitive sensitive region detector 620, and the spatio-temporal cognitive sensitive region detector 630 will be described in detail below.

7 is a flowchart of a method for detecting a spatial/temporal cognitive sensitive area according to an example.

The step 320 described above with reference to FIG. 3 may include steps 710, 720 and 730. At this time, the input video may be a reference video.

Step 340 described above with reference to FIG. 3 may include steps 710, 720 and 730. At this time, the input video may be a comparison video.

In operation 710, the spatial cognitive sensitive region detector 610 may detect a spatial cognitive sensitive region from an image of an input video. The spatial cognitive sensitive region detection unit 610 may extract information for each image of the input video.

In operation 720, the temporal cognitive sensitive region detector 620 may detect the temporal cognitive sensitive region from a plurality of images of the input video. The plurality of images may be a group of pictures (GOP) or a group of features. The temporal cognitive sensitive region detection unit 620 may extract information in units of a plurality of images of the input video.

In operation 730, the spatiotemporal cognitive sensitive region detector 630 may detect the spatiotemporal cognitive sensitive region from the spatio-temporal slice of the input video. The space-time slice may be a structure in which the GOP is spatially divided. The space-time cognitive sensitive region detector 630 may extract information in units of space-time slices of the input video.

Alternatively, the spatio-temporal cognitive sensitive region detector 630 may be configured through a combination of the spatial cognitive sensitive region detector 610 and the temporal cognitive sensitive region detector 620.

8 shows the structure of the spatial/temporal cognitive sensitive area change calculator according to an example.

The spatial/temporal cognitive sensitive region change calculator 230 may include a spatial cognitive sensitive region change calculator 810, a temporal cognitive sensitive region change calculator 820, and a spatiotemporal cognitive sensitive region change calculator 830. .

In the subjective picture quality evaluation, which evaluates picture quality on a pair of reference video and evaluation video, such as DSCQS, the evaluator can evaluate picture quality by relatively considering changes in a cognitively sensitive region in two videos. In order to use these characteristics, the spatial/temporal cognitive sensitive area change calculator 230 may detect a change such as a difference between the cognitively sensitive area of the reference video and the cognitively sensitive area of the comparison video.

The functions and operations of the spatial cognitive sensitive region change calculator 810, the temporal cognitive sensitive region change calculator 820, and the spatiotemporal cognitive sensitive region change calculator 830 are described in detail below.

9 is a flowchart of a method for calculating spatial/temporal cognitive sensitive area change according to an example.

The step 350 described above with reference to FIG. 3 may include steps 910, 920 and 930.

In operation 910, the spatial cognitive sensitive region change calculator 810 may calculate a change in the spatial cognitive sensitive region from images of the reference video and the comparative video. The spatial cognitive sensitive region change calculator 810 may extract information for each image of the reference video and the comparison video.

In operation 920, the temporal cognitive sensitive region change calculator 820 may calculate a change in the temporal cognitive sensitive region from a plurality of images of the reference video and the comparison video. The plurality of images may be a group of pictures (GOP) or a group of features. The temporal cognitive sensitive region change calculator 820 may extract information in units of a plurality of images of a reference video and a comparison video.

In operation 930, the spatial and temporal cognitive sensitive region change calculator 830 may calculate the spatial and temporal cognitive sensitive region change from the spatio-temporal slice of the reference video and the comparative video. The space-time slice may be a structure in which the GOP is spatially divided. The spatial and temporal cognitive sensitive region change calculator 830 may extract information in units of space-time slices of the reference video and the comparison video.

Alternatively, the spatiotemporal cognitive sensitive region change calculator 830 may be configured through a combination of the spatial cognitive sensitive region change calculator 810 and the temporal cognitive sensitive region change calculator 820.

10 shows a structure of a distortion calculation unit according to an example.

The distortion calculation unit 240 may include a spatial distortion calculation unit 1010 and a temporal distortion calculation unit 1020.

The functions and operations of the spatial distortion calculator 1010 and the temporal distortion calculator 1020 will be described in detail below.

11 is a flowchart of a distortion calculation method according to an example.

In step 1110, the spatial distortion calculation unit 1010 calculates the result from the spatial/temporal feature information extraction unit 210, the result from the spatial/temporal cognitive sensitive area detection unit 220, and the spatial/temporal cognitive sensitive area change calculation The spatial distortion can be calculated using the result from the unit 230.

In step 1120, the temporal distortion calculation unit 1020 calculates the result from the spatial/temporal feature information extraction unit 210, the result from the spatial/temporal cognitive sensitive region detection unit 220, and the spatial/temporal cognitive sensitive region change calculation The temporal distortion can be calculated using the result from the unit 230.

12 shows a structure of a spatial distortion calculator according to an example.

The spatial distortion calculation unit 1010 may include a reference video representative spatial information extraction unit 1210, a comparison video representative spatial information extraction unit 1220, and a representative spatial information difference calculation unit 1230.

The reference video representative spatial information extraction unit 1210 may include a reference video spatial information combination unit 1211, a spatial pooling unit 1212, and a temporal pooling unit 1213.

The comparison video representative spatial information extraction unit 1220 may include a comparison video spatial information combination unit 1221, a spatial pooling unit 1222, and a temporal pooling unit 1223.

The functions and operations of the reference video representative spatial information extraction unit 1210, the comparison video representative spatial information extraction unit 1220, and the representative spatial information difference calculation unit 1230 will be described in detail below.

13 is a flowchart of spatial distortion calculation according to an example.

For convenience of description, the embodiment may be described for a case where the spatial/temporal feature information extractor 210 includes only the spatial feature information extractor 410. Unlike this description, the spatial/temporal feature information extractor 210 may include two or more of the spatial feature information extractor 410, the temporal feature information extractor 420, and the spatiotemporal feature information extractor 430. It might be.

The step 1110 described above with reference to FIG. 11 may include steps 1310, 1320 and 1330.

In step 1310, the reference video representative spatial information extraction unit 1210 includes: 1) spatial feature information of the reference video extracted from the reference video, 2) spatial perception sensitive areas of the reference video detected from the reference video, and 3) reference video And representative spatial information of the reference video may be extracted by using a change between the spatial cognitive sensitive area of the reference video and the spatial cognitive sensitive area of the comparative video calculated from the comparison video.

Step 1310 may include steps 1311, 1312 and 1313.

In step 1311, the reference video spatial information combining unit 1211 combines the reference video by combining 1) spatial feature information of the reference video, 2) spatial cognitive sensitive area of the reference video, and 3) spatial cognitive sensitive area change. Spatial information can be generated. The change of the spatial cognitive sensitive region may mean a change between the spatial cognitive sensitive region of the reference video and the spatial cognitive sensitive region of the comparative video.

In other words, the combined spatial information of the reference video may be spatial feature information of the reference video in which the recognition importance of the region is reflected.

The reference video spatial information combining unit 1211 may allow the spatial characteristic information of the reference video to reflect the cognitive importance of the area by using 1) spatial cognitive sensitive areas of the reference video and 2) spatial cognitive sensitive areas. In other words, the spatial characteristic information of the reference video may be reflected per region by the reference video spatial information combining unit 1211.

In step 1312, the spatial pooling unit 1212 may extract representative spatial information of the reference video from the combined spatial information of the reference video resulting from the reference video spatial information combining unit 1211.

The spatial pooling unit 1212 may extract representative spatial information of each image of the reference video from the spatial feature information of the reference video in which the recognition importance of the region is reflected.

For example, representative spatial information of the image may be an average value of spatial information of the image.

For example, the representative spatial information of the image may be a standard deviation of the spatial information of the image.

For example, the spatial pooling unit 1212 may extract representative spatial information of an image using spatial pooling used in an automatic image quality measurement method.

In step 1313, the temporal pooling unit 1213 may extract representative spatial information of the reference video (or GOP of the reference video) from representative spatial information of each image of the reference video (or GOP of the reference video). .

For example, the representative spatial information of the reference video (or GOP of the reference video) may be an average value of spatial information of the reference video (or GOP of the reference video).

For example, the representative spatial information of the reference video (or GOP of the reference video) may be a standard deviation of the spatial information of the reference video (or GOP of the reference video).

For example, the temporal pooling unit 1213 may extract representative spatial information of the reference video (or GOP of the reference video) using temporal pooling used in the automatic image quality measurement method.

In step 1320, the representative video representative spatial information extraction unit 1220 includes spatial characteristic information of the comparison video extracted from the comparison video, 2) spatial recognition sensitive area of the comparison video detected from the comparison video, and 3) reference video and comparison. Representative spatial information of the comparison video can be extracted using the change between the spatial cognitive sensitive area of the reference video calculated from the video and the spatial cognitive sensitive area of the comparison video.

Step 1320 may include steps 1321, 1322, and 1323.

In step 1321, the comparison video spatial information combining unit 1221 combines the comparison video by combining 1) spatial feature information of the comparison video, 2) spatial cognitive sensitivity area of the comparison video, and 3) spatial cognitive sensitivity area change. Spatial information can be generated. The change of the spatial cognitive sensitive region may mean a change between the spatial cognitive sensitive region of the reference video and the spatial cognitive sensitive region of the comparative video.

In other words, the combined spatial information of the comparison video may be spatial feature information of the comparison video in which the recognition importance of the region is reflected.

The comparison video spatial information combining unit 1221 may use 1) the spatial cognitive sensitive area of the comparison video and 2) the spatial cognitive sensitivity area to change the spatial feature information of the comparative video to reflect the cognitive importance of the area. In other words, the spatial characteristic information of the comparative video may be reflected per region by the comparison video spatial information combining unit 1221.

In step 1322, the spatial pooling unit 1222 may extract representative spatial information of the comparison video from the combined spatial information of the comparison video, which is the result from the comparison video spatial information combining unit 1221.

Representative spatial information of each image of the comparison video may be extracted from spatial feature information of the comparison video in which the recognition importance of the region is reflected.

For example, representative spatial information of the image may be an average value of spatial information of the image.

For example, the representative spatial information of the image may be a standard deviation of the spatial information of the image.

For example, the spatial pooling unit 1222 may extract representative spatial information of an image using spatial pooling used in an automatic image quality measurement method.

In step 1323, the temporal pooling unit 1223 may extract representative spatial information of the comparison video (or GOP of the comparison video) from representative spatial information of each image of the comparison video (or GOP of the comparison video). .

For example, representative spatial information of the comparison video (or GOP of the comparison video) may be an average value of spatial information of the comparison video (or GOP of the comparison video).

For example, the representative spatial information of the comparison video (or GOP of the comparison video) may be a standard deviation of the spatial information of the comparison video (or GOP of the comparison video).

For example, the temporal pooling unit 1223 may extract representative spatial information of the comparison video (or GOP of the comparison video) using temporal pooling used in the automatic image quality measurement method.

In operation 1330, the representative spatial information difference calculator 1230 may calculate a difference between the representative spatial information of the reference video and the representative spatial information of the comparison video.

The spatial distortion described above with reference to FIG. 3 may be a difference between representative spatial information of a reference video (or an image of the reference video) and representative spatial information of a comparison video (or image of the comparison video).

14 shows a structure of a spatial feature information extracting unit according to an example.

The spatial feature information extraction unit 410 may include a horizontal/vertical feature detection unit 1410 and a horizontal/vertical feature detection unit 1420.

The functions and operations of the horizontal/vertical feature detection unit 1410 and the horizontal/vertical feature detection unit 1420 will be described in detail below.

15 is a flowchart of spatial feature information extraction according to an example.

Step 510 described above with reference to FIG. 5 may include steps 1510 and 1520.

In step 1510, the horizontal/vertical feature detector 1410 may detect horizontal/vertical features of spatial feature information from each image of the input video.

The horizontal/vertical feature may mean one or more of a horizontal feature and a vertical feature.

Generally, an area having a large contrast, such as an object boundary, may have high cognitive sensitivity.

According to these characteristics, the horizontal/vertical feature detector 1410 may derive a region having high cognitive sensitivity from the image using an edge detection method such as a sobell operation of image processing.

The Sobel operation is an example, and another operation for extracting a region having high contrast sensitivity may be used as an edge detection method.

The horizontal/vertical feature detection unit 1410 may apply the edge detection method in the horizontal direction and the vertical direction, respectively. The horizontal/vertical feature detection unit 1410 may derive information on a horizontal edge value using a horizontal edge detection method for an image. In addition, the horizontal/vertical feature detection unit 1410 may derive information on a vertical edge value using an edge detection method in a vertical direction with respect to an image.

The horizontal/vertical feature detection unit 1410 may derive the final edge information using a horizontal edge value and a vertical edge value.

For example, the final edge information may be a square root of a sum of squares of horizontal edge values and squares of vertical edge values.

In step 1520, the feature detection unit 1420 other than the horizontal/vertical direction may detect a feature of the spatial feature information in a direction other than the horizontal/vertical direction from each image of the input video.

In order to consider human characteristics that are more sensitive to horizontal/vertical edges, and blocking artifacts due to encoding, are generated in the form of horizontal/vertical edges, the horizontal/vertical feature detection unit 1410 detects results generated by the edge detection method From the horizontal/vertical component, strong edges can be separated. The horizontal/vertical feature detected by the horizontal/vertical feature detector 1410 may be a strong edge as a separate horizontal/vertical component.

Alternatively, the horizontal/vertical feature detector 1410 may separate a region in which a vertical edge value derived in the middle from the final edge information is greater than or equal to a threshold and a region in which horizontal edge values are greater than or equal to a threshold. The horizontal/vertical feature detected by the horizontal/vertical feature detection unit 1410 may be such a separated area. The horizontal/vertical direction feature detected by the non-horizontal/vertical feature detection unit 1420 may be the rest of the separated area in the final edge information.

For example, the horizontal/vertical feature may be a horizontal-vertical edge map (HVM). A feature for a direction other than horizontal/vertical may be an edge map (EM) (excluding information on a horizontal/vertical edge).

16 shows a structure of a spatial cognitive sensitive region detection unit according to an example.

The spatial cognitive sensitive area detection unit 610 may include a spatial randomness calculation unit 1610 and a background flatness calculation unit 1620.

The functions and operations of the spatial complexity calculator 1610 and the background flatness calculator 1620 are described in detail below.

17 is a flowchart of spatial cognitive sensitive area detection according to an example.

Step 710 described above with reference to FIG. 7 may include steps 1710 and 1720.

In step 1710, the spatial complexity calculator 1610 may calculate a spatial complexity of pixels or blocks of an image of an input video. The spatial complexity calculator 1610 may calculate the spatial complexity in units of pixels or blocks for an image of the input video.

The spatial complexity calculator 1610 may calculate the spatial complexity of the pixel or block using the similarity of the surroundings of the pixel or block.

For convenience of description, the result of processing by the spatial complexity calculator 1610 may be referred to as a spatial randomness map (SRM). In other words, the spatial complexity calculator 1610 may generate an SRM by calculating spatial complexity of pixels or first blocks of an image of an input video. The SRM may include spatial complexity of pixels or first blocks of an image of the input video.

In operation 1720, the background flatness calculator 1620 may calculate a background flatness of pixels or blocks of an image of an input video. The background flatness calculator 1620 may calculate a background flatness in units of blocks for an image of an input video.

The size of the second block used as a unit in the background flatness calculator 1620 may be larger than the size of the first block used as a unit in the spatial complexity calculator 1610.

The result of the processing by the background flatness calculating unit 1620 may be referred to as a smoothness map (SM). In other words, the background flatness calculator 1620 may generate an SM by calculating the background flatness of the second blocks of the image of the input video. SM may include background flatness of pixels or blocks of an image of the input video.

18 shows a measurement of spatial complexity of an image according to an example.

In the following, the operation of the spatial complexity calculator 1610 is described.

In FIG. 18, a prediction model used for measurement of spatial complexity is illustrated. In the image, the central region Y can be predicted from the peripheral regions X. In FIG. 18, the central area Y is shown as a circle filled inside, and the peripheral areas X are shown as circles filled with a diagonal line inside. Arrows can indicate predictions. At this time, the region from which the arrow starts may be used for prediction of the region indicated by the arrow.

For example, an area can be a pixel or block. In addition, the area may be a unit in which the spatial complexity calculator 1610 measures spatial complexity.

When the region is a block, the spatial complexity calculator 1610 may downsample an image of the input video so that each block corresponds to one pixel of the spatial complexity map.

Hereinafter, a case where the area is a pixel, that is, a case where the spatial complexity calculator 1610 measures spatial complexity in units of pixels will be described.

As illustrated in FIG. 18, the spatial complexity calculator 1610 may predict the central region Y from the peripheral region X. This prediction can be expressed as Equation 1 below. The peripheral area X may mean a set of neighboring pixels surrounding the central area Y.

[Equation 1]

u may mean a spatial location. H may be a transformation matrix that provides optimal prediction.

The symbol "^" may indicate that a value is generated by prediction.

For example, H can be obtained through the optimization method of the minimum average error as shown in Equation 2 below.

[Equation 2]

R _XY may be a cross-correlation matrix of X and Y. R _X may be a correlation matrix of X.

R- ¹ _X may be the inverse matrix of R _X.

R- ¹ _X can be obtained using an approximated pseudo-inverse matrix technique as shown in Equation 3 below.

[Equation 3]

Finally, the spatial complexity calculator 1610 can obtain the SRM using Equation 4 below. SR ( u ) may indicate the degree of spatial randomness.

[Equation 4]

u may be a vector representing position information of ( x , y ) pixels.

Through Equation 4, the spatial complexity calculator 1610 may numerically grasp the degree of spatial randomness in the input image in units of pixels.

In the following, the operation of the background flatness calculating unit 1620 is described.

The background flatness calculating unit 1620 may calculate a background flatness in units of blocks.

When the unit used in the calculation of the SRM is the first block, the second block that is the unit of calculation of the background flatness calculation unit 1620 is horizontal and vertical compared to the block that is the unit used in the calculation of the SRM. It may have to be more than 2 times larger. That is, when the size of the first block that is a unit of calculation of the spatial complexity calculator 1610 is ( w , h ), the second block that is a unit of calculation of the background flatness calculator 1620 is (2 w , 2 h) ).

In calculating the background flatness of the block, the background flatness calculator 1620 may calculate the background flatness of the block using pixel values of pixels of the SRM corresponding to the inside of the block. The background flatness of the block can be calculated according to Equation 5 below.

[Equation 5]

N _lc may indicate the number of pixels having a spatial randomness lower than a threshold in a block to be calculated (ie, pixels having a low complexity). W _b ² may represent an area of the block (ie, the number of pixels in the block).

In the following, operations of the reference video spatial information combining unit 1211 and the comparison video spatial information combining unit 1221 are described. Hereinafter, the reference video spatial information combining unit 1211 and the comparative video spatial information combining unit 1221 are abbreviated as spatial information combining units. The description of the spatial information combining unit may be applied to the reference video spatial information combining unit 1211 and the comparison video spatial information combining unit 1221, respectively. The input video to the spatial information combining unit may be interpreted as a reference video for the reference video spatial information combining unit 1211, and may be interpreted as a comparison video for the comparison video spatial information combining unit 1221.

The spatial information combining unit may combine the inputted three information into one information. The three pieces of information inputted here may be 1) spatial characteristic information of the input video, 2) spatial cognitive sensitive area of the input video, and 3) spatial cognitive sensitive area change.

The spatial information combining unit may further emphasize the spatial characteristics of the cognitive sensitive region, and the spatial characteristics of the cognitive desensitization region may be weaker. In addition, a change in the spatial cognitive sensitive area in the reference video and the comparative video may mean that important information has been lost or artifacts that significantly affect image quality have occurred. Accordingly, when the spatial cognitive sensitive region is changed in the reference video and the comparative video, the spatial information combining unit may emphasize the spatial characteristics of the spatially cognitive sensitive region.

The inputs to the spatial information combining section are 1) HVM (that is, horizontal/vertical component feature), 2) EM (that is, feature other than horizontal/vertical feature), 3) SRM (that is, spatial complexity) and 4) SM ( That is, background flatness). When the inputs to the spatial information combining unit are HVM, EM, SRM, and SM, Equations 6 to 9 below can be established.

[Equation 6]

[Equation 7]

[Equation 8]

[Equation 9]

Equations 6 and 7 may represent a combination of spatial information on an image of a reference video.

Equations 8 and 9 may represent a combination of spatial information on an image of a comparison video.

ref may represent an image of a reference video. comp may represent an image of a comparison video. x and y may represent the coordinates of the pixel of the image. The SVW may be cognitive weight (or spatial visual weighting) for existing spatial information.

The symbol "'" may indicate updated values. For example, EM ' _ref ( x , y ) may indicate the updated value of EM for a pixel. Here, the pixel may be a pixel having coordinates ( x , y ) in an image of the reference video.

Alternatively, the symbol "'" may indicate that the SVW has been updated as it is applied. For example, EM _'ref may be an EM (EM weighted) the generated weighted by weighting the EM for the image of the reference video.

The symbol "·" can mean a product. The symbol "·" may mean multiplying the values of the two positions (in other words, matrices) at the same position (in other words, elements of the matrix).

The spatial information combining unit may update the EM for the reference video by applying a cognitive weight to the EM for the reference video.

The spatial information combining unit may update the HVM for the reference video by applying cognitive weights to the HVM for the reference video.

The spatial information combining unit may update the EM for the comparison video by applying a cognitive weight to the EM for the comparison video.

The spatial information combining unit may update the HVM for the comparison video by applying cognitive weights to the HVM for the comparison video.

For example, the cognitive weight for spatial feature information of the image of the reference video used in Equations 6 and 7 may be defined as Equation 10 below.

[Equation 10]

" w " may indicate cognitive weights.

For convenience of explanation, the same cognitive weights are used in Equations 6 and 7, but different cognitive weights can be used in Equations 6 and 7, respectively.

For example, the cognitive weight for spatial feature information of the image of the comparison video used in Equations 8 and 9 may be defined as Equation 11 below.

[Equation 11]

For convenience of explanation, the same cognitive weights are used in Equations 8 and 9, but different cognitive weights can be used in Equations 8 and 9, respectively.

The cognitive weights of Equations 10 and 11 may not reflect changes in the spatial cognitive sensitive area. Equations 12 and 13 below may represent updated cognitive weights that combine changes in spatial cognitive sensitivity regions. In other words, Equations 12 and 13 may represent cognitive weights reflecting changes in spatial cognitive sensitivity regions.

[Equation 12]

[Equation 13]

In Equation 10, Equation 11, Equation 12, and Equation 13 described above, it has been described that the same cognitive weight is applied to an image of a reference video and an image of a comparison video. Alternatively, different cognitive weights may be applied to the video of the reference video and the video of the comparison video, respectively.

19 shows the structure of a temporal distortion calculation unit according to an example.

The temporal distortion calculation unit 1020 may include a reference video representative temporal information calculation unit 1910, a comparative video representative temporal information calculation unit 1920, and a representative temporal information difference calculation unit 1930.

The reference video representative temporal information calculating unit 1910 may include a reference video temporal information combining unit 1911, a spatial pooling unit 1912, and a temporal pooling unit 1913.

The comparison video representative temporal information calculating unit 1920 may include a comparison video temporal information combining unit 1921, a spatial pooling unit 1922, and a temporal pooling unit 1923.

The functions and operations of the reference video representative temporal information calculator 1910, the comparative video representative temporal information calculator 1920, and the representative temporal information difference calculator 1930 will be described in detail below.

20 is a flowchart of temporal distortion calculation according to an example.

For convenience of description, the embodiment may be described for a case where the spatial/temporal feature information extractor 210 includes only the temporal feature information extractor 420. Unlike this description, the spatial/temporal feature information extractor 210 may include two or more of the spatial feature information extractor 410, the temporal feature information extractor 420, and the spatiotemporal feature information extractor 430. It might be.

Step 1120 described above with reference to FIG. 11 may include steps 2010, 2020 and 2030.

In step 2010, the reference video representative temporal information calculator 1910 includes: 1) temporal characteristic information of the reference video extracted from the reference video, 2) temporal perception sensitive areas of the reference video detected from the reference video, and 3) reference video And a change between the temporal cognitive sensitive region of the reference video calculated from the comparative video and the temporal cognitive sensitive region of the comparative video, to calculate representative temporal information of the reference video.

Step 2010 may include steps 2011, 2012 and 2013.

In step 2011, the reference video temporal information combining unit 1911 combines the reference video by combining 1) temporal feature information of the reference video, 2) temporal cognitive sensitive area of the reference video, and 3) changes of the temporal cognitive sensitive area. Generated temporal information. The change of the temporal cognitive sensitive region may mean a change between the temporal cognitive sensitive region of the reference video and the temporal cognitive sensitive region of the comparative video.

In other words, the combined temporal information of the reference video may be temporal characteristic information of the reference video in which the recognition importance of the region is reflected.

The reference video temporal information combining unit 1911 may allow the temporal characteristic information of the reference video to reflect the cognitive importance of the area by using 1) temporal cognitive sensitive areas of the reference video and 2) temporal cognitive sensitive areas. In other words, the temporal characteristic information of the reference video may be reflected per region by the reference video temporal information combining unit 1911.

In step 2012, the spatial pooling unit 1912 can extract representative temporal information of the reference video from the combined temporal information of the reference video, which is the result from the reference video temporal information combining unit 1911.

The spatial pooling unit 1912 may extract representative temporal information of each image of the reference video from temporal characteristic information of the reference video reflecting the recognition importance of the region.

For example, representative temporal information of the image may be an average value of temporal information of the image.

For example, the representative temporal information of the image may be a standard deviation of temporal information of the image.

For example, the spatial pooling unit 1912 may extract representative temporal information of an image using spatial pooling used in an automatic image quality measurement method.

In step 2013, the temporal pooling unit 1913 may extract representative temporal information of the reference video (or GOP of the reference video) from representative temporal information of each image of the reference video (or GOP of the reference video). .

For example, the representative temporal information of the reference video (or GOP of the reference video) may be an average value of temporal information of the reference video (or GOP of the reference video).

For example, the representative temporal information of the reference video (or GOP of the reference video) may be a standard deviation of temporal information of the reference video (or GOP of the reference video).

For example, the temporal pooling unit 1913 may extract representative temporal information of the reference video (or GOP of the reference video) using temporal pooling used in the automatic image quality measurement method.

In step 2020, the comparison video representative temporal information calculation unit 1920 includes temporal characteristic information of the comparison video extracted from the comparison video, 2) temporal recognition sensitive area of the comparison video detected from the comparison video, and 3) reference video and comparison. The representative temporal information of the comparison video may be calculated using the change between the temporal sensitive area of the reference video calculated from the video and the temporal cognitive sensitive area of the comparison video.

Step 2020 may include steps 2021, 2022 and 2023.

In step 2021, the comparison video temporal information combining unit 1921 combines the comparison video by combining 1) temporal characteristic information of the comparative video, 2) temporal cognitive sensitive area of the comparative video, and 3) changes of the temporal cognitive sensitive area. Generated temporal information. The change of the temporal cognitive sensitive region may mean a change between the temporal cognitive sensitive region of the reference video and the temporal cognitive sensitive region of the comparative video.

In other words, the combined temporal information of the comparison video may be temporal characteristic information of the comparison video in which the recognition importance of the region is reflected.

The comparison video temporal information combining unit 1921 may allow the temporal characteristic information of the comparative video to reflect the cognitive importance of the region by using 1) temporal cognitive sensitive regions of the comparative video and 2) temporal cognitive sensitive regions. That is, the comparison video temporal information combining unit 1921 may reflect the cognitive importance of temporal feature information of the comparison video for each region.

In step 2022, the spatial pooling unit 1922 may extract representative temporal information of the comparison video from the combined temporal information of the comparison video resulting from the comparison video temporal information combining unit 1921.

The spatial pooling unit 1922 may extract representative temporal information of each image of the comparative video from temporal characteristic information of the comparative video in which the recognition importance of the region is reflected.

For example, representative temporal information of the image may be an average value of temporal information of the image.

For example, the representative temporal information of the image may be a standard deviation of temporal information of the image.

For example, the spatial pooling unit 1922 may extract representative temporal information of an image using spatial pooling used in an automatic image quality measurement method.

In one embodiment, step 2022 by the spatial pooling portion 1922 may be omitted.

In step 2023, the temporal pooling unit 1923 may extract representative temporal information of the comparison video (or GOP of the comparison video) from representative temporal information of each image of the comparison video (or GOP of the comparison video). .

For example, representative temporal information of the comparison video (or GOP of the comparison video) may be an average value of temporal information of the comparison video (or GOP of the comparison video).

For example, the representative temporal information of the comparison video (or GOP of the comparison video) may be a standard deviation of temporal information of the comparison video (or GOP of the comparison video).

For example, the temporal pooling unit 1923 may extract representative temporal information of the comparison video (or GOP of the comparison video) using temporal pooling used in the automatic image quality measurement method.

In operation 2030, the representative temporal information difference calculator 1930 may calculate a difference between the representative temporal information of the reference video and the representative temporal information of the comparison video.

The temporal distortion described above with reference to FIG. 3 may be a difference between representative temporal information of the reference video and representative temporal information of the comparison video.

21 shows a structure of a temporal feature information extracting unit according to an example.

The temporal feature information extraction unit 420 may include a vertical motion information extraction unit 2110 and a horizontal motion information extraction unit 2120.

The functions and operations of the vertical motion information extraction unit 2110 and the horizontal motion information extraction unit 2120 will be described in detail below.

22 is a flowchart of temporal feature information extraction according to an example.

Step 520 described above with reference to FIG. 5 may include steps 2210 and 2220.

In operation 2210, the vertical motion information extraction unit 2110 may extract vertical motion information of temporal feature information from a plurality of images of an input video.

According to a person's cognitive characteristics, a person may be sensitive to sudden movements between frames. In order to consider these cognitive characteristics, the vertical motion information extraction unit 2110 may extract vertical motion information as shown in Equation 14 below.

[Equation 14]

v _x ( i , j , t ) may be a movement speed in the x- axis direction. R is the frame rate per second. R may be 1/ Δt .

Equation 14 can reflect the cognitive characteristic judder distortion in the motion vector. The vertical motion information extraction unit 2110 may reflect cognitive distortion judder distortion by extracting motion information using Equation 14.

In step 2220, the horizontal motion information extraction unit 2120 may extract horizontal motion information of temporal feature information from a plurality of images of the input video.

The horizontal motion information extraction unit 2120 may extract horizontal motion information by substituting the motion speed in the y- axis direction into Equation 14 described above.

23 shows a structure of a temporal cognitive sensitive region detection unit according to an example.

The temporal cognitive sensitive region detection unit 620 may include a spatial randomness calculation unit 2310 and a background flatness calculation unit 2320.

The functions and operations of the spatial complexity calculation unit 2310 and the background flatness calculation unit 2320 will be described in detail below.

24 is a flowchart of temporal cognitive sensitive area detection according to an example.

Step 720 described above with reference to FIG. 7 may include steps 2410 and 2420.

The temporal cognitive sensitive region detector 620 may detect a temporal cognitive sensitive region with respect to the GOP. The GOP may be composed of two images, a previous image and a current image.

In step 2410, the spatial complexity calculator 2310 may calculate the spatial complexity of the current image of the GOP of the input video.

At this time, the method for calculating the spatial complexity of the spatial complexity calculator 1610 of the spatial cognitive sensitive region detector 610 described above may also be used in the spatial complexity calculator 2310.

Alternatively, the spatial complexity calculator 2310 may calculate the spatial complexity of the previous image of the GOP of the input video.

Alternatively, the spatial complexity calculator 2310 may calculate the spatial complexity of the weighted average of the current image and the previous image of the GOP of the input video.

In step 2420, the background flatness calculating unit 2320 may calculate the background flatness for the current image of the GOP of the input video.

At this time, the method of calculating the background flatness of the background flatness calculating unit 1620 of the spatial cognitive sensitive region detector 610 described above may also be used in the background flatness calculating unit 2320.

Alternatively, the background flatness calculating unit 2320 may calculate the background flatness with respect to a previous image of the GOP of the input video.

Alternatively, the background flatness calculating unit 2320 may calculate the background flatness with respect to the weighted average of the current image and the previous image of the GOP of the input video.

25 illustrates a configuration for measuring image quality using a deep neural network for measuring cognitive image quality according to an embodiment.

The spatial/temporal feature information extraction unit 210, the distortion calculation unit 240, and the index calculation unit 250 described above with reference to FIG. 2 may be replaced with the deep neural network 2510 for cognitive image quality measurement. The processor 210 may drive the deep neural network 2510 for cognitive image quality measurement.

The cognitive image quality of the comparison video may be measured by the deep neural network 2510 measuring the cognitive image quality instead of the units described above with reference to FIG. 2. At this time, a reference video and a comparison video may be input to the deep neural network 2510 for cognitive quality measurement, 1) a spatial/temporal cognitive sensitive area of the reference video, 2) a spatial/temporal cognitive sensitive area of the comparative video, and 3) Changes between the spatial/temporal cognitive sensitive region of the reference video and the spatial/temporal cognitive sensitive region of the comparison video may be input.

In other words, when the cognitive image quality measurement deep neural network 2510 measures cognitive image quality, information for measuring cognitive image quality is not generated by itself in the cognitive image quality measurement deep neural network 2510 in which a reference video and a comparison video are input. , May be provided by the spatial/temporal cognitive sensitive region detection unit 220 and the spatial/temporal cognitive sensitive region change calculation unit 230 described in the embodiment.

As the information from the outside is input, the cognitive image quality measurement deep neural network 2510 can better learn cognitive characteristics related to the reference video and the comparative video in the process of learning the deep neural network, and the cognitive image quality after the end of learning Can provide higher measurement reliability in the evaluation of.

The function and operation of the cognitive quality measurement deep neural network 2510 will be described in detail below.

26 is a flowchart of a method for measuring image quality using a deep neural network for measuring cognitive image quality according to an embodiment.

Step 2610 may correspond to step 320 described above with reference to FIG. 3. Duplicate description is omitted.

Step 2620 may correspond to step 340 described above with reference to FIG. 3. Duplicate description is omitted.

Step 2630 may correspond to step 350 described above with reference to FIG. 3. Duplicate description is omitted.

In step 2640, 1) reference video, 2) comparative video, 3) spatial/temporal cognitive sensitive area of reference video, 4) spatial/temporal cognitive sensitive area of reference video and 5) spatial/temporal cognitive sensitivity of reference video The change between the region and the spatial/temporal cognitive sensitive region of the comparison video may be input to the deep neural network 2510 for cognitive quality measurement.

The cognitive quality measurement deep neural network 2510 includes 1) a reference video, 2) a comparative video, 3) a spatial/temporal cognitive sensitive area of the reference video, 4) a spatial/temporal cognitive sensitive area of the comparative video, and 5) a spatial/ The results of the image quality measurement can be generated using the change between the temporal cognitive sensitive region and the spatial/temporal cognitive sensitive region of the comparison video.

The device described above may be implemented with hardware components, software components, and/or combinations of hardware components and software components. For example, the devices and components described in the embodiments include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors (micro signal processors), microcomputers, field programmable arrays (FPAs), It may be implemented using one or more general purpose computers or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and/or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodied in the transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium.

The computer-readable recording medium may include information used in embodiments according to the present invention. For example, a computer-readable recording medium may include a bitstream, and the bitstream may include information described in embodiments according to the present invention.

The computer-readable recording medium may include a non-transitory computer-readable medium.

The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded in the medium may be specially designed and configured for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specifically configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and/or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

100: encoding device
110: processing unit
120: communication unit
170: sensor

Claims (20)

A communication unit that receives a reference video and a comparison video; And
Processing unit for generating a result of the quality measurement for the comparison video
Including,
The processing unit changes the feature information of the reference video, the feature information of the comparison video, the cognitive sensitive region of the reference video, the cognitive sensitive region of the comparison video, the cognitive sensitive region of the reference video, and the cognitive sensitive region of the comparison video An image processing apparatus that calculates distortion using and generates a result of the image quality measurement based on the distortion. According to claim 1,
The feature information of the reference video includes at least one of spatial feature information of the reference video, temporal feature information of the reference video, and spatial and temporal feature information of the reference video,
The feature information of the comparison video includes at least one of spatial feature information of the comparison video, temporal feature information of the comparison video, and spatiotemporal feature information of the comparison video. According to claim 2,
The processing unit extracts the spatiotemporal feature information from the spatiotemporal slice of the input video,
The input video is at least one of the reference video and the comparison video,
The space-time slice is an image processing apparatus in which a group of pictures (GOP) of the input video is spatially divided. According to claim 2,
The processing unit detects horizontal and vertical features of the spatial feature information from an image of an input video,
The processing unit detects a characteristic for a direction other than a horizontal direction and a vertical direction from the image. The method of claim 4,
The processing unit derives a region having high cognitive sensitivity from the image using an edge detection method. The method of claim 5,
The edge detection method is a Sobel operation image processing apparatus. The method of claim 5,
The horizontal feature and the vertical feature are horizontal-vertical edge maps,
The feature for the directions other than the horizontal direction and the vertical direction is an image processing apparatus that is an edge map in which information on horizontal edges and vertical edges is excluded. The method of claim 7,
The processing unit updates the horizontal-vertical edge map using a first cognitive weight,
The processing unit updates the edge map from which information on the horizontal edge and the vertical edge is excluded by using a second cognitive weight. The method of claim 8,
The first cognitive weight and the second cognitive weight are image processing devices that reflect changes in the spatial cognitive sensitive region. According to claim 1,
The cognitive sensitive region of the reference video includes at least one of a spatial cognitive sensitive region of the reference video, a temporal cognitive sensitive region of the reference video, and a spatial and temporal cognitive sensitive region of the reference video,
The cognitive sensitive region of the comparison video includes at least one of a spatial cognitive sensitive region of the comparison video, a temporal cognitive sensitive region of the comparison video, and a spatial and temporal cognitive sensitive region of the comparison video. The method of claim 10,
The processing unit generates a spatial complexity map by calculating spatial complexity of pixels or first blocks of an image of an input video,
The input video is the reference video or the comparison video,
The processing unit generates an flatness map by calculating background flatnesses of second blocks of an image of the input video, According to claim 1,
The change is an image processing apparatus that is a difference between a cognitive sensitive region of the reference video and a cognitive sensitive region of the comparison video. The method of claim 12,
The distortion includes one or more of spatial distortion, temporal distortion, and spatial and temporal distortion. The method of claim 13,
The processing unit extracts representative spatial information of the reference video by using spatial characteristic information of the reference video, spatial cognitive sensitive areas and spatial cognitive sensitive areas of the reference video,
The processing unit extracts representative spatial information of the comparison video by using the spatial characteristic information of the comparison video, the spatial cognitive sensitive area and the spatial cognitive sensitive area of the comparison video,
The processing unit calculates the spatial distortion,
The spatial distortion is a difference between representative spatial information of the reference video and representative spatial information of the comparison video,
The change of the spatial cognitive sensitive region is a change between the spatial cognitive sensitive region of the reference video and the spatial cognitive sensitive region of the comparison video. The method of claim 14,
The processor generates combined spatial information of the reference video by combining spatial feature information of the reference video, spatial cognitive sensitive area of the reference video, and changes in the spatial cognitive sensitive area,
The processing unit extracts representative spatial information of each image of the reference video from the combined spatial information,
The processing unit extracts representative spatial information of the reference video from representative spatial information of each image of the reference video. The method of claim 13,
The processing unit extracts representative temporal information of the reference video using temporal characteristic information of the reference video, temporal cognitive sensitive areas and temporal cognitive sensitive areas of the reference video,
The processing unit extracts representative temporal information of the comparison video by using temporal characteristic information of the comparison video, temporal cognitive sensitive area and temporal cognitive sensitive area of the comparison video,
The processing unit calculates the temporal distortion,
The temporal distortion is a difference between the representative temporal information of the reference video and the representative temporal information of the comparison video,
The change of the temporal cognitive sensitive region is a change between the temporal cognitive sensitive region of the reference video and the temporal cognitive sensitive region of the comparison video. The method of claim 16,
The processor generates combined temporal information of the reference video by combining temporal characteristic information of the reference video, temporal cognitive sensitive area of the reference video, and changes in the temporal cognitive sensitive area,
The processing unit extracts representative temporal information of each image of the reference video from the combined temporal information,
The processing unit extracts representative temporal information of the reference video from representative temporal information of each image of the reference video. Extracting feature information of the reference video;
Detecting a cognitive sensitive area of the reference video:
Extracting feature information of the comparison video;
Detecting a cognitive sensitive region of the comparison video;
Calculating a change between a cognitive sensitive region of the reference video and a cognitive sensitive region of the comparison video;
Calculating distortion using feature information of the reference video, feature information of the comparison video, a cognitive sensitive region of the reference video, a cognitive sensitive region of the comparison video, and the change; And
Generating a result of image quality measurement based on the distortion
Image processing method comprising a. A computer-readable recording medium containing a program for processing the image processing method of claim 18. A communication unit that receives a reference video and a comparison video; And
Processing unit to drive deep neural network for cognitive quality measurement
Including,
The processing unit detects the cognitive sensitive region of the reference video and the cognitive sensitive region of the comparison video, calculates a change between the cognitive sensitive region of the reference video and the cognitive sensitive region of the comparative video,
The cognitive sensitive region of the reference video, the cognitive sensitive region of the comparison video, and the change are input to the deep neural network for measuring cognitive image quality,
The cognitive image quality measurement deep neural network is an image processing apparatus that generates a result of image quality measurement using a cognitive sensitive region of the reference video, a cognitive sensitive region of the comparison video, and the change.

Images