KR20220140669A - Method and apparatus for measuring image quality base on perceptual sensitivity - Google Patents

Abstract

Disclosed are a method and a device for measuring the image quality considering perceptual sensitivity. A spatial distortion analysis unit generates spatial distortion information based on spatial feature information of an original image, spatial feature information of a comparison image, and spatial recognition sensitivity-based information of the original image. A temporal distortion analysis unit generates spatial distortion information based on temporal feature information of the original image, the temporal feature information of the comparison image, and the temporal perception sensitivity-based information of the original image. An index calculation unit generates an image quality index for the subjective image quality based on the temporal distortion information and the spatial distortion information.

Description

Image quality measurement method and apparatus considering cognitive sensitivity

The following embodiments relate to a method and apparatus for image processing, and in more detail, a method and apparatus for measuring image quality in consideration of cognitive sensitivity are disclosed.

With the continuous development of the information and communication industry, a broadcasting service having a high definition (HD) resolution has spread worldwide. Through this proliferation, many users have become accustomed to high-resolution and high-definition images and/or videos.

In order to satisfy users' demand for high image quality, many organizations are spurring the development of next-generation imaging devices. User interest in High Definition TV (HDTV) and Full HD (FHD) TV, as well as Ultra High Definition (UHD) TV, which has a resolution four times higher than that of FHD TV has increased, and with this increase in interest, image encoding/decoding technology for an image having higher resolution and image quality is required. In addition, in the video service, the "quality" of the video is becoming a very important competitive edge.

As research fields related to image quality, there are image compression fields that must provide optimal image quality under limited capacity, and super-resolution image processing fields that improve image quality.

In the field of image compression and image processing, an index indicating image quality is required to verify the performance of the developed technology or to select a mode within the developed technology. In addition, as these indicators better reflect subjective image quality performance, the developed technology can provide superior subjective image quality performance.

Full reference quality indicators such as the conventional Mean Squared Error (MSE), Peak Signal to Noise Ratio (PSNR) and Structural SIMilarity (SSIM) As a method of measuring the difference between target images, although it has the advantage of being intuitive, there is a case where there is a large difference between the measured value and the perceived image quality according to the index, and relatively low accuracy in comparison between images with different characteristics It has the disadvantage of being visible.

An embodiment may provide an apparatus and method for extracting various temporal feature information and spatial feature information from an image, and predicting image quality by applying a weight based on cognitive sensitivity suitable to the extracted feature information.

An embodiment may provide an apparatus and method for discriminating a sensitive region from a perceptual point of view within an image and generating an index in consideration of temporal distortion and spatial distortion in the determined region.

An embodiment may provide an apparatus and method for calculating a reliability index even in comparison between images having similar characteristics to the actual perceived quality of an image and having different characteristics.

In one aspect, there is provided a spatial distortion analyzer for generating spatial distortion information based on spatial characteristic information of the original image, spatial characteristic information of the comparison image, and spatial perception sensitivity-based information of the original image; a temporal distortion analyzer for generating spatial distortion information based on temporal characteristic information of the original image, temporal characteristic information of the comparison image, and temporal recognition sensitivity-based information of the original image; and an index calculator configured to generate a picture quality index for subjective picture quality based on the temporal distortion information and the spatial distortion information.

In addition to this, another method, apparatus, system for implementing the present invention, and a computer readable recording medium for recording a computer program for executing the method are further provided.

An apparatus and method are provided for extracting various temporal and spatial characteristic information from an image, and predicting image quality by applying a weight based on cognitive sensitivity suitable to the extracted feature information.

An apparatus and method are provided for discriminating a sensitive region from a cognitive point of view within an image and generating an index that considers temporal distortion and spatial distortion in the discriminated region.

An apparatus and method for calculating an index of reliability even in comparison between images having different characteristics and having a similar tendency to the actual perceived image quality of an image are provided.

1 illustrates a structure of an electronic device according to an exemplary embodiment.
2 illustrates a structure of a processing unit according to an example.
3 is a flowchart of a method of deriving an index for image quality according to an embodiment.
4 illustrates that a pixel is not determined as a strong HV pixel by a dominant angle of the pixel according to an example.
5 illustrates that a pixel is determined to be a strong HV pixel by a dominant angle of the pixel according to an example.
6 illustrates a comparison between the degree of distortion on the HV plane and actual perceived image quality according to an example.
7 illustrates a method of determining a degree of distortion due to block artifacts according to a cognitive viewpoint according to an example.
8 illustrates a spatial frame and a temporal frame of an image according to an example.
9 illustrates a row plane among temporal frames according to an example.
10 illustrates applying a cognitive sensitivity-based weight to a degree of distortion with respect to a column plane of temporal frames according to an example.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. It should be understood that embodiments are different but need not be mutually exclusive.

Terms used in the embodiments may be interpreted based on the actual meaning of the terms and the contents of the present specification rather than the simple names of terms.

In embodiments, the connection relationship between the specific part and the other part may include an indirect connection relationship between the specific part and the other part via another part in addition to the direct connection relationship between the two parts. The same reference numerals provided in each figure may refer to the same member.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0016] Reference is made to the accompanying drawings, which show by way of illustration specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. In addition, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description set forth below is not intended to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all scope equivalents to those claimed.

Like reference numerals in the drawings refer to the same or similar functions throughout the various aspects. The shapes and sizes of elements in the drawings may be exaggerated for clearer description.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings in order to enable those of ordinary skill in the art to easily practice the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0010] Reference is made to the accompanying drawings, which illustrate specific embodiments by way of example. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that various embodiments are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. In addition, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the detailed description set forth below is not intended to be taken in a limiting sense, and the scope of exemplary embodiments, if properly described, is limited only by the appended claims, along with all scope equivalents to those as claimed.

Like reference numerals in the drawings refer to the same or similar functions throughout the various aspects. The shapes and sizes of elements in the drawings may be exaggerated for clearer description.

Terms used in the examples are for describing the examples and are not intended to limit the present invention. In embodiments, the singular also includes the plural unless the phrase specifically dictates otherwise. As used herein, “comprises” and/or “comprising” refers to the presence of one or more other components, steps, operations and/or elements mentioned. Or addition is not excluded, and it means that an additional configuration may be included in the scope of the technical spirit of the exemplary embodiments or the practice of the exemplary embodiments. When a component is referred to as being “connected” or “connected” to another component, the two components may be directly connected or connected to each other, but in the above 2 It should be understood that other components may exist in the middle of the components.

Terms such as first and second may be used to describe various components, but the above components should not be limited by the above terms. The above terms are used to refer to one component as distinct from another component. For example, without departing from the scope of rights, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

In addition, components shown in the embodiments are shown independently to represent different characteristic functions, and it does not mean that each component is made of separate hardware or only one software component unit. That is, each component is listed as each component for convenience of description. For example, at least two components among the components may be combined into one component. Also, one component may be divided into a plurality of components. Integrated embodiments and separate embodiments of each of these components are also included in the scope of rights without departing from the essence.

In addition, some of the components are not essential components to perform essential functions, but may be optional components only to improve performance. Embodiments may be implemented including only components essential for implementing the essence of the embodiment, and for example, structures in which optional components are excluded, such as components used only to improve performance, are also included in the scope of rights.

Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings in order to enable those of ordinary skill in the art to easily implement the embodiments. In describing the embodiments, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present specification, the detailed description thereof will be omitted.

The embodiment relates to an index for measuring the perceived quality of a video, and it is possible to predict the quality of an image by extracting various temporal characteristic information and spatial characteristic information from an image, and applying a weighting based on cognitive sensitivity appropriate to the extracted characteristic information. have.

Since these proposed indexes discriminate sensitive areas from a cognitive point of view within the image, and consider temporal and spatial distortions in the identified areas, the index calculated for the image shows a tendency similar to the actual perceived quality of the image. can have Therefore, the proposed index can have reliability even in comparison between images of different characteristics.

1 illustrates a structure of an electronic device according to an exemplary embodiment.

The electronic device 100 may include at least a portion of the processing unit 110 , the communication unit 120 , and the storage unit 130 as components. The components may communicate with each other via one or more communication buses or signal lines.

The electronic device 100 may be an image processing device that derives an index for image quality.

Components illustrated for the electronic device 100 in FIG. 1 may be merely examples. Not all of the illustrated components may be essential for the electronic device 100 . The electronic device 100 may have more or fewer components than those shown in FIG. 1 . Also, two or more components shown in FIG. 1 may be combined. Also, the components may be configured or arranged differently than shown in FIG. 1 . Each component may be implemented in hardware including one or more signal processing and/or application specific integrated circuits (ASICs), implemented in software, or a combination of hardware and software.

The processing unit 110 may process a task required for the operation of the electronic device 100 . The processing unit 110 may execute codes of operations or steps of the processing unit 110 described in the embodiments.

The processing unit 110 may generate and process a signal, data, or information input to the electronic device 100 , output from the electronic device 100 , or generated in the electronic device 100 , and the signal, data or information related inspection, comparison, and judgment can be performed. In other words, in the embodiment, generation and processing of data or information and inspection, comparison, and judgment related to data or information may be performed by the processing unit 110 .

For example, the processing unit 110 may be at least one processor.

The processor may be a hardware processor, or a central processing unit (CPU). There may be a plurality of processors. Alternatively, the processor may include a plurality of cores, and may provide multi-tasking for simultaneously executing a plurality of processes and/or a plurality of threads. At least some of the steps of the embodiments may be performed in parallel for a plurality of objects via a plurality of processors, a plurality of cores, a plurality of processes and/or a plurality of threads.

For example, the processing unit 110 may execute codes of operations or steps of the electronic device 100 described in the embodiments.

For example, the processing unit 110 may run a program. The processing unit 110 may execute codes constituting a program. The program may include an operating system (OS) of the electronic device 100 , a system program, an application, and an app.

Also, the processing unit 110 may control other components of the electronic device 100 for the functions of the processing unit 110 described above.

The communication unit 120 may receive data or information used for the operation of the electronic device 100 , and may transmit data or information used for the operation of the electronic device 100 .

The communication unit 120 may transmit data or information to another device in a network to which the electronic device 100 is connected, and may receive data or information from the other device. That is, in the embodiment, transmission or reception of data or information may be performed by the communication unit 120 .

For example, the communication unit 120 may be a networking chip, a networking interface, or a communication port.

The storage unit 130 may store data or information used for the operation of the electronic device 100 . In an embodiment, data or information possessed by the electronic device 100 may be stored in the storage unit 130 .

For example, the storage unit 130 may be a memory. The storage unit 130 may include a built-in storage medium such as a RAM and a flash memory, and may include a removable storage medium such as a memory card.

The storage unit 130 may store at least one program. The processing unit 110 may execute at least one program. The processor 110 may read the code of at least one program from the storage 130 and execute the read code.

Operations, functions, and features of the processing unit 110 , the communication unit 120 , and the storage unit 130 of the electronic device 100 will be described in detail below with reference to the embodiments.

2 illustrates a structure of a processing unit according to an example.

The processing unit 110 includes an image feature information extraction unit 210 , a spatial perception sensitivity calculation unit 220 , a temporal perception sensitivity calculation unit 230 , a spatial distortion analysis unit 240 , a temporal distortion analysis unit 250 , and an index calculation. part 260 may be included.

Alternatively, the image feature information extraction unit 210 , the spatial perception sensitivity calculation unit 220 , the temporal perception sensitivity calculation unit 230 , the spatial distortion analysis unit 240 , the temporal distortion analysis unit 250 , and the index calculation unit 260 . ) may be at least one program executed by the processing unit 210 , a part of at least one program, or at least one module.

An original image may be input to the image feature information extraction unit 210 , the spatial perception sensitivity calculation unit 220 , and the temporal perception sensitivity calculation unit 230 .

A comparison image may be input to the image feature information extraction unit 210 .

The spatial characteristic information of the original image and spatial characteristic information of the comparison image output from the image characteristic information extraction unit 210 may be input to the spatial distortion analyzer 240 .

The temporal characteristic information of the original image and temporal characteristic information of the comparison image output from the image characteristic information extraction unit 210 may be input to the temporal distortion analyzer 250 .

An image quality index may be output from the index calculation unit 260 .

of the image feature information extraction unit 210 , the spatial perception sensitivity calculation unit 220 , the temporal perception sensitivity calculation unit 230 , the spatial distortion analysis unit 240 , the temporal distortion analysis unit 250 , and the index calculation unit 260 . Specific operations and functions are described in more detail below.

3 is a flowchart of a method of deriving an index for image quality according to an embodiment.

The communication unit 120 may receive the original image and the comparison image.

The comparison image may be an encoded image. Alternatively, the comparison image may be an image transmitted through a network or an image to which a specified image processing is applied. Alternatively, the comparison image may be a damaged image.

In operation 310, the image characteristic information extractor 210 may generate spatial characteristic information of the original image, spatial characteristic information of the comparison image, temporal characteristic information of the original image, and temporal characteristic information of the comparison image.

The spatial feature information of the original image may include an SI plane and an HV plane of the original image.

The spatial feature information of the comparison image may include an SI plane and an HV plane of the comparison image.

The image feature information extractor 210 may generate spatial feature information of the image by using a spatial information filter defined as in Equation 1 below.

[Formula 1]

h(x) may be a filter coefficient value for the x-th pixel of the image. When the size of the filter is L, the value of x may range from -L/2 to L/2.

Alternatively, the image feature information extractor 210 may generate spatial feature information of the image by using various edge extraction filters that can know the structure information of the image.

A plane obtained by applying such a filter in the horizontal and vertical directions of the image may be referred to as I _H and I _V , respectively.

Also, the SI plane may be defined as in Equation 2 below.

[Equation 2]

The SI plane may include information about the dominant structure in the image. Accordingly, the SI plane may be spatial feature information that can be significantly utilized in predicting the perceived quality of an image.

The image feature information extractor 210 may obtain a Horizontal-Vertical (HV) plane that discriminates edges in which a vertical component and a horizontal component in the image are particularly strong, along with the SI plane. The image feature information extractor 210 may utilize the HV plane to predict the degree of distortion due to block artifacts generated when an image is compressed.

The construction of the HV plane will be described in detail below with reference to FIGS. 4 and 5 .

The temporal characteristic information of the original image may include temporal frame information of the original image. The temporal characteristic information of the target image may include temporal frame information of the target image.

In operation 320, the spatial perception sensitivity calculator 220 may generate spatial perception sensitivity-based information on the original image by using the original image.

The spatial perception sensitivity-based information on the original image may include a spatial perception sensitivity-based weight map for the original image.

In operation 330 , the temporal cognitive sensitivity calculator 230 may generate temporal cognitive sensitivity-based information on the original image by using the original image.

The temporal perceptual sensitivity-based information may include a temporal perceptual sensitivity-based weight map for the original image.

The temporal perceptual sensitivity calculator 230 may determine a region in which a central movement of the original image is large in consideration of the cognitive characteristics of the original image. The temporal cognitive sensitivity calculation unit 230 may generate information on a region having a large determined motion.

The temporal recognition sensitivity-based information may include information on a region with a large movement.

The temporal cognitive sensitivity calculator 230 may generate a residual (or difference) image between adjacent frames of the original image in order to determine a region in which the central movement of the original image is large.

In order to give a weight to the center of the original image, the temporal cognitive sensitivity calculator 230 may multiply the residual image between adjacent frames by the weight function of the Gaussian distribution as in Equation 3 .

[Equation 3]

N may be the total number of rows or the total number of columns. n may be the row or column that is the current object.

The temporal cognitive sensitivity calculator 230 may use the residual image to determine a region in which the center of the original image has a large movement.

In operation 340 , the spatial distortion analyzer 240 may receive spatial feature information of the original image and spatial feature information of the comparison image from the image feature information extractor 210 .

The spatial feature information of the original image may include an SI plane and an HV plane of the original image.

The spatial feature information of the comparison image may include an SI plane and an HV plane of the comparison image.

Also, the spatial distortion analyzer 240 may receive cognitive sensitivity-based information on the original image from the spatial cognitive sensitivity calculator 220 .

The perceptual sensitivity-based information on the original image may include a spatial perceptual sensitivity-based weight map with respect to the original image.

The spatial distortion analyzer 240 may generate spatial distortion information based on spatial feature information of the original image, spatial feature information of the comparison image, and cognitive sensitivity-based information on the original image.

The spatial distortion analyzer 240 may analyze the degree of spatial distortion based on spatial feature information of the original image, spatial feature information of the comparison image, and cognitive sensitivity-based information on the original image, and a numerical value related to the degree of spatial distortion can create

The spatial distortion information may include a result of analysis on the degree of spatial distortion, and may include numerical values related to the degree of spatial distortion.

The spatial distortion analyzer 240 may perform a comparison between the SI plane of the original image and the SI plane of the comparison image to derive a degree of difference between the SI plane of the original image and the SI plane of the comparison image.

The degree of difference between the SI plane of the original image and the SI plane of the comparison image may indicate a degree of decrease or improvement of overall sharpness in the images.

The spatial distortion analyzer 240 may derive a degree of difference between the HV plane of the original image and the HV plane of the comparison image by performing a comparison between the HV plane of the original image and the HV plane of the comparison image.

A degree of a difference between the HV plane of the original image and the HV plane of the comparison image may indicate a degree of occurrence of block artifacts in the images.

In comparing the planes of the images, the degree of difference between the images may be a difference value between the images. In other words, deriving the degree of difference between the images by comparing the planes of the images may be calculating a difference value between the images.

Also, the degree of difference between the images may be derived using a ratio comparison function as in Equation 4 below.

[Equation 4]

Also, the degree of difference between the images may be derived using a log comparison function as in Equation 5 below.

[Equation 5]

f _o may be an SI plane value or an HV plane value of the original image.

f _p may be an SI plane value or an HV plane value of a comparison image.

The ratio comparison function and log comparison function may reflect a visual masking effect appearing in an image at a certain level. Such visual masking may mean that when the complexity of the specified region in the original image is high, the degree of difference in the specified region appearing in the SI plane or the HV plane is less perceptually revealed.

A comparison between the degree of distortion on the HV plane and the actual perceived picture quality will be described in more detail below with reference to FIGS. 6 and 7 .

The spatial distortion information may include HV plane distortion information and SI plane distortion information. The spatial distortion analyzer 240 may transmit the HV plane distortion information and the SI plane distortion information to the index calculator 260 .

The HV plane distortion information may be information indicating a degree of distortion with respect to the HV plane. In the HV plane distortion information, the degree of distortion on the HV plane may be converted into one representative number through a weighted average or a percentile. Alternatively, the degree of distortion with respect to the HV plane may be used and transmitted as HV plane distortion information as it is in a planar form.

The SI plane distortion information may be information indicating a degree of distortion with respect to the SI plane. The SI plane distortion information may be one in which the degree of distortion on the SI plane is converted into one representative number through a weighted average or percentile. Alternatively, the degree of distortion with respect to the SI plane may be used and transmitted as SI plane distortion information as it is in a planar form.

In operation 350 , the temporal distortion analyzer 250 may receive temporal characteristic information of the original image and temporal characteristic information of the comparison image from the image characteristic information extraction unit 210 .

The temporal characteristic information of the original image may include temporal frames of the original image. The temporal characteristic information of the comparison image may include temporal frames of the comparison image.

The temporal distortion analyzer 250 may receive temporal perception sensitivity-based information on the original image from the temporal perception sensitivity calculator 230 .

The temporal perceptual sensitivity-based information on the original image may include a temporal perceptual sensitivity-based weight map for the original image.

The temporal distortion analyzer 250 may generate temporal distortion information based on temporal frames of the original image, temporal frames of the comparison image, and a weight map based on temporal perception sensitivity of the original image.

The temporal distortion analyzer 250 may analyze the degree of temporal distortion based on the temporal frames of the original image, temporal frames of the comparison image, and a weight map based on temporal perception sensitivity for the original image, and Related numerical values may be output to the index calculator 260 .

The temporal distortion information may include a result of analysis on the degree of temporal distortion, and may include numerical values related to the temporal distortion.

In comparing the temporal frames of the original image and the temporal frames of the comparison image, the temporal distortion analyzer 250 may calculate a difference value between the temporal frames of the original image and the temporal frames of the comparison image, and the spatial distortion analyzer ( 250), temporal frames of the original image and temporal frames of the comparison image may be compared using various other indices including comparison functions.

The temporal distortion analyzer 250 may receive information about a region in which the center of the original image has a large movement from the temporal cognitive sensitivity calculator 230 .

The temporal distortion information may include information on a region in which motion is large.

The temporal distortion analyzer 250 may calculate the degree of distortion with respect to the row plane of temporal frames.

Also, the temporal distortion analyzer 250 may calculate the degree of distortion in the column plane of temporal frames.

The row plane and column plane of the frames are described in more detail below with reference to FIGS. 8 and 9 .

The temporal distortion analyzer 250 calculates the degree of distortion with respect to the row plane and the degree of distortion with respect to the column plane by using information on the area in which the central motion of the original image is large, in a region where the center of the image has a lot of motion. can be given great weight.

The weighting will be described in more detail below with reference to FIG. 10 .

By assigning such weights, the degree of distortion of temporal specific information can be measured more accurately from a cognitive viewpoint.

The temporal distortion information may include row plane distortion information and column plane distortion information of the calculated temporal frames.

The temporal distortion analyzer 250 may transmit the calculated row plane distortion information and column plane distortion information of the temporal frames to the index calculator 160 .

The row plane distortion information may be information indicating a degree of distortion with respect to the row plane. The row plane distortion degree may be one in which the row plane distortion degree is converted into one representative number through a weighted average or a percentile. Alternatively, the degree of distortion with respect to the row plane may be used and transmitted as row plane distortion information in a planar form.

The column plane distortion information may be information indicating the degree of distortion with respect to the column plane. The degree of column plane distortion may be one in which the degree of distortion with respect to the column plane is converted into one representative number through a weighted average or percentile. Alternatively, the degree of distortion with respect to the column plane may be used and transmitted as column plane distortion information as it is in a planar form.

In operation 360 , the index calculator 260 may generate a picture quality index for subjective picture quality based on the temporal distortion information and the spatial distortion information.

The quality index may be a single number predicting subjective image quality.

The index calculation unit 260 may calculate a final image quality index by applying a weight based on an existing cognitive visual model or the like to temporal distortion information and/or spatial distortion information.

Alternatively, the indicator calculator 260 may learn a method of combining temporal distortion information and spatial distortion information by applying machine learning based on actual subjective image quality evaluation data.

For example, the spatial distortion information transmitted by the spatial distortion analysis unit 240 and the temporal distortion information transmitted by the temporal distortion analysis unit 250 determine the degree of distortion with respect to the feature information in the form of a numeric vector. In this case, the indicator calculator 260 may be trained to output the quality indicator using a neural network in the form of a fully connected (FC) layer.

For example, when the spatial distortion information transmitted by the spatial distortion analysis unit 240 and the temporal distortion information transmitted by the temporal distortion analysis unit 250 have the degree of distortion with respect to the feature information as a whole, index calculation The unit 260 may be trained to output a picture quality index using a neural network in the form of a convolutional neural network (CNN).

4 illustrates that a pixel is not determined as a strong HV pixel by a dominant angle of the pixel according to an example.

5 illustrates that a pixel is determined to be a strong HV pixel by a dominant angle of the pixel according to an example.

4 and 5, it is shown whether a pixel is judged to be a strong HV pixel depending on the dominant angle of the pixel.

For HV plane calculation, first, the image feature information extractor 210 may derive the size of the vertical component and the horizontal component of each pixel of the pixels of the image.

The image feature information extractor 210 may obtain a dominant angle of a pixel through an arctangent operation using a size of a vertical component and a size of a horizontal component of the pixel.

The image feature information extraction unit 210 may construct the HV plane by determining whether the obtained dominant angle θ is close to the vertical direction or the horizontal direction.

The image feature information extractor 210 may determine whether the pixel is a strong HV pixel by determining whether the obtained dominant angle θ is close to the vertical direction or the horizontal direction.

The image feature information extractor 210 may classify the pixel as a strong HV pixel when the dominant angle θ of the pixel is close to the vertical direction or the horizontal direction. For example, the image feature information extractor 210 may classify the pixel as a strong HV pixel when the dominant angle θ of the pixel is smaller than the threshold θ _threshold . For example, the image feature information extractor 210 may classify the pixel as not a strong HV pixel when the dominant angle θ of the pixel is equal to or greater than the threshold θ _threshold .

In obtaining the dominant angle of the pixel and determining whether the pixel is a strong HV pixel, the method shown in Equation 6 below may be used instead of the arc tangent operation, which has high computational complexity.

[Equation 6]

According to Equation 6, a larger value among I _H and I _V may be a denominator, and a smaller value may be a numerator. That is, irrespective of the vertical direction and the horizontal direction, a value serving as a criterion for judging a pixel may be a value in which a larger value among I _H and I _V is a denominator and a smaller value is a numerator.

Whether the horizontal component or the vertical component of the pixel is strong may be determined by comparing a value serving as a criterion for determining the pixel with tan(θ _threshold ) serving as a threshold value.

6 illustrates a comparison between the degree of distortion on the HV plane and actual perceived image quality according to an example.

In the case of an SI plane that calculates the degree of improvement or decrease in overall sharpness in an image, a numerical value calculated using the comparison functions described above with reference to step 340 of FIG. 8 may show a tendency similar to the perceived image quality. .

On the other hand, according to at least specific cases, in the case of the HV plane for measuring block artifact distortion, the numerical value calculated using the above-described comparison functions may be different from the actual cognitive distortion.

6 may be an example of a comparison between the degree of distortion on the HV plane and actual perceived image quality.

As shown in FIG. 6 , from a numerical point of view, it can be seen that distortion due to block artifacts is high in the area indicated by a yellow circle. However, from a perceptual point of view, distortion may be more prominent in the edges and texture areas within the flat background represented by the green circles than in the flat areas represented by the yellow circles.

In other words, in a flat area represented by a yellow circle, even if the degree of distortion is high, the perceived image quality may not be significantly deteriorated. In the edge and texture areas indicated by green circles, even if the degree of distortion is low, the perceived image quality may be greatly deteriorated.

As part of a solution to these examples, a region in which encoding distortion is conspicuous may be determined.

Referring to the above-described step 320 , the spatial perception sensitivity-based information may include information on a region where encoding distortion is conspicuous.

The spatial cognitive sensitivity calculation unit 220 may determine a region where encoding distortion such as block artifacts are conspicuous in the original image according to a cognitive viewpoint, and transmit information of the determined region to the spatial distortion analysis unit 240 . can

When considered from a spatial point of view, a region where distortion is conspicuous may be a region in which edges or textures are not random, and are flat or regular. When a structure such as an edge or a texture of an image is deformed due to encoding distortion in such an area, the structure deviates from a pattern expected from a perceptual point of view, so that the deformation of the structure is easily noticeable.

Accordingly, by discriminating regions that are not spatially random in an image, determining an edge or a texture region existing in the region, and giving a high weight, it is possible to further consider information loss on a cognitively important region.

The spatial randomness can be calculated for a pixel as in Equation 7 below.

[Equation 7]

Here, SR(u) may represent the degree of spatial randomness in the pixel X(u).

Y(u) may mean a set of neighboring pixels surrounding the pixel X(u).

R _xy may mean a cross-correlation matrix between X(u) and Y(u).

R _x ^-1 may be an inverse of a correlation matrix of X(u).

Through Equation 7 or the like, the spatial cognitive sensitivity calculator 220 may numerically determine the degree of spatial randomness in the input image in units of pixels. In addition, the spatial cognitive sensitivity calculator 220 may determine a flat and regular region having a low degree of randomness in the image through application of a threshold or the like.

The spatial cognitive sensitivity calculation unit 220 may complete the cognitive sensitivity-based map by finding the regions of edges and textures existing in the region based on the determined region having a low degree of randomness.

The spatial perception sensitivity calculator 220 may use a difference curvature as shown in Equations 8 and 9 below to determine the edge of the image and the area of the texture.

[Equation 8]

[Equation 9]

Here, G _x,i may mean a first gradient value in the x-axis direction of the i-th pixel in the image.

G _y,i may mean a primary gradient value in the y-axis direction of the i-th pixel in the image.

Here, G _xx,i may mean a secondary gradient value in the x-axis direction of the i-th pixel in the image.

G _yy,i may mean a secondary gradient value in the y-axis direction of the i-th pixel in the image.

G _xy,i may mean that the primary gradient in the y-axis direction is applied to the primary gradient value in the x-axis direction of the i-th pixel in the image.

N and E may mean values of thresholds experimentally derived for defining an image region.

IND may mean an impulse function that assigns a value of 1 to a case in which an inequality within the function is satisfied.

When the value of r _i for the i-th pixel is determined by Equation 8, the type of the area of the i-th pixel is determined as one of flat, edge, and texture according to the value determined by Equation 9 can be

The edge or texture region determined through Equation 8 (that is, the region of pixels in which the value of r ₁ is 1 or 2) is a structure such as an edge and a texture that exists on a flat and regular background, and encoding distortion occurs within the region. When it occurs, it can mean an easily visible area from a cognitive point of view.

As described above, the spatial cognitive sensitivity calculation unit 220 can determine a region where encoding distortion such as block artifacts are conspicuous in the original image according to the cognitive viewpoint, and the spatial distortion analysis unit uses the information of the determined region. It can be transmitted to (240).

An area where encoding distortion is conspicuous may include an edge or a texture area. As described above, the region where encoding distortion is conspicuous is transmitted from the spatial cognitive sensitivity calculator 220 to the spatial distortion analyzer 240 and is used to predict image quality degradation due to spatial distortion.

7 illustrates a method of determining a degree of distortion due to block artifacts according to a cognitive viewpoint according to an example.

7 exemplifies a method of deriving the degree of distortion on the HV plane by applying a weight based on cognitive sensitivity to the degree of distortion on the HV plane.

The spatial distortion analyzer 240 calculates the degree of distortion on the HV plane, and uses the information provided from the spatial cognitive sensitivity calculator 220 to give greater weight to the distortion in the edge and texture region on a flat background. have. Through this process, the degree of distortion due to block artifacts from a cognitive viewpoint can be measured more accurately.

8 illustrates a spatial frame and a temporal frame of an image according to an example.

9 illustrates a row plane among temporal frames according to an example.

As shown in FIG. 9 , a row plane and a column plane may largely exist in a temporal frame of an image.

The first row plane may be a plane in which first rows of frames are gathered as illustrated in FIG. 9 .

The first row plane may be a plane in which the first rows of frames are gathered as shown in FIG. 9 .

In order to reduce the amount of computation, the plane may be extracted by being spaced apart at a specified interval. R/8 may represent a specified interval. For example, R may be the number of frames.

10 illustrates applying a cognitive sensitivity-based weight to a degree of distortion with respect to a column plane of temporal frames according to an example.

In Fig. 10, the column plane is shown on the left, and the volume of motion weights is shown on the right.

The motion weight volume may be the product of frame-by-frame and center-biased Gaussian weights. In other words, the motion weight volume may be obtained by multiplying the residual image between adjacent frames described in step 330 by the weight function of the Gaussian distribution.

The volume of column planes and motion weights can be multiplied plane-wise. That is, the motion weight map for Difference _row1 and Difference _row1 among the column planes may be multiplied. Difference _row1 may be a difference in the first row.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium.

The computer-readable recording medium may contain information used in the embodiments according to the present invention. For example, the computer-readable recording medium may include a bitstream, and the bitstream may include the information described in the embodiments according to the present invention.

The computer-readable recording medium may include a non-transitory computer-readable medium.

The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

100: electronic device
110: processing unit
120: communication department
130: storage
210: image feature information extraction unit
220: spatial perception sensitivity calculation unit
230: temporal cognitive sensitivity calculation unit
240: spatial distortion analysis unit
250: temporal distortion analysis unit
260: indicator calculation unit

Claims (20)

a distortion analyzer for generating distortion information by using the information on the original image and the information on the comparison image; and
and an index calculator configured to generate an index for at least one of the original image and the comparison image based on the distortion information. According to claim 1,
The information on the original image includes spatial feature information of the original image,
The information on the comparison image includes spatial feature information of the comparison image. 3. The method of claim 2,
The distortion information is generated using spatial perception sensitivity-based information of the original image. According to claim 1,
The information on the original image includes temporal characteristic information of the original image,
The information on the comparison image includes temporal characteristic information of the comparison image. 5. The method of claim 4,
The distortion information is generated using information based on temporal perception sensitivity of the original image. According to claim 1,
The distortion information includes temporal distortion information and spatial distortion information. 7. The method of claim 6,
The indicator is an image quality indicator generated based on the temporal distortion information and the spatial distortion information. generating distortion information by using the information on the original image and the information on the comparison image; and
generating an index for at least one of the original image and the comparison image based on the distortion information
A method of deriving an indicator including 9. The method of claim 8,
The information on the original image includes spatial feature information of the original image,
The information on the comparison image includes spatial feature information of the comparison image. 10. The method of claim 9,
The distortion information is an index derivation method that is generated using spatial perception sensitivity-based information of the original image. 9. The method of claim 8,
The information on the original image includes temporal characteristic information of the original image,
The information on the comparison image is an indicator deriving method including temporal characteristic information of the comparison image. 12. The method of claim 11,
The distortion information is an indicator derivation method that is generated using information based on temporal perception sensitivity of the original image. 9. The method of claim 8,
The distortion information is an indicator derivation method including temporal distortion information and spatial distortion information. 14. The method of claim 13,
The index is an index derivation method that is a quality index generated based on the temporal distortion information and the spatial distortion information. A computer-readable recording medium for storing a bitstream for image decoding, the bitstream comprising:
information about the original video;
Information about comparison images
including,
Distortion information is generated using the information on the original image and the information on the comparison image,
A computer-readable recording medium generating an index for at least one of the original image and the comparison image based on the distortion information. 17. The method of claim 16,
The information on the original image includes spatial feature information of the original image,
The information on the comparison image is a computer-readable recording medium including spatial characteristic information of the comparison image. 17. The method of claim 16,
The distortion information is a computer-readable recording medium that is generated using spatial perception sensitivity-based information of the original image. 17. The method of claim 16,
The information on the original image includes temporal characteristic information of the original image,
The information on the comparison image is a computer-readable recording medium including temporal characteristic information of the comparison image. 19. The method of claim 18,
The distortion information is a computer-readable recording medium that is generated using information based on temporal perception sensitivity of the original image. 17. The method of claim 16,
The distortion information is a computer-readable recording medium including temporal distortion information and spatial distortion information.

Images