KR20210061298A - Method and apparatus for image processing and perceptual qualitty enhancement based on perceptual characteristic - Google Patents

Abstract

Disclosed are a method and an apparatus for image processing such as encoding and decoding of images. The image processing apparatus performs quantization based on the minimum just detectable difference (JND) on an input image, and performs the detection of a cognitively sensitive region, the detection of an image quality deterioration region, the determination of a cognitive deterioration region, and the improvement of perceived image quality within the input image. In detecting the cognitively sensitive region, at least one of randomness, masking characteristics and attention and concentration characteristics is used. In the detection of an image quality deterioration region, at least one of boundary strength information, JND level information, and edge information is used. In the improvement of perceived image quality, at least one of adjustment of quantization parameter values and denoising based on machine learning is used.

Description

Method and apparatus for image processing and cognitive quality improvement based on cognitive characteristics {METHOD AND APPARATUS FOR IMAGE PROCESSING AND PERCEPTUAL QUALITTY ENHANCEMENT BASED ON PERCEPTUAL CHARACTERISTIC}

The following embodiments relate to an image processing method and apparatus, and to a method and apparatus for processing image compression and improving perceived quality based on cognitive characteristics.

Through the continuous development of the information and communication industry, broadcast services with high definition (HD) resolution have spread worldwide. Through this proliferation, many users have become accustomed to high-resolution and high-definition images and/or videos.

In order to satisfy users' demands for high image quality, many organizations are spurring the development of next-generation imaging devices. Users' interest in High Definition TV (HDTV) and Full HD (FHD) TV, as well as Ultra High Definition (UHD) TV, which has a resolution of 4 times or more compared to FHD TV This has increased, and as interest increases, an image encoding/decoding technology for an image having a higher resolution and quality is required.

An image encoding/decoding apparatus and method is an inter prediction technology, an intra prediction technology, and a transform in order to perform encoding/decoding on a high-resolution and high-definition image. And a quantization technique and an entropy coding technique. The inter prediction technique may be a technique for predicting a value of a pixel included in a target picture using a temporally previous picture and/or a temporally later picture. The intra prediction technique may be a technique for predicting a value of a pixel included in a target picture using information on a pixel in the target picture. The transform and quantization technique may be a technique for compressing energy of a residual signal. The entropy encoding technique may be a technique of allocating a short code to a symbol with a high frequency of appearance and a long code to a symbol having a low frequency of appearance.

In image encoding, the concept of a minimum detectable difference (JND) or a minimum detectable threshold (JND) is used.

JND can refer to the point at which a person perceives a change or stimulus. In general, JND can mean the point at which 50% of people perceive a change.

In the JND used in image encoding, only the first JND position considering only the difference in perceptual quality between the source image and the target image is modeled and used. Accordingly, when a quantization value exceeds a threshold defined by the first JND in encoding an image, the effect of image compression using the JND may be insignificant. In addition, there may be little gain that can be obtained through JND from the point when the viewer feels the deterioration of image quality, and thus, there may be few coding bits that can be used to improve the image quality.

An embodiment may provide a method and an apparatus for image processing based on cognitive characteristics.

According to an exemplary embodiment, the compression rate of an image may be improved without deteriorating the perceived image quality by removing cognitive redundancy in the image compression process, and the perceived image quality of the compressed image may be improved.

An embodiment may provide a method and apparatus for processing image quantization using multi-level JND intervals and multi-level JND thresholds.

In one side, performing an alphabetization based on a minimum detectable difference (JND) with respect to the input image; And improving the perceived image quality on the input image.

In addition to this, another method, apparatus, and system for implementing the present invention, and a computer-readable recording medium for recording a computer program for executing the method are further provided.

A method and apparatus for image processing based on cognitive characteristics are provided.

According to an exemplary embodiment, the compression rate of an image may be improved without deteriorating the perceived image quality by removing cognitive redundancy in the image compression process, and the perceived image quality of the compressed image may be improved.

A method and apparatus for processing an image quantization using multi-level JND intervals and multi-level JND thresholds are provided.

1 is a block diagram showing a configuration according to an embodiment of an encoding apparatus to which the present invention is applied.
2 is a block diagram showing a configuration according to an embodiment of a decoding apparatus to which the present invention is applied.
3 is a structural diagram of an encoding apparatus according to an embodiment.
4 is a structural diagram of a decoding apparatus according to an embodiment.
5 illustrates a JND position and a JND section according to an example.
6 shows quantization using a JND threshold according to an example.
7 shows a JND threshold in a frequency domain according to an example.
8 is a flowchart of an image processing method according to an exemplary embodiment.
9 shows multi-level JND positions according to an example.
10 illustrates a first JND threshold according to a frequency coefficient position according to an example.
11 shows a second JND threshold according to a frequency coefficient position according to an example.
12 illustrates a machine learning-based network learning step according to an example.
13 illustrates grasping a JND section using a machine learning-based network according to an example.
14 shows 8x8 DCT coefficients of an input image according to an example.
15 illustrates an analysis of cognitive characteristics for an input image according to an example.
16 shows an example of a JND threshold determined according to cognitive characteristic analysis.
17 illustrates another example of a JND threshold according to cognitive characteristic analysis.
18 illustrates luminance and weights of blocks of a picture according to an example.
19 illustrates block types and weights of blocks of a picture according to an example.
20 illustrates a first JND threshold in a frequency domain according to an example.
21 illustrates a second JND threshold in a frequency domain according to an example.
22 shows additional quantization using a JND threshold in an example.
23 illustrates quantization values according to positions of frequency coefficients according to an example.
24 illustrates 8x8 DCT coefficients of a reconstructed image according to an example.
25 illustrates a process of improving cognitive image quality according to an example.
26 illustrates detection of randomness in units of pixels, according to an example.
27 illustrates detection of randomness in units of blocks, according to an example.
28 illustrates a model of a blocking artifact across a block boundary according to an example.
29 illustrates a process of determining a cognitive deterioration region according to an example.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to a specific embodiment, it should be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention.

For a detailed description of exemplary embodiments described below, reference is made to the accompanying drawings, which illustrate specific embodiments as examples. These embodiments are described in detail sufficient to enable a person skilled in the art to practice the embodiments. It should be understood that the various embodiments are different from each other but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present invention in relation to one embodiment. In addition, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the detailed description to be described below is not intended to be taken in a limiting sense, and the scope of exemplary embodiments, if appropriately described, is limited only by the appended claims, along with all scopes equivalent to those claimed by the claims.

Like reference numerals in the drawings refer to the same or similar functions over several aspects. The shapes and sizes of elements in the drawings may be exaggerated for clearer explanation.

In the present invention, terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. The term and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

When a component is referred to as being "connected" or "connected" to another component, the above two components may be directly connected or connected to each other, but the above 2 It should be understood that other components may exist in the middle of the components of the dog. When a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that no other component exists between the two components.

Components shown in the embodiments of the present invention are independently illustrated to represent different characteristic functions, and does not mean that each component is formed of separate hardware or a single software component unit. That is, each component is listed and included as each component for convenience of explanation, and at least two components of each component are combined to form a single component, or one component is divided into a plurality of components to perform a function. It is possible to perform, and integrated embodiments and separate embodiments of each of these components are also included in the scope of the present invention without departing from the essence of the present invention.

In addition, the description of "including" a specific configuration in the exemplary embodiments does not exclude configurations other than the specific configurations described above, and additional configurations are not limited to the implementation of the exemplary embodiments or the technical idea of the exemplary embodiments. It means that it can be included in the scope.

The terms used in the present invention are used only to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present invention, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or a combination thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof does not preclude in advance. That is, in the present invention, the description of "including" a specific configuration does not exclude configurations other than the corresponding configuration, and means that additional configurations may also be included in the scope of the implementation of the present invention or the scope of the technical idea of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings in order to enable those of ordinary skill in the art to easily implement the embodiments. In describing the embodiments, when it is determined that a detailed description of a related known configuration or function may obscure the subject matter of the present specification, a detailed description thereof will be omitted. In addition, the same reference numerals are used for the same elements in the drawings, and redundant descriptions of the same elements are omitted.

Hereinafter, an image may mean one picture constituting a video, and may represent a video itself. For example, "encoding and/or decoding of an image" may mean "encoding and/or decoding of a video", and may mean "encoding and/or decoding of one of the images constituting a video" May be.

Hereinafter, the terms "video" and "motion picture" may be used with the same meaning, and may be used interchangeably.

Hereinafter, the target image may be an encoding target image that is a target of encoding and/or a decoding target image that is a target of decoding. Also, the target image may be an input image input through an encoding device or an input image input through a decoding device.

Hereinafter, the terms "image", "picture", "frame" and "screen" may be used with the same meaning, and may be used interchangeably.

Hereinafter, the target block may be an encoding target block, which is an object of encoding, and/or a decoding object block, which is an object of decoding. In addition, the target block may be a current block that is currently a target of encoding and/or decoding. For example, the terms "target block" and "current block" may be used with the same meaning, and may be used interchangeably.

Hereinafter, the terms "block" and "unit" may be used with the same meaning, and may be used interchangeably. Or “block” may represent a specific unit.

Hereinafter, the terms "region" and "segment" may be used interchangeably.

Hereinafter, the specific signal may be a signal indicating a specific block. For example, the original signal may be a signal representing a target block. The prediction signal may be a signal representing a prediction block. The residual signal may be a signal indicating a residual block.

In embodiments, each of the specified information, data, flag and element, attribute, etc. may have a value. A value of "0" such as information, data, flags, elements, and attributes may represent a logical false or a first predefined value. That is to say, the value "0", false, logical false, and the first predefined value may be replaced with each other and used. A value of "1" such as information, data, flags, elements, attributes, etc. may represent a logical true or a second predefined value. That is to say, the value "1", true, logical true, and the second predefined value may be used interchangeably.

When a variable such as i or j is used to indicate a row, column, or index, the value of i may be an integer greater than or equal to 0, or may be an integer greater than or equal to 1. That is to say, in embodiments, rows, columns, and indexes may be counted from 0 and may be counted from 1.

In the following, terms used in the embodiments are described.

Encoder: refers to a device that performs encoding.

Decoder: refers to a device that performs decoding.

Unit: A unit may represent a unit for encoding and decoding an image. The terms "unit" and "block" may be used with the same meaning, and may be used interchangeably.

-The unit may be an MxN array of samples. Each of M and N may be a positive integer. A unit can often mean an arrangement of two-dimensional samples.

-In encoding and decoding of an image, a unit may be an area generated by division of one image. In other words, the unit may be a specified area within one image. One image may be divided into a plurality of units. Alternatively, the unit may mean the divided part when one image is divided into subdivided parts and encoding or decoding of the divided part is performed.

-In encoding and decoding of an image, a predefined processing for a unit may be performed according to the type of the unit.

-Depending on the function, the type of unit is a macro unit, a coding unit (CU), a prediction unit (PU), a residual unit, a transform unit (TU), etc. It can be classified as Alternatively, depending on the function, the unit is a block, a macroblock, a coding tree unit, a coding tree block, a coding unit, a coding block, and a prediction unit. It may mean a (Prediction Unit), a prediction block, a residual unit, a residual block, a transform unit, a transform block, and the like.

-A unit may mean information including a luma component block, a chroma component block corresponding thereto, and a syntax element for each block in order to distinguish it from a block.

-The size and shape of the unit can vary. In addition, the unit may have various sizes and various shapes. In particular, the shape of the unit may include not only a square, but also a geometric figure that can be expressed in two dimensions, such as a rectangle, a trapezoid, a triangle, and a pentagon.

-Also, the unit information may include at least one of a unit type, a unit size, a unit depth, a unit encoding order, and a unit decoding order. For example, the type of the unit may indicate one of CU, PU, residual unit and TU.

-One unit can be further divided into sub-units with a smaller size than the unit.

Depth: Depth may mean the degree of division of a unit. In addition, the unit depth may indicate the level at which the unit exists when the unit is expressed in a tree structure.

-The unit division information may include the depth related to the depth of the unit. Depth may indicate the number and/or degree to which a unit is divided.

-In the tree structure, the depth of the root node is the shallowest and the depth of the leaf node is the deepest.

-One unit may be hierarchically divided into a plurality of sub-units while having depth information based on a tree structure. That is to say, a unit and a sub-unit generated by the division of the unit may correspond to a node and a child node of the node, respectively. Each divided sub-unit can have a depth. Since the depth indicates the number and/or degree of division of the unit, the division information of the sub-unit may include information on the size of the sub-unit.

-In the tree structure, the highest node may correspond to the first undivided unit. The highest node may be referred to as a root node. Also, the highest node may have a minimum depth value. In this case, the uppermost node may have a depth of level 0.

-A node with a depth of level 1 may indicate a unit created as the first unit is divided once. A node with a depth of level 2 may represent a unit created as the first unit is divided twice.

-A node having a depth of level n may indicate a unit generated when the first unit is divided n times.

-The leaf node may be the lowest node and may be a node that cannot be further divided. The depth of the leaf node may be at the maximum level. For example, the predefined value of the maximum level may be 3.

-QT depth may indicate the depth for quad division. BT depth may indicate the depth for binary division. The TT depth may indicate the depth for the three-way division.

Sample: A sample may be a base unit constituting a block. A sample may be expressed as values ranging ^{from 0 to 2 Bd} -1 according to a bit depth (Bd).

-The sample can be a pixel or a pixel value.

-Hereinafter, the terms "pixel", "pixel" and "sample" may be used with the same meaning, and may be used interchangeably.

Coding Tree Unit (CTU): A CTU may be composed of one luminance component (Y) coding tree block and two chrominance component (Cb, Cr) coding tree blocks related to the luminance component coding tree block. have. In addition, the CTU may mean including the blocks and a syntax element for each block of the blocks.

-Each coding tree unit is a quad tree (QT), a binary tree (BT), and a ternary tree (TT) to construct sub-units such as a coding unit, a prediction unit, and a transform unit. It can be partitioned using one or more partitioning schemes.

-CTU may be used as a term for referring to a pixel block, which is a processing unit in a process of decoding and encoding an image, as in splitting an input image.

Coding Tree Block (CTB): A coding tree block may be used as a term for referring to any one of a Y coding tree block, a Cb coding tree block, and a Cr coding tree block.

Neighbor block: A neighboring block may mean a block adjacent to the target block. The neighboring block may mean a reconstructed neighboring block.

-Hereinafter, the terms "peripheral block" and "adjacent block" may be used with the same meaning, and may be used interchangeably.

Spatial neighbor block: The spatial neighboring block may be a block spatially adjacent to the target block. The neighboring blocks may include spatial neighboring blocks.

-The target block and the spatial neighboring block may be included in the target picture.

-Spatial neighboring blocks may refer to blocks whose boundaries are in contact with the target block or blocks located within a predetermined distance from the target block.

-Spatial neighboring block may mean a block adjacent to the vertex of the target block. Here, the block adjacent to the vertex of the target block may be a block vertically adjacent to a neighboring block horizontally adjacent to the target block or a block horizontally adjacent to a neighboring block vertically adjacent to the target block.

Temporal neighbor block: The temporal neighbor block may be a block that is temporally adjacent to the target block. The neighboring blocks may include temporal neighboring blocks.

-Temporal neighboring blocks may include co-located blocks (col blocks).

-The collocated block may be a block in a co-located picture (col picture) that has already been reconstructed. The position of the collocated block within the collocated picture may correspond to the position within the target picture of the target block. Alternatively, the position of the collocated block in the collocated picture may be the same as the position in the target picture of the target block. The collocated picture may be a picture included in the reference picture list.

-The temporal neighboring block may be a block that is temporally adjacent to the spatial neighboring block of the target block.

Prediction unit: may mean a base unit for prediction such as inter prediction, intra prediction, inter compensation, intra compensation, and motion compensation.

-One prediction unit may be divided into a plurality of partitions or sub prediction units having a smaller size. The plurality of partitions may also be a base unit for performing prediction or compensation. A partition generated by division of the prediction unit may also be a prediction unit.

Prediction unit partition: This may mean a form in which a prediction unit is divided.

Reconstructed neighboring unit: The reconstructed neighboring unit may be a unit that has already been decoded and reconstructed around the target unit.

-The reconstructed neighboring unit may be a spatial neighboring unit or a temporal neighboring unit to the target unit.

-The reconstructed spatial neighboring unit may be a unit within the target picture and already reconstructed through encoding and/or decoding.

-The reconstructed temporal neighboring unit may be a unit in the reference image and already reconstructed through encoding and/or decoding. The position of the reconstructed temporal neighboring unit in the reference image may be the same as the position in the target picture of the target unit or may correspond to the position in the target picture of the target unit.

Parameter set: The parameter set may correspond to header information among structures in the bitstream. For example, the parameter set may include a video parameter set, a sequence parameter set, a picture parameter set, and an adaptation parameter set.

In addition, the parameter set may include slice header information and tile header information.

Rate-distortion optimization: The encoding apparatus uses a combination of the size of the coding unit, the prediction mode, the size of the prediction unit, motion information, and the size of the transform unit to provide high coding efficiency. You can use distortion optimization.

-The rate-distortion optimization method can calculate the rate-distortion cost of each combination in order to select an optimal combination among the above combinations. Rate-distortion cost can be calculated using Equation 1 below. In general, a combination in which the rate-distortion cost is minimized may be selected as an optimal combination in the rate-distortion optimization method.

[Equation 1]

-D can represent distortion. D may be a mean square error of the difference values between the original transform coefficients and the reconstructed transform coefficients in the transform unit.

-R can represent a rate. R may represent a bit rate using related context information.

-λ can represent Lagrangian multiplier. R may include not only coding parameter information such as prediction mode, motion information, and coded block flag, but also bits generated by encoding transform coefficients.

-The encoding apparatus may perform processes such as inter prediction and/or intra prediction, transformation, quantization, entropy encoding, inverse quantization, and inverse transformation in order to calculate accurate D and R. These processes can greatly increase the complexity of the encoding device.

Bitstream: A bitstream may mean a sequence of bits including encoded image information.

Parameter set: The parameter set may correspond to header information among structures in the bitstream.

-The parameter set may include at least one of a video parameter set, a sequence parameter set, a picture parameter set, and an adaptation parameter set. In addition, the parameter set may include information on a slice header and information on a tile header.

Parsing: Parsing may mean determining a value of a syntax element by entropy decoding a bitstream. Alternatively, parsing may mean entropy decoding itself.

Symbol: It may mean at least one of a syntax element, a coding parameter, a transform coefficient, and the like of a coding target unit and/or a decoding target unit. Also, the symbol may mean an object of entropy encoding or a result of entropy decoding.

Reference picture: A reference picture may mean an image referenced by a unit for inter prediction or motion compensation. Alternatively, the reference picture may be an image including a reference unit referenced by the target unit for inter prediction or motion compensation.

Hereinafter, the terms "reference picture" and "reference image" may be used with the same meaning, and may be used interchangeably.

Reference picture list: The reference picture list may be a list including one or more reference pictures used for inter prediction or motion compensation.

-The types of reference picture lists are List Combined (LC), List 0 (List 0; L0), List 1 (List 1; L1), List 2 (List 2; L2), and List 3 (List 3; L3). ), etc.

-One or more reference picture lists may be used for inter prediction.

Inter prediction indicator: The inter prediction indicator may indicate the direction of inter prediction for the target unit. Inter prediction may be one of one-way prediction and two-way prediction. Alternatively, the inter prediction indicator may indicate the number of reference pictures used when generating the prediction unit of the target unit. Alternatively, the inter prediction indicator may mean the number of prediction blocks used for inter prediction or motion compensation for a target unit.

Reference picture index: The reference picture index may be an index indicating a specific reference picture in the reference picture list.

Motion Vector (MV): The motion vector may be a two-dimensional vector used for inter prediction or motion compensation. The motion vector may mean an offset between the target image and the reference image.

-For example, MV can be expressed in the form _{of (mv x} , mv _{y ).} mv _x may represent a horizontal component, and mv _y may represent a vertical component.

Search range: The search region may be a two-dimensional region in which MV is searched during inter prediction. For example, the size of the search area may be MxN. Each of M and N may be a positive integer.

Motion vector candidate: A motion vector candidate may mean a block as a prediction candidate or a motion vector of a block as a prediction candidate when predicting a motion vector.

-The motion vector candidate may be included in the motion vector candidate list.

Motion vector candidate list: The motion vector candidate list may mean a list constructed by using one or more motion vector candidates.

Motion vector candidate index: The motion vector candidate index may mean an indicator indicating a motion vector candidate in the motion vector candidate list. Alternatively, the motion vector candidate index may be an index of a motion vector predictor.

Motion information: Motion information includes not only motion vector, reference picture index and inter prediction indicator, but also reference picture list information, reference picture, motion vector candidate, motion vector candidate index, merge candidate and merge index, etc. It may mean information including at least one of.

Transform unit: The transform unit may be a basic unit in residual signal encoding and/or residual signal decoding such as transform, inverse transform, quantization, inverse quantization, transform coefficient encoding and transform coefficient decoding. One transform unit may be divided into a plurality of transform units having a smaller size.

Scaling: Scaling may mean a process of multiplying a transform coefficient level by a factor.

-As a result of scaling on the transform coefficient level, a transform coefficient may be generated. Scaling may also be referred to as dequantization.

Quantization Parameter (QP): The quantization parameter may mean a value used when generating a transform coefficient level for a transform coefficient in quantization. Alternatively, the quantization parameter may mean a value used when generating a transform coefficient by scaling a transform coefficient level in inverse quantization. Alternatively, the quantization parameter may be a value mapped to a quantization step size.

Delta quantization parameter: The delta quantization parameter means a differential value between a predicted quantization parameter and a quantization parameter of a target unit.

Scan: Scan may refer to a method of arranging the order of coefficients in a unit, block, or matrix. For example, arranging a two-dimensional array into a one-dimensional array may be referred to as a scan. Alternatively, arranging the one-dimensional array in the form of a two-dimensional array may also be referred to as a scan or an inverse scan.

Transform coefficient: The transform coefficient may be a coefficient value generated by performing transformation in the encoding apparatus. Alternatively, the transform coefficient may be a coefficient value generated by performing at least one of entropy decoding and inverse quantization in the decoding apparatus.

-A quantized level generated by applying quantization to a transform coefficient or a residual signal or a quantized transform coefficient level may also be included in the meaning of the transform coefficient.

Quantized level: The quantized level may mean a value generated by quantizing a transform coefficient or a residual signal in an encoding apparatus. Alternatively, the quantized level may mean a value that is an object of inverse quantization in performing inverse quantization in the decoding apparatus.

-The quantized transform coefficient level resulting from transform and quantization may also be included in the meaning of the quantized level.

Non-zero transform coefficient: The non-zero transform coefficient may mean a transform coefficient having a non-zero value or a transform coefficient level having a non-zero value. Alternatively, the non-zero transform coefficient may mean a transform coefficient in which the size of a value is not 0 or a transform coefficient level in which the size of a value is not zero.

Quantization matrix: The quantization matrix may mean a matrix used in a quantization process or an inverse quantization process in order to improve subjective or objective quality of an image. The quantization matrix may also be referred to as a scaling list.

Quantization matrix coefficient: The quantization matrix coefficient may mean each element in the quantization matrix. The quantization matrix coefficient may also be referred to as a matrix coefficient.

Default matrix: The default matrix may be a quantization matrix predefined by an encoding device and a decoding device.

Non-default matrix: The non-default matrix may be a quantization matrix that is not predefined in an encoding device and a decoding device. The non-default matrix may be signaled from the encoding device to the decoding device.

Signaling: Signaling may indicate that information is transmitted from an encoding device to a decoding device. Alternatively, signaling may mean including information in a bitstream or a recording medium. Information signaled by the encoding device may be used by the decoding device.

1 is a block diagram showing a configuration according to an embodiment of an encoding apparatus to which the present invention is applied.

The encoding device 100 may be an encoder, a video encoding device, or an image encoding device. A video may include one or more images. The encoding apparatus 100 may sequentially encode one or more images of a video.

Referring to FIG. 1, the encoding apparatus 100 includes an inter prediction unit 110, an intra prediction unit 120, a switch 115, a subtractor 125, a transform unit 130, a quantization unit 140, and entropy encoding. A sub 150, an inverse quantization unit 160, an inverse transform unit 170, an adder 175, a filter unit 180, and a reference picture buffer 190 may be included.

The encoding apparatus 100 may encode a target image using an intra mode and/or an inter mode.

Also, the encoding apparatus 100 may generate a bitstream including encoding information through encoding on a target image, and may output the generated bitstream. The generated bitstream may be stored in a computer-readable recording medium, and may be streamed through a wired/wireless transmission medium.

When an intra mode is used as the prediction mode, the switch 115 may be switched to intra. When the inter mode is used as the prediction mode, the switch 115 may be switched to inter.

The encoding apparatus 100 may generate a prediction block for a target block. Also, after the prediction block is generated, the encoding apparatus 100 may encode a residual between the target block and the prediction block.

When the prediction mode is an intra mode, the intra prediction unit 120 may use a pixel of an already coded/decoded block adjacent to the target block as a reference sample. The intra prediction unit 120 may perform spatial prediction for the target block using the reference sample, and may generate prediction samples for the target block through spatial prediction.

The inter prediction unit 110 may include a motion prediction unit and a motion compensation unit.

When the prediction mode is the inter mode, the motion prediction unit can search for an area that best matches the target block from the reference image in the motion prediction process, and derives a motion vector for the target block and the searched area using the searched area. can do.

The reference picture may be stored in the reference picture buffer 190, and may be stored in the reference picture buffer 190 when the reference picture is encoded and/or decoded.

The motion compensation unit may generate a prediction block for the target block by performing motion compensation using a motion vector. Here, the motion vector may be a 2D vector used for inter prediction. In addition, the motion vector may represent an offset between the target image and the reference image.

When the motion vector has a non-integer value, the motion prediction unit and the motion compensation unit may generate a prediction block by applying an interpolation filter to a partial region of the reference image. In order to perform inter prediction or motion compensation, a method of motion prediction and motion compensation of a PU included in the CU based on the CU is a skip mode, a merge mode, and an advanced motion vector prediction. Prediction (AMVP) mode and a current picture reference mode may be determined, and inter prediction or motion compensation may be performed according to each mode.

The subtractor 125 may generate a residual block, which is a difference between the target block and the prediction block. The residual block may also be referred to as a residual signal.

The residual signal may mean a difference between the original signal and the predicted signal. Alternatively, the residual signal may be a signal generated by transforming or quantizing, or transforming and quantizing a difference between the original signal and the predicted signal. The residual block may be a residual signal for each block.

The transform unit 130 may transform the residual block to generate a transform coefficient, and may output the generated transform coefficient. Here, the transform coefficient may be a coefficient value generated by performing transform on the residual block.

In performing the conversion, the conversion unit 130 may use one of a plurality of predefined conversion methods.

A plurality of predefined transformation methods may include Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), and Karhunen-Loeve Transform (KLT) based transformation. have.

The transform method used for transforming the residual block may be determined according to at least one of coding parameters for the target block and/or the neighboring block. For example, the transformation method may be determined based on at least one of an inter prediction mode for a PU, an intra prediction mode for a PU, a size of a TU, and a shape of a TU. Alternatively, transformation information indicating a transformation method may be signaled from the encoding device 100 to the decoding device 200.

When the transform skip mode is applied, the transform unit 130 may omit the transform of the residual block.

By applying quantization to a transform coefficient, a quantized transform coefficient level or a quantized level may be generated. Hereinafter, in embodiments, a quantized transform coefficient level and a quantized level may also be referred to as transform coefficients.

The quantization unit 140 may generate a quantized transform coefficient level or a quantized level by quantizing a transform coefficient according to a quantization parameter. The quantization unit 140 may output the generated quantized transform coefficient level or the generated quantized level. In this case, the quantization unit 140 may quantize the transform coefficient using a quantization matrix.

The entropy encoding unit 150 may generate a bitstream by performing entropy encoding according to a probability distribution based on values calculated by the quantization unit 140 and/or coding parameter values calculated during an encoding process. . The entropy encoder 150 may output the generated bitstream.

The entropy encoder 150 may perform entropy encoding on information about pixels of an image and information for decoding an image. For example, information for decoding an image may include a syntax element or the like.

When entropy coding is applied, a small number of bits may be allocated to a symbol having a high probability of occurrence, and a large number of bits may be allocated to a symbol having a low probability of occurrence. As symbols are represented through such allocation, the size of a bitstring for symbols to be encoded may be reduced. Accordingly, compression performance of image encoding may be improved through entropy encoding.

In addition, the entropy encoding unit 150 includes exponential golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding for entropy encoding. A coding method such as Arithmetic Coding (CABAC) can be used. For example, the entropy encoding unit 150 may perform entropy encoding using a Variable Length Coding/Code (VLC) table. For example, the entropy encoder 150 may derive a binarization method for a target symbol. In addition, the entropy encoder 150 may derive a probability model of a target symbol/bin. The entropy encoding unit 150 may perform arithmetic encoding using the derived binarization method, a probability model, and a context model.

The entropy encoder 150 may change a coefficient of a form of a two-dimensional block into a form of a one-dimensional vector through a transform coefficient scanning method in order to encode the quantized transform coefficient level.

The coding parameter may be information required for encoding and/or decoding. The coding parameter may include information that is encoded by the encoding device 100 and transmitted from the encoding device 100 to the decoding device, and may include information that can be inferred during an encoding or decoding process. For example, as information transmitted to the decoding device, there is a syntax element.

The coding parameter, like a syntax element, may include information (or flags, indexes, etc.) that is encoded by the encoding device and signaled from the encoding device to the decoding device, as well as information derived during the encoding process or the decoding process. have. In addition, the coding parameter may include information required for encoding or decoding an image. For example, the size of the unit/block, the depth of the unit/block, the division information of the unit/block, the division structure of the unit/block, information indicating whether the unit/block is divided into a quad tree, and the unit/block is binary. Information indicating whether the tree is divided, the direction of division of the binary tree type (horizontal or vertical direction), the division type of the binary tree type (symmetrical division or asymmetrical division), and whether the unit/block is divided into a ternary tree shape. Information represented, split direction in the form of a ternary tree (horizontal or vertical direction), prediction method (intra prediction or inter prediction), intra prediction mode/direction, reference sample filtering method, prediction block filtering method, prediction block boundary filtering method, filtering Filter tap, filter coefficient of filtering, inter prediction mode, motion information, motion vector, reference picture index, inter prediction direction, inter prediction indicator, reference picture list, reference picture, motion vector predictor, motion vector prediction candidate, motion vector candidate List, information indicating whether to use merge mode, merge candidate, merge candidate list, information indicating whether to use skip mode, type of interpolation filter, filter tap of interpolation filter, filter coefficient of interpolation filter, motion Vector size, motion vector expression accuracy, transform type, transform size, information indicating whether to use a first-order transform, information indicating whether to use an additional (quaternary) transform, a first-order transform index, a second-order transform index, and residuals Information indicating the presence or absence of a signal, a coded block pattern, a coded block flag, a quantization parameter, a quantization matrix, information about an intra-loop filter, whether or not an intra-loop filter is applied. Information, intra-loop filter coefficients, intra-loop filter taps, intra-loop filter shape/form, information indicating whether to apply a deblocking filter, deblocking filter coefficients, deblocking filter Tap, deblocking filter strength, deblocking filter shape/type Information indicating whether to apply adaptive sample offset, adaptive sample offset value, adaptive sample offset category, adaptive sample offset type, whether to apply adaptive in-loop filter, adaptive In-loop filter coefficient, adaptive intra-loop filter tap, adaptive intra-loop filter shape/shape, binarization/inverse binarization method, context model, context model determination method, context model update method, whether to perform regular mode, Whether to perform bypass mode, context bin, bypass bin, transform coefficient, transform coefficient level, transform coefficient level scanning method, image display/output order, slice identification information, slice type, slice division information, tile identification information, At least one of a tile type, tile split information, picture type, bit depth, information on a luminance signal, and information on a color difference signal, a combined form, or statistics may be included in the coding parameter. The prediction method may represent one of an intra prediction mode and an inter prediction mode.

The residual signal may represent a difference between the original signal and the predicted signal. Alternatively, the residual signal may be a signal generated by transforming a difference between the original signal and the predicted signal. Alternatively, the residual signal may be a signal generated by transforming and quantizing a difference between the original signal and the predicted signal. The residual block may be a residual signal for the block.

Here, signaling the flag or index means that the encoding device 100 transmits the entropy-encoded flag or entropy-encoded index generated by performing entropy encoding on the flag or index in a bitstream. It may mean to include, and the decoding apparatus 200 may mean obtaining a flag or index by performing entropy decoding on an entropy-encoded flag or entropy-encoded index extracted from the bitstream. .

Since encoding through inter prediction is performed by the encoding apparatus 100, the encoded target image may be used as a reference image for other image(s) to be processed later. Accordingly, the encoding apparatus 100 may reconstruct or decode the encoded target image again, and store the reconstructed or decoded image in the reference picture buffer 190 as a reference image. Inverse quantization and inverse transformation of a target image encoded for decoding may be processed.

The quantized level may be inversely quantized in the inverse quantization unit 160 and may be inversely transformed in the inverse transform unit 170. The inverse quantized and/or inverse transformed coefficient may be summed with the prediction block through the adder 175, and a reconstructed block may be generated by summing the inverse quantized and/or inverse transformed coefficient and the prediction block. Here, the inverse quantized and/or inverse transformed coefficient may mean a coefficient in which at least one of dequantization and inverse-transformation has been performed, and may mean a reconstructed residual block.

The reconstructed block may pass through the filter unit 180. The filter unit 180 reconstructs at least one of a deblocking filter, a sample adaptive offset (SAO), and an adaptive loop filter (ALF). It can be applied to a picture. The filter unit 180 may also be referred to as an in-loop filter.

The deblocking filter may remove block distortion occurring at the boundary between blocks. In order to determine whether to apply the deblocking filter, it may be determined whether to apply the deblocking filter to the target block based on the pixel(s) included in several columns or rows included in the block.

When the deblocking filter is applied to the target block, the applied filter may be different according to the required strength of the deblocking filtering. In other words, a filter determined according to the strength of the deblocking filtering among different filters may be applied to the target block. When the deblocking filter is applied to the target block, one of a strong filter and a weak filter may be applied to the target block according to the required strength of the deblocking filtering.

In addition, when vertical filtering and horizontal filtering are performed on the target block, horizontal filtering and vertical filtering may be processed in parallel.

SAO may add an appropriate offset to a pixel value of a pixel to compensate for a coding error. The SAO may perform correction using an offset for a difference between the original image and the image to which the deblocking is applied in a pixel unit of the deblocking image. To perform offset correction for an image, a method of dividing pixels included in an image into a certain number of areas, determining an area to be offset among the divided areas, and applying an offset to the determined area can be used. In addition, a method of applying an offset in consideration of edge information of each pixel of an image may be used.

The ALF may perform filtering based on a value obtained by comparing the reconstructed image and the original image. After dividing the pixels included in the image into predetermined groups, a filter to be applied to each divided group may be determined, and filtering may be performed differentially for each group. For the luminance signal, information related to whether to apply the adaptive loop filter may be signaled for each CU. The shape and filter coefficient of ALF to be applied to each block may be different for each block. Alternatively, regardless of the characteristics of the block, a fixed ALF may be applied to the block.

The reconstructed block or reconstructed image that has passed through the filter unit 180 may be stored in the reference picture buffer 190. The reconstructed block that has passed through the filter unit 180 may be a part of the reference picture. In other words, the reference picture may be a reconstructed picture composed of reconstructed blocks that have passed through the filter unit 180. The stored reference picture can be used for inter prediction later.

2 is a block diagram showing a configuration according to an embodiment of a decoding apparatus to which the present invention is applied.

The decoding device 200 may be a decoder, a video decoding device, or an image decoding device.

Referring to FIG. 2, the decoding apparatus 200 includes an entropy decoding unit 210, an inverse quantization unit 220, an inverse transform unit 230, an intra prediction unit 240, an inter prediction unit 250, and a switch 245. , An adder 255, a filter unit 260, and a reference picture buffer 270 may be included.

The decoding apparatus 200 may receive a bitstream output from the encoding apparatus 100. The decoding apparatus 200 may receive a bitstream stored in a computer-readable recording medium, and may receive a bitstream streamed through a wired/wireless transmission medium.

The decoding apparatus 200 may perform intra mode and/or inter mode decoding on a bitstream. In addition, the decoding apparatus 200 may generate a reconstructed image or a decoded image through decoding, and may output the generated reconstructed image or a decoded image.

For example, switching to an intra mode or an inter mode according to a prediction mode used for decoding may be performed by the switch 245. When the prediction mode used for decoding is the intra mode, the switch 245 may be switched to the intra mode. When the prediction mode used for decoding is the inter mode, the switch 245 may be switched to the inter mode.

The decoding apparatus 200 may obtain a reconstructed residual block by decoding the input bitstream, and may generate a prediction block. When the reconstructed residual block and the prediction block are obtained, the decoding apparatus 200 may generate a reconstructed block to be decoded by adding the reconstructed residual block and the prediction block.

The entropy decoder 210 may generate symbols by performing entropy decoding on a bitstream based on a probability distribution on the bitstream. The generated symbols may include a symbol in the form of a quantized transform coefficient level. Here, the entropy decoding method may be similar to the entropy encoding method described above. For example, the entropy decoding method may be a reverse process of the entropy encoding method described above.

The entropy decoder 210 may change a coefficient in the form of a one-dimensional vector into a form of a two-dimensional block through a transform coefficient scanning method in order to decode the quantized transform coefficient level.

For example, the coefficients may be changed into a 2D block shape by scanning the coefficients of the block using the upper right diagonal scan. Alternatively, it may be determined which of the upper-right diagonal scan, vertical scan, and horizontal scan will be used according to the size of the block and/or the intra prediction mode.

The quantized coefficient may be inverse quantized by the inverse quantization unit 220. The inverse quantization unit 220 may generate an inverse quantized coefficient by performing inverse quantization on the quantized coefficient. In addition, the inverse quantized coefficient may be inversely transformed by the inverse transform unit 230. The inverse transform unit 230 may generate a reconstructed residual block by performing an inverse transform on an inverse quantized coefficient. As a result of performing inverse quantization and inverse transformation on the quantized coefficients, a reconstructed residual block may be generated. In this case, the inverse quantization unit 220 may apply a quantization matrix to the quantized coefficients in generating the reconstructed residual block.

When the intra mode is used, the intra prediction unit 240 may generate a prediction block by performing spatial prediction using pixel values of an already decoded block around the target block.

The inter prediction unit 250 may include a motion compensation unit. Alternatively, the inter prediction unit 250 may be referred to as a motion compensation unit.

When the inter mode is used, the motion compensation unit may generate a prediction block by performing motion compensation using a motion vector and a reference image stored in the reference picture buffer 270.

When the motion vector has a non-integer value, the motion compensation unit may apply an interpolation filter to a partial region of the reference image and may generate a prediction block using the reference image to which the interpolation filter is applied. The motion compensation unit may determine which of a skip mode, a merge mode, an AMVP mode, and a current picture reference mode is the motion compensation method used for the PU included in the CU based on the CU to perform motion compensation, and the determined mode According to the motion compensation can be performed.

The reconstructed residual block and prediction block may be added through an adder 255. The adder 255 may generate a reconstructed block by adding the reconstructed residual block and the prediction block.

The reconstructed block may pass through the filter unit 260. The filter unit 260 may apply at least one of the deblocking filter, SAO, and ALF to the reconstructed block or the reconstructed image. The reconstructed image may be a picture including a reconstructed block.

The reconstructed image that has passed through the filter unit 260 may be output by the encoding device 100 and may be used by the encoding device 100.

The reconstructed image that has passed through the filter unit 260 may be stored as a reference picture in the reference picture buffer 270. The reconstructed block that has passed through the filter unit 260 may be a part of the reference picture. In other words, the reference picture may be an image composed of reconstructed blocks that have passed through the filter unit 260. The stored reference picture can then be used for inter prediction.

3 is a structural diagram of an encoding apparatus according to an embodiment.

The encoding device 300 may correspond to the encoding device 100 described above.

The encoding device 300 includes a processing unit 310, a memory 330, a user interface (UI) input device 350, a UI output device 360, and a storage that communicate with each other through a bus 390. It may include 340. In addition, the encoding apparatus 300 may further include a communication unit 320 connected to the network 399.

The processing unit 310 may be a central processing unit (CPU), a semiconductor device that executes processing instructions stored in the memory 330 or the storage 340. The processing unit 310 may be at least one hardware processor.

The processing unit 310 may generate and process signals, data, or information input to the encoding device 300, output from the encoding device 300, or used inside the encoding device 300. Inspection, comparison, and judgment related to data or information can be performed. That is, in the embodiment, the generation and processing of data or information, and inspection, comparison, and determination related to the data or information may be performed by the processing unit 310.

The processing unit 310 includes an inter prediction unit 110, an intra prediction unit 120, a switch 115, a subtractor 125, a transform unit 130, a quantization unit 140, an entropy encoding unit 150, and inverse quantization. A unit 160, an inverse transform unit 170, an adder 175, a filter unit 180, and a reference picture buffer 190 may be included.

Inter prediction unit 110, intra prediction unit 120, switch 115, subtractor 125, transform unit 130, quantization unit 140, entropy encoding unit 150, inverse quantization unit 160, At least some of the inverse transform unit 170, the adder 175, the filter unit 180, and the reference picture buffer 190 may be program modules and may communicate with an external device or system. Program modules may be included in the encoding apparatus 300 in the form of an operating system, an application program module, and other program modules.

Program modules may be physically stored on various known storage devices. In addition, at least some of these program modules may be stored in a remote storage device capable of communicating with the encoding device 300.

Program modules are routines, subroutines, programs, objects, components, and data that perform functions or operations according to an embodiment or implement abstract data types according to an embodiment. The structure (data structure) may be included, but is not limited thereto.

The program modules may be composed of instructions or codes executed by at least one processor of the encoding apparatus 300.

The processing unit 310 includes an inter prediction unit 110, an intra prediction unit 120, a switch 115, a subtractor 125, a transform unit 130, a quantization unit 140, an entropy encoding unit 150, and inverse quantization. Commands or codes of the unit 160, the inverse transform unit 170, the adder 175, the filter unit 180, and the reference picture buffer 190 may be executed.

The storage unit may represent the memory 330 and/or the storage 340. The memory 330 and the storage 340 may be various types of volatile or nonvolatile storage media. For example, the memory 330 may include at least one of a ROM 331 and a RAM 332.

The storage unit may store data or information used for the operation of the encoding device 300. In an embodiment, data or information of the encoding device 300 may be stored in the storage unit.

For example, the storage unit may store pictures, blocks, lists, motion information, inter prediction information, and bitstreams.

The encoding apparatus 300 may be implemented in a computer system including a recording medium that can be read by a computer.

The recording medium may store at least one module required for the encoding apparatus 300 to operate. The memory 330 may store at least one module, and at least one module may be configured to be executed by the processing unit 310.

A function related to communication of data or information of the encoding device 300 may be performed through the communication unit 320.

For example, the communication unit 320 may transmit the bitstream to the decoding device 400 to be described later.

4 is a structural diagram of a decoding apparatus according to an embodiment.

The decoding device 400 may correspond to the decoding device 200 described above.

The decoding apparatus 400 includes a processing unit 410, a memory 430, a user interface (UI) input device 450, a UI output device 460, and a storage that communicate with each other through a bus 490. (440) may be included. In addition, the decoding apparatus 400 may further include a communication unit 420 connected to the network 499.

The processing unit 410 may be a central processing unit (CPU), a semiconductor device that executes processing instructions stored in the memory 430 or the storage 440. The processing unit 410 may be at least one hardware processor.

The processing unit 410 may generate and process signals, data, or information input to the decoding device 400, output from the decoding device 400, or used inside the decoding device 400. Inspection, comparison, and judgment related to data or information can be performed. That is to say, in the embodiment, generation and processing of data or information, and inspection, comparison, and determination related to data or information may be performed by the processing unit 410.

The processing unit 410 includes an entropy decoding unit 210, an inverse quantization unit 220, an inverse transform unit 230, an intra prediction unit 240, an inter prediction unit 250, a switch 245, an adder 255, and a filter. It may include a sub 260 and a reference picture buffer 270.

Entropy decoding unit 210, inverse quantization unit 220, inverse transform unit 230, intra prediction unit 240, inter prediction unit 250, switch 245, adder 255, filter unit 260 and At least some of the reference picture buffers 270 may be program modules and may communicate with an external device or system. Program modules may be included in the decoding device 400 in the form of an operating system, an application program module, and other program modules.

Program modules may be physically stored on various known storage devices. In addition, at least some of these program modules may be stored in a remote storage device capable of communicating with the decoding device 400.

Program modules are routines, subroutines, programs, objects, components, and data that perform functions or operations according to an embodiment or implement abstract data types according to an embodiment. The structure (data structure) may be included, but is not limited thereto.

The program modules may be composed of instructions or codes executed by at least one processor of the decoding apparatus 400.

The processing unit 410 includes an entropy decoding unit 210, an inverse quantization unit 220, an inverse transform unit 230, an intra prediction unit 240, an inter prediction unit 250, a switch 245, an adder 255, and a filter. Commands or codes of the unit 260 and the reference picture buffer 270 may be executed.

The storage unit may represent the memory 430 and/or the storage 440. The memory 430 and the storage 440 may be various types of volatile or nonvolatile storage media. For example, the memory 430 may include at least one of a ROM 431 and a RAM 432.

The storage unit may store data or information used for the operation of the decoding apparatus 400. In an embodiment, data or information of the decoding apparatus 400 may be stored in the storage unit.

For example, the storage unit may store pictures, blocks, lists, motion information, inter prediction information, and bitstreams.

The decoding apparatus 400 may be implemented in a computer system including a recording medium that can be read by a computer.

The recording medium may store at least one module required for the decoding apparatus 400 to operate. The memory 430 may store at least one module, and at least one module may be configured to be executed by the processing unit 410.

A function related to communication of data or information of the decoding device 400 may be performed through the communication unit 420.

For example, the communication unit 420 may receive a bitstream from the encoding device 300.

Image compression based on cognitive characteristics

In an embodiment to be described later, a method of improving the compression rate without deteriorating the perceived image quality by reducing cognitively unnecessary cognitive redundancy in image encoding by using human cognitive characteristics will be described. In addition, in an embodiment to be described later, a method of improving the perceived image quality using coding bits saved by reducing cognitive redundancy will be described.

In an embodiment, not only a first JND defining a difference between the source image and the target image, but also JNDs of various levels capable of defining a difference between the reconstructed image generated in the process of compressing the image and the target image may be defined. According to this definition, even when an image reconstructed in the process of compressing an image exceeds the threshold defined at JND level 1, the compression rate can be improved by using the JND threshold defined at another JND level.

In an embodiment, cognitive quality may be improved by using the saved coding bits by reducing cognitive redundancy.

In addition, according to an embodiment, the compression rate may be improved without deteriorating the perceived image quality by removing cognitive redundancy in the process of compressing an image.

5 illustrates a JND position and a JND section according to an example.

In general, according to the perceived visual characteristics of a person regarding the distortion of the image quality, as shown in FIG. 5, even if the distortion value changes within a specific bit rate range, a person perceives the same distortion as an image of the same quality. .

The JND position generally represents the point at which a person begins to feel the difference from the object being compared.

The first JND position refers to a point at which a person starts to feel a difference or change compared to the original image.

The second JND position refers to a point at which a person feels a difference or change again compared to the image at the first JND position.

In other words, the first JND position and the second JND position may mean a point at which a person feels a change in cognitive distortion.

The JND section may be a section of a distortion value that a person recognizes as the same image even if the distortion value changes. In other words, even if the bit rate or distortion value of the image changes within one JND section, a person perceives it as an image of the same quality.

The JND position appears differently depending on the characteristics of the video. Modeling for the first JND position has been widely conducted.

6 shows quantization using a JND threshold according to an example.

The JND threshold may mean a distortion value that causes the original image or the comparison target image to reach a specific JND position.

The JND threshold may be a distortion value of a size that most people do not feel a difference in perceived image quality when a specific distortion value is subtracted or applied in the spatial or frequency domain of the input image.

For example, as illustrated in FIG. 6, when a value corresponding to the first JND threshold is subtracted or applied to the image, distortion of the image may reach the first JND position. That is, the first JND threshold may be a value that causes the distortion of the image to reach the first JND position when it is subtracted from the image or applied to the image.

That is, as shown in FIG. 6, when a value smaller than the first JND threshold is subtracted or applied to the image, the distortion value measured in the traditional method for the image may change, but the perceived distortion value of the image does not change. I can.

For example, the traditional method may include Sum of Absolute Difference (SAD) and Mean Squared Error (MSE).

The JND threshold may vary depending on the cognitive characteristics of the image. The JND threshold model used for compression is mainly made by subtracting a specific distortion from the input image to reduce information (ie, perceived redundancy).The size of the frequency value in the frequency domain is reduced by the JND threshold, while maintaining the image quality. A method of reducing encoding bits for an encoded image may be used.

7 shows a JND threshold in a frequency domain according to an example.

7 illustrates JND thresholds in the frequency domain. Even if the JND threshold corresponding to the frequency coefficient of the video is reduced in the video, the viewer can hardly feel the change in the perceived quality of the input video.

As described above, in image compression, in general, only the first JND threshold that minimizes the difference in perceived quality between the original image and the compressed image may be used.

When the value of the quantization parameter (QP) in the image compression process exceeds the first JND threshold, a difference between the quality of the reconstructed image and the quality of the original image occurs, and the first JND threshold is not useful. I can lose.

Even in this case, if the quality of the reconstructed image becomes the criterion of the final image quality, the JND section to which the quality of the reconstructed image belongs is derived, and additional quantization using the JND threshold for the derived JND section is performed. Can be applied to the compression of. Through this compression, the compression rate can be improved.

In embodiments, a method of modeling the JND threshold and JND positions of various levels such as the second level and the third level, as well as the first JND threshold and the first JND position using human cognitive characteristics, and the input image and reconstruction A method of determining the JND section of the recorded video is presented.

In addition, in the embodiments, an improvement in perceived quality of a compressed image is described.

In embodiments, through detection of the cognitive sensitive region and the image quality deterioration region, the region requiring the improvement of the perceived image quality is determined, the cognitive sensitive region and the image quality deterioration region are divided, and various cognitive factors required in determining each region Are explained.

In embodiments, a method of applying a technique for improving perceived image quality is described. Perceptual quality can be improved by using the coding bits saved through JND quantization. Methods for improving the perceived quality in encoding are described.

8 is a flowchart of an image processing method according to an exemplary embodiment.

The image processing method of the embodiment may be regarded as an image encoding method based on cognitive characteristics or an image compression method based on cognitive characteristics.

The processing unit 310 may be a quantization unit 140.

In steps 810 and 820, the processing unit 310 may perform JND-based quantization on the input image.

In step 810, the processor 310 may perform modeling on the multilevel JND thresholds and the multilevel JND intervals.

The multi-level JND intervals may mean a plurality of JND intervals for a plurality of JNDs, such as a second JND and a third JND, as well as the first JND.

The multilevel JND thresholds may mean a plurality of thresholds leading to a plurality of intervals or JND positions for a plurality of JNDs.

The JND position and the JND section can be determined based on the point at which people perceive deterioration of the picture quality through subjective picture quality evaluation, and the JND threshold can be made in the form of modeling the difference between the JND position or the video belonging to the JND section and the original video. have.

In an embodiment, the JND threshold may be determined in consideration of a person's perception characteristic of an image, and the size of the JND threshold may vary according to the characteristics of the image.

In one embodiment, the processor 310 determines a JND model by performing modeling on multilevel JND thresholds and multilevel JND intervals using at least one of 1) subjective image quality evaluation and mathematical modeling, and 2) machine learning. I can.

In an embodiment, the processor 310 may determine the JND model by performing modeling on multilevel JND thresholds and multilevel JND intervals using subjective image quality evaluation and mathematical modeling.

In an embodiment, the processor 310 may determine the JND model by performing modeling on multilevel JND thresholds and multilevel JND intervals using machine learning.

In step 820, the processor 310 may perform a suppression on the input image using the determined JND model.

In step 820, the processing unit 310 may analyze the cognitive characteristics of the input image.

The input image may be a reconstructed image.

In step 820, the processor 310 may derive a difference value between the input image and the original image.

In an embodiment, the processor 310 may calculate a difference value between the input image and the original image by using a sum of difference values between pixels of the input image and pixels of the original image.

In an embodiment, the processor 310 may calculate a difference value between the input image and the original image by using a weighted sum of difference values between the pixels of the input image and the pixels of the original image. .

The processor 310 determines which JND section and which JND threshold among the multilevel JND sections and multilevel JND thresholds modeled in step 810 by using the difference value between the derived input image and the original image. You can decide.

In determining the JND section for the input image, the JND section and the JND threshold corresponding to the input image may vary according to the cognitive characteristics of the input image.

For example, even when two reconstructed images have the same difference values from the original image, JND intervals and JND thresholds corresponding to the input images may be different according to cognitive characteristics of the input images.

In an embodiment, the processor 310 may determine a JND section and a JND threshold corresponding to the input image by using the contrast sensitivity characteristic of the input image.

In an embodiment, the processor 310 may determine a JND section and a JND threshold corresponding to the input image by using the masking characteristic of the input image.

In an embodiment, the processor 310 may determine a JND section and a JND threshold corresponding to the input image by using the attention and concentration characteristics of the input image.

In an embodiment, the processor 310 may determine a JND section and a JND threshold corresponding to the input image using edge and texture information of the input image.

In an embodiment, in performing reduction using the determined JND model, the processor 310 may use at least one of reduction through pre-processing of an original input image and reduction of a residual image.

In an embodiment, in performing the reduction using the determined JND model, the processor 310 may use the reduction through pre-processing of the original input image.

In one embodiment, the processing unit 310 may use the reduction of the residual image in performing reduction using the determined JND model. The processing unit 310 performs quantization on the input image using the determined JND threshold. can do.

The above quantization may be additional quantization for an input image that is a reconstructed image.

As additional quantization is performed, the distortion value of the input image may be moved to the JND position of the next level of the input image, and through this movement, the bit rate may be reduced while maintaining the perceived image quality.

The decoding of the reconstructed image generated by the above-described quantization may be performed by the decoding apparatus 200. In addition, inverse quantization corresponding to the above-described quantization may be performed by the inverse quantization unit 220 of the decoding apparatus 200.

In an embodiment, in performing reduction using the determined JND model, the processor 310 may perform reduction on a residual signal.

In steps 830, 840, and 850, the processing unit 310 may improve the perceived quality of the input image.

In step 830, the processor 310 may detect a cognitive sensitive region in the input image.

In an embodiment, the processor 310 may use at least one of 1) randomness, 2) masking characteristics, and 3) attention and concentration characteristics in detecting a cognitive sensitive region.

In an embodiment, the processing unit 310 may use randomness in detecting a cognitive sensitive region.

In an embodiment, the processing unit 310 may use a masking characteristic in detecting a cognitive sensitive region.

In an embodiment, the processing unit 310 may use attention and concentration characteristics in detecting a cognitive sensitive region.

In step 840, the processing unit 310 may detect an image quality deterioration region in the input image.

In step 840, the processor 310 may detect the image quality deterioration region using at least one of 1) boundary strength (BS) information, 2) JND level information, and 3) edge information. .

In an embodiment, the processing unit 310 may detect an image quality deterioration region using BS information.

In an embodiment, the processing unit 310 may detect an image quality deterioration region using the JND level information.

In an embodiment, the processing unit 310 may detect an image quality deterioration region using edge information.

In step 850, the processor 310 may determine a perceived deterioration region of the input image.

In step 860, the processor 310 may improve the perceived image quality of the input image.

In an embodiment, the processor 310 may improve the perceived image quality by using at least one of 1) adjusting a QP value and 2) removing noise based on machine learning.

In an embodiment, the processor 310 may improve the perceived image quality by using the adjustment of the QP value.

In an embodiment, the processor 310 may improve cognitive image quality using noise removal based on machine learning.

9 shows multi-level JND positions according to an example.

Modeling of the multi-level JND thresholds and multi-level JND intervals of the above-described step 810 will be described with reference to FIG. 8.

In step 810, the processing unit 310 may perform modeling of the JND interval, the JND position, and the JND threshold. The above modeling can be divided into 1) a method using subjective image quality evaluation and mathematical modeling, and 2) a method using machine learning.

How to use subjective picture quality assessment and mathematical modeling

The processing unit 310 may generate various images while increasing the size of the deterioration of the original image. The processor 310 may derive an image corresponding to each JND position of the multi-level JND positions corresponding to the original image through subjective image quality evaluation of the generated images.

For example, the multi-level JND positions may include a first JND position, a second JND position, and a third position.

The processing unit 310 may actually determine a difference value reaching each JND position through analysis of the difference between the image corresponding to each JND position and the original image.

For example, the difference value for the JND position may be simply a sum of difference values in the spatial domain and the frequency domain between the original image and the reconstructed image corresponding to the JND position. In addition, the sum of these difference values may be defined as a JND threshold leading to a JND position and a JND interval.

The sum of the difference values in the spatial domain between the image A and the image B may be defined as in Equation 2 below.

[Equation 2]

The sum of the difference values in the frequency domain between the image A and the image B may be defined as in Equation 3 below.

[Equation 3]

MATD can represent the mean of absoluted transformed differences.

SATD can represent the sum of absoluted transformed differences.

SSTE may represent the sum of squared transformed errors.

Diff ( i , j ) can be defined as in Equation 4 below.

[Equation 4]

DiffT(i, j) can be defined as in Equation 5 below.

i and j may represent coordinates of pixels of an image in a spatial domain. The image (i, j) may represent a value of a pixel whose coordinates in the image are (i, j).

[Equation 5]

i and j may represent coordinates of pixels of an image in the frequency domain.

10 illustrates a first JND threshold according to a frequency coefficient position according to an example.

11 shows a second JND threshold according to a frequency coefficient position according to an example.

For the frequency domain, a difference between an original image transformed for each frequency coefficient and a deteriorated image corresponding to a specific JND may be modeled. For each JND position, a difference value between the frequency coefficient of the original image and the frequency coefficient between the deteriorated image may be set as a JND threshold for reaching the JND position and the JND section of the specific JND.

For example, the difference between the original image and corresponding coefficients in the frequency domain of the image corresponding to a specific JND position may be modeled as shown in FIGS. 10 and 11 through mathematical modeling such as a simple average or linear regression.

10 may represent a difference between corresponding coefficients of an original image and an image corresponding to a first JND position.

11 may represent a difference between corresponding coefficients of an original image and an image corresponding to a second JND position.

This difference can be seen as the JND threshold corresponding to the JND position.

If the difference between the corresponding coefficients in the frequency domain of the original image and the deteriorated image (that is, the reconstructed image) is smaller than the first JND threshold defined above, the quality or distortion of the deteriorated image is the first JND section. It can be seen that it is within.

How to use machine learning

12 illustrates a machine learning-based network learning step according to an example.

The processing unit 310 may generate various images while increasing the size of the deterioration of the original image. The processor 310 may derive an image corresponding to each JND position of the multi-level JND positions corresponding to the original image through subjective image quality evaluation of the generated images.

For example, the multi-level JND positions may include a first JND position, a second JND position, and a third position.

The processing unit 310 may include a machine learning-based network or may operate a machine learning-based network.

The machine learning-based network can perform various machine learning using the original image and images corresponding to multi-level JND positions.

13 illustrates grasping a JND section using a machine learning-based network according to an example.

When the machine-learning-based network performs learning as described with reference to FIG. 12, the machine-learning-based network displays the input image among multi-level JND positions and multi-level JND intervals for an image input to the machine learning-based network The corresponding JND position and/or JND section can be classified and output.

Determination of JND threshold and JND section through characteristic analysis of input image

14 shows 8x8 DCT coefficients of an input image according to an example.

15 illustrates an analysis of cognitive characteristics for an input image according to an example.

In step 820 described above with reference to FIG. 8, the processor 310 may derive a difference value between the input image and the original image.

The processing unit 310 may calculate a difference value to find the JND section of the input image (ie, the reconstructed image), and may use the following methods.

The processor 310 may find the JND section of the input image by using the sum of the difference values.

When the input image is the image A and the original image is the image B, the sum of the difference values in the spatial domain between the image A and the image B may be defined as in Equation 6 below.

[Equation 6]

The sum of the difference values in the frequency domain between the image A and the image B may be defined as in Equation 7 below.

[Equation 7]

MATD can represent the mean of absoluted transformed differences.

SATD can represent the sum of absoluted transformed differences.

SSTE may represent the sum of squared transformed errors.

Diff ( i , j ) can be defined as in Equation 8 below.

[Equation 8]

DiffT(i, j) can be defined as in Equation 9 below.

i and j may represent coordinates of pixels of an image in a spatial domain. The image (i, j) may represent a value of a pixel whose coordinates in the image are (i, j).

[Equation 9]

i and j may represent coordinates of pixels of an image in the frequency domain.

The processor 310 may find the JND section of the input image by using the sum to which the difference values are weighted.

The processing unit 310 may add a weight item to the equations of Equations 5 to 9 described above, and may define a sum to which the weights of the difference values are assigned.

For example, when a difference value in the frequency domain is used, a smaller weight may be given to the difference value in the high frequency domain than the difference value in the low frequency domain based on the fact that the human is insensitive to high-frequency changes due to human perception characteristics. I can.

A weighted sum in the spatial domain between the image A and the image B may be defined as in Equation 10 below.

[Equation 10]

A weighted sum in the frequency domain between the image A and the image B may be defined as in Equation 11 below.

[Equation 11]

16 shows an example of a JND threshold determined according to cognitive characteristic analysis.

17 illustrates another example of a JND threshold according to cognitive characteristic analysis.

In step 820 described with reference to FIG. 8, the processor 310 uses the size of the difference value between the derived input image and the original image to determine the multi-level JND sections and multi-level JND sections modeled in step 810. Among the level JND thresholds, it is possible to determine which JND interval and which JND threshold corresponds to.

The processing unit 310 may calculate the JND threshold by analyzing the cognitive characteristics of the input image, and may determine the JND section of the input image according to the calculated threshold. In this manner, the JND interval and the JND threshold may be determined differently according to various cognitive characteristics.

The JND threshold JND _Threshold through the analysis of the cognitive characteristics of the input image can be expressed in the form of a product of JND _{Basic and weights.}

JND _Basic may be a default value of the JND threshold derived through subjective image quality evaluation.

The weights may be values derived by analyzing various cognitive characteristics such as contrast sensitivity, masking characteristics, attention and concentration characteristics of the input image.

For example, the JND threshold JND _Threshold may be expressed in the form of Equation 12 below.

[Equation 12]

JND _Basic is a default value of the JND threshold obtained through subjective quality evaluation, and may be a threshold matrix.

W _Contrast is a weight obtained in consideration of the contrast sensitivity characteristic of the input image, and may be a weight matrix.

W _Masking is a weight obtained in consideration of the masking characteristic of the input image, and may be a weight matrix.

W _Attention is a weight obtained in consideration of the attention and concentration characteristics of the input image, and may be a weight matrix.

For example, when the masking characteristic of the input image is large, the value of the weight W _Masking for the masking characteristic may increase, and the JND threshold increases as the value of W _{Masking increases.}

The main cognitive characteristics used in the analysis of the cognitive characteristics of the input image

The main cognitive characteristics used in the analysis of the cognitive characteristics of the input image are described below.

1) Contrast sensitivity characteristics: According to the cognitive characteristics of the human eye, the human eye can generally have a high sensitivity to changes in contrast for regions with low spatial frequencies, and contrast for regions with high spatial frequencies. May have low sensitivity to change.

That is, people may feel more sensitively to a change in a low-frequency component than a change in a high-frequency component in the frequency domain, and may hardly feel a change in contrast at a frequency above a specified frequency size.

Therefore, when calculating the JND threshold using this cognitive characteristic, different weights may be assigned to different frequency domains. In addition, the size of the weight multiplied by the JND threshold may vary according to the size of the sum of the weights of the frequency domains with respect to the difference value. In other words, the size of the weight multiplied by the JND threshold may be determined based on the size of the sum of the weights of the frequency domains with respect to the difference value.

2) Contrast masking characteristic: The contrast masking characteristic may be a term indicating that a person's cognitive characteristics are changed in response to a change in distortion or a size of distortion according to an image characteristic. In general, a change in distortion in a flat area can be recognized well according to a person's cognitive characteristics, but on the contrary, a change in distortion may not be recognized well in a textured area.

Accordingly, in calculating a distortion value based on such cognitive characteristics, weights for blocks may be different for each block according to texture and/or edge characteristics of blocks. In other words, the weight for the block may be determined based on the texture characteristic and/or the boundary characteristic of the block.

3) Temporal masking characteristic: The temporal masking characteristic may be a term indicating a phenomenon in which the recognition rate for distortion occurring in a fast moving object decreases as the temporal frequency of an image increases according to a person's cognitive characteristics. Based on this cognitive characteristic, weights for blocks may be different for each block according to a time frequency and a motion size in an image. In other words, the weight for the block may be determined based on the time frequency and the size of the motion in the image.

4) Attention and concentration characteristics: Attention and concentration characteristics may be terms that refer to a phenomenon in which a person's cognitive visual system focuses on or ignores a specific environment or object. Attention and concentration characteristics can be largely classified into two as follows according to a factor that attracts attention. In an embodiment, the weight imposed may be changed according to whether the attention and concentration factors are included in the image. In other words, the weight may be determined based on whether attention and concentration factors are included in the image.

4-1) Bottom-up attention (or, reflexive attention or exogenous attention)

The factor of bottom-up attention can be referred to as a low-level salient feature/raw sensory input.

The factor of bottom-up attention can refer to a factor that causes the shift of attention to a trait of potentially significant importance to occur rapidly or unconsciously.

Bottom-up attention may refer to attracting attention with a sharp change in factors such as color, shape, movement, contrast, and size, rather than a factor created with special intention.

4-2) Top-down attention (or voluntary attention or endogenous attention)

The factor of top-down attention may be a purpose-oriented cognitive factor such as signs and sign language, and may mean a factor that attracts attention by prior knowledge or specific expectations.

Assigning weights according to cognitive characteristics

In the following, specific methods of assigning weights using the above-described cognitive characteristics will be described.

1) How to use the contrast sensitivity feature

The weight may be assigned according to the size of the spatial frequency in consideration of the cognitive characteristics in the frequency space.

The cognitively sensitive area may be determined based on a contrast sensitivity function (CSF) as shown in Equation 13 below.

[Equation 13]

ω _i,j may represent the spatial frequency magnitude at coordinates (i, j) in the frequency domain.

a , b and c may be predefined constants. For example, a may be 1.33, b may be 0.11, and c may be 0.1.

The 8x8 matrix H _i,j ( ω _i,j ) of Equation 14 below may be a result derived from Equation 13 above, normalized to a maximum value.

[Equation 14]

The 8x8 matrix H _i,j ( ω _i,j ) may be a weight matrix based on Equation 13, and may be a contrast sensitivity weight matrix.

As shown in the matrix H _i,j ( ω _i,j ), elements belonging to positions having a relatively smaller frequency size may have relatively large values.

The contrast sensitivity value derived as the matrix H _i,j ( ω _i,j ) can be used as the weight.

Equation 13 is an example for obtaining the contrast sensitivity, and may be used to determine the contrast sensitivity of another type of diet reflecting the contrast sensitivity characteristic.

The contrast sensitivity weighting matrix, such as the matrix H _i,j ( ω _i,j ), may be a weighting matrix multiplied by a threshold matrix of basic JND thresholds.

2) How to use masking properties

The masking effect may mean a phenomenon in which a recognition rate for another signal or other stimulus decreases, or another signal or other stimulus is not recognized at all by a specific signal or a specific stimulus.

In the spatial domain, the masking effect may mean a phenomenon in which an error (that is, a signal) occurring in an area having a complex texture becomes more difficult to recognize than an error occurring in a smooth area.

In the time domain, the masking effect may mean a phenomenon in which the recognition rate for an error occurring in the above frames decreases as the difference in luminance between successive frames increases.

In an embodiment, the weight may be determined using characteristics of luminance masking, contrast masking, and temporal masking.

18 illustrates luminance and weights of blocks of a picture according to an example.

2-1) Determination of weights using luminance masking characteristics

As illustrated in FIG. 18, the input image may be divided into a plurality of blocks. In an embodiment, the input signal may be a block.

Each block may have an average luminance value, and a weight may be assigned to the block according to the average luminance value of the block.

Considering the luminance adaptation characteristic of the block, a high weight may be given to a cognitively sensitive area.

As shown in Equation 15 below, a cognitively sensitive area may be determined based on the average luminance value of the block. In addition, a weight for a block may be determined according to the range of the calculated average luminance value.

The weight for the block may be determined as in Equation 15 below.

[Equation 15]

는 블록의 평균 휘도 값(average intensity value)일 수 있다.

May be an average intensity value of the block.

ω may be the weight for the block. ω _i,j may be a weight matrix for the block.

In constructing a weight or weight matrix for calculating distortion, a small weight may be assigned to a block when the average luminance value of a block falls within a range of a dark or bright area.

In Equation 15, when ω _i,j is a weight or weight matrix, when the average luminance value of the block belongs to the middle region (that is, when the average luminance value of the block is greater than 60 and less than 170), the weight for the block Can be 1. When the average luminance value of the block does not belong to the middle region, the weight may be defined by an equation in Equation 15. For example, when the average luminance value of the block does not belong to the middle region, the weight for the block may be less than 1.

Equation 15 may be an example of determining a weight for a block. The method of determining the weight for the block may be defined as another type of equation reflecting the luminance adaptation characteristic.

2-2) Determination of weights using contrast masking characteristics

19 illustrates block types and weights of blocks of a picture according to an example.

As illustrated in FIG. 19, the input image may be divided into a plurality of blocks. In an embodiment, the input signal may be a block.

A weight may be assigned to a block according to the contrast masking characteristic of the block. A high weight may be assigned to a cognitively sensitive area in consideration of the contrast masking characteristic of the block.

The size of the average edge pixel density may be defined, and a cognitively sensitive area may be determined based on the size of the average edge pixel density. The boundary pixel density may indicate the number of boundary pixels of a block or may be proportional to the number of boundary pixels. Also, the boundary pixel density may be in inverse proportion to the size of the block.

는 아래의 수식 16과 같이 결정될 수 있다.Boundary pixel density of the average of the block

Can be determined as in Equation 16 below.

[Equation 16]

The boundary pixel may be detected through a boundary pixel detection operator such as a sobel or canny.

에 기반하여 결정될 수 있다. 계산된 평균의 경계 픽셀 밀집도

가 속하는 범위에 따라서, 블록의 블록 타입이 결정될 수 있다. 블록의 블록 타입에 기반하여 블록에 적용되는 가중치 또는 가중치 행렬이 결정될 수 있다.The block type of the block is the density of the boundary pixels of the block.

Can be determined based on Boundary pixel density of the calculated average

The block type of the block may be determined according to the range to which is belong. A weight applied to a block or a weight matrix may be determined based on the block type of the block.

The block type of the block may be determined as shown in Equation 17 below.

[Equation 17]

α and β can be real values.

및 기정의된 값들 간의 비교의 결과에 의해 플레인(plane), 경계(edge) 및 텍스처(texture) 중 하나로 분류될 수 있다.The block type of the block is the average boundary pixel density, as illustrated in Equation 17.

And a plane, an edge, and a texture based on a result of comparison between predefined values.

That is, in constructing the weight matrix, if there are many textures in a block that is an input signal, the recognition rate for image quality deterioration or distortion may be lowered. In this case, it is determined that a person reacts cognitively less sensitively to image quality degradation or distortion in the block, so that a lower weight may be set for the block. Conversely, if there are few textures in the block as the input signal, a high weight may be set for the block.

The weight for the block may be set as shown in Equation 18 below.

[Equation 18]

In Equation 18, the size of the block may be 8x8.

ω may be the weight for the block. Hω _i,j may be a weight matrix for the block.

Equation 18 may be an example in which the weight of the block is determined according to the density of the boundary pixel.

When the block type is a texture, when the number of boundary pixels of the block exceeds a specified number (eg, 16), it may be determined as a block with a boundary. When the block type is a texture, when the number of boundary pixels of the block is less than or equal to the specified number, it may be determined as a block without a boundary.

When the block type is a texture, a higher weight may be given to a block with a boundary compared to a block without a boundary. In other words, if the number of boundary pixels of a block whose block type is determined to be a texture is less than or equal to the specified number, the block can be recognized as a texture block having a relatively low boundary pixel density, and a relatively low boundary pixel density. As it is recognized as a texture block having a, a low weight may be given.

Equations 17 and 18 may be examples of determining a block type for a block and a weight according to the block type. A block type for a block and a method of determining a weight according to the block type may be defined as another type of equation reflecting the contrast masking characteristic.

2-3) Determination of weights using time masking characteristics

The input image may be divided into a plurality of blocks. In an embodiment, the input signal may be a block.

A weight may be assigned to a block according to the time masking characteristic of the block. A high weight may be assigned to a cognitively sensitive area in consideration of the temporal masking characteristic of the block.

A cognitively sensitive region may be determined through a correlation between a frame rate representing a time frequency characteristic and a motion vector of a block.

That is, in constructing the weight matrix, as the frame rate of the input image and the absolute size of the motion vector of the block increase, the recognition rate for image quality deterioration or distortion may decrease. In this case, it is determined that a person reacts cognitively less sensitively to image quality degradation or distortion in the block, so that a lower weight may be set for the block. Conversely, as the frame rate of the input image and the absolute size of the motion vector of the block are smaller, a higher weight may be set for the block.

Table 1 shows an example of a weight according to a frame rate and an absolute size of a motion vector of a block.

[Table 1]

Table 1 may indicate a weight for a block determined according to a frame rate and an absolute size of a motion vector of the block.

As shown in Table 1, as the frame rate increases, it may be determined that the recognition rate for distortion or screen deterioration decreases, and according to this determination, a smaller weight may be assigned to the block. In addition, as the absolute size of the motion vector increases, it may be determined that the recognition rate for distortion or screen deterioration decreases, and according to this determination, a smaller weight may be assigned to the block.

Table 1 may be an example of determining a weight according to a frame rate and an absolute size of a motion vector using a temporal masking characteristic. The method of determining the weight according to the frame rate and the absolute size of the motion vector may be defined as another type of equation reflecting the temporal masking characteristic, or may be expressed as a different value.

3) Determination of weights using attention and concentration characteristics

The weight may be determined differently depending on whether factors such as color, shape, movement, contrast, and size that attract attention due to a sudden change or prominence of factors are not generated in the input image with special intention.

As a purpose-oriented cognitive factor, such as a sign or sign language, in the input image, a weight may be determined differently depending on whether there are factors that attract attention by prior knowledge or a specific expectation.

For example, when such factors appear in an image, these factors may be regarded as cognitively and visually important factors, and a low weight may be given to the image according to these considerations. When a low weight is given, the JND threshold may be lowered, and the size of additional quantization using the JND threshold may be reduced.

4) Determination of weight using boundary/texture information

In the frequency domain, the directionality of the dominant coefficients can appear a lot as vertically shaped boundaries and textures in the spatial domain (ie, the spatial domain). That is, when a large number of vertical boundary components appear in the spatial domain, a large coefficient value (ie, a strong coefficient) with respect to the horizontal component may be expressed in the frequency domain.

Using this characteristic, a weight multiplied by a basic JND threshold may be determined differently according to a boundary and a texture direction in a spatial domain.

For example, when a vertical component appears strong in a spatial domain, small weights may be given to coefficients corresponding to a horizontal position in the frequency domain. In other words, when the vertical component of the coefficients of the image appears strongly in the spatial domain, the JND threshold model can reduce the threshold for the horizontal component in the frequency domain.

Through this process, when quantization using the JND threshold, less quantization can be applied to coefficients corresponding to the horizontal component in the frequency domain. By applying less quantization to the coefficients corresponding to the horizontal component in the frequency domain, the quantization is smaller for the vertical component in the spatial domain, and the deterioration of the image quality on the vertical component can be reduced.

20 illustrates a first JND threshold in a frequency domain according to an example.

21 illustrates a second JND threshold in a frequency domain according to an example.

The first JND threshold and the second JND threshold in the frequency domain may be derived through various criteria and principles as described above.

The JND threshold may be the same as a quantization value for further quantization.

The JND thresholds shown in FIGS. 20 and 21 are a result of multiplying the weights according to cognitive characteristics of the input image and the reconstructed image, and the sizes of the JND thresholds may be different from each other.

22 shows additional quantization using a JND threshold in an example.

As shown in FIG. 22, additional quantization using the JND threshold may be performed on the encoded image, and the bit rate for the encoded image may be reduced while the perceived image quality is maintained by the additional quantization using the JND threshold. have.

23 illustrates quantization values according to positions of frequency coefficients according to an example.

24 illustrates 8x8 DCT coefficients of a reconstructed image according to an example.

The DCT coefficients of the reconstructed image of FIG. 24 may be a difference between the DCT coefficients of the input image of FIG. 14 and quantization values according to positions of the frequency coefficients of FIG. 23.

In the positive encoding process, the processor 310 may calculate a difference value to determine the JND section of the reconstructed image, and compare the calculated difference value with the JND threshold.

As described above, the difference value for comparison with the JND threshold may be a simple sum of coefficients or a weighted sum. JND thresholds can also be expressed as one value as they are summed in the same way.

When the summed difference values are smaller than the sum of the first JND thresholds, the image may belong to the first JND interval, and the first JND threshold may be used as an additional quantization value for the image.

When the summed difference values are greater than the sum of the first JND thresholds and smaller than the sum of the second JND thresholds, the image may belong to the second JND interval, and the second JND threshold may be used as an additional quantization value for the image. .

25 illustrates a process of improving cognitive image quality according to an example.

The image of the original is shown in the upper left of FIG. 25.

In the middle of the upper part of FIG. 25, a map of a cognitive sensitive region is shown.

In the upper right of FIG. 25, a deterioration region map generated by performing CU-based encoding is illustrated.

A cognitive deterioration area map is shown in the lower right of FIG. 25.

In the lower left of FIG. 25, an image according to the application of the image quality enhancement technique based on the perceived image quality degradation map is shown.

Detection of cognitive sensitive areas using randomness

26 illustrates detection of randomness in units of pixels, according to an example.

27 illustrates detection of randomness in units of blocks, according to an example.

In step 830 described above with reference to FIG. 8, the processing unit 310 may detect a cognitive sensitive region in the input image.

The cognitive sensitive area may mean a cognitively sensitive area. In general, when image quality deterioration occurs in a cognitive sensitive region, people may react more sensitively to the image quality deterioration. In determining the cognitive sensitive region, an index for determining a region requiring improved image quality may be used.

In an embodiment, the processing unit 310 may use randomness in detecting a cognitive sensitive region.

Randomness can be a measure of how irregular a given area is.

Randomness may be a value calculated by measuring how well the current area can be restored from the surrounding area. Here, the peripheral area may be a block or a pixel.

In general, in terms of cognitive vision, people may be poorly aware of or tend to ignore deterioration appearing in objects or areas with large randomness. Since the perception sensitivity to distortion appearing in a region having low randomness is high, the region randomness may be considered preferentially in determining the image quality enhancement region.

Randomness can be detected in units of pixels or blocks.

Detection of cognitive sensitive areas using masking characteristics

In step 830 described above with reference to FIG. 8, the processing unit 310 may use the masking characteristic in detecting the cognitive sensitive region.

As described above, various weight values for an image or a transformed image may be defined based on the masking characteristic. Here, the defined weight value may mean that the larger the weight value, the lower the cognitive sensitivity. Accordingly, an area having a low weight can be determined as a cognitive sensitive area.

Detection of cognitive sensitive areas using attention and concentration characteristics

In step 830 described above with reference to FIG. 8, the processing unit 310 may use attention and concentration characteristics in detecting the cognitive sensitive region.

Like the masking characteristic, a cognitively conspicuous region may be determined as a cognitive sensitive region based on the above-defined attention and concentration characteristics.

Detection of image quality deterioration areas

In step 840 described above with reference to FIG. 8, the processing unit 310 may detect an image quality deterioration region in the input image.

Detection of image quality deterioration regions using boundary strength (BS)

28 illustrates a model of a blocking artifact across a block boundary according to an example.

The BS may be used to detect blocking artifacts between blocks generated due to transformation and quantization, and may be used in a deblocking filter.

As illustrated in Table 2 below, block modes, conditions according to the value of BS, and pixels to be modified (ie, to be filtered), the larger the value of BS, the greater the blocking artifacts occurring between blocks. I can.

[Table 2]

Accordingly, in detecting the image quality deterioration region, the processing unit 310 may designate a unit having a value less than or equal to the predefined BS value as the image quality deterioration region. Here, the unit may be one or more (adjacent) pixels and/or one or more (adjacent) blocks, and the unit may have a specific shape such as a square or a rectangle.

Detection of image quality deterioration areas using JND level information

As described above in the definition of JND, in general, the larger the JND level of the image, the greater the deterioration of the image may be. Accordingly, when the JND level of the specified region of the image is greater than or equal to the predefined level, the processor 310 may determine that noticeable deterioration has occurred in the region and may designate the region as a deterioration region. Here, the area may be the unit described above.

The JND level of an image or region can be determined as follows.

1) The processing unit 310 may determine the JND level of an image or region through a machine learning discriminator based on machine learning learned using a database in which information on various JND levels is defined.

2) The processing unit 310 may determine the JND level of the image or region using the JND threshold modeled through subjective image quality evaluation.

The modeled JND threshold JND _k ( i , j ) can be used using Equation 19 below.

[Equation 19]

K may represent the JND level.

i and j may represent the positions of the coefficients in the frequency domain.

Δ c ( i , j ) may be a difference value between coefficients of change of the original image and the deteriorated image (or reconstructed image).

JND _k ( i , j ) may be a JND threshold at the JND level k. That is, JND ₁ ( i , j ) may be a threshold value of positions of i and j in the first JND threshold model.

를 JND 레벨 k에서의 인지 오차 발생률이라고 간주하면, 처리부(310)는

가 1을 초과할 경우 해당하는 JND 레벨이 도과한 것으로 판단할 수 있다. 즉,

가 1을 초과하고

가 1의 이하인 경우, 해당하는 영상 또는 영역의 JND 레벨은 2가 될 수 있다.In Equation 19,

Considering the rate of occurrence of the recognition error at the JND level k, the processing unit 310

If is greater than 1, it can be determined that the corresponding JND level has exceeded. In other words,

Is greater than 1 and

When is less than or equal to 1, the JND level of the corresponding image or region may be 2.

Detection of image quality deterioration areas using main edge information

In general, people can react more sensitively to changes in vertical and/or horizontal edges than to changes in diagonal edges (i.e., oblique effect). Therefore, if the horizontal and/or vertical edges are prominent in the image or region, and the corresponding edge component disappears in the reconstructed image or region, the processing unit 310 converts the image or region into a deteriorated region. You can decide.

For example, the processing unit 310 may determine a value of the size of the edge of the block in the horizontal direction and the vertical direction through the edge operator, and when the value is greater than or equal to a specified reference value, the block is horizontally and/or It can be determined that the vertical edge component is present. The processor 310 may determine that deterioration has occurred in the block when the size of the corresponding horizontal and/or vertical edge in the reconstructed image that has undergone transformation and quantization decreases beyond a reference value.

29 illustrates a process of determining a cognitive deterioration region according to an example.

In FIG. 29, a process of determining a cognitive deterioration region using information about randomness and BS is illustrated.

The image of the original is shown on the upper left of FIG. 29.

A randomness map is shown in the middle of the top of FIG. 29.

In the upper right of FIG. 29, a BS map generated by performing CU-based encoding is shown.

A cognitive deterioration area map is shown in the lower part of FIG. 29.

In determining the cognitive deterioration region, the processing unit 310 uses the information on the cognitive deterioration region and the information on the deterioration region as described above, when deterioration occurs in the cognition sensitive region, the cognitive deterioration region in which the deterioration has occurred as a cognitive deterioration region You can decide.

Improvement of cognitive image quality

In step 860 described above with reference to FIG. 8, the processing unit may improve the perceived image quality.

The processing unit 310 may improve the picture quality of the cognitive degradation region by using the coding bits saved through the JND reduction described above.

Improvement of perceived quality through adjustment of QP value

In an embodiment, the processor 310 may use the saved coding bits to adjust the value of QP.

In encoding an image, the processor 310 may improve the perceived quality of the cognitive degradation region by allocating a lower QP than the predetermined QP to the cognitive degradation region.

In determining the value of the QP, the processing unit 31 may 1) allocate a specified value to the QP, 2) adjust the value of the QP until the JND level in the cognitive deterioration area is lowered, and 3) the phenomenon of cognitive deterioration. You can adjust the value of QP until disappears. In this case, such adjustment may be limited so that the number of bits increased by adjusting the value of QP is less than the number of bits saved through JND reduction.

Improvement of cognitive image quality through noise removal based on machine learning

Currently, various types of machine learning-based noise removal techniques are being proposed. The processor 310 may apply a noise removal technique based on machine learning to improve perceived image quality.

The processing unit 310 may generate a noise removal neural network based on machine learning optimized for the learning data by using the determined cognitive deterioration region as training data in the application of the noise removal technique based on machine learning. In addition, a noise canceling neural network can be used to improve cognitive quality.

In the above-described embodiments, the methods are described on the basis of a flow chart as a series of steps or units, but the present invention is not limited to the order of steps, and certain steps may occur in a different order or concurrently with other steps as described above. I can. In addition, those of ordinary skill in the art understand that the steps shown in the flowchart are not exclusive, other steps are included, or one or more steps in the flowchart may be deleted without affecting the scope of the present invention. You can understand.

The embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded in the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in the computer software field.

The computer-readable recording medium may contain information used in embodiments according to the present invention. For example, the computer-readable recording medium may include a bitstream, and the bitstream may include information described in embodiments according to the present invention.

The computer-readable recording medium may include a non-transitory computer-readable medium.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magnetic-optical media such as floptical disks. media), and a hardware device specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the processing according to the present invention, and vice versa.

In the above, the present invention has been described by specific matters such as specific elements and limited embodiments and drawings, but this is provided only to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , Anyone having ordinary knowledge in the technical field to which the present invention pertains can make various modifications and variations from these descriptions.

Therefore, the spirit of the present invention is limited to the above-described embodiments and should not be defined, and all modifications that are equally or equivalent to the claims as well as the claims to be described later fall within the scope of the spirit of the present invention. I would say.

Claims (1)

Performing alphabetization based on a just noticeable difference (JND) with respect to the input image; And
Enhancement of perceived quality of the input image
Containing, image processing method.

Images