KR102508428B1 - Method and apparatus for encoding and decoding video - Google Patents

Abstract

An encoding device is disclosed. The encoding apparatus obtains a prediction block for encoding corresponding to a current block of an input image based on a communication unit, a memory in which a noise model is stored, and a noise model, and obtains a difference signal obtained based on a difference between the current block and the prediction block. and a processor for performing quantization on , encoding the quantized differential signal to obtain an encoded video signal, and outputting a stream including noise metadata corresponding to a prediction block and the encoded video signal to a decoding device through a communication unit.

Description

Encoding, decoding method and device { METHOD AND APPARATUS FOR ENCODING AND DECODING VIDEO }

The present invention relates to an encoding and decoding method and apparatus, and more particularly, to an encoding and decoding method and apparatus considering noise of an image.

Recently, high-resolution images such as UHD (Ultra High Definition) images have been popularized, and post-processing performance of images has improved with the development of electronic technology.

In order to provide a high-resolution image, high-efficiency image compression technology is required, and thus research and development are being actively conducted.

However, existing image compression technologies have a problem in that they focus only on research and development of encoding and decoding algorithms without considering post-processing performance of electronic devices for images.

Accordingly, a need has arisen to improve processing speed and performance by considering the post-processing performance of an electronic device when encoding an image and omitting unnecessary encoding and decoding processes.

SUMMARY OF THE INVENTION The present invention has been made in accordance with the above-described needs, and an object of the present invention is to provide an encoding and decoding method and apparatus for performing encoding in consideration of post-processing of an electronic device.

An encoding apparatus according to an embodiment of the present invention for achieving the above object obtains a prediction block for encoding corresponding to a current block of an input image based on a communication unit, a memory in which a noise model is stored, and the noise model, Quantization is performed on a difference signal obtained based on a difference between a current block and the prediction block, encoding of the quantized difference signal is performed to obtain an encoded video signal, and noise metadata corresponding to the prediction block and the encoded video are obtained. and a processor outputting a stream including a signal to a decoding device through the communication unit, and the noise model is obtained by learning a noise post-processing process based on a plurality of sample images.

Here, the noise model includes noise pattern information according to at least one of a different coding unit (CU) size, a coding index, and a bit rate of a coding coefficient of each of the plurality of sample images, The processor may obtain the prediction block corresponding to the current block based on the noise pattern information.

Here, the processor obtains a prediction block retaining noise information corresponding to the noise pattern information among the noise information included in the current block, and the noise metadata includes noise information corresponding to the noise pattern information. can do.

The noise pattern information includes at least one of information on processable noise learned based on each of the plurality of sample images and information on intensity of the processable noise, and the information on processable noise is , ringing noise, flickering noise, blocking noise, and blurring noise.

Also, when the information on the decoding device is received through the communication unit, the processor may obtain the prediction block corresponding to the current block based on the noise model and the information on the decoding device.

In addition, the processor may obtain the noise model by performing a convolution neural network (CNN) on the plurality of sample images.

Meanwhile, when an encoded video signal and noise metadata are received from a communication unit and an encoding device through the communication unit, the electronic device according to an embodiment of the present disclosure decodes the encoded video signal, and based on the noise metadata, the electronic device A processor that performs post-processing on a decoded video signal, wherein the encoded video signal is an encoded video signal in a state in which noise information preset in an original video based on a noise model is maintained, and the noise model includes a plurality of A noise post-processing process is learned based on the sample image, and the processor performs the post-processing by filtering noise corresponding to the predetermined noise information included in the noise metadata in the decoded image signal.

Here, the noise metadata includes at least one of information on noise that can be processed in a noise post-processing process and information on the strength of the noise that can be processed, and the processor performs a decoded image based on the noise metadata. At least one of ringing noise filtering, flickering noise filtering, blocking noise filtering, and blurring noise filtering may be performed.

In addition, the electronic device may further include a display, and the processor may provide the post-processed image through the display.

Meanwhile, an encoding method of an encoding device storing a noise model according to an embodiment of the present disclosure includes obtaining a prediction block for encoding corresponding to a current block of an input image based on the noise model, the current block and the prediction block. Performing quantization on a difference signal obtained based on a difference between blocks, encoding the quantized difference signal to obtain an encoded video signal, and noise metadata corresponding to the prediction block and the encoded video signal. and outputting the stream to a decoding device, wherein the noise model is learned through a noise post-processing process based on a plurality of sample images.

Here, the noise model includes noise pattern information according to at least one of a different coding unit (CU) size, a coding index, and a bit rate of a coding coefficient of each of the plurality of sample images, In the obtaining of the prediction block, the prediction block corresponding to the current block may be obtained based on the noise pattern information.

In addition, the obtaining of the prediction block may include obtaining a prediction block maintaining noise information corresponding to the noise pattern information among the noise information included in the current block, and the noise metadata corresponding to the noise pattern information. noise information may be included.

The noise pattern information includes at least one of information on processable noise learned based on each of the plurality of sample images and information on intensity of the processable noise, and the information on processable noise is , ringing noise, flickering noise, blocking noise, and blurring noise.

In the obtaining of the prediction block, when the information on the decoding device is received, the prediction block corresponding to the current block may be obtained based on the noise model and the information on the decoding device.

The method may further include acquiring the noise model by performing a convolution neural network (CNN) on the plurality of sample images.

Meanwhile, a decoding method of an electronic device according to an embodiment of the present disclosure includes receiving an encoded video signal and noise metadata from an encoding device, decoding the encoded video signal, and decoding the encoded video signal based on the noise metadata. and performing post-processing on the video signal, wherein the encoded video signal is an encoded video signal in a state in which noise information preset in an original video is maintained based on a noise model, and the noise model is a plurality of sample images. A noise post-processing process is learned based on, and the performing of the post-processing is performed by filtering noise corresponding to the predetermined noise information included in the noise metadata in the decoded video signal to perform the post-processing. can be done

Here, the noise metadata includes at least one of information on noise that can be processed in a noise post-processing process and information on the strength of the noise that can be processed, and the performing of the post-processing includes the noise metadata Based on the decoded image, at least one of ringing noise filtering, flickering noise filtering, blocking noise filtering, and blurring noise filtering may be performed.

The method may further include providing the post-processed image through a display.

According to various embodiments of the present invention as described above, it is possible to minimize time and resources required for encoding and decoding, and improve performance and image quality by maintaining filterable noise according to post-processing of the electronic device.

1 is a block diagram for explaining an encoding apparatus according to an embodiment of the present disclosure.
2 is a block diagram showing the configuration of an encoding device to aid understanding of the present disclosure.
3 is a block diagram for explaining a noise model according to an embodiment of the present disclosure.
4 is a block diagram for explaining an electronic device according to an embodiment of the present disclosure.
5 is a block diagram showing the configuration of an electronic device to help understanding of the present disclosure.
6 is a flowchart illustrating an encoding method of an encoding apparatus according to an embodiment of the present disclosure.
7 is a flowchart illustrating a decoding method of an electronic device according to an embodiment of the present disclosure.

Terms used in this specification will be briefly described, and the present disclosure will be described in detail.

The terms used in the embodiments of the present disclosure have been selected from general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but they may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technologies, and the like. . In addition, in a specific case, there is also a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the disclosure. Therefore, terms used in the present disclosure should be defined based on the meaning of the term and the general content of the present disclosure, not simply the name of the term.

Embodiments of the present disclosure may apply various transformations and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, and should be understood to include all transformations, equivalents, and substitutes included in the spirit and scope of technology disclosed. In describing the embodiments, if it is determined that a detailed description of a related known technology may obscure the subject matter, the detailed description will be omitted.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. Terms are only used to distinguish one component from another.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "comprise" or "consist of" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other It should be understood that the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof is not precluded.

In the present disclosure, a “module” or “unit” performs at least one function or operation, and may be implemented in hardware or software or a combination of hardware and software. In addition, a plurality of "modules" or a plurality of "units" are integrated into at least one module and implemented by at least one processor (not shown), except for "modules" or "units" that need to be implemented with specific hardware. It can be.

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily carry out the present disclosure. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. And in order to clearly describe the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

1 is a block diagram for explaining an encoding apparatus according to an embodiment of the present disclosure.

As shown in FIG. 1 , the encoding device 100 includes a communication unit 110 , a memory 120 and a processor 130 .

The encoding device 100 is a device that encodes an image and converts it into another signal form. Here, an image is composed of a plurality of frames, and each frame may include a plurality of pixels. For example, the encoding device 100 may be a device for compressing an unprocessed original video. However, it is not limited thereto, and the encoding device 100 may be a device that changes a pre-encoded image into another signal type.

In particular, the encoding device 100 may perform encoding by dividing each frame constituting an image into a plurality of blocks. The encoding apparatus 100 may perform encoding through temporal or spatial prediction, transformation, quantization, filtering, entropy encoding, and the like in units of blocks.

According to an embodiment, the encoding device 100 includes a communication unit 110.

The communication unit 110 is a component that performs communication with various types of external devices according to various types of communication methods. For example, the communication unit 110 communicates with an external device using wired/wireless LAN, WAN, Ethernet, Bluetooth, Zigbee, IEEE 1394, Wi-Fi, or Power Line Communication (PLC). can be performed.

In particular, the encoding device 100 may receive an image through the communication unit 110 and output an encoded video signal to a decoding device through the communication unit 110 . For example, the communication unit 110 may transmit a stream (or bit stream) including noise metadata and an encoded video signal to the decoding device 200 . A detailed description of the noise metadata will be described later.

The memory 120 may store a noise model. Here, the noise model may be a noise post-processing process learned based on a plurality of sample images.

As an example, the noise model may be a machine-learned model using a plurality of sample images each having a different coding unit (CU) size, coding index, and coding coefficient. The noise model may include information on noise that can be processed through post-processing of the image and information on the intensity of noise that can be processed. For example, it may be assumed that a blocking artifact, a ringing artifact, and the like are included in an image. The encoding device 100 may process or remove artifacts and noise when encoding an image. The encoding apparatus 100 according to an embodiment of the present disclosure may encode without processing or removing some noises in a process of encoding an image based on information on processable noise in a post-processing process of the image.

The noise model may include noise pattern information according to at least one of a coding unit (CU) size of an image, a coding index, and a bit rate of coding coefficients, and the noise pattern information is a plurality of sample images It may include at least one of information about the processable noise and information about the strength of the processable noise learned based on each. For example, the noise model may be a model obtained by performing Convolution Neural Network (CNN) training on a plurality of sample images. Here, CNN is a multilayer neural network with a special connection structure designed for voice processing and image processing.

According to an embodiment, information on processable noise may include at least one of ringing noise, flickering noise, blocking noise, and blurring noise. there is.

The memory 120 stores various data such as an O/S (Operating System) software module for driving the encoding device 100, an encoding algorithm, a noise model, a CNN learning module, and the like.

The memory 120 may be implemented as an internal memory such as a ROM or RAM included in the processor 130 or may be implemented as a separate memory from the processor 130 . In this case, the memory 120 may be implemented in the form of a memory embedded in the encoding device 100 or in the form of a removable memory in the encoding device 100 depending on the data storage purpose. For example, data for driving the encoding device 100 is stored in a memory embedded in the encoding device 100, and data for an extended function of the encoding device 100 is detachable from the encoding device 100. It can be stored in available memory. On the other hand, in the case of memory embedded in the encoding device 100, it is implemented in the form of non-volatile memory, volatile memory, hard disk drive (HDD) or solid state drive (SSD), etc., and is removable from the encoding device 100. In the case of , it may be implemented in the form of a memory card (eg, micro SD card, USB memory, etc.), an external memory (eg, USB memory) connectable to a USB port, and the like.

The processor 130 controls the overall operation of the encoding device 100 .

According to an embodiment, the processor 130 may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON). However, it is not limited thereto, and the central processing unit ( central processing unit (CPU)), micro controller unit (MCU), micro processing unit (MPU), controller, application processor (AP), or communication processor (CP), ARM processor In addition, the processor 140 may be implemented as a system on chip (SoC) having a built-in processing algorithm, a large scale integration (LSI), or an FPGA ( It may be implemented in the form of a field programmable gate array).

The processor 130 according to an embodiment may obtain a prediction block for encoding corresponding to a current block of an input image based on a noise model stored in the memory 120 .

The noise model may include noise pattern information according to at least one of a coding unit (CU) size, a coding index, and a bit rate of coding coefficients. For example, the noise model may include, as noise pattern information, at least one of processable noise and information about the strength of processable noise in a post-processing process according to the size of the CU.

The noise model may be one in which a noise post-processing process is learned based on a plurality of sample images each having a different coding unit (CU) size, coding index, and coding coefficient bit rate.

The processor 130 may obtain noise pattern information according to at least one of a size of a coding unit (CU) of a current block of an input image, a coding index, and a bit rate of coding coefficients. Here, the noise pattern information may include at least one of information on noise that can be processed and information on the intensity of noise that can be processed during post-processing of the current block. For example, if the current block includes at least one of ringing noise, flickering noise, blocking noise, and blurring noise, different Information on whether or not each noise can be processed and the intensity that can be processed for each different noise may be included.

The processor 130 may obtain a prediction block for encoding corresponding to the current block of the input image based on the noise model, and may perform quantization on the obtained differential signal based on a difference between the current block and the prediction block.

The processor 130 according to an embodiment may obtain an encoded video signal by encoding the quantized differential signal, and output noise metadata and an encoded video signal corresponding to a prediction block to a decoding device.

Here, the noise metadata corresponding to the prediction block may include information on noise that can be processed based on noise pattern information and information on the intensity of noise that can be processed.

For example, the processor 130 may perform processing on the current block according to at least one of a coding unit (CU) size, a coding index, and a bit rate of coding coefficients of the current block. Noise pattern information including at least one of noise information and processable noise intensity information may be obtained, and a prediction block may be obtained based on the acquired noise pattern information. The noise metadata may include at least one of information about noise that can be processed and information about the strength of noise that can be processed during post-processing of the current block based on acquired noise pattern information or prediction blocks.

For example, the processor 130 processes flickering noise according to at least one of a coding unit (CU) size, a coding index, and a bit rate of coding coefficients in a current block. Information (or noise pattern information) on availability and processability strength may be obtained, and a prediction block and noise metadata may be obtained based on the acquired information.

Here, the prediction block may be a block that retains noise information corresponding to noise pattern information among noise information included in the current block. For example, if the decoding device can process ringing noise during post-processing according to the noise pattern information corresponding to the current block, the encoding device 100 can perform ringing noise on the basis of a predicted block that retains the same encoding can be performed. Accordingly, the encoding device 100 may perform encoding in consideration of the noise that can be processed by the decoding device and the noise intensity that can be processed by the decoding device. Since the encoding device 100 maintains and encodes noise that can be processed by the decoding device in the input image, the encoding speed can be reduced and resources required for encoding can be efficiently utilized. A detailed description of the prediction block and the encoding process will be given in FIG. 2 .

When information on the decoding device is received through the communication unit 110, the processor 130 according to an embodiment of the present disclosure may obtain a prediction block corresponding to the current block based on the noise model and the information on the decoding device. there is.

The noise model may be a model learned based on a plurality of sample images without considering the post-processing performance of the decoding device. Accordingly, the noise model may include, as noise pattern information, information about processable noise and processable noise strength during post-processing in a general decoding apparatus without specifying the decoding apparatus. The encoding device 100 according to an embodiment of the present disclosure may receive information about a decoding device and obtain a prediction block based on the received information and a noise model. Here, the information on the decoding device may include information on noise and noise strength that the decoding device can process during post-processing.

Meanwhile, FIG. 1 briefly illustrates various components by taking the case where the encoding device 100 is a device having functions such as a communication function and a control function. Accordingly, it goes without saying that, depending on embodiments, some of the components shown in FIG. 1 may be omitted or changed, and other components may be further added.

2 is a block diagram showing the configuration of the encoding device 100 to help understanding of the present disclosure.

As shown in FIG. 2 , the encoding device 100 includes a noise model 211, a motion estimation unit 212, a motion compensation unit 212, an intra prediction unit 220, a switch 215, and a subtractor 225 , transform unit 230, quantization unit 240, entropy encoding unit 250, inverse quantization unit 260, inverse transform unit 270, adder 275, filter unit 280, and reference image buffer 290 includes

The encoding device 100 may perform encoding by dividing each frame of the input image into a plurality of blocks. The encoding apparatus 100 may perform encoding through temporal or spatial prediction, transformation, quantization, filtering, entropy encoding, and the like in units of blocks.

Prediction means generating a prediction block similar to the current block to be encoded. Here, the block unit to be encoded may be a Coding Tree Unit (CTU), Coding Unit (CU), Prediction Unit (PU), or Transform Unit (TU). However, these terms are only used for convenience of description of the present disclosure, and are not limited thereto.

Prediction is divided into temporal prediction and spatial prediction. Temporal prediction means inter-screen prediction. The encoding device 100 may store some reference pictures that are highly correlated with an image to be currently encoded and perform inter-prediction using the reference pictures. That is, the encoding device 100 may generate a prediction block from a reference image decoded after encoding at a previous time. In this case, it is said that the encoding device 100 operates in an Inter mode among coding indexes.

When operating in the inter mode, the motion estimation unit 211 may search for a block having the highest temporal correlation with the current block in the reference image stored in the reference image buffer 290 . The motion estimation unit 211 may interpolate the reference image and search for a block having the highest temporal correlation with the current block in the interpolated image.

Here, the reference image buffer 290 is a space for storing a reference image. The reference image buffer 290 is used only when inter-prediction is performed, and may store some reference images having a high correlation with an image to be currently encoded. The reference image may be an image generated by sequentially transforming, quantizing, inverse-quantizing, inverse-transforming, and filtering residual blocks to be described later. That is, the reference image may be a decoded image after encoding.

The motion compensation unit 212 may generate a prediction block based on motion information about a block having the highest temporal correlation with the current block found by the motion estimation unit 211 . Here, the motion information may include a motion vector, a reference picture index, and the like.

Spatial prediction means intra-picture prediction. The intra predictor 220 may generate a prediction value for the current block by performing spatial prediction from encoded neighboring pixels in the current image. In this case, the encoding device 100 is said to operate in an intra mode.

Inter mode or intra mode may be determined in units of Coding Units (CUs). Here, the coding unit may include at least one prediction unit. When the mode is determined, the position of the switch 215 may be changed to correspond to the mode.

Meanwhile, a reference image decoded after encoding in temporal prediction may be an image to which filtering is applied, but adjacent pixels decoded after encoding in spatial prediction may be pixels to which filtering is not applied.

The subtractor 225 may generate a residual signal or residual block by obtaining a difference between a target block and a prediction block obtained from temporal prediction or spatial prediction. The differential signal may be a block from which redundancy is largely removed by a prediction process, or may be a block including information to be encoded because prediction is not perfectly performed.

The transform unit 230 may output a transform coefficient in the frequency domain by transforming the difference signal after intra- or inter-prediction in order to remove spatial redundancy. At this time, the unit of transformation is a transform unit (TU), which is determined regardless of the prediction unit. For example, a frame including a plurality of differential signals may be divided into a plurality of transform units regardless of a prediction unit, and the transform unit 230 may perform transform for each transform unit. The division of the transform unit may be determined according to bit rate optimization.

The conversion unit 230 may perform conversion to concentrate the energy of each conversion unit in a specific frequency domain. For example, the transform unit 230 may concentrate data into a low-frequency region by performing DCT (Discrete Cosine Transform)-based transform on each transform unit. Alternatively, the transform unit 230 may perform a discrete Fourier transform (DFT)-based transform or a discrete sine transform (DST)-based transform.

The quantization unit 240 may perform quantization on transform coefficients and approximate the transform coefficients to a predetermined number of representative values. That is, the quantization unit 240 may map input values within a specific range to one representative value. In this process, high-frequency signals that are not well recognized by humans are removed, and information loss may occur.

The quantization unit 240 may use one of uniform quantization and non-uniform quantization methods according to a probability distribution of input data or a purpose of quantization. For example, the quantization unit 240 may use a uniform quantization method when the probability distribution of the input data is uniform. Alternatively, the quantization unit 240 may use a non-uniform quantization method when the probability distribution of the input data is not uniform.

The entropy encoding unit 250 may reduce the amount of data by variably allocating a symbol length according to a symbol occurrence probability for data input from the quantization unit 240 . That is, the entropy encoding unit 250 may generate a stream by expressing input data as a variable length bit string composed of 0 and 1 based on a probability model.

For example, the entropy encoder 250 may represent input data by allocating a small number of bits to a symbol with a high probability of occurrence and a large number of bits to a symbol with a low probability of occurrence. Accordingly, the size of a bit string of input data may be reduced, and compression performance of video encoding may be improved.

The entropy encoding unit 250 may perform entropy encoding using variable length coding such as Huffman coding or exponential Gollum coding or arithmetic coding.

The inverse quantization unit 260 and the inverse transformation unit 270 may receive the quantized transform coefficients, inverse quantize them, and then perform inverse transformation to generate a reconstructed differential signal.

The adder 275 may generate a reconstructed block by adding the reconstructed differential signal and a prediction block obtained from temporal prediction or spatial prediction.

Meanwhile, in the encoding process, adjacent blocks are quantized by different quantization parameters, and thus block boundaries may be degraded. This phenomenon is called blocking artifacts, which is one of the important factors in evaluating image quality. A filtering process may be performed to reduce this degradation.

The filter unit 280 may apply at least one of a deblocking filter, a sample adaptive offset (SAO), and an adaptive loop filter (ALF) to the reconstructed image. The filtered reconstructed image may be stored in the reference image buffer 290 and used as a reference image.

Based on the noise model 211, the encoding apparatus 100 according to an embodiment of the present disclosure may obtain a prediction block corresponding to the current block from a reference image on which a filtering process for reducing degradation has not been performed.

For example, blocking artifacts (or blocking noise) may be processed or removed during post-processing of an image in a decoding device. Accordingly, the encoding apparatus 100 may obtain a prediction block having blocking degradation based on the noise model 211 .

The encoding device 100 may obtain a prediction block corresponding to the current block from a reference image to which filtering for noise, degradation, and the like is not performed. Since the encoding device 100 can obtain a prediction signal and output a stream without filtering, encoding time and resources required for encoding can be efficiently managed.

Here, the stream may include noise metadata and an encoded video signal. The encoded video signal may refer to a signal obtained by entropy coding a quantized differential signal.

Noise metadata may include noise information corresponding to noise pattern information based on a prediction block. For example, the noise metadata performs decoding on an encoded video signal based on at least one of the size of a coding unit (CU) of a current block, a coding index, and a bit rate of coding coefficients; During post-processing of the decoded signal, information on noise that can be processed and information on the strength of the noise may be included.

Meanwhile, a motion estimation unit 212, a motion compensation unit 212, an intra prediction unit 220, a switch 215, a subtractor 225, a transform unit 230, a quantization unit 240, an entropy encoding unit 250 ), the inverse quantization unit 260, the inverse transform unit 270, the adder 275, and the filter unit 280 may mean functions and operations performed by the processor 130, of course.

3 is a block diagram for explaining a noise model according to an embodiment of the present disclosure.

According to FIG. 3, each of a plurality of sample images has a bit rate of coding indexes and coding coefficients such as CU (Coding Unit), QP (Quantization Parameter), Inter/Intra mode, etc. etc. may be different.

The encoding device 100 may learn a noise post-processing process based on a plurality of sample images. Depending on the learning result, information about whether each noise can be processed and the processing intensity can be obtained.

For example, the encoding device 100 determines whether each of ringing noise, flickering noise, blocking noise, and blurring noise can be processed, and processing strength information can be obtained, etc. Here, the information obtained by the encoding device 100 includes at least one of processable noise and information about the intensity of processable noise as the decoding device performing decoding or an external electronic device performs post-processing on the image. can do. It may refer to information about noise that can be filtered when an unspecified electronic device performs post-processing of an image according to a learning result, rather than limiting a specific electronic device.

Based on the learning result, the encoding device 100 may obtain a prediction block according to at least one of a coding unit (CU) size, a coding index, and a bit rate of coding coefficients according to a current block. there is. Here, the obtained prediction block may be a prediction block that retains noise information that can be processed or removed during post-processing according to a learning result. As another example, if noise information that cannot be processed or removed is included in the current block during post-processing according to a learning result, the encoding device 100 may obtain an encoded video signal by performing filtering on the noise information.

The encoding apparatus 100 according to an embodiment of the present disclosure may obtain a prediction block and acquire noise metadata including noise information that can be processed or removed during post-processing.

Noise metadata may be transmitted to a decoding device as a stream together with an encoded video signal. The decoding device may perform post-processing (or filtering) on the decoded video signal based on the noise metadata.

The encoding device 100 according to an embodiment of the present disclosure may perform CNN on a plurality of sample images and store a noise model according to a learning result. Here, CNN is a multilayer neural network with a special connection structure designed for voice processing and image processing. In particular, CNN can filter images in various ways through pixel preprocessing and recognize characteristics of images.

4 is a block diagram for explaining an electronic device according to an embodiment of the present disclosure.

As shown in FIG. 4 , the electronic device 400 includes a communication unit 410 and a processor 420 .

The electronic device 400 may be implemented as various types of devices capable of performing decoding on an encoded video signal. For example, it can be implemented as a set-top box, a Blu-ray player, and the like. However, it is not limited thereto and may be implemented as a display device capable of displaying a decoded video signal.

The communication unit 410 may communicate with the encoding device 100 . Specifically, the communication unit 410 may receive a stream from the encoding device 100 . In particular, the communication unit 410 may receive an encoded video signal and noise metadata.

The communication unit 410 communicates with the encoding device 100 using wired/wireless LAN, WAN, Ethernet, Bluetooth, Zigbee, IEEE 1394, Wi-Fi, or Power Line Communication (PLC). can be performed.

The electronic device 400 according to an embodiment may communicate with an external electronic device having an output unit, such as a speaker or a display, and transmit a decoded video signal to the external electronic device having an output unit through the communication unit 410. may be

When the encoded video signal and noise metadata are received from the encoding device 100 through the communication unit 410, the processor 420 decodes the encoded video signal and performs post-processing on the decoded video signal based on the noise metadata. can do. Here, the encoded video signal may be an encoded video signal while preserving noise information preset in the original video based on the noise model in the encoding apparatus 100 .

In particular, the processor 420 may perform post-processing by filtering noise corresponding to predetermined noise information included in noise metadata in the decoded video signal.

Here, the noise metadata may include at least one of information on noise that can be processed in a noise post-processing process and information on the intensity of noise that can be processed. The processor 420 performs ringing noise filtering, flickering noise filtering, blocking noise filtering, and blurring noise filtering on the decoded video signal based on the noise metadata. At least one of these may be performed.

Displays (not shown) are various types of displays such as liquid crystal displays (LCDs), organic light emitting displays (OLEDs), liquid crystal on silicon (LCoS), or digital light processing (DLPs). can be implemented as However, it is not limited thereto, and of course it can be implemented in various types of displays capable of displaying images.

In particular, the electronic device 400 may perform post-processing on the decoded video signal and output the post-processed video signal through a display.

Meanwhile, FIG. 4 simplifies various components by taking a case where the electronic device 400 is a device having functions such as a communication function and a control function. Accordingly, depending on embodiments, some of the components shown in FIG. 4 may be omitted or changed, and other components may be further added.

5 is a block diagram illustrating the configuration of an electronic device 400 to help understanding of the present disclosure.

As shown in FIG. 4 , the electronic device 400 includes an entropy decoding unit 510, an inverse quantization unit 520, an inverse transform unit 530, an adder 535, an intra prediction unit 540, a motion compensation unit ( 550), a switch 555, a filter unit 560, and a reference image buffer 570.

The electronic device 400 may reconstruct an image by performing decoding on the encoded video signal received from the encoding device 100 . The electronic device 400 may perform decoding through entropy decoding, inverse quantization, inverse transformation, and filtering in block units.

The entropy decoding unit 510 may generate quantized transform coefficients by entropy decoding the input stream. In this case, the entropy decoding method may be a method in which the method used in the entropy encoding unit 150 in FIG. 2 is reversely applied.

The inverse quantization unit 520 may perform inverse quantization by receiving the quantized transform coefficient. That is, according to the operation of the quantization unit 140 and the inverse quantization unit 520, an input value within a specific range is changed to any one reference input value within a specific range, and in this process, the input value and any one reference input value are changed. Errors can occur as much as the difference.

The inverse transformation unit 530 inversely transforms the data output from the inverse quantization unit 520, and may perform inverse transformation by applying the method used in the transformation unit 130 inversely. The inverse transform unit 530 may generate a restored differential signal by performing inverse transform.

The adder 535 may generate a reconstructed block by adding the reconstructed differential signal and the predicted block. Here, the prediction block may be a block generated in an inter mode or an intra mode.

When operating in the inter mode, the motion compensator 550 may receive motion information about a current block to be decoded from the encoding device 100 and generate a prediction block based on the received motion information. Here, the motion compensator 550 may generate a prediction block from a reference image stored in the reference image buffer 570 . The motion information may include a motion vector of a block having the highest temporal correlation with the target block, a reference picture index, and the like.

Here, the reference image buffer 570 may store some reference images that have a high correlation with an image to be currently decoded. The reference image may be an image generated by filtering the reconstruction block described above. That is, the reference image may be a decoded image after encoding.

When operating in the intra mode, the intra prediction unit 540 may generate a prediction value for the current block by performing spatial prediction from encoded adjacent pixels in the current image.

Meanwhile, the position of the switch 555 may be changed according to the mode of the current block.

The filter unit 560 may apply at least one of a deblocking filter, SAO, and ALF to the reconstructed image. The filtered reconstructed image may be stored in the reference image buffer 570 and used as a reference image.

Meanwhile, the filter unit 560 may perform filtering based on noise metadata received from the encoding device 100 . For example, in obtaining a prediction block based on the noise model 211, the encoding apparatus 100 determines the size of a coding unit (CU) of the current block, a coding index, and coding coefficients. A prediction block and noise metadata according to at least one of the bit rates may be obtained. Since the noise pattern information includes information on processable noise and noise intensity during post-processing of the decoded video signal of the electronic device 400, the filter unit 560 performs the encoding device 100 based on the noise metadata. Filtering may be performed on noise that is not processed (or removed) in .

As described above, the encoding device 100 may compress an input image through an encoding process and transmit the compressed image to the electronic device 400 . The electronic device 400 may reconstruct an image by decoding the compressed image.

6 is a flowchart illustrating an encoding method of an encoding apparatus according to an embodiment of the present disclosure.

In the encoding method of the encoding device in which the noise model shown in FIG. 6 is stored, a prediction block for encoding corresponding to a current block of an input image is obtained based on the noise model (S610).

Subsequently, quantization is performed on the obtained differential signal based on the difference between the current block and the predicted block (S620).

Subsequently, an encoded video signal is obtained by encoding the quantized differential signal (S630).

Subsequently, a stream including noise metadata and an encoded video signal corresponding to the prediction block is output to the decoding device (S640).

Here, the noise model may be a noise post-processing process learned based on a plurality of sample images.

A noise model according to an embodiment of the present disclosure is noise pattern information according to at least one of a different coding unit (CU) size, a coding index, and a bit rate of a coding coefficient of each of a plurality of sample images Including, in step S610 of obtaining a prediction block, a prediction block corresponding to the current block may be obtained based on the noise pattern information.

In addition, in step S610 of obtaining a prediction block, a prediction block that retains noise information corresponding to noise pattern information among noise information included in the current block is obtained, wherein the noise metadata is noise information corresponding to noise pattern information. information may be included.

Here, the noise pattern information includes at least one of information on processable noise learned based on each of a plurality of sample images and information on intensity of processable noise, and the information on processable noise includes ringing noise ( It may include at least one of ringing noise, flickering noise, blocking noise, and blurring noise.

In step S610 of obtaining a prediction block according to an embodiment, when information on a decoding device is received, a prediction block corresponding to the current block may be obtained based on the noise model and the information on the decoding device.

The encoding method according to an embodiment may further include obtaining a noise model by performing a convolution neural network (CNN) on a plurality of sample images.

7 is a flowchart illustrating a decoding method of an electronic device according to an embodiment of the present disclosure.

According to the decoding method of the electronic device shown in FIG. 7, upon receiving the encoded video signal and the noise metadata from the encoding device, the encoded video signal is decoded (S710).

Subsequently, post-processing is performed on the decoded video signal based on the noise metadata (S720).

Here, the encoded video signal is an encoded video signal in a state in which noise information preset in the original video is maintained based on the noise model, and the noise model is a noise post-processing process learned based on a plurality of sample images. In step S720 of performing processing, post-processing may be performed by filtering noise corresponding to predetermined noise information included in noise metadata in the decoded video signal.

Here, the noise metadata includes at least one of processable noise information and processable noise intensity information in the noise post-processing process, and the post-processing step S720 includes decoding based on the noise metadata. At least one of ringing noise filtering, flickering noise filtering, blocking noise filtering, and blurring noise filtering may be performed on the image.

The decoding method according to an embodiment of the present disclosure may further include providing a post-processed image through a display.

Meanwhile, various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by a processor itself. According to software implementation, embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

Meanwhile, computer instructions for performing processing operations according to various embodiments of the present disclosure described above may be stored in a non-transitory computer-readable medium. Computer instructions stored in such a non-transitory computer readable medium may cause a specific device to perform processing operations according to various embodiments described above when executed by a processor.

A non-transitory computer readable medium is a medium that stores data semi-permanently and is readable by a device, not a medium that stores data for a short moment, such as a register, cache, or memory. Specific examples of the non-transitory computer readable media may include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

Although the preferred embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the specific embodiments described above, and is common in the technical field belonging to the present disclosure without departing from the gist of the present disclosure claimed in the claims. Of course, various modifications are possible by those with knowledge of, and these modifications should not be individually understood from the technical spirit or perspective of the present disclosure.

100: encoding device 110: communication unit
120: memory 130: processor
400: electronic device 410: communication unit
420: processor

Claims (18)

communications department;
a memory in which a noise model is stored;
Obtaining a prediction block for encoding corresponding to a current block of an input image based on the noise model;
Perform quantization on a differential signal obtained based on a difference between the current block and the prediction block;
Encoding the quantized differential signal to obtain an encoded video signal;
A processor outputting a stream including noise metadata corresponding to the prediction block and the encoded video signal to a decoding device through the communication unit;
The noise model is one in which a noise post-processing process is learned based on a plurality of sample images. According to claim 1,
The noise model,
Includes noise pattern information according to at least one of a different coding unit (CU) size of each of the plurality of sample images, a coding index, and a bit rate of a coding coefficient;
the processor,
An encoding device that obtains the prediction block corresponding to the current block based on the noise pattern information. According to claim 2,
the processor,
Obtaining a prediction block maintaining noise information corresponding to the noise pattern information among the noise information included in the current block;
The noise metadata,
Encoding device comprising noise information corresponding to the noise pattern information. According to claim 2,
The noise pattern information,
Includes at least one of information on processable noise learned based on each of the plurality of sample images and information on intensity of the processable noise;
Information on the processable noise,
An encoding device comprising at least one of ringing noise, flickering noise, blocking noise, and blurring noise. According to claim 1,
the processor,
When information on the decoding device is received through the communication unit, the prediction block corresponding to the current block is obtained based on the noise model and the information on the decoding device. According to claim 1,
the processor,
An encoding device for obtaining the noise model by performing a convolution neural network (CNN) on the plurality of sample images. In electronic devices,
communications department; and
When an encoding video signal and noise metadata are received from an encoding device through the communication unit, a processor that decodes the encoded video signal and performs post-processing on the decoded video signal based on the noise metadata;
The encoded video signal,
Noise information preset in the original video based on the noise model is an encoded video signal while being maintained,
The noise model is learned through a noise post-processing process based on a plurality of sample images,
the processor,
The electronic device performs the post-processing by filtering noise corresponding to the predetermined noise information included in the noise metadata in the decoded video signal. According to claim 7,
The noise metadata,
Including at least one of information about noise that can be processed in a noise post-processing process and information about the intensity of the noise that can be processed;
the processor,
At least one of ringing noise filtering, flickering noise filtering, blocking noise filtering, and blurring noise filtering is performed on the decoded image based on the noise metadata to do, electronic devices. According to claim 7,
further comprising a display;
the processor,
An electronic device that provides the post-processed image through the display. In the encoding method of the encoding device in which the noise model is stored,
obtaining a prediction block for encoding corresponding to a current block of an input image based on the noise model;
performing quantization on a differential signal obtained based on a difference between the current block and the prediction block;
encoding the quantized differential signal to obtain an encoded video signal; and
And outputting a stream including noise metadata corresponding to the prediction block and the encoded video signal to a decoding device,
The noise model is one in which a noise post-processing process is learned based on a plurality of sample images. According to claim 10,
The noise model,
Includes noise pattern information according to at least one of a different coding unit (CU) size of each of the plurality of sample images, a coding index, and a bit rate of a coding coefficient;
Obtaining the prediction block,
Acquiring the prediction block corresponding to the current block based on the noise pattern information. According to claim 11,
Obtaining the prediction block,
Obtaining a prediction block retaining noise information corresponding to the noise pattern information among the noise information included in the current block;
The noise metadata,
Encoding method comprising noise information corresponding to the noise pattern information. According to claim 11,
The noise pattern information,
Includes at least one of information on processable noise learned based on each of the plurality of sample images and information on intensity of the processable noise;
Information on the processable noise,
An encoding method comprising at least one of ringing noise, flickering noise, blocking noise, and blurring noise. According to claim 10,
Obtaining the prediction block,
If the information on the decoding device is received, obtaining the prediction block corresponding to the current block based on the noise model and the information on the decoding device. According to claim 10,
Acquiring the noise model by performing a convolution neural network (CNN) on the plurality of sample images; further comprising the encoding method. In the decoding method of the electronic device,
Receiving an encoding video signal and noise metadata from an encoding device;
decoding the encoded video signal; and
Performing post-processing on the decoded video signal based on the noise metadata;
The encoded video signal,
Noise information preset in the original video based on the noise model is an encoded video signal while being maintained,
The noise model is learned through a noise post-processing process based on a plurality of sample images,
In the post-processing step,
The decoding method of performing the post-processing by filtering noise corresponding to the predetermined noise information included in the noise metadata from the decoded video signal. According to claim 16,
The noise metadata,
Including at least one of information about noise that can be processed in a noise post-processing process and information about the intensity of the noise that can be processed;
In the post-processing step,
At least one of ringing noise filtering, flickering noise filtering, blocking noise filtering, and blurring noise filtering is performed on the decoded image based on the noise metadata , decoding method. According to claim 16,
Further comprising, a decoding method comprising: providing the image on which the post-processing has been performed through a display.

Images