WO2019011011A1 - Procédé, dispositif et système de traitement d'image - Google Patents

Procédé, dispositif et système de traitement d'image Download PDF

Info

Publication number
WO2019011011A1
WO2019011011A1 PCT/CN2018/081678 CN2018081678W WO2019011011A1 WO 2019011011 A1 WO2019011011 A1 WO 2019011011A1 CN 2018081678 W CN2018081678 W CN 2018081678W WO 2019011011 A1 WO2019011011 A1 WO 2019011011A1
Authority
WO
WIPO (PCT)
Prior art keywords
image block
reconstructed
current
reconstructed image
encoded
Prior art date
Application number
PCT/CN2018/081678
Other languages
English (en)
Chinese (zh)
Inventor
张红
杨海涛
刘杉
吴枫
李跃
刘�东
李厚强
Original Assignee
华为技术有限公司
中国科学技术大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司, 中国科学技术大学 filed Critical 华为技术有限公司
Publication of WO2019011011A1 publication Critical patent/WO2019011011A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/117—Filters, e.g. for pre-processing or post-processing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/124—Quantisation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/13—Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/132—Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/90—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
    • H04N19/91—Entropy coding, e.g. variable length coding [VLC] or arithmetic coding

Definitions

  • the present application relates to image processing technologies, and in particular, to an image processing method, device, and system.
  • FIG. 1 is a schematic diagram of a digital video provided by the present application.
  • the digital video is composed of a multi-frame digital image.
  • 2 is a schematic diagram of a digital image provided by the present application.
  • the image is composed of 12*16 pixels, wherein each pixel is referred to as one pixel, and 12*16 represents image resolution.
  • the image resolution of 2K video is 1920*1080
  • the image resolution of 4K video is 3840*2160.
  • the original video contains a large amount of data, which is not suitable for storage and transmission, and requires efficient video compression coding techniques to compress the original data.
  • FIG. 3 is a schematic diagram of encoding of an encoding end provided by the present application.
  • the encoding process of the encoding end includes: after receiving the video by the encoding end, dividing the image into multiple to-bes for each frame image constituting the video. Encode image blocks.
  • the image block is first reconstructed by reference (the reference reconstructed image block is used to provide a reference pixel required for the current image block to be encoded, and the reference pixel is used to predict the current image block to be encoded).
  • the coded image block is predicted to obtain a prediction signal of the current image block to be coded; the prediction signal is subtracted from the original signal of the current image block to be coded to obtain a residual signal. After prediction, the magnitude of the residual signal is much smaller than the original signal.
  • the residual signal is transformed and quantized. After transform quantization, the transform quantized coefficients are obtained, and the quantized coefficients and other indication information in the encoding are encoded by the entropy coding technique to obtain a code stream. Further, the encoding end further needs to reconstruct the current image block to be encoded, so as to provide reference pixels for encoding the subsequent image block to be encoded.
  • the encoding end needs to perform inverse quantization and inverse transform on the transform quantized coefficients of the current image block to be encoded to obtain a reconstructed residual signal, and the reconstructed residual signal is
  • the prediction signals corresponding to the current image block to be encoded are added to obtain a reconstructed signal of the current image block to be encoded, and the reconstructed image block is obtained according to the reconstructed signal.
  • the reconstructed image block can predict a subsequent image block to be encoded.
  • the residual signal is transformed to obtain a transform coefficient, and the transform coefficient has information loss after being quantized, and the information loss is irreversible.
  • the inversely quantized transform coefficients are distorted, so that the reconstructed signal is inconsistent with the original signal.
  • This compression method is lossy compression. Therefore, for lossy compression, after the reconstructed image block is obtained, the reconstructed image block needs to be filtered to remove some distortion introduced by lossy compression, such as blockiness, ringing effect, and the like.
  • a DBK filter in the H.264, H.265 standard can be used.
  • the SAO filter in H.265, and the ALF filter in the next generation standard, etc. can be used.
  • the residual signal is obtained by using a lossless transform operation to obtain transform coefficients, and the transform coefficients are subjected to entropy coding without performing quantization operations.
  • lossless compression filtering operations are generally not performed. Further, after each image block of the current image is reconstructed, a reconstructed image is obtained, wherein the reconstructed image can predict other subsequent frame images.
  • FIG. 4 is a schematic diagram of decoding of a decoding end provided by the present application. As shown in FIG. 4, after acquiring a code stream, the decoding end first performs entropy decoding on the code stream to obtain a transform quantized coefficient of the current image block to be reconstructed, and then transforms the quantized coefficient. Performing inverse quantization and inverse transform to obtain a reconstructed residual signal of the current image block to be reconstructed.
  • FIG. 5 is a schematic diagram of encoding of the encoding end provided by the present application. As shown in FIG. 5, the encoding end uses downsampling processing on the entire image, and then Each image block to be encoded in the downsampled image is encoded to obtain a code stream. The resolution of the reconstructed image block corresponding to each image block to be encoded is a downsampling resolution.
  • the decoding end parses the code stream, and the resolution of each image block to be reconstructed is the downsampling resolution, and the resolution of the corresponding reconstructed image block is also the downsampling resolution, and the decoding end needs to adopt the reconstructed image block. Sampling processing to obtain a reconstructed image block of the original resolution.
  • the encoding end uses downsampling processing for the entire image.
  • the characteristics of each image block included in the entire image may be different.
  • some image blocks may be relatively flat, and the encoding end is suitable for downsampling processing.
  • Some image blocks may have more details, in which case the downsampling process will lose these details. This will result in poor coding on the encoding side.
  • this will result in some reconstructed image blocks obtained by the decoding end being relatively ambiguous, that is, the reconstructed image blocks obtained by the decoding end are not effective.
  • the application provides an image processing method, device and system, so that the reconstructed image block obtained by the decoding end has better effect, and the encoding effect of the encoding end is better.
  • the present application provides an image processing method, including: parsing a code stream to obtain an encoding mode of a current image block to be reconstructed of a current image, a residual signal of a current image block to be reconstructed, and a current image block to be reconstructed.
  • Each reference reconstructs a pixel in the image block, wherein the encoding mode is the original resolution encoding mode or the down sampling encoding mode, and the current image block to be reconstructed corresponds to M reference reconstructed image blocks, and M is a positive integer greater than or equal to 1; Determining, by a coding mode of the image block to be reconstructed and a pixel in the M reference reconstructed image blocks, a plurality of reference pixels of the current image block to be reconstructed; generating a prediction signal of the current image block to be reconstructed according to the plurality of reference pixels; The residual signal generates a reconstructed signal of the current image block to be reconstructed, and reconstructs the current image block to be reconstructed according to the reconstructed signal to obtain a current reconstructed image block.
  • the present application considers that image blocks have respective characteristics, and the encoding end uses different encoding methods for them. Based on this, the decoding end encodes the current image block to be reconstructed and M. The two coding factors of the reference reconstructed image block are taken into consideration to reconstruct the current image block to be reconstructed. Thereby, the reconstructed image block obtained by the decoding end is more effective.
  • the method further includes: performing, on the current reconstructed image block, the pixels of the adjacent reconstructed image block required for performing the upsampling process on the current reconstructed image block. Upsampling processing.
  • the upsampling process is performed after all the image blocks of the current image are reconstructed; correspondingly, the code stream includes: a coding mode of each of the reference reconstructed image blocks in the M reference reconstructed image blocks; Determining a plurality of reference pixels of the image block to be reconstructed according to the coding mode of the image block to be reconstructed and the pixels in the M reference reconstructed image blocks, including: coding according to the current image block to be reconstructed, and encoding of the M reference reconstructed image blocks The mode and the M reference reconstructed pixels in the image block determine a plurality of reference pixels of the current image block to be reconstructed.
  • the upsampling process is performed after all the reconstructed image blocks required for performing the upsampling process of the current reconstructed image block are reconstructed; correspondingly, the code stream includes: M reference reconstructed image blocks And determining, by the encoding mode of the image block to be reconstructed and the pixels in the M reference reconstructed image blocks, determining a plurality of reference pixels of the current image block to be reconstructed, including: according to the current image block to be reconstructed The coding mode, the coding mode of the M reference reconstructed image blocks, and the pixels in the M reference reconstructed image blocks determine a plurality of reference pixels of the current image block to be reconstructed.
  • the upsampling process of the current reconstructed image block includes: performing upsampling on the current reconstructed image block according to a pixel of a part of the adjacent reconstructed image block that is currently reconstructed in the adjacent reconstructed image block. Processing; if another part of the adjacent reconstructed image block of the currently uncompleted reconstruction in the required adjacent reconstructed image block has been reconstructed, then the partial boundary of the currently reconstructed image block is quadraminated according to another partial adjacent reconstructed image block.
  • a sampling process in which a partial boundary of a currently reconstructed image block is adjacent to another partially adjacent reconstructed image block.
  • the upsampling process of the current reconstructed image block includes: performing up-sampling the current reconstructed image block according to a pixel of a portion of the adjacent reconstructed image block that is currently reconstructed in the required adjacent reconstructed image block. Processing; if all the image blocks of the current image have been reconstructed, performing a second upsampling process on a partial boundary of the currently reconstructed image block according to another partially adjacent reconstructed image block in the desired adjacent reconstructed image block And another partial adjacent reconstructed image block is an image block that is not reconstructed when the current up-sampling image block is subjected to the first upsampling process; a partial boundary of the current reconstructed image block is adjacent to another partially adjacent reconstructed image block.
  • a part of the adjacent reconstructed image block is an upper image block and a left image block of the current reconstructed image block, and another partial adjacent reconstructed image block is a lower image block and a right image block of the current reconstructed image block; or, a part of the adjacent image block
  • the reconstructed image block is the upper left image block, the upper image block, the upper right image block and the left image block of the current reconstructed image block
  • the other adjacent reconstructed image block is the right image block of the current reconstructed image block, the lower left image block, and the lower image block and The bottom right image block.
  • a partial boundary of the currently reconstructed image block is a right boundary and a lower boundary of the currently reconstructed image block.
  • the required adjacent reconstructed image block includes an upper image block, a lower image block, a left image block, and a right image block of the current reconstructed image block; or the desired adjacent reconstructed image block includes the current reconstructed image block.
  • the method further includes: identifying that the current reconstructed image block has completed the upsampling process. It is thus possible to avoid repeated upsampling of the currently reconstructed image block.
  • the reference pixels include: if the encoding mode of the image block to be reconstructed is the original resolution encoding mode, and the M reference reconstructed image blocks include the reference reconstructed image block whose encoding mode is the original resolution encoding mode, then the encoding mode is original Determining the reference pixel of the current image block to be reconstructed in the pixel of the reference reconstructed image block of the resolution coding mode; if the coding mode of the current image block to be reconstructed is the original resolution coding mode, and the M reference reconstructed image blocks include the coding mode
  • the reference reconstructed image block of the sampling and coding mode acquires pixels required for reconstructing the current image block to be reconstructed from the pixels of the reference reconstructed image block whose encoding mode is the down sampling coding mode, and reconstructs the current image block to be reconstructed.
  • the required pixels are subjected to upsampling processing to obtain reference pixels of the current image block to be reconstructed;
  • the coding mode of the image block to be reconstructed is a down-sampling coding mode, and the M reference reconstructed image blocks include a reference reconstructed image block whose coding mode is a down-sampling coding mode, and the pixel of the reference reconstructed image block whose coding mode is the down-sampling coding mode.
  • Determining a reference pixel of the current image block to be reconstructed if the encoding mode of the current image block to be reconstructed is a downsampling coding mode, and the M reference reconstructed image blocks include a reference reconstructed image block whose encoding mode is the original resolution encoding mode, then The pixel required to reconstruct the current image block to be reconstructed is obtained from the pixels of the reference reconstructed image block of the original resolution coding mode, and the pixels required for reconstructing the current image block to be reconstructed are downsampled to The reference pixel of the current image block to be reconstructed is obtained.
  • determining a plurality of reference pixels of the current image block to be reconstructed according to the current coding mode of the image block to be reconstructed and the pixels in the M reference reconstructed image blocks including: if the current to be reconstructed
  • the encoding mode of the image block is the original resolution encoding mode, and the pixels required for reconstructing the current image block to be reconstructed are acquired in the pixels of the M reference reconstructed image blocks, and the current waiting is determined in the M reference reconstructed image blocks.
  • Reconstructing a reference pixel of the image block if the encoding mode of the current image block to be reconstructed is a down sampling coding mode, the pixels required for reconstructing the current image block to be reconstructed are downsampled to obtain a reference of the current image block to be reconstructed. Pixel.
  • the reference pixels of the currently reconstructed image block can be effectively determined, thereby realizing the reconstruction of the currently reconstructed image block.
  • the present application provides an image processing method, including: acquiring an encoding mode of a current image block to be encoded of a current image, and a pixel in each reference reconstructed image block corresponding to a current image block to be encoded, where the encoding mode is In the original resolution coding mode or the down sampling coding mode, the current image block to be coded corresponds to M reference reconstructed image blocks, and M is a positive integer greater than or equal to 1; according to the current coding mode of the image block to be encoded and M reference reconstructed image blocks a pixel in the image, determining a plurality of reference pixels of the current image block to be encoded; generating a prediction signal of the current image block to be encoded according to the plurality of reference pixels; acquiring an encoded signal of the current image block to be encoded, wherein when the image block to be currently encoded is When the encoding mode is the original resolution encoding mode, the encoded signal is the original signal of the current image block to be encoded,
  • the encoded signal is the original signal of the current image block to be encoded. a signal obtained after downsampling processing; generating a current to-be-programmed based on the predicted signal and the encoded signal The residual signal of the code image block; the residual signal pair is encoded.
  • the present application considers that the image blocks to be encoded have respective characteristics, and the encoding manners used by the encoding end are also different. Based on this, the encoding end encodes the current image block to be reconstructed. And the encoding method of the M reference reconstructed image blocks are taken into consideration to encode the current image block to be encoded. Thereby the encoding effect of the encoding end is better.
  • the method further includes: generating a reconstructed signal of the current image block to be encoded, and reconstructing the current image block to be encoded according to the reconstructed signal to obtain a current reconstructed image block; if the encoding mode of the current reconstructed image is a downsampling encoding mode, based on the current The pixels of the adjacent reconstructed image block required for the up-sampling process are reconstructed, and the currently reconstructed image block is subjected to upsampling processing.
  • the upsampling process is performed after all the image blocks of the current image are reconstructed.
  • the plurality of reference pixels of the image block to be reconstructed include: obtaining an encoding manner of each of the reference reconstructed image blocks of the M reference reconstructed image blocks; and encoding the encoding manner of the M reference reconstructed image blocks according to the current encoding mode of the image block to be encoded.
  • the pixels in the M reference reconstructed image blocks determining a plurality of reference pixels of the current image block to be encoded.
  • the upsampling process is performed after all the reconstructed image blocks required for performing the upsampling process of the current reconstructed image block are reconstructed; correspondingly, according to the coding mode of the current image block to be encoded and the And determining, by the pixels in the M reference reconstructed image blocks, the plurality of reference pixels of the current image block to be reconstructed, including: acquiring an encoding manner of each of the reference reconstructed image blocks in the M reference reconstructed image blocks; according to the current image block to be encoded
  • the coding mode, the coding mode of the M reference reconstructed image blocks, and the pixels in the M reference reconstructed image blocks determine a plurality of reference pixels of the current image block to be encoded.
  • the upsampling process of the current reconstructed image block includes: performing upsampling on the current reconstructed image block according to a pixel of a part of the adjacent reconstructed image block that is currently reconstructed in the adjacent reconstructed image block. Processing; if another part of the adjacent reconstructed image block of the currently uncompleted reconstruction in the required adjacent reconstructed image block has been reconstructed, then the partial boundary of the currently reconstructed image block is quadraminated according to another partial adjacent reconstructed image block.
  • a sampling process in which a partial boundary of a currently reconstructed image block is adjacent to another partially adjacent reconstructed image block.
  • the upsampling process of the current reconstructed image block includes: performing up-sampling the current reconstructed image block according to a pixel of a portion of the adjacent reconstructed image block that is currently reconstructed in the required adjacent reconstructed image block. Processing; if all the image blocks of the current image have been reconstructed, performing a second upsampling process on a partial boundary of the currently reconstructed image block according to another partially adjacent reconstructed image block in the desired adjacent reconstructed image block And another partial adjacent reconstructed image block is an image block that is not reconstructed when the current up-sampling image block is subjected to the first upsampling process; a partial boundary of the current reconstructed image block is adjacent to another partially adjacent reconstructed image block.
  • a part of the adjacent reconstructed image block is an upper image block and a left image block of the current reconstructed image block, and another partial adjacent reconstructed image block is a lower image block and a right image block of the current reconstructed image block; or, a part of the adjacent image block
  • the reconstructed image block is the upper left image block, the upper image block, the upper right image block and the left image block of the current reconstructed image block
  • the other adjacent reconstructed image block is the right image block of the current reconstructed image block, the lower left image block, and the lower image block and The bottom right image block.
  • a partial boundary of the currently reconstructed image block is a right boundary and a lower boundary of the currently reconstructed image block.
  • the required adjacent reconstructed image block includes an upper image block, a lower image block, a left image block, and a right image block of the current reconstructed image block; or the desired adjacent reconstructed image block includes the current reconstructed image block.
  • the method further includes: identifying that the current reconstructed image block has completed the upsampling process. It is thus possible to avoid repeated upsampling of the currently reconstructed image block.
  • the reference pixels include: if the encoding mode of the current image block to be encoded is the original resolution encoding mode, and the M reference reconstructed image blocks include the reference reconstructed image block whose encoding mode is the original resolution encoding mode, then the encoding mode is original Determining the reference pixel of the current image block to be encoded in the pixel of the reference reconstructed image block of the resolution coding mode; if the current coding mode of the image block to be encoded is the original resolution coding mode, and the M reference reconstructed image blocks include the coding mode
  • the reference reconstructed image block of the sampling and coding mode acquires pixels required for reconstructing the current reconstructed image block from the pixels of the reference reconstructed image block whose encoding mode is the downsampling coding mode, and reconstructs the current image block to be encoded.
  • the required pixels are subjected to upsampling processing to obtain reference pixels of the current image block to be encoded;
  • the encoding mode of the coded image block is a down-sampling coding mode
  • the M reference reconstructed image blocks include a reference reconstructed image block whose coding mode is a down-sampling coding mode, and is in a pixel of a reference reconstructed image block whose coding mode is a down-sampling coding mode.
  • Determining a reference pixel of the current image block to be encoded if the encoding mode of the current image block to be encoded is a downsampling coding mode, and the M reference reconstructed image blocks include a reference reconstructed image block whose encoding mode is the original resolution encoding mode, the encoding is performed.
  • the method is: acquiring pixels required for reconstructing the current image block to be encoded in the pixels of the reference reconstructed image block of the original resolution coding mode, and performing downsampling processing on the pixels required for reconstructing the current image block to be encoded, to obtain The reference pixel of the current image block to be encoded.
  • determining, according to the current coding mode of the image block to be encoded and the pixels in the M reference reconstructed image blocks, multiple reference pixels of the current image block to be encoded including: if currently coded If the encoding mode of the image block is the original resolution encoding mode, the reference pixels of the current image block to be encoded are determined in the pixels of the M reference reconstructed image blocks; if the encoding mode of the current image block to be encoded is the down sampling encoding mode, then The pixels required for reconstructing the current image block to be encoded are acquired in the pixels of the M reference reconstructed image blocks, and the pixels required for reconstructing the current image block to be encoded are subjected to down sampling processing to obtain the current image block to be encoded. Reference pixel.
  • the reference pixels of the currently reconstructed image block can be effectively determined by the above two optional methods, thereby realizing reconstruction of the currently reconstructed image block.
  • the obtaining the encoding mode of the current image block to be encoded includes: determining a first encoding cost when the current image block to be encoded adopts the original resolution encoding mode; determining a second when the current image block to be encoded adopts the down sampling encoding mode Encoding cost; the coding mode corresponding to the smaller coding cost of the first coding cost and the second coding cost is used as the coding mode of the current image block to be coded.
  • the coding mode used by the image block to be coded is a coding mode with a small coding cost, thereby reducing the coding complexity of the coding end, thereby improving the coding efficiency of the coding end.
  • the present application provides an image processing apparatus, including: a parsing module, configured to parse a code stream, to obtain an encoding mode of a current image block to be reconstructed of a current image, a residual signal of a current image block to be reconstructed, and a current waiting Reconstructing pixels in each reference reconstructed image block corresponding to the image block, wherein the encoding mode is the original resolution encoding mode or the down sampling encoding mode, and the current image block to be reconstructed corresponds to M reference reconstructed image blocks, where M is greater than or equal to 1 a positive integer; a determining module, configured to determine, according to an encoding manner of the current image block to be reconstructed and pixels in the M reference reconstructed image blocks, a plurality of reference pixels of the current image block to be reconstructed; and a generating module, configured to a pixel is used to generate a prediction signal of the current image block to be reconstructed; a reconstruction module is configured to generate a
  • the present application provides an image processing apparatus, including: an obtaining module, configured to acquire an encoding manner of a current image block to be encoded of a current image and a pixel in each reference reconstructed image block corresponding to a current image block to be encoded,
  • the encoding mode is the original resolution encoding mode or the down sampling encoding mode.
  • the current image block to be encoded corresponds to M reference reconstructed image blocks, and M is a positive integer greater than or equal to 1.
  • the determining module is configured to use the current image block to be encoded.
  • Encoding mode and pixels in the M reference reconstructed image blocks determining a plurality of reference pixels of the current image block to be encoded; generating a module, configured to generate a prediction signal of the current image block to be encoded according to the plurality of reference pixels;
  • An encoded signal for acquiring a current image block to be encoded wherein when the encoding mode of the current image block to be encoded is the original resolution encoding mode, the encoded signal is the original signal of the current image block to be encoded, when the image block to be currently encoded is When the coding mode is the downsampling coding mode, the coded signal is the original signal of the current image block to be coded.
  • the signal obtained after processing the samples; generating module is further configured to generate a residual signal of a current image block to be encoded and a prediction signal based on the encoded signal; an encoding module for encoding the residual signal.
  • the present application provides an image processing system, comprising: the image processing device according to the third aspect, and the image processing device according to the fourth aspect.
  • the present application provides an image processing apparatus including a decoder configured to:
  • a pixel in the image determining a plurality of reference pixels of the image block to be reconstructed; generating a prediction signal of the current image block to be reconstructed according to the plurality of reference pixels; generating a reconstruction signal of the current image block to be reconstructed according to the prediction signal and the residual signal, and according to The reconstructed signal reconstructs the current image block to be reconstructed to obtain a current reconstructed image block.
  • the present application provides an image processing apparatus including an encoder configured to:
  • the encoding mode of the current image block to be encoded of the current image and the pixel in each reference reconstructed image block corresponding to the current image block to be encoded, where the encoding mode is the original resolution encoding mode or the down sampling encoding mode, and the current image to be encoded
  • the block corresponds to M reference reconstructed image blocks, and M is a positive integer greater than or equal to 1; determining a plurality of reference pixels of the current image block to be encoded according to the encoding mode of the current image block to be encoded and the pixels in the M reference reconstructed image blocks Generating a prediction signal of the current image block to be encoded according to the plurality of reference pixels; acquiring an encoded signal of the current image block to be encoded, wherein when the encoding mode of the current image block to be encoded is the original resolution encoding mode, the encoded signal is currently The original signal of the image block is encoded.
  • the encoded signal is a signal obtained by down-sampling the original signal of the current image block to be encoded; and is generated according to the prediction signal and the encoded signal.
  • the residual signal of the current image block to be encoded encoding the residual signal.
  • the present application provides a computer storage medium for storing computer software instructions for use in the image processing apparatus of the third aspect or the sixth aspect, which is configured to perform the third aspect or the sixth aspect described above. Designed program.
  • the present application provides a computer program product comprising instructions which, when executed by a computer, cause the computer to perform the functions performed by the image processing apparatus of the third aspect or the sixth aspect.
  • the present application provides a computer storage medium for storing computer software instructions for use in the image processing device of the fourth aspect or the seventh aspect, which is configured to perform the fourth aspect or the seventh aspect described above. Designed program.
  • the present application provides a computer program product comprising instructions which, when executed by a computer, cause the computer to perform the functions performed by the image processing apparatus of the fourth aspect or the seventh aspect.
  • the present application provides an image processing method, device, and system. Since the present application considers that image blocks have respective characteristics, the encoding end uses different encoding methods for them, and based on this, the decoding end will image the current image to be reconstructed.
  • the encoding method of the block and the encoding mode of the M reference reconstructed image blocks are taken into consideration to reconstruct the current image block to be reconstructed. Thereby, the reconstructed image block obtained by the decoding end is more effective.
  • the encoding end takes into account two factors of the encoding mode of the current image block to be reconstructed and the encoding mode of the M reference reconstructed image blocks to encode the current image block to be encoded. Thereby the encoding effect of the encoding end is better.
  • FIG. 1 is a schematic diagram of a digital video provided by the present application.
  • FIG. 2 is a schematic diagram of a digital image provided by the present application.
  • FIG. 3 is a schematic diagram of coding of an encoding end provided by the present application.
  • FIG. 4 is a schematic diagram of decoding of a decoding end provided by the present application.
  • FIG. 5 is a schematic diagram of coding of an encoding end provided by the present application.
  • FIG. 6 is a schematic diagram of an image being encoded according to an embodiment of the present application.
  • FIG. 7 is a schematic diagram of a reference pixel template according to an embodiment of the present application.
  • FIGS. 8A and 8B are schematic diagrams of a Planar mode according to an embodiment of the present application.
  • FIG. 9 is a schematic diagram of specific directions of 33 kinds of angle prediction modes according to an embodiment of the present application.
  • FIG. 10 is a schematic diagram of image down sampling according to an embodiment of the present application.
  • FIG. 11 is a schematic diagram of image up sampling according to an embodiment of the present application.
  • FIG. 12 is a schematic diagram of image up sampling according to an embodiment of the present application.
  • FIG. 13 is a flowchart of an image processing method according to an embodiment of the present application.
  • FIG. 14 is a schematic diagram of a 4-neighbor pixel provided by an embodiment of the present application.
  • FIG. 15 is a schematic diagram of an 8-neighbor pixel provided by an embodiment of the present application.
  • FIG. 16 is a schematic diagram of image upsampling according to an embodiment of the present disclosure.
  • FIG. 17 is a schematic diagram of image upsampling according to another embodiment of the present disclosure.
  • FIG. 18 is a schematic diagram of image upsampling according to an embodiment of the present disclosure.
  • FIG. 19 is a schematic diagram of image upsampling according to another embodiment of the present application.
  • FIG. 20 is a schematic diagram of a current image block to be reconstructed and a reference reconstructed image block according to an embodiment of the present disclosure
  • FIG. 21 is a schematic diagram of a current image block to be reconstructed and a reference reconstructed image block according to an embodiment of the present disclosure
  • FIG. 22 is a schematic diagram of a current image block to be reconstructed and a reference reconstructed image block according to an embodiment of the present disclosure
  • FIG. 23 is a schematic diagram of a current image block to be reconstructed and a reference reconstructed image block according to an embodiment of the present disclosure
  • FIG. 24 is a flowchart of an image processing method according to another embodiment of the present application.
  • FIG. 25 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present disclosure.
  • FIG. 26 is a schematic structural diagram of an image processing apparatus according to another embodiment of the present disclosure.
  • FIG. 27 is a schematic structural diagram of an image processing system provided by the present application.
  • Digital video is a video recorded in digital form.
  • Digital video consists of multi-frame digital images. Often the original video contains a large amount of data, which is not suitable for storage and transmission, and requires efficient video compression coding techniques to compress the original data.
  • Video compression technology achieves compression by eliminating video redundancy.
  • Video redundancy mainly includes the following items: spatial redundancy, temporal redundancy, visual redundancy, and information entropy redundancy.
  • Spatial redundancy It is the most important data redundancy for static images. It means that the amplitudes of adjacent pixels are similar in an image. This spatial coherence is called spatial correlation or spatial redundancy.
  • the spatial redundancy is mainly eliminated by the intra prediction method.
  • the intra prediction method refers to using the correlation of the video spatial domain to predict the pixels of the current image block by using the pixels of the reference reconstructed image block to achieve the purpose of removing the video spatial redundancy. .
  • Temporal redundancy It is the redundancy that is often included in video sequences. Since adjacent images of video often contain the same or similar background and moving objects, only the spatial position of the moving object is slightly different. The high correlation of data is called time redundancy. Temporal redundancy is mainly eliminated by inter-frame prediction techniques, which refer to the use of pixels of temporally adjacent images to predict the current pixel.
  • Visual redundancy The human visual system is insensitive to changes in image detail, and even if these subtle changes are lost, the human eye does not feel it.
  • the visual redundancy is mainly eliminated by transforming and transforming technology, which refers to transforming the image signal into the frequency domain for processing, and performing data expression and bit redistribution according to the contribution of the different frequency signals to the visual quality, so that the space can be corrected. Unreasonable expression of uniform sampling over a domain.
  • the need to remove visual redundancy is taken into consideration, and the quantization operation is omitted, and the excessively high-frequency component expression is omitted to achieve effective compression.
  • Information entropy redundancy It can be known from information theory that as a pixel representing image data, it is only necessary to allocate the corresponding number of bits according to the size of its information entropy, and for each pixel of image data, it is difficult to obtain its information at the time of image acquisition. Entropy, therefore, is generally expressed by the same number of bits per pixel, so there must be redundancy. Information entropy redundancy is mainly solved by entropy coding technology. Entropy coding technology distributes different bit numbers for data with different information entropy through the information entropy distribution of statistical coefficients.
  • the current mainstream video compression coding architecture is a hybrid coding architecture.
  • different technologies are adopted to eliminate redundancy, and these technologies are combined to form a hybrid architecture of video coding.
  • the encoding side divides the image into image blocks to be encoded for each frame of the image constituting the video.
  • the current image block to be encoded is first predicted by referring to the reconstructed image block to obtain a prediction signal of the current image block to be encoded; and the residual signal is obtained by subtracting the prediction signal from the original signal of the current image block to be encoded. .
  • the magnitude of the residual signal is much smaller than the original signal.
  • the residual signal is transformed and quantized.
  • the transform quantized coefficients are obtained, and the quantized coefficients and other indication information in the encoding are encoded by the entropy coding technique to obtain a code stream. Further, the encoding end further needs to reconstruct the current image block to be encoded, so as to provide reference pixels for encoding the subsequent image block to be encoded.
  • the encoding end needs to perform inverse quantization and inverse transform on the transform quantized coefficients of the current image block to be encoded to obtain a reconstructed residual signal, and the reconstructed residual signal is
  • the prediction signals corresponding to the current image block to be encoded are added to obtain a reconstructed signal of the current image block to be encoded, and the reconstructed image block is obtained according to the reconstructed signal.
  • the decoding end after acquiring the code stream, the decoding end first performs entropy decoding on the code stream to obtain a transformed quantized coefficient of the current image block to be reconstructed, and then inverse quantizes and inverse transforms the transformed quantized coefficient to obtain an image to be reconstructed.
  • the residual signal of the reconstruction of the block Predicting the current image block to be reconstructed by referring to the reconstructed image block, obtaining a prediction signal of the current image block to be reconstructed, and then adding the prediction signal and the reconstructed residual signal to obtain a reconstruction signal of the current image block to be reconstructed, and then according to The reconstructed signal obtains a current reconstructed image block corresponding to the current image block to be reconstructed.
  • the encoding end first downsamples the image of each frame. As shown in FIG. 5, the encoding end uses downsampling processing on the entire image, and then each of the images subjected to the downsampling process.
  • the coded image block is encoded to obtain a code stream.
  • the resolution of the reconstructed image block corresponding to each image block to be encoded is a downsampling resolution.
  • the decoding end parses the code stream, and the resolution of each image block to be reconstructed is the downsampling resolution, and the resolution of the corresponding reconstructed image block is also the downsampling resolution, and the decoding end needs to adopt the reconstructed image block. Sampling processing to obtain a reconstructed image block of the original resolution.
  • the encoding end and the decoding end both involve predicting the current image block (the current image block to be encoded or the current image block to be reconstructed) by referring to the reconstructed image block, and obtaining a prediction signal of the current image block.
  • the prediction mode (mainly the intra prediction method) of the current image block may adopt the prior art, as follows:
  • FIG. 6 is a schematic diagram of an image being encoded according to an embodiment of the present application.
  • the image includes a plurality of image blocks, where the encoding order of the image is: from top to bottom, from left to right. right.
  • image blocks C, B, D, E, and A represent reconstructed image blocks that have been reconstructed
  • image block F is the current image block to be encoded
  • other regions in the image are uncoded image regions.
  • H.265 supports dividing the current image block to be encoded into smaller sub-image blocks for prediction operations.
  • the partition structure of the sub-image block is a quadtree structure, that is, one image block can be divided into four sub-image blocks, and each sub-image block can be further divided into four sub-image blocks.
  • FIG. 6 it is assumed that the current image block to be encoded is divided into 7 sub-image blocks for prediction operation, and the current image block to be encoded can also be divided into more sub-image blocks for prediction operations.
  • FIG. 7 is a schematic diagram of a reference pixel template according to an embodiment of the present application. As shown in FIG. 7, P 1, 1 , P 2 , 1 ...
  • the sub-image block to be encoded may be the sub-image in FIG. Block 1.
  • other reference pixels R 0,0 , R 1,0 ... R 2N+1,0 ... R 0,2N constitute a reference pixel template, assuming that the image is to be encoded.
  • the sub-image block is the sub-image block 1 in FIG. 6, in which case some of the reference pixels are the pixels of the last row of the reference reconstructed image block B, and the other portion of the pixels are the rightmost side of the reference reconstructed image block A A column of pixels.
  • some of the reference pixels are pixels of the lower plurality of rows included in the reference reconstructed image block B, and the other partial pixels are pixels of the right plurality of columns included in the reference reconstructed image block A. That is, the present application does not limit the reference pixel template.
  • FIG. 8A and FIG. 8B are schematic diagrams of the Planar mode provided by an embodiment of the present application. As shown in FIG. 8, two linear filters in the horizontal and vertical directions are used, respectively. Two predicted values with And with The average value is used as a prediction signal for the pixel (x, y).
  • the DC mode is applicable to a large area flat area, and the prediction signal of the current sub-image block to be encoded can be obtained from the average value of the reference pixels on the left and the top thereof, as shown in FIG. 7, the prediction of each pixel in the sub-image block to be encoded.
  • the signal can be obtained by averaging R 0,1 ,..., R 0,N , R 1,0 ,..., R N,0 .
  • FIG. 9 is a schematic diagram of specific directions of 33 kinds of angle prediction modes according to an embodiment of the present invention. As shown in FIG. 9 , the 33 angle prediction modes are classified into a horizontal type mode (2 to 17) and a vertical type mode (18 to 34). ). Where V0 (mode 26) and H0 (mode 10) represent the vertical and horizontal directions, respectively, and the prediction directions of the remaining angle prediction modes can be regarded as an angular offset in the vertical or horizontal direction.
  • the angle prediction process is illustrated by taking the vertical direction V0 (26) as an example.
  • the vertical direction prediction is to predict the current sub-image block to be encoded using a row of reference pixels adjacent to the sub-image block to be currently encoded, which is currently to be encoded.
  • the position of the reference pixel may be the position between two adjacent reference pixels. If this is the case, a reference pixel needs to be interpolated between the two reference pixels according to the calculated position.
  • a prediction signal is generated from the obtained reference pixels.
  • the present application also relates to image downsampling processing and image upsampling processing.
  • the image downsampling process involves three aspects of information: 1. Downsampling ratio; 2. Downsampling position; 3. Filter used for downsampling.
  • the downsampling ratio refers to the ratio of the original image to the downsampled image, which can be described in the horizontal direction and the vertical direction, respectively.
  • the image signal can be downsampled in the horizontal direction by 2:1, the vertical direction is downsampled by 4:1; or the horizontal direction is not downsampled, the vertical direction is downsampled by 2:1; or both horizontal and vertical directions are performed under 2:1. Sampling, etc.
  • the down sampling position refers to the positional relationship between the down sampling point and the original sampling point.
  • the position of the down sampling point may be the same as the position of some of the original sampling points, or the down sampling point falls between several original sampling points.
  • the downsampling filter may be a 3-lobe Lanczos filter, a Bilinear filter, a Bicubic, a Gauss filter, or the like.
  • FIG. 10 is a schematic diagram of image down sampling according to an embodiment of the present invention. It is assumed that the sampling ratios in the horizontal direction and the vertical direction are both 2:1. In the horizontal direction, the position of the down sampling point falls to the left of the two original sampling points. The original sampling point position, in the vertical direction, the down sampling point falls on the original sampling point position above the two original sampling points. As shown in Figure 10, the circled circle indicates the position of the downsampling point.
  • the downsampling filter is as follows:
  • the filter is a simple low-pass filter that can be thought of as a two-dimensional filter or as two one-dimensional filters. If used as a two-dimensional filter, downsampling in both horizontal and vertical directions can be done simultaneously in one filtering operation. As shown in FIG. 10, when downsampling point A is downsampled, eight adjacent original sampling points (circles circled by a triangle) are used, and the pixel value of the down sampling point A is calculated according to the above filter. If it is treated as two one-dimensional filters, it is necessary to first perform down-sampling in the horizontal or vertical direction, and then down-sample the results of the horizontal or vertical down-sampling in the completed horizontal or vertical direction. As shown in FIG.
  • downsampling point A when downsampling point A is down-sampled, firstly, one original sampling point is used to perform horizontal down sampling, and then the down-sampling point A is used to sample one original sample. The point is downsampled in the vertical direction, and the pixel value of the down sample point A is calculated according to the above filter. Using the same method, the entire 16*16 image block is downsampled. The final downsampling result is shown in Figure 10. The position of each downsampled point is shown by the circle in the box, and the pixel value of the downsampled point is The value after passing the filter operation. As shown in FIG. 10, the resolution of the downsampled image block is 8*8.
  • the upsampling process involves three aspects of information: 1. Upsampling ratio; 2. Upsampling position; 3 Filter used for upsampling.
  • the upsampling ratio refers to the ratio of the image before the upsampling to the image after the upsampling, which can be described in the horizontal direction and the vertical direction, respectively.
  • the upsampled image signal can be upsampled in the horizontal direction by 1:2, the vertical direction is 1:4 upsampled; or the horizontal direction is not upsampled, the vertical direction is 1:2 upsampled; or both horizontal and vertical directions are Perform 1:2 upsampling, etc.
  • the upsampling position is a positional relationship between the sampled point after the upsampling and the sampled point before the upsampling.
  • FIG. 11 is a schematic diagram of image upsampling provided in an embodiment of the present application, as shown in FIG.
  • the horizontal 1:2 upsampling ratio is used, and the position of the upsampled sample point can be on the right side of the sample point before the upsampling, where ⁇ represents the position of the sampled point after upsampling, and the circle represents the position of the sampled point before the upsampling.
  • the horizontal 1:2 upsampling ratio is used, and the position of the upsampled sample point can be to the left of the sample point before the upsampling, where ⁇ represents the position of the sampled point after upsampling, and the circle represents the position before the upsampling.
  • the sampling point location It should be noted that the position of the sampled point after the upsampling should correspond to the position selection of the above-mentioned down sampling point. For example, when performing downsampling, the position of the down sampling point is selected as the position of the original sampling point on the left side thereof. Then, when performing upsampling, the position of the sampled point after upsampling is selected as the position of the sampling point (downsampling point) before the upper right sampling.
  • the upsampling filter may be a DCTIF filter, a bilinear interpolation filter, a sinc filter, or the like.
  • the upsampling process will be described below by taking an image block with a resolution of 8*8 (i.e., the image block after downsampling described above) as an example. Assume that the upsampling ratios in the horizontal direction and the vertical direction are both 1:2.
  • the position of the sampled point after upsampling is the position of the sampling point before the right upsampling, and in the vertical direction, the upsampling
  • the position of the subsequent sampling point is the position of the sampling point before the upper upsampling, where the horizontal and vertical directions are respectively sampled as an example, and the DCTIF filter is taken as an example to illustrate the upsampling process.
  • the DCTIF filter is (-1, 4, -11, 40, 40, -11, 4, -1), assuming that the horizontal direction is currently being sampled.
  • Figure 11 assuming that the B3 sample point needs to be inserted, the following is used.
  • the formula determines the pixel value of B3:
  • FIG. 12 is a schematic diagram of image upsampling according to an embodiment of the present invention. As shown in FIG. 12, x represents a sampled point after upsampling, and a circle represents a sample point before upsampling.
  • the encoding end uses downsampling processing for the entire image.
  • the characteristics of each image block included in the entire image may be different.
  • some image blocks may be relatively flat, and the encoding end is suitable for downsampling processing.
  • Some image blocks may have more details, in which case the downsampling process will lose these details. This will result in poor coding on the encoding side.
  • this will result in some reconstructed image blocks obtained by the decoding end being relatively ambiguous, that is, the reconstructed image blocks obtained by the decoding end are not effective.
  • the present application provides an image processing method, device, and system.
  • the present application is based on the coding diagrams of FIG. 3 and FIG. 5.
  • the coding mode of the image block to be encoded included in one image may be the original resolution coding mode as shown in FIG.
  • the original resolution encoding method refers to directly performing an encoding operation on the current image block to be encoded.
  • the downsampling coding mode refers to performing downsampling processing on the current image block to be encoded, and then performing encoding operation on the currently sampled image block to be coded.
  • the texture image block adopts the original resolution coding mode
  • the smooth image block adopts the down sampling coding mode.
  • the encoding side needs to mark the encoding method used by each image block to be encoded, and write the label into the code stream.
  • the decoding end is caused to perform corresponding operations on the image block to be reconstructed according to the mark.
  • the main idea of the present application is that the decoding end determines the current to be reconstructed according to the coding mode of the current image block to be reconstructed, the coding mode of the M reference reconstructed image blocks corresponding to the current image block to be reconstructed, and the pixels in the M reference reconstructed image blocks.
  • a plurality of reference pixels of the image block generate a prediction signal according to the plurality of reference pixels, thereby reconstructing the image block to be reconstructed.
  • the encoding end determines the current image block to be encoded according to the encoding mode of the current image block to be encoded, the encoding mode of the M reference reconstructed image blocks corresponding to the current image block to be encoded, and the pixels in the M reference reconstructed image blocks.
  • the reference pixels generate a prediction signal according to the plurality of reference pixels, thereby obtaining a residual signal, and encoding the residual signal.
  • FIG. 13 is a flowchart of an image processing method according to an embodiment of the present disclosure. As shown in FIG. 13, the method includes:
  • Step S1301 Parsing the code stream to obtain an encoding mode of the current image block to be reconstructed of the current image, a residual signal of the current image block to be reconstructed, and a pixel in each reference reconstructed image block corresponding to the current image block to be reconstructed;
  • the decoding end parses the code stream, where the code stream carries the label corresponding to the encoding mode of the current image block to be reconstructed, and the transform quantization coefficient of the current image block to be reconstructed.
  • the decoding end may perform inverse quantization and inverse transform on the transform quantized coefficients to obtain a residual signal of the current image block to be reconstructed.
  • the current image block to be reconstructed corresponds to M reference reconstructed image blocks, and M is a positive integer greater than or equal to 1.
  • the reference reconstructed image block is used to determine a plurality of reference pixels of the image block to be reconstructed, wherein the plurality of reference pixels are used to generate a prediction signal of the current reconstructed image block.
  • the reference reconstructed image block is specifically related to which prediction image block and the prediction mode employed by the decoding end. When any of the above-described prediction modes in 35 is employed, reference may be made to the reference pixel template as shown in FIG.
  • Step S1302 Determine a plurality of reference pixels of the current image block to be reconstructed according to the coding mode of the current image block to be reconstructed and the pixels in the M reference reconstructed image blocks.
  • the reference reconstructed image block has the same resolution as the current image block to be reconstructed, at least one reference pixel is directly determined in the reference reconstructed image block; if the current image block to be reconstructed is the original resolution, the resolution of the reference reconstructed image block is determined.
  • Rate the downsampling resolution obtain at least one pixel required for reconstructing the current reconstructed image block from the reference reconstructed image block, and perform upsampling processing on at least one pixel required for reconstructing the current reconstructed image block
  • At least one pixel required for reconstruction of the image block to be reconstructed is currently subjected to downsampling processing to obtain at least one reference pixel of the current image block to be reconstructed.
  • Step S1303 Generate a prediction signal of the current image block to be reconstructed according to the plurality of reference pixels
  • Step S1304 Generate a reconstruction signal of the current image block to be reconstructed according to the prediction signal and the residual signal, and reconstruct the current image block to be reconstructed according to the reconstruction signal to obtain a current reconstructed image block.
  • the prediction mode of the current image block to be reconstructed is generated according to the plurality of reference pixels, and any one of the prediction modes in the foregoing 35 prediction mode may be used, and of course, other prediction modes in the prior art may also be used. Make restrictions. Finally, the reconstructed residual signal is added to the prediction signal to obtain a reconstructed signal of the current image block to be reconstructed, and the current image block to be reconstructed is reconstructed according to the reconstructed signal to obtain a current reconstructed image block.
  • the present application considers that image blocks have their own characteristics, and the encoding end uses different encoding methods for them. Based on this, the decoding end takes the factor of the encoding mode of the image block to be reconstructed into consideration to reconstruct The image block to be reconstructed currently. Thereby, the reconstructed image block obtained by the decoding end is more effective.
  • the image processing method further includes: pixels of the adjacent reconstructed image block required for performing the upsampling process on the current reconstructed image block, and the current reconstructed image block Perform upsampling processing.
  • the pixels of the adjacent reconstructed image blocks are mainly used for upsampling the partial boundaries of the currently reconstructed image block, and the portions of the current reconstructed image block except the above partial boundaries are used. The pixels are upsampled.
  • the decoding end uses the filter to perform upsampling processing on the current reconstructed image block, the adjacent reconstructed image blocks required for the current reconstructed image block are also different for different filters.
  • FIG. 14 is a A schematic diagram of a 4-neighbor pixel provided by an embodiment, as shown in FIG. 14, the adjacent reconstructed image block required for the current reconstructed image block includes: an upper image block, a lower image block, a left image block, and a right image of the current reconstructed image block. Piece. It is assumed that the filter is a Convolutional Neural Network (CNN) filter. In this case, the adjacent reconstructed image block required for reconstructing the image block is as follows: FIG.
  • CNN Convolutional Neural Network
  • the adjacent reconstructed image block includes: an upper image block, a lower image block, a left image block, a right image block, an upper left image block, a lower left image block, an upper right image block, and a lower right image block of the current reconstructed image block.
  • the current coding order from top to bottom, from left to right coding order
  • the lower image block, the right image block, the lower left image block, and the lower right image block of the current reconstructed image block have not yet been reconstructed.
  • the technique is to achieve upsampling processing by copying the pixels of the current reconstructed image block itself, but this way will cause the current reconstructed image block after the upsampling process to have a problem that the right and lower boundaries are discontinuous.
  • the present application provides the following four options:
  • the upsampling process is performed after all the reconstructed image blocks required for the upsampling process of the current reconstructed image block are reconstructed; correspondingly, the code stream includes: each of the M reference reconstructed image blocks. Referring to the coding mode of the reconstructed image block; determining, according to the current coding mode of the image block to be reconstructed and the pixels in the M reference reconstructed image blocks, the plurality of reference pixels of the current image block to be reconstructed, including: coding according to the current image block to be reconstructed The manner, the coding manner of the M reference reconstructed image blocks, and the pixels in the M reference reconstructed image blocks determine a plurality of reference pixels of the current image block to be reconstructed.
  • the upsampling process is performed after all the image blocks of the current image are reconstructed.
  • the code stream includes: a coding mode of each reference reconstructed image block in the M reference reconstructed image blocks;
  • the coding mode of the image block and the pixels in the M reference reconstructed image blocks determine a plurality of reference pixels of the current image block to be reconstructed, including: coding manner according to the current image block to be reconstructed, coding mode of the M reference reconstructed image blocks, and
  • the M reference reconstructed pixels in the image block determine a plurality of reference pixels of the current image block to be reconstructed.
  • Manner 3 performing a upsampling process on the currently reconstructed image block according to a pixel of a part of the adjacent reconstructed image block that is currently reconstructed in the required adjacent reconstructed image block; if the required adjacent reconstructed image block is in the If another part of the adjacent reconstructed image block that has not been reconstructed has been reconstructed, the partial boundary of the currently reconstructed image block is subjected to a second upsampling process according to another part of the adjacent reconstructed image block, wherein a part of the boundary of the currently reconstructed image block is Another portion of the adjacent reconstructed image blocks are contiguous.
  • the method of performing the upsampling process on the current reconstructed image block includes: performing an upsampling process on the currently reconstructed image block according to the pixels of the part of the adjacent reconstructed image block that are currently reconstructed in the required adjacent reconstructed image block; If all the image blocks of the current image have been reconstructed, the partial boundary of the currently reconstructed image block is subjected to secondary upsampling according to another partially adjacent reconstructed image block in the desired adjacent reconstructed image block, wherein A portion of the adjacent reconstructed image block is an image block that is not reconstructed when the first upsampling process is performed on the current reconstructed image block; a partial boundary of the current reconstructed image block is adjacent to another partially adjacent reconstructed image block.
  • the partial boundary of the current reconstructed image block satisfies the condition that, in the first upsampling process performed on the current reconstructed image block, another part of the adjacent reconstructed image block required for the partial boundary is not reconstructed.
  • a partial boundary of the currently reconstructed image block is a right boundary and a lower boundary of the currently reconstructed image block.
  • the part of the adjacent reconstructed image block is an upper image block and a left image block of the current reconstructed image block
  • the another partial adjacent reconstructed image block is a lower image block and a right side of the current reconstructed image block.
  • the part of the adjacent reconstructed image block is an upper left image block, an upper image block, an upper right image block and a left image block of the current reconstructed image block, and the other partially adjacent reconstructed image block is a right side of the current reconstructed image block.
  • the adjacent reconstructed image blocks required to reconstruct the image block are also different for different filters.
  • the adjacent reconstructed image block required for the current reconstructed image block includes: an upper image block, a lower image block, a left image block, and a right image block of the current reconstructed image block.
  • the adjacent reconstructed image block required for the current reconstructed image block includes: an upper image block, a lower image block, a left image block, a right image block, an upper left image block, a lower left image block, and an upper right of the currently reconstructed image block.
  • Image block and bottom right image block is also different for different filters.
  • FIG. 16 is a schematic diagram of image upsampling provided by an embodiment of the present application.
  • the adjacent reconstructed image block 1, the adjacent reconstructed image block 2, and the adjacent reconstructed image block required for reconstructing the image block B are as shown in FIG. 3 and the adjacent reconstructed image block 4 have all been reconstructed.
  • the current reconstructed image block B is subjected to upsampling processing, as shown in FIG. 16, wherein a circle in B indicates a sampling point before upsampling, and ⁇ indicates a sampling point after upsampling.
  • the currently reconstructed image block itself is a boundary image block of an image, in this case, even if the required adjacent reconstructed image block is reconstructed, it is necessary to copy the current reconstructed image when performing the upsampling process.
  • Pixel For example, as shown in FIG. 14, when the currently reconstructed image block is the rightmost image block of an image, its right image block does not exist, so the rightmost column included in the currently reconstructed image block can be included.
  • the pixels are copied to achieve upsampling processing.
  • other methods may also be used for the upsampling process, which is not limited in this application.
  • each adjacent reconstructed image block required for reconstructing the image block may be a downsampling coding mode or an original resolution coding mode, when performing the upsampling process on the current reconstructed image block, Divided into the following two cases:
  • the current reconstructed image block may be directly upsampled according to the pixels in the adjacent reconstructed image block.
  • the coding mode of an adjacent reconstructed image block is the original resolution coding mode
  • at least one pixel required for the upsampling process in the pixels of the adjacent reconstructed image block may be acquired, and the pixels are downsampled, and The current reconstructed image block is subjected to upsampling processing according to at least one pixel after the downsampling process.
  • the adjacent reconstructed image block is mainly used for performing upsampling processing on a partial boundary of the currently reconstructed image block (the partial boundary is different according to the filter), for example, as shown in FIG. 16, the adjacent reconstructed image block 3
  • the downsampling coding mode is adopted.
  • the right boundary of the current reconstructed image block B can be upsampled directly by using the pixels included in the adjacent reconstructed image block 3.
  • the adjacent reconstructed image block 4 adopts the original resolution encoding mode, the pixels required for the upsampling process included in the adjacent reconstructed image block 4 need to be downsampled, or the adjacent reconstructed image block 4 is subjected to downsampling processing.
  • FIG. 17 is a schematic diagram of image upsampling according to another embodiment of the present application. As shown in FIG. 17, this is a case of 8 neighborhood pixels, and the decoding side samples the current reconstructed image block C by using an upsampling processing method and 4 neighborhoods. The method in the case of a pixel is similar, and will not be described here.
  • the current reconstructed image block may be subjected to upsampling processing, and the current reconstructed image block may be identified as having completed the upsampling process.
  • the current reconstructed image block is subjected to upsampling processing according to a certain rule.
  • the upsampling process is based on the case of 4 neighborhood pixels
  • the current reconstructed image block may be subjected to upsampling processing.
  • the upsampling process is based on the case of 8 neighborhood pixels, once the lower right image block of the currently reconstructed image block is reconstructed, the current reconstructed image block may be subjected to upsampling processing.
  • the adjacent reconstructed image blocks it needs have been reconstructed, based on which, for any reconstructed image using downsampling coding
  • the blocks can be upsampled.
  • the specific upsampling process is similar to the mode of the foregoing mode 1, and the application is not described herein again.
  • the upsampling process performed on the current reconstructed image block includes two upsampling processes.
  • the first upsampling process is: performing a upsampling process on the currently reconstructed image block according to the pixels of a part of the adjacent reconstructed image blocks of the currently completed reconstructed image in the required adjacent reconstructed image block.
  • the second upsampling process is: if another part of the adjacent reconstructed image block of the currently uncompleted reconstruction in the required adjacent reconstructed image block has been reconstructed, the currently reconstructed image block is reconstructed according to another partial adjacent reconstructed image block.
  • the partial boundary is subjected to a second upsampling process in which the partial boundary of the currently reconstructed image block is contiguous with another partially adjacent reconstructed image block.
  • the current reconstructed image block of the downsampling coding mode is saved, and the reference pixels are provided for subsequent prediction of the other image blocks to be reconstructed.
  • the decoding end may copy the pixels of the rightmost column or columns included in the current reconstructed image block to obtain interpolated pixels. Upsampling processing is performed on the right boundary included in the currently reconstructed image block by these interpolated pixels. The decoding end may also copy the pixels of the lowermost row or rows included in the current reconstructed image block to obtain interpolated pixels. Upsampling processing is performed on the lower boundary included in the currently reconstructed image block by these interpolated pixels.
  • FIG. 18 is a schematic diagram of image upsampling according to an embodiment of the present invention.
  • the current reconstructed image block is image block A
  • the current reconstructed image block A is subjected to the second upsampling process
  • the DCTIF filter performs upsampling processing.
  • the current reconstructed image block A has completed the first upsampling process.
  • the right four reference pixels required for each x do not exist. If the adjacent reconstructed image block C is reconstructed, the second upsampling process is performed on the right boundary of the currently reconstructed image block according to the adjacent reconstructed image block C.
  • the upsampling processing method here is the same as the above-described upsampling processing method, and details are not described herein again.
  • the current reconstructed image block is the image block B
  • the DCTIF filter is currently used for the upsampling process
  • the current reconstructed image block B has been completed for the first time.
  • the upsampling process when using the DCTIF filter for upsampling processing as described above, requires four pixels above and below, then in the first upsampling process, the lower four rows of the currently reconstructed image block B ⁇ ( ⁇ indicates The four reference pixels required for the sample point after the first upsampling process are incomplete. For example, for the bottom row x, the lower four reference pixels required for each x do not exist.
  • the second upsampling process is performed on the lower boundary of the currently reconstructed image block according to the adjacent reconstructed image block C.
  • the upsampling processing method here is the same as the above-described upsampling processing method, and details are not described herein again.
  • FIG. 19 is a schematic diagram of image upsampling according to another embodiment of the present application. As shown in FIG. 19, this is the case of 8 neighborhood pixels, and the decoding end is on the right boundary of the currently reconstructed image block A and the current reconstructed image block B.
  • the upsampling processing method used in the lower boundary is similar to the method in the case of 4 neighborhood pixels, and will not be described here.
  • the current reconstructed image block may be identified as having completed the upsampling process.
  • the current reconstructed image block is subjected to upsampling processing according to a certain rule.
  • the upsampling process is based on the case of 4 neighborhood pixels
  • the current upsampling image block may be subjected to the second upsampling process.
  • the upsampling process is based on the case of 8 neighborhood pixels
  • the current upsampling image block may be subjected to a second upsampling process.
  • the processed reconstructed image block can be subjected to the second upsampling process.
  • the specific upsampling process is similar to the mode of the foregoing mode 3, and details are not described herein again.
  • the current reconstructed image block of the downsampling coding mode is saved, and the reference pixels are provided for subsequent prediction of the other image blocks to be reconstructed.
  • step S1302 is described in detail based on the above manners 1 and 2:
  • Step S1302 specifically includes the following four cases:
  • the encoding mode of the image block to be reconstructed is the original resolution encoding mode
  • the M reference reconstructed image blocks include the reference reconstructed image block whose encoding mode is the original resolution encoding mode
  • the encoding mode is the original resolution encoding.
  • the reference pixels of the current image block to be reconstructed are determined in the pixels of the reference reconstructed image block.
  • Case 2 If the encoding mode of the image block to be reconstructed is the original resolution encoding mode, and the M reference reconstructed image blocks include the reference reconstructed image block whose encoding mode is the down sampling encoding mode, the encoding mode is the down sampling encoding mode.
  • the pixels required for reconstructing the current image block to be reconstructed are obtained from the pixels of the reconstructed image block, and the pixels required for reconstructing the current image block to be reconstructed are subjected to upsampling processing to obtain a reference of the current image block to be reconstructed. Pixel.
  • Case 3 If the coding mode of the current image block to be reconstructed is a downsampling coding mode, and the M reference reconstructed image blocks include a reference reconstructed image block whose coding mode is a downsampling coding mode, the coding mode is a reference of the downsampling coding mode. A reference pixel of the current image block to be reconstructed is determined in the pixels of the reconstructed image block.
  • Case 4 If the coding mode of the current image block to be reconstructed is a downsampling coding mode, and the M reference reconstructed image blocks include a reference reconstructed image block whose coding mode is the original resolution coding mode, the coding mode is the original resolution coding mode.
  • the pixels required to reconstruct the current image block to be reconstructed are acquired in the pixels of the reference reconstructed image block, and the pixels required for reconstructing the current image block to be reconstructed are downsampled to obtain a reference of the current image block to be reconstructed. Pixel.
  • FIG. 20 is a schematic diagram of a current image block to be reconstructed and a reference reconstructed image block according to an embodiment of the present application.
  • the M reference reconstructed image blocks corresponding to the image block E to be reconstructed are reference reconstructed image blocks A, B, C, and D.
  • the image block E also needs to use an image.
  • the image block below the block A, but in the current coding order, the image block below the image block A has not been reconstructed, and the prior art realizes the prediction of the image block E by copying the image block A. This application does not elaborate on this.
  • the reference reconstructed image blocks A, B, C and D since their encoding methods are all original resolution encoding methods, and the current encoding mode of the image block E to be reconstructed is also the original resolution encoding mode, it can be reconstructed directly from the reference.
  • the reference pixels of the current image block to be reconstructed are determined in image blocks A, B, C, and D. For example, as shown in FIG. 20, the pixels in the frame are reference pixels.
  • FIG. 21 is a schematic diagram of a current image block to be reconstructed and a reference reconstructed image block according to an embodiment of the present application. As shown in FIG. 21, it is assumed that the reference pixel template shown in FIG. 7 is used in the embodiment of the present application.
  • the M reference reconstructed image blocks corresponding to the image block E to be reconstructed are reference reconstructed image blocks A, B, C, and D, wherein the current image to be reconstructed is encoded in the original resolution encoding mode, and the reference reconstructed image is used.
  • the coding mode of the block B is a downsampling coding mode.
  • the pixel of the lowermost row in the reference reconstructed image block B needs to be acquired, and then the acquired pixel is subjected to upsampling processing to obtain a reference pixel of the current reconstructed image block E.
  • the pixels in the frame are all reference pixels.
  • the reference pixels of the current reconstructed image block E and the like may be determined according to the plurality of rows of pixels in the reference reconstructed image block B, which is not limited in this application.
  • FIG. 22 is a schematic diagram of a current image block to be reconstructed and a reference reconstructed image block according to an embodiment of the present invention.
  • the M reference reconstructed image blocks corresponding to the image block E to be reconstructed are reference reconstructed image blocks A, B, C, and D, wherein the current image to be reconstructed is encoded in the original resolution encoding mode, and the reference reconstructed image is used.
  • the coding modes of blocks A, B and D are all original sampling coding modes.
  • the reference pixels of the current image block to be reconstructed can be directly determined from the reference reconstructed image blocks A, B, C and D.
  • the pixels in the frame are reference pixels.
  • FIG. 23 is a schematic diagram of a current image block to be reconstructed and a reference reconstructed image block according to an embodiment of the present invention. As shown in FIG. 23, it is assumed that the reference pixel template shown in FIG. 7 is used in the embodiment of the present application.
  • the M reference reconstructed image blocks corresponding to the image block E to be reconstructed are reference reconstructed image blocks A, B, C, and D, wherein the current image to be reconstructed is encoded in a downsampling mode, and the reference reconstructed image block is referenced.
  • the encoding mode of B is the original resolution encoding mode.
  • the pixels in the frame are all reference pixels.
  • the downsampling process and the upsampling process involved in the above four cases can be performed by the prior art method, which is not limited in this application.
  • any image block to be reconstructed using the downsampling coding mode after the reconstructed image block is reconstructed, the first upsampling process is performed, so any one The resolution of the reconstructed image block is the original resolution.
  • the step S1302 includes: if the coding mode of the image block to be reconstructed is the original resolution coding mode, determining the reference of the current image block to be reconstructed in the pixels of the M reference reconstructed image blocks. Pixel; if the encoding mode of the image block to be reconstructed is a downsampling coding mode, the pixels required for reconstructing the current image block to be reconstructed are acquired in the pixels of the M reference reconstructed image blocks, and the image block to be reconstructed is currently obtained.
  • the pixels required for reconstruction are subjected to downsampling processing to obtain reference pixels of the current image block to be reconstructed; or, if a neighboring reconstructed image block required for the current image block to be reconstructed is subjected to downsampling coding, the phase is saved. If the encoding mode of the image block to be reconstructed is the down sampling encoding mode, the reference pixel of the current image block to be reconstructed may be directly determined in the previously saved pixel.
  • the reference pixels of the currently reconstructed image block can be effectively determined by the above method, thereby realizing reconstruction of the currently reconstructed image block.
  • the above mainly introduces the image processing method at the decoding end.
  • the image processing method at the encoding end will be described below.
  • FIG. 24 is a flowchart of an image processing method according to another embodiment of the present application. As shown in FIG. 24, the method includes:
  • Step S2401 Obtain an encoding manner of a current image block to be encoded of a current image, and a pixel in each reference reconstructed image block corresponding to the current image block to be encoded;
  • Step S2402 determining, according to an encoding manner of the current image block to be encoded and pixels in the M reference reconstructed image blocks, a plurality of reference pixels of the current image block to be encoded;
  • Step S2403 Generate a prediction signal of a current image block to be encoded according to a plurality of reference pixels
  • Step S2404 Acquire an encoded signal of the current image block to be encoded, wherein when the encoding mode of the current image block to be encoded is the original resolution encoding mode, the encoded signal is the original signal of the current image block to be encoded, and the current image block to be encoded When the coding mode is the downsampling coding mode, the coded signal is a signal obtained by down-sampling the original signal of the current image block to be coded;
  • Step S2405 Generate a residual signal of the current image block to be encoded according to the prediction signal and the encoded signal;
  • Step S2406 Encoding the residual signal.
  • the encoding mode is the original resolution encoding mode or the down sampling encoding mode.
  • the current image block to be encoded corresponds to M reference reconstructed image blocks, and M is a positive integer greater than or equal to 1.
  • the reference reconstructed image block has the same resolution as the current image block to be reconstructed, at least one reference pixel is directly determined in the reference reconstructed image block; if the current image block to be reconstructed is the original resolution, the resolution of the reference reconstructed image block is determined.
  • Rate the downsampling resolution obtain at least one pixel required for reconstructing the current reconstructed image block from the reference reconstructed image block, and perform upsampling processing on at least one pixel required for reconstructing the current reconstructed image block
  • At least one pixel required for reconstruction of the image block to be encoded is currently subjected to downsampling processing to obtain at least one reference pixel of the current image block to be reconstructed.
  • the reference reconstructed image block is specifically related to which prediction image block and the prediction mode employed by the decoding end.
  • any of the above-described prediction modes in 35 reference may be made to the reference pixel template as shown in FIG.
  • the prediction signal of the current image block to be reconstructed is generated according to the plurality of reference pixels, and any one of the prediction modes in the above-mentioned 35 mode may be used. Of course, other prediction modes in the prior art may also be used, which is not limited in this application. .
  • a residual signal of the current image block to be encoded is generated according to the prediction signal and the encoded signal; encoding the residual signal includes: transforming and quantizing the residual signal. After transform quantization, the transform quantized coefficients are obtained, and the quantized coefficients and other indication information in the encoding are encoded by the entropy coding technique to obtain a code stream.
  • the present application considers that the image blocks to be coded have their own characteristics, and the encoding end uses different encoding methods for them. Based on this, the encoding end takes into account the factor of the encoding mode of the image block to be reconstructed. To encode the current image block to be encoded. Thereby the encoding effect of the encoding end is better.
  • the method further includes: generating a reconstructed signal of the current image block to be encoded, and reconstructing the current image block to be encoded according to the reconstructed signal to obtain a current reconstructed image block; if the encoding mode of the current reconstructed image block is a downsampling coding mode, The current reconstructed image block is subjected to upsampling processing based on the pixels of the adjacent reconstructed image block required for the up-sampling process of the current reconstructed image block.
  • the generating the reconstructed signal of the current image block to be encoded includes: transforming and quantizing the residual signal of the current image block to be encoded, obtaining a quantized coefficient of the current image block to be encoded, performing inverse quantization and inverse transform on the quantized coefficient, and reconstructing
  • the residual signal is added to the reconstructed residual signal and the predicted signal of the current image block to be encoded to obtain a reconstructed signal of the current image block to be encoded.
  • the current reconstructed image block is subjected to upsampling processing in order to obtain the current reconstructed image block of the original resolution, thereby preparing for the subsequent interframe prediction technique.
  • Method 1 The upsampling process is performed after all the image blocks of the current image are reconstructed; correspondingly, the current image block to be reconstructed is determined according to the coding mode of the current image block to be encoded and the pixels in the M reference reconstructed image blocks.
  • the plurality of reference pixels include: obtaining an encoding manner of each of the M reference reconstructed image blocks; encoding according to the current image block to be encoded, encoding manners of the M reference reconstructed image blocks, and M references The pixels in the image block are reconstructed to determine a plurality of reference pixels of the current image block to be encoded.
  • the upsampling process is performed after all the reconstructed image blocks required for the upsampling process of the current reconstructed image block are reconstructed; correspondingly, according to the coding mode of the current image block to be encoded and M reference reconstructions.
  • Determining, by a pixel in the image block, a plurality of reference pixels of the image block to be reconstructed comprising: acquiring an encoding mode of each of the reference reconstructed image blocks in the M reference reconstructed image blocks; and encoding according to the current image block to be encoded, M Referring to the coding mode of the reconstructed image block and the pixels in the M reference reconstructed image blocks, a plurality of reference pixels of the current image block to be encoded are determined.
  • the third method of performing the upsampling process on the current reconstructed image block includes: performing an upsampling process on the currently reconstructed image block according to a pixel of a part of the adjacent reconstructed image block that is currently reconstructed in the required adjacent reconstructed image block; If another part of the adjacent reconstructed image block of the currently uncompleted reconstructed image block in the adjacent reconstructed image block has been reconstructed, the partial boundary of the current reconstructed image block is subjected to secondary upsampling processing according to another partial adjacent reconstructed image block. Where a portion of the boundary of the currently reconstructed image block is contiguous with another portion of the adjacent reconstructed image block.
  • the method of performing the upsampling process on the current reconstructed image block includes: performing an upsampling process on the currently reconstructed image block according to the pixels of the part of the adjacent reconstructed image block that are currently reconstructed in the required adjacent reconstructed image block; If all the image blocks of the current image have been reconstructed, the partial boundary of the currently reconstructed image block is subjected to secondary upsampling according to another partially adjacent reconstructed image block in the desired adjacent reconstructed image block, wherein A portion of the adjacent reconstructed image block is an image block that is not reconstructed when the first upsampling process is performed on the current reconstructed image block; a partial boundary of the current reconstructed image block is adjacent to another partially adjacent reconstructed image block.
  • the part of the adjacent reconstructed image block is an upper image block and a left image block of the current reconstructed image block
  • the another partial adjacent reconstructed image block is a lower image block and a right side of the current reconstructed image block.
  • the part of the adjacent reconstructed image block is an upper left image block, an upper image block, an upper right image block and a left image block of the current reconstructed image block, and the other partially adjacent reconstructed image block is a right side of the current reconstructed image block.
  • a partial boundary of the current reconstructed image block is a right boundary and a lower boundary of the current reconstructed image block.
  • the required adjacent reconstructed image block includes an upper image block, a lower image block, a left image block, and a right image block of the current reconstructed image block; or, the desired adjacent reconstructed image block includes the current
  • the upper image block, the lower image block, the left image block, the right image block, the upper left image block, the lower left image block, the upper right image block, and the lower right image block of the image block are reconstructed.
  • step S2402 specifically includes the following four situations:
  • the encoding mode of the current image block to be encoded is the original resolution encoding mode
  • the M reference reconstructed image blocks include the reference reconstructed image block whose encoding mode is the original resolution encoding mode
  • the encoding mode is the original resolution encoding.
  • the reference pixels of the current image block to be encoded are determined in the pixels of the reference reconstructed image block.
  • Case 2 If the encoding mode of the current image block to be encoded is the original resolution encoding mode, and the M reference reconstructed image blocks include the reference reconstructed image block whose encoding mode is the down sampling encoding mode, the encoding mode is the down sampling encoding mode. Obtaining a pixel required for reconstructing the current reconstructed image block in a pixel of the reconstructed image block, and performing upsampling processing on a pixel required for reconstructing the current image block to be encoded to obtain a reference pixel of the current image block to be encoded .
  • Case 3 If the encoding mode of the current image block to be encoded is the downsampling coding mode, and the M reference reconstructed image blocks include the reference reconstructed image block whose coding mode is the downsampling coding mode, the coding mode is the reference of the downsampling coding mode.
  • a reference pixel of the current image block to be encoded is determined in the pixels of the reconstructed image block.
  • Case 4 If the encoding mode of the current image block to be encoded is a downsampling coding mode, and the M reference reconstructed image blocks include a reference reconstructed image block whose encoding mode is the original resolution encoding mode, the encoding mode is the original resolution encoding mode.
  • the pixels required for reconstructing the current image block to be encoded are acquired in the pixels of the reference reconstructed image block, and the pixels required for reconstructing the current image block to be encoded are subjected to down sampling processing to obtain a reference of the current image block to be encoded. Pixel.
  • the step S2402 includes: if the encoding mode of the current image block to be encoded is the original resolution encoding mode, determining the reference pixel of the current image block to be encoded in the pixels of the M reference reconstructed image blocks; If the encoding mode of the current image block to be encoded is the down sampling coding mode, the pixels required for reconstructing the current image block to be encoded are acquired in the pixels of the M reference reconstructed image blocks, and the current image block to be encoded is reconstructed. The required pixels are subjected to downsampling processing to obtain reference pixels of the current image block to be encoded.
  • the method further includes: identifying that the current reconstructed image block has completed the upsampling process.
  • the current reconstructed image block is subjected to upsampling processing according to a certain rule.
  • the upsampling process is based on the case of 4 neighborhood pixels
  • the upsampling process is based on the case of 8 neighborhood pixels
  • the upsampling process is based on the case of 8 neighborhood pixels
  • the upsampling process is based on the case of 8 neighborhood pixels, once the lower right image block of the currently reconstructed image block is reconstructed, the current reconstructed image block may be subjected to upsampling processing.
  • the acquiring the encoding mode of the current image block to be encoded in step S2401 includes: determining a first encoding cost when the current image block to be encoded adopts the original resolution encoding mode; determining when the current image block to be encoded adopts the down sampling encoding mode The second coding cost; the coding mode corresponding to the smaller coding cost of the first coding cost and the second coding cost is used as the coding mode of the current image block to be coded.
  • the coding end may use a Rate Distortion Optimization (RDO) method to calculate the coding cost of the current coded image block.
  • RDO Rate Distortion Optimization
  • the RDO method is an optimized method to improve video compression performance.
  • the principle is to optimize the video's lossy (picture quality) and bit rate (the amount of data required for encoding) to achieve an optimal balance point.
  • this algorithm was originally used in video compression encoders, it can also be used for various multimedia encodings including video, video, audio, etc., as long as the encoding takes into account both quality and file size.
  • the coding mode adopted by the image block to be coded is a coding mode with a small coding cost, thereby reducing the coding complexity of the coding end, thereby improving the coding efficiency of the coding end.
  • FIG. 25 is a schematic structural diagram of an image processing device according to an embodiment of the present disclosure.
  • the device includes: a parsing module 2501, configured to parse a code stream to obtain a current image block to be reconstructed of a current image. a coding mode, a residual signal of the current image block to be reconstructed, and a pixel in each reference reconstructed image block corresponding to the current image block to be reconstructed, wherein the coding mode is a primary resolution coding mode or a down sampling coding
  • the current image block to be reconstructed corresponds to M reference reconstructed image blocks, and M is a positive integer greater than or equal to 1.
  • the determining module 2502 is configured to use the encoding mode of the current image block to be reconstructed and the M reference reconstructed image blocks. a pixel that determines a plurality of reference pixels of the image block to be reconstructed; a generating module 2503, configured to generate a prediction signal of the current image block to be reconstructed according to the plurality of reference pixels; and a reconstruction module 2504, configured to generate the signal according to the prediction signal and the residual signal The reconstructed signal of the image block to be reconstructed is reconstructed, and the current image block to be reconstructed is reconstructed according to the reconstructed signal to obtain a current reconstructed image block.
  • the method further includes: a processing module 2505.
  • the processing module 2505 is configured to: if the encoding mode of the currently reconstructed image block is the down sampling encoding mode, the pixels of the adjacent reconstructed image block required for the upsampling process based on the current reconstructed image block, and the current reconstructed image block Perform upsampling processing.
  • the processing module 2505 is specifically configured to: perform the upsampling process after all image blocks of the current image are reconstructed.
  • the code stream includes: a coding mode of each of the M reference reconstructed image blocks; the determining module 2502 is specifically configured to: according to the coding mode of the current image block to be reconstructed, Determining a plurality of reference pixels of the current image block to be reconstructed by encoding the M reference reconstructed image blocks and pixels in the M reference reconstructed image blocks.
  • the processing module 2505 is specifically configured to perform the upsampling process after all the reconstructed image blocks required for performing the upsampling process of the current reconstructed image block are reconstructed.
  • the code stream includes: a coding mode of each of the M reference reconstructed image blocks; the determining module 2502 is specifically configured to: according to the coding mode of the current image block to be reconstructed, Determining a plurality of reference pixels of the current image block to be reconstructed by encoding the M reference reconstructed image blocks and pixels in the M reference reconstructed image blocks.
  • the determining module 2502 is specifically configured to: if the encoding mode of the image block to be reconstructed is the original resolution encoding mode, and the M reference reconstructed image blocks include the reference reconstructed image block whose encoding mode is the original resolution encoding mode, Determining a reference pixel of the current image block to be reconstructed in a pixel of the reference reconstructed image block whose encoding mode is the original resolution encoding mode; if the current encoding mode of the image block to be reconstructed is the original resolution encoding mode, and M reference reconstructed image blocks Include a reference reconstructed image block whose coding mode is a down-sampling coding mode, and obtain a pixel required for reconstructing the current image block to be reconstructed from the pixels of the reference reconstructed image block whose coding mode is the down-sampling coding mode, and wait for the current pixel
  • the pixels required for reconstructing the image block for performing reconstruction are subjected to upsampling processing to obtain reference
  • the pixels required for reconstructing the current image block to be reconstructed are acquired from the pixels of the reference reconstructed image block whose encoding mode is the original resolution encoding mode, and are required for reconstructing the current image block to be reconstructed.
  • the pixels are downsampled to obtain reference pixels of the current image block to be reconstructed.
  • the processing module 2505 is specifically configured to: perform an upsampling process on the current reconstructed image block according to a pixel of a part of the adjacent reconstructed image block that is currently reconstructed in the required adjacent reconstructed image block; If another part of the adjacent reconstructed image block of the current incomplete reconstruction in the required adjacent reconstructed image block has been reconstructed, the partial boundary of the current reconstructed image block is secondarily upsampled according to another partial adjacent reconstructed image block. Processing, wherein a partial boundary of the currently reconstructed image block is contiguous with the another portion of the adjacent reconstructed image block.
  • the processing module 2505 is specifically configured to: perform a once-sampling process on the currently reconstructed image block according to a pixel of a part of the adjacent reconstructed image block that is currently reconstructed in the required adjacent reconstructed image block; After all the image blocks have been reconstructed, the partial boundary of the current reconstructed image block is subjected to secondary upsampling according to another partially adjacent reconstructed image block in the desired adjacent reconstructed image block, wherein another part is adjacent
  • the reconstructed image block is an image block that is not reconstructed when the first upsampling process is performed on the currently reconstructed image block; a partial boundary of the current reconstructed image block is adjacent to another partially adjacent reconstructed image block.
  • the determining module 2502 is specifically configured to: if the encoding mode of the image block to be reconstructed is the original resolution encoding mode, determine the reference pixels of the current image block to be reconstructed in the pixels of the M reference reconstructed image blocks; If the encoding mode of the image block to be reconstructed is the down sampling encoding mode, acquiring pixels required for reconstructing the current image block to be reconstructed in the pixels of the M reference reconstructed image blocks, and The pixels required for reconstruction of the image block to be reconstructed are subjected to downsampling processing to obtain reference pixels of the current image block to be reconstructed.
  • the part of the adjacent reconstructed image block is an upper image block and a left image block of the current reconstructed image block
  • the another partial adjacent reconstructed image block is a lower image block and a right side of the current reconstructed image block.
  • the part of the adjacent reconstructed image block is an upper left image block, an upper image block, an upper right image block, and a left image block of the current reconstructed image block
  • the another partial adjacent reconstructed image block is the current reconstructed image block.
  • a partial boundary of the current reconstructed image block is a right boundary and a lower boundary of the current reconstructed image block.
  • the required adjacent reconstructed image block includes an upper image block, a lower image block, a left image block, and a right image block of the current reconstructed image block; or the desired adjacent reconstructed image block includes the current reconstructed image.
  • the processing module 2505 is further configured to: after performing the upsampling process on the current reconstructed image block, identify that the current reconstructed image block has completed the upsampling process.
  • the image processing device provided by the present application can perform the image processing method corresponding to FIG. 13 and the optional manner of the method, and the implementation principle and technical effects are similar, and details are not described herein again.
  • FIG. 26 is a schematic structural diagram of an image processing device according to another embodiment of the present disclosure. As shown in FIG. 26, the device includes:
  • the obtaining module 2601 is configured to acquire an encoding manner of a current image block to be encoded of a current image, and a pixel in each reference reconstructed image block corresponding to the current image block to be encoded, where the encoding mode is a primary resolution encoding manner. Or a downsampling coding mode, where the current image block to be encoded corresponds to M reference reconstructed image blocks, and M is a positive integer greater than or equal to 1.
  • the determining module 2602 is configured to determine, according to the encoding manner of the current image block to be encoded and the pixels in the M reference reconstructed image blocks, a plurality of reference pixels of the current image block to be encoded.
  • the generating module 2603 is configured to generate, according to the plurality of reference pixels, a prediction signal of the current image block to be encoded.
  • the acquiring module 2601 is further configured to acquire an encoded signal of the current image block to be encoded, where the encoded signal is a device when the encoding mode of the current image block to be encoded is the original resolution encoding mode.
  • the original signal of the current image block to be encoded when the encoding mode of the current image block to be encoded is the down sampling encoding mode, the encoded signal is after the original signal of the current image block to be encoded is subjected to down sampling processing. The signal obtained.
  • the generating module 2603 is further configured to generate a residual signal of the current image block to be encoded according to the prediction signal and the encoded signal.
  • the encoding module 2604 is configured to encode the residual signal.
  • a processing module 2605 is also included.
  • the generating module 2603 is further configured to: generate a reconstructed signal of the current image block to be encoded, and reconstruct the current image block to be encoded according to the reconstructed signal to obtain a current reconstructed image block; and the processing module 2605 is configured to perform current reconstruction.
  • the coding mode of the image block is the downsampling coding mode, and the current reconstructed image block is subjected to upsampling processing based on the pixels of the adjacent reconstructed image block required for performing the upsampling process on the current reconstructed image block.
  • the processing module 2605 is specifically configured to: after the image blocks of the current image are all reconstructed, perform the upsampling process; correspondingly, the determining module 2602 is specifically configured to: acquire the M Refers to an encoding manner of each reference reconstructed image block in the reconstructed image block; according to the encoding manner of the current image block to be encoded, the encoding manner of the M reference reconstructed image blocks, and the M reference reconstructed image blocks a pixel, determining a plurality of reference pixels of the current image block to be encoded.
  • the processing module 2605 is specifically configured to perform the upsampling process after all the reconstructed image blocks required for performing the upsampling process on the current reconstructed image block are reconstructed.
  • the determining module 2602 is specifically configured to: if the encoding mode of the current image block to be encoded is the original resolution encoding mode, and the M reference reconstructed image blocks include the reference reconstructed image block whose encoding mode is the original resolution encoding mode, Determining a reference pixel of the current image block to be encoded in a pixel of the reference reconstructed image block whose encoding mode is the original resolution encoding mode; if the current encoding mode of the image block to be encoded is the original resolution encoding mode, and M reference reconstructed image blocks Include a reference reconstructed image block whose coding mode is a down-sampling coding mode, and obtain, from a pixel of a reference reconstructed image block whose coding mode is a down-sampling coding mode, a pixel required for reconstructing the current reconstructed image block, and the current to be encoded
  • the pixels required for reconstruction of the image block are subjected to upsampling processing to obtain reference pixels
  • the processing module 2605 is specifically configured to: perform an upsampling process on the currently reconstructed image block according to a pixel of a part of the adjacent reconstructed image block that is currently reconstructed in the required adjacent reconstructed image block; If another part of the adjacent reconstructed image block of the current incomplete reconstruction in the required adjacent reconstructed image block has been reconstructed, the partial boundary of the current reconstructed image block is subjected to the second upsampling process according to another partial adjacent reconstructed image block. Wherein, a partial boundary of the currently reconstructed image block is adjacent to another partially adjacent reconstructed image block.
  • the processing module 2605 is specifically configured to: perform an upsampling process on the currently reconstructed image block according to a pixel of a part of the adjacent reconstructed image block that is currently reconstructed in the required adjacent reconstructed image block; If all the image blocks of the current image have been reconstructed, then the partial boundary of the current reconstructed image block is subjected to secondary upsampling according to another partially adjacent reconstructed image block in the desired adjacent reconstructed image block, wherein the other portion
  • the adjacent reconstructed image block is an image block that is not reconstructed when the first upsampling process is performed on the current reconstructed image block; a partial boundary of the current reconstructed image block is adjacent to another partially adjacent reconstructed image block.
  • the determining module 2502 is specifically configured to: if the encoding mode of the current image block to be encoded is the original resolution encoding mode, determine the reference pixels of the current image block to be encoded in the pixels of the M reference reconstructed image blocks; If the encoding mode of the current image block to be encoded is the down sampling coding mode, acquiring pixels required for reconstructing the current image block to be encoded in the pixels of the M reference reconstructed image blocks, and The pixels required for reconstructing the current image block to be encoded are subjected to down sampling processing to obtain reference pixels of the current image block to be encoded.
  • the part of the adjacent reconstructed image block is an upper image block and a left image block of the current reconstructed image block
  • the another partial adjacent reconstructed image block is a lower image block and a right side of the current reconstructed image block.
  • the part of the adjacent reconstructed image block is an upper left image block, an upper image block, an upper right image block and a left image block of the current reconstructed image block, and the other partially adjacent reconstructed image block is a right side of the current reconstructed image block.
  • a partial boundary of the current reconstructed image block is a right boundary and a lower boundary of the current reconstructed image block.
  • the required adjacent reconstructed image block includes an upper image block, a lower image block, a left image block, and a right image block of the current reconstructed image block; or the desired adjacent reconstructed image block includes the current
  • the upper image block, the lower image block, the left image block, the right image block, the upper left image block, the lower left image block, the upper right image block, and the lower right image block of the image block are reconstructed.
  • the processing module 2605 is further configured to: after performing the upsampling process on the current reconstructed image block, identify that the current reconstructed image block has completed the upsampling process.
  • the acquiring module 2601 is specifically configured to: determine a first encoding cost when the current image block to be encoded adopts the original resolution encoding mode; and determine a second encoding cost when the current image block to be encoded adopts the down sampling encoding mode; The coding mode corresponding to the smaller coding cost of the first coding cost and the second coding cost is used as the coding mode of the current image block to be encoded.
  • the image processing device provided by the present application can perform the image processing method corresponding to FIG. 24 and the optional manner of the method, and the implementation principle and technical effects are similar, and details are not described herein again.
  • FIG. 27 is a schematic structural diagram of an image processing system provided by the present application. As shown in FIG. 27, the system includes: an image processing device 2701 of the above decoding end, and an image processing device 2702 of the encoding end.
  • the present application provides an image processing apparatus including: a processor and a memory for storing executable instructions of the processor; wherein the processor can perform the image processing method corresponding to FIG. 13 and an optional manner of the method .
  • the implementation principle and technical effect are similar, and will not be described here.
  • the present application provides an image processing apparatus including: a processor and a memory for storing executable instructions of the processor; wherein the processor can execute the image processing method corresponding to FIG. 24 and an optional manner of the method .
  • the implementation principle and technical effect are similar, and will not be described here.
  • the image processing device of the decoding end of the image processing system provided by the present application may perform the image processing method corresponding to FIG. 13 and an optional manner of the method, and the image processing device of the encoding end may perform the image processing method corresponding to FIG. 24 described above and The optional method and the technical effect of the method are similar, and are not described here.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

La présente invention concerne un procédé, un dispositif et un système de traitement d'image. Le procédé consiste à : analyser un flux de code pour acquérir un mode d'encodage d'un bloc d'image actuel à reconstruire d'une image actuelle, un signal résiduel du bloc d'image actuel à reconstruire, et un pixel de chaque bloc d'image de reconstruction de référence correspondant au bloc d'image actuel à reconstruire, le mode d'encodage étant un mode d'encodage de résolution d'origine ou un mode d'encodage de sous-échantillonnage, et le bloc d'image actuel à reconstruire correspondant à M blocs d'image de reconstruction de référence ; déterminer de multiples pixels de référence du bloc d'image actuel à reconstruire selon le mode d'encodage du bloc d'image actuel à reconstruire et des pixels des M blocs d'image de reconstruction de référence ; produire un signal de prédiction selon les multiples pixels de référence ; et produire un signal de reconstruction selon le signal de prédiction et le signal résiduel, et reconstruire le bloc d'image actuel à reconstruire selon le signal de reconstruction pour obtenir un bloc d'image reconstruit actuel. Par conséquent, le bloc d'image reconstruit obtenu par un terminal de décodage fonctionne mieux.
PCT/CN2018/081678 2017-07-13 2018-04-03 Procédé, dispositif et système de traitement d'image WO2019011011A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710572139.7 2017-07-13
CN201710572139.7A CN109257608B (zh) 2017-07-13 2017-07-13 图像处理方法、设备及系统

Publications (1)

Publication Number Publication Date
WO2019011011A1 true WO2019011011A1 (fr) 2019-01-17

Family

ID=65000880

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/081678 WO2019011011A1 (fr) 2017-07-13 2018-04-03 Procédé, dispositif et système de traitement d'image

Country Status (3)

Country Link
CN (1) CN109257608B (fr)
TW (1) TWI672941B (fr)
WO (1) WO2019011011A1 (fr)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111641831B (zh) * 2019-03-01 2023-09-01 杭州海康威视数字技术股份有限公司 帧内预测方法、装置、设备及存储介质
CN112235568B (zh) * 2019-07-15 2024-03-26 华为技术有限公司 图像编码方法、解码方法、装置和存储介质
CN111586410B (zh) * 2020-06-02 2022-04-19 浙江大华技术股份有限公司 视频编码方法、解码方法及其相关装置
CN113329228B (zh) * 2021-05-27 2024-04-26 杭州网易智企科技有限公司 视频编码方法、解码方法、装置、电子设备及存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105430417A (zh) * 2014-09-22 2016-03-23 中兴通讯股份有限公司 编码方法、解码方法、装置及电子设备
CN105556963A (zh) * 2013-10-14 2016-05-04 联发科技（新加坡）私人有限公司 用于hevc范围扩展的残差差分脉冲编码调制方法
US20170064323A1 (en) * 2011-10-26 2017-03-02 Intellectual Discovery Co., Ltd. Scalable video coding method and apparatus using intra prediction mode
WO2017069419A1 (fr) * 2015-10-22 2017-04-27 엘지전자 주식회사 Procédé et appareil de prédiction intra dans un système de codage vidéo

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104170379B (zh) * 2011-10-24 2016-01-20 英孚布瑞智有限私人贸易公司 用于图像解码的方法和装置
US9998727B2 (en) * 2012-09-19 2018-06-12 Qualcomm Incorporated Advanced inter-view residual prediction in multiview or 3-dimensional video coding

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170064323A1 (en) * 2011-10-26 2017-03-02 Intellectual Discovery Co., Ltd. Scalable video coding method and apparatus using intra prediction mode
CN105556963A (zh) * 2013-10-14 2016-05-04 联发科技（新加坡）私人有限公司 用于hevc范围扩展的残差差分脉冲编码调制方法
CN105430417A (zh) * 2014-09-22 2016-03-23 中兴通讯股份有限公司 编码方法、解码方法、装置及电子设备
WO2017069419A1 (fr) * 2015-10-22 2017-04-27 엘지전자 주식회사 Procédé et appareil de prédiction intra dans un système de codage vidéo

Also Published As

Publication number Publication date
CN109257608B (zh) 2021-11-30
CN109257608A (zh) 2019-01-22
TW201909635A (zh) 2019-03-01
TWI672941B (zh) 2019-09-21

Similar Documents

Publication Publication Date Title
TWI681670B (zh) 影像處理方法、設備及系統
CN108028941B (zh) 用于通过超像素编码和解码数字图像的方法和装置
JP2022008296A (ja) 省略符号化を用いた映像符号化及び復号化装置及びその方法
TWI672941B (zh) 影像處理方法、設備及系統
KR20200044653A (ko) 딥 뉴럴 네트워크를 이용한 영상의 ａｉ 부호화 및 ａｉ 복호화 방법, 및 장치
TWI681672B (zh) 圖像處理方法、設備及系統
TWI575932B (zh) 混成視訊解碼器、混成視訊編碼器及資料串流
US20130272391A1 (en) Method and Apparatus of Quantization Matrix Coding
WO2018158735A1 (fr) Procédé et appareil de codage et de décodage d'images numériques ou de flux vidéo
KR20200050284A (ko) 영상 적응적 양자화 테이블을 이용한 영상의 부호화 장치 및 방법
US8249372B2 (en) Methods and devices for coding and decoding multidimensional digital signals
KR20230107627A (ko) 후처리 제어를 이용한 비디오 디코딩
US20240040160A1 (en) Video encoding using pre-processing
JP2022503990A (ja) ビデオ画像成分予測方法および装置、コンピュータ記憶媒体
US20240137513A1 (en) Application of Super Resolution
US20240236325A9 (en) Super resolution downsampling
US20240137517A1 (en) Super Resolution Position and Network Structure
WO2024078635A1 (fr) Procédés et rapports de sous-échantillonnge pour codage vidéo basé sur la super-résolution
US20240137577A1 (en) Super Resolution Upsampling and Downsampling
CN118433404A (zh) 视频解码方法、视频编码方法及译码设备
WAVELET AN EFFICIENT IMAGE DOWNSAMPLING TECHNIQUE USING GENETIC ALGORITHM AND DISCRETE WAVELET TRANSFORM

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18832344

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18832344

Country of ref document: EP

Kind code of ref document: A1