WO2021169408A1

WO2021169408A1 - Image processing method and apparatus, and electronic device and storage medium

Info

Publication number: WO2021169408A1
Application number: PCT/CN2020/128207
Authority: WO
Inventors: 王晶; 白博; 葛运英
Original assignee: 华为技术有限公司
Priority date: 2020-02-26
Filing date: 2020-11-11
Publication date: 2021-09-02
Also published as: CN113313774A

Abstract

The embodiments of the present application relate to the technical field of image processing in the technical field of computer vision in the field of artificial intelligence. Provided are an image processing method and apparatus, and an electronic device and a storage medium. A pre-processing link including a plurality of preset strategies is provided, wherein during encoding, a target strategy is selected from among the plurality of preset strategies to pre-process an original image, and then, an obtained image to be compressed is compressed; if a different target strategy is selected, the bit rate of obtained compressed data is also different, thereby achieving the aim of one compression model corresponding to a plurality of bit rates; and compression is carried out by using a deep learning image compression framework, and the compression performance is improved. In addition, a reverse processing link of pre-processing is provided, wherein during decoding, compressed data is decompressed first, and then, reverse processing of the pre-processing is carried out on an obtained restored image by using a reverse strategy of the target strategy, such that the visual quality of a reconstructed image is basically unchanged. Therefore, by means of the embodiments of the present application, one compression model corresponding to a plurality of bit rates can be realized while a compression effect is ensured.

Description

Image processing method, device, electronic equipment and storage medium

This application claims the priority of a Chinese patent application filed with the State Intellectual Property Office on February 26, 2020, the application number is 202010120792.1, and the application name is "Image processing methods, devices, electronic equipment and storage media", the entire content of which is incorporated by reference Incorporated in this application.

Technical field

The embodiments of the present application relate to the field of image processing technology in the field of computer vision technology in the field of artificial intelligence, and specifically, to an image processing method, device, electronic device, and storage medium.

Background technique

Image compression is extremely important for data storage and transmission. Uncompressed images will take up a huge amount of storage space and at the same time will bring huge pressure on transmission. The reason why the image can be compressed is because there is redundant information in the image. The redundant information mainly includes: spatial redundancy caused by the correlation between adjacent pixels in the image, and spectrum caused by the correlation between different color planes or spectrum bands. Redundancy, etc. The purpose of image compression is to reduce the number of bits required to represent the image by removing these redundant information.

In practical applications, in addition to the requirements for the compression effect and the need to ensure that the visual quality of the picture is basically unchanged, the bit rate required by different applications may also be different. The bit rate refers to the amount of compressed data of the picture displayed per second, so , How to achieve a compression model corresponding to multiple code rates while ensuring the compression effect is a problem that researchers need to solve.

Summary of the invention

The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device, and storage medium to solve the problem of how to achieve a compression model corresponding to multiple code rates while ensuring the compression effect.

In a first aspect, an embodiment of the present application provides an image processing method. The image processing method includes: acquiring an original image; in response to an operation on the original image, determining a target strategy from a plurality of preset strategies, of which at least two The code rate of the compressed data corresponding to the preset strategy is different; according to the target strategy, the original image is preprocessed to obtain the image to be compressed; the preset deep learning image compression framework is used to perform the compression on the image to be compressed. Compressed to obtain the compressed data, wherein the compressed data is used to obtain a restored image by decompressing through a preset deep learning image decompression framework, and the restored image is used to perform a restoration based on the reverse strategy of the target strategy The reverse processing of the preprocessing obtains a reconstructed image corresponding to the original image.

The image processing method provided by the embodiment of this application first sets up a pre-processing link, including multiple preset strategies. When compressing, select a target strategy from the multiple preset strategies to preprocess the original image, and then perform pre-processing on the obtained image to be compressed. For image compression, the selected target strategy is different, and the code rate of the compressed data will be different, so as to achieve the purpose of one compression model corresponding to multiple code rates; secondly, the deep learning image compression framework is used for compression to improve the compression performance; At the same time, when the compressed data is used for decompression, first use the deep learning image decompression framework to decompress the compressed data, and then use the reverse strategy of the target strategy to do the reverse processing of the preprocessing of the obtained restored image, so that the reconstructed image The visual quality is basically unchanged. Therefore, the embodiments of the present application can realize that one compression model corresponds to multiple code rates while ensuring the compression effect.

Optionally, the target strategy includes a first instruction and a first parameter corresponding to the first instruction; the step of preprocessing the original image according to the target strategy to obtain the image to be compressed includes : Preprocess the original image according to the first instruction and the first parameter to obtain the image to be compressed.

In the embodiment of the present application, different first instructions and first parameters are used to preprocess the original image, and the corresponding compressed data has different code rates.

Optionally, the first instruction includes a global zoom instruction, and the first parameter includes a global zoom factor and a zoom kernel; and the original image is preprocessed according to the first instruction and the first parameter , The step of obtaining the image to be compressed includes: performing global scaling on the original image in accordance with the global scaling instruction, global scaling factor, and scaling kernel to obtain the image to be compressed.

In the embodiment of the present application, when the first command is a global zoom command, different global zoom coefficients are used to perform global zoom on the original image, and the code rates of the corresponding compressed data are different. The smaller the global scaling factor, the smaller the bit rate of compressed data.

Optionally, the first instruction includes an adaptive scaling instruction, and the first parameter includes a block parameter; and the original image is preprocessed according to the first instruction and the first parameter to obtain The step of the image to be compressed includes: dividing the original image according to the block parameters to obtain a plurality of image blocks; according to the adaptive scaling instruction and the image characteristics of each image block, each Each of the image blocks is adaptively scaled to obtain the image block to be compressed corresponding to each of the image blocks, wherein the image to be compressed includes a plurality of image blocks to be compressed, and the image feature is used to determine the image The zoom factor of the block.

In the embodiment of the present application, the adaptive zoom instruction refers to a method of adaptively zooming the original image. Adaptive scaling of the original image refers to the image feature of the original image (for example, color feature, texture feature, shape feature, etc.), the area with different image features is scaled to different degrees, for example, the background area is more zoomed. The foreground area is less zoomed.

Performing adaptive scaling for each image block refers to determining the scaling factor of each image block according to the image characteristics corresponding to each image block, and then performing block reduction or block enlargement according to the respective scaling factors. The image block with more image features has a larger scaling factor; the image block with fewer image features has a smaller scaling factor. That is, smooth image blocks are scaled more, and unsmooth image blocks are scaled less.

When the first instruction is an adaptive scaling instruction, in order to ensure compression performance, an image block with more image features has a larger scaling factor, and an image block with fewer image features has a smaller scaling factor. By adjusting the block parameters, one compression model can correspond to multiple code rates.

Optionally, the first instruction includes a blur processing instruction, and the first parameter includes a blur kernel; and the original image is preprocessed according to the first instruction and the first parameter to obtain the The step of the image to be compressed includes: performing blur processing on the original image according to the blur processing instruction and the blur kernel to obtain the image to be compressed.

In the embodiment of the present application, when the first instruction is a fuzzy processing instruction, by adjusting the fuzzy kernel, compression of different code rates can be realized. The larger the fuzzy kernel scale, the smaller the bit rate of compressed data.

Optionally, the first instruction includes an image degradation instruction, and the first parameter includes an image degradation parameter; and the original image is preprocessed according to the first instruction and the first parameter to obtain the The step of the image to be compressed includes: performing image degradation on the original image according to the image degradation instruction and the image degradation parameter to obtain the image to be compressed.

In the embodiment of the present application, when the first instruction is an image degradation instruction, by adjusting the image degradation parameter, compression at different code rates can be realized. The larger the image degradation parameter, the smaller the bit rate of the compressed data.

Optionally, the first instruction includes an image separation instruction and a first post-processing instruction, and the first parameter includes an image separation parameter corresponding to the image separation instruction and an image separation parameter corresponding to the first post-processing instruction The first post-stage processing parameter; the step of preprocessing the original image according to the first instruction and the first parameter to obtain the image to be compressed includes: according to the image separation instruction and the The image separation parameter is used to perform image separation on the original image to obtain an edge image and a texture image; according to the first post-processing instruction and the first post-processing parameter, the edge image and the texture image are At least one of global scaling, adaptive scaling, blur processing, and image degradation is performed to obtain an edge image to be compressed and a texture image to be compressed, wherein the image to be compressed includes the edge image to be compressed and the edge image to be compressed Texture image.

In the embodiment of the present application, when the first instruction is an image separation instruction and a first post-processing instruction, by adjusting the image separation parameter and the first post-processing parameter, compression at different bit rates can be realized.

Optionally, the first instruction includes an image segmentation instruction and a second post-processing instruction, and the first parameter includes a segmentation category corresponding to the image segmentation instruction and a second post-processing instruction corresponding to the second post-processing instruction. Two post-stage processing parameters; the step of preprocessing the original image according to the first instruction and the first parameter to obtain the image to be compressed includes: following the image segmentation instruction and the Segmentation category, image segmentation is performed on the original image to obtain multiple image regions; according to the second post-processing instruction and the second post-processing parameters, the multiple image regions are globally zoomed and automatically At least one of scaling, blur processing, and image degradation is adapted to obtain a to-be-compressed image area corresponding to each of the image areas, where the to-be-compressed image includes a plurality of to-be-compressed image areas.

In the embodiment of the present application, when the first instruction is an image segmentation instruction and a second post-stage processing instruction, by adjusting the segmentation category and the second post-stage processing parameter, compression at different code rates can be realized.

Optionally, the deep learning image compression framework includes a first deep neural network, a quantization model, and an entropy coding model; the step of compressing the image to be compressed using a preset deep learning image compression framework to obtain compressed data , Including: using the first deep neural network to perform feature extraction on the image to be compressed to obtain image features; using the quantization model to quantize the image features to obtain compressed features; using the entropy coding model to Entropy coding is performed on the compressed feature to obtain the compressed data.

In a second aspect, an embodiment of the present application further provides an image processing method, the image processing method includes: obtaining compressed data, wherein the compressed data is obtained by compressing an image to be compressed using a preset deep learning image compression framework, The image to be compressed is obtained by preprocessing the original image according to a target strategy, and the target strategy is determined from a plurality of preset strategies in response to an operation on the original image, and at least two of the preset strategies correspond to The code rate of the compressed data is different; the compressed data is decompressed using a preset deep learning image decompression framework to obtain a restored image; the reverse strategy corresponding to the target strategy is obtained; according to the reverse strategy Performing the reverse processing of the pre-processing on the restored image to obtain a reconstructed image corresponding to the original image.

Optionally, the step of obtaining a reverse strategy corresponding to the target strategy includes: obtaining the target strategy, the target strategy including a first instruction and a first parameter corresponding to the first instruction; The corresponding relationship between the first instruction and the preset instruction determines the second instruction; the second parameter is determined according to the first parameter and the preset parameter calculation rule, wherein the reverse strategy includes the second instruction and the The second parameter corresponding to the second instruction.

Optionally, the step of performing reverse processing of the preprocessing on the restored image according to the reverse strategy to obtain a reconstructed image corresponding to the original image includes: following the second instruction and the The second parameter is to perform reverse processing of the pre-processing on the restored image to obtain a reconstructed image corresponding to the original image.

Optionally, the first instruction includes a global zoom instruction, the first parameter includes a global zoom factor and a zoom core; the second instruction includes a global zoom instruction, and the second parameter includes the reciprocal of the global zoom factor and zoom Core; the step of performing the pre-processing reverse processing on the restored image according to the second instruction and the second parameter to obtain a reconstructed image corresponding to the original image includes: according to the The global zoom instruction, the reciprocal of the global zoom coefficient, and the zoom kernel perform global zoom on the restored image to obtain the reconstructed image.

Optionally, the first instruction includes an adaptive scaling instruction, the first parameter includes a block parameter; the second instruction includes an adaptive scaling instruction, and the second parameter includes a splicing associated with the block parameter. Parameters; the restored image includes a plurality of restored image blocks; the restored image is subjected to the reverse processing of the pre-processing in accordance with the second instruction and the second parameter to obtain a corresponding to the original image The step of reconstructing the image includes: performing adaptive scaling on each restored image block according to the adaptive scaling instruction and the image characteristics of each restored image block, to obtain a corresponding to each restored image block The image block to be reconstructed, wherein the image feature is used to determine the scaling factor of the restored image block; and a plurality of image blocks to be reconstructed are spliced according to the splicing parameter to obtain the reconstructed image.

Optionally, the first instruction includes a fuzzy processing instruction, and the first parameter includes a fuzzy core; the second instruction includes a deblurring processing instruction, and the second parameter includes a deblurring core corresponding to the fuzzy core; The step of performing the reverse processing of the pre-processing on the restored image according to the second instruction and the second parameter to obtain a reconstructed image corresponding to the original image includes: according to the deblurring The processing instruction and the deblurring kernel perform deblurring processing on the restored image to obtain the reconstructed image.

Optionally, the first instruction includes an image degradation instruction, the first parameter includes an image degradation parameter; the second instruction includes an image enhancement instruction, and the second parameter includes an image enhancement parameter; The second instruction and the second parameter, the step of performing the reverse processing of the pre-processing on the restored image to obtain a reconstructed image corresponding to the original image includes: following the image enhancement instruction and the image enhancement Parameter, performing the image enhancement on the restored image to obtain the reconstructed image.

Optionally, the first instruction includes an image separation instruction and a first post-processing instruction, and the first parameter includes an image separation parameter corresponding to the image separation instruction and an image separation parameter corresponding to the first post-processing instruction The first post-processing parameter; the second instruction includes the reverse instruction of the image separation instruction and the reverse instruction of the first post-processing instruction, and the second parameter includes the reverse instruction of the image separation instruction The reverse parameter of the image separation parameter corresponding to the direction instruction and the reverse parameter of the first post-processing parameter corresponding to the reverse instruction of the first post-processing instruction; the restored image includes a restored edge image And a restored texture image; the step of performing the reverse processing of the pre-processing on the restored image according to the second instruction and the second parameter to obtain a reconstructed image corresponding to the original image includes: According to the reverse instruction of the first post-stage processing instruction and the reverse parameter of the first post-stage processing parameter, both the restored edge image and the restored texture image are subjected to the reverse processing of global scaling and adaptive At least one of the reverse processing of scaling, deblurring processing, and image enhancement to obtain the edge image to be reconstructed and the texture image to be reconstructed; according to the reverse instruction of the image separation instruction and the reverse parameter of the image separation parameter, Image fusion is performed on the edge image to be reconstructed and the texture image to be reconstructed to obtain the reconstructed image.

Optionally, the first instruction includes an image segmentation instruction and a second post-processing instruction, and the first parameter includes a segmentation category corresponding to the image segmentation instruction and a second post-processing instruction corresponding to the second post-processing instruction. Two post-stage processing parameters; the second instruction includes a reverse instruction of the image segmentation instruction and a reverse instruction of the second post-stage processing instruction, and the second parameter includes the same as the second post-processing instruction The reverse instruction corresponding to the reverse parameter of the second post-stage processing parameter; the restored image includes a plurality of restored image areas and the position coordinates of each restored image area; the second instruction and The second parameter, the step of performing the reverse processing of the pre-processing on the restored image to obtain a reconstructed image corresponding to the original image includes: a reverse instruction according to the second post-processing instruction and The inverse parameter of the second post-stage processing parameter performs at least one of global scaling inverse processing, adaptive scaling inverse processing, deblurring processing, and image enhancement for each of the restored image regions, Obtain the to-be-reconstructed image area corresponding to each of the restored image areas; according to the reverse instruction of the image segmentation instruction and the position coordinates of each of the restored image areas, stitch the multiple to-be-reconstructed image areas to obtain the Reconstruct the image.

Optionally, the deep learning image decompression framework includes a second deep neural network, an inverse quantization model, and an entropy decoding model; the preset deep learning image decompression framework is used to decompress the compressed data to obtain restoration The image step includes: using the entropy decoding model to perform entropy decoding on the compressed data to obtain compressed features; using the inverse quantization model to dequantize the compressed features to obtain image features; and using the second depth The neural network restores the image features to obtain the restored image.

Optionally, the image processing method further includes: using at least one of a super-resolution algorithm, a deblurring algorithm, a dehazing algorithm, and a denoising algorithm to process the reconstructed image, so as to improve the vision of the reconstructed image. Effect.

In a third aspect, an embodiment of the present application also provides an image processing method. The image processing method includes: acquiring an original image; in response to an operation on the original image, determining a target strategy from a plurality of preset strategies, wherein at least two The code rates of the compressed data corresponding to each of the preset strategies are different; according to the target strategy, the original image is preprocessed to obtain the image to be compressed; the preset deep learning image compression framework is used for the image to be compressed Perform compression to obtain the compressed data; use a preset deep learning image decompression framework to decompress the compressed data to obtain a restored image; obtain the reverse strategy corresponding to the target strategy; The restored image is subjected to the reverse processing of the pre-processing to obtain a reconstructed image corresponding to the original image.

In a fourth aspect, an embodiment of the present application also provides an image processing device, the image processing device includes: an image acquisition module for acquiring an original image; a response module for responding to an operation on the original image, from multiple presets The target strategy is determined in the strategy, wherein at least two of the preset strategies have different code rates for the compressed data; the preprocessing module is used to preprocess the original image according to the preset strategy to obtain the image to be compressed Compression module, used to compress the image to be compressed using a preset deep learning image compression framework to obtain the compressed data, wherein the compressed data is used to decompress the preset deep learning image decompression framework A restored image is obtained by compression, and the restored image is used to perform reverse processing of the pre-processing based on the reverse strategy of the preset strategy to obtain a reconstructed image corresponding to the original image.

In a fifth aspect, an embodiment of the present application also provides an image processing device. The image processing device includes a sequence obtaining module for obtaining compressed data, wherein the compressed data is processed using a preset deep learning image compression framework. The compressed image is obtained by compression, the image to be compressed is obtained by preprocessing the original image according to a target strategy, and the target strategy is determined from a plurality of preset strategies in response to an operation on the original image, and at least two The code rate of the compressed data corresponding to the preset strategy is different; a decompression module for decompressing the compressed data using a preset deep learning image decompression framework to obtain a restored image; a reverse strategy obtaining module , Used to obtain the reverse strategy corresponding to the target strategy; a post-processing module, used to perform the pre-processing reverse processing on the restored image according to the reverse strategy to obtain a reconstruction corresponding to the original image image.

In a sixth aspect, an embodiment of the present application also provides an image processing device, the image processing device includes: an image acquisition module for acquiring an original image; a response module for responding to an operation on the original image, from multiple presets The target strategy is determined in the strategy, wherein at least two of the preset strategies have different code rates for the compressed data; the preprocessing module is used to preprocess the original image according to the preset strategy to obtain the image to be compressed Compression module, used to compress the image to be compressed using a preset deep learning image compression framework to obtain the compressed data; decompression module, use a preset deep learning image decompression framework to perform compression on the compressed data Decompress to obtain a restored image; a reverse strategy obtaining module for obtaining a reverse strategy corresponding to the target strategy; a post-processing module for performing the reverse of the preprocessing on the restored image according to the reverse strategy To obtain a reconstructed image corresponding to the original image.

In a seventh aspect, an embodiment of the present application also provides an electronic device, the electronic device includes: one or more processors; a memory, used to store one or more programs, when the one or more programs are When executed by one or more processors, the one or more processors implement the image processing method of the first aspect or the second aspect or the third aspect.

In an eighth aspect, an embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the image processing method of the first aspect or the second aspect or the third aspect is implemented.

In a ninth aspect, the embodiments of the present application also provide a computer program product, including instructions, which when run on a computer, cause the computer to execute the image processing method of the first aspect or the second aspect or the third aspect.

In a tenth aspect, an embodiment of the present application further provides a chip system. The chip system includes a processor and may also include a memory for implementing the image processing method of the first aspect or the second aspect or the third aspect. The chip system can be composed of chips, or it can include chips and other discrete devices.

For the beneficial effects of the above-mentioned second aspect to the tenth aspect and the implementation manners thereof, reference may be made to the description of the beneficial effects of the method and implementation manners of the first aspect.

Description of the drawings

Fig. 1 is a schematic diagram of a JPEG image compression framework provided by the prior art.

FIG. 2 is a schematic diagram of an image compression framework based on Auto-encoder provided by the prior art.

Fig. 3 is a schematic diagram of an image compression framework based on RNN provided by the prior art.

FIG. 4 is a schematic diagram of an overall flow of an image processing method provided by an embodiment of the application.

FIG. 5 is a schematic flowchart of an image processing method provided by an embodiment of the application.

FIG. 6 is a schematic flowchart of step S103 in the image processing method provided in FIG. 5.

FIG. 7 is a schematic diagram of another overall flow of an image processing method provided by an embodiment of the application.

FIG. 8 is an example diagram of a compression curve corresponding to the image processing method provided by an embodiment of the application.

FIG. 9 is a schematic diagram of another overall flow of an image processing method provided by an embodiment of the application.

FIG. 10 is a schematic diagram of another overall flow of an image processing method provided by an embodiment of the application.

FIG. 11 is a schematic diagram of another overall flow of an image processing method provided by an embodiment of the application.

FIG. 12 is a schematic diagram of another overall flow of an image processing method provided by an embodiment of the application.

FIG. 13 is a schematic diagram of another overall flow of an image processing method provided by an embodiment of the application.

FIG. 14 is a schematic flowchart of step S104 in the image processing method provided in FIG. 5.

FIG. 15 is a schematic flowchart of another image processing method provided by an embodiment of the application.

FIG. 16 is a schematic flowchart of step S202 in the image processing method provided in FIG. 15.

FIG. 17 is a schematic flowchart of step S203 in the image processing method provided in FIG. 15.

FIG. 18 is a schematic flowchart of step S204 in the image processing method provided in FIG. 15.

FIG. 19 is a schematic flowchart of another image processing method provided by an embodiment of the application.

FIG. 20 is a schematic flowchart of another image processing method provided by an embodiment of the application.

FIG. 21 is a schematic diagram of an application of the image processing method provided by an embodiment of the application.

FIG. 22 is a schematic diagram of another application of the image processing method provided by an embodiment of the application.

FIG. 23 is a schematic diagram of another application of the image processing method provided by an embodiment of the application.

FIG. 24 is a schematic diagram of a composition of an image processing device provided by an embodiment of the application.

FIG. 25 is a schematic diagram of another composition of an image processing apparatus provided by an embodiment of the application.

FIG. 26 is a schematic diagram of another composition of an image processing apparatus provided by an embodiment of the application.

FIG. 27 is a schematic diagram of the composition of an electronic device provided by an embodiment of the application.

Detailed ways

In order to make the above objectives, features and advantages of the present application more obvious and understandable, specific embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

Image compression is mainly divided into two categories: lossy compression and lossless compression. Lossless compression is mainly used in scenes that require very precise image details, such as authentication signature image processing, archive image processing, and part of medical image processing. Lossy compression utilizes the human eye’s insensitivity to high-frequency signals, and coarsely quantizes high-frequency components in transform coding. At the same time, the surrounding pixel values can be used to predict the current pixel value, which greatly reduces the amount of data that needs to be encoded. The image compression described is all lossy compression.

Traditional image compression methods include JPEG, JPEG2000, BPG, etc. The following takes JPEG compression as an example to introduce the traditional image compression methods.

Please refer to FIG. 1. FIG. 1 shows a schematic diagram of a JPEG image compression framework. The compression framework includes two parts: an encoding process and a decoding process. Among them, the encoding process includes: First, the original image (for example, RGB three-channel image) undergoes Discrete Cosine Transform (DCT) to transform the image features into the frequency domain space, so that the low-frequency information in the image that has a significant impact on the image quality Separate it from high-frequency information to reduce data redundancy; then, through quantization to remove high-frequency information that has less impact on image quality, reducing storage space; and then Huffman encoding the quantized integer to obtain an encoded JPEG stream. The decoding process is opposite to the encoding process, including: the encoded JPEG code stream is entropy-decoded and dequantized to obtain a floating-point number, and then the floating-point number is transformed from the frequency domain space to the pixel space through inverse discrete cosine transform to obtain a reconstructed image.

At present, traditional image compression methods have been widely used. However, since part of the spatial and frequency domain information of the image is removed during encoding, the visual quality of the reconstructed image is poor; at the same time, these methods are all aimed at certain types of images. Features are manually designed and cannot adapt to the emerging new media types, such as virtual reality images, panoramic images, and square images. Therefore, how to further improve compression performance on the basis of traditional compression methods is a problem that researchers are concerned about.

In recent years, with the development of deep learning technology, especially the successful application of convolutional neural networks in image processing and computer vision, it has become possible to use deep learning technology for image compression. Compared with traditional image compression methods, image compression methods based on deep learning can jointly optimize codec, quantization, and entropy estimation on the one hand, so that the overall performance of compression is optimal; on the other hand, it can provide diversified codecs. The method can realize intelligent coding and decoding for different tasks, thereby effectively improving the compression performance of the image.

Image compression methods based on deep learning mainly include: Auto-encoder-based methods and Recurrent Neural Network (RNN)-based methods. The two methods are briefly introduced below.

Please refer to FIG. 2. FIG. 2 shows a schematic diagram of an image compression framework based on Auto-encoder. For this compression framework, when encoding, the original image is input to the encoding network, undergoes spatial transformation, and obtains encoded data through quantization, and then obtains compressed data through entropy encoding. When decoding, the compressed data is subjected to entropy decoding and dequantization, and then input to the decoding network, and the data is converted back to the image space through the decoding network to obtain a reconstructed image. The encoding network and the decoding network are both Convolutional Neural Networks (CNN), and the two constitute an Auto-encoder.

When the compression framework is trained, the coding network and the decoding network can be jointly optimized, and the reconstructed Loss can be obtained by comparing the original image and the reconstructed image; the code rate Loss can be obtained by estimating the entropy of the encoded data; the bit rate Loss can be adjusted and the reconstruction The weight of Loss is used to train models with different bit rates. Therefore, after the training is completed, a model is only applicable to one bit rate. That is, only one code rate of compressed data can be output for a kind of input image. If multiple code rates of compressed data need to be output, multiple models must be trained, which severely limits the application. Because there are various bandwidth and storage requirements in practical applications, outputting compressed data with various code rates is very important for practical applications.

The above-mentioned code rate is also called compression rate, which refers to the code length required for unit pixel coding. Generally, the higher the bit rate, the clearer the reconstructed image, the larger the storage space required for compressed data, and the higher the bandwidth required to transmit compressed data.

Please refer to FIG. 3, which shows a schematic diagram of an image compression framework based on RNN. The compression framework is a cyclic compression framework based on residual input, that is, in the first cycle, the encoder (Encoder) inputs the original image, and the decoder (Decoder) outputs the first reconstructed image. In the second cycle, the encoder inputs the residuals between the original image and the first reconstructed image, and the decoder outputs the compressed residuals, which are superimposed with the reconstructed image output from the previous time to obtain the second reconstructed image. By analogy, each time the cyclic encoder inputs the residual of the previous reconstructed image and the original image.

For this compression framework, the code rate is proportional to the number of cycles, so the code rate can be controlled by controlling the number of cycles, and one model can be applied to multiple code rates. However, the residual is not conducive to compression, so the RNN-based method has poor compression effect.

It can be seen from the above that the Auto-encoder-based method has a good compression effect, but a model is only suitable for one bit rate. The RNN-based method can realize that one model is suitable for multiple code rates, but the compression effect is not good. In practical applications, it is not only necessary to ensure that the visual quality of the picture is basically unchanged, but also that the bit rate is adjustable. Therefore, how to realize a compression model corresponding to multiple code rates while ensuring the compression effect is a problem that needs to be solved urgently.

In view of the above-mentioned problems, the inventor found in research that when an image is compressed and decompressed, part of the information needs to be restored through the compressed code stream, and the other part of the information can be derived from prior knowledge. Based on this idea, please refer to FIG. 4. Based on the existing deep learning image compression method, the embodiment of the present application adds a pre-processing link in the encoding process, and correspondingly, a post-processing link in the decoding process. When encoding, do preprocessing and then compression, and when decoding, do decompression and then post-processing. Post-processing refers to the process of reasoning with pre-processing as prior knowledge, that is, the reverse processing of pre-processing, so that the visual quality of the reconstructed image is basically unchanged.

At the same time, in the embodiment of the present application, multiple preset strategies are set in advance in the preprocessing step, and the target strategy is selected from the multiple preset strategies for preprocessing during encoding. The selected target strategy is different, the code rate of the compressed data will be different. When decoding the restored image after decompression, the reverse strategy of the target strategy is used to do the reverse processing of preprocessing. In this way, while ensuring the compression effect, one compression model corresponds to multiple code rates.

The implementation of the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

Please refer to FIG. 5. FIG. 5 is a schematic flowchart of an image processing method provided by an embodiment of the application. The image processing method is applied to the encoding end, for example, it may be an electronic device with encoding function, and the image processing method may include the following steps:

S101. Obtain an original image.

The original image can be image data that needs to be compressed in order to save storage space or meet bandwidth transmission requirements. For example, the raw data of the video stream output by the camera inside the camera, the pictures in the terminal album, the pictures in the cloud album, etc.

At the same time, the original image refers to uncompressed image data, and the data format of the original image can be RGB, YUV, CMYK. If the compression task corresponds to compressed image data (e.g., JPEG image), the corresponding decoder (e.g., JPEG decoder) needs to be used to decode the compressed image data (e.g., JPEG image) into the original image. Compress again.

S102: In response to an operation on the original image, determine a target strategy from a plurality of preset strategies, where at least two preset strategies correspond to different code rates of compressed data.

The encoding process of the embodiment of the present application adds a pre-processing step, and the pre-processing step has a plurality of preset strategies set in advance, and the preset strategies include methods and parameters for pre-processing the original image. For example, methods and parameters for global scaling of the original image, methods and parameters for blurring the original image, and methods and parameters for image enhancement of the original image.

Different preset strategies can be set to correspond to different code rates, that is, if different preset strategies are used to preprocess the original image, the code rates of the compressed data obtained may be different. However, in actual applications, different preset strategies may correspond to the same bit rate. For example, the method and parameters for global scaling of the original image and the method and parameters for blurring the original image are preprocessed according to the two preset strategies. The bit rate of the compressed data obtained after compression may be the same. That is, these two preset strategies correspond to the same bit rate. Therefore, in actual applications, it is only necessary to ensure that the code rates of the compressed data corresponding to at least two preset strategies are different.

The target strategy refers to any one of a plurality of preset strategies, and the target strategy is related to the user's operation on the original image.

The user's operation on the original image refers to the user's selection operation on the original image. The association relationship between the selection operation and the preset strategy can be preset, that is, one selection operation is preset to be associated with at least one preset strategy. For example, the relationship between the selection operation and the preset strategy is shown in Table 1 below:

Table 1

选择操作1 Select operation 1	预设策略1、预设策略2、预设策略3… Preset strategy 1, Preset strategy 2, Preset strategy 3...
选择操作2Select operation 2	预设策略1、预设策略2、预设策略3… Preset strategy 1, Preset strategy 2, Preset strategy 3...
选择操作3Select operation 3	预设策略1、预设策略2、预设策略3… Preset strategy 1, Preset strategy 2, Preset strategy 3...

…

Among them, the code rate of the compressed data corresponding to the preset strategy associated with the same selection operation (for example, selection operation 1) is similar, and the compression corresponding to the preset strategy associated with different selection strategies (for example, selection operation 1, selection operation 2) The bit rate of the data is different. In other words, each selection operation corresponds to a series of preset strategies, for example, the preset strategy for global zooming of the original image, the preset strategy for blurring the original image, etc., which are only different from different selection operations (for example, Selection operation 1, selection operation 2) are associated with different parameters of the same preset strategy (for example, preset strategy 1).

At the same time, an option is set for the user to select, and an option represents a user's compression requirement. For example, setting the "high", "medium", and "low" options respectively represents the compression quality desired by the user as high, medium, and low. When the user selects an option, it makes a selection operation. For example, when the user selects the "high" option, it makes a selection operation with high compression quality. For example, for the raw data of the video stream output by the camera's internal camera, if the user wants high compression quality, they can pre-select the "high" option before taking a picture.

For the original image, when the user selects an option, that is, a selection operation is performed on the original image, and then the preset strategy associated with the selection operation can be used as the target strategy according to the association relationship between the selection operation and the preset strategy. If the selection operation is associated with multiple preset strategies, then one of the multiple preset strategies will be found as the target strategy with the best effect. For example, the original image can be preprocessed and compressed according to each preset strategy, and then a compressed data with the best effect can be selected, and the preset strategy corresponding to the compressed data with the best effect can be used as the target strategy.

It should be pointed out that the selection operation includes the default operation, which means that the user did not select any option. When the user's operation on the original image is the default operation, one of the preset strategies associated with the default operation is found as the target strategy with the best effect. For example, if the default operation is high compression quality, then one of the preset strategies associated with high compression quality will be found as the target strategy with the best effect.

S103: Preprocess the original image according to the target strategy to obtain the image to be compressed.

The target strategy can be, but is not limited to, global scaling methods and parameters, block scaling methods and parameters, global blur methods and parameters, block blur methods and parameters, global enhancement methods and parameters, and block enhancement methods And one or more of the parameters. Correspondingly, the preprocessing may be, but is not limited to, one or more of global scaling, block scaling, global blur, block blur, global enhancement, block enhancement, and the like. For example, if the target strategy is the method and parameters of global scaling and the method and parameters of block blur, the preprocessing is global scaling and block blur.

The image to be compressed refers to the image obtained after preprocessing the original image according to the target strategy.

S104. Compress the image to be compressed using a preset deep learning image compression framework to obtain compressed data, where the compressed data is used to decompress the preset deep learning image decompression framework to obtain a restored image, and the restored image is used to obtain a restored image based on the target. The reverse strategy of the strategy performs the reverse processing of the preprocessing to obtain the reconstructed image corresponding to the original image.

Compressed data refers to the code stream obtained after the original image is preprocessed and compressed. The target strategy and compressed data can be stored or transmitted as a file. When it is necessary to decode compressed data into an image, first use the deep learning image decompression framework to decompress the compressed data into a restored image, and then infer the reverse strategy of the target strategy according to the target strategy used in preprocessing in the encoding process, and follow The reverse strategy of the target strategy performs reverse processing of preprocessing on the restored image, and finally generates a reconstructed image corresponding to the original image.

The aforementioned deep learning image compression framework and deep learning image decompression framework can be the Auto-encoder-based image compression framework shown in Figure 1, or the RNN-based image compression framework shown in Figure 2, or the field Other image compression frameworks based on deep learning that technicians may use.

On the basis of Fig. 5, please refer to Fig. 6, S103 may include the following detailed steps:

S1031: Preprocess the original image according to the first instruction and the first parameter to obtain the image to be compressed.

The target strategy includes a first instruction and a first parameter corresponding to the first instruction. The first instruction refers to a method for preprocessing the original image, and the first parameter refers to a parameter corresponding to the method for preprocessing the original image.

For example, the first instruction may be, but is not limited to, one or more of methods such as global scaling, block scaling, global blur, block blur, global enhancement, and block enhancement of the original image. The first parameter may be, but is not limited to, one or more of global scaling parameters, block scaling parameters, global blur parameters, block blur parameters, global enhancement parameters, block enhancement parameters and other parameters.

The original image can be preprocessed by using traditional image processing algorithms, for example, traditional image interpolation algorithms, Gaussian filtering, and so on. It is also possible to pre-process the original image with a pre-trained deep learning network, for example, a deep convolutional neural network, a convolutional layer, a pooling layer, and so on.

The process of preprocessing the original image according to the first instruction and the first parameter will be introduced as an example below.

In one embodiment, when the first instruction is a global zoom instruction and the first parameter is a global zoom factor and a zoom kernel, the preprocessing is global zoom;

According to the first instruction and the first parameter, the process of preprocessing the original image to obtain the image to be compressed may include:

According to the global zoom instruction, the global zoom factor and the zoom kernel, the original image is globally zoomed to obtain the image to be compressed.

The global zoom instruction refers to a method of global zooming of the original image. Global zooming is to reduce or enlarge the image as a whole. The preprocessing of the encoding process is just the opposite of the postprocessing of the decoding process. For example, the encoding process is: first reduce the original image and then compress, then the decoding process is: first decompress and then enlarge the restored image.

The global zoom factor is the number of times that the image is reduced or enlarged as a whole, and the global zoom factor can be represented by n. If n<1, it means that the entire image is reduced; if n>1, it means that the entire image is enlarged.

The scaling kernel includes linear interpolation, bilinear interpolation, and so on.

For example, referring to Figure 7, assuming that the global scaling factor is n (n<1), and the scaling core is the core corresponding to downsampling, the image is reduced by downsampling during the encoding process. The input is the original image, and the output is the long An image to be compressed whose sum width is n (n<1) times the original image size.

The reason why global zoom can be used in the image compression process is that a certain degree of zoom has little effect on image quality. The following experiments are used to prove:

Experimental data: CLIC public data set (330 photos);

Experimental process: Using the resize function, the original image is reduced by one time, and the restored image is doubled;

Experimental results: average MS-SSIM = 0.9947, with little effect.

Among them, MS-SSIM (Multi-Scale-Structural Similarity Index, multi-scale structural similarity) is an image compression quality evaluation index used to evaluate the similarity between the original image and the reconstructed image, and its value range is 0 to 1. The closer to 1 means that the reconstructed image is closer to the original image.

The image compression quality evaluation index is used to evaluate the image quality of the compressed image. In addition to the MS-SSIM mentioned above, it also includes PSNR (Peak Signal to Noise Ratio) and SSIM (structural similarity index, structural similarity) Wait. The higher the value of PSNR and SSIM, the smaller the distortion and the better the quality of the image after compression.

Using different global zoom factors to global zoom the original image, the bit rate of the corresponding compressed data will also be different. That is, using the same compression model, multiple code rate compression can be achieved only by adjusting the global scaling factor. The smaller the n, the smaller the scale of the image to be compressed, and the smaller the code rate of the compressed data; the larger the n, the larger the scale of the image to be compressed, and the larger the code rate of the compressed data.

For example, please refer to Figure 8. The left image is the experimental result of the Kodak data set, and the right image is the experimental result of the CLIC data set. In the figure, the vertical axis is MS-SSIM, and the horizontal axis is BPP (bits per pixel), which represents the number of bits consumed by each pixel. The smaller the BPP, the smaller the bit rate. The curve corresponding to GSM-org is the initial compression curve. GSM-newMSSSIM, GSM-newMSSSIM-0.25, and GSM-newMSSSIM-0.5 represent the compression curves with global compression coefficients of 1, 0.25, and 0.5, respectively. It can be clearly seen from the figure that using the same compression model, by adjusting the global scaling factor (1, 0.25, 0.5), the compression of 3 code rates can be achieved under the premise of ensuring the compression performance.

In another embodiment, when the first instruction is an adaptive scaling instruction and the first parameter is a block parameter, the preprocessing is to block first and then adaptive scaling;

First, divide the original image according to the block parameters to obtain multiple image blocks;

Then, according to the adaptive scaling instruction and the image characteristics of each image block, each image block is adaptively scaled to obtain the image block to be compressed corresponding to each image block, where the image to be compressed includes multiple images to be compressed Block, the image feature is used to determine the scaling factor of the image block.

The block parameter is a parameter used to characterize how to divide the original image. The block parameter can be represented by M×N, where M is the horizontal block parameter, and N is the vertical block parameter. For example, if the block parameter is 3×3, it means that the original image is divided into 9 image blocks of 3×3. At the same time, each image block after block division has a corresponding position vector (i, j), i represents the i-th image block in the horizontal direction, and j represents the j-th image block in the horizontal direction.

The image feature of the image block may be one or more of the color feature, texture feature, and shape feature of the image block. The color feature and texture feature are used to describe the surface properties of the object corresponding to the image block. The shape feature includes the contour feature and the area feature. The contour feature includes the outer boundary feature of the object, and the area feature includes the shape and area feature of the object.

The adaptive zoom command refers to a method of adaptively zooming the original image. Based on the image characteristics of the original image (for example, color characteristics, texture characteristics, shape characteristics, etc.), areas with different image characteristics can be scaled to different degrees, such as , The background area is zoomed more, and the foreground area is zoomed less.

Adaptive scaling for each image block means to determine the scaling factor of each image block according to the corresponding image feature (for example, color feature, texture feature, shape feature, etc.) of each image block, and then according to the respective scaling The coefficient performs block reduction or block enlargement. Generally, in order to ensure compression performance, image blocks with more image features (for example, color, texture, etc.) have a larger scaling factor; image blocks with fewer image features (for example, color, texture, etc.) have a smaller scaling factor. That is, smooth image blocks are scaled more, and unsmooth image blocks are scaled less.

For example, referring to Figure 9, assuming the block parameter is 4×3, the original image is first divided into 12 image blocks in the encoding process; then each image block is determined according to the image characteristics (for example, color, texture, etc.) The zoom factor of each image block. For example, according to the texture of the (2,2)th image block and the (1,1)th image block, the scaling factor of the (2,2)th image block is determined to be 1, the (2,2)th image block The zoom factor is 0.25.

In another embodiment, when the first instruction is a fuzzy processing instruction and the first parameter is a fuzzy kernel, the preprocessing is fuzzy processing;

According to the blur processing instruction and the blur kernel, the original image is subjected to blur processing to obtain the image to be compressed.

The blur processing instruction refers to a method of performing blur processing on the original image. The blur processing instruction can be, but is not limited to, a median function, an average function, a Gaussian function, and the like. The blur processing can be, but is not limited to, median blur, mean blur, Gaussian blur, and the like. Correspondingly, the fuzzy kernel can be, but is not limited to, a median template, an average template, a Gaussian template, and the like. The blur kernel is a kind of convolution kernel, which is actually a matrix. The original image and the blur kernel are convolved to blur the original image.

In the encoding process, blurring is performed before compression. For example, please refer to Figure 10 to perform Gaussian filtering on the original image and then compress it. Gaussian function can be used for smoothing filtering. The input is the original image and the output is the filtered image to be compressed.

By adjusting the fuzzy kernel, compression of different code rates can be achieved. Generally, the larger the scale of the fuzzy kernel (for example, Gaussian template) is, the more blurred the picture after filtering, and the smaller the bit rate of the compressed data; the smaller the scale of the fuzzy kernel (for example, the Gaussian template), the clearer the picture after filtering, and the better the compressed data. The bit rate is also larger. The following is proved by experiment:

The initial code rate and compression performance of the model are as follows:

BPP: 0.5048, PSNR: 28.7966, MS-SSIM: 0.9860;

After adjusting the scale of the Gaussian template, the new code rate and compression performance are as follows:

The Gaussian template scale is 3: BPP: 0.3100, PSNR: 26.8661, MS-SSIM: 0.9593;

The Gaussian template scale is 5: BPP: 0.3657, PSNR: 26.8711, MS-SSIM: 0.9667.

Obviously, the larger the Gaussian template scale, the larger the BPP, and the better the image quality after compression; that is, by adjusting the Gaussian template scale (3, 5), compression of 3 code rates is achieved.

In another embodiment, when the first instruction is an image degradation instruction and the first parameter is an image degradation parameter, the preprocessing is image degradation;

According to the image degradation instruction and the image degradation parameter, perform image degradation on the original image to obtain the image to be compressed.

The image degradation instruction refers to the method of degrading the original image. The image degradation is performed before compression in the encoding process. For example, please refer to Figure 11. The original image is degraded and then compressed. The input is the original image, and the output is the degraded image. The image to be compressed.

By adjusting the image degradation parameters, compression at different bit rates can be achieved. Generally, the larger the image degradation parameter, the smaller the bit rate of the compressed data.

In another embodiment, when the first instruction is an image separation instruction and a first post-processing instruction, the first parameter is an image separation parameter corresponding to the image separation instruction and a first post-processing instruction corresponding to the first post-processing instruction When processing parameters, the pre-processing is image separation first and then the first post-processing;

First, perform image separation on the original image according to the image separation instructions and image separation parameters to obtain edge images and texture images;

Then, according to the first post-stage processing instructions and the first post-stage processing parameters, at least one of global scaling, adaptive scaling, blur processing, and image degradation is performed on the edge image and the texture image to obtain the edge image to be compressed and the edge image to be compressed. Compressed texture image, where the image to be compressed includes edge image to be compressed and texture image to be compressed.

The image separation instruction refers to the method of image separation of the original image. Image separation can be to separate the original image into edge image and texture image according to the texture characteristics of the original image, and then perform the first post-processing on the edge image and texture image respectively . The first post-stage processing can be one or more of global scaling, adaptive scaling, blur processing, and image degradation. The detailed process of global zooming, adaptive zooming, blur processing, and image degradation can be referred to the foregoing description, and will not be repeated here.

The first post-processing instruction refers to the method of performing the first post-processing on the original image. The first post-processing instruction can be one or more of the global zoom instruction, the adaptive zoom instruction, the blur processing instruction, and the image degradation instruction. kind. Correspondingly, the first post-stage processing parameter may be one or more of the global zoom factor and zoom kernel, block parameter, blur kernel, and image degradation parameter.

The encoding process is the first stage of image separation, then the first stage of processing and then compression. For example, please refer to Figure 12, first the image is separated into edge images and texture images, and then Gaussian filtering is performed on the edge images and texture images respectively. The Gaussian template scale of the texture image can be smaller, and the Gaussian template scale of the edge image can be larger, to obtain the edge image to be compressed and the texture image to be compressed.

By adjusting the image separation parameters and the first post-stage processing parameters, compression at different bit rates can be achieved.

In another embodiment, when the first instruction is an image segmentation instruction and a second post-processing instruction, the first parameter is the segmentation category corresponding to the image segmentation instruction and the second post-processing instruction corresponding to the second post-processing instruction For parameters, the preprocessing is first image segmentation and then the second post-processing stage;

First, perform image segmentation on the original image according to the image segmentation instruction and segmentation category to obtain multiple image regions;

Then, according to the second post-processing instruction and the second post-processing parameters, at least one of global scaling, adaptive scaling, blur processing, and image degradation is performed on multiple image regions to obtain the corresponding image region The image area to be compressed, where the image to be compressed includes a plurality of image areas to be compressed.

The image segmentation instruction refers to the method of image segmentation of the original image. Image segmentation is the technology and process of dividing the image into a number of specific and unique areas and proposing objects of interest. Image segmentation can be based on the segmentation category, the original image is divided into several image areas, each image area has a corresponding location coordinates (x, y). The position coordinates can be the coordinates of each edge point of the corresponding image area, or the coordinates of the center point of the corresponding image area.

The segmentation categories can be foreground and background, or categories of all targets in the foreground, such as humans, animals, plants, and so on. Generally, the image segmentation method can be, but is not limited to, threshold-based segmentation, region-based segmentation, edge-based segmentation, and so on.

In the encoding process, the original image can be divided into multiple image regions (for example, foreground and background) according to the segmentation category, and then the second post-processing is performed on each image region separately. The second post-stage processing can be one or more of global scaling, adaptive scaling, blur processing, and image degradation. The detailed process of global zooming, adaptive zooming, blur processing, and image degradation can be referred to the foregoing description, and will not be repeated here.

The second post-processing instruction refers to the method of performing the second post-processing on the original image. The second post-processing instruction can be one or more of global scaling instructions, adaptive scaling instructions, blur processing instructions, and image degradation instructions. kind. Correspondingly, the second post-stage processing parameter may be one or more of the global zoom factor and zoom kernel, block parameter, blur kernel, and image degradation parameter.

The encoding process is to segment the image first, then process the second stage and then compress it. For example, referring to Figure 13, the image is first segmented into the foreground (ie flies) and the background, and then Gaussian filtering is performed on the foreground and the background respectively. The Gaussian template scale of the foreground can be smaller, and the Gaussian template scale of the background can be larger, to obtain the foreground to be compressed and the background to be compressed.

By adjusting the segmentation category and the second post-stage processing parameters, compression at different bit rates can be achieved.

The deep learning image compression framework includes the first deep neural network, quantization model and entropy coding model. On the basis of FIG. 5, please refer to FIG. 14. S104 may include the following detailed steps:

S1041: Perform feature extraction on the compressed image using the first deep neural network to obtain image features.

The first deep neural network may be a fully connected neural network, CNN, CNN variants, RNN, RNN variants, etc., or may be other deep neural networks that may be used by those skilled in the art. The variants of CNN can be DCNN (Dilated Convolutions Neural Network), IDCNN (Iteration Dilated Convolutions Neural Network), etc. RNN variants can be LSTM (Long Short-Term Memory), GRU (Gated Recurrent Unit), etc.

The first deep neural network is used for feature extraction of the image to be compressed to obtain image features.

S1042: Use the quantization model to quantize the image feature to obtain the compressed feature.

The quantization model is used to discretize the compressed features to save storage space and facilitate further entropy coding.

S1043: Entropy coding the compressed feature using the entropy coding model to obtain compressed data.

After obtaining the compression features, use entropy coding to further reduce the amount of data, and the entropy coding model can use arithmetic coding and so on.

Please refer to FIG. 15, which shows another schematic flowchart of an image processing method provided by an embodiment of the present application. The image processing method is applied to the decoding end, for example, it may be an electronic device with a decoding function, and the image compression method may include the following steps:

S201: Obtain compressed data, where the compressed data is obtained by compressing a to-be-compressed image using a preset deep learning image compression framework, and the to-be-compressed image is obtained by preprocessing the original image according to a target strategy, and the target strategy is a response to the original image The operation is determined from a plurality of preset strategies, and the code rates of the compressed data corresponding to at least two preset strategies are different.

When a user wants to view or send a picture of a terminal album, or view or download a picture of a cloud album, the terminal or the cloud will decompress the corresponding compressed data into a restored image. At the same time, in order to make the reconstructed image and the original image as consistent as possible, it is necessary to process the restored image according to the reverse processing of preprocessing.

S202: Decompress the compressed data using a preset deep learning image decompression framework to obtain a restored image.

S203: Obtain a reverse strategy corresponding to the target strategy.

S204: Perform reverse processing of pre-processing on the restored image according to the reverse strategy to obtain a reconstructed image corresponding to the original image.

The aforementioned deep learning image decompression framework includes a second deep neural network, an inverse quantization model, and an entropy decoding model. On the basis of FIG. 15, please refer to FIG. 16, S202 may include the following detailed steps:

S2021: Entropy decoding the compressed data using the entropy decoding model to obtain compression features.

S2022: Inversely quantize the compressed features using an inverse quantization model to obtain image features.

S2023: Use the second deep neural network to restore the image features to obtain a restored image.

The second deep neural network is used to transform and learn image features, so as to restore the frequency domain information to the pixel domain without loss, and obtain a restored image.

The second deep neural network may be a fully connected neural network, CNN, CNN variants, RNN, RNN variants, etc., or other deep neural networks that may be used by those skilled in the art. The CNN variants can be DCNN, IDCNN, etc., and the RNN variants can be LSTM, GRU, etc.

On the basis of FIG. 15, please refer to FIG. 17, S203 may include the following detailed steps:

S2031: Obtain a target strategy, where the target strategy includes a first instruction and a first parameter corresponding to the first instruction.

S2032: Determine the second command according to the corresponding relationship between the first command and the preset command.

S2033: Determine a second parameter according to the first parameter and a preset parameter calculation rule, where the reverse strategy includes a second instruction and a second parameter corresponding to the second instruction.

The correspondence between the first instruction and the second instruction can be preset to determine the second instruction according to the first instruction. For example, in the instruction correspondence, the global zoom instruction corresponds to the global zoom instruction, the blur processing instruction corresponds to the deblur processing instruction, and so on. At the same time, the corresponding relationship between the first parameter and the second parameter is preset to determine the second parameter according to the first parameter. For example, in the parameter corresponding relationship, the first parameter is the global scaling factor and the scaling kernel, and the second parameter is the global scaling factor. The reciprocal and zoom kernel etc.

On the basis of FIG. 15, please refer to FIG. 18. S204 may include the following detailed steps:

S2041: Perform reverse processing of preprocessing on the restored image according to the second instruction and the second parameter to obtain a reconstructed image corresponding to the original image.

The second instruction refers to a reverse processing method of preprocessing the restored image, and the second parameter refers to a parameter corresponding to the reverse processing method of preprocessing the original image.

Traditional image processing algorithms can be used to perform reverse processing of the restored image, for example, traditional image interpolation algorithms, Gaussian filtering, super-resolution algorithms, etc. It is also possible to use a pre-trained deep learning network to perform reverse processing of preprocessing the restored image.

The process of reverse processing of preprocessing the restored image according to the second instruction and the second parameter will be described below with an example.

In one embodiment, if the first instruction is a global scaling instruction, the first parameter is a global scaling factor and a scaling kernel, and the preprocessing is global scaling, then the second instruction is a global scaling instruction, and the second parameter is the reciprocal of the global scaling factor. And zooming core, the reverse processing of pre-processing is global zooming;

According to the second instruction and the second parameter, the reverse process of preprocessing the restored image to obtain the reconstructed image corresponding to the original image may include:

According to the global zoom instruction, the reciprocal of the global zoom factor and the zoom kernel, the restored image is globally zoomed to obtain a reconstructed image.

For example, in conjunction with Figure 7, during the decoding process, an up-sampling method is used to enlarge the image, the input is a restored image, and the output is a reconstructed image whose length and width are the same size as the original image.

In this case, the super-resolution algorithm can also be used to perform the reverse processing of the preprocessing of the restored image to obtain the reconstructed image.

In another embodiment, if the first instruction is an adaptive scaling instruction, the first parameter is a block parameter, and the preprocessing is to block first and then adaptive scaling, then the second instruction is an adaptive scaling instruction, and the second parameter is The splicing parameters related to the block parameters, the reverse processing of the pre-processing is first adaptive scaling and then splicing;

First, according to the adaptive scaling instructions and the image characteristics of each restored image block, each restored image block is adaptively scaled to obtain the image block to be reconstructed corresponding to each restored image block, where the image feature is used to determine the restoration The zoom factor of the image block;

Then, the multiple image blocks to be reconstructed are spliced according to the splicing parameters to obtain a reconstructed image.

The stitching parameter is associated with the block parameter, including the position vector (i, j) corresponding to each image block after the block, that is, the position vector (i, j) corresponding to each restored image block.

For example, in conjunction with Figure 9, in the decoding process, the zoom factor of each restored image block is determined according to the image characteristics (for example, color, texture, etc.) of each restored image block, and then the position vector corresponding to each restored image block ( i, j) Perform splicing and output the reconstructed image.

In another embodiment, if the first instruction is a fuzzy processing instruction, the first parameter is a fuzzy kernel, and the preprocessing is a fuzzy processing, then the second instruction is a de-blurring instruction, and the second parameter is a de-blurring kernel corresponding to the fuzzy kernel. , The reverse processing of pre-processing is de-blurring processing;

According to the deblurring processing instruction and the deblurring kernel, the restored image is deblurred to obtain the reconstructed image.

The deblurring processing can be, but is not limited to, edge detection, image sharpening, deep learning image restoration, etc.; correspondingly, the deblurring kernel can be a sharpening kernel and the like.

If the encoding process performs blurring before compression, the decoding process performs deblurring after decompression. For example, in conjunction with Figure 10, first decompress and then sharpen the image, you can use the Laplacian sharpening function or deep learning network for sharpening, the input is the restored image, and the sharpened reconstructed image is output.

In another embodiment, if the first instruction is an image degradation instruction, the first parameter is an image degradation parameter, and the preprocessing is image degradation, then the second instruction is the image enhancement instruction, and the second parameter is an image enhancement parameter;

According to the image enhancement instructions and the image enhancement parameters, image enhancement is performed on the restored image to obtain a reconstructed image.

The purpose of image enhancement is to improve the visual effect of the image, or to convert the image into a form more suitable for human observation and machine analysis and recognition, so as to obtain more useful information from the image.

The image enhancement method may be, but is not limited to, histogram equalization, contrast enhancement, gamma transformation, noise smoothing, sharpening, and the like. The image enhancement instructions can be, but are not limited to, transformation functions, Laplacian operators, and so on.

If the encoding process performs image degradation before compression, the decoding process performs image enhancement after decompression. For example, in conjunction with Figure 11, the decoding process is first decompression and then image enhancement. Deep learning network post-processing can be used. The input is a restored image and the output is a reconstructed image.

In another embodiment, if the first instruction is an image separation instruction and a first post-processing instruction, the first parameter is an image separation parameter corresponding to the image separation instruction and an image separation parameter corresponding to the first post-processing instruction. The first post-stage processing parameter, the preprocessing is the image separation first and then the first post-stage processing; the second instruction is the reverse instruction of the image separation instruction and the reverse instruction of the first post-processing instruction, and the second parameter is the image separation instruction. The reverse parameter of the image separation parameter corresponding to the reverse instruction of the separation instruction and the reverse parameter of the first post-processing parameter corresponding to the reverse instruction of the first post-processing instruction, the reverse processing of the preprocessing is first first Reverse processing and image fusion of post-processing;

First, according to the reverse instruction of the first post-processing instruction and the reverse parameter of the first post-processing parameter, both the restored edge image and the restored texture image are subjected to the reverse processing of global scaling, the reverse processing of adaptive scaling, At least one of deblurring and image enhancement to obtain an edge image to be reconstructed and a texture image to be reconstructed;

Then, according to the reverse instruction of the image separation instruction and the reverse parameter of the image separation parameter, image fusion is performed on the edge image to be reconstructed and the texture image to be reconstructed to obtain a reconstructed image.

The reverse command of the first post-stage processing command may be one or more of the reverse command of the global zoom command, the reverse command of the adaptive zoom command, the deblur processing command, and the image enhancement command. Correspondingly, the first post-stage processing parameter may be one or more of the reciprocal of the global zoom factor and zoom kernel, stitching parameter, deblurring kernel, and image enhancement parameter.

If the encoding process is first image separation and then the first stage of processing and then compression, then the decoding process is the reverse processing of first decompression and then the first post-stage processing and then image fusion. For example, referring to Figure 12, in the decoding process, image sharpening is performed on the decompressed restored edge image and the restored texture image respectively, and then the two sharpened images are image fused and output as a reconstructed image.

In another embodiment, if the first instruction is an image segmentation instruction and a second post-processing instruction, the first parameter is the segmentation category corresponding to the image segmentation instruction and the second post-processing instruction corresponding to the second post-processing instruction Parameter, the preprocessing is first image segmentation and then the second post-stage processing; then the second instruction is the reverse instruction of the image segmentation instruction and the reverse instruction of the second post-stage processing instruction, and the second parameter is the same as the second post-stage processing instruction The reverse instruction corresponding to the reverse parameter of the second post-stage processing parameter, the reverse processing of the pre-processing is the reverse processing of the second post-processing first and then splicing; the restored image includes multiple restored image regions and each restored image The location coordinates of the area;

First, in accordance with the reverse instruction of the second post-processing instruction and the reverse parameter of the second post-processing parameter, each restored image area is subjected to the reverse processing of the global zoom, the reverse processing of the adaptive zoom, and the deblurring At least one of processing and image enhancement to obtain the image area to be reconstructed corresponding to each restored image area;

Then, according to the reverse instruction of the image segmentation instruction and the position coordinates of each restored image area, a plurality of image areas to be reconstructed are spliced to obtain the reconstructed image.

If the encoding process is first image segmentation and then second stage processing and then compression, then the decoding process is first decompression and then reverse processing of the second post-stage processing and then splicing. For example, in conjunction with Figure 13, in the decoding process, the decompressed restored foreground and restored background are respectively sharpened, and then the two sharpened image regions are spliced and output as a reconstructed image.

In a possible situation, due to the influence of preprocessing, the subjective visual effect of generating the reconstructed image may be poor. Therefore, on the basis of FIG. 15, FIG. 19 is a schematic flowchart of another image processing method provided by an embodiment of the application. Referring to FIG. 19, after S204, the image processing method may further include the following steps:

S205: Use at least one of a super-resolution algorithm, a deblurring algorithm, a dehazing algorithm, and a denoising algorithm to process the reconstructed image, so as to improve the visual effect of the reconstructed image.

Please refer to FIG. 20, which shows another schematic flowchart of an image processing method provided by an embodiment of the present application. The image processing method is applied to the encoding and decoding end, for example, it may be an electronic device with encoding and decoding functions, and the image processing method may include the following steps:

S301: Obtain an original image.

S302: In response to an operation on the original image, determine a target strategy from a plurality of preset strategies, where at least two preset strategies have different code rates for the compressed data.

S303: Perform preprocessing on the original image according to the target strategy to obtain the image to be compressed.

S304: Compress the image to be compressed using a preset deep learning image compression framework to obtain compressed data.

S305: Decompress the compressed data using a preset deep learning image decompression framework to obtain a restored image.

S306: Obtain a reverse strategy corresponding to the target strategy.

S307: Perform reverse processing of preprocessing the restored image according to the reverse strategy to obtain a reconstructed image corresponding to the original image.

The detailed implementation of S301 to S307 can be referred to the description of the foregoing embodiment, which will not be repeated here.

The application scenarios of the image processing method provided in the embodiments of the present application will be introduced below with examples.

In an application scenario, please refer to Figure 21. The user uses the terminal camera to take a picture. Before taking the picture, you can set the compression quality of the camera. For example, select "Compression Quality (9/10)". There are 10 levels of compression quality, 1 means the worst, 10 means the best, 9/10 means the compression quality is 9, if not selected, it is the default compression quality, and "compression quality (10/10)" in the figure is the default compression quality . Then the camera takes pictures, and the camera outputs the raw data of the video stream. Since the compression level selected by the user is "compression quality (9/10)", it can automatically change from the one corresponding to the "compression quality (9/10)" according to the raw data of the video stream. The target strategy with the best effect is found out of the series of preset strategies, and the raw video stream data is preprocessed and compressed according to the target strategy, and a compressed file including the compressed data and the target strategy is generated and stored to save the storage space of the terminal. When the user wants to view a certain picture stored in the terminal, the terminal will compress the file and display it after decoding.

In another application scenario, please refer to Figure 22. The user uploads the picture of the terminal album to the cloud album, and can select the compression quality before uploading, for example, select "Compression quality (9/10)", the same as above, if you do not select It is the default compression quality, and "compression quality (10/10)" in the figure is the default compression quality. After uploading the picture to the cloud, if the picture in the terminal album is compressed (for example, a .jpg file), the cloud must first parse the picture into an original image (for example, in YUV format). The cloud automatically finds the target strategy with the best compression effect according to the compression level selected by the user "compression quality (9/10)" and the picture, and compresses the original image after preprocessing according to the target strategy, and generates a compression including the compressed data and the target strategy File and store to save cloud storage space.

When a user wants to download or preview a certain picture of a cloud album, the cloud obtains the compressed file, which is decoded for the user to download or preview. At the same time, if the picture uploaded by the user is a file in a specific format, for example, a .jpg file, the cloud will process the reconstructed image into a specific format for the user to download or preview. Alternatively, the cloud can also provide a corresponding decoder, and the user directly downloads the compressed file, and then decodes it with the decoder provided by the cloud after downloading.

In another application scenario, please refer to Figure 23, the user sends a picture of the terminal album (for example, picture A) to other terminals, and can select the compression quality before sending, for example, select "compression quality (9/10)" , Same as above, if not selected, it is the default compression quality, and "compression quality (10/10)" in the figure is the default compression quality. The sender terminal automatically selects and finds the target strategy with the best compression effect according to the compression level "compression quality (9/10)" selected by the user and the selected picture, and preprocesses the picture according to the target strategy and compresses it to generate compressed data and The compressed file of the target strategy is transmitted to the receiving terminal to save transmission bandwidth. At the same time, if the picture of the terminal album has been compressed (for example, a .jpg file), the sender terminal must first parse the picture into an original image (for example, YUV format) before preprocessing and compression. When the receiver wants to download "Picture A", the receiver terminal obtains the compressed file, which is decoded for the user to download.

In order to execute the corresponding steps in the foregoing image processing method embodiment and each possible implementation manner, possible implementation manners of the image processing apparatus are given below.

Please refer to FIG. 24. FIG. 24 is a schematic diagram of the composition of an image processing apparatus 100 according to an embodiment of the application. The image processing apparatus 100 is applied to the encoding end, and may be, for example, an electronic device with encoding function. The image processing device 100 includes an image acquisition module 101, a response module 102, a preprocessing module 103, and a compression module 104.

The image acquisition module 101 is used to acquire an original image.

The response module 102 is configured to determine a target strategy from a plurality of preset strategies in response to an operation on the original image, wherein the code rates of the compressed data corresponding to at least two preset strategies are different.

The preprocessing module 103 is used to preprocess the original image according to the target strategy to obtain the image to be compressed.

The compression module 104 is configured to compress the image to be compressed using a preset deep learning image compression framework to obtain compressed data, where the compressed data is used to decompress the preset deep learning image decompression framework to obtain a restored image, and restore the image Reverse processing for preprocessing based on the reverse strategy of the target strategy obtains the reconstructed image corresponding to the original image.

In one embodiment, the target strategy includes a first instruction and a first parameter corresponding to the first instruction;

The preprocessing module 103 is specifically configured to preprocess the original image according to the first instruction and the first parameter to obtain the image to be compressed.

Optionally, the first instruction includes a global zoom instruction, and the first parameter includes a global zoom factor and a zoom kernel;

The preprocessing module 103 executes the preprocessing of the original image according to the first instruction and the first parameter to obtain the image to be compressed, which may include: global scaling of the original image according to the global scaling instruction, the global scaling factor and the scaling kernel, Get the image to be compressed.

Optionally, the first instruction includes an adaptive scaling instruction, and the first parameter includes a block parameter;

The preprocessing module 103 executes the preprocessing of the original image according to the first instruction and the first parameter to obtain the image to be compressed, which may include: dividing the original image according to the block parameters to obtain multiple image blocks; The zoom instruction and the image feature of each image block are adaptively scaled for each image block to obtain the image block to be compressed corresponding to each image block. The image to be compressed includes multiple image blocks to be compressed, and the image feature is used for To determine the zoom factor of the image block.

Optionally, the first instruction includes a fuzzy processing instruction, and the first parameter includes a fuzzy kernel;

The preprocessing module 103 performs preprocessing on the original image according to the first instruction and the first parameter to obtain the image to be compressed, which may include: performing blur processing on the original image according to the blur processing instruction and blur kernel to obtain the image to be compressed .

Optionally, the first instruction includes an image degradation instruction, and the first parameter includes an image degradation parameter;

The pre-processing module 103 performs pre-processing on the original image according to the first instruction and the first parameter to obtain the image to be compressed, which may include: performing image degradation on the original image according to the image degradation instruction and image degradation parameters to obtain the image to be compressed. image.

Optionally, the first instruction includes an image separation instruction and a first post-processing instruction, and the first parameter includes an image separation parameter corresponding to the image separation instruction and a first post-processing parameter corresponding to the first post-processing instruction;

The preprocessing module 103 performs preprocessing on the original image according to the first instruction and the first parameter to obtain the image to be compressed, which may include: performing image separation on the original image according to the image separation instruction and image separation parameters to obtain the edge image And texture images; according to the first post-processing instructions and first post-processing parameters, perform at least one of global scaling, adaptive scaling, blur processing, and image degradation on both the edge image and the texture image to obtain the edge image to be compressed And the texture image to be compressed, where the image to be compressed includes the edge image to be compressed and the texture image to be compressed.

Optionally, the first instruction includes an image segmentation instruction and a second post-processing instruction, and the first parameter includes a segmentation category corresponding to the image segmentation instruction and a second post-processing parameter corresponding to the second post-processing instruction;

The preprocessing module 103 executes preprocessing of the original image according to the first instruction and the first parameter to obtain the image to be compressed, which may include: image segmentation of the original image according to the image segmentation instruction and segmentation category to obtain multiple images Area; according to the second post-processing instruction and the second post-processing parameters, at least one of global scaling, adaptive scaling, blur processing, and image degradation is performed on multiple image regions, to obtain the corresponding to each image region A compressed image area, where the image to be compressed includes a plurality of image areas to be compressed.

In one embodiment, the deep learning image compression framework includes a first deep neural network, a quantization model, and an entropy coding model;

The compression module 104 is specifically configured to use the first deep neural network to perform feature extraction on the compressed image to obtain the image feature; use the quantization model to quantize the image feature to obtain the compressed feature; use the entropy coding model to entropy encode the compressed feature to obtain the compression data.

Please refer to FIG. 25. FIG. 25 is a schematic diagram of the composition of an image processing apparatus 200 according to an embodiment of the application. The image processing apparatus 200 is applied to a decoding end, and may be, for example, an electronic device with a decoding function. The image processing device 200 includes a sequence obtaining module 201, a decompression module 202, a reverse strategy obtaining module 203, and a post-processing module 204.

The sequence obtaining module 201 is used to obtain compressed data, where the compressed data is obtained by compressing the image to be compressed using a preset deep learning image compression framework, and the image to be compressed is obtained by preprocessing the original image according to the target strategy. The target strategy is It is determined from a plurality of preset strategies in response to the operation on the original image, that the code rates of the compressed data corresponding to at least two preset strategies are different.

The decompression module 202 is configured to decompress the compressed data using a preset deep learning image decompression framework to obtain a restored image.

The reverse strategy obtaining module 203 is used to obtain the reverse strategy corresponding to the target strategy.

The post-processing module 204 is configured to perform reverse processing of preprocessing the restored image according to the reverse strategy to obtain a reconstructed image corresponding to the original image.

In one embodiment, the deep learning image decompression framework includes a second deep neural network, an inverse quantization model, and an entropy decoding model;

The decompression module 202 is specifically configured to: use the entropy decoding model to entropy decode the compressed data to obtain compressed features; use the inverse quantization model to dequantize the compressed features to obtain image features; use the second deep neural network to restore the image features, Get the restored image.

In one embodiment, the reverse strategy obtaining module 203 is specifically configured to: obtain a target strategy, the target strategy including a first instruction and a first parameter corresponding to the first instruction; Two instructions; the second parameter is determined according to the first parameter and the preset parameter calculation rule, wherein the reverse strategy includes the second instruction and the second parameter corresponding to the second instruction.

In one embodiment, the post-processing module 204 is specifically configured to perform reverse processing of pre-processing on the restored image according to the second instruction and the second parameter to obtain a reconstructed image corresponding to the original image.

Optionally, the first instruction includes a global zoom instruction, the first parameter includes a global zoom factor and a zoom core; the second instruction includes a global zoom instruction, and the second parameter includes the reciprocal of the global zoom factor and the zoom core;

The post-processing module 204 executes the reverse processing of preprocessing the restored image according to the second instruction and the second parameter to obtain the reconstructed image corresponding to the original image, including: following the global scaling instruction, the inverse of the global scaling factor, and scaling Core, global zoom the restored image to obtain the reconstructed image.

Optionally, the first instruction includes an adaptive scaling instruction, the first parameter includes a block parameter; the second instruction includes an adaptive scaling instruction, and the second parameter includes a splicing parameter associated with the block parameter; the restored image includes a plurality of restored image blocks ；

The post-processing module 204 executes the reverse process of preprocessing the restored image according to the second instruction and the second parameter to obtain the reconstructed image corresponding to the original image, including: following the adaptive scaling instruction and each restored image block Image feature, each restored image block is adaptively scaled to obtain the image block to be reconstructed corresponding to each restored image block, where the image feature is used to determine the scaling factor of the restored image block; multiple to be reconstructed according to the stitching parameters The image blocks are stitched together to obtain a reconstructed image.

Optionally, the first instruction includes a fuzzy processing instruction, and the first parameter includes a fuzzy kernel; the second instruction includes a deblurring processing instruction, and the second parameter includes a deblurring kernel corresponding to the fuzzy kernel;

The post-processing module 204 executes the reverse process of preprocessing the restored image according to the second instruction and the second parameter to obtain the reconstructed image corresponding to the original image, including: according to the deblurring processing instruction and the deblurring kernel, the restoration The image is deblurred to obtain a reconstructed image.

Optionally, the first instruction includes an image degradation instruction, and the first parameter includes an image degradation parameter; the second instruction includes an image enhancement instruction, and the second parameter includes an image enhancement parameter;

The post-processing module 204 performs the reverse processing of preprocessing the restored image according to the second instruction and the second parameter to obtain the reconstructed image corresponding to the original image, including: according to the image enhancement instruction and the image enhancement parameter, the restored image Perform image enhancement to obtain a reconstructed image.

The second instruction includes the reverse instruction of the image separation instruction and the reverse instruction of the first post-stage processing instruction, and the second parameter includes the reverse parameter of the image separation parameter corresponding to the reverse instruction of the image separation instruction and the reverse instruction of the first post-stage The inverse instruction of the processing instruction corresponds to the inverse parameter of the first post-stage processing parameter; restoring the image includes restoring the edge image and restoring the texture image;

The post-processing module 204 executes the reverse processing of preprocessing the restored image according to the second instruction and the second parameter to obtain the reconstructed image corresponding to the original image, including: the reverse instruction and the reverse instruction according to the first post-processing instruction The inverse parameter of the first post-processing parameter is to perform at least one of global scaling inverse processing, adaptive scaling inverse processing, deblurring processing, and image enhancement on both the restored edge image and the restored texture image to obtain the The edge image and the texture image to be reconstructed are reconstructed; according to the reverse instruction of the image separation instruction and the reverse parameter of the image separation parameter, the edge image to be reconstructed and the texture image to be reconstructed are image fused to obtain the reconstructed image.

The second instruction includes the reverse instruction of the image segmentation instruction and the reverse instruction of the second post-processing instruction, and the second parameter includes the reverse parameter of the second post-processing parameter corresponding to the reverse instruction of the second post-processing instruction ; The restored image includes multiple restored image areas and the position coordinates of each restored image area;

The post-processing module 204 executes the reverse processing of preprocessing the restored image according to the second instruction and the second parameter to obtain the reconstructed image corresponding to the original image, including: the reverse instruction and the reverse instruction according to the second post-processing instruction The second post-processing parameter is the inverse parameter, and each restored image area is subjected to at least one of the inverse processing of global scaling, the inverse processing of adaptive scaling, the deblurring processing, and the image enhancement to obtain each restored image area. The image area to be reconstructed corresponding to the image area; according to the reverse instruction of the image segmentation instruction and the position coordinates of each restored image area, a plurality of image areas to be reconstructed are spliced to obtain the reconstructed image.

In an embodiment, the post-processing module 204 is further configured to process the reconstructed image using at least one of a super-resolution algorithm, a deblurring algorithm, a dehazing algorithm, and a denoising algorithm to improve the visual effect of the reconstructed image.

Please refer to FIG. 26. FIG. 26 is a schematic diagram of the composition of an image processing apparatus 300 according to an embodiment of the application. The image processing device 300 is applied to the encoding and decoding end, and may be, for example, an electronic device with encoding and decoding functions. The image processing device 300 includes an image acquisition module 301, a response module 302, a preprocessing module 303, a compression module 304, a decompression module 305, a reverse strategy acquisition module 306, and a post-processing module 307.

The image acquisition module 301 is used to acquire the original image.

The response module 302 is configured to determine a target strategy from a plurality of preset strategies in response to an operation on the original image, wherein the code rates of the compressed data corresponding to at least two preset strategies are different.

The preprocessing module 303 is used to preprocess the original image according to the target strategy to obtain the image to be compressed.

The compression module 304 is configured to compress the image to be compressed using a preset deep learning image compression framework to obtain compressed data.

The decompression module 305 is configured to decompress the compressed data using a preset deep learning image decompression framework to obtain a restored image.

The reverse strategy obtaining module 306 is used to obtain the reverse strategy corresponding to the target strategy.

The post-processing module 307 is configured to perform reverse processing of preprocessing the restored image according to the reverse strategy to obtain a reconstructed image corresponding to the original image.

Those skilled in the art can clearly understand that, for the convenience and conciseness of the description, the working process of the image processing apparatus 100, 200, 300 described above can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

Please refer to FIG. 27, which is a schematic diagram of the composition of an electronic device 10 provided by an embodiment of the application. The electronic device 10 may be a terminal, a server, etc. The electronic device 10 includes a processor 11, a memory 12, and a bus 13. The processor 11 passes through the bus. 13 is connected to the memory 12.

The memory 12 is used to store programs. For example, the image processing device 100 shown in FIG. 24 includes at least one operating system that can be stored in the memory 12 in the form of software or firmware or solidified in the electronic device 10 , The software function module in the OS). After receiving the execution instruction, the processor 11 executes the program to implement the image processing method applied to the encoding end disclosed in the foregoing embodiment.

The memory 12 may include a high-speed random access memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM).

The processor 11 may be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the above method can be completed by an integrated logic circuit of hardware in the processor 11 or instructions in the form of software. The aforementioned processor 11 may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a microcontroller unit (Microcontroller Unit, MCU), a complex programmable logic device (Complex Programmable Logic Device, CPLD), and an on-site programmable logic device (CPLD). Programmable gate array (Field-Programmable Gate Array, FPGA), embedded ARM and other chips.

The embodiments of the present application also provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the image processing method disclosed in the above-mentioned embodiments is implemented.

The embodiments of the present application also provide a computer program product, including instructions, which when run on a computer, cause the computer to execute the image processing method disclosed in the above embodiments.

The embodiments of the present application provide a chip system. The chip system includes a processor and may also include a memory for implementing the image processing method disclosed in the foregoing embodiments. The chip system can be composed of chips, or it can include chips and other discrete devices.

Although this application is disclosed as above, this application is not limited to this. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of this application. Therefore, the protection scope of this application shall be subject to the scope defined by the claims.

Claims

An image processing method, characterized in that the image processing method includes:

Get the original image;

In response to the operation on the original image, a target strategy is determined from a plurality of preset strategies, wherein at least two of the preset strategies have different code rates for the compressed data;

Preprocessing the original image according to the target strategy to obtain the image to be compressed;

Use a preset deep learning image compression framework to compress the image to be compressed to obtain the compressed data, wherein the compressed data is used to decompress the preset deep learning image decompression framework to obtain a restored image, so The restored image is used to perform reverse processing of the pre-processing based on the reverse strategy of the target strategy to obtain a reconstructed image corresponding to the original image.
The image processing method according to claim 1, wherein the target strategy comprises a first instruction and a first parameter corresponding to the first instruction;

The step of preprocessing the original image according to the target strategy to obtain the image to be compressed includes:

According to the first instruction and the first parameter, the original image is preprocessed to obtain the image to be compressed.
The image processing method according to claim 2, wherein the first instruction includes a global zoom instruction, and the first parameter includes a global zoom factor and a zoom kernel;

The step of preprocessing the original image according to the first instruction and the first parameter to obtain the image to be compressed includes:

According to the global zoom instruction, the global zoom factor and the zoom kernel, the original image is globally zoomed to obtain the image to be compressed.
The image processing method according to claim 2, wherein the first instruction includes an adaptive scaling instruction, and the first parameter includes a block parameter;

The step of preprocessing the original image according to the first instruction and the first parameter to obtain the image to be compressed includes:

Divide the original image according to the block parameters to obtain multiple image blocks;

According to the adaptive scaling instruction and the image characteristics of each image block, each image block is adaptively scaled to obtain the image block to be compressed corresponding to each image block, wherein the to-be-compressed image block is obtained. The compressed image includes a plurality of image blocks to be compressed, and the image feature is used to determine the scaling factor of the image block.
The image processing method according to claim 2, wherein the first instruction includes a blur processing instruction, and the first parameter includes a blur kernel;

The step of preprocessing the original image according to the first instruction and the first parameter to obtain the image to be compressed includes:

According to the blur processing instruction and the blur kernel, the original image is subjected to blur processing to obtain the image to be compressed.
The image processing method according to claim 2, wherein the first instruction includes an image degradation instruction, and the first parameter includes an image degradation parameter;

The step of preprocessing the original image according to the first instruction and the first parameter to obtain the image to be compressed includes:

Perform image degradation on the original image according to the image degradation instruction and the image degradation parameter to obtain the image to be compressed.
The image processing method according to claim 2, wherein the first instruction includes an image separation instruction and a first post-processing instruction, and the first parameter includes an image separation parameter and an image separation parameter corresponding to the image separation instruction. A first post-stage processing parameter corresponding to the first post-stage processing instruction;

The step of preprocessing the original image according to the first instruction and the first parameter to obtain the image to be compressed includes:

Performing image separation on the original image according to the image separation instruction and the image separation parameter to obtain an edge image and a texture image;

According to the first post-processing instruction and the first post-processing parameter, at least one of global scaling, adaptive scaling, blur processing, and image degradation is performed on the edge image and the texture image to obtain The edge image to be compressed and the texture image to be compressed, wherein the image to be compressed includes the edge image to be compressed and the texture image to be compressed.
The image processing method according to claim 2, wherein the first instruction includes an image segmentation instruction and a second post-processing instruction, and the first parameter includes a segmentation category corresponding to the image segmentation instruction and The second post-stage processing parameter corresponding to the second post-stage processing instruction;

The step of preprocessing the original image according to the first instruction and the first parameter to obtain the image to be compressed includes:

Performing image segmentation on the original image according to the image segmentation instruction and the segmentation category to obtain multiple image regions;

According to the second post-processing instruction and the second post-processing parameter, at least one of global scaling, adaptive scaling, blur processing, and image degradation is performed on the multiple image regions to obtain each The image area to be compressed corresponds to the image area, wherein the image to be compressed includes a plurality of image areas to be compressed.
The image processing method according to any one of claims 1-8, wherein the deep learning image compression framework includes a first deep neural network, a quantization model, and an entropy coding model;

The step of compressing the image to be compressed using a preset deep learning image compression framework to obtain compressed data includes:

Performing feature extraction on the image to be compressed by using the first deep neural network to obtain image features;

Quantify the image features by using the quantization model to obtain compressed features;

Entropy coding the compression feature by using the entropy coding model to obtain the compressed data.
An image processing method, characterized in that the image processing method includes:

Obtain compressed data, wherein the compressed data is obtained by compressing an image to be compressed using a preset deep learning image compression framework, and the image to be compressed is obtained by preprocessing the original image according to a target strategy, and the target strategy is response The operation on the original image is determined from a plurality of preset strategies, and the code rates of the compressed data corresponding to at least two of the preset strategies are different;

Decompress the compressed data using a preset deep learning image decompression framework to obtain a restored image;

Obtain the reverse strategy corresponding to the target strategy;

Perform the reverse processing of the pre-processing on the restored image according to the reverse strategy to obtain a reconstructed image corresponding to the original image.
The image processing method according to claim 10, wherein the step of obtaining the reverse strategy corresponding to the target strategy comprises:

Obtaining the target strategy, where the target strategy includes a first instruction and a first parameter corresponding to the first instruction;

Determining the second instruction according to the corresponding relationship between the first instruction and the preset instruction;

The second parameter is determined according to the first parameter and a preset parameter calculation rule, wherein the reverse strategy includes the second instruction and the second parameter corresponding to the second instruction.
11. The image processing method according to claim 11, wherein the step of performing reverse processing of the pre-processing on the restored image according to the reverse strategy to obtain a reconstructed image corresponding to the original image ,include:

According to the second instruction and the second parameter, the reverse processing of the pre-processing is performed on the restored image to obtain a reconstructed image corresponding to the original image.
The image processing method according to claim 12, wherein the first instruction includes a global zoom instruction, the first parameter includes a global zoom factor and a zoom kernel; the second instruction includes a global zoom instruction, the The second parameter includes the reciprocal of the global zoom factor and the zoom kernel;

The step of performing the reverse processing of the pre-processing on the restored image according to the second instruction and the second parameter to obtain a reconstructed image corresponding to the original image includes:

According to the global zoom instruction, the reciprocal of the global zoom coefficient, and the zoom kernel, the restored image is globally zoomed to obtain the reconstructed image.
The image processing method according to claim 12, wherein the first instruction includes an adaptive scaling instruction, the first parameter includes a block parameter; the second instruction includes an adaptive scaling instruction, and the second instruction includes an adaptive scaling instruction. The second parameter includes the splicing parameter associated with the block parameter; the restored image includes a plurality of restored image blocks;

The step of performing the reverse processing of the pre-processing on the restored image according to the second instruction and the second parameter to obtain a reconstructed image corresponding to the original image includes:

According to the adaptive scaling instruction and the image characteristics of each restored image block, each restored image block is adaptively scaled to obtain a to-be-reconstructed image block corresponding to each restored image block, wherein, The image feature is used to determine the scaling factor of the restored image block;

The multiple image blocks to be reconstructed are spliced according to the splicing parameters to obtain the reconstructed image.
The image processing method according to claim 12, wherein the first instruction includes a blur processing instruction, the first parameter includes a blur kernel; the second instruction includes a deblur processing instruction, and the second parameter Including a deblurring kernel corresponding to the blurring kernel;

The step of performing the reverse processing of the pre-processing on the restored image according to the second instruction and the second parameter to obtain a reconstructed image corresponding to the original image includes:

According to the deblurring processing instruction and the deblurring kernel, deblurring the restored image is performed to obtain the reconstructed image.
The image processing method according to claim 12, wherein the first instruction includes an image degradation instruction, the first parameter includes an image degradation parameter; the second instruction includes an image enhancement instruction, and the second parameter Including image enhancement parameters;

The step of performing the reverse processing of the pre-processing on the restored image according to the second instruction and the second parameter to obtain a reconstructed image corresponding to the original image includes:

According to the image enhancement instruction and the image enhancement parameter, the image enhancement is performed on the restored image to obtain the reconstructed image.
The image processing method according to claim 12, wherein the first instruction includes an image separation instruction and a first post-processing instruction, and the first parameter includes an image separation parameter and an image separation parameter corresponding to the image separation instruction. A first post-stage processing parameter corresponding to the first post-stage processing instruction;

The second instruction includes a reverse instruction of the image separation instruction and a reverse instruction of the first post-processing instruction, and the second parameter includes the image corresponding to the reverse instruction of the image separation instruction The reverse parameter of the separation parameter and the reverse parameter of the first post-processing parameter corresponding to the reverse instruction of the first post-processing instruction;

The restored image includes a restored edge image and a restored texture image;

The step of performing the reverse processing of the pre-processing on the restored image according to the second instruction and the second parameter to obtain a reconstructed image corresponding to the original image includes:

According to the reverse instruction of the first post-stage processing instruction and the reverse parameter of the first post-stage processing parameter, both the restored edge image and the restored texture image are subjected to the reverse processing of global scaling and adaptive At least one of reverse processing of scaling, deblurring processing, and image enhancement to obtain an edge image to be reconstructed and a texture image to be reconstructed;

According to the reverse instruction of the image separation instruction and the reverse parameter of the image separation parameter, image fusion is performed on the edge image to be reconstructed and the texture image to be reconstructed to obtain the reconstructed image.
The image processing method according to claim 12, wherein the first instruction includes an image segmentation instruction and a second post-processing instruction, and the first parameter includes a segmentation category corresponding to the image segmentation instruction and The second post-stage processing parameter corresponding to the second post-stage processing instruction;

The second instruction includes a reverse instruction of the image segmentation instruction and a reverse instruction of the second post-processing instruction, and the second parameter includes a reverse instruction corresponding to the second post-processing instruction The reverse parameter of the second post-stage processing parameter;

The restored image includes a plurality of restored image areas and the position coordinates of each restored image area;

The step of performing the reverse processing of the pre-processing on the restored image according to the second instruction and the second parameter to obtain a reconstructed image corresponding to the original image includes:

According to the reverse instruction of the second post-processing instruction and the reverse parameter of the second post-processing parameter, the reverse processing of global scaling and the reverse of adaptive scaling are performed on each of the restored image regions. At least one of processing, deblurring, and image enhancement, to obtain an image area to be reconstructed corresponding to each restored image area;

According to the reverse instruction of the image segmentation instruction and the position coordinates of each restored image area, a plurality of image areas to be reconstructed are spliced to obtain the reconstructed image.
The image processing method according to any one of claims 10-18, wherein the deep learning image decompression framework includes a second deep neural network, an inverse quantization model, and an entropy decoding model;

The step of decompressing the compressed data using a preset deep learning image decompression framework to obtain a restored image includes:

Entropy decoding the compressed data using the entropy decoding model to obtain compression features;

Using the inverse quantization model to perform inverse quantization on the compressed features to obtain image features;

The second deep neural network is used to restore the image features to obtain the restored image.
The image processing method according to any one of claims 10-18, wherein the image processing method further comprises:

At least one of a super-resolution algorithm, a de-blurring algorithm, a de-hazing algorithm, and a de-noising algorithm is used to process the reconstructed image to improve the visual effect of the reconstructed image.
An image processing method, characterized in that the image processing method includes:

Get the original image;

In response to the operation on the original image, a target strategy is determined from a plurality of preset strategies, wherein at least two of the preset strategies have different code rates for the compressed data;

Preprocessing the original image according to the target strategy to obtain the image to be compressed;

Compress the image to be compressed by using a preset deep learning image compression framework to obtain the compressed data;

Decompress the compressed data using a preset deep learning image decompression framework to obtain a restored image;

Obtain the reverse strategy corresponding to the target strategy;

Perform the reverse processing of the pre-processing on the restored image according to the reverse strategy to obtain a reconstructed image corresponding to the original image.
An image processing device, characterized in that the image processing device includes:

The image acquisition module is used to acquire the original image;

The response module is configured to determine a target strategy from a plurality of preset strategies in response to an operation on the original image, wherein at least two of the preset strategies have different code rates for the compressed data;

The preprocessing module is used to preprocess the original image according to a preset strategy to obtain the image to be compressed;

The compression module is used to compress the image to be compressed using a preset deep learning image compression framework to obtain the compressed data, wherein the compressed data is used to decompress the preset deep learning image decompression framework A restored image is obtained, and the restored image is used to perform reverse processing of the pre-processing based on the reverse strategy of the preset strategy to obtain a reconstructed image corresponding to the original image.
An image processing device, characterized in that the image processing device includes:

A sequence obtaining module for obtaining compressed data, wherein the compressed data is obtained by compressing an image to be compressed using a preset deep learning image compression framework, and the image to be compressed is obtained by preprocessing the original image according to the target strategy, The target strategy is determined from a plurality of preset strategies in response to an operation on the original image, and the code rates of the compressed data corresponding to at least two of the preset strategies are different;

The decompression module is used to decompress the compressed data using a preset deep learning image decompression framework to obtain a restored image;

The reverse strategy obtaining module is used to obtain the reverse strategy corresponding to the target strategy;

The post-processing module is configured to perform the pre-processing reverse processing on the restored image according to the reverse strategy to obtain a reconstructed image corresponding to the original image.
An image processing device, characterized in that the image processing device includes:

The image acquisition module is used to acquire the original image;

The response module is configured to determine a target strategy from a plurality of preset strategies in response to an operation on the original image, wherein at least two of the preset strategies have different code rates for the compressed data;

The preprocessing module is used to preprocess the original image according to a preset strategy to obtain the image to be compressed;

A compression module, configured to compress the image to be compressed using a preset deep learning image compression framework to obtain the compressed data;

The decompression module uses a preset deep learning image decompression framework to decompress the compressed data to obtain a restored image;

The reverse strategy obtaining module is used to obtain the reverse strategy corresponding to the target strategy;

The post-processing module is configured to perform the pre-processing reverse processing on the restored image according to the reverse strategy to obtain a reconstructed image corresponding to the original image.
An electronic device, characterized in that, the electronic device includes:

One or more processors;

The memory is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors realize any one of claims 1-9 The image processing method, or the image processing method according to any one of claims 10-20, or the image processing method according to claim 21.
A computer-readable storage medium with a computer program stored thereon, wherein the computer program implements the image processing method according to any one of claims 1-9 when executed by a processor, or, as claimed in claim 10. The image processing method according to any one of -20, or the image processing method according to claim 21.