WO2024104504A1

WO2024104504A1 - Image processing method and apparatus, and electronic device and storage medium

Info

Publication number: WO2024104504A1
Application number: PCT/CN2024/072113
Authority: WO
Inventors: 陈方栋; 潘冬萍; 孙煜程; 武晓阳
Original assignee: 杭州海康威视数字技术股份有限公司
Priority date: 2022-11-14
Filing date: 2024-01-12
Publication date: 2024-05-23
Also published as: CN118042163A

Abstract

An image processing method and apparatus, and an electronic device and a storage medium, which relate to the field of image processing, and solve the problem of the image quality of different permission areas in an image being relatively poor after the image is decoded and filtered. The image processing method, which is executed by an electronic device, comprises: dividing an image to be processed into a plurality of image sub-areas, and acquiring the permission level of each image sub-area; and if the permission levels of at least two adjacent image sub-areas among the plurality of image sub-areas are different, performing filtering processing on a connection boundary between the at least two image sub-areas.

Description

Image processing method, device, electronic device and storage medium

Technical Field

The present application relates to the field of image processing, and in particular to an image processing method, device, electronic device and storage medium.

Background technique

In video encoding and decoding, the purpose of filtering is to achieve smooth noise reduction and detail removal within the video image block, and to preserve the image edges to the greatest extent possible. Usually, electronic devices filter the boundaries of multiple areas in the image to preserve the image edges. However, different users have different viewing permissions for different areas in the image. When a low-authority user does not have permission to view a high-authority area in the image, the electronic device will not filter the boundary between the high-authority area and the low-authority area during the video decoding process, resulting in poor image quality.

Summary of the invention

The present application provides an image processing method, device, electronic device and storage medium, which solve the problem that the image quality of different permission areas in the image is poor after decoding and filtering.

This application adopts the following technical solution.

In a first aspect, the present application provides an image processing method, which is executed by an electronic device, and includes: dividing an image to be processed into multiple image sub-regions, and obtaining an authority level for each image sub-region; if the authority levels of at least two adjacent image sub-regions among the multiple image sub-regions are different, filtering the adjacent boundaries between the at least two image sub-regions.

In this embodiment, the electronic device divides the image to be processed into multiple image sub-regions, obtains the authority level of each image sub-region, and judges the authority level of each image sub-region. If the authority levels of at least two adjacent image sub-regions in the multiple image sub-regions are different, it means that the two image sub-regions may still not be filtered after filtering in the decoding process, and the quality of the image to be processed is not high. Therefore, based on the decoding information such as the division of the authority region, the electronic device performs filtering again on the adjacent boundaries between at least two image sub-regions, thereby improving the image quality of different authority regions in the image to be processed.

In some embodiments, the permission levels include a first permission level, a second permission level, and a zero permission level; an image sub-region of the zero permission level is an image sub-region that can be viewed by any user.

In some embodiments, before dividing the image to be processed into multiple image sub-regions and obtaining the permission level of each image sub-region, the process includes: receiving a decoded code stream of the image to be processed, decoding based on the decoded code stream, and obtaining the decoded image to be processed.

In some embodiments, the filtering process includes at least one of: deblocking effect vertical filtering, deblocking effect horizontal filtering, sample adaptive compensation filtering, adaptive loop filtering and neural network filtering.

In some embodiments, there is a vertical boundary between at least two image sub-regions, and the filtering process includes vertical filtering for deblocking effects. The filtering process for the boundary between at least two image sub-regions includes: determining a target area to be filtered on the vertical side of a first sub-region among the at least two image sub-regions; the vertical side is the left side or right side of the boundary between the first sub-region, and the target area to be filtered is any one of the following: an area to be filtered on the right side of the boundary, an area to be filtered on the left side of the boundary, an area to be filtered on the right side of the boundary, and an area to be filtered on the left side of the boundary; and performing vertical filtering for deblocking effects on the target area to be filtered.

In some embodiments, the at least two adjacent image sub-regions further include a second sub-region, and the second sub-region is horizontally adjacent to the first sub-region, where horizontal adjacent means that the second sub-region is located on the left or right side of the first sub-region in the horizontal direction. The at least two adjacent image sub-regions have different authority levels, including: the authority level of the first sub-region is lower than the authority level of the second sub-region; or, the authority level of the first sub-region is higher than the authority level of the second sub-region; or, the authority level of the first sub-region is not equal to the authority level of the second sub-region; or, the authority level of the first sub-region is not equal to the authority level of the second sub-region, and the filtering strength of the adjacent boundaries is greater than or equal to the preset filtering strength.

In some embodiments, there is a horizontal boundary between at least two image sub-regions, and the filtering process includes horizontal filtering for deblocking effects. The filtering process for the boundary between at least two image sub-regions includes: determining a target area to be filtered on the horizontal side of a first sub-region among the at least two image sub-regions; the horizontal side is the upper side or the lower side of the boundary between the first sub-region, and the target area to be filtered is any one of the following: an area to be filtered on the lower side of the boundary, an area to be filtered on the upper side of the boundary, an area to be filtered on the lower side of the boundary, and an area to be filtered on the upper side of the boundary; and performing horizontal filtering for deblocking effects on the target area to be filtered.

In some embodiments, the at least two adjacent image sub-regions further include a third sub-region, and the third sub-region is vertically adjacent to the first sub-region, where vertical adjacent means that the third sub-region is located above or below the first sub-region in the horizontal direction. The at least two adjacent image sub-regions have different authority levels, including: the authority level of the first sub-region is lower than the authority level of the third sub-region; or, the authority level of the first sub-region is higher than the authority level of the third sub-region; or, the authority level of the first sub-region is not equal to the authority level of the third sub-region; or, the authority level of the first sub-region is not equal to the authority level of the third sub-region, and the filtering strength of the adjacent boundaries is greater than or equal to the preset filtering strength.

In some embodiments, the filtering process includes sample adaptive compensation filtering, and the filtering process on the boundary between at least two image sub-regions includes: determining the target area to be filtered from the surrounding side of the first sub-region in the at least two image sub-regions according to the set filtering order; the surrounding side includes: the left side, the upper side, the right side and the lower side of the boundary between the first sub-region and the other sub-regions, the authority level of the first sub-region is different from the authority level of the other sub-regions, and the target area to be filtered is any one of the following: the first sub-region The first side area in the area compared to the connected boundary, the second side area outside the first sub-area compared to the connected boundary, the first side area and the second side area; according to the set filtering order, sample adaptive compensation filtering is performed on the target area to be filtered.

In some embodiments, the filtering process includes adaptive loop filtering, and the above-mentioned filtering process is performed on the adjacent boundary between at least two image sub-regions, including: determining the target area to be filtered from the surrounding side of the first sub-region in the at least two image sub-regions according to the set filtering order; the surrounding side includes, in sequence: the left side, the top side, the right side and the bottom side of the adjacent boundary between the first sub-region and other sub-regions, the authority level of the first sub-region is different from the authority level of other sub-regions, and the target area to be filtered is any one of the following: the first side area within the first sub-region compared to the adjacent boundary, the second side area outside the first sub-region compared to the adjacent boundary, the first side area and the second side area; according to the set filtering order, the target area to be filtered is adaptively loop filtered.

In some embodiments, based on the decoded information of the image to be processed, it is determined whether a first sub-region of at least two image sub-regions needs to be subjected to neural network-based filtering processing; when the first sub-region needs to be subjected to neural network-based filtering processing, the reconstructed pixel value of the first sub-region and the authority level of the first sub-region are input into the neural network; and the filtered reconstructed pixels of the first sub-region are obtained as output.

In some embodiments, when the first sub-region needs to undergo neural network-based filtering processing, the reconstructed pixel value of the first sub-region and the authority level of the first sub-region are input into the neural network, and the permission level of at least one image sub-region adjacent to the first sub-region is input into the neural network.

In some embodiments, the electronic device may obtain information of an image sub-region that is lower than or equal to the user's authority level according to the user's authority level; the information of the image region includes pixel values of the image sub-region.

In some embodiments, if the authority levels of at least two adjacent image sub-regions among the multiple image sub-regions are different, filtering processing is performed on the connecting boundary between the at least two image sub-regions, including: filtering processing of the brightness channel component of the connecting boundary between the at least two image sub-regions; or, filtering processing of the chrominance channel component of the connecting boundary between the at least two image sub-regions; or, filtering processing of the brightness channel component and the chrominance channel component of the connecting boundary between the at least two image sub-regions.

In a second aspect, the present application provides an image processing device, which includes: an image division unit and an image filtering unit; the image division unit is used to divide the image to be processed into multiple image sub-regions; an image decoding unit is used to obtain the authority level of each image sub-region; and the image filtering unit is used to filter the adjacent boundaries between at least two image sub-regions if the authority levels of at least two adjacent image sub-regions among the multiple image sub-regions are different.

In some embodiments, the permission level of each image sub-region is used to indicate: whether each image sub-region is a set permission region, and if each image sub-region is a permission region, the electronic device determines the permission level of each image sub-region; the permission level includes a first permission level and a second permission level.

In some embodiments, the image decoding unit is further used to receive a decoded code stream of the image to be processed, perform decoding based on the decoded code stream, and obtain the decoded image to be processed.

In some embodiments, the image filtering unit is further used to determine a target area to be filtered on the vertical side of a first sub-region among at least two image sub-regions; the vertical side is the left or right side of a connected boundary of the first sub-region, and the target area to be filtered is any one of the following: an area to be filtered on the right side of the connected boundary, an area to be filtered on the left side of the connected boundary, an area to be filtered on the right side of the connected boundary, and an area to be filtered on the left side; the image filtering unit is further used to perform vertical filtering to remove blocking effects on the target area to be filtered.

In some embodiments, the at least two adjacent image sub-regions further include a second sub-region, and the second sub-region is horizontally adjacent to the first sub-region, where horizontal adjacent means that the second sub-region is located on the left or right side of the first sub-region in the horizontal direction. The at least two adjacent image sub-regions have different authority levels, including: the authority level of the first sub-region is lower than the authority level of the second sub-region; or the authority level of the first sub-region is higher than the authority level of the second sub-region; or the authority level of the first sub-region is not equal to the authority level of the second sub-region; or the authority level of the first sub-region is not equal to the authority level of the second sub-region, and the filtering strength of the adjacent boundaries is greater than or equal to the preset filtering strength.

In some embodiments, the image filtering unit is further used to determine a target area to be filtered on the horizontal side of a first sub-region among at least two image sub-regions; the horizontal side is the upper side or the lower side of the adjacent boundary of the first sub-region, and the target area to be filtered is any one of the following: an area to be filtered on the lower side of the adjacent boundary, an area to be filtered on the upper side of the adjacent boundary, an area to be filtered on the lower side of the adjacent boundary, and an area to be filtered on the upper side of the adjacent boundary; the image filtering unit is further used to perform horizontal filtering to remove blocking effects on the target area to be filtered.

In some embodiments, the at least two adjacent image sub-regions further include a third sub-region, and the third sub-region is vertically adjacent to the first sub-region, where vertical adjacent means that the third sub-region is located on the upper side or the lower side of the first sub-region in the horizontal direction. The at least two adjacent image sub-regions have different authority levels, including: the authority level of the first sub-region is lower than the authority level of the third sub-region; or the authority level of the first sub-region is higher than the authority level of the third sub-region; or the authority level of the first sub-region is not equal to the authority level of the third sub-region; or the authority level of the first sub-region is not equal to the authority level of the third sub-region, and the filtering strength of the adjacent boundaries is greater than or equal to the preset filtering strength.

In some embodiments, the image filtering unit is further used to determine the target area to be filtered from the surrounding side of the first sub-region of at least two image sub-regions according to the set filtering order; the surrounding side includes: the left side, the upper side, the right side and the lower side of the boundary between the first sub-region and the other sub-regions, the authority level of the first sub-region is different from the authority level of the other sub-regions, and the target area to be filtered is as follows Any one: the first side area within the first sub-area compared to the connected boundary, the second side area outside the first sub-area compared to the connected boundary, the first side area and the second side area; the image filtering unit is also used to perform sample adaptive compensation filtering on the target area to be filtered according to the set filtering order.

In some embodiments, the image filtering unit is also used to determine the target area to be filtered from the surrounding side of the first sub-region in at least two image sub-regions according to a set filtering order; the surrounding side includes, in sequence: the left side, top side, right side and bottom side of the boundary between the first sub-region and other sub-regions, the authority level of the first sub-region is different from the authority level of other sub-regions, and the target area to be filtered is any one of the following: the first side area in the first sub-region compared to the connected boundary, the second side area outside the first sub-region compared to the connected boundary, the first side area and the second side area; the image filtering unit is also used to perform adaptive loop filtering on the target area to be filtered according to the set filtering order.

In some embodiments, the image filtering unit is also used to determine whether it is necessary to perform neural network-based filtering processing on the first sub-region of at least two image sub-regions based on the decoded information of the image to be processed; when the first sub-region needs to perform neural network-based filtering processing, the reconstructed pixel value of the first sub-region and the authority level of the first sub-region are input into the neural network; and the filtered reconstructed pixels of the first sub-region are obtained as output.

In some embodiments, the image filtering unit is further used to input the authority level of at least one image sub-region adjacent to the first sub-region into the neural network.

In some embodiments, the image filtering unit is further used to filter the brightness channel component of the adjacent boundary between at least two image sub-regions; or, to filter the chrominance channel component of the adjacent boundary between at least two image sub-regions; or, to filter the brightness channel component and the chrominance channel component of the adjacent boundary between at least two image sub-regions.

In a third aspect, the present application provides an electronic device, comprising: a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code includes computer instructions; wherein, when the processor executes the computer instructions, the image processing device executes the image processing method as in the first aspect and any possible implementation thereof.

In a fourth aspect, the present application provides a computer-readable storage medium, which includes: computer software instructions; when the computer software instructions are executed in an image processing device, the image processing device implements the image processing method of the first aspect and any possible implementation method thereof.

In a fifth aspect, the present application provides a computer program product, which, when executed on an image processing device, enables the image processing device to execute the image processing method of the first aspect and any possible implementation thereof. The beneficial effects of the second to fifth aspects can be referred to the corresponding description of the first aspect, and will not be repeated here.

Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of an image processing flow framework provided by the present application;

FIG2 is a schematic diagram of a region for deblocking filtering provided by the present application;

FIG3 is a schematic diagram of a filter block boundary provided by the present application;

FIG4 is a schematic diagram of a filter provided by the present application;

FIG5 is a schematic diagram of the structure of an image processing device provided by the present application;

FIG6 is a schematic diagram of a flow chart of an image processing method provided by the present application;

FIG7 is a schematic diagram of a detection area provided by the present application;

FIG8 is a schematic diagram of a flow chart of another image processing method provided by the present application;

FIG9 is a schematic diagram of a filtering area provided by the present application;

FIG10 is a schematic diagram of another filtering area provided by the present application;

FIG11 is a schematic diagram of a flow chart of another image processing method provided by the present application;

FIG12 is a schematic diagram of another filtering area provided by the present application;

FIG13 is a schematic diagram of a flow chart of another image processing method provided by the present application;

FIG14 is a schematic diagram of another filtering area provided by the present application;

FIG15 is a schematic diagram of a flow chart of another image processing method provided by the present application;

FIG16 is a schematic diagram of another filtering area provided by the present application;

FIG17 is a schematic diagram of a flow chart of another image processing method provided by the present application;

FIG18 is a schematic diagram of a flow chart of another image processing method provided by the present application;

FIG19 is a schematic diagram of the hardware structure of an electronic device provided in this application.

Detailed ways

The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

It should be noted that, in the embodiments of the present application, words such as “exemplarily” or “for example” are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical items or similar items with basically the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order.

In order to enable those skilled in the art to better understand the technical solution provided by the embodiments of the present application, some technical terms involved in the embodiments of the present application and the main processes of video encoding and decoding are briefly described below.

1. Technical terms

1. Deblocking filter (DBF): used to remove the block boundary effect caused by block coding, including vertical deblocking filter and horizontal deblocking filter.

2. Sample adaptive offset (SAO): By classifying the pixel values of the samples and the gradient values of the surrounding blocks, different compensation values are added to the pixel values of each category, so that the reconstructed image is closer to the original image. The basic principle of SAO is to add negative values to compensate the peak pixels in the reconstructed curve and add positive values to supplement the trough pixels. SAO uses the Collect Transfer unit (CTU) as the basic unit, including two major types of compensation: edge compensation (EO) and sideband compensation (BO). In addition, parameter fusion technology is introduced. Sample offset compensation filtering includes basic sample value offset compensation (the above-mentioned SAO), enhanced sample value offset compensation (ESAO), and cross-component sample value offset compensation (CCSAO, only for chroma).

3. Adaptive loop filter (ALF): The reconstructed image is enhanced and filtered through the Wiener filter to make the reconstructed image closer to the original image. The adaptive loop filter can enable the enhanced adaptive correction filter (EALF).

4. Neural Network (NN): A neural network is a computational model consisting of a large number of interconnected nodes (or neurons). In an artificial neural network, a neuron processing unit can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. The types of processing units in the network are divided into three categories: input units, output units, and hidden units. The input unit receives signals and data from the outside world; the output unit realizes the output of the system processing results; the hidden unit is a unit between the input and output units that cannot be observed from outside the system. The connection weights between neurons reflect the connection strength between units, and the representation and processing of information are reflected in the connection relationship of the network processing units. Artificial neural network is a non-programmed, brain-like information processing method. Its essence is to obtain a parallel distributed information processing function through the transformation and dynamic behavior of the network, and imitate the information processing function of the human brain nervous system to varying degrees and levels. At present, in the field of video processing, commonly used neural networks include convolutional neural networks (CNN), recurrent neural networks (RNN), fully connected networks, etc.

2. The main process of video encoding and decoding

As shown in FIG. 1 , taking video coding as an example, video coding generally includes processes such as prediction, transformation, quantization, and entropy coding. Furthermore, the coding process can also be implemented according to the framework in FIG. 1 .

Among them, prediction can be divided into intra-frame prediction and inter-frame prediction. Intra-frame prediction uses the surrounding coded blocks as references to predict the current uncoded block, effectively removing redundancy in the spatial domain. Inter-frame prediction uses the neighboring coded images to predict the current image, effectively removing redundancy in the temporal domain.

Transformation refers to converting an image from the spatial domain to the transform domain, and using transform coefficients to represent the image. Most images contain many flat areas and slowly changing areas. Appropriate transformation can transform the image from a dispersed distribution in the spatial domain to a relatively concentrated distribution in the transform domain, remove the frequency domain correlation between signals, and cooperate with the quantization process to effectively compress the bit stream.

Entropy coding is a lossless coding method that can transform a series of element symbols into a binary code stream for transmission or storage. The input symbols may include quantized transform coefficients, motion vector information, prediction mode information, transform quantization related syntax, etc. Entropy coding can effectively remove the redundancy of video element symbols.

The above is an introduction based on encoding as an example. The process of video decoding is relative to that of video encoding, that is, video decoding usually includes entropy decoding, prediction, inverse quantization, inverse transformation, filtering and other processes. The implementation principles of each process are the same or similar to entropy encoding. Among them, filtering processing includes deblocking filtering, sample adaptive compensation filtering, adaptive loop filtering and filtering based on neural network.

3. DBF Filtering

The implementation of DBF filtering is briefly described below.

DBF filtering processing includes two processes: filtering decision and filtering operation.

The filtering decision includes: 1) obtaining the boundary filter strength (BS value); 2) filtering switch decision; 3) filtering strength selection. For chroma, only step 1) exists, and the BS value of brightness is directly reused. For chroma, filtering is performed only when the BS value is 2 (that is, at least one of the image areas on both sides adopts intra mode).

The filtering operations include: 1) strong filtering and weak filtering for luminance; 2) filtering for chrominance.

The boundary to be filtered that meets one of the following conditions does not need DBF filtering:

Condition 1: If the boundary to be filtered is an image boundary, then the boundary does not need to be filtered.

Condition 2: If the boundary to be filtered is a slice boundary and the cross-slice loop filtering enable flag is 0, then filtering is not required for this boundary.

Condition 3: If the boundary to be filtered is a luminance filtering boundary, and the luminance filtering boundary is not a boundary of a luminance coding block or a luminance transform block, then the boundary does not need to be filtered.

Condition 4: If the boundary to be filtered is a chroma filter boundary, and the chroma filter boundary is not a boundary of a chroma coding block or a chroma transform block, and the luminance filter boundary corresponding to the chroma filter boundary is not a boundary of a luminance coding block, then the boundary does not need to be filtered.

Condition 5: If the boundary to be filtered is a luminance filtering boundary, and the sub-block transform flag of the coding unit where the luminance filtering boundary is located is 1, and the luminance filtering boundary is not a boundary of a luminance coding block, then the boundary does not need to be filtered.

Condition 6: If the string copy intra prediction mode flag of the coding unit where the boundary to be filtered is located is 1, then the boundary does not need to be filtered.

DBF filtering generally performs deblocking vertical filtering and deblocking horizontal filtering in units of 8x8, and DBF filtering filters at most 3 pixels on both sides of the border of the current block (image area to be filtered), and uses at most 4 pixels on both sides of the border for filtering, so vertical/horizontal filtering of different blocks does not affect each other and can be performed in parallel. As shown in Figure 2, for the current 8x8 block, vertical filtering is first performed on the 3 columns on the left side of the current block and the 3 columns on the right side of the left block, and then horizontal filtering is performed on the 3 rows on the upper side of the current block and the 3 rows on the lower side of the upper block.

(I) The derivation process of boundary strength, i.e. BS value, is as follows:

As shown in Fig. 3, a schematic diagram of the vertical side connecting boundary (vertical boundary) and the horizontal side connecting boundary (horizontal boundary) of the current block is shown, and the 8 pixel samples on both sides of the vertical boundary/horizontal boundary are recorded as p ₀ , p ₁ , p ₂ , p ₃ and q ₀ , q ₁ , q ₂ , q ₃ respectively. It should be noted that when calculating the boundary filter strength BS value, the boundary filter strength BS value is calculated for the pixel points of the current block, that is, the boundary is the boundary of the pixel points of the current block, rather than the boundary of the current luminance block or the current chrominance block.

Boundary filter strength determination method 1:

If all of the following conditions are met, the boundary filter strength BS value is equal to 0.

a) The quantization coefficients of the transform blocks of the coding units where p ₀ and q ₀ are located are all 0. If p ₀ (or q ₀ ) is a luma sample and the coding unit where p ₀ (or q ₀ ) is located only contains luma samples, then the coding unit where p ₀ (or q ₀ ) is located refers to the luma coding unit containing p ₀ (or q ₀ ); if p ₀ (or q ₀ ) is a chroma sample and the coding unit where p ₀ (or q ₀ ) is located only contains chroma samples, then the coding unit where p ₀ (or q ₀ ) is located refers to the coding unit containing the luma sample corresponding to p ₀ (or q ₀ ); otherwise (i.e., the coding unit where p ₀ or q ₀ is located contains both luma samples and chroma samples), then the coding unit where p ₀ (or q ₀ ) is located refers to the coding unit containing p ₀ (or q ₀ ).

b) The prediction type of the coding unit where p ₀ and q ₀ are located is not intra frame.

c) Let _BP and _BQ be the 4×4 luminance coding blocks where _p0 and _q0 are located respectively, and the motion information of _BP and _BQ satisfies the following conditions 1 and 2 at the same time or satisfies conditions 3, 4 and 5 at the same time.

1) The L0 reference indexes of the spatial motion information storage units corresponding to _BP and _BQ are both equal to -1, or the reference frames corresponding to the L0 reference indexes of the spatial motion information storage units corresponding to _BP and _BQ are the same frame and the differences of all components of the L0 motion vectors of the spatial motion information storage units are less than an integer pixel.

2) The L1 reference indexes of the spatial motion information storage units corresponding to _BP and _BQ are both equal to -1, or the reference frames corresponding to the L1 reference indexes of the spatial motion information storage units corresponding to _BP and _BQ are the same frame and the differences of all components of the L1 motion vectors of the spatial motion information storage units are less than an integer pixel.

3) Meet one of the following conditions:

The L0 reference index of the spatial motion information storage unit corresponding to _BP is equal to -1 and the L1 reference index of the spatial motion information storage unit corresponding to _BQ is equal to -1.

The reference frame corresponding to the L0 reference index of the spatial motion information storage unit corresponding to _BP and the reference frame corresponding to the L1 reference index of the spatial motion information storage unit corresponding to _BQ are the same frame, and the differences between all components of the L0 motion vector of the spatial motion information storage unit corresponding to _BP and the L1 motion vector of the spatial motion information storage unit corresponding to _BQ are less than an integer pixel.

4) Meet one of the following conditions:

Condition 1: The L0 reference index of the spatial motion information storage unit corresponding to B _Q is equal to -1 and the L1 reference index of the spatial motion information storage unit corresponding to _BP is equal to -1.

Condition 2: The reference frame corresponding to the L0 reference index of the spatial motion information storage unit corresponding to B _Q and the reference frame corresponding to the L1 reference index of the spatial motion information storage unit corresponding to _BP are the same frame and the differences between all components of the L0 motion vector of the spatial motion information storage unit corresponding to B _Q and the L1 motion vector of the spatial motion information storage unit corresponding to _BP are less than an integer pixel.

5) The reference frame corresponding to the L0 reference index of the spatial motion information storage unit corresponding to _BP and the reference frame corresponding to the L0 reference index of the spatial motion information storage unit corresponding to _BQ are not the same frame; the reference frame corresponding to the L1 reference index of the spatial motion information storage unit corresponding to _BP and the reference frame corresponding to the L1 reference index of the spatial motion information storage unit corresponding to _BQ are not the same frame.

Otherwise, the boundary filter strength Bs value is calculated according to the second boundary filter strength determination method.

Boundary filter strength determination method 2:

Step 1: Calculate the average quantization parameter _QPav of the coding unit where _p0 and _q0 are located. If it is a luma sample, the quantization parameter of the luma coding block should be used; if it is a chroma sample, the quantization parameter of the chroma coding block should be used. Let the quantization parameter of the coding unit where _p0 is located be is QP _p , the quantization parameter of the coding unit where q ₀ is located is QP _q , and the average quantization parameter is:

Step 2: Calculate indexes IndexA and IndexB.

Step 3: Look up the table according to IndexA and IndexB to get the values of α' and β' respectively, and then get the values of α and β according to BitDepth.

Exemplarily, “>>” is a right shift operation, which is used to replace division, that is, “>>5” is equivalent to division by 25 (that is, 32). In addition, in the embodiment of the present application, multiplication (that is, “*”) can be replaced by left shift in actual implementation. For example, a multiplied by 4 can be replaced by left shifting 2 bits, that is, by a<<2; a multiplied by 10 can be replaced by (a<<3)+(a<<1).

Exemplarily, “<<” is a left shift operation, which is used to replace multiplication, that is, “a<<2” is equivalent to multiplication by 22 (ie, 4).

For example, when a division operation is implemented by shifting, the operation result is usually directly rounded.

That is, when the calculation result is a non-integer between N and N+1, the result is N. Considering that when the decimal part is greater than 0.5, the accuracy of taking the result as N+1 will be higher, therefore, in order to improve the accuracy of the determined pixel value, when performing the calculation, the numerator of the above weighted sum can be added with 1/2 of the denominator (i.e., the dividend) to achieve the effect of rounding.

Step 4: If DeblockingFilterType is 1 and Abs(p0-q0) is greater than or equal to 4×α, Bs is equal to 0; otherwise, calculate the Bs value as follows.

1) Set the values of fL and fR to 0 and calculate fS.

2) Determine the Bs value based on fS, and the following situations exist.

Case 1: when fS is equal to 6, if Abs(p ₀ -p ₁ ) is less than or equal to β/4 and Abs(q ₀ -q ₁ ) is less than or equal to β/4 and Abs(p ₀ -q ₀ ) is less than α, then the Bs value is equal to 4; otherwise, the Bs value is equal to 3.

Case 2: When fS is equal to 6 and DeblockingFilterType is equal to 1, if Abs(p0-p1) is less than or equal to β/4 and Abs(q0-q1) is less than or equal to β/4 and Abs(p0-p3) is less than or equal to β/2 and Abs(q0-q3) is less than or equal to β/2 and Abs(p0-q0) is less than α, then the Bs value is equal to 4; otherwise, the Bs value is equal to 3.

Case 3: When fS is equal to 5, if _p0 is equal to _p1 and _q0 is equal to _q1 , then the Bs value is equal to 3; otherwise, the Bs value is equal to 2.

Case 4: When fS is equal to 5 and DeblockingFilterType is equal to 1, if p0 is equal to p1 and q0 is equal to q1 and Abs(p2-q2) is less than α, then the Bs value is equal to 3; otherwise, the Bs value is equal to 2.

Case 5: When fS is equal to 4, if fL is equal to 2, the Bs value is equal to 2; otherwise, the Bs value is equal to 1.

Case 6: When fS is equal to 3, if Abs(p ₁ -q ₁ ) is less than β, the Bs value is equal to 1; otherwise, the Bs value is equal to 0.

Case 7: When fS is other value, Bs value is equal to 0.

3) If the Bs value obtained in step 2) is not equal to 0 and the filtered boundary is a chroma coding block boundary, the Bs value is reduced by 1.

(II) Determine the deblocking filter adjustment parameters

Deblocking filter adjustment parameters DbrThresold, DbrOffset0, DbrOffset1, DbrAltOffset0 and DbrAltOffset1 are determined.

If the current boundary is a vertical boundary and PictureDbrVEnableFlag is 1, or if the current boundary is a horizontal boundary and PictureDbrHEnableFlag is 1, the value of PictureDbrEnableFlag is 1; otherwise, the value of PictureDbrEnableFlag is 0.

If the current boundary is a vertical boundary and PictureAltDbrVEnableFlag is 1, or if the current boundary is a horizontal boundary and PictureAltDbrHEnableFlag is 1, the value of PictureAltDbrEnableFlag is 1; otherwise, the value of PictureAltDbrEnableFlag is 0.

For vertical borders:

For horizontal borders:

(III) Boundary filtering process when the brightness component Bs is equal to 4

When the value of the boundary filter strength Bs is 4, the calculation process of filtering _p0 , _p1 , _p2 and _q0 , _q1 , _q2 is as follows ( _P0 , _P1 , _P2 and _Q0 , _Q1 , _Q2 are the values after filtering):

If the value of PictureDbrEnableFlag is 1, the values of P ₀ , P ₁ , P ₂ , Q ₀ , Q ₁ and Q ₂ are adjusted according to the following “(IX) Deblocking filter adjustment process”.

(IV) Boundary filtering process when the brightness component Bs is equal to 3

When the value of the boundary filter strength Bs is 3, the calculation process of filtering _p0 , _p1 and _q0 , _q1 is as follows ( _P0 , _P1 and _Q0 , _Q1 are the values after filtering):

If the value of PictureDbrEnableFlag is 1, the values of P ₀ , P ₁ , Q ₀ and Q ₁ are adjusted according to the following “(IX) Deblocking filter adjustment process”.

(V) Boundary filtering process when the brightness component Bs is equal to 2

When the value of the boundary filter strength Bs is 2, the calculation process for filtering _p0 and _q0 is as follows ( _P0 and _Q0 are the values after filtering):

If the value of PictureDbrEnableFlag is 1, adjust the values of _P0 and _Q0 according to the following "(IX) Deblocking filter adjustment process".

(VI) Boundary filtering process when the brightness component Bs is equal to 1

When the value of the boundary filter strength Bs is 1, the calculation process of the p ₀ and q ₀ filtering is as follows (P ₀ and Q ₀ are the values after filtering):

(VII) Boundary filtering process when the brightness component Bs is equal to 0

When the value of the boundary filter strength Bs is 0 and the value of PictureAltDbrEnableFlag is 1, the calculation process for the filter adjustment of p ₀ and q ₀ is as follows (P ₀ and Q ₀ are the values after the filter adjustment):

(VIII) Boundary filtering process when the chrominance component Bs is greater than 0

When the value of the boundary filter strength Bs is greater than 0, the calculation process of the p ₀ and q ₀ filtering is as follows (P ₀ and Q ₀ are the values after filtering):

When the value of the boundary filter strength Bs is equal to 3, the calculation process of the _p1 and _q1 filtering is as follows ( _P1 and _Q1 are the filtered values):

(IX) Deblocking filter adjustment process

Adjust the values of _Pi and _Qi (i can be 0, 1 or 2):

Clip1(x) means to limit x to the range [0,2^(bit_depth)-1] (including 0 and 2^(bit_depth)-1). bit_depth represents the bit depth of the image, which is usually 8, 10, 12, etc.

4. SAO Filter

SAO filtering is used to eliminate the ringing effect. The ringing effect is caused by the quantization distortion of the high-frequency AC coefficient. After decoding, ripples will appear around the edge. The larger the transform block size, the more obvious the ringing effect. The basic principle of SAO is to add negative values to compensate the peak pixels in the reconstructed curve and add positive values to supplement the trough pixels. SAO uses the Collect Transfer unit (CTU) as the basic unit, including two types of compensation forms: edge offset (EO) and sideband offset (BO). In addition, parameter fusion technology is introduced.

The sample offset compensation filtering includes basic sample offset compensation (the above-mentioned SAO), enhanced sample offset compensation (the above-mentioned ESAO), and cross-component sample offset compensation (the above-mentioned CCSAO, which is performed only for chrominance).

The basic process of SAO:

If the current component of the current block enables SAO, the basic sample value offset compensation unit is first derived, and then the basic sample value offset compensation information corresponding to the current basic sample value offset compensation unit is derived, and finally the components of each sample in the current basic sample value offset compensation unit are operated to obtain the offset sample value. Otherwise, the value of the corresponding component of the filtered sample is directly used as the value of the sample component after offset.

The basic process of ESAO (ESAO and SAO are mutually exclusive, that is, if SAO is enabled, ESAO is not enabled):

If ESAO is enabled for the current component of the current block, the enhanced sample value offset compensation unit is first derived, and then each component of each sample in the enhanced sample value offset compensation unit is operated to obtain the offset sample value; otherwise, the value of the corresponding component of the filtered sample is directly used as the value of the sample component after offset.

The basic process of CCSAO:

If CCSAO is enabled for the current component of the current block, the cross-component sample value offset compensation unit is first derived, and then the component samples in the current cross-component sample value offset compensation unit are operated to obtain the cross-component offset sample value. Otherwise, the value of the component corresponding to the offset sample is directly used as the value of the sample component after the cross-component offset.

For the chroma pixel that needs to be CCSAO, the classification of the current chroma pixel is determined based on the luminance value (the above-mentioned Y5) after deblocking filtering (adjustment) at the corresponding position and the current chroma pixel (the pixel after ESAO). Based on the classification, the compensation value of the current chroma pixel is determined, and the current chroma pixel is added with the compensation value to obtain the chroma pixel value after CCSAO filtering.

5. ALF Filtering

ALF filtering is the optimal filter in the mean square sense calculated based on the original signal and the distorted signal, that is, the Wiener filter. The adaptive correction filter (ALF) can enable the enhanced adaptive correction filter (EALF) or not. Positive filtering.

The basic process of ALF is as follows:

If the current component of the current image does not enable ALF, the value of the offset sample component is directly used as the value of the corresponding reconstructed sample component; otherwise, adaptive correction filtering is performed on the corresponding offset sample component.

The unit of the adaptive correction filter is an adaptive correction filter unit derived from the maximum coding unit, which is processed in sequence according to the raster scanning order. First, the adaptive correction filter coefficients of each component are decoded, then the adaptive correction filter unit is derived, and the adaptive correction filter coefficient index of the brightness component of the current adaptive correction filter unit is determined. Finally, the brightness and chrominance components of the adaptive correction filter unit are adaptively corrected to obtain the reconstructed sample.

If EALF is enabled for the current image, each group of filter coefficients of the ALF of the current image has 15, and the shape is shown in (a) of Figure 4. The maximum number of adaptive correction filters for the brightness component of the current image is 64. Otherwise, each group of filter coefficients of the ALF of the current image has 9, and the shape is shown in (b) of Figure 4. The maximum number of adaptive correction filters for the brightness component of the current image is 16.

Regardless of DBF filtering, SAO filtering or ALF filtering, they are all classified based on the current pixel value of the adjacent boundary to be filtered, or the relationship between the pixel value of the current block and the pixel value of the adjacent blocks on the surrounding side of the current block (such as the level of authority), and then different filtering operations are performed based on different categories.

As mentioned above, in video coding and decoding, the purpose of filtering is to achieve smooth noise reduction and detail removal within a video image block, and to retain the image edge to the greatest extent. Generally, electronic devices filter the boundaries of multiple regions in an image to retain the image edge.

In order to protect the data security of users, general images can be set with permission areas (for example, high permission level areas, low permission level areas) and non-permission areas (zero permission level areas), and the permission areas can include multiple permission areas of different high and low levels. A permission area refers to an area where only users with relevant permissions can correctly decode the image content of the area. Exemplarily, only users with high permissions can correctly view image areas of any permission level, while users with only low permissions cannot view image areas of higher permission levels, and can only view image areas of low permission levels and image areas of zero permission level areas.

Therefore, since different users have different viewing permissions for different areas in the image, when a low-authority user does not have viewing permissions for high-authority areas in the image, the electronic device will not filter the adjacent boundaries of image areas with different permission levels during the video decoding process, resulting in poor image quality.

To address this problem, the embodiments of the present application provide an image processing method, device, electronic device, and storage medium. The method divides the image to be processed into multiple image sub-regions through an electronic device, obtains the authority level of each image sub-region, and judges the authority level of each image sub-region. If the authority levels of at least two adjacent image sub-regions in the multiple image sub-regions are different, it means that the above two image sub-regions may still not be filtered after filtering in the decoding process, and the quality of the image to be processed is not high. Therefore, based on the decoding information such as the division of the authority region, the electronic device can filter the adjacent boundaries between at least two image sub-regions again, which can improve the image quality of different authority regions in the image to be processed.

The image processing method provided in the embodiment of the present application may be applied to an image processing device 11 as shown in FIG. 5 . The image processing device 11 includes an image dividing unit 101 , an image filtering unit 102 , and an image decoding unit 103 .

The image division unit 101 is used to divide the image to be processed into a plurality of image sub-regions.

The image decoding unit 103 is used to obtain the permission level of each image sub-region.

The image filtering unit 102 is configured to perform filtering processing on a boundary between at least two adjacent image sub-regions among the plurality of image sub-regions if the authority levels of at least two adjacent image sub-regions are different.

The image decoding unit 103 is further configured to receive a decoded code stream of the image to be processed, perform decoding based on the decoded code stream, and obtain a decoded image to be processed.

In some embodiments, the image filtering unit 102 is further used to determine a target area to be filtered on the vertical side of a first sub-region among at least two image sub-regions; the vertical side is the left or right side of a connected boundary of the first sub-region, and the target area to be filtered is any one of the following: an area to be filtered on the right side of the connected boundary, an area to be filtered on the left side of the connected boundary, an area to be filtered on the right side of the connected boundary, and an area to be filtered on the left side; the image filtering unit 102 is further used to perform vertical filtering to remove blocking effects on the target area to be filtered.

In some embodiments, the image filtering unit 102 is further used to determine a target area to be filtered on the horizontal side of a first sub-region among at least two image sub-regions; the horizontal side is the upper side or the lower side of the adjacent boundary of the first sub-region, and the target area to be filtered is any one of the following: an area to be filtered on the lower side of the adjacent boundary, an area to be filtered on the upper side of the adjacent boundary, an area to be filtered on the lower side of the adjacent boundary, and an area to be filtered on the upper side; the image filtering unit 102 is further used to perform horizontal filtering to remove blocking effects on the target area to be filtered.

In some embodiments, the image filtering unit 102 is further used to determine the target area to be filtered from the surrounding side of the first sub-region of the at least two image sub-regions according to the set filtering order; the surrounding side includes: the left side, the upper side, the right side and the lower side of the boundary between the first sub-region and the other sub-regions, the authority level of the first sub-region is different from the authority level of the other sub-regions, and the target area to be filtered is any one of the following: the first side area in the first sub-region compared to the connected boundary, the second side area outside the first sub-region compared to the connected boundary, the first side area and the second side area; the image filtering unit 102 is further used to filter the target area to be filtered according to the set filtering order. The filter area performs sample adaptive compensation filtering.

In some embodiments, the image filtering unit 102 is also used to determine the target area to be filtered from the surrounding side of the first sub-region in at least two image sub-regions according to a set filtering order; the surrounding side includes, in sequence: the left side, top side, right side and bottom side of the boundary between the first sub-region and other sub-regions, the authority level of the first sub-region is different from the authority level of other sub-regions, and the target area to be filtered is any one of the following: the first side area in the first sub-region compared to the connected boundary, the second side area outside the first sub-region compared to the connected boundary, the first side area and the second side area; the image filtering unit 102 is also used to perform adaptive loop filtering on the target area to be filtered according to the set filtering order.

In some embodiments, the image filtering unit 102 is also used to determine whether it is necessary to perform neural network-based filtering processing on the first sub-region of at least two image sub-regions based on the decoded information of the image to be processed; when the first sub-region needs to perform neural network-based filtering processing, the reconstructed pixel value of the first sub-region and the authority level of the first sub-region are input into the neural network; and the filtered reconstructed pixels of the first sub-region are obtained as output.

In some embodiments, the image filtering unit 102 is further configured to input the permission level of at least one image sub-region adjacent to the first sub-region into the neural network.

In some embodiments, the image filtering unit 102 is further used to filter the brightness channel component of the adjacent boundary between at least two image sub-regions; or, to filter the brightness channel component of the adjacent boundary between at least two image sub-regions; or, to filter the brightness channel component and the chroma channel component of the adjacent boundary between at least two image sub-regions.

FIG6 is a flowchart of an image processing method provided in an embodiment of the present application. Exemplarily, the image processing method provided in the present application can be applied to the image processing device shown in FIG5. As shown in FIG6, the image processing method provided in the present application can specifically include the following steps:

S101. Divide an image to be processed into a plurality of image sub-regions, and obtain the permission level of each image sub-region.

The permission levels are divided into at least two different permission levels.

In some embodiments, an image may be provided with a permission area and a non-permission area (zero permission level area). The image sub-area of the non-permission area (zero permission level area) is an image sub-area that any user can view. Among them, the permission area may include multiple permission areas of different high and low levels. For example, the permission area may include a first permission level area (high permission level area), a second permission level area (medium permission level area), and a third permission level area (low permission level area). The permission level of the first permission level area is higher than the permission level of the second permission level area, and the permission level of the second permission level area is higher than the permission level of the third permission level area. Each permission level may also be divided into multiple sub-permission levels, which is not specifically limited in this embodiment.

In some embodiments, the permission level of the target area in the image is configured according to the category of the target area. For example, the face of a person in the image is a high permission level area, the upper body is a low permission level area, and the lower body is a zero permission level area; the license plate part of the vehicle image is a high permission level area, the window part is a low permission level area, and the body part is a zero permission level area, etc.

In some embodiments, the permission levels of image sub-regions in the image to be processed can be divided into high permission level areas, low permission level areas and zero permission level areas. For example, as shown in Figure 7, multiple image sub-regions may include low permission level areas, high permission level areas and zero permission level areas.

It is understandable that different users have different viewing permissions for different areas in the image. High-authority users can view image sub-areas at any permission level (for example, high-authority level, low-authority level areas, and zero-authority level areas), low-authority users can view image sub-areas at low-authority levels and image sub-areas at zero-authority levels, and unauthorized users can only view image sub-areas at zero-authority levels.

In some embodiments, before dividing the image to be processed into a plurality of image sub-regions, the image decoding unit 103 is used to receive a decoded code stream of the image to be processed, and decoding is performed based on the decoded code stream to obtain the decoded image to be processed.

In other embodiments, when the image to be processed includes a target image region, when the target image region is used as a filtering processing object, the image decoding unit 103 receives a decoded code stream of the image to be processed, and decodes the target image region based on the decoded code stream. After the target image region is decoded, it is not necessary to wait for the entire image to be processed to be decoded, and the target image region can be directly divided into multiple image sub-regions to obtain the permission level of each image sub-region.

In other embodiments, when the image to be processed includes a target image area, when the target image area is used as a filtering processing object, the target image area can be taken as the center and expanded to the surrounding side (left side, right side, upper side or lower side) to obtain a detection area with a width of CW and a height of CH, and the above detection area is divided to obtain multiple NxN (N is preferably 2 or 4 or 8) pixel units of image sub-block areas, wherein the authority level in each image sub-block area is the same. Among them, the expansion pixel units of the detection area in the four directions can be different or completely the same.

Exemplarily, as shown in (a) of FIG. 8 , the detection area may be centered on the target image area and extended to the left and upward by T (T is preferably 4, 8, or 16) unit pixels.

Exemplarily, as shown in (b) of FIG8 , the detection area may also be increased by an upper right area on the detection area of (a) of FIG8 as the overall detection area.

S102: If the authority levels of at least two adjacent image sub-regions among the plurality of image sub-regions are different, filtering is performed on a boundary (boundary to be filtered) between the at least two image sub-regions.

In some embodiments, when the image to be processed includes a target image area that is a filtering object, after the target image area is decoded using the image decoding unit 103, the target image area can be directly divided into multiple image sub-regions. If the authority levels of at least two adjacent image sub-regions among the multiple image sub-regions are different, the connecting boundary between the two image sub-regions can be directly filtered without waiting for the entire image to be processed to be decoded before filtering.

In some embodiments, the at least two adjacent image sub-regions may be two adjacent image sub-regions, or may be two adjacent image sub-regions that are not adjacent but are relatively close to each other.

In some embodiments, the boundary between at least two image sub-regions may be filtered for the luminance channel component; or, the boundary between at least two image sub-regions may be filtered for the chrominance channel component; or, the boundary between at least two image sub-regions may be filtered for both the luminance channel component and the chrominance channel component.

In some embodiments, the image processing device can obtain image area information of any permission level in the image to be processed, as well as image area information of non-authorization areas; or, the image processing device can obtain image area information of the second permission level in the image to be processed, as well as image area information of non-authorization areas.

It is understandable that the image to be processed is divided into multiple image sub-regions, and the authority level of each image sub-region is obtained, so as to judge the authority level of each image sub-region. If the authority levels of at least two adjacent image sub-regions in the multiple image sub-regions are different, it means that the two image sub-regions may still not be filtered after filtering in the decoding process, and the quality of the image to be processed is not high. Therefore, based on the decoding information such as the division of the authority region, the boundary between at least two image sub-regions is filtered again, which can improve the image quality of different authority regions in the image to be processed.

In some embodiments, if there is a vertically connected boundary between the at least two image sub-regions (i.e., the positional relationship between the two image sub-regions is horizontally connected), the filtering process includes deblocking effect vertical filtering, and if the DBF filtering conditions are met, the deblocking effect vertical filtering can be performed on the boundary between the at least two image sub-regions. Please refer to Figure 9. The decoding method provided in this embodiment includes the following steps Sa1 to Sa2.

Sa1. Determine a target area to be filtered on a vertical side of a first sub-region in at least two image sub-regions.

Among them, the vertical side is the left or right side of the boundary of the first sub-region, and the target area to be filtered is any one of the following: the area to be filtered on the right side of the boundary, the area to be filtered on the left side of the boundary, the area to be filtered on the right side of the boundary and the area to be filtered on the left side.

Sa2. Perform vertical filtering to remove the blocking effect on the target area to be filtered.

In some embodiments, the at least two adjacent image sub-regions further include a second sub-region, and the second sub-region is horizontally adjacent to the first sub-region, where horizontal adjacent means that the second sub-region is located on the left or right side of the first sub-region in the horizontal direction. The permission levels of the at least two adjacent image sub-regions are different, including any of the following situations:

1. The authority level of the first sub-area is lower than the authority level of the second sub-area.

2. The authority level of the first sub-area is higher than the authority level of the second sub-area.

3. The permission level of the first sub-area is not equal to the permission level of the second sub-area.

4. The authority level of the first sub-area is not equal to the authority level of the second sub-area, and the filtering strength (BS value) of the adjacent boundaries is greater than or equal to the preset filtering strength.

As shown in (a) of FIG. 10 , when the second sub-region is located on the left side of the first sub-region, a deblocking vertical filtering is performed on the target area to be filtered according to the different authority levels of the at least two adjacent image sub-regions, and a detailed description is given.

Embodiment 1: When the authority level of the first sub-region is lower than the authority level of the second sub-region, the target area to be filtered is the area to be filtered on the right side of the adjacent boundary (within the first sub-region), and the target area to be filtered is subjected to vertical filtering for deblocking effect. The area to be filtered on the right side is a pixel point area with a width of n columns to the right of the adjacent boundary, and n is preferably 1.

Embodiment 2: When the authority level of the first sub-region is higher than the authority level of the second sub-region, the target area to be filtered is the area to be filtered on the right side of the adjacent boundary (within the first sub-region), and the target area to be filtered is subjected to vertical filtering for deblocking effect. The area to be filtered on the right side is a pixel point area with a width of n columns to the right of the adjacent boundary, and n is preferably 1.

Embodiment 3: When the authority level of the first sub-region is not lower than the authority level of the second sub-region, the target area to be filtered is the right side area to be filtered (within the first sub-region) of the adjacent boundary, and the target area to be filtered is subjected to vertical filtering for deblocking effect. The right side area to be filtered is a pixel point area with a width of n columns to the right of the adjacent boundary, and n is preferably 1.

Embodiment 4: When the authority level of the first sub-region is not equal to the authority level of the second sub-region, and the BS value of the adjacent boundary is greater than or equal to 1, the target area to be filtered is the area to be filtered on the right side of the adjacent boundary (within the first sub-region), and the target area to be filtered is subjected to vertical filtering for deblocking effect. Among them, the area to be filtered on the right side is a pixel point area with a width of n columns to the right of the adjacent boundary, and n is preferably 1.

As shown in (b) of FIG10 , in the first to fourth embodiments, the target area to be filtered is the right side of the adjacent boundary. The side area to be filtered (within the first sub-area).

Embodiment 5: When the authority level of the first sub-region is lower than the authority level of the second sub-region, the target area to be filtered is the area to be filtered on the left side of the adjacent boundary (within the second sub-region), and the target area to be filtered is subjected to vertical filtering for deblocking effect. The area to be filtered on the left side is a pixel point area with a width of n columns to the left of the adjacent boundary, and n is preferably 1.

Embodiment 6: When the authority level of the first sub-region is higher than the authority level of the second sub-region, the target area to be filtered is the area to be filtered on the left side of the adjacent boundary (within the second sub-region), and the target area to be filtered is subjected to vertical filtering for deblocking effect. The area to be filtered on the left side is a pixel point area with a width of n columns to the left of the adjacent boundary, and n is preferably 1.

Embodiment 7: When the authority level of the first sub-region is not equal to the authority level of the second sub-region, the target area to be filtered is the area to be filtered on the left side of the adjacent boundary (within the second sub-region), and the target area to be filtered is subjected to vertical filtering for deblocking effect. The area to be filtered on the left side is a pixel point area with a width of n columns to the left of the adjacent boundary, and n is preferably 1.

Embodiment 8: When the authority level of the first sub-region is not equal to the authority level of the second sub-region, and the BS value of the adjacent boundary is greater than or equal to 1, the target area to be filtered is the area to be filtered on the left side of the adjacent boundary (within the second sub-region), and the target area to be filtered is subjected to vertical filtering for deblocking effect. Among them, the area to be filtered on the left side is a pixel point area with a width of n columns to the left of the adjacent boundary, and n is preferably 1.

As shown in (c) of FIG. 10 , in the fifth to eighth embodiments, the target region to be filtered is the region to be filtered on the left side of the adjacent boundary (within the second sub-region).

Embodiment 9: When the authority level of the first sub-region is lower than the authority level of the second sub-region, the target area to be filtered is the right area to be filtered and the left area to be filtered (within the first region and the second sub-region) of the adjacent boundary, and the target area to be filtered is subjected to vertical filtering for deblocking effect. The target area to be filtered is a pixel point area with a width of 2n, which is n columns to the right and n sides to the left of the adjacent boundary, and n is preferably 1.

Embodiment 10: When the authority level of the first sub-region is higher than the authority level of the second sub-region, the target area to be filtered is the right area to be filtered and the left area to be filtered (within the first region and the second sub-region) of the adjacent boundary, and the target area to be filtered is subjected to vertical filtering for deblocking effect. The target area to be filtered is a pixel point area with a width of 2n, which is n columns to the right and n sides to the left of the adjacent boundary, and n is preferably 1.

Embodiment 11: When the authority level of the first sub-region is not equal to the authority level of the second sub-region, the target area to be filtered is the right area to be filtered and the left area to be filtered (within the first region and the second sub-region) of the adjacent boundary, and the target area to be filtered is subjected to vertical filtering for deblocking effect. The target area to be filtered is a pixel point area with a width of 2n, which is n columns to the right and n sides to the left of the adjacent boundary, and n is preferably 1.

Embodiment 12: When the authority level of the first sub-region is not equal to the authority level of the second sub-region, and the BS value of the adjacent boundary is greater than or equal to 1, the target area to be filtered is the area to be filtered on the right side and the area to be filtered on the left side of the adjacent boundary (within the first region and the second sub-region), and the target area to be filtered is subjected to vertical filtering for deblocking effect. The target area to be filtered is a pixel area with a width of 2n, which is n columns to the right and n sides to the left of the adjacent boundary, and n is preferably 1.

As shown in (d) of FIG. 10 , in Embodiments 9 to 12, the target region to be filtered is the right region to be filtered and the left region to be filtered (within the first region and the second sub-region) of the adjacent boundaries.

It is understandable that if there is a vertical boundary between the at least two image sub-regions, and the filtering process includes vertical filtering for deblocking effects, the target area to be filtered on the vertical side of the first sub-region in the at least two image sub-regions can be determined, and the target area to be filtered can be vertically filtered for deblocking effects. Among them, based on the authority levels of the two image sub-regions, the corresponding target area to be filtered can be selected for vertical filtering for deblocking effects. In the case where the image processing device obtains the information of the corresponding image area according to the authority level of the user, the method can meet the needs of users of any authority level to perform vertical filtering for deblocking effects on the decoded image again, improve the image quality of different authority areas in the image to be processed, and thus improve the user experience at each authority level.

In some embodiments, if there is a horizontal boundary between the at least two image sub-regions (i.e., the positional relationship between the two image sub-regions is vertically connected), the filtering process includes deblocking effect horizontal filtering, and when the DBF filtering conditions are met, the deblocking effect horizontal filtering is performed on the boundary between the at least two image sub-regions. Please refer to Figure 11. The decoding method provided in this embodiment includes the following steps Sb1 to Sb2.

Sb1. Determine a target area to be filtered on the horizontal side of a first sub-region in at least two image sub-regions.

Among them, the horizontal side is the upper side or lower side of the boundary of the first sub-region, and the target area to be filtered is any one of the following: the upper side area to be filtered of the connected boundary, the lower side area to be filtered of the connected boundary, the upper side area to be filtered of the connected boundary and the lower side area to be filtered.

Sb2. Perform deblocking horizontal filtering on the target area to be filtered.

In some embodiments, the at least two adjacent image sub-regions further include a third sub-region, and the third sub-region is vertically adjacent to the first sub-region, where vertically adjacent means that the third sub-region is located above or below the first sub-region in the horizontal direction. The permission levels of the at least two adjacent image sub-regions are different, including any of the following situations:

1. The authority level of the first sub-area is lower than the authority level of the third sub-area.

2. The authority level of the first sub-area is higher than the authority level of the third sub-area.

3. The permission level of the first sub-area is not equal to the permission level of the third sub-area.

4. The authority level of the first sub-area is not equal to the authority level of the third sub-area, and the filtering strength (BS value) of the adjacent boundaries is greater than or equal to the preset filtering strength.

As shown in (a) of FIG. 12 , the third sub-region is located on the upper side of the first sub-region. The following describes in detail the deblocking horizontal filtering of the target area to be filtered according to the different authority levels of the at least two adjacent image sub-regions.

Embodiment 13: When the authority level of the first sub-region is lower than the authority level of the third sub-region, the target area to be filtered is the lower area to be filtered of the adjacent boundary (within the first sub-region), and the target area to be filtered is subjected to horizontal filtering for deblocking effect. The lower area to be filtered is a pixel point area with a width of n rows upward from the adjacent boundary, and n is preferably 1.

Embodiment 14: When the authority level of the first sub-region is higher than the authority level of the third sub-region, the target area to be filtered is the lower area to be filtered of the adjacent boundary (within the first sub-region), and the target area to be filtered is subjected to horizontal filtering for deblocking effect. The lower area to be filtered is a pixel point area with a width of n rows upward from the adjacent boundary, and n is preferably 1.

Embodiment 15: When the authority level of the first sub-region is not lower than the authority level of the third sub-region, the target area to be filtered is the lower area to be filtered of the adjacent boundary (within the first sub-region), and the target area to be filtered is subjected to horizontal filtering for deblocking effect. The lower area to be filtered is a pixel point area with a width of n rows upward from the adjacent boundary, and n is preferably 1.

Embodiment 16: When the authority level of the first sub-region is not equal to the authority level of the third sub-region, and the BS value of the adjacent boundary is greater than or equal to 1, the target area to be filtered is the lower area to be filtered of the adjacent boundary (within the first sub-region), and the target area to be filtered is subjected to horizontal filtering for deblocking effect. The lower area to be filtered is a pixel point area with a width of n rows upward from the adjacent boundary, and n is preferably 1.

As shown in (b) of FIG. 12 , in Embodiment 13 to Embodiment 16 , the target region to be filtered is the region to be filtered at the lower side of the adjacent boundary (within the first sub-region).

Embodiment 17: When the authority level of the first sub-region is lower than the authority level of the third sub-region, the target area to be filtered is the upper area to be filtered of the adjacent boundary (within the third sub-region), and the target area to be filtered is subjected to horizontal filtering for deblocking effect. The upper area to be filtered is a pixel point area with a width of n rows below the adjacent boundary, and n is preferably 1.

Embodiment 18: When the authority level of the first sub-region is higher than the authority level of the third sub-region, the target area to be filtered is the upper area to be filtered of the adjacent boundary (within the third sub-region), and the target area to be filtered is subjected to horizontal filtering for deblocking effect. The upper area to be filtered is a pixel point area with a width of n rows below the adjacent boundary, and n is preferably 1.

Embodiment 19: When the authority level of the first sub-region is not equal to the authority level of the third sub-region, the target area to be filtered is the upper area to be filtered of the adjacent boundary (within the third sub-region), and the target area to be filtered is subjected to horizontal filtering for deblocking effect. The upper area to be filtered is a pixel point area with a width of n rows below the adjacent boundary, and n is preferably 1.

Embodiment 20: When the authority level of the first sub-region is not equal to the authority level of the third sub-region, and the BS value of the adjacent boundary is greater than or equal to 1, the target area to be filtered is the upper area to be filtered of the adjacent boundary (within the third sub-region), and the target area to be filtered is subjected to horizontal filtering for deblocking effect. The upper area to be filtered is a pixel point area with a width of n rows below the adjacent boundary, and n is preferably 1.

As shown in (c) of FIG. 12 , in Embodiments 17 to 20, the target region to be filtered is the region to be filtered on the upper side of the adjacent boundary (within the third sub-region).

Embodiment 21: When the authority level of the first sub-region is lower than the authority level of the third sub-region, the target area to be filtered is the lower area to be filtered and the upper area to be filtered (in the first region and the third sub-region) of the adjacent boundary, and the target area to be filtered is subjected to horizontal filtering for deblocking effect. The target area to be filtered is the pixel point area with a width of 2n, which is n rows upward and n sides downward of the adjacent boundary, and n is preferably 1.

Embodiment 22: When the authority level of the first sub-region is higher than the authority level of the third sub-region, the target area to be filtered is the lower area to be filtered and the upper area to be filtered (in the first region and the third sub-region) of the adjacent boundary, and the target area to be filtered is subjected to horizontal filtering for deblocking effect. The target area to be filtered is the pixel point area with a width of 2n, which is n rows upward and n sides downward of the adjacent boundary, and n is preferably 1.

Embodiment 23: When the authority level of the first sub-region is not equal to the authority level of the third sub-region, the target area to be filtered is the lower area to be filtered and the upper area to be filtered (in the first region and the third sub-region) of the adjacent boundary, and the target area to be filtered is subjected to horizontal filtering for deblocking effect. The target area to be filtered is the pixel point area with a width of 2n, which is n rows upward and n sides downward of the adjacent boundary, and n is preferably 1.

Embodiment 24: When the authority level of the first sub-region is not equal to the authority level of the third sub-region, and the BS value of the adjacent boundary is greater than or equal to 1, the target area to be filtered is the lower area to be filtered and the upper area to be filtered (in the first region and the third sub-region) of the adjacent boundary, and the target area to be filtered is subjected to horizontal filtering for deblocking effect. The target area to be filtered is the pixel point area with a width of 2n, which is n rows upward and n sides downward of the adjacent boundary, and n is preferably 1.

Among them, as shown in (d) of Figure 12, in Examples 21 to 24, the target area to be filtered is the upper area to be filtered and the lower area to be filtered of the adjacent boundaries (inside the first area and in the third sub-area).

It is understandable that if there is a horizontal boundary between the at least two image sub-regions mentioned above, and the filtering process includes horizontal filtering for deblocking effects, the target area to be filtered on the horizontal side of the first sub-region in the at least two image sub-regions can be determined, and the target area to be filtered can be subjected to horizontal filtering for deblocking effects. Among them, based on the authority levels of the two image sub-regions, the corresponding target area to be filtered can be selected for horizontal filtering for deblocking effects. In the case where the image processing device obtains the information of the corresponding image area according to the authority level of the user, the method can meet the needs of users of any authority level to perform horizontal filtering for deblocking effects on the decoded image again, improve the image quality of different authority areas in the image to be processed, and thus improve the user experience at each authority level.

In some embodiments, the filtering process includes sample adaptive compensation filtering. If the authority levels of at least two adjacent image sub-regions among multiple image sub-regions are different, sample adaptive compensation filtering is performed on the adjacent boundary between the at least two image sub-regions. Please refer to Figure 13. The decoding method provided in this embodiment includes the following steps Sc1 to Sc2.

Sc1. According to a set filtering order, determine a target area to be filtered from the surrounding side of a first sub-area in at least two image sub-areas.

Among them, the surrounding side includes, in sequence: the left side, top side, right side and bottom side of the border between the first sub-region and other sub-regions. When the authority level of the first sub-region is different from the authority level of other sub-regions, the target area to be filtered is any one of the following: the first side area within the first sub-region compared to the border, the second side area outside the first sub-region compared to the border, the first side area and the second side area.

In some embodiments, the permission levels of other sub-regions around the first sub-region may be determined in sequence according to the set filtering order, and when the permission levels of other sub-regions around the first sub-region are different from the permission level of the first sub-region, the target area to be filtered is determined. The target area to be filtered may include the following two situations:

Case 1: If other sub-regions with different authority levels from the first sub-region are located on the vertical side of the first sub-region, the first side region is a region with a width of n columns of pixels from the adjacent boundary toward the first sub-region; the second side region is a region with a width of n columns of pixels from the adjacent boundary toward the other sub-regions.

Case 2: If other sub-regions with different authority levels from the first sub-region are located on the horizontal side of the first sub-region, the first side region is a region with a width of n columns of pixels from the adjacent boundary toward the first sub-region; the second side region is a region with a width of n rows of pixels from the adjacent boundary toward the other sub-regions.

The above two situations can exist at the same time.

Exemplarily, as shown in FIG14 , the first sub-region is surrounded by an adjacent sub-region A on the left, an adjacent sub-region B on the upper left, an adjacent sub-region C on the upper side, and an adjacent sub-region D on the upper right. According to the filtering order from right to left and then from top to bottom, the authority levels of other sub-regions around the first sub-region are judged in turn. When the authority levels of the adjacent sub-regions A and C are different from the authority level of the first sub-region, and when the authority levels of the adjacent sub-regions B and D are the same as the authority level of the first sub-region, the target area to be filtered is determined to be a column pixel area with a width of 1 in the left direction and a column pixel area with a width of 1 in the right direction of the border connecting with the adjacent sub-region A, and a row pixel area with a width of 1 in the upward direction and a row pixel area with a width of 1 in the downward direction of the border connecting with the adjacent sub-region C, that is, the area of the shaded part shown in FIG14 .

In some other embodiments, when the authority level of other sub-regions around the first sub-region is higher than the authority level of the first sub-region, the target area to be filtered is determined. The target area to be filtered may include the following two situations:

Case 1: If other sub-regions with higher authority levels than the first sub-region are located on the vertical side of the first sub-region, the first side region is a region with a width of n columns of pixels from the adjacent boundary toward the first sub-region; the second side region is a region with a width of n columns of pixels from the adjacent boundary toward the other sub-regions.

Case 2: If other sub-regions with higher authority levels than the first sub-region are located on the horizontal side of the first sub-region, the first side region is a region with a width of n columns of pixels from the adjacent boundary toward the first sub-region; the second side region is a region with a width of n rows of pixels from the adjacent boundary toward the other sub-regions.

The above two situations can exist at the same time.

Sc2. Perform sample adaptive compensation filtering on the target area to be filtered according to the set filtering order.

In some embodiments, as shown in FIG. 14 above, sample adaptive compensation filtering is performed on the target area to be filtered, that is, the shaded area in FIG. 14 , in a filtering order from right to left and then from top to bottom.

In some embodiments, the adjacent sub-regions around the first sub-region that have not been filtered retain their original pixel values. For example, as shown in FIG. 14 , the pixel values of the borders between the first sub-region and the adjacent sub-region B and the borders between the first sub-region and the adjacent sub-region D remain unchanged.

It is understandable that if the authority levels of at least two adjacent image sub-regions in a plurality of image sub-regions are different, the target area to be filtered can be determined from the surrounding side of the first sub-region of the at least two image sub-regions according to the set filtering order, and the target area to be filtered can be subjected to sample adaptive compensation filtering according to the set filtering order. Among them, based on the authority level of the adjacent sub-regions on the surrounding side of the first sub-region, the corresponding target area to be filtered can be selected for sample adaptive compensation filtering. In the case where the image processing device obtains the information of the corresponding image area according to the authority level of the user, the method can meet the needs of users of any authority level to perform sample adaptive compensation filtering on the decoded image again, improve the image quality of different authority areas in the image to be processed, and thus improve the user experience at each authority level.

In some embodiments, the filtering process includes adaptive loop filtering. If the authority levels of at least two adjacent image sub-regions among multiple image sub-regions are different, adaptive loop filtering is performed on the adjacent boundary between the at least two image sub-regions. Please refer to Figure 15. The decoding method provided in this embodiment includes the following steps Sd1 to Sd2.

Sd1. Determine a target area to be filtered from the surrounding side of a first sub-area in at least two image sub-areas according to a set filtering order.

The above two situations can exist at the same time.

Exemplarily, as shown in FIG16 , the first sub-region is surrounded by an adjacent sub-region A on the left, an adjacent sub-region B on the upper left, an adjacent sub-region C on the upper side, and an adjacent sub-region D on the upper right. According to the filtering order from right to left and then from top to bottom, the authority levels of other sub-regions around the first sub-region are judged in turn. When the authority levels of the adjacent sub-regions B, C, and D are different from the authority level of the first sub-region, and when the authority level of the adjacent sub-region A is the same as the authority level of the first sub-region, the target area to be filtered is determined to be a row pixel area with a width of 1 in the downward direction and a row pixel area with a width of 1 in the upward direction of the boundary connecting the adjacent sub-regions B, C, and D, that is, the area of the shaded part shown in FIG16 .

The above two situations can exist at the same time.

Sd2: According to the set filtering order, adaptive loop filtering is performed on the target area to be filtered.

In some embodiments, as shown in FIG. 16 above, adaptive loop filtering is performed on the target area to be filtered, that is, the shaded area in FIG. 16 , in a filtering order from right to left and then from top to bottom.

In some embodiments, the adjacent sub-regions around the first sub-region that have not been filtered retain their original pixel values. For example, as shown in FIG16 , the pixel values at the border between the first sub-region and the adjacent sub-region A remain unchanged.

It is understandable that if the authority levels of at least two adjacent image sub-regions in a plurality of image sub-regions are different, the target area to be filtered can be determined from the surrounding side of the first sub-region of the at least two image sub-regions according to the set filtering order, and the target area to be filtered can be subjected to sample adaptive compensation filtering according to the set filtering order. Among them, based on the authority level of the adjacent sub-regions on the surrounding side of the first sub-region, the corresponding target area to be filtered can be selected for adaptive loop filtering. In the case where the image processing device obtains the information of the corresponding image area according to the authority level of the user, the method can meet the needs of users of any authority level to perform adaptive loop filtering on the decoded image again, improve the image quality of different authority areas in the image to be processed, and thus improve the user experience at each authority level.

In some embodiments, the filtering process includes filtering process based on a neural network. If the authority levels of at least two adjacent image sub-regions among multiple image sub-regions are different, the connecting boundary between the at least two image sub-regions can be filtered based on a neural network. Please refer to Figure 17. The decoding method provided in this embodiment includes the following steps Se1 to Se2.

Se1. When the first sub-region needs to be subjected to filtering processing based on a neural network, the pixel value of the first sub-region and the authority level of the first sub-region are input into the neural network.

In some embodiments, the pixel value of the first sub-region may be a reconstructed pixel value, and the reconstructed pixel value may be the pixel value of the first sub-region after decoding and filtering, and may be the pixel value after decoding and other filtering (such as deblocking vertical filtering). The pixel value is not specifically limited in this embodiment.

In some embodiments, when the first sub-region needs to be filtered based on a neural network, the reconstructed pixel value of the first sub-region and the authority level of the first sub-region, as well as other decoded information, are input into the neural network. The other decoded information includes the above-mentioned boundary filter strength Bs value, the quantization coefficient of the coding unit transform block, etc. This embodiment is not specifically limited.

In some embodiments, the neural network may be a convolutional neural network (CNN) or other neural networks, which are not specifically limited in this embodiment.

In some embodiments, step Se1 may include step Se12.

Se12. Input the authority level of at least one image sub-region adjacent to the first sub-region into the neural network.

Exemplarily, the permission level of the adjacent sub-region A, the permission level of the adjacent sub-region B, the permission level of the adjacent sub-region C, and the permission level of the adjacent sub-region D around the first sub-region in FIG. 16 are all input into the neural network.

Se2. Obtain the filtered pixels of the first sub-region of the output.

As shown in FIG18 , the reconstructed pixel value of the first sub-region, the authority level of the first sub-region, the authority level of at least one image sub-region adjacent to the first sub-region, and other decoding information are input into the neural network to obtain the output filtered reconstructed pixel of the first sub-region.

It is understandable that, based on the level of authority of the image sub-region, the corresponding target area to be filtered can be selected for filtering based on the neural network. In the case where the image processing device obtains the information of the corresponding image area according to the user's authority level, this method can meet the needs of users of any authority level to perform filtering based on the neural network on the decoded image again, improve the image quality of different authority areas in the image to be processed, and thus improve the user experience at each authority level.

In some embodiments, the process of executing Sa1 to Sa2, Sb1 to Sb2, Sc1 to Sc2, and Sd1 to Sd2 can be performed in sequence, that is, the output of Sa1 to Sa2 is the input of Sb1 to Sb2, the output of Sb1 to Sb2 is the input of Sc1 to Sc2, and the output of Sc1 to Sc2 is the input of Sd1 to Sd2.

In other embodiments, the process of executing Sa1 to Sa2, Sb1 to Sb2, Sc1 to Sc2, Sd1 to Sd2, and Se1 to Se2 may be performed in sequence, that is, the output of Sa1 to Sa2 is the input of Sb1 to Sb2, the output of Sb1 to Sb2 is the input of Sc1 to Sc2, the output of Sc1 to Sc2 is the input of Sd1 to Sd2, and the output of Sd1 to Sd2 is the input of Se1 to Se2.

In other embodiments, the above-mentioned Sa1 to Sa2, Sb1 to Sb2, Sc1 to Sc2, Sd1 to Sd2, and Se1 to Se2 may be performed in any combination.

It is understandable that, based on the level of authority of the image sub-region, the corresponding target area to be filtered can be selected for the above filtering process. And according to the filtering process requirements of the image to be processed, the pixel points on one side of the adjacent boundary of the image sub-region can be selected for filtering process or the pixel points on both sides can be filtered process. At the same time, different filtering process methods can also be combined, and the image to be processed can be filtered in the order of combination, so as to improve the image quality of different authority areas in the image to be processed, thereby improving the user experience.

The present application also provides an electronic device, as shown in FIG19, which is a schematic diagram of the structure of an electronic device provided by the present application, and the electronic device 12 includes a processor 201 and a communication interface 202. The processor 201 and the communication interface 202 are coupled to each other. It is understandable that the communication interface 202 can be a transceiver or an input-output interface. Optionally, the electronic device 12 can also include a memory 203 for storing instructions executed by the processor 201 or storing input data required by the processor 201 to run the instructions or storing data generated after the processor 201 runs the instructions.

In some embodiments, the processor 201 and the communication interface 202 are used to perform the functions of the above-mentioned image division unit 101, the image filtering unit 102, and the image decoding unit 103.

The specific connection medium between the communication interface 202, the processor 201 and the memory 203 is not limited in the embodiment of the present application. In FIG. 17 , the communication interface 202, the processor 201 and the memory 203 are connected by a bus 204, and the bus 204 is represented by a bold line in FIG. 19 . The connection mode between other components is only for schematic illustration and is not limited thereto. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one bold line is used in FIG. 19 , but it does not mean that there is only one bus or one type of bus.

The memory 203 can be used to store software programs and modules, such as program instructions/modules corresponding to the decoding method or encoding method provided in the embodiment of the present application. The processor 201 executes the software programs and modules stored in the memory 203 to perform various functional applications and data processing. The communication interface 202 can be used to communicate signaling or data with other devices. In the present application, the electronic device 12 can have multiple communication interfaces 202.

It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), a neural processing unit (NPU) or a graphic processing unit (GPU), or may be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.

The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device or a terminal device. Of course, the processor and the storage medium can also be present in a network device or a terminal device as discrete components.

In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

In the various embodiments of the present application, if there is no special explanation and logical conflict, the terms and/or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships. In the present application, "at least one" means one or more, and "more" means two or more. "And/or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and/or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In the text description of the present application, the character "/" generally indicates that the previous and next associated objects are in an "or" relationship; in the formula of the present application, the character "/" indicates that the previous and next associated objects are in a "division" relationship.

It is understood that the various numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application. The size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic.

Claims

An image processing method, characterized in that the image processing method is executed by an electronic device, comprising:

Divide the image to be processed into multiple image sub-regions, and obtain the permission level of each image sub-region;

If the authority levels of at least two adjacent image sub-regions among the multiple image sub-regions are different, filtering processing is performed on a connecting boundary between the at least two image sub-regions.
The method according to claim 1 is characterized in that the permission levels include a first permission level, a second permission level, and a zero permission level, and the image sub-area of the zero permission level is an image sub-area that can be viewed by users of any permission level.
The method according to claim 1 or 2, characterized in that before dividing the image to be processed into a plurality of image sub-regions and obtaining the permission level of each image sub-region, it comprises:

A decoded code stream of the image to be processed is received, and decoding is performed based on the decoded code stream to obtain the decoded image to be processed and decoding information of the image to be processed.
The method according to claim 1 or 2 is characterized in that the filtering processing includes: at least one of deblocking effect vertical filtering, deblocking effect horizontal filtering, sample adaptive compensation filtering, adaptive loop filtering and neural network filtering.
The method according to claim 4, characterized in that there is a vertical boundary between the at least two image sub-regions, the filtering process includes deblocking vertical filtering, and the filtering process on the boundary between the at least two image sub-regions includes:

Determine a target area to be filtered on a vertical side of a first sub-region of the at least two image sub-regions; the vertical side is a left side or a right side of a boundary between the first sub-regions, and the target area to be filtered is any one of the following: a right side area to be filtered on a boundary between the two sub-regions, a left side area to be filtered on a boundary between the two sub-regions, a right side area to be filtered on a boundary between the two sub-regions, and a left side area to be filtered on a boundary between the two sub-regions;

Performing a deblocking vertical filter on the target area to be filtered.
The method according to claim 5, characterized in that the at least two adjacent image sub-regions further include a second sub-region, the second sub-region is horizontally adjacent to the first sub-region, and the at least two adjacent image sub-regions have different authority levels, including:

The authority level of the first sub-area is lower than the authority level of the second sub-area;

Alternatively, the authority level of the first sub-area is higher than the authority level of the second sub-area;

Alternatively, the authority level of the first sub-area is not equal to the authority level of the second sub-area;

Alternatively, the authority level of the first sub-region is not equal to the authority level of the second sub-region, and the filtering strength of the adjacent boundary is greater than or equal to a preset filtering strength.
The method according to claim 4, characterized in that there is a horizontal boundary between the at least two image sub-regions, the filtering process includes deblocking effect horizontal filtering, and the filtering process on the boundary between the at least two image sub-regions comprises:

Determine a target area to be filtered on the horizontal side of a first sub-region of the at least two image sub-regions; the horizontal side is an upper side or a lower side of a boundary between the first sub-regions, and the target area to be filtered is any one of the following: a lower side area to be filtered of the boundary, an upper side area to be filtered of the boundary, a lower side area to be filtered of the boundary, and an upper side area to be filtered;

Performing a deblocking horizontal filter on the target area to be filtered.
The method according to claim 7, characterized in that the at least two adjacent image sub-regions further include a third sub-region, the third sub-region is vertically adjacent to the first sub-region, and the at least two adjacent image sub-regions have different authority levels, including:

The authority level of the first sub-area is lower than the authority level of the third sub-area;

Alternatively, the authority level of the first sub-area is higher than the authority level of the third sub-area;

Alternatively, the authority level of the first sub-area is not equal to the authority level of the third sub-area;

Alternatively, the authority level of the first sub-region is not equal to the authority level of the third sub-region, and the filtering strength of the adjacent boundary is greater than or equal to a preset filtering strength.
The method according to claim 4, characterized in that the filtering process comprises sample adaptive compensation filtering, and the filtering process on the boundary between the at least two image sub-regions comprises:

According to the set filtering order, a target area to be filtered is determined from the surrounding side of the first sub-region in the at least two image sub-regions; the surrounding side sequentially includes: the left side, the upper side, the right side and the lower side of the boundary between the first sub-region and other sub-regions, the authority level of the first sub-region is different from the authority level of the other sub-regions, and the target area to be filtered is any one of the following: a first side area in the first sub-region compared to the boundary, a second side area outside the first sub-region compared to the boundary, the first side area and the second side area;

According to the set filtering order, sample adaptive compensation filtering is performed on the target area to be filtered.
The method according to claim 4, characterized in that the filtering process comprises adaptive loop filtering, and the filtering process on the boundary between the at least two image sub-regions comprises:

According to the set filtering order, a target area to be filtered is determined from the surrounding side of the first sub-region in the at least two image sub-regions; the surrounding side sequentially includes: the left side, the upper side, the right side and the lower side of the boundary between the first sub-region and other sub-regions, the authority level of the first sub-region is different from the authority level of the other sub-regions, and the target area to be filtered is any one of the following: a first side area in the first sub-region compared to the boundary, a second side area outside the first sub-region compared to the boundary, the first side area and the second side area;

According to the set filtering order, adaptive loop filtering is performed on the target area to be filtered.
The method according to claim 4, further comprising:

Determining whether it is necessary to perform a neural network-based filtering process on a first sub-region of the at least two image sub-regions according to the decoded information of the image to be processed;

In the case where the first sub-region needs to be subjected to filtering processing based on a neural network, the reconstructed pixel value of the first sub-region and the authority level of the first sub-region are input into the neural network;

Obtain output filtered reconstructed pixels of the first sub-region.
The method according to claim 11, characterized in that, when the first sub-region needs to be subjected to filtering processing based on a neural network, the pixel value of the first sub-region and the authority level of the first sub-region are input into the neural network, and further comprising:

The authority level of at least one image sub-region adjacent to the first sub-region is input into the neural network.
The method according to claim 1 is characterized in that the electronic device obtains information of an image sub-region that is lower than or equal to the user's authority level based on the user's authority level; the information of the image sub-region includes a pixel value of the image sub-region.
The method according to claim 1, characterized in that if the authority levels of at least two adjacent image sub-regions among the multiple image sub-regions are different, filtering the connected boundary between the at least two image sub-regions comprises:

Performing filtering processing on the brightness channel component on the boundary between the at least two image sub-regions;

Alternatively, filtering of the chrominance channel component is performed on the boundary between the at least two image sub-regions;

Alternatively, filtering processing of the brightness channel component and the chrominance channel component is performed on the adjacent boundary between the at least two image sub-regions.
An image processing device, characterized in that the image processing device comprises: an image division unit, an image filtering unit, and an image decoding unit;

The image division unit, the image filtering unit, and the image decoding unit are used to implement the method according to any one of claims 1 to 14.
An electronic device, characterized in that it includes a processor and a memory, wherein the memory is used to store computer instructions, and the processor is used to call and execute the computer instructions from the memory to implement any one of claims 1 to 14.
A non-transitory computer-readable storage medium, characterized in that a computer program or instruction is stored in the storage medium, and when the computer program or instruction is executed by an electronic device, the method described in any one of claims 1 to 14 is implemented.