CN117640959A

CN117640959A - Image encoding method, image decoding device, and computer storage medium

Info

Publication number: CN117640959A
Application number: CN202311353601.6A
Authority: CN
Inventors: 江东; 林聚财; 彭双; 方诚; 张雪; 殷俊
Original assignee: Zhejiang Dahua Technology Co Ltd
Current assignee: Zhejiang Dahua Technology Co Ltd
Priority date: 2023-10-18
Filing date: 2023-10-18
Publication date: 2024-03-01

Abstract

The application proposes an image encoding method, an image decoding method, an image encoder, an image decoding apparatus, and a computer storage medium. The image encoding method includes: acquiring an image to be encoded; selecting a target privacy protection coding level according to the image to be coded; generating a non-private image and a private image based on the image to be encoded and the private area information; encoding the non-private image according to the target privacy protection encoding hierarchy to generate a non-private code stream; encoding the privacy image according to the target privacy protection encoding hierarchy to generate a privacy code stream; generating an encoding code stream of the image to be encoded based on the privacy code stream and the non-privacy code stream; the privacy protection coding hierarchy comprises a maximum coding unit hierarchy, a coding unit hierarchy and an image hierarchy. By the image coding method, various privacy protection schemes are provided, the method is suitable for compression efficiency required to be coded, and the image coding efficiency is improved.

Description

Image encoding method, image decoding device, and computer storage medium

Technical Field

The present invention relates to the field of video encoding and decoding technologies, and in particular, to an image encoding method, an image decoding method, an image encoder, an image decoding apparatus, and a computer storage medium.

Background

The video image data size is relatively large, and video pixel data (RGB, YUV, etc.) is usually required to be compressed, and the compressed data is called a video code stream, and the video code stream is transmitted to a user terminal through a wired or wireless network and then decoded and watched. The whole video coding flow comprises the processes of block division, prediction, transformation, quantization, coding and the like.

In video acquisition, the image content of a part or all of an image area of an acquired image relates to privacy, and in order to protect the privacy of a user, the image content of the privacy area needs to be shielded, but the content of the privacy area can be restored for the user with the authority to view the privacy area.

However, the compression efficiency required for different application scenarios is different. If the compression efficiency is high, the existing privacy protection scheme is difficult to be used for a scene with low hardware configuration, and the falling of the privacy protection scheme is seriously hindered; if the compression efficiency is low, memory and computational overhead are easily wasted, resulting in a lower coding efficiency of the privacy preserving scheme.

Disclosure of Invention

In order to solve the above technical problems, the present application proposes an image encoding method, an image decoding method, an image encoder, an image decoding apparatus, and a computer storage medium.

In order to solve the above technical problems, the present application proposes an image encoding method, which includes:

acquiring an image to be encoded;

selecting a target privacy protection coding level according to the image to be coded;

generating a non-private image and a private image based on the image to be encoded and the private area information;

encoding the non-private image according to the target privacy protection encoding hierarchy to generate a non-private code stream;

encoding the privacy image according to the target privacy protection encoding hierarchy to generate a privacy code stream;

generating an encoding code stream of the image to be encoded based on the privacy code stream and the non-privacy code stream;

the privacy protection coding hierarchy comprises a maximum coding unit hierarchy, a coding unit hierarchy and an image hierarchy.

The target privacy protection coding level is a maximum coding unit level or a coding unit level;

the encoding the non-private image according to the target privacy protection encoding hierarchy to generate a non-private code stream includes:

Dividing the non-private image into a first private area and a first non-private area according to the maximum coding unit level or the coding unit level;

encoding the first privacy zone and the first non-privacy zone respectively to obtain a privacy zone code stream of the first privacy zone and a non-privacy zone code stream of the first non-privacy zone;

acquiring a non-privacy code stream of the non-privacy image based on the privacy zone code stream and the non-privacy zone code stream;

the reference information encoded in the first non-privacy zone does not include the encoded information of the first privacy zone.

The privacy image is encoded according to the target privacy protection encoding hierarchy, and a privacy code stream is generated;

dividing the privacy image into a second privacy area and a second non-privacy area according to the maximum coding unit level or the coding unit level;

encoding the second privacy zone to obtain an independent privacy code stream of the second privacy zone;

and acquiring the privacy code stream of the privacy image based on the independent privacy code stream.

The target privacy protection coding level is the coding unit level;

The dividing the non-private image into a first private area and a first non-private area according to the maximum coding unit level or the coding unit level includes:

dividing the non-privacy image into a plurality of first layer coding units according to the coding unit level;

judging whether each first layer coding unit simultaneously comprises a privacy area and a non-privacy area;

if yes, the first layer coding unit which simultaneously comprises the privacy area and the non-privacy area is continuously divided into a plurality of second layer coding units until each coding unit only comprises the privacy area or the non-privacy area;

taking an encoding unit only comprising a privacy area as a privacy encoding unit of the non-privacy image;

and taking the coding unit only comprising the non-privacy area as a non-privacy coding unit of the non-privacy image.

The target privacy protection coding level is the maximum coding unit level;

dividing the non-privacy image into a plurality of maximum coding units according to the maximum coding unit level;

Taking the coding units only comprising the non-private areas as non-private coding units of the non-private image, and taking the rest coding units as private coding units of the non-private image;

the encoding the first privacy zone and the first non-privacy zone respectively to obtain a privacy zone code stream of the first privacy zone and a non-privacy zone code stream of the first non-privacy zone includes:

encoding the non-privacy encoding unit to obtain the non-privacy region code stream;

and encoding the privacy encoding unit to obtain the privacy region code stream.

The target privacy protection coding level is an image level;

inputting the non-privacy image into a basic layer of a spatial domain scalable video encoder according to the image hierarchy, and acquiring a non-privacy code stream of the non-privacy image;

the encoding the privacy image according to the target privacy protection encoding hierarchy to generate a privacy code stream includes:

inputting the privacy image into an enhancement layer of the airspace scalable video encoder according to the image hierarchy, and acquiring a privacy code stream of the privacy image;

The generating the encoding code stream of the image to be encoded based on the private code stream and the non-private code stream includes:

and interleaving and fusing the private code stream and the non-private code stream into the code stream of the image to be coded by using the spatial scalable video encoder.

The step of inputting the privacy image into the enhancement layer of the spatial domain scalable video encoder according to the image level to obtain the privacy code stream of the privacy image comprises the following steps:

acquiring a non-privacy reconstructed image of the non-privacy code stream according to the image hierarchy;

and inputting the privacy image and the non-privacy reconstructed image into an enhancement layer of the airspace scalable video encoder, and obtaining a privacy code stream of the privacy image.

encoding a non-privacy area of the non-privacy image according to the target privacy protection encoding hierarchy to generate a first non-privacy code stream;

encoding the privacy area of the non-privacy image according to the shielding image content to generate a second non-privacy code stream;

generating the non-private code stream based on the first non-private code stream and the second non-private code stream.

Wherein, the image coding method further comprises:

acquiring privacy zone coordinates based on the privacy zone information;

generating a privacy zone syntax by using the privacy zone coordinates;

determining a privacy protection scheme based on the target privacy protection encoding hierarchy;

generating a privacy protection scheme syntax using the privacy protection scheme;

and generating a code stream organization form syntax based on the code stream organization form of the privacy code stream and/or the non-privacy code stream.

To solve the above technical problem, the present application further proposes an image encoder, the image encoder including a memory and a processor coupled to the memory; wherein the memory is configured to store program data, and the processor is configured to execute the program data to implement the image encoding method as described above.

In order to solve the above technical problems, the present application further provides an image decoding method, where the image encoding method includes:

acquiring a code stream to be decoded;

acquiring a target privacy protection decoding level based on the code stream to be decoded;

responding to the privacy authority, and acquiring a privacy code stream to be decoded and a non-privacy code stream to be decoded from the code stream to be decoded;

decoding the privacy code stream to be decoded according to the target privacy protection decoding hierarchy to generate a decoding privacy image;

Decoding the non-private code stream to be decoded according to the target privacy protection decoding hierarchy to generate a decoded non-private image;

generating a decoded image based on the decoded private image and the decoded non-private image;

the privacy protection decoding level comprises a maximum coding unit level, a coding unit level and an image level.

To solve the above technical problem, the present application further proposes an image decoder, the image decoder including a memory and a processor coupled to the memory; wherein the memory is configured to store program data, and the processor is configured to execute the program data to implement the image decoding method as described above.

To solve the above technical problem, the present application further proposes a computer storage medium for storing program data, which when executed by a computer, is configured to implement the above-mentioned image encoding method, and/or image decoding method.

Compared with the prior art, the beneficial effects of this application are: the image encoder acquires an image to be encoded and compression efficiency to be encoded; selecting a target privacy protection coding level according to the compression efficiency; generating a non-private image and a private image based on the image to be encoded and the private area information; encoding the non-private image according to the target privacy protection encoding hierarchy to generate a non-private code stream; encoding the privacy image according to the target privacy protection encoding hierarchy to generate a privacy code stream; generating an encoding code stream of the image to be encoded based on the privacy code stream and the non-privacy code stream; the privacy protection coding hierarchy comprises a maximum coding unit hierarchy, a coding unit hierarchy and an image hierarchy. By the image coding method, various privacy protection schemes are provided, the method is suitable for compression efficiency required to be coded, and the image coding efficiency is improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Wherein:

FIG. 1 is a flowchart illustrating an embodiment of an image encoding method provided in the present application;

FIG. 2 is a flow diagram of a multi-privacy preserving mode scheme provided herein;

FIG. 3 is a schematic diagram of privacy zone information provided herein;

FIG. 4 is a schematic flowchart showing a specific step S14 of the image encoding method shown in FIG. 1;

FIG. 5 is a flow diagram of a CU level reference limiting privacy preserving scheme provided herein;

fig. 6 is a schematic diagram of LCU privacy mapping provided herein;

FIG. 7 is a schematic diagram of various block partition types provided herein;

FIG. 8 is a flow chart of the code tree partitioning process provided herein;

FIG. 9 is a schematic diagram of the partitioning result of the code tree partitioning process of FIG. 8;

FIG. 10 is a schematic diagram of a flexible LCU processing sequence provided herein;

Fig. 11 is a flow chart of an airspace SVC privacy protection scheme with interlayer reference provided in the present application;

FIG. 12 is a schematic view of a privacy preserving scheme supporting front-end occlusion images provided herein;

FIG. 13 is a schematic view of a privacy preserving scheme supporting back-end mask images provided herein;

FIG. 14 is a flowchart of an embodiment of an image decoding method provided in the present application;

FIG. 15 is a schematic diagram of an embodiment of an image encoder provided herein;

FIG. 16 is a schematic diagram illustrating the structure of an embodiment of an image decoder provided herein;

fig. 17 is a schematic structural diagram of an embodiment of a computer storage medium provided in the present application.

Detailed Description

The following description of the technical solutions in the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.

The terms "first," "second," "third," "fourth" and the like in the description and in the claims of this application and in the above-described figures, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that embodiments of the present application described herein may be implemented, for example, in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

The coding and decoding method with multiple privacy protection modes is suitable for different application scenes, and various privacy protection schemes can also support flexible privacy area shielding processing.

Referring to fig. 1 and fig. 2, fig. 1 is a flow chart of an embodiment of an image encoding method provided in the present application, and fig. 2 is a flow chart of a multi-privacy preserving mode scheme provided in the present application.

As shown in fig. 2, the image encoding method of the present application is divided into three parts: multi-privacy preserving scheme, privacy zone processing, syntactic expression of multi-privacy preserving scheme.

Specifically, a multi-privacy protection scheme: in the multi-privacy protection scheme, a plurality of privacy protection schemes are designed, the requirements of different privacy protection schemes on computing resources and compression efficiency are different, and a user can select a proper privacy protection scheme according to actual service and scene.

Masking image processing: for users without privacy authorities, the privacy area can be flexibly shielded at the encoding end or the decoding end according to actual requirements.

Syntactic expression of multiple privacy preserving schemes: the method comprises a privacy zone syntax, a privacy protection scheme syntax and a code stream organization form syntax which are respectively used for representing the organization forms of privacy zone information, a selected privacy protection scheme, a privacy zone code stream and a non-privacy zone code stream.

As shown in fig. 1, the specific steps of the multi-privacy protection scheme in the image encoding method provided in the present application are as follows:

step S11: and acquiring an image to be encoded.

In the embodiment of the application, the image encoder provides a plurality of privacy protection schemes with different compression efficiencies, so that a user can select different privacy protection schemes according to the computing resources of the user equipment or the compression efficiency. Where compression efficiency is proportional to the computational resources required, i.e., the higher the compression efficiency, the greater the computational resources required.

Step S12: and selecting a target privacy protection coding level according to the image to be coded.

In the embodiment of the application, the user can adaptively acquire the optimal privacy protection scheme according to the compression efficiency or the calculation resources by inputting the required compression efficiency or the calculation resources possessed by the image encoder, so that the efficiency of image encoding is improved.

Assuming that N privacy preserving schemes exist, each privacy preserving scheme has different compression efficiency, compression efficiency of the nth privacy preserving scheme > =compression efficiency of the n+1th privacy preserving scheme. Meanwhile, in general, the computation resource requirement of the nth privacy protection scheme > =the computation resource requirement of the n+1th privacy protection scheme, and the computation resource requirement generally includes both of the computation power and the memory consumption.

The privacy protection scheme of four different compression efficiencies is designed, and the privacy protection scheme is respectively an ultrahigh compression efficiency scheme, a high compression efficiency scheme, a common compression efficiency scheme and a low compression efficiency scheme. The privacy protection coding levels for different privacy protection schemes are different, for example, an ultra-high compression efficiency scheme corresponds to a coding unit level, a high compression efficiency scheme corresponds to a maximum coding unit level, and a normal compression efficiency scheme and a low compression efficiency scheme correspond to an image level.

Specifically, privacy preserving schemes of ultra-high compression efficiency, including but not limited to: 1) Privacy protection scheme for CU-level reference restrictions: in the codec process, a non-private CU cannot refer to any information of an encoded private CU.

Privacy preserving schemes with high compression efficiency, including but not limited to: 1) Privacy protection scheme for LCU level reference restrictions: during the encoding and decoding process, the non-private LCU cannot refer to any information of the encoded private LCU.

Privacy preserving schemes for common compression efficiencies include, but are not limited to: 1) Spatial domain SVC privacy protection scheme with interlayer reference: the base layer in spatial SVC is used to encode non-private images, the enhancement layer is used to encode private images, and the encoded reconstructed base layer images can be referenced in the enhancement layer image encoding and decoding process.

Privacy preserving schemes of low compression efficiency, including but not limited to: 1) Spatial SVC privacy protection scheme without interlayer reference: the base layer in the spatial SVC is used for encoding non-privacy images, the enhancement layer is used for encoding privacy images, and the encoded reconstructed base layer images cannot be referred in the encoding and decoding processes of the enhancement layer images; 2) SEI-based privacy protection scheme: the non-private image stream is normally transmitted, while the private image stream is transmitted via SEI information.

The image encoder selects a target privacy protection coding level according to the compression efficiency, namely one of the four privacy protection schemes, and continuously executes the following steps according to the selected privacy protection scheme:

step S13: generating a non-private image and a private image based on the image to be encoded and the private region information.

In the embodiment of the application, the image encoder acquires an input video image, namely an image to be encoded, and then performs privacy protection processing on the image to be encoded according to the privacy zone information. The image to be coded is a non-privacy image, and the image to be coded which is not subjected to privacy protection treatment is a privacy image.

Specifically, the privacy zone represented by the privacy zone information is generally a rectangular zone. As shown in fig. 3, fig. 3 is a schematic diagram of privacy zone information provided in the present application. The general privacy zone description information is { x_top_left_privacy [ i ], y_top_left_privacy [ i ], x_right_bottom_privacy [ i ], y_right_bottom_privacy [ i ] }.

Wherein x_top_left_private [ I ], y_top_left_private [ I ] respectively represent a horizontal coordinate and a vertical coordinate of an upper left corner of an I-th privacy zone in an image, x_right_bottom_private [ I ], y_right_bottom_private [ I ] respectively represent a horizontal coordinate and a vertical coordinate of an upper right corner of the I-th privacy zone in the image, i= {0,1, …, I-1}, and I is the number of privacy zones. The size information (width_private [ i ] and height_private [ i ] of the privacy area) and the location information of the privacy area can be uniquely deduced from each other, as shown in the following formula:

width_privacy＝x_right_bottom_privacy[i]-x_top_left_privacy[i]

height_privacy＝y_right_bottom_privacy[i]-y_top_left_privacy[i]

step S14: and encoding the non-private image according to the target privacy protection encoding hierarchy to generate a non-private code stream.

In the embodiment of the present application, when the target privacy protection encoding level determined by the image encoder is the maximum encoding unit level or the encoding unit level, steps S141 to S143 are specifically executed.

With continued reference to fig. 4, fig. 4 is a schematic flowchart illustrating a specific process of step S14 of the image encoding method shown in fig. 1.

As shown in fig. 4, when the target privacy protection coding level is the maximum coding unit level or the coding unit level, step S14 specifically includes the following steps:

step S141: the non-private image is divided into a first private region and a first non-private region by a maximum coding unit level or coding unit level.

In the embodiment of the application, the image encoder divides the non-privacy image into a plurality of encoding units, the encoding unit corresponding to the largest encoding unit level is LCU (logic coding unit), and the encoding unit corresponding to the encoding unit level is CU (coding unit).

Specifically, when the target privacy preserving encoding hierarchy is an encoding unit hierarchy, the image encoder needs to divide encoding units for non-private images, i.e., CU division restriction. As shown in fig. 5, fig. 5 is a schematic flow chart of the CU-level reference restricted privacy protection scheme provided in the present application.

Wherein, CU partition limit: if the currently divided CU area includes both a private area and a non-private area, the current CU cannot be a leaf node of the partition tree, and the current CU must continue to partition. Finally, each CU in the division result contains only a private area or a private area, a coding unit containing a private area as a private CU, and a coding unit containing a non-private area as a non-private CU.

When the target privacy protection coding level is the maximum coding unit level, the image encoder needs to divide the non-privacy image into maximum coding units, namely, LCU division limitation.

Wherein, the image encoder constructs an LCU mapping chart: marking each LCU as a private LCU or a non-private LCU, if the LCU area contains the private area, considering the LCU as the private LCU, otherwise, judging the LCU as the non-private LCU. Specifically, as shown in fig. 6, fig. 6 is a schematic diagram of LCU privacy mapping provided in the present application. In the figure, each box is an LCU, the private LCU is marked 1, and the non-private LCU is marked 0.

Specifically, the present application may adopt a block division structure of qtbt+eqt, QT is a quadtree division, BT is a binary tree division, EQT is an i-shaped quadtree division, and the division modes are 5 in total, as shown in fig. 7. From left to right: QT partition (split_quad), BT vertical partition (split_bi_ver), BT horizontal partition (split_bi_hor), EQT vertical partition (split_eqt_ver), EQT horizontal partition (split_eqt_hor).

The schematic diagram of the partition nodes in the block partition process is shown in fig. 8:

step 1. The partition tree starts with the CTU (coding tree unit), i.e. the CTU root node in the figure.

Step2, carrying out QT division recursively, and if QT nodes carry out QT division, generating four QT nodes; if the QT division is not performed, the QT node is a QT leaf node.

Step3, for QT leaf nodes (or EQT nodes or BT nodes), determining whether to divide or not, and if not dividing, determining the nodes as CU leaf nodes; if the partition is performed (the EQT or BT node starts from the QT leaf node), it is determined whether the EQT partition or the BT partition is performed, if the EQT partition (the partition direction may be horizontal or vertical), four EQT nodes are generated, and if the BT partition (the partition direction may be horizontal or vertical), two BT nodes are generated. In addition, the EQT and BT divisions can be recursively performed as well, but the QT division cannot be performed.

The schematic diagram of the division results is shown in fig. 9, the left side is the division result of a certain CTU, the reference numerals are CU sequences, and different colors are used for distinguishing the division types; the right side is the partition tree corresponding to the partition result, the black circles are non-CU leaf nodes, and the circles with labels (corresponding to the left side labels) and gray (corresponding to the left side partition type) are CU leaf nodes.

Step S142: and respectively encoding the first privacy zone and the first non-privacy zone to obtain a privacy zone code stream of the first privacy zone and a non-privacy zone code stream of the first non-privacy zone.

In the embodiment of the present application, when the target privacy protection coding level is a coding unit level, in the coding and decoding process of the CU in the non-privacy area, when spatial neighboring/non-neighboring information and temporal information need to be acquired, if the information is the CU from the privacy area, the information is limited to be unavailable, and the information may be any coding and decoding information, including, but not limited to, an intra-prediction mode, an inter-prediction mode, a block vector, a motion vector, a reconstructed pixel, and the like.

When the target privacy protection coding level is the maximum coding unit level, the image coder firstly codes and decodes the non-privacy LCU and then codes and decodes the privacy LCU in the coding and decoding process. In the coding and decoding process of each type of LCU, coding and decoding are performed in sequence from left to right and from top to bottom.

As shown in fig. 10, the white non-private LCU group is firstly encoded and decoded, and the specific processing sequence is shown in the box; and then coding and decoding the gray privacy LCU group, wherein the specific processing sequence is shown in the box by the numbers.

LCU level reference limits: in the encoding and decoding process of the non-private LCU, when the CU of the non-private LCU needs to refer to spatial neighboring/non-neighboring, temporal CU information, if the referred CU belongs to the private LCU, the referred information may not be acquired, and the referred information may be any encoding and decoding information, including, but not limited to, intra-prediction mode, inter-prediction mode, block vector, motion vector, reconstructed pixel, and the like.

Step S143: and acquiring a non-privacy code stream of the non-privacy image based on the privacy zone code stream and the non-privacy zone code stream.

After the image encoder acquires the private area code stream and the non-private area code stream at the CU level or the LCU level, the private area code stream and the non-private area code stream are fused to obtain the non-private code stream of the non-private image.

In another embodiment, when the target privacy preserving encoding hierarchy is an image hierarchy, the image encoder may further select a privacy preserving scheme of a general compression efficiency or a privacy preserving scheme of a low compression efficiency. The privacy protection scheme of the common compression efficiency is an airspace SVC (Scalable Video Coding) privacy protection scheme with interlayer reference and scalable video coding. The hidden protection scheme with low compression efficiency is an airspace SVC privacy protection scheme without interlayer reference.

Taking a privacy protection scheme with a common compression efficiency as an example, an image encoding method of an image level is introduced, and referring to fig. 11 specifically, fig. 11 is a schematic flow diagram of an airspace SVC privacy protection scheme with interlayer reference provided in the present application.

As shown in fig. 11, the privacy preserving scheme of the general compression efficiency specifically includes the steps of:

step1: the original image, i.e. the image to be encoded, generates a non-private image and a private image based on the private region information (front-end detection).

Step2: at the encoding end: the non-private image and the private image are taken as input of a spatial SVC encoder, namely a spatial scalable video encoder, a base layer is used for encoding the non-private image, an enhancement layer is used for encoding the private image, and finally a code stream (comprising a non-private code stream and a private code stream) is output. Wherein, the base layer reconstructed image (i.e. the non-privacy reconstructed image) can be referred to in the encoding and decoding process of the enhancement layer. In addition, privacy zone information may be selectively transmitted.

Step3: at the decoding end: the non-private code stream is used as a base layer decoding input, the private code stream is used as an enhancement layer decoding input, in the same decoding process, the enhancement layer can refer to the base layer reconstruction value, and finally the non-private image and the private image (which are output when the privacy right exists) are output.

Correspondingly, the hidden protection scheme with low compression efficiency is similar to the spatial SVC privacy protection scheme with interlayer reference described above, except that the enhancement layer cannot refer to the base layer during the coding and decoding process.

As shown in fig. 2, the masking image processing in the image encoding method provided in the present application specifically includes: in the encoding and decoding of the non-private image, the mask image processing may include front-end mask image processing, back-end mask image processing; in particular, when the mask image processing is not performed, the privacy zone image is filled with a fixed value or reconstructed pixel value or the like in the encoding or decoding process of the non-privacy image.

1. Front-end mask image processing: when front-end occlusion image processing is supported, in the encoding process of non-private images, the privacy zone needs to encode the occlusion image and transmit in the code stream, wherein the occlusion image content includes, but is not limited to, fixed value padding, reconstructed pixel value padding, mosaic images, custom images, and the like.

2. Back-end mask image processing: when back-end mask image processing is supported, in the decoding process of the non-private image, mask image content is added to the private area according to the decoded private area information, wherein the back-end mask image content includes, but is not limited to, fixed value padding, pixel value padding, mosaic images, custom images, and the like.

In a specific embodiment, the non-mask image processing, that is, if the privacy region needs to be encoded in the encoding process of the non-privacy image, the privacy region is filled with a fixed value; in the decoding process of the non-private image, the private area is filled with a fixed value. Let the fixed value be val, val=2 < < (bitdepth-1), where bitdepth is the bit depth when processing the image.

In another specific embodiment, as shown in fig. 12, in the front-end mask image processing, in the encoding process of the non-private image, the embodiment processes the private area into the custom mosaic content, and encodes and transmits the custom mosaic content.

In still another specific embodiment, as shown in fig. 13, in the decoding process of the non-private image, the embodiment acquires the private area information, and processes the content of the private area into the custom mosaic content through the back-end mask image processing.

Step S15: and encoding the privacy image according to the target privacy protection encoding hierarchy to generate a privacy code stream.

In the embodiment of the present application, when the target privacy protection coding level determined by the image encoder is the maximum coding unit level or the coding unit level, the non-private image is encoded first, and then the private image is encoded. In the encoding and decoding process of the privacy image, the division structure adopts the division result of the non-privacy image, but only encodes the privacy CU or the privacy LCU.

Step S16: and generating an encoding code stream of the image to be encoded based on the privacy code stream and the non-privacy code stream.

In the embodiment of the present application, as shown in fig. 2, the syntax expression of the multi-privacy protection scheme in the image encoding method provided in the present application is specifically as follows: a multi-privacy preserving scheme syntax comprising 1) a privacy zone syntax; 2) Privacy preserving scheme syntax; 3) The code stream organizes formal syntax.

Specifically, the privacy zone syntax expression includes, but is not limited to:

SEI (Supplemental Enhancement Information, media supplemental enhancement information) identifies privacy zone coordinates: coordinates of the privacy zone are transmitted through the SEI information.

CU-level syntax identification: in each CU, whether the current CU is a private CU or a non-private CU is marked, wherein the area corresponding to the private CU is a private area, and the area corresponding to the non-private CU is a non-private area.

Mask map identification: each pixel point label belongs to a private area or a non-private area.

Privacy preserving scheme syntax, including but not limited to: privacy protection scheme switch syntax and schema syntax are set at the video parameter set (VPS, video parameter set), sequence parameter set (SPS, sequence Paramater Set), picture parameter set (PPS, picture parameter set), picture Header (PH), where the switch syntax is used to indicate the enablement of the privacy protection scheme and the schema syntax is used to indicate what privacy protection scheme is enabled.

The bitstream organization form syntax includes, but is not limited to: the private and non-private streams employ different NAL identifications and the private and non-private streams employ the same NAL identifications.

The NAL type syntax element nal_unit_type, whose syntax (existing syntax) is shown in the following table, is used to distinguish the data payload type in the NAL:

in one specific embodiment: the privacy zone syntax is expressed as CU-level syntax identification.

Specifically, syntax private_cu_flag is added to each CU, and private_cu_flag=0 indicates that the CU is a non-private CU, and private_cu_flag=1 indicates that the CU is a private CU.

In another specific embodiment: the privacy preserving scheme syntax specifically includes:

switching syntax: the addition of a syntax privacy_enable_flag in the sequence parameter set, privacy_enable_flag=0 indicates that the privacy protection scheme is not enabled, and privacy_enable_flag=1 indicates that the privacy protection scheme is enabled.

Pattern syntax: and adding a syntax privacy_mode_idx in the sequence parameter set, wherein the value range of the syntax privacy_mode_idx is 0 to 3, when the syntax privacy_enable_flag=1, the syntax privacy_mode_idx is valid, and otherwise, the syntax privacy_mode_idx is invalid.

When private_enable_flag=1, private_mode_idx=0/1/2/3 represents a privacy protection scheme among a privacy protection scheme using super-efficient compression efficiency/a privacy protection scheme of normal compression efficiency/a privacy protection scheme of low compression efficiency, respectively.

In yet another specific embodiment: code stream organization form syntax: the private and non-private streams are identified by different NALs.

Specifically, when NAL type syntax nal_unit_type=19 and 20 in the present embodiment, the NAL is considered to be a non-private image and a coded slice of a private image as shown in the following table. The patch_data_rbsp () is represented as a slice load, and the load contains relevant information of decoded pictures:

in the embodiment of the application, an image encoder acquires an image to be encoded and compression efficiency to be encoded; selecting a target privacy protection coding level according to the compression efficiency; generating a non-private image and a private image based on the image to be encoded and the private area information; encoding the non-private image according to the target privacy protection encoding hierarchy to generate a non-private code stream; encoding the privacy image according to the target privacy protection encoding hierarchy to generate a privacy code stream; generating an encoding code stream of the image to be encoded based on the privacy code stream and the non-privacy code stream; the privacy protection coding hierarchy comprises a maximum coding unit hierarchy, a coding unit hierarchy and an image hierarchy. By the image coding method, various privacy protection schemes are provided, the method is suitable for compression efficiency required to be coded, and the image coding efficiency is improved.

The privacy protection scheme provided by the application can comprise multiple privacy protection schemes with different efficiencies, different choices are provided for the user, and different privacy protection schemes can be selected according to the conditions of the computing resource, the network bandwidth and the like of the self equipment. The privacy protection scheme provided by the application can provide flexible shielding image processing.

With continued reference to fig. 14, fig. 14 is a flowchart illustrating an embodiment of an image decoding method provided in the present application.

As shown in fig. 14, the image decoding method provided in the present application specifically includes the following steps:

step S21: and obtaining a code stream to be decoded.

Step S22: and acquiring a target privacy protection decoding level based on the code stream to be decoded.

Step S23: and responding to the privacy authority, and acquiring the privacy code stream to be decoded and the non-privacy code stream to be decoded from the code stream to be decoded.

Step S24: and decoding the privacy code stream to be decoded according to the target privacy protection decoding level to generate a decoding privacy image.

Step S25: and decoding the non-private code stream to be decoded according to the target privacy protection decoding level to generate a decoded non-private image.

Step S26: a decoded image is generated based on the decoded private image and the decoded non-private image.

The image decoding process in steps S21 to S26 of the present application and the image encoding process in the above embodiment are inverse processes, and will not be described here again.

It will be appreciated by those skilled in the art that in the above-described method of the specific embodiments, the written order of steps is not meant to imply a strict order of execution but rather should be construed according to the function and possibly inherent logic of the steps.

In order to implement the above image encoding method, the present application further proposes an image encoder, and specifically please refer to fig. 15, fig. 15 is a schematic structural diagram of an embodiment of the image encoder provided in the present application.

The image encoder 400 of the present embodiment includes a processor 41, a memory 42, an input-output device 43, and a bus 44.

The processor 41, the memory 42 and the input/output device 43 are respectively connected to the bus 44, and the memory 42 stores program data, and the processor 41 is configured to execute the program data to implement the image encoding method described in the above embodiment.

In the present embodiment, the processor 41 may also be referred to as a CPU (Central Processing Unit ). The processor 41 may be an integrated circuit chip with signal processing capabilities. The processor 41 may also be a general purpose processor, a digital signal processor (DSP, digital Signal Process), an application specific integrated circuit (ASIC, application Specific Integrated Circuit), a field programmable gate array (FPGA, field Programmable Gate Array) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The general purpose processor may be a microprocessor or the processor 41 may be any conventional processor or the like.

In order to implement the above image decoding method, the present application further proposes an image decoder, and specifically please refer to fig. 16, fig. 16 is a schematic structural diagram of an embodiment of the image decoder provided in the present application.

The image decoder 500 of the present embodiment includes a processor 51, a memory 52, an input-output device 53, and a bus 54.

The processor 51, the memory 52, and the input/output device 53 are respectively connected to the bus 54, and the memory 52 stores program data, and the processor 51 is configured to execute the program data to implement the image decoding method described in the above embodiment.

In the present embodiment, the processor 51 may also be referred to as a CPU (Central Processing Unit ). The processor 51 may be an integrated circuit chip with signal processing capabilities. Processor 51 may also be a general purpose processor, a digital signal processor (DSP, digital Signal Process), an application specific integrated circuit (ASIC, application Specific Integrated Circuit), a field programmable gate array (FPGA, field Programmable Gate Array) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The general purpose processor may be a microprocessor or the processor 51 may be any conventional processor or the like.

Still further, with continued reference to fig. 17, fig. 17 is a schematic structural diagram of an embodiment of the computer storage medium provided in the present application, in which the computer program 61 is stored in the computer storage medium 600, and the computer program 61 is configured to implement the image encoding method and/or the image decoding method according to the above embodiments when executed by a processor.

Embodiments of the present application are implemented in the form of software functional units and sold or used as a stand-alone product, which may be stored on a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied in essence or a part contributing to the prior art or all or part of the technical solution, in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The foregoing description is only of embodiments of the present application, and is not intended to limit the scope of the patent application, and all equivalent structures or equivalent processes using the descriptions and the contents of the present application or other related technical fields are included in the scope of the patent application.

Claims

1. An image encoding method, characterized in that the image encoding method comprises:

acquiring an image to be encoded;

2. The image encoding method according to claim 1, wherein,

3. The image encoding method according to claim 2, wherein,

4. The image encoding method according to claim 2, wherein,

the target privacy protection coding level is the coding unit level;

5. The image encoding method according to claim 2, wherein,

the target privacy protection coding level is the maximum coding unit level;

6. The image encoding method according to claim 1, wherein,

the target privacy protection coding level is an image level;

7. The image encoding method according to claim 6, wherein,

inputting the privacy image into an enhancement layer of the spatial domain scalable video encoder according to the image hierarchy, and acquiring a privacy code stream of the privacy image, wherein the method comprises the following steps:

8. The image encoding method according to claim 1, wherein,

9. The image encoding method according to claim 1, wherein,

the image encoding method further includes:

acquiring privacy zone coordinates based on the privacy zone information;

generating a privacy zone syntax by using the privacy zone coordinates;

10. An image encoder comprising a memory and a processor coupled to the memory;

Wherein the memory is for storing program data and the processor is for executing the program data to implement the image encoding method as claimed in any one of claims 1 to 9.

11. An image decoding method, characterized in that the image encoding method comprises:

acquiring a code stream to be decoded;

12. An image decoder, comprising a memory and a processor coupled to the memory;

wherein the memory is for storing program data and the processor is for executing the program data to implement the image decoding method of claim 11.

13. A computer storage medium for storing program data which, when executed by a computer, is adapted to carry out the image encoding method of any one of claims 1 to 9 and/or the image decoding method of claim 11.