CN113489991B

CN113489991B - Video encoding method, video encoding device, video encoding apparatus, and storage medium

Info

Publication number: CN113489991B
Application number: CN202110745137.XA
Authority: CN
Inventors: 刘学青; 董磊; 刘宗宗
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2021-06-30
Filing date: 2021-06-30
Publication date: 2023-04-07
Anticipated expiration: 2041-06-30
Also published as: CN113489991A

Abstract

The embodiment of the application provides a video coding method, a video coding device and a storage medium, wherein a composite video stream simultaneously comprising an original frame and a privacy protection frame is coded, the original frame refers to the privacy protection frame for coding, and the original frame is not used as a reference frame during coding, so that the original frame does not need to be stored, the video stream is compounded and time-domain layered in preprocessing, the memory consumption required by the storage of the reference frame is reduced for time-domain layered interleaving reference in a reference relation, the correlation between the privacy protection frame and the original frame is effectively utilized, the privacy protection frame corresponding to the original frame is referred to for coding, most of reference areas are completely the same, the number of coded code streams is greatly reduced, and the code stream storage and transmission cost is reduced; moreover, privacy protection processing is realized in the video encoder through hardware logic, so that the processing pressure of a processor can be reduced, the memory read-write bandwidth of image data is reduced, and the encoding efficiency is greatly improved.

Description

Video encoding method, video encoding device, video encoding apparatus, and storage medium

Technical Field

The present application relates to the field of video coding technologies, and in particular, to a video coding method, an apparatus, a video coding device, and a storage medium.

Background

With the development of scientific technology and the improvement of safety consciousness of people, monitoring equipment in public places is more and more, and intelligent detection technology is more and more applied to monitoring scenes. In addition, people pay more and more attention to personal privacy protection under monitoring equipment, and the demand of video privacy protection technology is higher and higher. The video privacy protection is that the image of the privacy area is not recognizable by processing such as coding, blocking or scrambling the local privacy area in the video image, and other areas keep the original content to be normally visible. However, image processing for protecting video privacy has the characteristic that protected contents are not recoverable, and when public safety hazards occur, if the original contents cannot be recovered by a monitoring video, the evidence obtaining work is greatly hindered.

In order to solve the problems of privacy protection and evidence collection, in the related technology, collected video streams are processed into two paths of same video streams, one path of video stream is not subjected to privacy processing, an original video frame is reserved for coding, and the other path of video stream is subjected to privacy protection processing and codes the video frame after privacy protection.

However, by using the above method, two paths of video streams are respectively encoded, which may cause a problem of large resource consumption in the encoding process, and also causes a large consumption of network bandwidth resources in the data transmission process.

Disclosure of Invention

An object of the embodiments of the present application is to provide a video encoding method, a video encoding device, a video encoding apparatus, and a storage medium, so as to reduce resource consumption in an encoding process. The specific technical scheme is as follows:

in a first aspect, an embodiment of the present application provides a video encoding method applied to a video encoding apparatus, where the video encoding apparatus includes a processor and a video encoder, and the method includes:

the processor acquires a video stream to be processed, and configures each video frame in the video stream to be processed into a privacy protection frame and an original frame respectively to obtain a composite video stream; setting the interframe coding reference mode of the privacy protection frame as referable and the privacy protection type as protection, setting the interframe coding reference mode of the original frame as unreferenceable and the privacy protection type as unprotected, and thus obtaining configuration information; the processor sending the composite video stream and the configuration information to the video encoder;

the video encoder carries out privacy protection on the video frames with the privacy protection types in the composite video stream based on the configuration information, and encodes the video frames with the privacy protection types; coding the video frame of which the privacy protection type is unprotected in the composite video stream to obtain a coded video data stream, wherein the interframe coding reference mode is a referable video frame and is allowed to be used as a reference frame for coding other video frames; inter-coded reference modes are video frames that are not referable and do not allow reference frames for encoding as other video frames.

In one possible embodiment, the video encoder includes a control module, a privacy processing module, and an encoding module;

the video encoder carries out privacy protection on the video frames with the privacy protection types being protected in the composite video stream based on the configuration information, and encodes the video frames with the privacy protection; encoding the video frame of which the privacy protection type is unprotected in the composite video stream to obtain an encoded video data stream, wherein the encoding process comprises the following steps:

for a video frame to be coded currently in the composite video stream, if the privacy protection type of the video frame in the configuration information is unprotected, the control module configures the privacy processing module to be closed, and if the privacy protection type of the video frame in the configuration information is protected, the control module configures the privacy processing module to be opened; if the interframe coding reference mode of the video frame is referable, configuring the coding module to set the video frame as a reference frame, and if the interframe coding reference mode of the video frame is unreferenceable, configuring the coding module to set the video frame as a non-reference frame;

the privacy processing module performs privacy processing on a video frame to be coded currently in the composite video stream under the condition of being opened, wherein the privacy processing module does not perform the privacy processing on the video frame to be coded currently under the condition of being closed;

and the coding module codes the video frame output by the privacy processing module and sets the currently coded video frame as a reference frame or a non-reference frame according to the configuration of the control module.

In a possible implementation manner, the video coding device further includes a memory, and the memory is used for storing the video stream to be processed and the reconstructed frame data of the reference frame.

In a possible embodiment, for each video frame in the video stream to be processed, configuring the video frame as a privacy protection frame and an original frame, respectively, to obtain a composite video stream, including:

for each video frame in the video stream to be processed, obtaining an M +1 frame video frame with the same content as the video frame based on the video frame, setting the frame type attribute of the M frame video frame as a privacy protection frame type to obtain an M privacy protection frame, and setting the frame type attribute of the remaining frame video frame as an original frame type to obtain an original frame, wherein M is the number of the privacy protection types;

and interweaving the privacy protection frame and the original frame corresponding to each video frame in the video stream to be processed into a composite video stream.

In one possible embodiment, the method further comprises:

the processor reads data with a preset unit data amount from the coded video data stream every time, and for the currently read data, if the data is data of an original frame, the currently read data is encrypted, the encrypted data is packaged, and if the data is data of a privacy protection frame, the data is directly packaged, so that a data stream to be sent is obtained, wherein the data stream to be sent comprises the packaged data of the original frame and the packaged data of the privacy protection frame;

and respectively sending the data stream to be sent to a common authority user and a high-level authority user.

In one possible embodiment, the method further comprises:

the processor reads data with preset unit data amount from the coded video data stream every time, and obtains a first code stream and a second code stream which have the same content with the currently read data based on the currently read data;

if the current first code stream is data of an original frame, encrypting the current first code stream, and packaging the encrypted data, and if the current first code stream is data of a privacy protection frame, directly packaging the current first code stream to obtain a data stream to be sent, wherein the data stream to be sent comprises packaging data of the original frame and packaging data of the privacy protection frame;

sending the data stream to be sent to a high-level authority user;

if the current second code stream is data of an original frame, discarding the data, and if the current second code stream is data of a privacy protection frame, encapsulating the current second code stream to obtain a privacy protection data stream, wherein the privacy protection data stream comprises encapsulated data of the privacy protection frame;

and sending the privacy protection data stream to a common authority user.

In a second aspect, an embodiment of the present application provides a video encoding apparatus, which includes a processor, a video encoder, and a memory;

the memory is used for storing a video stream to be processed;

the processor is configured to acquire a video stream to be processed, and configure each video frame in the video stream to be processed into a privacy protection frame and an original frame, respectively, to obtain a composite video stream; setting the interframe coding reference mode of the privacy protection frame as referable and the privacy protection type as protection, setting the interframe coding reference mode of the original frame as unreferenceable and the privacy protection type as unprotected, and thus obtaining configuration information; sending the composite video stream and the configuration information to the video encoder;

the video encoder is used for performing privacy protection on the video frames with the privacy protection types being protected in the composite video stream based on the configuration information and encoding the video frames with the privacy protection; coding the video frame of which the privacy protection type is unprotected in the composite video stream to obtain a coded video data stream, wherein the interframe coding reference mode is that a referable video frame is allowed to be used as a reference frame for coding other video frames; inter-coded reference modes are video frames that are not referable and do not allow reference frames for encoding as other video frames.

the control module is configured to, for a video frame to be currently encoded in the composite video stream, configure the privacy processing module to be turned off if the privacy protection type of the video frame in the configuration information is unprotected, and configure the privacy processing module to be turned on if the privacy protection type of the video frame in the configuration information is protected; if the interframe coding reference mode of the video frame is referable, configuring the coding module to set the video frame as a reference frame, and if the interframe coding reference mode of the video frame is unreferenceable, configuring the coding module to set the video frame as a non-reference frame;

the privacy processing module is used for carrying out privacy processing on a video frame to be coded currently in the composite video stream under the condition of opening, wherein the privacy processing module does not execute the privacy processing aiming at the video frame to be coded currently under the condition of closing;

and the coding module is used for coding the video frame output by the privacy processing module and setting the currently coded video frame as a reference frame or a non-reference frame according to the configuration of the control module.

In a possible implementation, the memory is further configured to store reconstructed frame data of the video stream to be processed and the reference frame.

In a possible implementation, the processor is specifically configured to: for each video frame in the video stream to be processed, obtaining an M +1 frame video frame with the same content as the video frame based on the video frame, setting the frame type attribute of the M frame video frame as a privacy protection frame type to obtain an M privacy protection frame, and setting the frame type attribute of the remaining frame video frame as an original frame type to obtain an original frame, wherein M is the number of the privacy protection types; and interweaving the privacy protection frame and the original frame corresponding to each video frame in the video stream to be processed into a composite video stream.

In one possible implementation, the processor is further configured to: reading data with a preset unit data amount from the coded video data stream every time, encrypting the currently read data and packaging the encrypted data if the data is data of an original frame aiming at the currently read data, and directly packaging the data if the data is data of a privacy protection frame so as to obtain a data stream to be sent, wherein the data stream to be sent comprises packaged data of the original frame and packaged data of the privacy protection frame; and respectively sending the data stream to be sent to a common authority user and a high-level authority user.

In one possible implementation, the processor is further configured to: reading data with preset unit data amount from the coded video data stream every time, and obtaining a first code stream and a second code stream which have the same content with the currently read data based on the currently read data; if the current first code stream is data of an original frame, encrypting the current first code stream, and packaging the encrypted data, and if the current first code stream is data of a privacy protection frame, directly packaging the current first code stream to obtain a data stream to be sent, wherein the data stream to be sent comprises packaging data of the original frame and packaging data of the privacy protection frame; sending the data stream to be sent to a high-level authority user; if the current second code stream is data of an original frame, discarding the data, and if the current second code stream is data of a privacy protection frame, encapsulating the current second code stream to obtain a privacy protection data stream, wherein the privacy protection data stream comprises encapsulated data of the privacy protection frame; and sending the privacy protection data stream to a common authority user.

In a third aspect, an embodiment of the present application provides a video decoding method, where the method includes:

obtaining a video stream to be decoded, wherein the video stream to be decoded comprises: packaging data of an original frame and packaging data of a privacy protection frame;

under the condition that a decoding key for an original frame is acquired, decoding encapsulated data of the original frame in the video stream to be decoded by using the decoding key to obtain the original frame;

and decoding the encapsulated data of the privacy protection frame in the video stream to be decoded to obtain the privacy protection frame under the condition that a decoding key for the original frame is not acquired.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, and the computer program, when executed by a processor, implements a video encoding method or a video decoding method as described in any of the present application.

The embodiment of the application has the following beneficial effects:

the video coding method, the video coding device and the storage medium provided by the embodiment of the application are characterized in that the video coding device comprises a processor and a video encoder, wherein the processor acquires a video stream to be processed, and configures each video frame in the video stream to be processed into a privacy protection frame and an original frame respectively to obtain a composite video stream; setting the interframe coding reference mode of the privacy protection frame as referable and the privacy protection type as protection, and setting the interframe coding reference mode of the original frame as unreferenceable and the privacy protection type as unprotected, thereby obtaining configuration information; the processor sends the composite video stream and the configuration information to the video encoder; the video encoder carries out privacy protection on the video frames with the privacy protection type being protected in the composite video stream based on the configuration information, and encodes the video frames after the privacy protection; coding a video frame of which the privacy protection type is unprotected in the composite video stream to obtain a coded video data stream, wherein an interframe coding reference mode is a referable video frame and is allowed to be used as a reference frame for coding other video frames; inter-coded reference modes are video frames that are not referable and do not allow reference frames for encoding as other video frames.

The method comprises the steps that a path of composite video stream which simultaneously comprises an original frame and a privacy protection frame is coded, the original frame refers to the privacy protection frame for coding, and the original frame is not used as a reference frame during coding, so that the original frame does not need to be stored, the video stream is compounded and time-domain layered in pre-processing, memory consumption required by reference frame storage is reduced for time-domain layered interleaving reference on a reference relation, the correlation between the privacy protection frame and the original frame is effectively utilized, the privacy protection frame corresponding to the privacy protection frame reference is coded, most original reference areas are completely the same, the number of coded code streams is favorably reduced greatly, and the code stream storage and transmission cost is reduced; moreover, privacy protection processing is realized in the video encoder through hardware logic, so that the processing pressure of a processor can be reduced, the memory read-write bandwidth of image data is reduced, and the encoding efficiency is greatly improved. Of course, it is not necessary for any product or method of the present application to achieve all of the above advantages at the same time.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art according to the drawings.

Fig. 1 is a schematic diagram of a video encoding method according to an embodiment of the present application;

FIG. 2 is a diagram illustrating a structure of a code stream in the related art;

FIG. 3 is a diagram illustrating a related art SVC-TX2 model and SVC-TX4 model inter-frame coding timing reference relationship;

fig. 4 is a schematic diagram illustrating a video stream transmission process in a video encoding method according to an embodiment of the present application;

FIG. 5 is a schematic diagram of a flowchart of a pre-coding processing unit according to an embodiment of the present disclosure;

FIG. 6a is a schematic structural diagram of an encoding end and a decoding end according to an embodiment of the present application;

FIG. 6b is a schematic structural diagram of an encoding end and a decoding end according to an embodiment of the present application;

FIG. 6c is a block diagram of a video encoder according to an embodiment of the present application;

FIG. 7a is a diagram illustrating a memory access scenario in the related art;

FIG. 7b is a diagram illustrating a memory access scenario in an embodiment of the present application;

fig. 8 is a schematic diagram of a working flow chart of an encoding post-processing unit according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments that can be derived by one of ordinary skill in the art from the description herein are intended to be within the scope of the present disclosure.

In order to reduce resource consumption in an encoding process, an embodiment of the present application provides a video encoding method applied to a video encoding device, where the video encoding device includes a processor and a video encoder, and referring to fig. 1, the method includes:

s101, the processor acquires a video stream to be processed, and configures each video frame in the video stream to be processed into a privacy protection frame and an original frame respectively to obtain a composite video stream; setting the interframe coding reference mode of the privacy protection frame as referable and the privacy protection type as protection, setting the interframe coding reference mode of the original frame as unreferenceable and the privacy protection type as unprotected, and thus obtaining configuration information; the processor sends the composite video stream and the configuration information to the video encoder.

The video encoding method in the embodiment of the present application is applied to a video encoding device, and therefore can be implemented by the video encoding device, specifically, the video encoding device may be a video camera or a hard disk video recorder, and the like. The video stream to be processed is any video stream needing to be encoded.

The image contents of the privacy protection frame and the original frame configured by the video frame are the same as the image contents of the video frame, and the privacy protection frame and the original frame are different in type identification.

In one example, each video frame in the video stream to be processed may be processed and configured twice, interleaved into a composite video stream, for example, the first time processed into a privacy-preserving frame and the second time into an original frame, and also interleaved with each other in a reference system at the time of information configuration.

In other possible embodiments, each video frame in the video stream to be processed may be processed and configured multiple times to be interleaved into a composite video stream, for example, a privacy-preserving frame for protecting a first object is processed for the first time, a privacy-preserving frame for protecting a second object is processed for the second time, \ 8230 \ 8230;, an original frame is processed for the last time, and the reference relations are also interleaved with each other. In one example, the first object and the second object are different types of objects, for example, the first object is a human face, the second object is a license plate number, and the like.

step one, aiming at each video frame in the video stream to be processed, obtaining an M +1 frame video frame with the same content as the video frame based on the video frame, setting the frame type attribute of the M frame video frame as a privacy protection frame type to obtain an M privacy protection frame, and setting the frame type attribute of the remaining frame video frame as an original frame type to obtain an original frame, wherein M is the number of the privacy protection types.

And step two, interweaving privacy protection frames and original frames corresponding to all video frames in the video stream to be processed into a composite video stream.

Frame type attributes include, but are not limited to, NALU type, temporal priority, other protocol specified customizable fields, and the like. Taking the code stream of the h.265 standard as an example, the code stream structure can be as shown in fig. 2, and in the first layer, each Frame represents the code stream encoded by each Frame. In the second layer, a frame may be divided into one or more slices, and the code stream of each slice is an NAL unit, so that a frame of code stream may be composed of one or more NAL units. In the third layer, the Nal unit consists of NALU header and RBSP. The NALU header mainly carries the content characteristics of the current NAL unit, and the RBSP is the NALU payload and contains the main video compression data (slice header + slice data). In the fourth Layer, the NALU header consists of a reserved bit (F), a NALU type (NALU _ type), a Layer level (NLI) and a time domain level (NTI), and generally, the NALU header can directly and quickly judge the type of a frame to which a current segment code stream belongs from the NALU type to perform operations of frame extraction and the like of the time domain level; the lowest layer belongs to the specific values and numerical meanings of NALU types. For the h.265 standard code stream, the privacy protection frame type or the original frame type may be recorded in the NALU type.

S102, the video encoder carries out privacy protection on the video frames with the privacy protection types being protected in the composite video stream based on the configuration information, and encodes the video frames with the privacy protection; coding the video frame of which the privacy protection type is unprotected in the composite video stream to obtain a coded video data stream, wherein an interframe coding reference mode is a referable video frame and is allowed to be used as a reference frame for coding other video frames; inter-coded reference modes are video frames that are not referable and do not allow reference frames for encoding as other video frames.

The privacy protection can be performed on the video frame according to a preset privacy protection rule. The preset privacy protection rule can be set by self according to actual conditions, the sources of the privacy protection area coordinates include but are not limited to direct configuration of users, automatic detection of intelligent algorithms and the like, and the number of the privacy protection areas in each frame of privacy protection frame is not limited to 1. The privacy processing method for the privacy protection area includes but is not limited to scrambling, or blocking.

In one example, the setting of the inter-coding reference mode of the video frame may be implemented by configuring the basic register parameter and the reference relation register parameter. For example, the basic register parameter may be configured generally for the privacy protection frame, and when the reference relation register parameter is configured, a temporal scalable reference model supporting bitstream framing, such as an SVC-TX2 model or an SVC-TX4 model, is selected, and then the currently processed privacy protection frame is set as a temporal basic layer or an intermediate enhancement layer that can be referred to. The schematic diagrams of the SVC-TX2 model and the SVC-TX4 model may be that, as shown in fig. 3, the enhancement layer needs to refer to the base layer for coding and decoding, the SVC-TX2 model may refer to the base layer forward or backward, and the SVC-TX4 model may refer to the base layer forward or backward, where the base layer is identified by a light color, and the enhancement layer is represented by a dark color. And aiming at the original frame, the image content of the original frame is not subjected to privacy processing, a basic register is subjected to general configuration when a register parameter is configured, a time domain reference model which is the same as the privacy protection frame is selected when a reference relation register parameter is configured, and then the original frame is set as a time domain enhancement layer which cannot be referred to.

The video coding standards adopted by the coding include, but are not limited to, h.264, h.265, AVS2, and the like. The inter-coding reference mode of the original frame is non-referable and does not allow reference frames for encoding as other video frames, i.e. the original frame is an enhancement layer. The inter-frame coding reference mode of the privacy protection frame is referable, and allows a reference frame for coding as other video frames, that is, the privacy protection frame may be a base layer or an enhancement layer.

the video encoder carries out privacy protection on the video frames with the privacy protection types in the composite video stream based on the configuration information, and encodes the video frames with the privacy protection types; encoding the video frame of which the privacy protection type is unprotected in the composite video stream to obtain an encoded video data stream, wherein the encoding process comprises the following steps:

step one, aiming at a video frame to be coded currently in the composite video stream, if the privacy protection type of the video frame in the configuration information is unprotected, the control module configures the privacy processing module to be closed, and if the privacy protection type of the video frame in the configuration information is protected, the control module configures the privacy processing module to be opened; if the interframe coding reference mode of the video frame is referable, configuring the coding module to set the video frame as a reference frame, and if the interframe coding reference mode of the video frame is non-referable, configuring the coding module to set the video frame as a non-reference frame;

secondly, the privacy processing module carries out privacy processing on the video frame to be coded currently in the composite video stream under the condition of being opened, wherein the privacy processing module does not carry out the privacy processing on the video frame to be coded currently under the condition of being closed;

and thirdly, the encoding module encodes the video frame output by the privacy processing module and sets the currently encoded video frame as a reference frame or a non-reference frame according to the configuration of the control module.

In a possible implementation, the video encoding apparatus further includes a memory for storing reconstructed frame data of the reference frame and the video stream to be processed.

In one example, the memory may be used to store the encoded code stream and other custom data in addition to the video stream to be processed and the reconstructed frame data.

In the embodiment of the application, a path of composite video stream simultaneously comprising an original frame and a privacy protection frame is coded, the original frame refers to the privacy protection frame for coding, and the original frame is not used as a reference frame during coding, so the original frame does not need to be stored, the video stream is compounded and time-domain layered in preprocessing, the memory consumption required by the storage of the reference frame is reduced for time-domain layered interleaving reference on a reference relation, the correlation between the privacy protection frame and the original frame is effectively utilized, the original frame refers to the corresponding privacy protection frame for coding, most of reference areas are completely the same, the number of coded code streams is favorably and greatly reduced, and the storage and transmission cost of the code streams is reduced; moreover, the privacy protection processing is realized through hardware logic in the video encoder, the processing pressure of a processor can be reduced, the memory read-write bandwidth of image data is reduced, and the encoding efficiency is greatly improved.

After obtaining the encoded video data stream, the encoded video data stream also needs to be sent in a packaged manner, and in a possible implementation manner, the method further includes:

the processor reads data with a preset unit data amount from the coded video data stream every time, and for the currently read data, if the data is data of an original frame, the currently read data is encrypted, the encrypted data is packaged, and if the data is data of a privacy protection frame, the data is directly packaged, so that a data stream to be sent is obtained, wherein the data stream to be sent comprises the packaged data of the original frame and the packaged data of the privacy protection frame; and respectively sending the data stream to be sent to a common authority user and a high-level authority user.

The client of the ordinary authority user has authority limit, and only the privacy protection frame can be decoded and displayed, but the original frame cannot be displayed.

Reading the encoded video data stream output by the video encoder, and reading the code stream data of the preset unit data amount each time, wherein the granularity of the preset unit data amount can be set by self according to the actual situation, including but not limited to Byte level, NALU level or frame level. In one example, header information of current data may be parsed to obtain a type of a frame to which the current data belongs, for example, in h.264 and h.265 standards, whether the current data is raw frame data or privacy protection frame data is determined according to a NALU type or a time domain level in NALU header information. If the data is original frame data, the original frame code stream is encrypted independently, the encryption mode includes but is not limited to AES encryption and the like, and the encrypted data is encapsulated in an application layer. If the data is privacy protection frame data, directly performing application layer encapsulation; in one example, global encryption is selectively performed, and then the global encryption is packaged and output to a decoding end.

In one possible embodiment, referring to fig. 4, the method further comprises:

s201, the processor reads data with preset unit data amount from the coded video data stream every time, and obtains a first code stream and a second code stream which have the same content with the currently read data based on the currently read data;

s202, if the current first code stream is data of an original frame, encrypting the current first code stream, and packaging the encrypted data, and if the current first code stream is data of a privacy protection frame, directly packaging the current first code stream, so as to obtain a data stream to be sent, wherein the data stream to be sent comprises packaging data of the original frame and packaging data of the privacy protection frame;

s203, sending the data stream to be sent to a high-level authority user;

s204, if the current second code stream is the data of the original frame, discarding the data, and if the current second code stream is the data of the privacy protection frame, encapsulating the current second code stream to obtain a privacy protection data stream, wherein the privacy protection data stream comprises the encapsulated data of the privacy protection frame;

s205, sending the privacy protection data stream to a common authority user.

In the prior art, the frame extraction operation is to adapt to different bandwidth situations, for example, when the network bandwidth is smooth, frame extraction is not performed, a video stream of 60FPS (frames per second) is used, when the network bandwidth is congested, 1/2 video frames are extracted, i.e., a video stream of 30FPS is used, when the network bandwidth is highly congested, 1/4 video frames are extracted, i.e., a video stream of 15FPS is used, so that the normal playing of the video is ensured under the condition of sacrificing the frames per second.

In the embodiment of the application, the data of the privacy protection frame is extracted by using a frame extraction mode and is packaged into the privacy protection data stream, and the privacy protection data stream received by the ordinary authority user only contains the data of the privacy protection frame, so that the ordinary authority user can be prevented from acquiring the original frame, the privacy safety is ensured, and meanwhile, the bandwidth of the ordinary authority user is saved.

An embodiment of the present application further provides a video decoding method, where the method includes:

step one, obtaining a video stream to be decoded, wherein the video stream to be decoded comprises: the encapsulation data of the original frame and the encapsulation data of the privacy protection frame.

The video stream to be decoded may be a data stream to be transmitted sent by the encoding end in the above embodiment.

And secondly, decoding the encapsulated data of the original frame in the video stream to be decoded by using the decoding key under the condition of acquiring the decoding key aiming at the original frame to obtain the original frame.

And the decoding end decodes the encapsulated data of the original frame in the video stream to be decoded by using the decoding key to obtain the original frame under the condition of acquiring the decoding key of the original frame.

And thirdly, decoding the encapsulated data of the privacy protection frame in the video stream to be decoded to obtain the privacy protection frame under the condition that the decoding key for the original frame is not acquired.

The normal authority user does not know the decoding key, so that only the privacy protection frame can be decoded.

An embodiment of the present application further provides a video encoding apparatus, for example, as shown in fig. 6a, the video encoding apparatus is shown as an encoding end in the figure, and includes an encoding preprocessing unit, a video encoder, and an encoding postprocessing unit. The video coding device interleaves the privacy protection frame and the original frame into a composite video stream, then performs video coding, and divides the composite video stream into a base layer and an enhancement layer by using temporal hierarchy or temporal hierarchy reference relationship, wherein the enhancement layer may be one layer or multiple layers. The base layer is a privacy-preserving frame available for reference, the middle enhancement layer is another privacy-preserving frame available for reference, and the highest enhancement layer is an original frame unavailable for reference. And judging the code stream type when the coded code stream is subjected to post-processing, and performing privacy protection encryption operation again if the coded code stream is an original frame code stream, or else, performing no privacy protection encryption on the privacy protection frame code stream. The authority management can be realized by a decoding end which comprises an authority management unit, an advanced user decoding unit, a common user decoding unit and a display unit. The authority management unit judges the authority of the current user according to whether the privacy protection key is input or not, the advanced authority user with the key decodes the composite video stream formed by interweaving the privacy protection frame and the original frame after decryption, then extracts the privacy protection frame and only displays the original frame; and the ordinary authority user without the key only decodes the privacy protection frame video stream and directly displays the privacy protection frame video stream.

The video coding device comprises a pre-coding processing unit, a video coder and a post-coding processing unit.

An encoding preprocessing unit: processing and allocating each frame of video frames to be output at least twice to obtain a frame of original frames and at least one frame of privacy protection frames, and interleaving the frames into a composite video stream. Wherein the privacy-preserving frame is set to allow coding references for other video frames, and the original frame is set to disallow coding references for other video frames. In one example, taking two times as an example, the first time is processed into a privacy protection frame, the second time is an original frame, and the frames are also interleaved with each other in a reference system during information configuration. In one example, each frame may be processed and configured multiple times to be interleaved into a composite video stream, where the first time is to process privacy-protected frames protecting a first object, the second time is to process privacy-protected frames protecting a second object, \ 8230 \ 8230;, and the last time is to process original frames, which are also interleaved with each other in the reference relationship.

A video encoder: the video encoder in the present Application is a hardware encoder, and may be implemented by at least one of hardware such as an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and an SoC (System on Chip), and is configured to perform privacy protection and encoding processing on a video frame, and the supported standard protocols include, but are not limited to, h.264, h.265, h.266, AVS2, AVS3, AV1, and VP 9. Implementations of privacy protection include, but are not limited to, image coding, image occlusion, image scrambling, and the like. In the embodiment of the present application, for example, as shown in fig. 6b, a privacy protection function is implemented by a video encoder, where the video encoder includes a privacy processing module for performing privacy protection on a video frame; the video encoder also includes an encoding module for encoding the video frames.

An encoding post-processing unit: the unit comprises a code stream frame extracting component inside, the traditional usage is that frame extraction is carried out when the network bandwidth is in tension, in the application, the code stream frame extracting component is used for extracting frames and then judging the video frame type of the code stream, if the video frame type is an original frame code stream, privacy protection encryption operation is carried out again, the encryption mode includes but is not limited to AES encryption and the like, and if the video frame type is a privacy protection frame code stream, privacy protection encryption is not required or another set of secret key is used for encryption. After the privacy protection encryption is carried out on the code stream, the primary global encryption can be selectively carried out, so that the data stream to be sent is obtained, the security of the code stream in the transmission process is ensured, and then the data stream to be sent is output.

In one example, the pre-coding processing unit and the post-coding processing unit are implemented by a processor based on software logic.

The functions of the units are specifically described below:

the basic function of the encoding pre-processing unit is to transmit register parameters and image YUV data for an encoder according to encoding attributes configured by a user and an acquired input video sequence. The coding pre-processing unit of the patent also needs to process and configure input frames for multiple times, interweaves the input frames into a composite video stream of privacy protection frames and original frames, and configures register parameters into a reference relation capable of extracting frames in a time domain. The processing flow of the encoding pre-processing is shown in fig. 5, and the specific processing flow is described as follows:

step 1, reading the encoding attribute configured by the user from the user interface, and converting the abstract encoding attribute into a concrete parameter value.

And 2, acquiring an original frame image of a frame as an input frame aiming at the video stream to be processed.

And 3, configuring the basic register parameters and the reference relation register parameters of the current processing frame. The basic register parameters are configured generally, and when the reference relationship register parameters are configured, a time-domain scalable reference model supporting the bitstream framing is selected, including but not limited to an SVC-TX2 model, an SVC-TX4 model, and the like.

And 4, outputting the configured register parameters to a video encoder, and outputting the processed image content data, wherein the image content data format includes but is not limited to YUV420, YUV422 and other formats.

And 5, if the current input frame is configured for the last time, ending the processing of the current input frame, and otherwise, jumping to the step 3.

In the embodiment of the application, in the composite video stream, the privacy protection frame is skipped to refer to other privacy protection frames, and the original frame refers to the adjacent privacy protection frame.

The video encoder has the function of sequentially carrying out video encoding on each frame of image in the composite video stream sequence by adopting a video encoding standard to generate a compressed code stream. The standards supported by the video encoder include, but are not limited to, h.264, h.265, AVS2, etc. Video coding standard protocols H.264, H.265, AVS2 and the like all adopt a hybrid coding framework to divide an image into small blocks for coding, and the small blocks are collectively called coding blocks in the application.

The hardware structure of the video encoder can be as shown in fig. 6c, the sub-modules in the hardware encoder are pipelined, and the process flow of the whole hardware system viewed by a single coding block is as follows:

the control module reads the configuration information of the encoder register and outputs control parameters to other modules. Judging whether the current frame is an I frame or not, and only reading the image data of the current frame from the DDR without a reference frame; otherwise, reading the current frame image data from the DDR, and simultaneously reading the required reference frame data from the DDR according to the time domain reference relation.

The privacy processing module judges whether to perform privacy protection image processing on the coding block according to the register switch, and performs coding block image processing (including but not limited to image coding, image shielding, image scrambling and other processing modes) when the register switch is opened; otherwise, the Bypass mode is used for not processing the image of the coding block, and the data are directly output to a rear-stage module.

The coding module performs intra-frame prediction, inter-frame prediction, transform quantization, inverse transform inverse quantization, loop filtering, entropy coding and other sub-modules according to a coding standard by taking a coding block as a unit, and the part can be referred to a standard coder and is not described herein again.

Outputting the compressed code stream for encoding and post-processing, and writing the reconstructed image Data back to a memory, such as a DDR (Double Data Rate) memory, for reference of a subsequent frame.

The scheme of privacy protection image processing realized by the hardware logic of the video encoder not only can reduce the load of a processor and improve the system efficiency, but also has the advantage of reducing the memory read-write bandwidth.

For the case of dividing the time domain hierarchy into two layers, namely the base layer and the enhancement layer, the scheme of privacy protection by the processor accesses the memory as shown in fig. 7a, which shows that 3 times of writing and 3 times of reading are performed on the memory; in the present application, a schematic diagram of memory access by means of video encoder hardware is shown in fig. 7b, and 2 times of writing and 1 time of reading are performed on a memory, so that the memory read-write bandwidth is reduced by 50%.

The processing flow of the post-coding processing unit may be as shown in fig. 8, and includes:

step 1, reading a coded video data stream output by a video coder; the granularity of each read code stream includes but is not limited to Byte level, NALU level or frame level.

And 2, analyzing the code stream header information of the currently read data to acquire the type of the frame to which the current code stream belongs. Such as the NALU type or time domain hierarchy in the NALU header information in the h.264 and h.265 standards.

Step 3, judging the type of the current code stream data, and if the type of the current code stream data is the original frame code stream data, executing step 4; otherwise, the data is the privacy protection frame code stream data, and the step 5 is skipped.

And 4, independently encrypting the original frame code stream data in a mode including but not limited to AES encryption and the like.

And 5, performing application layer packaging on the current code stream data, optionally performing global encryption, and then packaging and outputting to a decoding end.

And 6, finishing the processing of the current code stream data.

In a possible implementation manner, the post-encoding processing unit may further send the code stream output by the video encoder in two paths, where one path performs frame extraction on the code stream and then packs the frame to output to the ordinary right user, and the other path packs the frame to output to the advanced right user. The code stream frame extraction operation is generally performed under the condition of network bandwidth shortage, and the code stream frame extraction performed in the application is to extract the original frame of the enhancement layer to generate the privacy protection data stream, which is an innovative use of the code stream frame extraction component in the application. The processing flow comprises the following steps:

And 2, analyzing the code stream header information of the currently read data to acquire the type of the frame to which the current code stream belongs. For example, the NALU is judged according to the NALU type or the time domain hierarchy in the NALU header information in the H.264 and H.265 standards.

And 3, dividing the current code stream into two paths for processing, wherein one path is connected with a common authority user, executing the step 4, the other path is connected with a high-level authority user, and jumping to the step 6.

And 4, butting the common authority user, executing the step 5 if the current code stream is the privacy protection frame code stream according to the frame type of the code stream, otherwise, performing frame extraction operation without sending the original frame code stream, and directly jumping to the step 7.

And 5, performing application layer packaging on the privacy protection frame code stream, packaging and outputting the privacy protection frame code stream to a common authority user, and jumping to the step 7.

And 6, butting the advanced authority users, directly packaging the code stream by an application layer, and packaging and outputting the code stream to the advanced authority users.

And 7, finishing the processing of the current code stream.

The method comprises the steps of coding a path of composite video stream simultaneously comprising an original frame and a privacy protection frame, coding the original frame by referring to the privacy protection frame, and coding the original frame not serving as a reference frame during coding, so that the original frame does not need to be stored; moreover, the privacy protection processing is realized through hardware logic in the video encoder, the processing pressure of a processor can be reduced, the memory read-write bandwidth of image data is reduced, and the encoding efficiency is greatly improved.

In an example, the pre-coding processing unit and the post-coding processing unit are implemented by a processor based on software logic, and the embodiment of the present application further provides a video coding apparatus, where the video coding apparatus includes a processor, a video encoder, and a memory;

the memory is used for storing a video stream to be processed;

the processor is used for acquiring a video stream to be processed, and configuring each video frame in the video stream to be processed into a privacy protection frame and an original frame respectively to obtain a composite video stream; setting the interframe coding reference mode of the privacy protection frame as referable and the privacy protection type as protection, and setting the interframe coding reference mode of the original frame as unreferenceable and the privacy protection type as unprotected, thereby obtaining configuration information; sending the composite video stream and the configuration information to the video encoder;

the video encoder is used for carrying out privacy protection on the video frames with the privacy protection types in the composite video stream based on the configuration information and encoding the video frames with the privacy protection; coding the video frame of which the privacy protection type is unprotected in the composite video stream to obtain a coded video data stream, wherein the interframe coding reference mode is that a referable video frame is allowed to be used as a reference frame for coding other video frames; inter-coded reference modes are video frames that are not referable and do not allow reference frames for encoding as other video frames.

the control module is configured to, for a video frame to be currently encoded in the composite video stream, configure the privacy processing module to be turned off if the privacy protection type of the video frame in the configuration information is unprotected, and configure the privacy processing module to be turned on if the privacy protection type of the video frame in the configuration information is protected; if the interframe coding reference mode of the video frame is referable, configuring the coding module to set the video frame as a reference frame, and if the interframe coding reference mode of the video frame is non-referable, configuring the coding module to set the video frame as a non-reference frame;

In one possible implementation, the processor is further configured to: reading data of a preset unit data volume from the coded video data stream each time, and obtaining a first code stream and a second code stream which have the same content with the currently read data based on the currently read data; if the current first code stream is data of an original frame, encrypting the current first code stream, and packaging the encrypted data, and if the current first code stream is data of a privacy protection frame, directly packaging the current first code stream to obtain a data stream to be sent, wherein the data stream to be sent comprises packaged data of the original frame and packaged data of the privacy protection frame; sending the data stream to be sent to a high-level authority user; if the current second code stream is data of an original frame, discarding the data, and if the current second code stream is data of a privacy protection frame, encapsulating the current second code stream to obtain a privacy protection data stream, wherein the privacy protection data stream comprises encapsulated data of the privacy protection frame; and sending the privacy protection data stream to a common authority user.

An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements any video encoding method in the present application.

In yet another embodiment provided herein, there is also provided a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the video encoding methods of the present application.

The method, the device, the storage medium, the computer program product and the like of the embodiments of the application can be applied to high-altitude parabolic monitoring scenes: in the application of high-altitude parabolic monitoring scenes, an electronic monitor is generally installed on the ground to monitor each floor of a high-rise building in real time from bottom to top. Because the installation height of the electronic monitor is generally approximate to the height of a low-rise building, the condition inside the low-rise building can be shot during real-time monitoring, and the privacy of residents on the low-rise building is leaked. By adopting the method, the device, the storage medium, the computer program product and the like of the embodiment of the application, the coordinates of the privacy protection area are directly configured according to the installation position of the electronic monitor, the security monitoring room is defined as a common authority user, the storage with the key is a high-level authority user, the privacy protection code stream output by the electronic monitor is decoded and then displayed in real time in the monitoring room, and the complete code stream is stored in the storage with the key. Therefore, the privacy of the users in the low-rise building is protected, and the requirement of checking the original video by using the secret key after the high-altitude parabolic event occurs is met.

The method, the device, the storage medium, the computer program product and the like of the embodiment of the application can be applied to security monitoring scenes: by applying the method, the equipment, the storage medium, the computer program product and the like of the embodiment of the application to the scenes of face recognition, behavior analysis and the like in the field of video monitoring, the intelligent algorithm automatically outputs the region coordinates needing privacy protection after face recognition, and codes or scrambles the face region appearing in the monitored video. The electronic monitor outputs the privacy protection code stream, the privacy protection code stream is decoded and then displayed in real time, the complete code stream is stored in a memory with a secret key, and the original video is checked during intervening investigation.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, digital subscriber line) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that includes one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), among others.

It should be noted that, in the present disclosure, technical features in various alternatives can be combined to form an alternative, and the alternative is within the scope of the disclosure. Relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.

The embodiments in the present specification are described in a related manner, each embodiment focuses on differences from other embodiments, and the same and similar parts in the embodiments are referred to each other.

The above description is only for the preferred embodiment of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A video encoding method applied to a video encoding apparatus, the video encoding apparatus comprising a processor and a video encoder, the method comprising:

the video encoder carries out privacy protection on the video frames with the privacy protection types being protected in the composite video stream based on the configuration information, and encodes the video frames with the privacy protection; coding the video frame of which the privacy protection type is unprotected in the composite video stream to obtain a coded video data stream, wherein an interframe coding reference mode is a referable video frame and is allowed to be used as a reference frame for coding other video frames; inter-coded reference modes are video frames that are not referable and do not allow reference frames for encoding as other video frames.

2. The method of claim 1, wherein the video encoder comprises a control module, a privacy processing module, and an encoding module;

3. The method of claim 2, wherein the video encoding device further comprises a memory for storing the video stream to be processed and reconstructed frame data of the reference frame.

4. The method according to claim 1, wherein the configuring, for each video frame in the video stream to be processed, the video frame as a privacy protection frame and an original frame respectively to obtain a composite video stream comprises:

5. The method of claim 1, further comprising:

6. The method of claim 1, further comprising:

if the current first code stream is data of an original frame, encrypting the current first code stream, and packaging the encrypted data, and if the current first code stream is data of a privacy protection frame, directly packaging the current first code stream to obtain a data stream to be sent, wherein the data stream to be sent comprises packaged data of the original frame and packaged data of the privacy protection frame;

sending the data stream to be sent to a high-level authority user;

and sending the privacy protection data stream to a common authority user.

7. A video encoding device, characterized in that the video encoding device comprises a processor, a video encoder and a memory;

the memory is used for storing a video stream to be processed;

8. The apparatus of claim 7, wherein the video encoder comprises a control module, a privacy processing module, and an encoding module;

9. The apparatus of claim 8, wherein the memory is further configured to store reconstructed frame data of the video stream to be processed and the reference frame.

10. The device of claim 7, wherein the processor is specifically configured to: for each video frame in the video stream to be processed, obtaining an M +1 frame video frame with the same content as the video frame based on the video frame, setting the frame type attribute of the M frame video frame as a privacy protection frame type to obtain an M privacy protection frame, and setting the frame type attribute of the remaining frame video frame as an original frame type to obtain an original frame, wherein M is the number of the privacy protection types; and interweaving the privacy protection frame and the original frame corresponding to each video frame in the video stream to be processed into a composite video stream.

11. The device of claim 7, wherein the processor is further configured to: reading data with a preset unit data amount from the coded video data stream every time, encrypting the currently read data and packaging the encrypted data if the data is data of an original frame aiming at the currently read data, and directly packaging the data if the data is data of a privacy protection frame so as to obtain a data stream to be sent, wherein the data stream to be sent comprises packaged data of the original frame and packaged data of the privacy protection frame; and respectively sending the data stream to be sent to a common authority user and a high-level authority user.

12. The device of claim 7, wherein the processor is further configured to: reading data with preset unit data amount from the coded video data stream every time, and obtaining a first code stream and a second code stream which have the same content with the currently read data based on the currently read data; if the current first code stream is data of an original frame, encrypting the current first code stream, and packaging the encrypted data, and if the current first code stream is data of a privacy protection frame, directly packaging the current first code stream to obtain a data stream to be sent, wherein the data stream to be sent comprises packaged data of the original frame and packaged data of the privacy protection frame; sending the data stream to be sent to a high-level authority user; if the current second code stream is data of an original frame, discarding the data, and if the current second code stream is data of a privacy protection frame, encapsulating the current second code stream to obtain a privacy protection data stream, wherein the privacy protection data stream comprises encapsulated data of the privacy protection frame; and sending the privacy protection data stream to a common authority user.

13. A method of video decoding, the method comprising:

obtaining a video stream to be decoded, wherein the video stream to be decoded comprises: packaging data of an original frame and packaging data of a privacy protection frame; wherein the video stream to be decoded is generated according to the method of any of claims 1-6;

14. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method of any one of claims 1-6, 13.