CN115278305B - Video processing method, video processing system, and storage medium - Google Patents
Video processing method, video processing system, and storage medium Download PDFInfo
- Publication number
- CN115278305B CN115278305B CN202210514646.6A CN202210514646A CN115278305B CN 115278305 B CN115278305 B CN 115278305B CN 202210514646 A CN202210514646 A CN 202210514646A CN 115278305 B CN115278305 B CN 115278305B
- Authority
- CN
- China
- Prior art keywords
- video stream
- video
- control instruction
- decoding capability
- frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 32
- 238000012545 processing Methods 0.000 title claims abstract description 20
- 238000004458 analytical method Methods 0.000 claims description 76
- 238000000034 method Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 5
- 230000000694 effects Effects 0.000 abstract description 12
- 238000010586 diagram Methods 0.000 description 9
- 239000000284 extract Substances 0.000 description 5
- 238000000605 extraction Methods 0.000 description 5
- 238000013459 approach Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 230000001276 controlling effect Effects 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234381—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/23418—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234363—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the spatial resolution, e.g. for clients with a lower screen resolution
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440263—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by altering the spatial resolution, e.g. for displaying on a connected PDA
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440281—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by altering the temporal resolution, e.g. by frame skipping
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The application relates to a video processing method, a video processing system and a storage medium, wherein the control instruction is used for instructing front-end equipment to adjust the inter-frame dependency relationship of reference frames in a video stream to obtain a coded first video stream; decoding the first video stream, wherein the first video stream supports the back-end device to decode in at least two video frame sequences, so that the problem that the back-end device cannot support a newly added channel under the condition that the accessed video stream meets the preview and/or playback effects in the related art is solved, and the back-end device can support the newly added channel.
Description
Technical Field
The present application relates to the field of video processing, and in particular, to a video processing method, a video processing system, and a storage medium.
Background
After the front-end equipment collects and generates the video stream, the video stream is pulled to the back-end equipment for analysis. In addition to the analysis of the back-end device, the video stream needs to be previewed or played back by the client device, so that the frame rate of the video stream is always the effect of guaranteeing the previewing or playing back preferentially. Some analysis algorithms have high requirements on frame rate, some analysis algorithms have low requirements on frame rate, and in order to save the performance cost of the analysis algorithms, in general, after decoding a video stream to obtain YUV, the YUV is transmitted to an analysis algorithm module by adopting a frame extraction or scaling mode.
Although the related art saves the analysis algorithm performance overhead, the overall channel of the backend device is much smaller. At a certain allocation ratio, the decoding capability and the analysis capability of the back-end device are relatively matched, i.e. are constrained by analysis specifications, for example, when the back-end device accesses 1080P video stream, the back-end device can support 16 channels to perform analysis (including decoding and analysis, and the frame extraction and scaling methods are applied) at a frame rate of 30 frames/second, when the back-end device accesses 4K video stream, the back-end device only supports 4 channels to perform analysis at a frame rate of 30 frames/second, and if the back-end device continues to add channels again to access video stream, the back-end device cannot support.
Aiming at the problem that in the related art, the backend device cannot support a newly added channel under the condition that the accessed video stream meets the preview and/or playback effects, no effective solution is proposed at present.
Disclosure of Invention
In this embodiment, a video processing method, a video processing system, and a storage medium are provided to solve the problem that in the related art, a backend device cannot support a newly added channel when an accessed video stream satisfies a preview and/or playback effect.
In a first aspect, in this embodiment, there is provided a video processing method applied to a backend device, including:
Transmitting a control instruction to front-end equipment, wherein the control instruction is used for instructing the front-end equipment to adjust the inter-frame dependency relationship of a reference frame in a video stream to obtain a coded first video stream;
decoding the first video stream, wherein the first video stream supports a backend device to decode in at least two sequential video frame sequences.
In some of these embodiments, before sending the control instruction to the head-end equipment, the method further comprises:
determining a current decoding capability and determining a desired decoding capability to be provided when decoding a video stream of the front-end device;
And comparing the current decoding capability with the expected decoding capability, and generating the control instruction according to a comparison result.
In some of these embodiments, the first video stream includes at least one group of pictures, each group of pictures including a plurality of non-key frames, comparing the current decoding capability to the desired decoding capability, and generating the control instruction based on the comparison result includes:
And generating a first control instruction under the condition that the current decoding capability is smaller than the expected decoding capability, wherein the first control instruction is used for indicating the front-end equipment to release inter-frame dependency relationship among partial non-key frames in each picture group of the first video stream.
In some of these embodiments, the method further comprises:
And generating a second control instruction under the condition that the current decoding capability is not less than the expected decoding capability, wherein the second control instruction is used for indicating the front-end equipment to increase the inter-frame dependency relationship among partial non-key frames in each picture group of the first video stream.
In some of these embodiments, where the current decoding capability is not less than the desired decoding capability, generating the second control instruction includes:
determining a maximum decoding capability, and judging whether the maximum decoding capability is greater than the expected decoding capability;
and generating the second control instruction when the maximum decoding capability is judged to be larger than the expected decoding capability.
In some of these embodiments, the method further comprises:
determining a current analysis capability and determining a desired analysis capability to be provided when analyzing the first video stream;
judging whether the current analysis capability is smaller than the expected analysis capability;
and refusing to access the front-end equipment under the condition that the current analysis capability is less than the expected analysis capability.
In some of these embodiments, decoding the first video stream includes:
Extracting part of the reference frames in the first video stream for decoding, and/or decoding all the reference frames in the first video stream.
In a second aspect, in this embodiment, there is provided a video processing method applied to a front-end device, including:
receiving a control instruction, and adjusting the inter-frame dependency relationship of a reference frame in a video stream according to the control instruction to obtain a first video stream;
And transmitting the first video stream to a back-end device, wherein the first video stream supports the back-end device to decode in at least two sequential video frame sequences.
In some embodiments, the first video stream includes at least one group of pictures, each group of pictures including a plurality of non-key frames, and adjusting the inter-frame dependencies of reference frames in the video stream according to the control instruction includes:
releasing the inter-frame dependency relationship between partial non-key frames in each picture group of the first video stream according to a first control instruction; or alternatively
And adding the inter-frame dependency relationship among part of non-key frames in each picture group of the first video stream according to a second control instruction.
In a third aspect, in this embodiment, there is provided a video processing system, including a front-end device and a back-end device, where the front-end device is connected to the back-end device, and the front-end device is configured to collect and encode a video stream to generate and send the video stream to the back-end device, and the back-end device is configured to perform the video processing method described in the first aspect.
In some embodiments, the front-end device is further configured to perform the video processing method according to the second aspect.
In some of these embodiments, the video processing system further comprises: a third party device configured to be able to decode all reference frames in the first video stream.
In a fourth aspect, in this embodiment, there is provided a storage medium having stored thereon a computer program which, when executed by a processor, implements the video processing method of the first aspect or the second aspect described above.
Compared with the related art, the video processing method, the video processing system and the storage medium provided in the embodiment send a control instruction to the front-end device, where the control instruction is used to instruct the front-end device to adjust the inter-frame dependency relationship of the reference frame in the video stream, so as to obtain the encoded first video stream; decoding the first video stream, wherein the first video stream supports the back-end device to decode in at least two video frame sequences, so that the problem that the back-end device cannot support a newly added channel under the condition that the accessed video stream meets the preview and/or playback effects in the related art is solved, and the back-end device can support the newly added channel.
The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the other features, objects, and advantages of the application.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:
FIG. 1 is a schematic diagram of a video frame sequence for full frame rate decoding in accordance with one embodiment of the present application;
FIG. 2 is a flowchart showing a video processing method according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a video frame sequence obtained after encoding parameter adjustment according to an embodiment of the present application;
FIG. 4 is a schematic diagram illustrating the adjustment of inter-frame dependencies of reference frames according to an embodiment of the application;
FIG. 5 is a flow chart of a back-end device analyzing a video stream according to an embodiment of the present application;
FIG. 6 is a flow chart illustrating the dynamic adjustment of inter-frame dependencies of reference frames according to an embodiment of the application;
FIG. 7 is a second flowchart of a video processing method according to an embodiment of the present application;
FIG. 8 is a schematic diagram of a video processing system according to an embodiment of the present application;
fig. 9 is a schematic diagram of a video processing system according to a preferred embodiment of the present application.
Detailed Description
The present application will be described and illustrated with reference to the accompanying drawings and examples for a clearer understanding of the objects, technical solutions and advantages of the present application.
Unless defined otherwise, technical or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terms "a," "an," "the," "these" and similar terms in this application are not intended to be limiting in number, but may be singular or plural. The terms "comprising," "including," "having," and any variations thereof, as used herein, are intended to encompass non-exclusive inclusion; for example, a process, method, and system, article, or apparatus that comprises a list of steps or modules (units) is not limited to the list of steps or modules (units), but may include other steps or modules (units) not listed or inherent to such process, method, article, or apparatus. The terms "connected," "coupled," and the like in this disclosure are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as used herein means two or more. "and/or" describes an association relationship of an association object, meaning that there may be three relationships, e.g., "a and/or B" may mean: a exists alone, A and B exist together, and B exists alone. Typically, the character "/" indicates that the associated object is an "or" relationship. The terms "first," "second," "third," and the like, as referred to in this disclosure, merely distinguish similar objects and do not represent a particular ordering for objects.
Before the method provided in this embodiment is described, it is necessary to describe the inventive concept of the present application. The frame rate of the video stream preferably ensures the effect of preview or playback, namely, the video stream generated after the front-end equipment codes does not change the frame rate and resolution, so that the third-party equipment decodes the video stream at the full frame rate. FIG. 1 is a schematic diagram of a video frame sequence provided for full frame rate decoding in accordance with an embodiment of the present application, wherein the I-frame is a complete image as shown in FIG. 1; p frames can be understood as patches on the complete image; i frame 1 can be directly decoded into one frame YUV (full image); for P frame 1, the first frame yuv+p frame 1 (patch) is required to be decoded together to generate a second frame YUV; for P frame 2, the second frame yuv+p frame 2 (patch) is required to be decoded together to generate a third frame YUV; the entire decoding order must be performed in the order of inter-frame dependencies of the reference frames, otherwise the latter YUV images may be problematic.
The back-end device has different frame rate requirements on video streams in a decoding stage and an analysis stage, and under the condition that the frame rate of the video streams is not changed, the initial frame rate of the video streams is 30 frames/second, and when the video streams generated by the front-end device are simultaneously analyzed by the back-end device and previewed and/or played back by the third-party device, the back-end device can achieve a better analysis effect only by 8 frames/second.
In order to solve the problem that the backend device cannot support the newly added channel in the case that the accessed video stream satisfies the preview and/or playback effect, it is proposed to consider: on the premise of not influencing the subsequent analysis, if the back-end equipment reduces the video stream frame rate during decoding, the back-end equipment adapts to the self decoding capability, and can support a new channel.
The problem is that the front-end equipment has only one coding format, on one hand, because of the existence of the third-party equipment, the video stream generated by the front-end equipment needs to meet the full frame rate decoding requirement, namely the frame rate of the video stream cannot be changed; on the other hand, when the backend device analyzes the video stream, it is necessary to first decode all P frames and then extract frames from YUV, instead of first extracting frames and then decoding. That is, there is a conflict between meeting preview and/or playback requirements and meeting analysis requirements between the backend device and the third party device; in the back-end device, there is a conflict between fully adapting the decoding capability and fully adapting the analysis capability.
In view of the above-mentioned technical problems and contradictions, in the present embodiment, a video processing method is provided and applied to a back-end device, and fig. 2 is a flowchart of the video processing method of the present embodiment, as shown in fig. 2, where the flowchart includes the following steps:
Step S201, a control instruction is sent to front-end equipment, wherein the control instruction is used for instructing the front-end equipment to adjust the inter-frame dependency relationship of a reference frame in a video stream, and an encoded first video stream is obtained;
Step S202, decoding a first video stream, wherein the first video stream supports the backend device to decode in at least two sequential video frame sequences.
In the encoding stage of the front-end device, the front-end device is controlled by the back-end device to adjust the inter-frame dependency relationship of the reference frames in the video stream, so as to obtain the encoded first video stream, and the first video stream can enable the back-end device to decode according to one of the video frame sequences in one order, or decode according to the video frame sequence in the original order, namely, the full frame rate decoding, under the condition that the frame rate of the video stream is not changed (the full frame rate is maintained).
The video frame sequences of at least two orders may be provided, namely a first video frame sequence and a second video frame sequence, the first video frame sequence being arranged in an order different from the full frame rate, the second video frame sequence being arranged in the full frame rate order. When the front-end equipment transmits the video stream to the back-end equipment, the back-end equipment decodes according to the first video frame sequence under the condition that the frame rate is unchanged.
The front-end device is controlled to encode by the back-end device, so that the inter-frame dependency relationship of reference frames in the video stream is dynamically adjustable, the frame rate equivalent to the first video frame sequence is dynamically adjustable, and in order to ensure a certain channel number, the frame rate of the first video frame sequence can be regulated down, so that the back-end device does not need to extract frames for analysis after decoding all P frames when analyzing the video stream, but directly extracts frames first and decodes the frames, the decoding performance overhead of the back-end device when decoding is reduced, and the video stream is adaptive to the decoding capability of the video stream, thereby supporting a newly added channel.
For example, when the total decoding capability is insufficient after the analysis channel accessed by the back-end device is increased, the front-end device may be controlled to adjust the encoding parameters, so as to obtain an encoded first video stream, where the intra-GOP frame reference relationship is shown in fig. 3, and the first video stream decodes and displays all frames when the third-party device previews and/or plays back, and the frame rate is still 30 frames/second. And only the frame of the black part is decoded when the back-end equipment analyzes, the frame rate is reduced to 1/3, and the decoding performance requirement is reduced.
In the steps S201 to S202, the back-end device controls the front-end device to adjust the inter-frame dependency relationship of the reference frames in the video stream, so as to obtain the video stream supporting decoding in at least two video frame sequences, thereby satisfying the requirement that the back-end device or the third-party device decodes the video stream at the full frame rate, realizing the preview and/or playback effects, and simultaneously satisfying the requirement that the back-end device decodes the video stream at the lower frame rate, supporting the newly added channels, and ensuring a certain channel number. Through the steps, the problem that the back-end equipment cannot support the newly added channel under the condition that the accessed video stream meets the preview and/or playback effect is solved, so that the back-end equipment can support the newly added channel.
In one embodiment, the back-end device will also determine the current decoding capability, and the desired decoding capability that needs to be provided when decoding the video stream of the front-end device, before sending the control instructions to the front-end device; and comparing the current decoding capability with the expected decoding capability, and generating a control instruction according to the comparison result.
The current decoding capability of the back-end device refers to the decoding capability of a channel of the back-end device for analyzing the video stream in the current stage, and the expected decoding capability refers to the decoding capability required to be provided according to the video stream specification of the front-end device which is accessed currently. The current decoding capability is determined according to the analysis specification of the current channel in the back-end equipment, wherein the analysis specification carries resolution information and frame rate information; the desired decoding capability is determined from resolution information and frame rate information carried by the video stream. Specifically, decoding capability = resolution wide resolution high frame rate.
The embodiment reasonably controls the front-end equipment to encode the video stream by comparing the current decoding capability with the expected decoding capability in real time, so that the generated first video stream is suitable for the current decoding capability of the back-end equipment.
In some embodiments of the present application, the video stream includes at least one group of pictures (GOP, group of Pictures), each group of pictures including a plurality of non-key frames, and the back-end device, when comparing the current decoding capability with the desired decoding capability, controls the front-end device to adjust the inter-frame dependencies of the reference frames in the video stream according to different conditions, in the following manner:
In one embodiment, the backend device compares the current decoding capability with the desired decoding capability, and generates a first control instruction if the current decoding capability is less than the desired decoding capability, where the first control instruction is used to instruct the front-end device to relieve inter-frame dependencies between portions of non-key frames in groups of pictures of the first video stream. The arrangement is such that the desired decoding capability does not exceed the decoding capability actually provided by the back-end device.
In one embodiment, the back-end device compares the current decoding capability with the desired decoding capability, and generates a second control instruction if the current decoding capability is not less than the desired decoding capability, where the second control instruction is used to instruct the front-end device to increase inter-frame dependencies between portions of non-key frames in groups of pictures of the first video stream. The configuration is that the expected decoding capability approaches to the decoding capability actually provided by the back-end equipment, and the decoding capability of the back-end equipment is fully exerted.
Fig. 4 is a schematic diagram of the principle of adjusting the inter-frame dependency of the reference frame in this embodiment, and as shown in fig. 4, taking the inter-frame dependency between partial non-key frames as an example, two video frame sequences are set, so that the backend device and the third party device can decode in the following two orders, where an I frame is a key frame and a P frame is a non-key frame:
When the video stream is supplied to the back-end equipment for analysis, a first sequence of video frames is employed: i frame 1, P frame 2, P frame 4, P frame 6, … …, I frame 2;
When the video stream is provided to a third party device for previewing and/or playback, a second sequence of video frames is employed: i frame 1, P frame 2, P frame 3, … …, I frame 2.
In this embodiment, in different situations, by controlling the adjustment direction of the inter-frame dependency relationship of the reference frames, the back-end device may be supported to decode at a lower frame rate or a higher frame rate, so that the decoding analysis performance of the channel in the back-end device is maximally exerted.
How the inter-frame dependencies of the reference frames are further controlled will be described below.
In one embodiment, the backend device compares the current decoding capability with the desired decoding capability, determines a maximum decoding capability if the current decoding capability is not less than the desired decoding capability, and determines whether the maximum decoding capability is greater than the desired decoding capability; and generating a second control instruction when the maximum decoding capability is judged to be larger than the expected decoding capability.
The current decoding capability is not less than the expected decoding capability, and represents that the decoding capability of the back-end equipment is remained, so that the front-end equipment coding is controlled based on the maximum decoding capability of the back-end equipment, and the inter-frame dependency relationship among partial non-key frames is increased in the first video stream so as to improve the frame rate of the first video frame sequence, so that the expected decoding capability approaches to the maximum decoding capability, and the decoding capability of the back-end equipment is fully exerted.
Further, in one embodiment, the backend device compares the current decoding capability with the desired decoding capability, determines a maximum decoding capability if the current decoding capability is not less than the desired decoding capability, and determines whether the maximum decoding capability is greater than the desired decoding capability; and under the condition that the maximum decoding capability is not greater than the expected decoding capability, determining that the frame rate of the first video frame sequence is in an optimal state and cannot be adjusted any more, and keeping unchanged.
In one embodiment, the backend device, after decoding the video stream in accordance with the first sequence of video frames, also scales the decoded video stream to a preset resolution for analysis.
In one embodiment, the backend device determines a current analysis capability and determines a desired analysis capability to be provided when analyzing the first video stream; judging whether the current analysis capability is smaller than the expected analysis capability; and refusing to access the front-end equipment under the condition that the current analysis capability is less than the expected analysis capability.
In this embodiment, the back-end device has a conflict between decoding and analysis, even if the first video frame sequence is obtained by adjusting the inter-frame dependency relationship of the reference frames in the video stream of the front-end device, so that the decoding capability actually provided by the back-end device meets the desired decoding capability, and if no additional analysis capability exists, the front-end device is refused to be accessed to the back-end device.
In one embodiment, the backend device may extract only a portion of the reference frames of the first video stream for decoding when decoding the first video stream; or the back-end equipment decodes all the reference frames in the first video stream; or a part of channels of the back-end equipment extract part of reference frames in the first video stream to decode, and the other part of channels decode all the reference frames in the first video stream; or the back-end equipment firstly extracts only part of the reference frames in the first video stream to decode, and then decodes all the reference frames in the first video stream.
Fig. 5 is a schematic flow chart of the back-end equipment analyzing the video stream according to an embodiment of the present application, as shown in fig. 5, the flow chart includes the following steps:
Step S501, receiving an encoded first video stream;
Step S502, decoding the first video stream to obtain a second video stream containing a plurality of frames YVU;
step S503, when the self analysis capability is insufficient, the second video stream is subjected to frame extraction processing, and part of YUV is discarded;
Step S504, scaling YUV in the second video stream to a preset resolution to obtain a third video stream;
Step S505, the third video stream is sent to an algorithm module for analysis;
Step S506, returning to the bounding box coordinates of the detection target;
And S507, carrying out matting encoding based on the bounding box coordinates of the detection target.
The video processing method of the present application will be described below with reference to specific application scenarios.
In the related art, the number of analysis channels and the analysis specification are set according to the hardware performance of the back-end device (a certain number of analysis channels can be reached at a certain resolution and frame rate in general). The resolution of the video collected by the front-end equipment is larger and larger, and a plurality of resolution combinations can exist on site, so that the analysis specification of the back-end equipment cannot be accurately given, because the analysis is carried out on the premise of decoding into YUV. In response to this problem, in some embodiments, a flow chart for dynamically adjusting the inter-frame dependency of the reference frame is provided, as shown in fig. 6, and the flow chart includes the following steps:
step S601, calculating the current decoding capability of an analysis channel in the back-end equipment at regular time;
Step S602, judging whether the current decoding capability is smaller than the expected decoding capability; if yes, go to step S604; if not, executing step S603;
Step S603, judging whether the maximum decoding capability is larger than the expected decoding capability; if yes, go to step S606; if not, executing step S605;
Step S604, indicating the front-end equipment to adjust the inter-frame dependency relationship of the reference frames, and reducing the frame rate of the first video frame sequence so that the decoding capability required to be provided does not exceed the maximum decoding capability;
step S605, maintaining the inter-frame dependency relationship of the original reference frame, and keeping the inter-frame dependency relationship unchanged;
Step S606 instructs the front-end device to adjust the inter-frame dependency of the reference frames, increasing the frame rate of the first video frame sequence such that the decoding capability to be provided approaches the maximum decoding capability.
By dynamically adjusting the inter-frame dependency relationship of the reference frames in the front-end equipment, an analysis specification independent of video resolution can be given, a certain number of analysis channels can be ensured to smoothly decode video streams on the premise of not influencing the resolution and the frame rate of video preview and/or playback, and the number of analysis channels can reach the set index in the factory specification of products under different resolution scenes.
The following embodiments will describe a method of adjusting decoding overhead by adjusting inter-frame dependencies of reference frames.
In some of these embodiments, the decoding and analysis capabilities of the backend device in the initial state are as follows:
decoding capability: 16 1080P,30 frames/sec, or 4K 30 frames/sec; analytical capability: 1080P 80 frames/sec.
The back-end equipment is accessed into 4 paths of 4K 30 frames/second video streams, the decoding capacity is remained by 0, the decoding capacity is reduced to 1080P for analysis after decoding, 120 frames/second video streams exceed the analysis capacity, and each path is subjected to frame extraction analysis according to 20 frames/second video streams.
The back-end equipment adds 4 paths of 4K 30 frames/second video again, changes into 8 paths of 4K 30 frames/second video, exceeds the decoding capability, and sends a control request to the front-end equipment, so that the 8 front-end equipment adjusts the inter-frame dependency relationship of the reference frames to enable the reference frames to support the frame rate of 15 frames/second for decoding, the decoding performance cost is 8 paths of 4K 15 frames/second, and after the video is scaled into 1080P, the intelligent analysis frame extraction strategy can be adjusted into 8 paths of 1080P 10 frames/second.
When the back-end device adds one path of 4K video on the former basis, assuming that the minimum requirement of the algorithm on the video stream is 1080P,10 frames/second, and the 9 paths of video streams at least need analysis capability of 90 frames/second, even if the decoding capability is met, no additional analysis capability exists, so that the 9 th path of analysis channel is refused to be added, and a failed prompt message is returned.
The method is only a scene with one algorithm and resolution, video streams with various resolutions and frame rates exist in the actual scene, various algorithms have different minimum resolution and frame rate requirements on the video streams, decoding overhead is adjusted by adjusting the inter-frame dependency of reference frames, and the decoding analysis performance of channels in the back-end equipment can be maximally exerted by combining a specific adjustment strategy.
In one embodiment of the present application, another video processing method is provided and applied to a front-end device, and fig. 7 is a flowchart of the video processing method of the present embodiment, as shown in fig. 7, where the flowchart includes the following steps:
Step S701, receiving a control instruction, and adjusting the inter-frame dependency relationship of a reference frame in a video stream according to the control instruction to obtain a first video stream;
In step S702, a first video stream is sent to a back-end device, where the first video stream supports decoding by the back-end device in at least two sequential video frame sequences.
In the encoding stage of the front-end device, the embodiment adjusts the inter-frame dependency relationship of the reference frames in the video stream by controlling the front-end device to obtain the encoded first video stream, where the first video stream can enable the back-end device to decode according to one of the video frame sequences in one order, or decode according to the video frame sequence in the original order, i.e., full-frame rate decoding, under the condition that the frame rate of the video stream is not changed (full-frame rate is maintained).
The video stream supporting decoding by at least two video frame sequences is obtained by controlling the front-end equipment to adjust the inter-frame dependency relationship of the reference frames in the video stream, so that the video stream is decoded at the full frame rate by the back-end equipment or the third-party equipment, the video stream is decoded at the lower frame rate by the back-end equipment while the preview and/or playback effects are realized, and the newly added channels are supported to ensure a certain channel number. Through the steps, the problem that the back-end equipment cannot support the newly added channel under the condition that the accessed video stream meets the preview and/or playback effect is solved, so that the back-end equipment can support the newly added channel.
In one embodiment, the first video stream includes at least one group of pictures, each group of pictures including a plurality of non-key frames, and the head-end device removes inter-frame dependencies between portions of the non-key frames in each group of pictures of the first video stream according to the first control instruction; or increasing the inter-frame dependency relationship among part of non-key frames in each picture group of the first video stream according to the second control instruction.
The video processing method performed in the front-end device is compatible with the back-end device, and therefore, specific examples of the video processing method performed by the front-end device may refer to examples described in the foregoing embodiments and alternative implementations of the back-end device, which are not described herein.
In an embodiment of the present application, there is further provided a video processing system, fig. 8 is a schematic structural diagram of the video processing system in this embodiment, and as shown in fig. 8, the system includes: the front-end device is connected with the back-end device, the front-end device is configured to collect and encode and generate a video stream, and send the video stream to the back-end device, and the back-end device is configured to execute the video processing method of any of the corresponding embodiments.
In an embodiment, the front-end device is further configured to perform the video processing method of any of the respective embodiments described above.
In one embodiment, the video processing system further comprises: and a third party device configured to be able to decode all reference frames in the first video stream.
In one embodiment, the backend device further comprises: the storage module can store the video stream which is not decoded, can also store the decoded video stream, and can also store the analysis result obtained after the video stream is analyzed.
In one embodiment, the front-end device may be an image capturing apparatus such as an IPC (webcam) or a dome camera, the back-end device is a terminal device or a platform with video analysis capability, the third-party device is a client/Web platform with video preview and/or playback function, and the client/Web platform may log directly into the front-end device to preview and/or play back video.
In addition, the third party device may be connected to a backend device, e.g., a client/Web platform may be connected to a terminal device or platform, which may log directly onto to access the analysis results.
Fig. 9 is a schematic structural diagram of a video processing system according to a preferred embodiment of the present application, and as shown in fig. 9, the video processing system includes: the system comprises an IPC, an analysis platform and a client/Web platform, wherein the analysis platform comprises a storage module, the IPC is connected with the analysis platform, and the IPC and the analysis platform are respectively connected with the client/Web platform.
The IPC is responsible for collecting pictures for coding; the analysis platform is responsible for pulling the video stream from the IPC to analyze, the storage module is responsible for storing the video stream and the analysis result, and in addition, the analysis platform also forwards the video stream to the client/Web platform; the client/Web platform can directly log in the IPC to preview and/or play back the video, or directly log in the analysis platform to access the analysis result.
It should be noted that, specific examples in this embodiment may refer to examples described in the foregoing embodiments and alternative implementations, and are not described in detail in this embodiment.
In addition, in combination with the video processing method provided in the above embodiment, a storage medium may be provided in this embodiment. The storage medium has a computer program stored thereon; the computer program, when executed by a processor, implements any of the video processing methods of the above embodiments.
It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to be limiting. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure in accordance with the embodiments provided herein.
It is to be understood that the drawings are merely illustrative of some embodiments of the present application and that it is possible for those skilled in the art to adapt the present application to other similar situations without the need for inventive work. In addition, it should be appreciated that while the development effort might be complex and lengthy, it would nevertheless be a routine undertaking of design, fabrication, or manufacture for those of ordinary skill having the benefit of this disclosure, and thus should not be construed as a departure from the disclosure.
The term "embodiment" in this disclosure means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive. It will be clear or implicitly understood by those of ordinary skill in the art that the embodiments described in the present application can be combined with other embodiments without conflict.
The above examples merely represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the patent claims. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.
Claims (8)
1. A video processing method applied to a back-end device, comprising:
Transmitting a control instruction to front-end equipment, wherein the control instruction is used for instructing the front-end equipment to adjust the inter-frame dependency relationship of a reference frame in a video stream to obtain a coded first video stream;
Decoding the first video stream, wherein the first video stream supports a backend device to decode in at least two sequential video frame sequences;
before sending the control instruction to the front-end equipment, the method further comprises:
determining a current decoding capability and determining a desired decoding capability to be provided when decoding a video stream of the front-end device;
Comparing the current decoding capability with the expected decoding capability, and generating the control instruction according to a comparison result;
The first video stream includes at least one group of pictures, each group of pictures including a plurality of non-key frames, comparing the current decoding capability with the desired decoding capability, and generating the control instruction according to the comparison result includes:
Generating a first control instruction under the condition that the current decoding capability is smaller than the expected decoding capability, wherein the first control instruction is used for indicating the front-end equipment to release inter-frame dependency relationship among partial non-key frames in each picture group of the first video stream;
generating a first control instruction in the case that the current decoding capability is less than the desired decoding capability, comprising:
Determining a decoding order according to a first video frame sequence and generating a first control instruction according to the decoding order under the condition that the current decoding capability is smaller than the expected decoding capability; non-key frames of odd digits are not included in the first video frame sequence;
the method further comprises the steps of: and generating a second control instruction under the condition that the current decoding capability is not less than the expected decoding capability, wherein the second control instruction is used for indicating the front-end equipment to increase the inter-frame dependency relationship among partial non-key frames in each picture group of the first video stream.
2. The video processing method according to claim 1, wherein generating a second control instruction in the case where the current decoding capability is not less than the desired decoding capability comprises:
determining a maximum decoding capability, and judging whether the maximum decoding capability is greater than the expected decoding capability;
and generating the second control instruction when the maximum decoding capability is judged to be larger than the expected decoding capability.
3. The video processing method of claim 1, wherein the method further comprises:
determining a current analysis capability and determining a desired analysis capability to be provided when analyzing the first video stream;
judging whether the current analysis capability is smaller than the expected analysis capability;
and refusing to access the front-end equipment under the condition that the current analysis capability is less than the expected analysis capability.
4. The video processing method of claim 1, wherein decoding the first video stream comprises:
Extracting part of the reference frames in the first video stream for decoding, and/or decoding all the reference frames in the first video stream.
5. A video processing method applied to a front-end device, comprising:
Receiving a control instruction, and adjusting the inter-frame dependency relationship of a reference frame in a video stream according to the control instruction to obtain a first video stream; the control instruction is sent by the back-end equipment, and the control instruction is generated by the back-end equipment according to the current decoding capability and a comparison result of the expected decoding capability required to be provided when the video stream of the front-end equipment is decoded;
Transmitting the first video stream to a back-end device, wherein the first video stream supports decoding by the back-end device in at least two sequential video frame sequences;
The first video stream includes at least one picture group, each picture group includes a plurality of non-key frames, and adjusting an inter-frame dependency relationship of a reference frame in the video stream according to the control instruction includes:
releasing the inter-frame dependency relationship between partial non-key frames in each picture group of the first video stream according to a first control instruction; the first control instruction is used for indicating the front-end equipment to release inter-frame dependency relationship among partial non-key frames in each picture group of the first video stream; the first control instruction is generated according to a decoding order determined by a first video frame sequence when the current decoding capability of the back-end equipment is smaller than the expected decoding capability, and the first video frame sequence does not contain non-key frames with odd digits; or alternatively
Adding inter-frame dependency relations among part of non-key frames in each picture group of the first video stream according to a second control instruction; wherein the second control instruction is generated by the backend device if the current decoding capability is not less than the desired decoding capability, and the second control instruction is for instructing the backend device to increase inter-frame dependencies between portions of non-key frames in groups of pictures of the first video stream.
6. A video processing system comprising a front-end device and a back-end device, the front-end device being connected to the back-end device, the front-end device being configured to be able to collect and encode a generated video stream and to send the video stream to the back-end device, the back-end device being configured to perform the video processing method of any one of claims 1 to 4; the head-end equipment is further configured to perform the video processing method of claim 5.
7. The video processing system of claim 6, wherein the video processing system further comprises: a third party device configured to be able to decode all reference frames in the first video stream.
8. A computer readable storage medium having stored thereon a computer program, characterized in that the computer program, when executed by a processor, implements the video processing method of any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210514646.6A CN115278305B (en) | 2022-05-12 | 2022-05-12 | Video processing method, video processing system, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210514646.6A CN115278305B (en) | 2022-05-12 | 2022-05-12 | Video processing method, video processing system, and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115278305A CN115278305A (en) | 2022-11-01 |
| CN115278305B true CN115278305B (en) | 2024-05-07 |
Family
ID=83758687
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210514646.6A Active CN115278305B (en) | 2022-05-12 | 2022-05-12 | Video processing method, video processing system, and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115278305B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102263942A (en) * | 2010-05-31 | 2011-11-30 | 苏州闻道网络科技有限公司 | Scalable video transcoding device and method |
| CN105023241A (en) * | 2015-07-29 | 2015-11-04 | 华南理工大学 | Fast image interpolation method for mobile terminal |
| CN108063973A (en) * | 2017-12-14 | 2018-05-22 | 浙江大华技术股份有限公司 | A kind of method for decoding video stream and equipment |
| CN111988561A (en) * | 2020-07-13 | 2020-11-24 | 浙江大华技术股份有限公司 | Adaptive adjustment method and device for video analysis, computer equipment and medium |
| WO2021007702A1 (en) * | 2019-07-12 | 2021-01-21 | Huawei Technologies Co., Ltd. | Video encoding method, video decoding method, video encoding device, and video decoding device |
| CN113473147A (en) * | 2021-05-17 | 2021-10-01 | 浙江大华技术股份有限公司 | Post-processing method and device of video code stream and computer readable storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9171577B1 (en) * | 2003-04-25 | 2015-10-27 | Gopro, Inc. | Encoding and decoding selectively retrievable representations of video content |
| US20120169883A1 (en) * | 2010-12-31 | 2012-07-05 | Avermedia Information, Inc. | Multi-stream video system, video monitoring device and multi-stream video transmission method |
| CN108111532A (en) * | 2018-01-05 | 2018-06-01 | 上海小蚁科技有限公司 | The control method for playing back and device of video stream data, storage medium, terminal |
-
2022
- 2022-05-12 CN CN202210514646.6A patent/CN115278305B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102263942A (en) * | 2010-05-31 | 2011-11-30 | 苏州闻道网络科技有限公司 | Scalable video transcoding device and method |
| CN105023241A (en) * | 2015-07-29 | 2015-11-04 | 华南理工大学 | Fast image interpolation method for mobile terminal |
| CN108063973A (en) * | 2017-12-14 | 2018-05-22 | 浙江大华技术股份有限公司 | A kind of method for decoding video stream and equipment |
| WO2021007702A1 (en) * | 2019-07-12 | 2021-01-21 | Huawei Technologies Co., Ltd. | Video encoding method, video decoding method, video encoding device, and video decoding device |
| CN111988561A (en) * | 2020-07-13 | 2020-11-24 | 浙江大华技术股份有限公司 | Adaptive adjustment method and device for video analysis, computer equipment and medium |
| CN113473147A (en) * | 2021-05-17 | 2021-10-01 | 浙江大华技术股份有限公司 | Post-processing method and device of video code stream and computer readable storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 结合帧率变换与HEVC标准的新型视频压缩编码算法;武其达;何小海;林宏伟;陶青川;吴笛;;自动化学报;20171212(09);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115278305A (en) | 2022-11-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP1177691B1 (en) | Method and apparatus for generating compact transcoding hints metadata | |
| JP5893002B2 (en) | Rate control in video coding. | |
| US20130022116A1 (en) | Camera tap transcoder architecture with feed forward encode data | |
| US20090290645A1 (en) | System and Method for Using Coded Data From a Video Source to Compress a Media Signal | |
| US20140362918A1 (en) | Tuning video compression for high frame rate and variable frame rate capture | |
| CN108810545B (en) | Method, apparatus, computer readable medium and electronic device for video encoding | |
| KR20070085745A (en) | Method and device for processing coded video data | |
| JP2013055587A (en) | Image processing apparatus, image processing method, and image processing system | |
| JPH11112973A (en) | Device and method for converting video signal | |
| US8139877B2 (en) | Image processing apparatus, image processing method, and computer-readable recording medium including shot generation | |
| JP4983917B2 (en) | Moving image distribution system, conversion device, and moving image distribution method | |
| EP2448287A1 (en) | Image processing device, image processing method, and image processing system | |
| JP2010035133A (en) | Moving image encoding apparatus and moving image encoding method | |
| US20090257736A1 (en) | Information processing apparatus and information processing method | |
| CN114157870A (en) | Encoding method, medium, and electronic device | |
| Makar et al. | Real-time video streaming with interactive region-of-interest | |
| CN115278305B (en) | Video processing method, video processing system, and storage medium | |
| JP4586328B2 (en) | Information processing system, information processing apparatus and method, recording medium, and program | |
| JP2008141354A (en) | Image coding apparatus and imaging apparatus | |
| Nakhaei et al. | Scene-level two-pass video rate controller for H. 265/HEVC standard | |
| JP4700992B2 (en) | Image processing device | |
| JP2009081622A (en) | Moving image compression encoder | |
| JP2005341093A (en) | Contents adaptating apparatus, contents adaptation system, and contents adaptation method | |
| JPH10145788A (en) | Video reproduction device | |
| Wang et al. | Attention information based spatial adaptation framework for browsing videos via mobile devices |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |