CN112954374A - Video data processing method and device, electronic equipment and storage medium - Google Patents
Video data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112954374A CN112954374A CN202110121083.XA CN202110121083A CN112954374A CN 112954374 A CN112954374 A CN 112954374A CN 202110121083 A CN202110121083 A CN 202110121083A CN 112954374 A CN112954374 A CN 112954374A
- Authority
- CN
- China
- Prior art keywords
- additional information
- video
- video data
- live
- data packet
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 36
- 230000005540 biological transmission Effects 0.000 claims abstract description 60
- 238000007906 compression Methods 0.000 claims abstract description 20
- 230000006835 compression Effects 0.000 claims abstract description 20
- 238000012545 processing Methods 0.000 claims description 27
- 238000000034 method Methods 0.000 claims description 17
- 238000009877 rendering Methods 0.000 claims description 8
- 238000000605 extraction Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 14
- 238000004891 communication Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 6
- 238000004806 packaging method and process Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012858 packaging process Methods 0.000 description 1
- 238000012856 packing Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
- H04N21/23106—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion involving caching operations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/235—Processing of additional data, e.g. scrambling of additional data or processing content descriptors
- H04N21/2355—Processing of additional data, e.g. scrambling of additional data or processing content descriptors involving reformatting operations of additional data, e.g. HTML pages
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/435—Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
- H04N21/4355—Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream involving reformatting operations of additional data, e.g. HTML pages on a television screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44012—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440218—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The embodiment of the invention provides a video data processing method and device, electronic equipment and a storage medium, and relates to the technical field of internet. The video data processing method comprises the following steps: sequentially extracting video additional information corresponding to each frame of live broadcast image in live broadcast video data, and storing the video additional information in a cache region; acquiring multiple frames of the video additional information from the buffer area and compressing the video additional information to obtain an additional information data packet; and respectively sending the additional information data packet and the live video data to corresponding clients. Therefore, the compression rate of the video additional information is improved, the size of the obtained additional information data packet is reduced, and transmission resources are occupied in the transmission process.
Description
Technical Field
The invention relates to the technical field of internet, in particular to a video data processing method and device, electronic equipment and a storage medium.
Background
With the development of internet technology, the playing content of video data is more and more diversified, and especially in a live scene, the playing of video data is no longer limited to a video picture itself, and video additional information (for example, outline information of a graphic element that a user pays attention to) of the video data is often required to be displayed.
Currently, the video server encodes, packages and transmits the extracted video additional information together with video pictures of the video data to a client used by a user. However, this approach requires that the resulting video additional information for each frame be encoded and packetized. Obviously, on one hand, the compression workload is increased, and on the other hand, the size of the transmission data packet still has an optimized space, which results in unnecessary waste of data transmission resources.
Disclosure of Invention
In view of the above, the present invention provides a video data processing method, an apparatus, an electronic device and a storage medium.
In order to achieve the above purpose, the embodiment of the present invention adopts the following technical solutions:
in a first aspect, the present invention provides a video data processing method, applied to a live broadcast server, where the video data processing method includes: sequentially extracting video additional information corresponding to each frame of live broadcast image in live broadcast video data, and storing the video additional information in a cache region; acquiring multiple frames of the video additional information from the buffer area and compressing the video additional information to obtain an additional information data packet; and respectively sending the additional information data packet and the live video data to corresponding clients.
It can be seen that, in the above embodiment, the video data processing method utilizes the characteristic that the difference between the video additional information of adjacent multiple live broadcast images in the same live broadcast video data is not very large, and selects to pack and compress the video additional information corresponding to the multiple live broadcast images together. That is, for the same amount of video additional information, the size of the obtained data packet can be obviously reduced after the data packet is packed and compressed by the scheme. On the basis, the additional information data packet and the live video data obtained after packaging are respectively sent to the client side for playing, so that the live video data and the additional information data packet can reach the client side, and meanwhile, the occupation of transmission resources is reduced.
In an optional embodiment, the step of obtaining multiple frames of the video additional information from the buffer for compression includes: acquiring transmission time delay of the live broadcast image transmitted to the client; acquiring the video additional information from the cache region for compression according to the transmission delay and by combining the acquisition time stamp of the video additional information; and the acquisition time stamp of the video additional information is the same as the acquisition time stamp of the corresponding live broadcast image.
It is understood that live video data in a live scene is collected in real time, and therefore, video additional information corresponding to live images is extracted in real time. In order to pack the multiple frames of video additional information extracted in real time, the video additional information extracted in real time needs to be placed into the buffer area first, so that the multiple frames of video additional information are stored in the buffer area and then compressed and packed. Although the more video additional information compressed together, the higher the compression rate, the more video additional information is obtained at the cost of longer time the video additional information stays in the buffer. However, the client needs to render the live image in the live video data synchronously with the corresponding video additional information.
Therefore, in the above embodiment, the video additional information is collected and compressed and packaged by using the inherent time delay in the live video data transmission process, so that the video additional information is prevented from being sent to the client later than the corresponding live video, and the live real-time performance is prevented from being influenced.
The collection and compression and packaging of the video additional information by using the inherent time delay in the live video data transmission process can be realized by adopting the following two modes:
in the first mode, the maximum time difference between the video additional information stored in the cache region is calculated according to the acquisition time stamp of the video additional information stored in the cache region; and under the condition that the maximum time difference is matched with the transmission time delay, the video additional information stored in the cache region is taken out and compressed.
In a second mode, under the condition that the sum of the storage duration of the video additional information written first in the buffer area and the transmission time consumption corresponding to the additional information data packet is equal to the transmission delay, all the video additional information in the buffer area is taken out and compressed.
In an optional implementation manner, the step of sending the additional information data packet and the live video data to corresponding clients respectively includes: generating a video data packet according to a live image of the live video data; sending the video data packet to the client through a video transmission channel; and transmitting the additional information data packet to the client through a data packet transmission channel.
In a second aspect, the present invention provides a video data processing method, applied to a client, where the video data processing method includes: receiving a video data packet and an additional information data packet corresponding to live video data; the additional information data packet is a data packet obtained by compressing multi-frame video additional information extracted from the live video data; the video data packet is generated based on a live image of the live video data; and after the video data packet and the additional information data packet are respectively decoded, acquiring the live broadcast image and the video additional information with corresponding identifications so as to synchronously render.
In an optional embodiment, the video data processing method further includes: periodically calculating transmission time delay according to the acquired acquisition time stamp and the universal time of the live broadcast image; and feeding back the transmission delay to a live broadcast server to guide the live broadcast server to create the additional information data packet.
In a third aspect, the present invention provides a video data processing apparatus applied to a live broadcast server, the video data processing apparatus comprising: the extraction module is used for sequentially extracting video additional information corresponding to each frame of live broadcast image in the live broadcast video data and storing the video additional information into a cache region; the compression module is used for acquiring multiple frames of the video additional information from the cache region and compressing the video additional information to obtain an additional information data packet; and the sending module is used for sending the additional information data packet and the live video data to corresponding clients respectively.
In a fourth aspect, the present invention provides a video data processing apparatus applied to a client, the video data processing apparatus comprising:
the receiving module is used for receiving a video data packet and an additional information data packet corresponding to the live video data; the additional information data packet is a data packet obtained by compressing multi-frame video additional information extracted from the live video data; the video data packet is generated based on a live image of the live video data;
and the rendering module is used for acquiring the live broadcast image and the video additional information with corresponding identifications after respectively decoding the video data packet and the additional information data packet so as to synchronously render.
In a fifth aspect, the present invention provides an electronic device comprising a processor and a memory, the memory storing machine executable instructions executable by the processor to implement the method of any one of the preceding embodiments.
In a sixth aspect, the invention provides a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of the preceding embodiments.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
Fig. 1 shows a live scene diagram provided in an embodiment of the present invention.
Fig. 2 shows a schematic diagram of an electronic device provided by an embodiment of the invention.
Fig. 3 is a flowchart illustrating steps of a video data processing method applied to a live server according to an embodiment of the present invention.
Fig. 4 is a diagram illustrating an example of encoding video additional information in the related art.
Fig. 5 is a diagram illustrating an example of merging and compressing additional video information according to an embodiment of the present invention.
Fig. 6 shows a flowchart of sub-steps of step S102 provided by the embodiment of the present invention.
Fig. 7 conceptually illustrates an exemplary view of the transmission of video data packets and additional information data packets to a viewer's client.
Fig. 8 shows a schematic diagram of a video data processing apparatus applied to a live server according to an embodiment of the present invention.
Fig. 9 is a flowchart illustrating steps of a video data processing method applied to a client according to an embodiment of the present invention.
Fig. 10 is a schematic diagram illustrating a video data processing apparatus applied to a client according to an embodiment of the present invention.
Icon: 100-an electronic device; 110-a memory; 120-a processor; 130-a communication module; 400-video data processing means; 401-an extraction module; 402-a compression module; 403-a sending module; 500-video data processing means; 501-a receiving module; 502-rendering module.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention.
It is noted that relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
In order to optimize a transmission scheme of video additional information corresponding to video data, particularly a transmission scheme of video additional information corresponding to live video data in a live scene. The embodiment of the invention provides a video data processing method and device, electronic equipment and a storage medium.
The application scene of the video data processing method and the device can be the scene where the image information acquired in real time needs to be acquired. For example, but not limited to, live scenes, monitored scenes, and the like may be included.
The following describes a video data processing method and apparatus, an electronic device 100, and a storage medium provided by embodiments of the present invention, taking an example in which the video data processing method and apparatus are applied to a live broadcast scene shown in fig. 1. As shown in fig. 1, a live client (also called a main broadcast) collects live image frames to generate live video data, and sends the live video data to a video server in the form of a video stream. And after the video server carries out relevant processing on the live video data, the live video data are sent to the corresponding audience client. The viewer client may be a client used by a viewer entering a live room created by a live client.
It should be noted that the live client and the viewer client are essentially clients that obtain services from the video server, and the difference is only in roles that are assumed in the live activities. Each client can switch between the live client and the viewer client according to the user's selection. For example, if a user a uses a client a to perform live broadcasting, the client a is a live broadcasting client, and a client B used by a user B who enters a live broadcasting room created by the user a is an audience client. After the user A downloads, when the client A enters a live broadcast room created by the user B, the client B used by the user B who is carrying out live broadcast is a live broadcast client, and the client A is an audience client.
The video data processing method and device provided by the embodiment of the invention can be applied to the live broadcast server and the corresponding audience client in the figure 1.
The live server may be, but is not limited to, a Personal Computer (PC), a server, a distributed computer, and the like. It is to be understood that the server is not limited to a physical server, and may also be a virtual machine on the physical server, a virtual machine built on a cloud platform, or the like, which can provide the same function as the server or the virtual machine.
The client may be an intelligent device with image capturing and displaying functions, for example, the client may include a mobile phone, an intelligent home device, a wearable device, an intelligent mobile device, a virtual reality device, an augmented reality device, or the like, and may also be a tablet computer, a laptop computer, or a built-in device in a motor vehicle, or the like.
For convenience of describing a hardware environment for operating the video data processing method and apparatus, the live broadcast server and the corresponding viewer client are both referred to as the electronic device 100 below.
Referring to fig. 2, fig. 2 is a block diagram of the electronic device 100. The electronic device 100 includes a memory 110, a processor 120, and a communication module 130. The memory 110, the processor 120 and the communication module 130 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines.
The memory 110 is used to store programs or data. The Memory 110 may be, but is not limited to, a Random Access Memory 110 (RAM), a Read Only Memory 110 (ROM), a Programmable Read Only Memory 110 (PROM), an Erasable Read Only Memory 110 (EPROM), an electrically Erasable Read Only Memory 110 (EEPROM), and the like.
The processor 120 is used to read/write data or programs stored in the memory 110 and perform corresponding functions.
The communication module 130 is configured to establish a communication connection between the electronic device 100 and another communication terminal through the network, and to transmit and receive data through the network.
It should be understood that the structure shown in fig. 2 is only a schematic structural diagram of the electronic device 100, and the electronic device 100 may also include more or fewer components than shown in fig. 2, or have a different configuration than shown in fig. 2. The components shown in fig. 2 may be implemented in hardware, software, or a combination thereof.
First embodiment
Referring to fig. 3, fig. 3 shows a video data processing method according to an embodiment of the invention. The video data processing method may be applied to a live server, and as shown in fig. 3, the video data processing method may include the following steps:
and S101, sequentially extracting video additional information corresponding to each frame of live broadcast image in the live broadcast video data, and storing the video additional information into a cache region.
The live video data may be a video stream received by a live server. In a live scene, the live video data may be a video stream collected and transmitted by a live client. In a monitoring scene, the live video data may be a video stream collected by a camera installed in a monitored field.
The live video data is composed of a plurality of frames of live images which are sequentially sequenced according to the acquisition time.
The video additional information may be information extracted based on the live image and interested by the user, or may be information written in relation to the live image. For example, in a live video scene, the video additional information may be clothes appearing in the live video image and a purchase link of the clothes appearing to be added to the live video image. Of course, the description information for the live image may also be, for example, the residual error between the live image and the last live image, and texture detail area information in the video frame.
The buffer may be a first-in first-out storage area for temporary data storage.
In some embodiments, the corresponding video additional information may be extracted based on each frame of live image of the live video data in turn. Each frame of video additional information corresponding to a live broadcast image (hereinafter simply referred to as a frame of video additional information for convenience of description) is extracted and written into the buffer.
Unlike the related art, the video additional information is extracted and then is not directly encoded and packaged with the live image, but the video additional information is written into the buffer area first. It can be understood that the live video data has a certain real-time property, each frame of live image in the live video data is collected in real time, and correspondingly, each frame of video additional information is also extracted in real time, and in order to accumulate multiple frames of video additional information, the extracted video additional information needs to be written into the buffer area.
Step S102, obtaining multi-frame video additional information from the buffer area and compressing to obtain an additional information data packet.
The additional information packet is a packet obtained by compressing the video additional information.
In some embodiments, after buffering the multiple frames of video additional information in the buffer area, the buffered multiple frames of video additional information may be taken out, and subjected to merging, compression, and packaging processes to obtain an additional information data packet.
In the related art, since the video additional information needs to be encoded and packaged together with the live video, it is equivalent to correspondingly generate an additional information data packet every time a frame of video additional information is extracted, as shown in fig. 4. Although transmission resources required for a single additional information packet are not large, since the compression rate of a single additional information packet is not high, a plurality of additional information packets occupy more transmission resources.
Since the picture contents of the adjacent live images are usually not very different, the corresponding video additional information extracted from the adjacent live images is not very different. Multiple frames of video supplementary information are combined and compressed together to obtain an integrated supplementary information data packet, as shown in fig. 5. Obviously, compared with the plurality of additional information data in fig. 4, the compression rate of the entire additional information data packet is improved, the corresponding data volume is smaller, and the required transmission resources are less.
As an embodiment, in step S102, if the amount of the video additional information stored in the buffer exceeds a specified threshold, all the video additional information in the buffer is taken out, merged, compressed, and packed to obtain an additional information data packet.
Although the more video additional information that is compressed together, the higher the compression rate. However, collecting more video additional information results in a longer time for the video additional information to stay in the buffer. However, the live image and the corresponding video additional information in the live video data also generally need to be rendered synchronously. Although there is a certain transmission delay in the process of sending live video data from the live server to the viewer client and then playing the live video data, if the video additional information is sent to the client later than the corresponding live image, the live video additional information will further aggravate the live video delay.
Therefore, in the process of collecting the video additional information by using the buffer area, the propagation delay of the live video data needs to be considered. That is, the propagation delay is used to trigger the packing of the video additional information in the buffer.
That is, as another embodiment, as shown in fig. 6, the step S102 may further include the following sub-steps:
and a substep S102-1 of obtaining the transmission time delay of the live broadcast image transmitted to the client.
In some embodiments, the sub-step S102-1 may determine the corresponding transmission delay according to a data transmission speed measured by the live server and the viewer client.
In other embodiments, the sub-step S102-1 may further reversely deduce a corresponding transmission delay according to a time point of decoding each frame of the live video frame fed back by the viewer client and an acquisition time point of the live video frame.
And a substep S102-2, acquiring the video additional information from the buffer zone according to the transmission delay and by combining the acquisition time stamp of the video additional information, and compressing the video additional information.
And the acquisition time stamp of the video additional information is the same as the acquisition time stamp of the corresponding live broadcast image.
As one way, the sub-step S102-2 may be:
(1) and calculating the maximum time difference between the video additional information stored in the cache region according to the acquisition time stamp of the video additional information stored in the cache region. It can be understood that the buffer area stores the video additional information in a first-in first-out mode, so that the maximum time difference can be obtained by calculating the acquisition time difference between the first-in video additional information and the last-in video additional information in the buffer area.
It should be noted that the processing time consumption of each frame of live image from the extraction of the video additional information to the storage in the buffer area is almost the same, that is, the corresponding acquisition time difference between the acquisition time stamps of two video additional information written in the buffer area also represents the time difference of the two video additional information written in the buffer area. Therefore, in the case of continuously processing the video stream, the corresponding maximum time difference in the buffer may represent the staying time of the first video additional information in the buffer. Of course, the maximum time difference also increases with the newly written video additional information in the buffer.
(2) And under the condition that the maximum time difference is matched with the transmission delay, the video additional information stored in the buffer area is taken out and compressed.
In some embodiments, the maximum time difference may be matched to the transmission delay such that the difference between the maximum time difference and the transmission delay does not exceed the maximum transmission time consumption for transmitting the additional information packet.
Alternatively, the sub-step S102-2 may be: and under the condition that the sum of the storage duration of the video additional information written first in the buffer area and the transmission time consumption corresponding to the additional information data packet is equal to the transmission delay, taking out all the video additional information in the buffer area, and carrying out merging, compression and packaging processing.
It is understood that the buffer is emptied after the video additional information in the buffer is fetched. I.e. new video additional information continues to be received and cached.
In addition, when the maximum time difference is not matched with the transmission delay yet, but the video additional information corresponding to the last frame of live image is received, the video additional information stored in the buffer area can be triggered to be taken out and compressed.
And step S103, respectively sending the additional information data packet and the live video data to corresponding clients.
In some embodiments, the additional information data packets and the live video data may be transmitted to the corresponding viewer client using different transmission links.
Alternatively, the video data packet may be generated from a live image of live video data. As shown in fig. 7, the video data packets are transmitted to the corresponding viewer clients through the video transmission channels. And after the additional information data packet is generated, transmitting the additional information data packet to the corresponding audience client through the data packet transmission channel. There is no necessary precedence between the two transmissions.
In order to perform the corresponding steps in the above embodiments and various possible manners, an implementation manner of the video data processing apparatus 400 is given below, and optionally, the video data processing apparatus 400 may adopt the device structure of the electronic device 100 shown in fig. 2. Further, referring to fig. 8, fig. 8 is a functional block diagram of a video data processing apparatus 400 according to an embodiment of the present invention. It should be noted that the basic principle and the resulting technical effects of the video data processing apparatus 400 provided in the present embodiment are the same as those of the above embodiments, and for the sake of brief description, no part of the present embodiment is mentioned, and reference may be made to the corresponding contents in the above embodiments. The video data processing apparatus 400 includes: an extraction module 401, a compression module 402 and a sending module 403.
The extracting module 401 is configured to sequentially extract video additional information corresponding to each frame of live broadcast image in the live broadcast video data, and store the video additional information in a buffer area.
In some embodiments, the above step S101 may be performed by the extraction module 401.
A compressing module 402, configured to obtain multiple frames of the video additional information from the buffer and compress the obtained additional information to obtain an additional information data packet.
In some embodiments, step S102 described above may be performed by the compression module 402.
A sending module 403, configured to send the additional information data packet and the live video data to corresponding clients respectively.
In some embodiments, the step S103 may be performed by the sending module 403.
Second embodiment
Referring to fig. 9, fig. 9 shows a video data processing method according to an embodiment of the invention. The video data processing method may be applied to a client, as shown in fig. 9, and the video data processing method may include the following steps:
step S201, a video data packet and an additional information data packet corresponding to the live video data are received.
The additional information data packet is a data packet obtained by compressing the multi-frame video additional information extracted from the live video data.
The video data packet is generated based on a live image of the live video data.
In some embodiments, the generation process of the additional information packet may refer to the description in the first embodiment.
Step S202, after the video data packet and the additional information data packet are decoded respectively, the live broadcast image and the video additional information with corresponding identification are obtained for synchronous rendering.
The corresponding identification is used for identifying the live broadcast image and the video additional information with the corresponding relation. There is usually a correspondence between the video additional information and the live image from which it was extracted.
In some embodiments, the corresponding identifiers may be the same capture timestamp, that is, the live image and the video additional information have the same capture timestamp, and then the representation has the corresponding identifier therebetween.
In other embodiments, the corresponding identifier may also be a live video frame sequence number written in the video additional information, so that it is convenient to determine a live video and the video additional information having a corresponding relationship.
In some embodiments, the video data packets are decoded upon receipt. Similarly, the additional information packet is decoded immediately after the additional information packet is received. And determining live broadcast images and video additional information with corresponding identifications in the decoded live broadcast images and video additional information so as to synchronously render.
In some embodiments, the video data processing method may further include:
step S301, periodically calculating transmission time delay according to the acquired acquisition time stamp and the universal time of the live broadcast image.
The universal time may be an objective time for the viewing client to decompress the live images. And calculating the transmission delay through recording the acquisition time stamp of the live broadcast image and the corresponding world time.
In other embodiments, in order to reduce the transmission pressure of the live video data, the capture timestamp corresponding to the live image may be extracted as the corresponding video additional information. For example, the video additional information includes a frame number and an acquisition timestamp of a corresponding live image. Therefore, the acquisition time stamp can be periodically acquired from the video additional information, the world time of the decoded live broadcast image corresponding to the frame number of the video additional information is inquired, and finally, the transmission delay is calculated according to the acquisition time stamp and the corresponding world time.
Step S302, the transmission delay is fed back to the live broadcast server to guide the live broadcast server to create an additional information data packet.
It should be noted that, the step S102 in the first embodiment may be referred to for instructing the live broadcast server to create the additional information data packet by using the transmission delay, and details are not described herein again.
In order to execute the corresponding steps in the above embodiments and various possible manners, an implementation manner of the video data processing apparatus 500 is given below, and optionally, the video data processing apparatus 500 may adopt the device structure of the electronic device 100 shown in fig. 2. Referring to fig. 10, fig. 10 is a functional block diagram of a video data processing apparatus 500 according to an embodiment of the present invention. It should be noted that the basic principle and the resulting technical effects of the video data processing apparatus 500 provided in the present embodiment are the same as those of the above embodiments, and for the sake of brief description, no part of the present embodiment is mentioned, and reference may be made to the corresponding contents in the above embodiments. The video data processing apparatus 500 includes: a receiving module 501 and a rendering module 502.
A receiving module 501, configured to receive a video data packet and an additional information data packet corresponding to live video data; the additional information data packet is a data packet obtained by compressing multi-frame video additional information extracted from the live video data; the video data packet is generated based on a live image of the live video data.
In some embodiments, the step S201 may be performed by the receiving module 501.
And a rendering module 502, configured to obtain the live broadcast image and the video additional information with the corresponding identifiers after decoding the video data packet and the additional information data packet, respectively, so as to perform rendering synchronously.
In some embodiments, the above step S202 may be performed by the rendering module 502.
In some embodiments, the video data processing apparatus further comprises:
and the calculation module is used for periodically calculating the transmission time delay according to the acquired acquisition time stamp of the live broadcast image and the universal time for decoding the live broadcast image.
And the feedback module is used for feeding back the transmission delay to the live broadcast server so as to guide the live broadcast server to create the additional information data packet.
Alternatively, the modules may be stored in the memory 110 shown in fig. 2 in the form of software or Firmware (Firmware) or be fixed in an Operating System (OS) of the electronic device 100, and may be executed by the processor 120 in fig. 2. Meanwhile, data, codes of programs, and the like required to execute the above-described modules may be stored in the memory 110.
In summary, embodiments of the present invention provide a video data processing method, an apparatus, an electronic device, and a storage medium. The video data processing method comprises the steps of sequentially extracting video additional information corresponding to each frame of live broadcast image in live broadcast video data and storing the video additional information into a cache region; acquiring multiple frames of the video additional information from the buffer area and compressing the video additional information to obtain an additional information data packet; and respectively sending the additional information data packet and the live video data to corresponding clients. The additional information data packet and the live video data obtained after packaging are respectively sent to the client for playing, so that the live video data and the additional information data packet can reach the client, and occupation of transmission resources is reduced.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, the functional modules in the embodiments of the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (11)
1. A video data processing method is applied to a live broadcast server, and comprises the following steps:
sequentially extracting video additional information corresponding to each frame of live broadcast image in live broadcast video data, and storing the video additional information in a cache region;
acquiring multiple frames of the video additional information from the buffer area and compressing the video additional information to obtain an additional information data packet;
and respectively sending the additional information data packet and the live video data to corresponding clients.
2. The method of claim 1, wherein the step of retrieving the frames of the video additional information from the buffer for compression comprises:
acquiring transmission time delay of the live broadcast image transmitted to the client;
acquiring the video additional information from the cache region for compression according to the transmission delay and by combining with the acquisition timestamp of the video additional information;
and the acquisition time stamp of the video additional information is the same as the acquisition time stamp of the corresponding live broadcast image.
3. The video data processing method according to claim 2, wherein the step of obtaining the video additional information from the buffer for compression according to the transmission delay and with reference to the collection timestamp of the video additional information comprises:
calculating the maximum time difference between the video additional information stored in the cache region according to the acquisition time stamp of the video additional information stored in the cache region;
and under the condition that the maximum time difference is matched with the transmission time delay, the video additional information stored in the cache region is taken out and compressed.
4. The video data processing method according to claim 2, wherein the step of obtaining the video additional information from the buffer for compression according to the transmission delay and with reference to the collection timestamp of the video additional information comprises:
and under the condition that the sum of the storage duration of the video additional information written first in the cache region and the transmission time consumption corresponding to the additional information data packet is equal to the transmission delay, taking out all the video additional information in the cache region and compressing.
5. The video data processing method according to claim 1, wherein the step of sending the additional information data packet and the live video data to the corresponding client, respectively, comprises:
generating a video data packet according to a live image of the live video data;
sending the video data packet to the client through a video transmission channel;
and transmitting the additional information data packet to the client through a data packet transmission channel.
6. A video data processing method is applied to a client, and comprises the following steps:
receiving a video data packet and an additional information data packet corresponding to live video data; the additional information data packet is a data packet obtained by compressing multi-frame video additional information extracted from the live video data; the video data packet is generated based on a live image of the live video data;
and after the video data packet and the additional information data packet are respectively decoded, acquiring the live broadcast image and the video additional information with corresponding identifications so as to synchronously render.
7. The video data processing method according to claim 6, wherein the video data processing method further comprises:
periodically calculating transmission time delay according to the acquired acquisition time stamp of the live broadcast image and the universal time for decoding the live broadcast image;
and feeding back the transmission delay to a live broadcast server to guide the live broadcast server to create the additional information data packet.
8. A video data processing apparatus, applied to a live server, the video data processing apparatus comprising:
the extraction module is used for sequentially extracting video additional information corresponding to each frame of live broadcast image in the live broadcast video data and storing the video additional information into a cache region;
the compression module is used for acquiring multiple frames of the video additional information from the cache region and compressing the video additional information to obtain an additional information data packet;
and the sending module is used for sending the additional information data packet and the live video data to corresponding clients respectively.
9. A video data processing apparatus applied to a client, the video data processing apparatus comprising:
the receiving module is used for receiving a video data packet and an additional information data packet corresponding to the live video data; the additional information data packet is a data packet obtained by compressing multi-frame video additional information extracted from the live video data; the video data packet is generated based on a live image of the live video data;
and the rendering module is used for acquiring the live broadcast image and the video additional information with corresponding identifications after respectively decoding the video data packet and the additional information data packet so as to synchronously render.
10. An electronic device comprising a processor and a memory, the memory storing machine executable instructions executable by the processor to perform the method of any one of claims 1 to 7.
11. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110121083.XA CN112954374B (en) | 2021-01-28 | 2021-01-28 | Video data processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110121083.XA CN112954374B (en) | 2021-01-28 | 2021-01-28 | Video data processing method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112954374A true CN112954374A (en) | 2021-06-11 |
| CN112954374B CN112954374B (en) | 2023-05-23 |
Family
ID=76238981
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110121083.XA Active CN112954374B (en) | 2021-01-28 | 2021-01-28 | Video data processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112954374B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114173171A (en) * | 2021-12-08 | 2022-03-11 | 上海智驾汽车科技有限公司 | Multi-channel big data information synchronous storage and playback method |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090175594A1 (en) * | 2008-01-07 | 2009-07-09 | Samsung Electronics Co., Ltd. | Method and apparatus for providing additional information of video using visible light communication |
| CN103905846A (en) * | 2012-12-25 | 2014-07-02 | 中国电信股份有限公司 | Content pushing method based on IPTV and server |
| WO2017088415A1 (en) * | 2015-11-25 | 2017-06-01 | 乐视控股（北京）有限公司 | Method, apparatus and electronic device for video content retrieval |
| WO2017206398A1 (en) * | 2016-05-31 | 2017-12-07 | 乐视控股（北京）有限公司 | Method and device for video sharing |
| CN107809682A (en) * | 2017-11-14 | 2018-03-16 | 北京威速科技有限公司 | It is a kind of to adjust the video flow control method for sending amount of buffered data |
| CN108668158A (en) * | 2017-03-31 | 2018-10-16 | 武汉斗鱼网络科技有限公司 | A kind of barrage synchronous method, client and barrage server |
| CN108989907A (en) * | 2018-08-22 | 2018-12-11 | 深圳市网心科技有限公司 | Live data streams processing method and system, electronic device and computer readable storage medium |
| CN110392274A (en) * | 2019-07-17 | 2019-10-29 | 咪咕视讯科技有限公司 | Information processing method, equipment, client, system and storage medium |
| CN111010614A (en) * | 2019-12-26 | 2020-04-14 | 北京奇艺世纪科技有限公司 | Method, device, server and medium for displaying live caption |
| CN111083515A (en) * | 2019-12-31 | 2020-04-28 | 广州华多网络科技有限公司 | Method, device and system for processing live broadcast content |
| WO2020103203A1 (en) * | 2018-11-19 | 2020-05-28 | 网宿科技股份有限公司 | Method and apparatus for synchronizing whiteboard with video, computing device, and storage medium |
| US10715860B1 (en) * | 2017-11-29 | 2020-07-14 | Twitch Interactive, Inc. | Video stream with additional content areas |
| CN112272303A (en) * | 2020-09-23 | 2021-01-26 | 北京五八信息技术有限公司 | Live content display method and device, electronic equipment and storage medium |
-
2021
- 2021-01-28 CN CN202110121083.XA patent/CN112954374B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090175594A1 (en) * | 2008-01-07 | 2009-07-09 | Samsung Electronics Co., Ltd. | Method and apparatus for providing additional information of video using visible light communication |
| CN103905846A (en) * | 2012-12-25 | 2014-07-02 | 中国电信股份有限公司 | Content pushing method based on IPTV and server |
| WO2017088415A1 (en) * | 2015-11-25 | 2017-06-01 | 乐视控股（北京）有限公司 | Method, apparatus and electronic device for video content retrieval |
| WO2017206398A1 (en) * | 2016-05-31 | 2017-12-07 | 乐视控股（北京）有限公司 | Method and device for video sharing |
| CN108668158A (en) * | 2017-03-31 | 2018-10-16 | 武汉斗鱼网络科技有限公司 | A kind of barrage synchronous method, client and barrage server |
| CN107809682A (en) * | 2017-11-14 | 2018-03-16 | 北京威速科技有限公司 | It is a kind of to adjust the video flow control method for sending amount of buffered data |
| US10715860B1 (en) * | 2017-11-29 | 2020-07-14 | Twitch Interactive, Inc. | Video stream with additional content areas |
| CN108989907A (en) * | 2018-08-22 | 2018-12-11 | 深圳市网心科技有限公司 | Live data streams processing method and system, electronic device and computer readable storage medium |
| WO2020103203A1 (en) * | 2018-11-19 | 2020-05-28 | 网宿科技股份有限公司 | Method and apparatus for synchronizing whiteboard with video, computing device, and storage medium |
| CN110392274A (en) * | 2019-07-17 | 2019-10-29 | 咪咕视讯科技有限公司 | Information processing method, equipment, client, system and storage medium |
| CN111010614A (en) * | 2019-12-26 | 2020-04-14 | 北京奇艺世纪科技有限公司 | Method, device, server and medium for displaying live caption |
| CN111083515A (en) * | 2019-12-31 | 2020-04-28 | 广州华多网络科技有限公司 | Method, device and system for processing live broadcast content |
| CN112272303A (en) * | 2020-09-23 | 2021-01-26 | 北京五八信息技术有限公司 | Live content display method and device, electronic equipment and storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114173171A (en) * | 2021-12-08 | 2022-03-11 | 上海智驾汽车科技有限公司 | Multi-channel big data information synchronous storage and playback method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112954374B (en) | 2023-05-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11729465B2 (en) | System and method providing object-oriented zoom in multimedia messaging | |
| CN106162235B (en) | For the method and apparatus of Switch Video stream | |
| CN110868600B (en) | Target tracking video plug-flow method, display method, device and storage medium | |
| EP2940940B1 (en) | Methods for sending and receiving video short message, apparatus and handheld electronic device thereof | |
| CN111866525A (en) | Multi-view video playing control method and device, electronic equipment and storage medium | |
| CN113542795B (en) | Video processing method and device, electronic equipment and computer readable storage medium | |
| CN112073543B (en) | Cloud video recording method and system and readable storage medium | |
| CN111479162B (en) | Live data transmission method and device and computer readable storage medium | |
| CN112948627B (en) | Alarm video generation method, display method and device | |
| WO2014113684A1 (en) | System, apparatus and method for sharing a screen having multiple visual components | |
| JPWO2004004363A1 (en) | Image encoding device, image transmitting device, and image photographing device | |
| CN112019905A (en) | Live broadcast playback method, computer equipment and readable storage medium | |
| CN109040786A (en) | Transmission method, device, system and the storage medium of camera data | |
| CN110784757A (en) | Game live broadcast method and device in live broadcast client, medium and electronic equipment | |
| JP2007336263A (en) | Image processing method, apparatus, and program | |
| US10033930B2 (en) | Method of reducing a video file size for surveillance | |
| CN112954374B (en) | Video data processing method and device, electronic equipment and storage medium | |
| CN112153401B (en) | Video processing method, communication device and readable storage medium | |
| CN113973224A (en) | Method for transmitting media information, computing device and storage medium | |
| CN112235600B (en) | Method, device and system for processing video data and video service request | |
| CN111093091A (en) | Video processing method, server and system | |
| CN111314350A (en) | Image storage system, storage method, calling system and calling method | |
| CN113660540B (en) | Image information processing method, system, display method, device and storage medium | |
| CN110798700B (en) | Video processing method, video processing device, storage medium and electronic equipment | |
| CN109417585B (en) | Method, system and computer readable storage medium for image transmission, image compression and image restoration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |