CN112954433B - Video processing method, device, electronic equipment and storage medium - Google Patents
Video processing method, device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112954433B CN112954433B CN202110137850.6A CN202110137850A CN112954433B CN 112954433 B CN112954433 B CN 112954433B CN 202110137850 A CN202110137850 A CN 202110137850A CN 112954433 B CN112954433 B CN 112954433B
- Authority
- CN
- China
- Prior art keywords
- video
- frame
- code stream
- data
- custom data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title abstract description 26
- 238000012545 processing Methods 0.000 claims abstract description 59
- 230000005540 biological transmission Effects 0.000 claims abstract description 42
- 238000004806 packaging method and process Methods 0.000 claims abstract description 13
- 238000000034 method Methods 0.000 claims description 44
- 230000001360 synchronised effect Effects 0.000 claims description 42
- 238000003780 insertion Methods 0.000 claims description 4
- 230000037431 insertion Effects 0.000 claims description 4
- 238000004891 communication Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 9
- 230000001960 triggered effect Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 8
- 238000012544 monitoring process Methods 0.000 description 6
- 238000011161 development Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000006837 decompression Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000153 supplemental effect Effects 0.000 description 1
- 230000036962 time dependent Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/435—Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
- H04N21/4355—Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream involving reformatting operations of additional data, e.g. HTML pages on a television screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/437—Interfacing the upstream path of the transmission network, e.g. for transmitting client requests to a VOD server
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440218—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/44213—Monitoring of end-user related data
- H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/44213—Monitoring of end-user related data
- H04N21/44222—Analytics of user selections, e.g. selection of programs or purchase activity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/643—Communication protocols
- H04N21/6437—Real-time Transport Protocol [RTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8547—Content authoring involving timestamps for synchronizing content
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Social Psychology (AREA)
- Computer Networks & Wireless Communication (AREA)
- Databases & Information Systems (AREA)
- Computer Security & Cryptography (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The application discloses a video processing method, a video processing device, electronic equipment and a storage medium, wherein video frames are obtained by carrying out coding processing on video data to be transmitted according to a video coding protocol based on a video transmission request; packaging video auxiliary data to be transmitted into a custom data frame; and obtaining a target video code stream according to the video frame and the custom data frame, and transmitting the target video code stream through a designated channel, so that video data and video auxiliary data are transmitted through the same channel, and the transmission flow of the video data and the video auxiliary data is simplified.
Description
Technical Field
The present disclosure relates to the field of video technologies, and in particular, to a video processing method, apparatus, electronic device, and storage medium.
Background
Video technology, i.e. dynamic image transmission technology, is known in the telecommunication field as video service or video service, and is often referred to in the computer world as multimedia communication, streaming media communication, etc. Video communication technology is a major technology for implementing and completing video services. With the rapid development of internet technology, video technology is increasingly widely used, such as video live broadcast, video conference, video lesson, video chat, etc.
Typically, the terminal encodes and transmits the acquired video data. In addition to video data, there are often video auxiliary data such as subtitles, audio, and data generated by various application scenes, which are required to be transmitted. The video data and the video auxiliary data need to be synchronously displayed, and the video auxiliary data and the video data are processed and transmitted separately, so that the process is complicated.
Disclosure of Invention
In view of the above, the present invention provides a video processing method, apparatus, electronic device, and storage medium.
In a first aspect, an embodiment of the present application provides a video processing method, where the method includes:
based on the video sending request, coding the video data to be sent according to a video coding protocol to obtain video frames;
packaging video auxiliary data to be transmitted into a custom data frame;
and obtaining a target video code stream according to the video frame and the custom data frame, and sending the target video code stream through a designated channel.
In a second aspect, embodiments of the present application further provide a video processing method, where the method includes:
acquiring a target video code stream through a designated channel based on a video playing request;
obtaining a video frame and a custom data frame according to the target video code stream;
Decoding the video frames according to a video coding protocol to obtain video data;
acquiring video auxiliary data encapsulated in the custom data frame;
and playing the video data and the video auxiliary data.
In a third aspect, embodiments of the present application further provide a video processing apparatus, where the apparatus includes:
the video frame acquisition module is used for carrying out coding processing on video data to be transmitted according to a video coding protocol based on a video transmission request to obtain a video frame;
the self-defined data frame acquisition module is used for packaging the video auxiliary data to be sent into self-defined data frames;
and the target video code stream acquisition module is used for acquiring a target video code stream according to the video frame and the custom data frame and transmitting the target video code stream through a designated channel.
In a fourth aspect, embodiments of the present application further provide an electronic device, including:
one or more processors;
a memory;
one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications configured to perform the method as described above.
In a fifth aspect, embodiments of the present application also provide a computer readable storage medium having program code stored therein, the program code being callable by a processor to perform a method as described above.
According to the technical scheme provided by the invention, video data to be transmitted are subjected to coding processing according to a video coding protocol on the basis of a video transmission request to obtain video frames; packaging video auxiliary data to be transmitted into a custom data frame; and obtaining a target video code stream according to the video frame and the custom data frame, and transmitting the target video code stream through a designated channel, so that video data and video auxiliary data are transmitted through the same channel, and the transmission flow of the video data and the video auxiliary data is simplified.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that are required for the description of the embodiments will be briefly introduced below, and it is apparent that the drawings in the following description are only some embodiments of the present application, but not all embodiments. All other embodiments and figures obtained by those skilled in the art without any inventive effort based on the embodiments herein fall within the scope of the present invention.
FIG. 1 illustrates a schematic diagram of an application environment in accordance with an embodiment of the present application;
fig. 2 is a schematic flow chart of a video processing method according to an embodiment of the present application;
fig. 3 is a schematic flow chart of a video processing method according to another embodiment of the present application;
fig. 4 is a schematic flow chart of a video processing method according to another embodiment of the present application;
fig. 5 is a schematic structural diagram of a target video code stream according to another embodiment of the present application;
fig. 6 shows a schematic flow chart of step S340 in yet another embodiment of the present application;
fig. 7 is a schematic flow chart of a video processing method according to still another embodiment of the present application;
FIG. 8 is a schematic flow chart of step S450 in another embodiment of the present application;
fig. 9 is a schematic flow chart of a video processing method according to still another embodiment of the present application;
fig. 10 is a block diagram showing a video processing apparatus according to an embodiment of the present application;
fig. 11 is a block diagram showing a video processing apparatus according to still another embodiment of the present application;
FIG. 12 is a block diagram of an electronic device according to an embodiment of the present application;
Fig. 13 shows a block diagram of a computer-readable storage medium according to an embodiment of the present application.
Detailed Description
In order to enable those skilled in the art to better understand the present application, the following description will make clear and complete descriptions of the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application.
With the rapid development of internet technology, video technology is increasingly used. In general, a terminal collects video data and encodes the video data, and the encoded video data is transmitted through a designated channel. However, in the video transmission process, not only video files, but also video auxiliary data such as subtitles, audio, data generated by various application scenes, and the like are often transmitted. The video auxiliary data needs to be presented synchronously with the video data. However, the above-mentioned video auxiliary data is transmitted separately from the encoded video data, which is cumbersome in process, and affects the synchronous display of the video data and the video auxiliary data due to network delay, packet loss, decoder performance, etc.
In order to improve the above problems, the inventors of the present application provide a video processing method, apparatus, electronic device, and storage medium. The method comprises the following steps: the method comprises the steps of obtaining video frames by encoding video data to be transmitted according to a video encoding protocol based on a video transmission request; packaging video auxiliary data to be transmitted into a custom data frame; and obtaining a target video code stream according to the video frame and the custom data frame, and transmitting the target video code stream through a designated channel, so that video data and video auxiliary data are transmitted through the same channel, and the transmission flow of the video data and the video auxiliary data is simplified.
The application environment of the video method provided by the embodiment of the invention is described below.
Referring to fig. 1, fig. 1 shows a video processing system according to an embodiment of the present application, where the video processing system includes: a terminal 100 and a server 200.
Terminal 100 includes, but is not limited to, a notebook computer, desktop computer, tablet computer, smart phone, wearable electronic device, and the like. The terminal 100 may include an image capturing device, such as a camera, that may capture raw video data. The terminal 100 may also include a sound collection device, such as a microphone, that may collect raw audio data. The terminal 100 may also include a display device, such as a display screen, that may be used to present video data. The terminal 100 may further comprise playing means for playing back video data and video auxiliary data. The terminal 100 may also include a sound playing device, such as a speaker, that may be used to present audio data. The terminal 100 may further include a storage device that may store data of the terminal 100, such as video data, audio data, subtitle data, and the like.
The server 200 may be a single server 200 or a cluster of servers 200 composed of a plurality of servers 200. The server 200 can realize functions such as information transmission and reception and information processing. For example, the server 200 may receive a target video bitstream transmitted by the video capture terminal 100. The server 200 may transmit the received target video bitstream to other playback terminals 100.
The terminal 100 and the server 200 may communicate with each other via the internet. Optionally, the internet described above uses standard communication techniques and/or protocols. The internet is typically the internet, but may be any network including, but not limited to, a local area network (Local Area Network, LAN), metropolitan area network (Metropolitan Area Network, MAN), wide area network (Wide Area Network, WAN), mobile, wired or wireless network, private network, or any combination of virtual private networks. In some embodiments, data exchanged over the network is represented using techniques and/or formats including HyperText Mark-up Language (HTML), extensible markup Language (Extensible Markup Language, XML), and the like. All or some of the links may also be encrypted using conventional encryption techniques such as secure socket layer (Secure Socket Layer, SSL), transport layer security (Transport Layer Security, TLS), virtual private network (Virtual Private Network, VPN), internet protocol security (Internet Protocol Security, IPsec), and the like. In other embodiments, custom and/or dedicated data communication techniques may also be used in place of or in addition to the data communication techniques described above.
Embodiments of the present application will be described in detail below with reference to the accompanying drawings.
Referring to fig. 2, an embodiment of the present application provides a video processing method, which may be applied to a terminal 100, and the embodiment describes a step flow of the terminal 100 side, and the method may include steps S110 to S130.
Step S110, based on the video sending request, the video data to be sent is coded according to a video coding protocol to obtain video frames.
The video data is a continuous image sequence, and is composed of continuous images. The video data may be acquired by an image acquisition device of the terminal 100.
In this embodiment, the terminal 100 may be installed with an application program that can acquire video data, such as a camera application program, an instant messaging application program, a live broadcast application program, and the like. The image capturing terminal 100 of the terminal 100 may be triggered to capture video data by invoking the related functions of the application program, for example, invoking the functions of recording video, video chat, and online live broadcast, where the image capturing device of the terminal 100 captures images at regular time to capture video data.
The video transmission request may include an identifier of the video data to be transmitted, and according to the identifier, the video data to be transmitted may be determined, where the video transmission request is used to instruct the terminal 100 to process the video data to be transmitted.
Alternatively, the video transmission request may be triggered by the user. The terminal 100 may acquire a video transmission request triggered by a user by monitoring an input operation of the user. The input operation of the user may be either contact or non-contact. For example, the user may operate by contact input, such as clicking a button, option, etc. associated with the video transmission, such as transmitting video, sharing video, video chat, live, etc. buttons or options. The terminal 100 can detect the clicked object of the user by monitoring the user operation, thereby acquiring the video transmission request corresponding to the object, and processing the video data corresponding to the object. For another example, the user may perform a non-contact input operation, such as a voice command, a gesture command, etc., and the terminal 100 may detect the voice command, the gesture command, etc. input by the user by monitoring the user operation, thereby obtaining a video transmission request corresponding to the voice command and the gesture command.
Alternatively, the video transmission request may be triggered by a specified event. An event triggering the video transmission request may be preset. For example, the specified event may be an event of ending video recording, and triggering the video sending request to process the video data to be sent after ending video recording. The specified event may also be an event that is sent periodically, and after the video recording is started, the video sending request is triggered periodically to process the video data to be sent. The specified event may also be an event of quantitative transmission, and after video recording is started, the video data to be transmitted triggers a video transmission request to process the video data to be transmitted when the video data to be transmitted reaches a threshold value.
The video data is uncompressed data, and the data format of the video data may be YUV format, RGB format, or the like. The data volume of the video data is huge, and the video data needs to be encoded and compressed to remove redundancy in space and time dimensions. The video data is convenient to transmit after being compressed, so that the space can be saved, and the transmission efficiency is improved. Video data may be compressed in accordance with a video coding protocol. The video coding protocol may be an h.264 coding protocol, an h.265 coding protocol, or the like, and performs coding processing on video data to be transmitted according to the video coding protocol to obtain video frames. Video frames may include multiple types according to different video encoding protocols, such as in the h.264 encoding protocol, I frames, P frames, B frames, etc. I (Intra-coded picture) frames are Intra-coded image frames, I frames represent key frames, and the data of this frame picture is completely reserved, and can be finished only by the frame data when decoding, which is also called Intra picture (Intra picture). P (Predictive-coded Picture) frames, i.e., forward Predictive coded image frames. The P-frame represents the difference between this frame and a previous key frame (or P-frame), and the difference defined by this frame needs to be overlapped with the previously buffered picture when decoding, so as to generate the final picture. P frames are also understood as difference frames, which have no complete picture data and only data that differ from the picture of the previous frame. B (Bidirectionally predicted picture) frames are bi-directionally predictive coded image frames. The B frame is a bi-directional difference frame, that is, the B frame records the difference between the present frame and the previous and subsequent frames. That is, in order to decode the B frame, not only the previous buffer picture but also the subsequent picture is obtained, and the final picture is obtained by superimposing the previous and subsequent pictures with the present frame data. It will be appreciated that video data may be encoded into different types of video frames according to different video encoding protocols, and the invention is not limited in this regard.
Step S120, the video auxiliary data to be sent is packaged into a custom data frame.
Wherein the video auxiliary data includes audio data, subtitle data, index data, and the like. The video auxiliary data may be data that is simultaneously acquired by the terminal 100 when acquiring video data, for example, when the image acquisition device of the terminal 100 acquires an image to acquire video data, the sound acquisition device of the terminal 100 may acquire sound to acquire audio data. The video auxiliary data may also be user-added data for presentation with the video data as it is being played, such as user-added subtitle data at a location associated with the video data.
Typically the video auxiliary data and the video data are processed separately and transmitted separately. On the one hand, the video auxiliary data and the video data need to be processed respectively, the process is complex, and CPU resources are occupied. On the other hand, the flow may cause the problems of arrival or packet loss of the video data and the video auxiliary data in sequence due to network delay, and the video data may be received and played, but the video auxiliary data is not received yet, which increases difficulty in synchronous display of the video data and the video auxiliary data.
To simplify the flow of video data and video auxiliary data transmission, embodiments of the present application address this difficulty by constructing custom data frames.
In embodiments of the present application, constructing custom data frames for transmitting video auxiliary data may be implemented, for example, in the h.264 protocol using SEI (Supplemental Enhancement Information ) data frames.
In the embodiment of the application, the user-defined data frames are utilized to bear and transmit the video auxiliary data to be transmitted, the user-defined data frames are inserted among the video frames, the user-defined data frames are transmitted together with the video frames, the flow of data processing can be simplified, the video frames are transmitted together with the user-defined data frames, and the influence of network delay is reduced.
The type of the custom data frame can be set to be distinguished from the video frame, and the video frame and the custom data frame can be distinguished according to the type of the frame during decoding.
In some embodiments, the video auxiliary data may be compressed and encapsulated into custom data frames. The audio data may be encoded, for example, according to an audio encoding protocol, which may be, for example, an AAC or the like protocol, to obtain audio frames. Further, the audio frames are encapsulated into custom data frames, so that space can be greatly saved, and the custom data frames can carry and transmit more data.
In other embodiments, the video auxiliary data may be directly encapsulated into the custom data frame without compression processing. The subtitle data may be encapsulated directly into custom data frames, for example.
In still other embodiments, the same custom data frame may encapsulate one or more types of video auxiliary data at the same time.
Optionally, a specified field may be added to the header of the custom data frame to distinguish the data type encapsulated by the custom data. For example, audio data may be represented by "01" and subtitle data may be represented by "11". The data type of the video auxiliary data encapsulated by the custom data frame can be rapidly identified according to the appointed field of the custom data frame, and the video auxiliary data is sent to the corresponding module for processing.
Step S130, a target video code stream is obtained according to the video frame and the custom data frame, and the target video code stream is sent through a designated channel.
In general, the transmission of video data and video auxiliary data may be based on a network protocol of data communication. Such as RTMP (Real Time Messaging Protocol, real-time messaging protocol). RTMP specifies that terminal 100 and server 200 can establish a network Connection (Net Connection) that represents the underlying connectivity between terminal 100 and server 200. A plurality of network streams may be established between the terminal 100 and the server 200, the network streams representing channels of multimedia data, such as video data, video auxiliary data, and the like.
In the prior art both video data and video auxiliary data are handled and transmitted separately. For example, video data is processed to obtain a video code stream, and audio data is processed to form an audio code stream. The video stream and the audio stream are transmitted over respective designated channels, e.g., by establishing respective network streams, respectively. In this embodiment, the target video code stream is obtained by combining the video frame and the custom data frame
In this embodiment, the target video bitstream is obtained from the video frame and the custom data frame. It is understood that the target video bitstream may be composed of a plurality of video frames and one or more custom data frames. The target video code stream is different from the existing video code stream in that custom data frames encapsulated with video auxiliary data are inserted between video frames, and the video auxiliary data can be added without damaging the standard of the original video code stream (namely, the video frames can still be identified and played), and the target video code stream can be transmitted through a designated channel, such as an established network stream, so that the video data and the video auxiliary data are transmitted through the same designated channel, and the transmission flow of the video data and the video auxiliary data is reduced.
According to the video processing method provided by the embodiment of the application, the video data to be transmitted are subjected to coding processing according to the video coding protocol based on the video transmission request to obtain the video frame; packaging video auxiliary data to be transmitted into a custom data frame; and obtaining a target video code stream according to the video frame and the custom data frame, and transmitting the target video code stream through a designated channel, so that video data and video auxiliary data are transmitted through the same channel, and the transmission flow of the video data and the video auxiliary data is simplified.
Referring to fig. 3, another embodiment of the present application provides a video processing method, which can be applied to the video processing terminal 100, and the present embodiment describes a step flow on the terminal 100 side, and the method may include steps S210 to S250.
Step S210, based on the video sending request, the video data to be sent is coded according to a video coding protocol to obtain video frames.
Step S220, the video auxiliary data to be sent is packaged into the custom data frame.
The specific description of step S210 to step S220 refer to step S110 to step S120, and the detailed description thereof is omitted herein.
Step S230, obtaining an initial video code stream according to all the obtained video frames.
The video data to be transmitted is processed by a video coding protocol to obtain a plurality of video frames. The plurality of video frames are arranged in time sequence to obtain an initial video code stream. Alternatively, the video data may be provided with a time stamp according to the acquisition time. The plurality of video frames may be arranged in time sequence of the time stamps to obtain an initial video bitstream.
Step S240, inserting the custom data frame into the initial video code stream to obtain the target video code stream.
And inserting a custom data frame between video frames of the initial video code stream to obtain a target video code stream.
The setting position of the custom data frame can be any inter-frame of the video frame, and finally the custom data frame and the video frame form a target video code stream, namely, the video data and the video auxiliary data can be transmitted through the same channel. When the target video code stream is received, the custom data frame and the video frame are distinguished according to the frame type. And processing the video auxiliary data encapsulated by the custom data frame respectively, processing the video data encapsulated by the video frame, and realizing synchronous display by synchronizing the video data and the video auxiliary data. The video auxiliary data is provided with a time stamp. The video assistance data may be time stamped according to the acquisition time. For example, when audio data is acquired, a time-dependent time stamp may be set at the audio data. Optionally, the timestamp of the video auxiliary data may also be user-defined. For example, the user adds a corresponding subtitle when the specific video data is displayed, and the time stamp of the subtitle data is the same as the time stamp of the specific video data. When synchronizing video data and video auxiliary data, synchronization may be performed according to corresponding time stamps.
Step S250, the target video code stream is sent through a designated channel.
According to the video processing method provided by the other embodiment of the application, the video data to be transmitted are subjected to coding processing according to the video coding protocol based on the video transmission request to obtain video frames; packaging video auxiliary data to be transmitted into a custom data frame; and obtaining a target video code stream according to the video frame and the custom data frame, and transmitting the target video code stream through a designated channel, so that video data and video auxiliary data are transmitted through the same channel, the transmission flow of the video data and the video auxiliary data is simplified, and the difficulty of synchronous display is reduced.
Referring to fig. 4, a further embodiment of the present application provides a video processing method, which can be applied to the video processing terminal 100, and the present embodiment describes a step flow on the video processing terminal 100 side, and the method may include steps S310 to S350.
Step S310, based on the video sending request, the video data to be sent is coded according to a video coding protocol to obtain video frames.
Step 320, the video auxiliary data to be sent is encapsulated into a custom data frame.
Step S330, obtaining an initial video code stream according to all the obtained video frames.
The specific description of step S310 to step S330 refer to step S210 to step S230, and the detailed description thereof is omitted herein.
Step S340, inserting the custom data frame into the initial video code stream according to the time stamp of the custom data frame to obtain the target video code stream.
The target video code stream can be transmitted through the same designated channel, so that the data transmission flow is simplified and the influence of network delay is reduced. However, when the target video code stream is received, the video frames and the custom data frames are respectively decoded to obtain video data, and the data packaged by the custom data frames are obtained to obtain video auxiliary data. When playing video, the video data and the video auxiliary data are synchronously played by means of the time stamp, and the playing device is required to have the function of synchronizing the video data and the video auxiliary data, and the additional synchronization function is required to be developed when the playing device is developed. Moreover, the synchronization is performed by means of the time stamp, and the video data and the video auxiliary data are gradually misplaced due to errors in the time stamp and the accumulation of errors in the playing process, so that the synchronization difficulty is high.
In order to solve the synchronization problem of video data and video playing data during playing, the embodiment of the application further inserts the custom data frame into the initial video code stream according to the timestamp of the custom data frame so as to obtain the target video code stream.
Fig. 5 shows a schematic structural diagram of a target video code stream according to another embodiment of the present application. In fig. 6, the target video bitstream 110 includes video frames 111 and custom data frames 112. The video frames 111 include a first video frame 111a, a second video frame 111b, a third video frame 111c, a fourth video frame 111d, and a fifth video frame 111e. The video frames 111 are arranged in the order of time stamps, in order: first video frame 111 a-second video frame 111 b-third video frame 111 c-fourth video frame 111 d-fifth video frame 111e. The custom data frames 112 include a first custom data frame 112a and a second custom data frame 112b. In this embodiment, the time stamp of the first custom data frame 112a follows the second video frame 111b and precedes the third video frame 111c, and thus the first custom data frame 112a is inserted between the second video frame 111b and the third video frame 111 c. The second custom data frame 112b is time stamped after the fourth video frame 111d and before the fifth video frame 111e, and therefore the second custom data frame 112b is inserted between the fourth video frame 111d and the fifth video frame 111e. The target video bitstream 110 is obtained by inserting the custom data frame 112 into the video frame 111 according to the timestamp of the custom data frame 112. It should be understood that the present application is not limited thereto, and the target video bitstream 110 may further include other necessary information, for example, the target video bitstream may further include a frame header for indicating a start position of the target video bitstream. The target video stream may further include a frame end for indicating an end position of the target video stream, and of course the target video stream may further include other necessary information, which is not limited herein.
In some embodiments, in the initial video stream, the video frames are arranged in a time-stamped order. And inserting the custom data frame into the initial video code stream according to the time stamp of the custom data to obtain a target video code stream. The video frames and the custom data frames in the target video code stream are arranged in time sequence. When processing the target video code stream, the target video code stream can be sequentially processed according to the arrangement sequence of the video frames and the custom data frames, namely, the first frame is a video frame, the first frame video frame is obtained, and the video data is obtained for playing; the second frame is a custom data frame, the custom data frame of the second frame is obtained, and the video auxiliary data is obtained for playing through processing. Therefore, the video data and the video auxiliary data can be processed synchronously without additional processing, the video frames and the custom data frames are processed sequentially according to the arrangement sequence, and the playing sequence is related to each other, and unlike the process of respectively playing, the video frames and the custom data frames are played according to the respective time stamps in isolation. In this way, even if there is an error in the time stamp, it is possible to obtain remedy and correction when sequentially processing the video frames and the custom data frames in this order. For example, the first video frame has a time stamp of 1 st second, the first custom data frame has a time stamp of 2 nd second, and the second video frame has a time stamp of 3 rd second. And if the video frame and the custom data frame are processed respectively to obtain video data and video auxiliary data, synchronously displaying the video data and the video auxiliary data.
The first video data is obtained from the first video frame and the second video data is obtained from the second video frame, the first video data is played at the 1 st second, and the second video data is played at the 3 rd second. And obtaining first video auxiliary data according to the first custom data frame, obtaining second video auxiliary data according to the second custom data frame, playing the first video auxiliary data at the 1 st second, playing the first video auxiliary data at the 2 nd second, and playing the second video auxiliary data at the 4 th second.
The theoretical play sequence is:
playing video data: first video data (1 st second) -second video data (3 rd second)
Playing video auxiliary data: first video auxiliary data (1 st second) -second video auxiliary data (2 nd second).
However, in practice, if the video data and the video auxiliary data are played separately, because the playing time of each frame of data is difficult to control accurately, there is a slight gap between the data of each frame, and after the gap is accumulated, the unsynchronized data becomes more and more obvious. Moreover, the video data is nonlinear, and the video auxiliary data may have linear data, which may also cause deviation. An out-of-sync condition occurs slowly, either as fast as video data is played or as fast as video auxiliary data is played.
The actual playing sequence is:
playing video data: first video data (1 st second) -second video data (3 rd second)
Playing video auxiliary data: first video auxiliary data (1 st second) -second video auxiliary data (theoretically 2 nd second, possibly due to deviation, eventually and second video data are played at 3 rd second, resulting in a playback dislocation)
Therefore, in the prior art, there is a need to develop a synchronization method, to continuously correct the deviation during the playing of the video data and the non-video data, to increase the playing speed of the video data when the video data is slow, and to decrease the playing speed of the video data when the video data is fast. Which increases the difficulty of synchronization.
In the embodiment of the application, the video frames and the custom data frames are sequentially processed according to the arrangement sequence, in the above example, the second video auxiliary data is processed after being processed, so that the dislocation situation that the second video auxiliary data and the second video data are played simultaneously can not occur, and the difficulty of synchronization is greatly reduced.
Specifically, referring to fig. 6, fig. 6 shows a schematic flow chart of step S340 in another embodiment of the present application, where in the embodiment of the present application, step S340 may include:
Step S341, judging whether a synchronous video frame of the self-defined data frame exists in the initial video code stream, wherein the synchronous video frame is a video frame with the same time stamp as the self-defined data frame in the initial video code stream.
Wherein the synchronized video frames are video frames having the same timestamp as the custom data frames. Both may be disposed adjacent. The error of the playing time of the two frames is not large.
Step S342, if yes, setting the custom data frame and the synchronous video frame adjacently in the initial video code stream.
Alternatively, the custom data frame and the synchronous video frame may be processed first and then the synchronous video frame, i.e., the custom data frame is disposed before the synchronous video frame. The video auxiliary data may be played first and then the video data may be played.
Alternatively, the custom data frame and the synchronous video frame may be processed first and then the custom data frame, i.e., the custom data frame is disposed after the synchronous video frame. The video data may be played first and then the video auxiliary data.
Optionally, in order to precisely control the playing time, the synchronous video frame and the custom data frame may be set adjacently, and then the playing sequence is set for control.
Since video frames can obtain video data which can be played through decoding processing, the decoding processing is complex and requires a long time. In order to further improve the playing efficiency, if the synchronous video frame of the custom data frame exists in the initial video code stream, the custom data frame and the synchronous video frame are adjacently arranged in the initial video code stream, and the custom data frame is arranged behind the synchronous video frame. Firstly, the synchronous video frames are decoded, and the playing efficiency is improved.
Step S343, if not, inserting the custom data frame into the initial video code stream according to the sequence of the time stamps.
If the condition that the time stamps are the same does not exist, the custom data frames are sequentially inserted according to the sequence of the time stamps.
Step S350, the target video code stream is sent through a designated channel.
According to the video processing method provided by the further embodiment of the application, the video frame is obtained by carrying out coding processing on video data to be transmitted according to a video coding protocol based on a video transmission request; packaging video auxiliary data to be transmitted into a custom data frame; and obtaining a target video code stream according to the video frame and the custom data frame, and transmitting the target video code stream through a designated channel, so that video data and video auxiliary data are transmitted through the same channel, the transmission flow of the video data and the video auxiliary data is simplified, and simultaneously, the difficulty of synchronous display is reduced by setting the sequence of the video frame and the video auxiliary frame.
Referring to fig. 7, a video processing method is provided in a further embodiment of the present application, which is applicable to the video processing terminal 100, and the present embodiment describes a step flow on the video processing terminal 100 side, and the method may include steps S410 to S420.
Step S410, based on the video transmission request, the video data to be transmitted is encoded according to the video encoding protocol to obtain a video frame.
Step S420, the video auxiliary data to be sent is encapsulated to the custom data frame.
Step S430, obtaining an initial video code stream according to all the obtained video frames.
Step S440, inserting the custom data frame into the initial video code stream according to the time stamp of the custom data frame to obtain a target video code stream.
The specific description of step S410 to step S440 refer to step S310 to step S340, and will not be described herein.
Step S450, setting the inter-frame sequence of the custom data frame according to the time stamp of the custom data frame, wherein the inter-frame sequence represents the play sequence of the custom data frame and the video frames adjacent to the custom data frame.
The inter-frame sequence may be set in a preset field of the custom data frame. For example, in the h.264 coding protocol, each frame has a NALU header, occupies one byte, and is divided into three parts: forbidden_zero_bit, nal_ref_idc, nal_unit_type, occupy 1,2,5 bits, respectively. Where nal_ref_idc may represent the relationship between the current frame and the adjacent frame (if a video frame), such as simultaneous playing, or the current frame playing first, or the adjacent frame playing first.
Specifically, referring to fig. 8, fig. 8 shows a flowchart of step S450 in a further embodiment of the present application, where step S450 may include:
in step S451, if the time stamp of the custom data frame is the same as the time stamp of the adjacent video frame, the inter-frame sequence of the custom data frame is set to be the first sequence, and the first sequence characterizes that the custom data frame and the adjacent video frame with the same time stamp are played simultaneously.
If the time stamps are the same, the custom data frame and the video frame with the same time stamp are played simultaneously, thereby ensuring accurate playing.
Step 452, if the time stamp of the custom data frame is different from the time stamp of the adjacent video frame, setting the inter-frame sequence of the custom data frame as a second sequence, and sequentially playing the custom data frame and the adjacent video frame according to the arrangement sequence in the target video code stream by the second sequence.
If the time stamps are different, the custom data frame and the adjacent video frames are sequentially played.
Step S460, the target video code stream is sent through the designated channel.
According to the video processing method provided by the further embodiment of the application, the video frame is obtained by carrying out coding processing on video data to be transmitted according to a video coding protocol based on a video transmission request; packaging video auxiliary data to be transmitted into a custom data frame; and obtaining a target video code stream according to the video frame and the custom data frame, and transmitting the target video code stream through a designated channel, so that video data and video auxiliary data are transmitted through the same channel, the flow of video data and video auxiliary data transmission is simplified, and accurate playing is ensured by setting the sequence among frames.
Referring to fig. 9, a video processing method is provided in a further embodiment of the present application, which can be applied to the video display terminal 100, and the embodiment describes a step flow on the video display terminal 100 side, and the method may include steps S510 to S550.
Step S510, based on the video playing request, the target video code stream is obtained through the designated channel.
In this embodiment, the terminal 100 may be in accordance with an application program, such as a video playing application program, an instant messaging application program, a live broadcast application program, etc., which may obtain a target video code stream. The terminal 100 may be triggered to acquire the target video bitstream by invoking the related functions of the application.
The video playing request may include an identifier of the video data to be acquired, and according to the identifier, a target video code stream corresponding to the data to be acquired may be determined, where the video playing request is used to instruct the terminal 100 to acquire the target video code stream through a specified channel.
Alternatively, the video play request may be triggered by the user. The terminal 100 may acquire a video play request triggered by a user by monitoring an input operation of the user. The input operation of the user may be either contact or non-contact. For example, the user may operate by contact input, such as clicking on a button, option, etc. associated with video playback, such as playing video, video chat, watching live, etc. The terminal 100 can detect the clicked object of the user by monitoring the user operation, thereby obtaining the video playing request corresponding to the object, so as to obtain the target video code stream corresponding to the object. For another example, the user may perform a non-contact input operation, such as a voice command, a gesture command, etc., and the terminal 100 may detect the voice command, the gesture command, etc. input by the user by monitoring the user operation, thereby obtaining a video play request corresponding to the voice command and the gesture command.
Alternatively, the video play request may be triggered by a specified event. An event triggering a video play request may be preset. For example, the specified event may be an event that the playing of the previous video ends, such as when watching a television show, automatically playing the next set of videos after the playing of one set of videos ends.
Step S520, obtaining video frames and custom data frames according to the target video code stream.
The target video code stream is composed of video frames and custom data frames, and the video frames and the custom data frames can be obtained according to the target video code stream.
Step S530, the video frames are decoded according to the video coding protocol to obtain video data.
Where video frames are compressed data, playing video requires the use of uncompressed video data. The video frames may be decoded according to a video encoding protocol to obtain video data. The video encoding protocol may be an h.264 encoding protocol, an h.265 encoding protocol, or the like.
Step S540, obtaining the video auxiliary data encapsulated in the custom data frame.
Wherein, the user-defined data frame is encapsulated with video auxiliary data, and the video auxiliary data can be obtained through decapsulation. The video auxiliary data may be compressed data or uncompressed data. When the acquired video auxiliary data is compressed data, further decompression processing is required for the compressed video auxiliary data. For example, the acquired video auxiliary data is an audio frame, and decoding processing is required for the audio frame to obtain audio data.
It can be appreciated that the video frames and the custom data frames can be processed sequentially according to the arrangement sequence of the video frames and the custom data frames in the target video stream.
Step S550, playing the video data and the video auxiliary data.
In some embodiments, the video data in the video frame and the video auxiliary data in the custom data frame may be acquired separately, and then the video data and the video auxiliary data are synchronized according to the time stamp, and then played.
In other embodiments, the video data and the video auxiliary data may be played sequentially in the order of the video frames and the custom data frames.
Optionally, if the video frame precedes the custom data frame, the video data is played first, and then the video auxiliary data is played.
Optionally, if the video frame follows the custom data frame, the video auxiliary data is played first, and then the video data is played.
In still other embodiments, if the custom data frame sets an inter-frame order, the playback is performed in the set inter-frame order.
Optionally, if the inter-frame sequence of the custom data frames is the first sequence, the video auxiliary data is played synchronously with the video data with the same time stamp.
Optionally, if the inter-frame sequence of the custom data frame is the second sequence, the video auxiliary data and the adjacent video data are sequentially played in the arrangement sequence.
According to the video processing method, video frames are obtained by encoding video data to be transmitted according to a video encoding protocol based on a video transmission request; packaging video auxiliary data to be transmitted into a custom data frame; and obtaining a target video code stream according to the video frame and the custom data frame, and transmitting the target video code stream through a designated channel, so that video data and video auxiliary data are transmitted through the same channel, and the transmission flow of the video data and the video auxiliary data is simplified.
Referring to fig. 10, a video processing apparatus 300 according to an embodiment of the invention is shown, where the video processing apparatus 300 includes: a video frame acquisition module 310, a custom data frame acquisition module 320, and a target video bitstream acquisition module 330.
A video frame acquisition module 310, configured to perform encoding processing on video data to be transmitted according to a video encoding protocol to obtain a video frame based on a video transmission request;
the custom data frame acquisition module 320 is configured to encapsulate video auxiliary data to be sent into custom data frames;
The target video code stream obtaining module 330 is configured to obtain a target video code stream according to the video frame and the custom data frame, and send the target video code stream through a designated channel.
Further, the video processing apparatus 300 further includes:
and the initial video code stream acquisition module is used for acquiring an initial video code stream according to all the acquired video frames.
And the custom data frame inserting module is used for inserting custom data frames into the initial video code stream to obtain a target video code stream.
And the target video code stream sending module is used for sending the target video code stream through a designated channel.
Further, the video processing apparatus 300 further includes a timestamp custom data insertion module.
And the time stamp custom data insertion module is used for inserting custom data frames into the initial video code stream according to the time stamps of the custom data frames so as to obtain a target video code stream.
Further, the timestamp custom data insertion module comprises a synchronous video frame judging unit, a first setting unit and a second setting unit.
And the synchronous video frame judging unit is used for judging whether synchronous video frames of the custom data frames exist in the initial video code stream or not, wherein the synchronous video frames are video frames with the same time stamp as the custom data frames in the initial video code stream.
And the first setting unit is used for adjacently setting the custom data frame and the synchronous video frame in the initial video code stream if the custom data frame and the synchronous video frame exist.
The first setting unit is further configured to, if the synchronous video frame of the custom data frame exists in the initial video code stream, set the custom data frame adjacent to the synchronous video frame in the initial video code stream, and set the custom data frame after the synchronous video frame.
And the second setting unit is used for inserting the custom data frames into the initial video code stream according to the sequence of the time stamps if the custom data frames do not exist.
Further, the video processing apparatus 300 further includes an inter-frame sequence setting module.
And the inter-frame sequence setting module is used for setting the inter-frame sequence of the custom data frame according to the time stamp of the custom data frame, wherein the inter-frame sequence represents the play sequence of the custom data frame and the video frames adjacent to the custom data frame.
Further, the inter-frame sequence setting module includes a first sequence unit and a second sequence unit.
And the first sequence unit is used for setting the inter-frame sequence of the custom data frame as a first sequence if the time stamp of the custom data frame is the same as the time stamp of the adjacent video frame, and the first sequence represents that the custom data frame and the adjacent video frame with the same time stamp are played simultaneously.
And the second sequence unit is used for setting the inter-frame sequence of the custom data frame as a second sequence if the time stamp of the custom data frame is different from the time stamp of the adjacent video frame, and the second sequence characterizes the sequential playing of the custom data frame and the adjacent video frame according to the arrangement sequence in the target video code stream.
Referring to fig. 11, a video processing apparatus 400 according to another embodiment of the present invention is shown, where the video processing apparatus 400 includes: a target video bitstream acquisition module 410, a video frame and custom data frame acquisition module 420, a video data acquisition module 430, a video auxiliary data acquisition module 440, and a playback module 450.
The target video code stream obtaining module 410 is configured to obtain a target video code stream through a designated channel based on the video playing request.
The video frame and custom data frame acquisition module 420 is configured to obtain a video frame and a custom data frame according to the target video bitstream.
The video data obtaining module 430 is configured to decode the video frame according to a video encoding protocol to obtain video data.
The video auxiliary data obtaining module 440 is configured to obtain video auxiliary data encapsulated in the custom data frame.
And a playing module 450 for playing the video data and the video auxiliary data.
Referring to fig. 12, based on the video processing method described above, another electronic device 500 including a processor capable of executing the video processing method described above is further provided, where the electronic device 500 further includes one or more processors 510, a memory 520, and one or more application programs. The memory 520 stores therein a program capable of executing the contents of the foregoing embodiments, and the processor 510 can execute the program stored in the memory.
Processor 510 may include one or more cores for processing data and a message matrix unit, among other things. The processor 510 utilizes various interfaces and lines to connect various portions of the overall electronic device, perform various functions of the electronic device, and process data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 520, and invoking data stored in the memory. Alternatively, the processor 510 may be implemented in hardware in at least one of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), programmable logic array (Programmable Logic Array, PLA). The processor 510 may integrate one or a combination of several of a central processing unit (Central Processing Unit, CPU), an image processor (Graphics Processing Unit, GPU), and a modem, etc. The CPU mainly processes an operating system, a user interface, an application program and the like; the GPU is used for being responsible for rendering and drawing of display content; the modem is used to handle wireless communications. It will be appreciated that the modem may not be integrated into the processor and may be implemented solely by a single communication chip.
The Memory 520 may include a random access Memory (Random Access Memory, RAM) or a Read-Only Memory (Read-Only Memory). Memory 520 may be used to store instructions, programs, code sets, or instruction sets. The memory 520 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a video play request, a video send request, etc.), instructions for implementing various method embodiments described below, and the like. The storage data area may also store data created by the terminal 100 in use (e.g., video data, video auxiliary data), etc.
Referring to fig. 13, a block diagram of a computer readable storage medium according to an embodiment of the present application is shown. The computer readable storage medium 600 has stored therein program code 610, said program code 610 being callable by a processor to perform the method described in the above method embodiments.
The computer readable storage medium 600 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 600 comprises a non-volatile computer readable medium (non-transitory computer-readable storage medium). The computer readable storage medium 600 has storage space for program code 610 that performs any of the method steps described above. These program code 610 can be read from or written to one or more computer program products. The program code may be compressed, for example, in a suitable form.
The video processing method, the video processing device, the electronic equipment and the storage medium provided by the invention are used for obtaining video frames by carrying out coding processing on video data to be transmitted according to a video coding protocol based on a video transmission request; packaging video auxiliary data to be transmitted into a custom data frame; and obtaining a target video code stream according to the video frame and the custom data frame, and transmitting the target video code stream through a designated channel, so that video data and video auxiliary data are transmitted through the same channel, and the transmission flow of the video data and the video auxiliary data is simplified.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and are not limiting thereof; although the present application has been described in detail with reference to the foregoing embodiments, one of ordinary skill in the art will appreciate that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not drive the essence of the corresponding technical solutions to depart from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims (8)
1. A method of video processing, the method comprising:
Based on the video sending request, coding the video data to be sent according to a video coding protocol to obtain video frames;
packaging video auxiliary data to be transmitted into a custom data frame;
obtaining an initial video code stream according to all the obtained video frames;
inserting the custom data frame into the initial video code stream according to the timestamp of the custom data frame to obtain a target video code stream; the target video code stream is used for sequentially processing according to the arrangement sequence of the video frames and the custom data frames when processing;
the target video code stream is sent through a designated channel,
inserting the custom data frame into the initial video code stream according to the timestamp of the custom data frame to obtain a target video code stream, wherein the method comprises the following steps:
judging whether a synchronous video frame of a self-defined data frame exists in the initial video code stream or not, wherein the synchronous video frame is a video frame with the same time stamp as the self-defined data frame in the initial video code stream;
if so, setting the custom data frame and the synchronous video frame adjacently in the initial video code stream to obtain a target video code stream;
If not, inserting the custom data frames into the initial video code stream according to the sequence of the time stamps so as to obtain the target video code stream.
2. The method of claim 1, wherein the disposing the custom data frame adjacent to the synchronized video frame in the initial video bitstream, if any, comprises:
if the synchronous video frame of the custom data frame exists in the initial video code stream, setting the custom data frame and the synchronous video frame adjacently in the initial video code stream, and setting the custom data frame behind the synchronous video frame.
3. The method of claim 1, wherein after inserting the custom data frame in the initial video bitstream according to the timestamp of the custom data frame to obtain a target video bitstream, the method further comprises:
setting the inter-frame sequence of the custom data frame according to the timestamp of the custom data frame, wherein the inter-frame sequence represents the play sequence of the custom data frame and the video frame adjacent to the custom data frame.
4. A method according to claim 3, wherein said setting the inter-frame order of the custom data frames according to the time stamps of the custom data frames comprises:
If the time stamp of the custom data frame is the same as the time stamp of the adjacent video frame, setting the inter-frame sequence of the custom data frame as a first sequence, wherein the first sequence represents that the custom data frame and the video frame with the same adjacent time stamp are played simultaneously;
if the time stamp of the custom data frame is different from the time stamp of the adjacent video frame, setting the inter-frame sequence of the custom data frame as a second sequence, wherein the second sequence represents that the custom data frame and the adjacent video frame are sequentially played according to the arrangement sequence in the target video code stream.
5. A method of video processing, the method comprising:
acquiring a target video code stream through a designated channel based on a video playing request; the target video code stream is obtained by inserting a custom data frame according to the time stamp of the custom data frame in an initial video code stream; the target video code stream is used for sequentially processing according to the arrangement sequence of video frames and the custom data frames when processing, and the video frames are obtained by encoding the video data to be transmitted according to a video encoding protocol based on a video transmission request; the initial video code stream is obtained through all obtained video frames;
Obtaining a video frame and a custom data frame according to the target video code stream;
decoding the video frames according to a video coding protocol to obtain video data;
acquiring video auxiliary data encapsulated in the custom data frame;
the target video code stream is obtained by inserting the custom data frame into an initial video code stream according to the timestamp of the custom data frame in the following way:
judging whether a synchronous video frame of a self-defined data frame exists in the initial video code stream or not, wherein the synchronous video frame is a video frame with the same time stamp as the self-defined data frame in the initial video code stream;
if so, setting the custom data frame and the synchronous video frame adjacently in the initial video code stream to obtain a target video code stream;
if not, inserting the custom data frames into the initial video code stream according to the sequence of the time stamps so as to obtain the target video code stream.
6. A video processing apparatus, the apparatus comprising:
the video frame acquisition module is used for carrying out coding processing on video data to be transmitted according to a video coding protocol based on a video transmission request to obtain a video frame;
The self-defined data frame acquisition module is used for packaging the video auxiliary data to be sent into self-defined data frames;
the initial video code stream acquisition module is used for acquiring an initial video code stream according to all the acquired video frames;
the time stamp custom data insertion module is used for judging whether a synchronous video frame of a custom data frame exists in the initial video code stream or not, wherein the synchronous video frame is a video frame which has the same time stamp as the custom data frame in the initial video code stream; if so, setting the custom data frame and the synchronous video frame adjacently in the initial video code stream to obtain a target video code stream; if not, inserting the custom data frames into the initial video code stream according to the sequence of the time stamps so as to obtain the target video code stream; the target video code stream is used for sequentially processing according to the arrangement sequence of the video frames and the custom data frames when processing;
and the target video code stream sending module is used for sending the target video code stream through a designated channel.
7. An electronic device, comprising:
One or more processors;
a memory;
one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications configured to perform the method of any of claims 1-4.
8. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a program code, which is callable by a processor for executing the method according to any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110137850.6A CN112954433B (en) | 2021-02-01 | 2021-02-01 | Video processing method, device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110137850.6A CN112954433B (en) | 2021-02-01 | 2021-02-01 | Video processing method, device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112954433A CN112954433A (en) | 2021-06-11 |
| CN112954433B true CN112954433B (en) | 2024-01-09 |
Family
ID=76241051
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110137850.6A Active CN112954433B (en) | 2021-02-01 | 2021-02-01 | Video processing method, device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112954433B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113507628A (en) * | 2021-06-30 | 2021-10-15 | 深圳市圆周率软件科技有限责任公司 | Video data processing method and related equipment |
| CN113923513B (en) * | 2021-09-08 | 2024-05-28 | 浙江大华技术股份有限公司 | Video processing method and device |
| CN115695858A (en) * | 2022-11-08 | 2023-02-03 | 天津萨图芯科技有限公司 | SEI encryption-based virtual film production video master film coding and decoding system, method and platform |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018033152A1 (en) * | 2016-08-19 | 2018-02-22 | 中兴通讯股份有限公司 | Video playing method and apparatus |
| CN109348252A (en) * | 2018-11-01 | 2019-02-15 | 腾讯科技（深圳）有限公司 | Video broadcasting method, video transmission method, device, equipment and storage medium |
-
2021
- 2021-02-01 CN CN202110137850.6A patent/CN112954433B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018033152A1 (en) * | 2016-08-19 | 2018-02-22 | 中兴通讯股份有限公司 | Video playing method and apparatus |
| CN109348252A (en) * | 2018-11-01 | 2019-02-15 | 腾讯科技（深圳）有限公司 | Video broadcasting method, video transmission method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112954433A (en) | 2021-06-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112954433B (en) | Video processing method, device, electronic equipment and storage medium | |
| US11653036B2 (en) | Live streaming method and system, server, and storage medium | |
| WO2019205872A1 (en) | Video stream processing method and apparatus, computer device and storage medium | |
| EP3806477A1 (en) | Video transcoding system and method, apparatus, and storage medium | |
| CN112073543B (en) | Cloud video recording method and system and readable storage medium | |
| CN110868600A (en) | Target tracking video plug-flow method, display method, device and storage medium | |
| CN112752115B (en) | Live broadcast data transmission method, device, equipment and medium | |
| CN111770390B (en) | Data processing method, device, server and storage medium | |
| CN112565224B (en) | Video processing method and device | |
| US11570226B2 (en) | Protocol conversion of a video stream | |
| WO2024022317A1 (en) | Video stream processing method and apparatus, storage medium, and electronic device | |
| CN112788431A (en) | Video playing method, device, system, medium and browser based on HTML5 | |
| CN109756744B (en) | Data processing method, electronic device and computer storage medium | |
| JP2006270634A (en) | Digital broadcast synchronizing reproducing apparatus, stream synchronization reproducing apparatus, and stream synchronization reproducing system | |
| CN110351576B (en) | Method and system for rapidly displaying real-time video stream in industrial scene | |
| CN116112620A (en) | Processing method and system for improving video stream multipath merging stability | |
| CN112565799B (en) | Video data processing method and device | |
| CN115278323A (en) | Display device, intelligent device and data processing method | |
| CN114257771A (en) | Video playback method and device for multi-channel audio and video, storage medium and electronic equipment | |
| CN112135163A (en) | Video playing starting method and device | |
| JP2010081227A (en) | Moving image decoder | |
| CN114979736B (en) | Display device and audio and video synchronization method | |
| CN110798725A (en) | Data processing method and device | |
| US20240080533A1 (en) | Cloud-Based Input Latency Measurement | |
| CN115209163B (en) | Data processing method and device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |