CN108174164A - A kind of real time video processing method and device - Google Patents
A kind of real time video processing method and device Download PDFInfo
- Publication number
- CN108174164A CN108174164A CN201810040486.XA CN201810040486A CN108174164A CN 108174164 A CN108174164 A CN 108174164A CN 201810040486 A CN201810040486 A CN 201810040486A CN 108174164 A CN108174164 A CN 108174164A
- Authority
- CN
- China
- Prior art keywords
- video frame
- current video
- control signaling
- triggering
- receiving terminal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
- H04N21/4345—Extraction or processing of SI, e.g. extracting service information from an MPEG stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/643—Communication protocols
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/647—Control signaling between network components and server or clients; Network processes for video distribution between server and clients, e.g. controlling the quality of the video stream, by dropping packets, protecting content from unauthorised alteration within the network, monitoring of network load, bridging between two different networks, e.g. between IP and wireless
- H04N21/64784—Data processing by the network
- H04N21/64792—Controlling the complexity of the content stream, e.g. by dropping packets
Abstract
The present invention provides a kind of real time video processing method and device, and this method includes：Transmitting terminal acquires current video frame；Detection is currently with the presence or absence of triggering control event；If in the presence of obtaining the corresponding control signaling of triggering control event, control signaling being inserted into current video frame, the current video frame after transmission insertion is to receiving terminal；If being not present, current video frame is directly transmitted to receiving terminal.Receiving terminal receives the current video frame of transmitting terminal transmission；Determine current video frame whether comprising control signaling；If so, current video frame is played according to control signaling；If not, directly play current video frame.Control signaling is inserted into video frame by transmitting terminal in the present invention, and control signaling is made to reach receiving terminal simultaneously with video frame.Realize that Video Rendering is synchronous with control signaling in receiving terminal.The triggering control event triggering that control signaling is actively triggered by user, increases transmitting terminal and the interactive means of receiving terminal two parties, improves the participation and Experience Degree of user in real-time video.
Description
Technical field
The present invention relates to technical field of data processing, in particular to a kind of real time video processing method and device.
Background technology
At present, it is required for carrying out the transmission of real-time video, hair in each application field such as monitoring system and instant messaging
Sending end also needs to transmission control signaling while real-time video is transmitted, and is regarded in real time so that receiving terminal is handled according to control signaling
Frequently.
Currently, when real-time video is transmitted in the relevant technologies, video flowing from control signaling is passed respectively in different channels
Defeated.By different channel transfers it is difficult to ensure that the corresponding control signaling of video flowing reaches receiving terminal simultaneously.In order to realize
Video flowing is synchronous with control signaling, buffering area is added in the relevant technologies, and carry out timestamp synchronization, but is handed in real-time video
Under mutual scene, higher time delay can be caused.And based on UDP (User Datagram Protocol, User Datagram Protocol
View) transmitting video-frequency flow and during control signaling, packet loss phenomenon is certainly existed, in order to ensure that the corresponding control signaling of video flowing is equal
Receiving terminal can be reached, can also accordingly increase the complexity of programming and realization.
Invention content
In view of this, the embodiment of the present invention is designed to provide a kind of real time video processing method and device, to solve
Video flowing transmits respectively from control signaling in different channels in the prior art, leads to video flowing and control signaling is nonsynchronous asks
Topic.
In a first aspect, an embodiment of the present invention provides a kind of real time video processing method, the method includes：
Acquire current video frame；
Detection is currently with the presence or absence of triggering control event；
If detecting triggering control event, the corresponding control signaling of the triggering control event is obtained, by the control
Signaling is inserted into the current video frame, transmits the current video frame after insertion operation to receiving terminal；
If triggering control event is not detected, the current video frame is directly transmitted to receiving terminal.
With reference to first aspect, an embodiment of the present invention provides the first possible realization method of above-mentioned first aspect,
In, it is described that the control signaling is inserted into the current video frame, including：
The current video frame is parsed, supplemental enhancement information cell S EI is oriented from the current video frame；
The control signaling is inserted into the SEI.
With reference to first aspect, an embodiment of the present invention provides second of possible realization method of above-mentioned first aspect,
In, the triggering control event is pre-set user behavior；The detection currently whether there is triggering control event, including：
Identify the user images that the current video frame includes；
Extract the characteristics of image of the user images；
According to described image feature, determine to whether there is pre-set user behavior in the user images；
If it is, determine that there is currently triggering control events；If it is not, then it determines that there is currently no triggerings to control thing
Part.
With reference to first aspect, an embodiment of the present invention provides the third possible realization method of above-mentioned first aspect,
In, the triggering control event is that there are the sound that loudness is more than default decibel threshold；The detection is currently with the presence or absence of triggering
Control event, including：
While acquiring the current video frame, the corresponding audio file of the current video frame is recorded；
Judge whether the loudness for the sound that the audio file includes is more than the default decibel threshold；
If it is, determine that there is currently triggering control events；If it is not, then it determines that there is currently no triggerings to control thing
Part.
Second aspect, an embodiment of the present invention provides a kind of real time video processing method, the method includes：
Receive the current video frame of transmitting terminal transmission；
Determine the current video frame whether comprising control signaling；
If it is, the current video frame is played according to the control signaling；
If it is not, then directly play the current video frame.
With reference to second aspect, an embodiment of the present invention provides the first possible realization method of above-mentioned second aspect,
In, it is described to determine whether the current video frame includes control signaling, including：
The current video frame is parsed, SEI is oriented from the current video frame；
It whether checks in the SEI comprising control signaling.
With reference to second aspect, an embodiment of the present invention provides second of possible realization method of above-mentioned second aspect,
In, it is described that the current video frame is played according to the control signaling, including：
Determine the corresponding operational order of the control signaling；
The operational order is performed while the current video frame is played.
The third aspect, an embodiment of the present invention provides a kind of real time video processing device, described device includes：
Acquisition module, for acquiring current video frame；
Detection module, it is current with the presence or absence of triggering control event for detecting；
Module is inserted into, for when the detection module detects triggering control event, obtaining the triggering control event
The control signaling is inserted into the current video frame by corresponding control signaling；
Transmission module is used for transmission the current video frame after insertion operation to receiving terminal；The transmission module is also used
In when triggering control event is not detected in the detection module, the current video frame is directly transmitted to receiving terminal.
With reference to the third aspect, an embodiment of the present invention provides the first possible realization method of the above-mentioned third aspect,
In, the insertion module includes：
For parsing the current video frame, supplemental enhancement information is oriented from the current video frame for positioning unit
Cell S EI；
Unit is inserted into, for the control signaling to be inserted into the SEI.
Fourth aspect, an embodiment of the present invention provides a kind of real time video processing device, described device includes：
Receiving module, for receiving the current video frame of transmitting terminal transmission；
Determining module, for determining the current video frame whether comprising control signaling；
Playing module, for when the determining module determine the current video frame include control signaling when, according to described
Control signaling plays the current video frame；And determine that the current video frame does not include control for working as the determining module
During signaling processed, the current video frame is directly played.
In embodiments of the present invention, transmitting terminal acquisition current video frame；Detection is currently with the presence or absence of triggering control event；If
In the presence of, the corresponding control signaling of triggering control event is obtained, control signaling is inserted into current video frame, it is current after transmission insertion
Video frame is to receiving terminal；If being not present, current video frame is directly transmitted to receiving terminal.Receiving terminal receives the current of transmitting terminal transmission
Video frame；Determine current video frame whether comprising control signaling；If so, current video frame is played according to control signaling；If
It is no, directly play current video frame.Control signaling is inserted into video frame by transmitting terminal in the present invention, makes the same video frame of control signaling
Reach receiving terminal simultaneously.Realize that Video Rendering is synchronous with control signaling in receiving terminal.Control signaling is touched by what user actively triggered
The triggering of control event is sent out, increases transmitting terminal and the interactive means of receiving terminal two parties, improves the ginseng of user in real-time video
With degree and Experience Degree.
For the above objects, features and advantages of the present invention is enable to be clearer and more comprehensible, preferred embodiment cited below particularly, and coordinate
Appended attached drawing, is described in detail below.
Description of the drawings
It in order to illustrate the technical solution of the embodiments of the present invention more clearly, below will be to needed in the embodiment attached
Figure is briefly described, it should be understood that the following drawings illustrates only certain embodiments of the present invention, therefore is not construed as pair
The restriction of range, for those of ordinary skill in the art, without creative efforts, can also be according to this
A little attached drawings obtain other relevant attached drawings.
Fig. 1 shows the flow chart of a kind of real time video processing method that the embodiment of the present invention 1 is provided；
Fig. 2 shows another flow charts of a kind of real time video processing method that the embodiment of the present invention 1 is provided；
Fig. 3 shows a kind of real-time video transmission schematic diagram that the embodiment of the present invention 1 is provided；
Fig. 4 shows the flow chart of a kind of real time video processing method that the embodiment of the present invention 2 is provided；
Fig. 5 shows another flow chart of a kind of real time video processing method that the embodiment of the present invention 2 is provided；
Fig. 6 shows the structure diagram of a kind of real time video processing device that the embodiment of the present invention 3 is provided；
Fig. 7 shows the structure diagram of a kind of real time video processing device that the embodiment of the present invention 4 is provided；
Fig. 8 shows the structure diagram of a kind of real time video processing device that the embodiment of the present invention 6 is provided；
Fig. 9 shows the structure diagram of a kind of real time video processing device that the embodiment of the present invention 8 is provided.
Specific embodiment
Purpose, technical scheme and advantage to make the embodiment of the present invention are clearer, below in conjunction with the embodiment of the present invention
The technical solution in the embodiment of the present invention is clearly and completely described in middle attached drawing, it is clear that described embodiment is only
It is part of the embodiment of the present invention, instead of all the embodiments.The present invention being usually described and illustrated herein in the accompanying drawings is real
Applying the component of example can be configured to arrange and design with a variety of different.Therefore, below to provide in the accompanying drawings the present invention
The detailed description of embodiment is not intended to limit the range of claimed invention, but is merely representative of the selected reality of the present invention
Apply example.Based on the embodiment of the present invention, institute that those skilled in the art are obtained under the premise of creative work is not made
There is other embodiment, shall fall within the protection scope of the present invention.
During in view of transmitting real-time video in the relevant technologies, video flowing from control signaling is passed respectively in different channels
Defeated, it is easy to appear nonsynchronous problems for the control signaling for causing video flowing corresponding.And in the prior art to understand
The certainly asynchronous problem is added to buffering area, and carries out timestamp synchronization, can so cause very high time delay.Based on this,
An embodiment of the present invention provides a kind of real time video processing method and devices, are described below by embodiment.
Embodiment 1
Referring to Fig. 1, an embodiment of the present invention provides a kind of real time video processing method, this method specifically includes following step
Suddenly：
Step 101：Acquire current video frame.
The executive agent of the embodiment of the present invention is the transmitting terminal of real-time video.The transmitting terminal can be transmitting terminal, can also
For server.In the application scenarios such as net cast, video monitoring or Video chat, transmitting terminal is to the figure of user or shooting area
As carrying out captured in real-time, the current video frame of user or shooting area is acquired.
Step 102：Detection is currently with the presence or absence of triggering control event, if it is, step 103 is performed, if it is not, then holding
Row step 104.
Trigger control event can be pre-set user behavior, be more than there are loudness default decibel threshold sound or it is default when
Carve etc..
When it is pre-set user behavior to trigger control event, after collecting current video frame, figure is carried out to current video frame
As processing, the user images that current video frame includes are identified；Extract the characteristics of image of user images；According to characteristics of image, determine
It whether there is pre-set user behavior in user images；If it is, determine that there is currently triggering control event, subsequent execution steps
103 operation；If it is not, then determine that there is currently no triggering control event, the operations of subsequent execution step 104.
In the embodiment of the present invention, it can also detect in the following way and whether there is pre-set user row in current video frame
To specifically include：Determine the corresponding body part of specific user's behavior.After collecting current video frame, to current video frame
Image procossing is carried out, user profile is extracted from current video frame, above-mentioned body part is intercepted from the user profile and is corresponded to
Contour images, calculate the similarity factor between the contour images and specific user's behavior.If the similarity factor is more than default system
Number threshold value, it is determined that there is currently triggering control events.If the similarity factor is less than or equal to predetermined coefficient threshold value, it is determined that when
It is preceding that there is no triggering control events.
Transmitting terminal also records the corresponding audio file of current video frame while current video frame is acquired.When triggering is controlled
Event processed is there are during the sound that loudness is more than default decibel threshold, judges whether the loudness for the sound that audio file includes is more than
Default decibel threshold；If it is, determine that there is currently triggering control event, the operations of subsequent execution step 103；If not,
Then determine that there is currently no triggering control event, the operations of subsequent execution step 104.
When it is predetermined time to trigger control event, while acquiring current video frame, system time is obtained, judges that this is
Whether the system time is above-mentioned preset time, if it is, determine there is currently triggering control event, subsequent execution step 103
Operation.If it is not, then determine that there is currently no triggering control event, the operations of subsequent execution step 104.
In addition to the sound or predetermined time that above-mentioned pre-set user behavior, loudness are more than default decibel threshold, triggering control thing
Part may be other defined trigger conditions in advance, and it can also be that there are the sound of specific decibel such as to trigger control event.This
When detection current video frame in the presence or absence of triggering control event when, determine the decibel point of the corresponding audio file of current video frame
Cloth information, this decibel of distributed intelligence can be that decibel is distributed scatter plot.It whether determines in this decibel of distributed intelligence comprising above-mentioned spy
Determine decibel, if comprising, it is determined that there is currently triggering control events.If do not include, it is determined that there is currently no triggerings to control thing
Part.The embodiment of the present invention does not limit triggering control event specifically, can carry out regulation triggering control thing in practical application according to demand
The concrete trigger condition of part.
Step 103:The corresponding control signaling of generation triggering control event, control signaling is inserted into current video frame, is passed
Current video frame after defeated insertion operation is to receiving terminal.
In embodiments of the present invention, the correspondence of configured in advance triggering control event and control signaling in transmitting terminal.
It determines after there is triggering control event in current video frame, from the correspondence of triggering control event and control signaling
It is middle to obtain corresponding control signaling.The control signaling of acquisition is inserted into current video frame.
Control signaling is inserted into current video frame by the embodiment of the present invention by operating as follows, is specifically included：
Current video frame is parsed, SEI (Supplemental Enhancement are oriented from current video frame
Information, supplemental enhancement information unit)；Control signaling is inserted into the SEI of current video frame.
After being inserted into control signaling in current video frame, current video frame is transferred to receiving terminal.Receiving terminal receives this
The control signaling is parsed after video frame from current video frame, and control signaling correspondence is performed when playing current video frame
Operational order.
In embodiments of the present invention, control signaling can be defined signaling identifier, such as 0 or 1 in advance.For example, hair
Configured in advance triggering control event " brandishing the right hand " and the correspondence of control signaling " 1 " in sending end, when detecting current video
When the image of user brandishes the right hand in frame, control signaling " 1 " is inserted into the SEI of current video frame, then by the current video
Frame is transferred to receiving terminal.The configured in advance correspondence of control signaling and operational order in receiving terminal, it is assumed that match in receiving terminal
The correspondence of control signaling " 1 " and operational order " window vibrations " is put, then receiving terminal is identified in current video frame and included
During control signaling " 1 ", vibrations play the window of current video frame.
It is above-mentioned when the current video frame for being inserted into control signaling is transferred to receiving terminal, in order to reduce in transmission process because of packet loss
The situation that control signaling is caused to be lost, the embodiment of the present invention are added to packet loss re-transmission policy in video transmitting procedure.To being inserted into
The current video frame of control signaling is enabled based on reliable UDP (User Datagram Protocol, user datagram transmission association
View) transmission, such as by KCP (fast and reliable agreement) come transmit be inserted into control signaling current video frame.
Worked as by KCP agreements come when transmitting the current video frame for being inserted into control signaling, control signaling is inserted by transmitting terminal
After preceding video frame, the current video frame is cached, and unique mark is distributed for the current video frame, it then should by KCP agreements
Current video frame and its unique mark are transferred to receiving terminal.After receiving terminal receives the current video frame and unique mark, by this
Unique mark returns to transmitting terminal.Transmitting terminal confirms that the current video frame is sent successfully when receiving the unique mark, continue into
The processing of row subsequent video frame.Do not have in a period of time if transmitting terminal is sent to receiving terminal after the current video frame and its unique mark
There is the unique mark for receiving receiving terminal return, then confirm the current video frame packet loss, this of transmitting terminal acquisition caching is current
The current video frame and its unique mark are sent to receiving terminal by video frame again by KCP agreements.Transmitting terminal should until confirming
Until current video frame is successfully sent to receiving terminal, the current video frame of caching is deleted, is further continued for carrying out subsequent video frame
Processing.
Step 104：Directly transmission current video frame is to receiving terminal.
When step 102 detects, there is currently no during triggering control event, current video frame directly to be transferred to receiving terminal.
After receiving terminal receives the current video frame, find not including control signaling in current video frame by parsing, then directly play
The current video frame.
When not including control signaling in current video frame, transmitting terminal can directly be incited somebody to action current by unreliable udp protocol
Video frame transmission is to receiving terminal, to improve the transmission speed of video frame.Certain transmitting terminal can also be passed by reliable UDP agreement
The defeated current video frame for being not inserted into control signaling.
For the ease of understanding real time video processing scheme provided in an embodiment of the present invention, carry out below in conjunction with the accompanying drawings specifically
It is bright.As shown in Fig. 2, S1：Acquire current video frame.S2：Encode current video frame.S3：Current video frame is written into control signaling
SEI in.S4：Judge whether comprising control signaling in current video frame, if so, step S5 is performed, if it is not, then performing step
Rapid S6.S5：Packet loss retransmission mechanism is enabled, current video frame is transmitted using the reliable UDPs such as KCP agreements agreement.S6：Using can not
Current video frame is transmitted by udp protocol.
All identical with current video frame for each frame of real-time video, transmitting terminal is according to the operation of step 101-104
Each frame of real-time video is handled, and be transferred to receiving terminal.The transmission schematic diagram of real-time video is illustrated in figure 3, it is real
When video in carry control signaling in the video frame that has, do not contain control signaling in some video frame.Carry regarding for control signaling
In frequency frame, control signaling is located in the SEI of video frame.Control signaling is inserted into video frame by the embodiment of the present invention so that control
Signaling processed can reach receiving terminal simultaneously with video frame, and it is synchronous with control signaling to realize Video Rendering.In addition, triggering control
Event can actively be triggered by user, is improved participation of the user in real-time video, is improved user experience.
In embodiments of the present invention, transmitting terminal acquisition current video frame；Detection is currently with the presence or absence of triggering control event；If
In the presence of, the corresponding control signaling of triggering control event is obtained, control signaling is inserted into current video frame, it is current after transmission insertion
Video frame is to receiving terminal；If being not present, current video frame is directly transmitted to receiving terminal.Transmitting terminal inserts control signaling in the present invention
Enter in video frame, control signaling is made to reach receiving terminal simultaneously with video frame.Realize that Video Rendering and control signaling are same in receiving terminal
Step.The triggering control event triggering that control signaling is actively triggered by user, increases transmitting terminal and the interaction of receiving terminal two parties
Means improve the participation and Experience Degree of user in real-time video.
Embodiment 2
Referring to Fig. 4, an embodiment of the present invention provides a kind of real time video processing method, this method specifically includes following step
Suddenly：
Step 201：Receive the current video frame of transmitting terminal transmission.
The executive agent of the embodiment of the present invention is the receiving terminal of real-time video.Forward sight is worked as in receiving terminal reception transmitting terminal transmission
Frequency frame.
Step 202：Current video frame is determined whether comprising control signaling, if it is, step 203 is performed, if it is not, then
Perform step 204.
After receiving terminal receives current video frame, current video frame is parsed, SEI is oriented from current video frame.It checks
Whether control signaling is included in the SEI of the current video frame.If comprising subsequent execution step 203, if being walked not comprising subsequent execution
Rapid 204.
Step 203：Current video frame is played according to control signaling.
When step 202 is determined to include control signaling in current video frame, the corresponding operational order of control signaling is determined；
Operational order is performed while current video frame is played.
In embodiments of the present invention, the correspondence of control signaling and operational order has been pre-configured in receiving terminal.When true
When making in current video frame comprising control signaling, corresponding operation is obtained in the correspondence from control signaling with operational order
Instruction, the operational order is performed when playing current video frame.
For example, it is assumed that the correspondence of control signaling " 1 " and operational order " window vibrations " has been pre-configured in receiving terminal,
When then determining to include control signaling " 1 " in current video frame, while current video frame is played by broadcast window, vibrations should
Broadcast window.
Transmitting terminal is transmitted across when sending the current video frame for being inserted into control signaling in order to reduce in above-described embodiment 1
Because packet loss leads to the situation of control signaling loss in journey, transmitting terminal enables packet loss re-transmission policy, i.e., works as to being inserted into control signaling
Preceding video frame enables the transmission based on reliable UDP agreement, such as the current video of insertion control signaling is transmitted by KCP agreements
Frame.When transmitting terminal transmits current video frame to receiving terminal by KCP agreements, also the unique mark of current video frame is transferred to
Receiving terminal.After receiving terminal receives the current video frame and unique mark, which is returned into transmitting terminal, so as to send
Termination confirms that the current video frame is sent successfully when receiving the unique mark, so as to continue the processing of subsequent video frame.If
Packet drop has occurred in the transmission of current video frame, then receiving terminal does not receive current video frame and its unique mark, so sends out
Sending end does not receive the unique mark of receiving terminal return after a certain time, then confirms the current video frame packet loss, and transmitting terminal will
The current video frame can be transmitted again, until confirming that the current video frame is successfully sent to receiving terminal.
Step 204：Directly play current video frame.
After receiving terminal receives the current video frame, find not including control signaling in current video frame by parsing, then
Directly play the current video frame.
For the ease of understanding the processing method of real-time video provided in an embodiment of the present invention, said below in conjunction with the accompanying drawings
It is bright.As shown in figure 5, A1：Analyze the current video frame of transmitting terminal transmission.A2：Judge whether include SEI in current video frame, such as
Fruit is then to perform step A3, if it is not, then performing step A6.A3：Whether judge in the SEI of current video frame comprising control letter
It enables, if it is, step A4 is performed, if it is not, then performing step A5.A4：Control signaling is returned to operation layer processing, in business
Layer performs the corresponding operational order of control signaling.A5：Show current video frame.
All identical with current video frame for each frame of real-time video, receiving terminal is according to the operation of step 201-204
Each frame of real-time video is handled and shown.Control signaling is inserted into video frame by transmitting terminal so that control signaling
Receiving terminal can be reached simultaneously with video frame, it is synchronous with control signaling to realize Video Rendering in receiving terminal.In addition, triggering control
Event processed can actively be triggered by user, is improved participation of the user in real-time video, is improved user experience.
In embodiments of the present invention, receiving terminal receives the current video frame of transmitting terminal transmission；Whether determine current video frame
Include control signaling；If so, current video frame is played according to control signaling；If not, directly play current video frame.This hair
Control signaling is inserted into video frame by bright middle transmitting terminal, and control signaling is made to reach receiving terminal simultaneously with video frame.In receiving terminal reality
Existing Video Rendering is synchronous with control signaling.The triggering control event triggering that control signaling is actively triggered by user, increases transmitting terminal
With the interactive means of receiving terminal two parties, the participation and Experience Degree of user in real-time video are improved.
Embodiment 3
Referring to Fig. 6, an embodiment of the present invention provides a kind of real time video processing device, which is used to perform above-mentioned implementation
The real time video processing method that example 1 is provided, the device include：
Acquisition module 30, for acquiring current video frame；
Detection module 31, it is current with the presence or absence of triggering control event for detecting；
Module 32 is inserted into, for when detection module detects triggering control event, it is corresponding to obtain triggering control event
Control signaling is inserted into current video frame by control signaling；
Transmission module 33 is used for transmission the current video frame after insertion operation to receiving terminal；Transmission module 33 is additionally operable to work as
When triggering control event is not detected in detection module 31, current video frame is directly transmitted to receiving terminal.
Above-mentioned insertion module 32 includes：
Positioning unit for parsing current video frame, orients supplemental enhancement information cell S EI from current video frame；
Unit is inserted into, for control signaling to be inserted into SEI.
In embodiments of the present invention, triggering control event is pre-set user behavior；Detection module 31 at this time are worked as identifying
The user images that preceding video frame includes；Extract the characteristics of image of user images；According to characteristics of image, determine in user images whether
There are pre-set user behaviors；If it is, determine that there is currently triggering control events；If it is not, then determine that there is currently no touch
Send out control event.
In embodiments of the present invention, triggering control event can also be that there are the sound that loudness is more than default decibel threshold；
The corresponding audio file of current video frame while for acquiring current video frame, is recorded in detection module 31 at this time；Judge audio
Whether the loudness for the sound that file includes is more than default decibel threshold；If it is, determine that there is currently triggering control events；Such as
Fruit is no, it is determined that there is currently no triggering control events.
In embodiments of the present invention, transmitting terminal acquisition current video frame；Detection is currently with the presence or absence of triggering control event；If
In the presence of, the corresponding control signaling of triggering control event is obtained, control signaling is inserted into current video frame, it is current after transmission insertion
Video frame is to receiving terminal；If being not present, current video frame is directly transmitted to receiving terminal.Transmitting terminal inserts control signaling in the present invention
Enter in video frame, control signaling is made to reach receiving terminal simultaneously with video frame.Realize that Video Rendering and control signaling are same in receiving terminal
Step.The triggering control event triggering that control signaling is actively triggered by user, increases transmitting terminal and the interaction of receiving terminal two parties
Means improve the participation and Experience Degree of user in real-time video.
Embodiment 4
Referring to Fig. 7, an embodiment of the present invention provides a kind of real time video processing device, which is used to perform above-mentioned implementation
The real time video processing method that example 2 is provided, the device include：
Receiving module 40, for receiving the current video frame of transmitting terminal transmission；
Determining module 41, for determining current video frame whether comprising control signaling；
Playing module 42, for when determining module 41 determine current video frame include control signaling when, according to control signaling
Play current video frame；And for when determining module 41 determines that current video frame does not include control signaling, directly playing and working as
Preceding video frame.
Above-mentioned determining module 41, for parsing current video frame, orients SEI from current video frame；Checking in SEI is
It is no to include control signaling.
When control signaling is included in current video frame, above-mentioned playing module 42, for determining the corresponding behaviour of control signaling
It instructs；Operational order is performed while current video frame is played.When determining that current video frame does not include control signaling, directly
Play current video frame.
In embodiments of the present invention, receiving terminal receives the current video frame of transmitting terminal transmission；Whether determine current video frame
Include control signaling；If so, current video frame is played according to control signaling；If not, directly play current video frame.This hair
Control signaling is inserted into video frame by bright middle transmitting terminal, and control signaling is made to reach receiving terminal simultaneously with video frame.In receiving terminal reality
Existing Video Rendering is synchronous with control signaling.The triggering control event triggering that control signaling is actively triggered by user, increases transmitting terminal
With the interactive means of receiving terminal two parties, the participation and Experience Degree of user in real-time video are improved.
Embodiment 5
An embodiment of the present invention provides a kind of nonvolatile computer storage media, which is stored with meter
Calculation machine executable instruction, the computer executable instructions can perform the real time video processing method in above-described embodiment 1, specifically hold
Row following methods：Acquire current video frame；Detection is currently with the presence or absence of triggering control event；If detecting triggering control event,
The corresponding control signaling of triggering control event is then obtained, control signaling is inserted into current video frame, after transmitting insertion operation
Current video frame is to receiving terminal；If triggering control event is not detected, current video frame is directly transmitted to receiving terminal.
The computer executable instructions of computer storage media storage determine there is triggering in current video frame after performing
During control event, the corresponding control signaling of triggering control event is inserted into current video frame so that control signaling is with working as forward sight
Frequency frame reaches receiving terminal simultaneously, realizes that Video Rendering is synchronous with control signaling.In addition, triggering control event can be by with householder
Dynamic triggering, improves participation of the user in real-time video, improves user experience.
Embodiment 6
As shown in figure 8, an embodiment of the present invention provides a kind of real time video processing device, the real time video processing device 60
Including：Processor 61, memory 62 and bus 63, memory 62 are stored with execute instruction, when the apparatus is operative, processor 61
It is communicated between memory 62 by bus 63, processor 61 performs the following execute instruction stored in memory 62：Acquisition is worked as
Preceding video frame；Detection is currently with the presence or absence of triggering control event；If detecting triggering control event, triggering control event is obtained
Control signaling is inserted into current video frame by corresponding control signaling, transmits the current video frame after insertion operation to receiving terminal；
If triggering control event is not detected, current video frame is directly transmitted to receiving terminal.
The real time video processing device performs above-metioned instruction by processor 61, determines there is triggering control in current video frame
During event processed, the corresponding control signaling of triggering control event is inserted into current video frame so that control signaling and current video
Frame reaches receiving terminal simultaneously, realizes that Video Rendering is synchronous with control signaling.In addition, triggering control event can be by user actively
Triggering, improves participation of the user in real-time video, improves user experience.
Embodiment 7
An embodiment of the present invention provides a kind of nonvolatile computer storage media, which is stored with meter
Calculation machine executable instruction, the computer executable instructions can perform the real time video processing method in above-described embodiment 2, specifically hold
Row following methods：Receive the current video frame of transmitting terminal transmission；Determine current video frame whether comprising control signaling；If so,
Current video frame is then played according to control signaling；If it is not, then directly play current video frame.
After the computer executable instructions of computer storage media storage perform, transmitting terminal determines to deposit in current video frame
When triggering control event, the corresponding control signaling of triggering control event is inserted into current video frame, and control letter will be inserted into
The current video frame of order is transferred to receiving terminal.Control signaling reaches receiving terminal simultaneously with current video frame, is realized in receiving terminal
Video Rendering is synchronous with control signaling.In addition, triggering control event can actively be triggered by user, user is improved real-time
Participation during video, improves user experience.
Embodiment 8
As shown in figure 9, an embodiment of the present invention provides a kind of real time video processing device, the real time video processing device 80
Including：Processor 81, memory 82 and bus 83, memory 82 are stored with execute instruction, when the apparatus is operative, processor 81
It is communicated between memory 82 by bus 83, processor 81 performs the following execute instruction stored in memory 82：Receive hair
The current video frame of sending end transmission；Determine current video frame whether comprising control signaling；If it is, it is played according to control signaling
Current video frame；If it is not, then directly play current video frame.
The real time video processing device performs above-metioned instruction by processor 81, and transmitting terminal determines exist in current video frame
When triggering control event, the corresponding control signaling of triggering control event is inserted into current video frame, and will be inserted into control signaling
Current video frame be transferred to receiving terminal.Control signaling reaches receiving terminal simultaneously with current video frame, realizes and regards in receiving terminal
Frequency renders synchronous with control signaling.In addition, triggering control event can actively be triggered by user, improve user and regarded in real time
Participation during frequency, improves user experience.
The real time video processing device that the embodiment of the present invention is provided can be equipment on specific hardware or be installed on
Software or firmware in equipment etc..The technique effect of the device that the embodiment of the present invention is provided, realization principle and generation is with before
It is identical to state embodiment of the method, to briefly describe, device embodiment part does not refer to part, can refer to phase in preceding method embodiment
Answer content.It is apparent to those skilled in the art that for convenience and simplicity of description, the system of foregoing description,
The specific work process of device and unit can refer to the corresponding process in above method embodiment, and details are not described herein.
In embodiment provided by the present invention, it should be understood that disclosed device and method, it can be by others side
Formula is realized.The apparatus embodiments described above are merely exemplary, for example, the division of the unit, only one kind are patrolled
Volume function divides, and can have other dividing mode in actual implementation, in another example, multiple units or component can combine or can
To be integrated into another system or some features can be ignored or does not perform.Another point, shown or discussed is mutual
Coupling, direct-coupling or communication connection can be INDIRECT COUPLING or communication link by some communication interfaces, device or unit
It connects, can be electrical, machinery or other forms.
The unit illustrated as separating component may or may not be physically separate, be shown as unit
The component shown may or may not be physical unit, you can be located at a place or can also be distributed to multiple
In network element.Some or all of unit therein can be selected according to the actual needs to realize the mesh of this embodiment scheme
's.
In addition, each functional unit in embodiment provided by the invention can be integrated in a processing unit, also may be used
To be that each unit is individually physically present, can also two or more units integrate in a unit.
If the function is realized in the form of SFU software functional unit and is independent product sale or in use, can be with
It is stored in a computer read/write memory medium.Based on such understanding, technical scheme of the present invention is substantially in other words
The part contribute to the prior art or the part of the technical solution can be embodied in the form of software product, the meter
Calculation machine software product is stored in a storage medium, is used including some instructions so that a computer equipment (can be
People's computer, server or network equipment etc.) perform all or part of the steps of the method according to each embodiment of the present invention.
And aforementioned storage medium includes：USB flash disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), arbitrary access are deposited
The various media that can store program code such as reservoir (RAM, Random Access Memory), magnetic disc or CD.
It should be noted that：Similar label and letter represents similar terms in following attached drawing, therefore, once a certain Xiang Yi
It is defined in a attached drawing, does not then need to that it is further defined and explained in subsequent attached drawing, in addition, term " the
One ", " second ", " third " etc. are only used for distinguishing description, and it is not intended that instruction or hint relative importance.
Finally it should be noted that：Embodiment described above, only specific embodiment of the invention, to illustrate the present invention
Technical solution, rather than its limitations, protection scope of the present invention is not limited thereto, although with reference to the foregoing embodiments to this hair
It is bright to be described in detail, it will be understood by those of ordinary skill in the art that：Any one skilled in the art
In the technical scope disclosed by the present invention, it can still modify to the technical solution recorded in previous embodiment or can be light
It is readily conceivable that variation or equivalent replacement is carried out to which part technical characteristic；And these modifications, variation or replacement, do not make
The essence of corresponding technical solution is detached from the spirit and scope of technical solution of the embodiment of the present invention.The protection in the present invention should all be covered
Within the scope of.Therefore, protection scope of the present invention should be based on the protection scope of the described claims.
Claims (10)
- A kind of 1. real time video processing method, which is characterized in that the method includes：Acquire current video frame；Detection is currently with the presence or absence of triggering control event；If detecting triggering control event, the corresponding control signaling of the triggering control event is obtained, by the control signaling It is inserted into the current video frame, transmits the current video frame after insertion operation to receiving terminal；If triggering control event is not detected, the current video frame is directly transmitted to receiving terminal.
- 2. according to the method described in claim 1, it is characterized in that, described be inserted into the current video frame by the control signaling In, including：The current video frame is parsed, supplemental enhancement information cell S EI is oriented from the current video frame；The control signaling is inserted into the SEI.
- 3. according to the method described in claim 1, it is characterized in that, the triggering control event is pre-set user behavior；It is described Detection currently whether there is triggering control event, including：Identify the user images that the current video frame includes；Extract the characteristics of image of the user images；According to described image feature, determine to whether there is pre-set user behavior in the user images；If it is, determine that there is currently triggering control events；If it is not, then determine that there is currently no triggering control events.
- 4. according to the method described in claim 1, it is characterized in that, the triggering control event, which is that there are loudness, to be more than default point The sound of shellfish threshold value；The detection currently whether there is triggering control event, including：While acquiring the current video frame, the corresponding audio file of the current video frame is recorded；Judge whether the loudness for the sound that the audio file includes is more than the default decibel threshold；If it is, determine that there is currently triggering control events；If it is not, then determine that there is currently no triggering control events.
- A kind of 5. real time video processing method, which is characterized in that the method includes：Receive the current video frame of transmitting terminal transmission；Determine the current video frame whether comprising control signaling；If it is, the current video frame is played according to the control signaling；If it is not, then directly play the current video frame.
- 6. according to the method described in claim 5, it is characterized in that, whether described determine the current video frame comprising control letter It enables, including：The current video frame is parsed, SEI is oriented from the current video frame；It whether checks in the SEI comprising control signaling.
- 7. according to the method described in claim 5, it is characterized in that, described play the current video according to the control signaling Frame, including：Determine the corresponding operational order of the control signaling；The operational order is performed while the current video frame is played.
- 8. a kind of real time video processing device, which is characterized in that described device includes：Acquisition module, for acquiring current video frame；Detection module, it is current with the presence or absence of triggering control event for detecting；Module is inserted into, is corresponded to for when the detection module detects triggering control event, obtaining the triggering control event Control signaling, the control signaling is inserted into the current video frame；Transmission module is used for transmission the current video frame after insertion operation to receiving terminal；The transmission module is additionally operable to work as When triggering control event is not detected in the detection module, the current video frame is directly transmitted to receiving terminal.
- 9. device according to claim 8, which is characterized in that the insertion module includes：Positioning unit for parsing the current video frame, orients supplemental enhancement information unit from the current video frame SEI；Unit is inserted into, for the control signaling to be inserted into the SEI.
- 10. a kind of real time video processing device, which is characterized in that described device includes：Receiving module, for receiving the current video frame of transmitting terminal transmission；Determining module, for determining the current video frame whether comprising control signaling；Playing module, for when the determining module determine the current video frame include control signaling when, according to the control Signaling plays the current video frame；And determine that the current video frame does not include control and believes for working as the determining module When enabling, the current video frame is directly played.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810040486.XA CN108174164A (en) | 2018-01-16 | 2018-01-16 | A kind of real time video processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810040486.XA CN108174164A (en) | 2018-01-16 | 2018-01-16 | A kind of real time video processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108174164A true CN108174164A (en) | 2018-06-15 |
Family
ID=62514865
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810040486.XA Pending CN108174164A (en) | 2018-01-16 | 2018-01-16 | A kind of real time video processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108174164A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108650438A (en) * | 2018-07-06 | 2018-10-12 | 深圳市鼎盛威电子有限公司 | Digital TV front-end host and digital signal converting system |
| CN109474365A (en) * | 2018-12-29 | 2019-03-15 | 深圳市柠檬互动科技有限公司 | A kind of frame synchronization UDP network synchronization method |
| CN110233716A (en) * | 2019-05-31 | 2019-09-13 | 北京文香信息技术有限公司 | A kind of communication interaction method, apparatus, storage medium, terminal device and server |
| CN114630138A (en) * | 2022-03-14 | 2022-06-14 | 上海哔哩哔哩科技有限公司 | Configuration information issuing method and system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104854872A (en) * | 2012-12-13 | 2015-08-19 | 索尼公司 | Transmission device, transmission method, reception device, and reception method |
| CN106488291A (en) * | 2016-11-17 | 2017-03-08 | 百度在线网络技术（北京）有限公司 | The method and apparatus of simultaneous display file in net cast |
| CN106506326A (en) * | 2016-09-30 | 2017-03-15 | 北京数字家圆科技有限公司 | A kind of video call method, terminal and system |
| CN106789916A (en) * | 2016-11-21 | 2017-05-31 | 广州视源电子科技股份有限公司 | Network transfer method and device, network transfer method and device based on UDP |
| WO2017115678A1 (en) * | 2015-12-28 | 2017-07-06 | Sony Corporation | Transmission apparatus, transmission method, reception apparatus, and reception method for high dynamic range video data |
| CN107135399A (en) * | 2017-03-29 | 2017-09-05 | 网易传媒科技（北京）有限公司 | Interactive control method, equipment and the computer-readable recording medium of video |
| CN107197194A (en) * | 2017-06-27 | 2017-09-22 | 维沃移动通信有限公司 | A kind of video call method and mobile terminal |
-
2018
- 2018-01-16 CN CN201810040486.XA patent/CN108174164A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104854872A (en) * | 2012-12-13 | 2015-08-19 | 索尼公司 | Transmission device, transmission method, reception device, and reception method |
| WO2017115678A1 (en) * | 2015-12-28 | 2017-07-06 | Sony Corporation | Transmission apparatus, transmission method, reception apparatus, and reception method for high dynamic range video data |
| CN106506326A (en) * | 2016-09-30 | 2017-03-15 | 北京数字家圆科技有限公司 | A kind of video call method, terminal and system |
| CN106488291A (en) * | 2016-11-17 | 2017-03-08 | 百度在线网络技术（北京）有限公司 | The method and apparatus of simultaneous display file in net cast |
| CN106789916A (en) * | 2016-11-21 | 2017-05-31 | 广州视源电子科技股份有限公司 | Network transfer method and device, network transfer method and device based on UDP |
| CN107135399A (en) * | 2017-03-29 | 2017-09-05 | 网易传媒科技（北京）有限公司 | Interactive control method, equipment and the computer-readable recording medium of video |
| CN107197194A (en) * | 2017-06-27 | 2017-09-22 | 维沃移动通信有限公司 | A kind of video call method and mobile terminal |
Non-Patent Citations (1)
| Title |
|---|
| 赵瑞玉等编著: "《现代交换原理与技术》", 31 July 2015, 西安电子科技大学出版社 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108650438A (en) * | 2018-07-06 | 2018-10-12 | 深圳市鼎盛威电子有限公司 | Digital TV front-end host and digital signal converting system |
| CN109474365A (en) * | 2018-12-29 | 2019-03-15 | 深圳市柠檬互动科技有限公司 | A kind of frame synchronization UDP network synchronization method |
| CN110233716A (en) * | 2019-05-31 | 2019-09-13 | 北京文香信息技术有限公司 | A kind of communication interaction method, apparatus, storage medium, terminal device and server |
| CN114630138A (en) * | 2022-03-14 | 2022-06-14 | 上海哔哩哔哩科技有限公司 | Configuration information issuing method and system |
| CN114630138B (en) * | 2022-03-14 | 2023-12-08 | 上海哔哩哔哩科技有限公司 | Configuration information issuing method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108174164A (en) | A kind of real time video processing method and device | |
| US9094571B2 (en) | Video chatting method and system | |
| CN103634199B (en) | The display methods and device of notification message | |
| CN106487742B (en) | Method and device for verifying source address validity | |
| CN110267053A (en) | Live broadcasting method, apparatus and system | |
| CN104796448B (en) | The data processing method and device of network system | |
| CN104394421A (en) | Video frame processing method and device | |
| CN106303331A (en) | video recording method, terminal and system | |
| CN103731631B (en) | The method, apparatus and system of a kind of transmitting video image | |
| CN108268364A (en) | Anomalous event back method, device and equipment | |
| WO2010033351A2 (en) | Improved audio and video testing methodology | |
| CN109640113A (en) | A kind of processing method and proxy server of dilatory video data | |
| CN108184136B (en) | Video converging method and device | |
| CN110932931A (en) | Detection method and device for network delay of data center | |
| CN105553939A (en) | Method and device for determining blockage of streaming media | |
| CN110247985A (en) | A kind of resource downloading method, device, electronic equipment and medium | |
| CN109821235B (en) | Game video recording method, device and server | |
| CN106604149A (en) | Method and device for displaying interaction condition of direct broadcast room | |
| CN110677718A (en) | Video identification method and device | |
| CN103197836A (en) | Interactive method, device and system of multimedia information | |
| CN110505490A (en) | Multi-medium data recognition methods and system | |
| AU2018275194A1 (en) | Temporal placement of a rebuffering event | |
| CN106341698A (en) | Video live broadcast processing method and device | |
| CN105898358A (en) | Method and device for sending video data | |
| CN106254456B (en) | Data synchronization method and data synchronization system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210302 Address after: Room 1702-03, Lantian Hesheng building, 32 Zhongguancun Street, Haidian District, Beijing 100082 Applicant after: BEIJING CENTURY TAL EDUCATION TECHNOLOGY Co.,Ltd. Address before: 102200 a1-05-001 maker Plaza, 338 Huilongguan East Street, Changping District, Beijing Applicant before: BEIJING SANTICLOUD TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180615 |
|
| RJ01 | Rejection of invention patent application after publication |