CN115550683A - Video data transmission method and device - Google Patents
Video data transmission method and device Download PDFInfo
- Publication number
- CN115550683A CN115550683A CN202110729347.XA CN202110729347A CN115550683A CN 115550683 A CN115550683 A CN 115550683A CN 202110729347 A CN202110729347 A CN 202110729347A CN 115550683 A CN115550683 A CN 115550683A
- Authority
- CN
- China
- Prior art keywords
- video
- link
- frame
- video data
- frames
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 294
- 238000000034 method Methods 0.000 title claims abstract description 108
- 238000003860 storage Methods 0.000 claims description 13
- 238000003780 insertion Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 8
- 238000004891 communication Methods 0.000 description 35
- 238000012545 processing Methods 0.000 description 18
- 230000006870 function Effects 0.000 description 16
- 238000010586 diagram Methods 0.000 description 15
- 238000007726 management method Methods 0.000 description 15
- 230000008569 process Effects 0.000 description 9
- 230000005236 sound signal Effects 0.000 description 9
- 230000001413 cellular effect Effects 0.000 description 5
- 238000001514 detection method Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000037431 insertion Effects 0.000 description 3
- 230000004931 aggregating effect Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 238000007667 floating Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000005266 casting Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The embodiment of the application provides a method and a device for transmitting video data, relates to the technical field of streaming media transmission, and can ensure smooth transmission of important information in the video data in the transmission process of the video data to a certain extent, wherein the method comprises the following steps: detecting the transmission quality of N links; determining video frames to be transmitted corresponding to the N links respectively according to the transmission quality of the N links; when the transmission quality of the kth link meets a preset condition, the video frames to be transmitted corresponding to the kth link are all video frames of the first video data; under the condition that the transmission quality of the kth link does not meet a preset condition, the video frames to be transmitted corresponding to the kth link are partial video frames in all the first video data; the video frame to be transmitted comprises all key data of first video data; and transmitting the video frames to be transmitted respectively corresponding to the N links in the N links.
Description
Technical Field
The embodiment of the application relates to the technical field of streaming media transmission, in particular to a video data transmission method and device.
Background
Video is applied more and more widely in real life, such as video call, live network broadcast, near-field screen projection and the like. The transmission of video data is an important factor affecting the playing quality of video data.
Currently, in the process of transmitting video data, a video sending end transmits data packets of video data on two links respectively to implement redundant transmission of video data, for example, all data packets of video data are transmitted on a first link, and all data packets of video data are also transmitted on a second link. In the two links, one link may be a main link, the other link is an auxiliary link, and data packets transmitted on the two links are backups for each other.
However, when the bandwidth of at least one of the two links for transmitting video data is limited (i.e. the two transmission links are strong and weak), the limited link may lose a part of video data when transmitting video data, and therefore, the two links can no longer back up each other, in this case, if a part of video data is lost due to other factors (for example, a receiving end is restarted) during the transmission of video data on the unlimited link, the part of video data is the same as the video data lost by the limited link, and the video data lost by the two links includes important information of video data, then the video receiving end may have problems of blocking, screen floating, and high delay when playing the received video data.
Disclosure of Invention
The embodiment of the application provides a method and a device for transmitting video data, which can ensure the normal transmission of important information in the video data in the transmission process of the video data to a certain extent.
In order to achieve the above purpose, the embodiment of the present application adopts the following technical solutions:
in a first aspect, an embodiment of the present application provides a method for transmitting video data, where the method includes: detecting the transmission quality of N links; the video data transmission method comprises the following steps that N links are used for transmitting first video data, the N links are links in different networks, the first video data comprise a plurality of video frames, and N is a positive integer greater than or equal to 2; determining video frames to be transmitted corresponding to the N links according to the transmission quality of the N links; when the transmission quality of the kth link meets a preset condition, the video frames to be transmitted corresponding to the kth link are all video frames of the first video data; under the condition that the transmission quality of the kth link does not meet a preset condition, the video frames to be transmitted corresponding to the kth link are partial video frames in all the first video data; the value of k is 1,2, \8230, any value of N, and the video frame to be transmitted comprises all key data of the first video data; and transmitting the video frames to be transmitted respectively corresponding to the N links in the N links.
In the transmission method of video data provided by the embodiment of the application, a video sending end detects transmission quality of N links for transmitting first video data, and determines video frames to be transmitted respectively corresponding to the N links according to the transmission quality of the N links, wherein under the condition that the transmission quality of a k-th link meets a preset condition, all video frames in the first video data are used as the video frames to be transmitted of the k-th link, and under the condition that the transmission quality of the k-th link does not meet the preset condition, part of the video frames in the first video data are used as the video frames to be transmitted of the k-th link; and the determined video frame to be transmitted comprises all key data in the first video data. So, when k link transmission quality was relatively poor, this part video frame in the first video data of transmission on k link included all key data in the first video data to can guarantee the normal transmission of key data in the first video data to a certain extent, and then solved video receiving terminal and appeared card pause, flower screen and high time delay scheduling problem when playing received video data.
In a possible implementation manner, the detecting the transmission quality of the kth link specifically includes: acquiring feedback information of a kth link, wherein the feedback information comprises transmission delay and/or retransmission times; and determining the transmission quality of the k link according to the feedback information of the k link.
When the feedback information is transmission delay, if the transmission delay of the kth link is less than the preset delay, the transmission quality of the kth link meets the preset condition; and if the transmission delay of the kth link is greater than or equal to the preset delay, the transmission quality of the kth link does not meet the preset condition.
When the feedback information is retransmission times, if the retransmission times of the kth link are less than preset times, the transmission quality of the kth link meets the preset condition; and if the retransmission times of the kth link are more than or equal to the preset times, the transmission quality of the kth link does not meet the preset condition.
When the feedback information is transmission delay and retransmission times, if the transmission delay of the kth link is less than a preset delay and the retransmission times of the kth link is less than the preset times, the transmission quality of the kth link meets a preset condition; and if the transmission delay of the kth link is greater than or equal to a preset threshold value and/or the retransmission times of the kth link are greater than or equal to preset times, the transmission quality of the kth link does not meet a preset condition.
In a possible implementation manner, determining a video frame to be transmitted corresponding to a kth link when the transmission quality of the kth link does not meet a preset condition specifically includes: and according to a preset proportion corresponding to the kth link and the sequence of the priority of the video frames of the first video data from high to low, selecting a part of the video frames of the first video data as the video frames to be transmitted corresponding to the kth link.
The preset proportion meets the following requirements:
wherein eta represents a predetermined ratio, num 1 Indicates the number of non-key video frames, num, in the video frames of the first video data 2 A total number of video frames representing the first video data; wherein the non-key video frames refer to video frames comprising non-key data.
In the method for transmitting video data provided by the embodiment of the application, a video sending end detects the transmission quality of a k link used for transmitting first video data, and determines a video frame to be transmitted corresponding to the k link according to the transmission quality of the k link, wherein under the condition that the transmission quality of the k link meets a preset condition, all video frames in the first video data are used as the video frames to be transmitted of the k link, and under the condition that the transmission quality of the k link does not meet the preset condition, part of the video frames in the first video data are used as the video frames to be transmitted of the k link; and the determined video frame to be transmitted comprises all key data in the first video data. Therefore, when the transmission quality of the kth link is poor, the part of video frames in the first video data transmitted on the kth link comprise all key data in the first video data, so that the normal transmission of the key data in the first video data can be ensured to a certain extent, and the problems of blockage, screen splash, high time delay and the like when a video receiving end plays the received video data are solved.
In a possible implementation manner, after the to-be-transmitted video frames respectively corresponding to the N links are transmitted in the N links, the method for transmitting video data provided in the embodiment of the present application further includes: detecting the transmission quality of the kth link; and adjusting the preset proportion corresponding to the kth link according to the transmission quality of the kth link.
In a possible implementation manner, the adjusting, according to the transmission quality of the kth link, the preset ratio corresponding to the kth link specifically includes: increasing the preset proportion under the condition that the transmission quality of the kth link meets a preset condition; and reducing the preset proportion under the condition that the transmission quality of the kth link does not meet the preset condition.
In the transmission method of video data provided by the embodiment of the application, a video sending end increases a preset proportion under the condition that the transmission quality of k links meets a preset condition; that is to say, the transmission quality of the kth link is better, the preset proportion is increased by the video sending end, so that the video sending end sends more video frames in the video data to the video receiving end, and the video receiving end can play high-quality video pictures according to more video frames in the video data.
In a possible implementation manner, a method for transmitting video data provided in an embodiment of the present application further includes: determining a video frame to be transmitted of the second video data corresponding to the Kth link according to the adjusted preset proportion corresponding to the Kth link; and transmitting the video frame to be transmitted corresponding to the k link in the k link.
In a possible implementation manner, a method for transmitting video data provided in an embodiment of the present application further includes: performing frequency domain layered coding on first video data to obtain a video frame of the first video data, wherein the video frame of the first video data comprises a basic frame and an extended frame of at least one key frame, and a basic frame and an extended frame of a plurality of forward frames; the non-key video frame comprises an extension frame of a key frame, a basic frame of a forward frame and an extension frame of the forward frame, and the key data of the first video data is the basic frame of the key frame.
In a possible implementation manner, a method for transmitting video data provided in an embodiment of the present application further includes: the priority of the video frames of the first video data is sequentially a basic frame of a key frame, a basic frame of a forward frame, an extended frame of the key frame and an extended frame of the forward frame from high to low.
In a possible implementation manner, a method for transmitting video data provided in an embodiment of the present application further includes: the ith video frame in the video frames of the first video data corresponding to the first link is a basic frame of the first video frame, and the ith video frame in the video frames of the first video data corresponding to the second link is an extended frame of the first video frame; the first link and the second link are two different links of the N links.
In summary, the arrangement order of the plurality of video frames of the first video data transmitted on different links is: in the video frame queues corresponding to the two different links, the video frames at the same position (or the same number or index) are the basic frame and the extended frame of the same video frame, the video sending end transmits the video frames in parallel on the two links according to the arrangement sequence of the video frames, the basic frame and the extended frame of the video frames can be quickly sent to the video receiving end, and for the video receiving end, the basic frame and the extended frame of the video frames can be quickly aggregated to obtain the decoded video frame.
In a possible implementation manner, a method for transmitting video data provided in an embodiment of the present application further includes: determining a retransmission video frame set, wherein the retransmission video frame set is an intersection of the video frames to be retransmitted corresponding to the N links; transmitting the video frames in the retransmission video frame set in the jth link in a queue-insertion transmission mode; wherein, the transmission quality of the j link is higher than that of other links, and the other links are links except the j link in the N links.
In a possible implementation manner, the transmitting a video frame in the retransmission video frame set in the jth link includes: and transmitting the video frames in the retransmission video frame set in the jth link according to the sequence of the priorities of the video frames in the retransmission video frame set from high to low.
In a possible implementation manner, the transmitting the video frames in the retransmitted video frame set in the jth link in the queue transmission manner includes: and inserting the retransmission video frame set into the head bit of the video frame queue to be transmitted corresponding to the j link.
In a possible implementation manner, before the video frames to be transmitted respectively corresponding to the N links are transmitted in the N links, the method further includes: and encrypting the video frames to be transmitted corresponding to the N links respectively.
In the embodiment of the application, on one hand, the video sending end transmits the video frames in the retransmission video frame set in an inserting queue mode, and preferentially transmits the video frames in the retransmission video frame set, so that the video receiving end can acquire lost video frames as soon as possible, the speed of the video receiving end for aggregating the video frames is improved, and the rapid aggregation of the video frames is realized; on the other hand, when the video sending end transmits the video frames in the retransmitted video frame set, the video sending end transmits the video frames in the retransmitted video frame set according to the sequence from top to bottom of the priority of the video frames in the retransmitted video set, and since the video frames with higher priority include more important data information, the retransmission of the video frames according to the priority can ensure that the important information in the first video data is not lost to some extent, so as to ensure the playing quality of the video data.
In a second aspect, an embodiment of the present application provides a video data transmission apparatus, including: the device comprises a detection module, a determination module and a transmission module; the detection module is used for detecting the transmission quality of the N links; the N links are used for transmitting first video data, the N links are links in different networks, the first video data comprise a plurality of video frames, and N is a positive integer greater than or equal to 2; the determining module is used for determining video frames to be transmitted corresponding to the N links according to the transmission quality of the N links; under the condition that the transmission quality of the kth link meets a preset condition, the video frames to be transmitted corresponding to the kth link are all video frames of the first video data; under the condition that the transmission quality of the kth link does not meet a preset condition, the video frames to be transmitted corresponding to the kth link are partial video frames in all the first video data; the value of k is 1,2, \8230, any value of \8230, and any value of N, and the video frame to be transmitted comprises all key data of the first video data; and the transmission module is used for transmitting the video frames to be transmitted corresponding to the N links in the N links.
In a possible implementation manner, the video data transmission apparatus provided in the embodiment of the present application further includes: an acquisition module; the obtaining module is configured to obtain feedback information of the kth link, where the feedback information includes transmission delay and/or retransmission times.
And the determining module is used for determining the transmission quality of the k link according to the feedback information of the k link.
When the feedback information is transmission delay, if the transmission delay of the kth link is less than the preset delay, the transmission quality of the kth link meets the preset condition; and if the transmission delay of the kth link is greater than or equal to the preset delay, the transmission quality of the kth link does not meet the preset condition.
When the feedback information is the retransmission times, if the retransmission times of the kth link is less than the preset times, the transmission quality of the kth link meets the preset condition; and if the retransmission times of the kth link are greater than or equal to the preset times, the transmission quality of the kth link does not meet the preset condition.
When the feedback information is transmission delay and retransmission times, if the transmission delay of the kth link is less than the preset delay and the retransmission times of the kth link is less than the preset times, the transmission quality of the kth link meets the preset condition; and if the transmission delay of the kth link is greater than or equal to a preset threshold value and/or the retransmission times of the kth link are greater than or equal to preset times, the transmission quality of the kth link does not meet a preset condition.
In a possible implementation manner, a video data transmission apparatus provided in an embodiment of the present application includes: a control module; and the control module is used for selecting a part of video frames in the video frames of the first video data as the video frames to be transmitted corresponding to the kth link according to the preset proportion corresponding to the kth link and the sequence of the priority of the video frames of the first video data from high to low.
The preset proportion meets the following requirements:
wherein eta represents a predetermined ratio, num 1 Indicates the number of non-key video frames, num, in the video frames of the first video data 2 Represents the firstA total number of video frames of a video data; wherein a non-key video frame refers to a video frame comprising non-key data.
In a possible implementation manner, a video data transmission apparatus provided in an embodiment of the present application includes: an adjustment module; the detection module is used for detecting the transmission quality of the kth link; and the adjusting module is used for adjusting the preset proportion corresponding to the kth link according to the transmission quality of the kth link.
In a possible implementation manner, in the video data transmission apparatus provided in the embodiment of the present application, the adjusting module is specifically configured to increase the preset ratio when the transmission quality of the kth link meets a preset condition; and reducing the preset proportion under the condition that the transmission quality of the kth link does not meet the preset condition.
In a possible implementation manner, the determining module is further configured to determine, according to the adjusted preset ratio corresponding to the K-th link, a to-be-transmitted video frame of the second video data corresponding to the K-th link; the transmission module is further configured to transmit the video frame to be transmitted corresponding to the kth link in the kth link.
In a possible implementation manner, a video data transmission apparatus provided in an embodiment of the present application includes: an encoding module; the encoding module is used for carrying out frequency domain layered encoding on the first video data to obtain a video frame of the first video data, wherein the video frame of the first video data comprises a basic frame and an extended frame of at least one key frame and a plurality of basic frames and extended frames of a forward frame; the non-key video frame includes an extended frame of a key frame, a base frame of a forward frame, and an extended frame of a forward frame, and the key data of the first video data is the base frame of the key frame.
In a possible implementation manner, in the video data transmission apparatus provided in the embodiment of the present application, the priority of the video frames of the first video data is, in order from low to high, an extended frame of a forward frame, an extended frame of a key frame, a base frame of the forward frame, and a base frame of the key frame.
In a possible implementation manner, in the video data transmission apparatus provided in the embodiment of the present application, an ith video frame in video frames of first video data corresponding to a first link is a base frame of the first video frame, and an ith video frame in video frames of the first video data corresponding to a second link is an extension frame of the first video frame; the first link and the second link are two different links of the N links.
In a possible implementation manner, the video data transmission apparatus provided in the embodiment of the present application further includes: a queue-insertion module; the determining module is further configured to determine a retransmission video frame set, where the retransmission video frame set is an intersection of the video frames to be retransmitted corresponding to the N links; a queue insertion module, configured to transmit a video frame in the retransmitted video frame set in a j-th link in a queue insertion transmission manner; wherein, the transmission quality of the j link is higher than that of other links, and the other links are links except the j link in the N links.
In a possible implementation manner, in the video data transmission apparatus provided in this embodiment of the present application, the transmission module is further configured to transmit the video frames in the retransmitted video frame set in the jth link according to an order from high priority to low priority of the video frames in the retransmitted video frame set.
In a possible implementation manner, the queue insertion module is specifically configured to insert the retransmission video frame set into a head of a to-be-transmitted video frame queue corresponding to the j-th link.
In a possible implementation manner, the video data transmission apparatus provided in the embodiment of the present application further includes an encryption module; and the encryption module is used for encrypting the video frames to be transmitted respectively corresponding to the N links.
In a third aspect, an embodiment of the present application provides a video data transmission apparatus, including a memory and a processor, where the memory is coupled to the processor; the memory is for storing computer program code, the computer program code comprising computer instructions; the computer instructions, when executed by the processor, cause the video data transmission apparatus to perform the method of any one of the first aspect and its possible implementations.
In a fourth aspect, an embodiment of the present application provides a computer storage medium, which includes computer instructions, when the computer instructions are executed on a video data transmission apparatus, cause the video data transmission apparatus to execute the method described in any one of the first aspect and possible implementation manners thereof.
In a fifth aspect, embodiments of the present application provide a computer program product, which, when run on a computer, causes the computer to execute the method described in any one of the first aspect and its possible implementation manners.
It should be understood that, for the technical effects achieved by the technical solutions of the second aspect to the fifth aspect and the corresponding possible implementations of the embodiments of the present application, reference may be made to the technical effects of the first aspect and the corresponding possible implementations.
Drawings
FIG. 1 is a schematic view of a screen projection method provided in an embodiment of the present application;
fig. 2 is a first schematic view of a scene using distributed cameras according to an embodiment of the present disclosure;
fig. 3 is a schematic view of a scene using a distributed camera according to an embodiment of the present disclosure;
fig. 4 is a first schematic view illustrating a transmission process of video data according to an embodiment of the present disclosure;
fig. 5 is a first schematic view illustrating an encoding method of video data according to an embodiment of the present disclosure;
fig. 6 is a first schematic diagram illustrating a transmission method of video data according to an embodiment of the present disclosure;
fig. 7 is a first schematic diagram illustrating a multi-path transmission architecture according to an embodiment of the present disclosure;
fig. 8 is a schematic structural diagram of a video data transmission apparatus according to an embodiment of the present application;
fig. 9 is a first schematic diagram illustrating a video data transmission method according to an embodiment of the present application;
fig. 10 is a schematic diagram illustrating a video data transmission method according to an embodiment of the present application;
fig. 11 is a first schematic diagram illustrating an ordering of a plurality of video frames according to an embodiment of the present application;
fig. 12 is a third schematic diagram illustrating a transmission method of video data according to an embodiment of the present application;
fig. 13 is a fourth schematic view illustrating a transmission method of video data according to an embodiment of the present application;
fig. 14 is a schematic diagram illustrating a video data transmission method according to an embodiment of the present application;
fig. 15 is a first schematic structural diagram of a video data transmission apparatus according to an embodiment of the present application;
fig. 16 is a schematic structural diagram of a video data transmission apparatus according to an embodiment of the present application.
Detailed Description
The term "and/or" herein is merely an association relationship describing an associated object, and means that there may be three relationships, for example, a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone.
The terms "first" and "second," and the like in the description and in the claims of the embodiments of the present application, are used for distinguishing between different objects and not for describing a particular order of the objects. For example, the first link and the second link, etc. are for distinguishing the links in different networks, and are not for describing a specific order of the links.
In the embodiments of the present application, the words "exemplary" or "such as" are used herein to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "such as" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the word "exemplary" or "such as" is intended to present concepts related in a concrete fashion.
In the description of the embodiments of the present application, the meaning of "a plurality" means two or more unless otherwise specified. For example, the plurality of video frames refers to two or more video frames.
With the development of technology, streaming media (such as video) is applied more and more widely in real life, and in daily life, video data transmission is involved in both a near-field screen projection scene and a distributed camera scene.
In the near-field screen projection scenario, the screen projection manner may include: mirror coordination, heterogeneous coordination, reverse control, and the like
And (3) image coordination: the method is characterized in that video pictures on a screen projection side are synchronously transmitted to a screen projection side in real time, so that the video pictures are synchronously played on the screen projection side.
For example, referring to (a) in fig. 1, the large screen 101 is a screen-projected side, the mobile phone 102 is a screen-projected side, and the screen projection manner of the mobile phone 102 and the large screen 101 is mirror coordination, for example, when a video call is performed by using the mobile phone 102, the mobile phone 102 transmits a video picture (specifically, video data corresponding to the video picture) during the video call to the large screen 101, and the large screen 101 can synchronously play the video picture of the mobile phone 102.
Heterology synergy: the method includes that a certain task running in a background of a screen projection side is synchronously transmitted to the screen projection side, so that the task is synchronously played on the screen projection side, and at the moment, the screen projection side can execute other tasks.
Exemplarily, referring to fig. 1 (b), the large screen 103 is a screen-projected side, the mobile phone 104 is a screen-projected side, a screen projection relationship between the mobile phone 104 and the large screen 103 is heterogeneous cooperation, a chat interface (a chat interface displayed by the mobile phone running a social application) and a video playing interface (an interface displayed by the mobile phone running a video playing application) are displayed on the mobile phone 104 in a screen-split manner, the mobile phone 104 can transmit a video picture displayed on the split screen 2 to the large screen 103, the large screen 103 synchronously plays the video picture displayed on the mobile phone 104, and after the screen projection is successful, a user can perform corresponding operation on the mobile phone 104 to set the video playing application to run in a background, so that the chat interface is displayed on the mobile phone 104 in a full screen manner.
Reverse control: the method comprises the steps of transmitting a picture displayed on a screen projection side to a screen to be projected, displaying the picture on the screen to be projected in real time, and then carrying out related operation on the screen to be projected by a user, so that the screen to be projected can respond to the operation and execute corresponding tasks.
Referring to fig. 1 (c), the computer 105 is the screen-projected side, the mobile phone 106 is the screen-projected side, the screen projection mode between the mobile phone 106 and the computer 105 is reverse control, the mobile phone 106 transmits the video frame displayed on the display screen of the mobile phone 106 to the computer 105 synchronously, the video frame is displayed on the display screen of the computer 105, then, the user can operate the displayed content on the computer 105, and the mobile phone 106 can respond to the operation of the user on the computer 105. For example, a user opens a certain video software in the mobile phone 106, the mobile phone 106 displays an interface including a plurality of video resources, and after the mobile phone 106 and the computer 105 establish a reverse-control screen-casting connection, the interface including the plurality of video resources is also synchronously displayed on the computer 105, and further, when the user opens a certain video resource in the plurality of video resources on the computer 105, the mobile phone 106 can respond to the opening operation of the video resource, the mobile phone 106 displays a video picture corresponding to the video resource, and simultaneously, the video picture displayed on the mobile phone 106 is also synchronously transmitted to the computer 105, so that the computer 105 displays the video picture.
In summary, in the screen projection scene, the screen projection side is a sending end of the video data, and the screen projection side is a receiving end of the video data.
In a distributed camera scene, video call, remote control mobile phone photographing, mobile phone live broadcasting, cross-device double-scene video recording and the like can be realized by using the distributed camera.
Fig. 2 (a) is a schematic view of a scene in which a video call is performed using a distributed camera, where a camera on a large screen 201 and a mobile phone 202 form a distributed camera, before the video call is performed using the mobile phone 202, the mobile phone 202 establishes a communication connection with the camera on the large screen 201, and since the pixels of the camera on the large screen 201 are higher than those of the camera of the mobile phone 202, when the video call is performed using the mobile phone 202, the camera on the large screen 201 is used as a camera for capturing a video picture during the video call, specifically, the video picture is captured by the camera on the large screen 201 and is synchronously transmitted to the mobile phone 202, and the mobile phone 202 displays the video picture in real time.
In the above video call scenario, the large screen 201 is a sending end of video data, and the mobile phone 202 is a receiving end of video data.
Fig. 2 (b) is a schematic diagram of a scene of remotely controlling a mobile phone to take a picture by using a distributed camera, wherein the smart watch 203 and the mobile phone 204 form a distributed camera; when the mobile phone 204 is placed at a far end for taking a picture, the mobile phone 204 establishes communication connection with the smart watch 203, the camera of the mobile phone 204 transmits the collected preview picture to the smart watch 203, the preview picture is synchronously displayed on the smart watch 203, and then the user can operate the smart watch 203 to control the mobile phone 204 to take a picture. For example, the user feels that the effect of the picture in the preview interface currently displayed by the smart watch 203 is better, and the user may select a "photo" button in the smart watch 203, so that the mobile phone 204 collects an image in response to the "photo" operation.
In the above scenario of remotely controlling the mobile phone to take a picture, the mobile phone 204 is a sending end of video data, and the smart watch 203 is a receiving end of video data.
Fig. 3 (a) is a schematic view of a scene in which a distributed camera is used for live broadcast of a mobile phone, where a rear camera of the mobile phone 301 and the mobile phone 302 form a distributed camera, and a pixel of the rear camera of the mobile phone 301 is higher than a front camera of the mobile phone 302, in a live broadcast scene, after the mobile phone 302 establishes a communication connection with the mobile phone 301, the mobile phone 302 runs a live broadcast APP, the rear camera of the mobile phone 301 collects a live broadcast picture (such as a landscape or a main broadcast), then the mobile phone 301 transmits the collected live broadcast picture to the mobile phone 302 in real time, and the mobile phone 302 plays the live broadcast picture synchronously.
In the above scenario of live broadcast of the mobile phone, the mobile phone 301 is a sending end of video data, and the mobile phone 302 is a receiving end of video data.
Fig. 3 (b) is a schematic view of a scene of performing cross-device double-view video recording by using a distributed camera, where a mobile phone 303, a mobile phone 305, and a camera 304 form a distributed camera, the mobile phone 303 and the camera 304 respectively establish communication connection with the mobile phone 305, after the double-view video recording is started, the mobile phone 303 transmits a close-view picture of a target scene acquired by a camera of the mobile phone 303 to the mobile phone 305, meanwhile, the camera 304 transmits a far-view picture of the acquired target scene to the mobile phone 305, and after the mobile phone 305 receives the pictures transmitted by the mobile phone 303 and the camera 304, the close-view picture and the far-view picture are displayed on a display screen of the mobile phone 305 in a split-screen manner, so as to implement the double-view video recording.
In the above-described scenario of cross-device double-shot recording, the cell phone 303 and the camera 304 are both a sending end of video data, and the cell phone 305 is a receiving end of video data.
It is understood that in the embodiment of the present application, the video data is a video stream, and includes a plurality of video frames, each of which is an image, and thus the video frames may also be referred to as image frames.
With reference to the schematic transmission flow diagram of video data shown in fig. 4 in combination with the multiple scenes related to video data transmission, first, a sending end encodes (pre-processes) the acquired video data (including multiple video frames), and commonly used encoding modes include time domain encoding and frequency domain encoding; secondly, the sending end encrypts the coded video data and sends the encrypted video data to the receiving end. When the receiving end receives the encrypted video data, firstly, the encrypted video data is reversely decrypted according to the encryption mode of the sending end to obtain decrypted video data; and finally, the receiving end renders the decoded video data and plays the video data.
Common coding schemes for video data may include time-domain coding and frequency-domain coding.
In the embodiment of the present application, the forward search frame is simply referred to as a forward frame, and the following embodiment will not be described.
Time domain coding: a small forward frame (i.e., a small P frame) and a large forward frame (a large P frame) of a video frame can be obtained by encoding an acquired video frame in a time-domain encoding manner, as shown in fig. 5 (a), where when a receiving end decodes a large P frame, it needs to rely on a decoding result of a previous large P frame of the large P frame and the small P frame corresponding to the large P frame.
And (3) frequency domain coding: encoding the acquired video frame into a base frame and an extended frame according to the frequency domain distribution of the video data, as shown in (b) of fig. 5; the basic frame of a video frame contains low-frequency data of the video frame, the extended frame of a video frame contains high-frequency data of the video frame, the size (data size) of the basic frame is different from that of the extended frame, specifically, the size ratio of the basic frame to the extended frame is between 1 and 4.
At present, video data can be transmitted by multiple links (i.e., multipath transmission), specifically, as shown in fig. 6, a video sending end transmits all data packets (i.e., data packets 1 to 8) of video data to be transmitted on a first link, and also transmits all data packets of video data to be transmitted on a second link, where the data packets transmitted on the first link and the second link are backups of each other, and when a data packet of one link is lost, a video receiving end can obtain the lost data packet from a data packet transmitted on the other link, so as to successfully complete decoding of the video data.
However, when the bandwidth of the second link of the two links is limited, a part of video data packets may be lost when the second link transmits the data packets of the video data to be transmitted, if the first link transmits the data packets of the video data to be transmitted, a part of video data packets is also lost due to other factors (for example, the receiving end is restarted), and the part of video data packets lost by the second link and the part of video data packets lost by the first link have an intersection, and the video data packets in the intersection may have important information in the video data to be transmitted, so that the receiving end cannot normally decode the video data to be transmitted, and the video receiving end may have problems of blocking, screen floating, and high delay when playing the received video data.
The two multipath transmission methods are implemented by a data flow module in a multipath transmission architecture, as shown in fig. 7, a transmission flow of a data packet specifically includes: after the acquisition module of the sending end acquires the data packet sequence, the data packet sequence is in communication connection with the receiving end through the communication module, and then the data flow module transmits the data packet sequence acquired by the acquisition module to the receiving end in a character flow or byte flow mode, wherein the method for transmitting the data packet sequence from the sending end to the receiving end can be any one of the multi-path transmission methods.
In the multipath transmission method in the embodiment of the present application, the same video data is transmitted on different links by using a parallel transmission mode on multiple links, and details are not described in the following.
Based on the problem that the smooth transmission of important information in video data cannot be guaranteed in the video data transmission process in the prior art, the embodiment of the application provides a video data transmission method and device, wherein a video transmission device or a video data sending end detects the transmission quality of N links; the N links are used for transmitting video data, and the N links are links in different networks; the video data comprises video frames of first video data, and N is a positive integer greater than or equal to 2; the video transmission device determines video frames to be transmitted corresponding to the N links according to the transmission quality of the N links; under the condition that the transmission quality of the kth link meets a preset condition, the video frame to be transmitted corresponding to the kth link is the video frame of the first video data; under the condition that the transmission quality of the kth link does not meet a preset condition, the video frame to be transmitted corresponding to the kth link is a partial video frame in the video frames of the first video data; the value of k is any one value of 1,2, \8230 \8230andN, and the determined video frame to be transmitted comprises all key data in the video frame of the first video data; and the video transmission device transmits the video frames to be transmitted respectively corresponding to the N links in the N links. By the technical scheme provided by the embodiment of the application, the smooth transmission of the key data (namely, important information) in the video data can be ensured to a certain extent.
For example, the video transmission apparatus in the embodiment of the present application may be a tablet computer, a mobile phone, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a Personal Digital Assistant (PDA), and the like, and the embodiment of the present application does not particularly limit the specific form of the video transmission apparatus.
An execution main body of the video data transmission method provided in the embodiment of the present application may be a device supporting video data transmission (hereinafter, the embodiment is simply referred to as a video transmission device), please refer to fig. 8, and the embodiment of the present application takes the video transmission device as a mobile phone 800 as an example to introduce the video transmission device provided in the embodiment of the present application. Of these, the cell phone 800 shown in fig. 8 is merely one example of a video transmission device, and the cell phone 800 may have more or fewer components than shown in the figure, may combine two or more components, or may have a different configuration of components. The various components shown in fig. 8 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.
As shown in fig. 8, the mobile phone 800 may include a processor 810, an external memory interface 820, an internal memory 821, a Universal Serial Bus (USB) interface 830, a charging management module 840, a power management module 841, a battery 842, an antenna 1, an antenna 2, a radio frequency module 850, a communication module 860, an audio module 870, a speaker 870A, a receiver 870B, a microphone 870C, a headset interface 870D, a sensor module 880, a button 890, a motor 891, an indicator 892, a camera 893, a display 894, and a Subscriber Identity Module (SIM) card interface 895, among others.
The sensor module 880 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and other sensors.
The controller may be the neural center and the command center of the cell phone 800. The controller can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of instruction fetching and instruction execution.
A memory may also be provided in processor 810 for storing instructions and data. In some embodiments, the memory in the processor 810 is a cache memory that may hold instructions or data that have just been used or recycled by the processor 810. If the processor 810 needs to use the instructions or data again, it can call directly from the memory. Avoiding repeated accesses reduces the latency of the processor 810, thereby increasing the efficiency of the system.
In some embodiments, processor 810 may include an interface. The interface may include an Integrated Circuit (I2C) interface, an Integrated Circuit built-in audio (I2S) interface, a Pulse Code Modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a general-purpose input/output (GPIO) interface, a SIM interface, and/or a USB interface, etc.
The I2C interface is a bidirectional synchronous serial bus including a Serial Data Line (SDL) and a Serial Clock Line (SCL). In some embodiments, processor 810 may include multiple sets of I2C buses. The processor 810 may be coupled to the touch sensor 280K, the charger, the flash, the camera 893, etc. through different I2C bus interfaces, respectively. For example: the processor 810 may be coupled to the touch sensor 280K through an I2C interface, so that the processor 810 and the touch sensor 280K communicate through an I2C bus interface, thereby implementing a touch function of the mobile phone 800.
The I2S interface may be used for audio communication. In some embodiments, processor 810 may include multiple sets of I2S buses. Processor 810 may be coupled to audio module 870 via an I2S bus to enable communication between processor 810 and audio module 870. In some embodiments, audio module 870 may communicate audio signals to communications module 860 via an I2S interface to enable answering a call via a bluetooth headset.
The PCM interface may also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, audio module 870 and communication module 860 may be coupled by a PCM bus interface. In some embodiments, the audio module 870 may also communicate audio signals to the communication module 860 via the PCM interface to enable answering a call via a bluetooth headset. Both the I2S interface and the PCM interface may be used for audio communication, with different sampling rates for the two interfaces.
MIPI interfaces may be used to connect processor 810 with peripheral devices such as display screen 894, camera 893, and the like. The MIPI interface includes a Camera Serial Interface (CSI), a Display Serial Interface (DSI), and the like. In some embodiments, the processor 810 and the camera 893 communicate through a CSI interface to implement the camera functions of the cell phone 800. The processor 810 and the display screen 894 communicate through a DSI interface to realize the display function of the mobile phone 800.
The interface connection relationship between the modules in the embodiment of the present application is only schematically illustrated, and does not constitute a limitation on the structure of the mobile phone 800. The mobile phone 800 may adopt different interface connection modes or a combination of multiple interface connection modes in this embodiment.
The charging management module 840 is configured to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 840 may receive charging input from a wired charger via the USB interface 830. In some wireless charging embodiments, the charging management module 840 may receive a wireless charging input through a wireless charging coil of the cell phone 800. The charging management module 840 may also supply power to the mobile phone 800 through the power management module 841 while charging the battery 842.
The power management module 841 is used to connect the battery 842, the charging management module 840 and the processor 810. The power management module 841 receives inputs from the battery 842 and/or the charge management module 840 and provides power to the processor 810, the internal memory 821, the external memory interface 820, the display 894, the camera 893, and the communication module 860, among other things. The power management module 841 may also be used to monitor parameters such as battery capacity, battery cycle count, battery state of health (leakage, impedance), etc. In some embodiments, the power management module 841 may also be disposed in the processor 810. In some embodiments, the power management module 841 and the charging management module 840 may also be disposed in the same device.
The wireless communication function of the mobile phone 800 can be implemented by the antenna 1, the antenna 2, the rf module 850, the communication module 860, a modem, a baseband processor, and the like.
The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the handset 800 may be used to cover a single or multiple communication bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: the cellular network antenna may be multiplexed into a wireless local area network diversity antenna. In some embodiments, the antenna may be used in conjunction with a tuning switch.
The rf module 850 may provide a communication processing module including a solution for wireless communication such as 2G/3G/4G/5G, which is applied to the mobile phone 800. The rf module 850 may include at least one filter, a switch, a power amplifier, a Low Noise Amplifier (LNA), and the like. The rf module 850 receives the electromagnetic wave from the antenna 1, and performs filtering, amplification, and other processing on the received electromagnetic wave, and transmits the electromagnetic wave to the modem for demodulation.
The communication module 860 may provide solutions for wireless communication applied to the mobile phone 800, including Wireless Local Area Networks (WLANs) (e.g., wireless fidelity (Wi-Fi) networks), bluetooth (BT), global Navigation Satellite System (GNSS), frequency Modulation (FM), near Field Communication (NFC), infrared (IR), and the like. For example, in the embodiment of the present application, the mobile phone 800 may access a Wi-Fi network through the wireless communication module 860.
The wireless communication module 860 may be one or more devices that integrate at least one communication processing module. The wireless communication module 860 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on electromagnetic wave signals, and transmits the processed signals to the processor 810. The wireless communication module 860 may also receive a signal to be transmitted from the processor 810, frequency modulate it, amplify it, and convert it into electromagnetic waves via the antenna 2 to radiate it.
The cell phone 800 implements a display function via the GPU, the display screen 894, and the application processor, etc. The GPU is a microprocessor for image processing, coupled to a display screen 894 and an application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 810 may include one or more GPUs that execute program instructions to generate or alter display information.
The display screen 894 is used to display images, video, and the like. The display screen 894 includes a display panel. For example, in this embodiment of the application, the display screen 894 may be used to display an application interface of the first application, such as a device sharing interface, a device searching interface, a two-dimensional code scanning interface, and the like.
The mobile phone 800 may implement a shooting function through the ISP, the camera 893, the video codec, the GPU, the display screen, and the application processor.
The camera 893 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image to the photosensitive element. The photosensitive element may be a Charge Coupled Device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. And the ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into image signal in standard RGB, YUV and other formats. In some embodiments, the cell phone 800 may include 1 or N cameras 893, N being a positive integer greater than 1.
The digital signal processor is used for processing digital signals, and can process other digital signals besides digital image signals. For example, when the cell phone 800 is in frequency bin selection, the digital signal processor is used to perform fourier transform or the like on the frequency bin energy.
Video codecs are used to compress or decompress digital video. The handset 800 may support one or more video codecs. Thus, the mobile phone 800 can play or record videos with various encoding formats, and in the present application, the mobile phone 800 encodes the video data by using a frequency domain encoder.
The external memory interface 820 may be used to connect an external memory card, such as a Micro SD card, to extend the storage capability of the mobile phone 800. The external memory card communicates with the processor 810 through the external memory interface 820 to implement data storage functions. For example, files such as music, video, etc. are saved in an external memory card.
The internal memory 821 may be used to store computer-executable program code, which includes instructions. The processor 810 executes various functional applications and data processing of the cellular phone 800 by executing instructions stored in the internal memory 821. For example, in the embodiment of the present application, the processor 810 may execute instructions stored in the internal memory 821, and the internal memory 821 may include a program storage area and a data storage area.
The mobile phone 800 may implement audio functions through the audio module 870, the speaker 870A, the receiver 870B, the microphone 870C, the headphone interface 870D, and the application processor. Such as music playing, recording, etc.
The audio module 870 is used to convert digital audio information into an analog audio signal output and also used to convert an analog audio input into a digital audio signal. The audio module 870 may also be used to encode and decode audio signals. In some embodiments, audio module 870 may be disposed in processor 810, or some functional modules of audio module 870 may be disposed in processor 810.
The speaker 870A, also called a "horn", is used to convert the audio electrical signal into a sound signal.
The receiver 870B, also called "earpiece", is used to convert the electrical audio signal into a sound signal.
The microphone 870C, also referred to as a "microphone," is used to convert sound signals into electrical signals.
The headphone interface 870D is used to connect a wired headphone. The earphone interface 870D may be the USB interface 830, or may be an open mobile phone platform (OMTP) standard interface of 3.5mm, or a CTIA (cellular telecommunications industry association) standard interface.
The keys 890 include a power-on key, a volume key, and the like. The keys 890 may be mechanical keys. Or may be touch keys. The cell phone 800 receives the key 890 input, generating key signal inputs related to user settings and function control of the cell phone 800.
The motor 891 may generate a vibration cue. The motor 891 may be used for incoming call vibration prompts, as well as for touch vibration feedback.
Indicator 292 may be an indicator light that may be used to indicate a state of charge, a change in charge, or may be used to indicate a message, missed call, notification, etc.
The SIM card interface 895 is used to connect a SIM. The SIM card can be attached to and detached from the cellular phone 800 by being inserted into the SIM card interface 895 or being pulled out from the SIM card interface 895. The cell phone 800 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 895 may support a Nano SIM card, a Micro SIM card, a SIM card, etc.
The methods in the following embodiments can be implemented in a video transmission apparatus having the above hardware structure, where the video transmission apparatus is a video sending end, and the methods in the following embodiments are described by taking the example where the video sending end is a mobile phone 800 shown in fig. 8.
An embodiment of the present application provides a method for transmitting video data, which is applied to a video transmitting end, and as shown in fig. 9, the method for transmitting video data may include S910 to S930.
S910, the video sending end detects the transmission quality of the N links for transmitting the first video data.
In this embodiment of the application, when a video sending end sends video data to a video receiving end, the video sending end may send the video data to the video receiving end through N links, respectively, where N is a positive integer greater than or equal to 2.
Based on the description of the foregoing embodiment, the first video data sent by the video sending end to the video receiving end includes a plurality of video frames, and the first video data is derived from one video segment in the video to be transmitted, for example, one video segment may be a group of pictures (GOP) or a plurality of groups of pictures (GOP), and one video segment may also be a video segment divided according to other methods, which is not specifically limited in this embodiment of the application.
Illustratively, a GOP is composed of 1 key frame (e.g., I frame) and 29 forward search frames (P frames).
It should be noted that the N links are links in different networks, that is, the N links belong to different networks. For example, the value of N is 2, and the 2 links for transmitting video data are link 1 and link 2, respectively, where link 1 may be a link in WIFI network 1, and link 2 may be a link in WIFI network 2.
In this embodiment of the present application, the influencing factors of the transmission quality of the link may include transmission delay, retransmission times, and the like of the link, and the video sending end may determine the transmission quality of the link according to at least one of the influencing factors, which will be described in detail in the following embodiments.
And S920, the video sending end determines video frames to be transmitted corresponding to the N links according to the transmission quality of the N links.
In the embodiment of the present application, one of N links is described, for example, the kth link is taken as an example to describe S920, where k is 1,2, \ 8230 \ 8230, and any one of N. It should be understood that a video frame to be transmitted for a link refers to a video frame to be transmitted over the link.
Specifically, the video sending end judges whether the transmission quality of a kth link meets a preset condition, and determines that video frames to be transmitted corresponding to the kth link are all video frames of the first video data under the condition that the transmission quality of the kth link meets the preset condition; and under the condition that the transmission quality of the kth link does not meet a preset condition (namely, the transmission quality of the kth link is poor), determining that a video frame to be transmitted corresponding to the kth link is a partial video frame of the first video data, wherein the video frame to be transmitted includes all key data in the first video data (the key data can be understood as important data in the first video data or data indispensable for decoding the first video data).
In the embodiment of the application, after the video frames to be transmitted corresponding to the N links are determined, the video sending end encrypts the video frames to be transmitted corresponding to the N links, wherein the encryption mode can be Hash algorithm encryption or symmetric encryption algorithm, and the encryption algorithm used specifically is not limited in the application.
Exemplarily, assuming that first video data is respectively transmitted on 2 links (link 1 and link 2), where the first video data includes 10 video frames, a video sending end first determines whether the transmission quality of the link 1 meets a preset condition, and if the transmission quality of the link 1 meets the preset condition, the 10 video frames are to-be-transmitted video frames of the link 1; if the link 1 does not satisfy the preset condition, determining a part of the video frames in the 10 video frames as video frames to be transmitted on the link 1, for example: and 6 video frames in the 10 video frames are determined as the video frames to be transmitted of the link 1. Similarly, the video transmission device also determines whether the transmission quality of the link 2 meets a preset condition, and if the transmission quality of the link 2 meets the preset condition, determines the 10 video frames as the video frames to be transmitted of the link 2; if the link 2 does not satisfy the preset condition, determining a part of the video frames in the 10 video frames as the video frames to be transmitted of the link 2, for example: and determining 8 video frames in the 10 video frames as the video frames to be transmitted of the link 2.
In the embodiment of the application, under the condition that the transmission quality of the link does not meet the preset condition, the video sending end transmits part of video frames in the first video data on the link, so that the transmission pressure of the link can be reduced, and the transmission efficiency of the link is improved.
And S930, the video sending end transmits the video frames to be transmitted corresponding to the N links on the N links respectively.
In the embodiment of the application, the video sending end transmits the video frames to be transmitted corresponding to the N links in parallel on the N different links.
With reference to the example in S920, the video frames to be transmitted of link 1 are 6 video frames of the first video data, and the video frames to be transmitted of link 2 are 8 video frames of the first video data, so that the video sending end transmits the 6 video frames on link 1 and transmits the 8 video frames on link 2 in parallel.
In the transmission method of video data provided by the embodiment of the application, a video sending end detects transmission quality of N links for transmitting first video data, and determines video frames to be transmitted respectively corresponding to the N links according to the transmission quality of the N links, wherein under the condition that the transmission quality of a k-th link meets a preset condition, all video frames in the first video data are used as the video frames to be transmitted of the k-th link, and under the condition that the transmission quality of the k-th link does not meet the preset condition, part of the video frames in the first video data are used as the video frames to be transmitted of the k-th link; and the determined video frame to be transmitted comprises all key data in the first video data. Therefore, when the transmission quality of the kth link is poor, the part of video frames in the first video data transmitted on the kth link comprises all the key data in the first video data, so that smooth transmission of the key data in the first video data can be ensured to a certain extent, and the problems of blockage, screen splash, high time delay and the like when a video receiving end plays the received video data are solved.
Optionally, with reference to fig. 9, as shown in fig. 10, before S910, the method for transmitting video data according to the embodiment of the present application further includes S1010.
S1010, the video sending end carries out frequency domain layered coding on the first video data to obtain a plurality of video frames of the first video data.
Optionally, the multiple video frames of the first video data are video frames obtained by encoding the first video data in the captured video stream by the video sending end, and as can be known from the description of the foregoing embodiment, the first video data (not encoded) includes at least one key frame and multiple forward frames.
In an implementation manner, the first video data may be encoded by using a frequency-domain layered coding manner to obtain a plurality of video frames of the first video data, and specifically, for any key frame in the first video data, the base frame of the key frame and the extension frame of the key frame are obtained after the frequency-domain layered coding, and similarly, for any forward frame in the first video data, the base frame of the forward frame and the extension frame of the forward frame are obtained after the frequency-domain layered coding, so that the plurality of video frames obtained after the first video data is encoded include the base frame and the extension frame of the at least one key frame, and the base frame and the extension frame of the plurality of forward frames.
For a detailed description of the method of frequency domain layered coding, reference may be made to the related description of the above embodiments, and details are not repeated here.
In this embodiment of the present application, the video frames of the first video data transmitted on each of the N links may be arranged in a certain order, and the arrangement orders of the video frames corresponding to the N links may be the same or different.
Optionally, in an implementation manner, taking a first link and a second link of the N links as an example, an arrangement order of a plurality of video frames corresponding to the first link is different from an arrangement order of a plurality of video frames corresponding to the second link, specifically, of the plurality of video frames of the first video data, an ith video frame corresponding to the first link is a base frame of the first video frame, and an ith video frame corresponding to the second link is an extension frame of the first video frame; wherein, the first link and the second link are two different links in the N links. Optionally, the first video frame may be a key frame or a forward frame.
Illustratively, fig. 11 is a schematic diagram of an ordering of a plurality of video frames on a first link and a plurality of video frames on a second link, in fig. 11, a 1 st video frame on the first link is a base frame of a key frame (may be referred to as a first key frame), a 1 st video frame on the second link is an extension frame of the first key frame, for example, a 4 th video frame on the first link is an extension frame of a forward frame (may be referred to as a first forward frame), and a 4 th video frame on the second link is a base frame of the first forward frame.
In summary, the sequence of the plurality of video frames of the first video data transmitted on different links is: in the video frame queues corresponding to the two different links, the video frames at the same position (or the same number or index) are the basic frame and the extended frame of the same video frame, the video sending end transmits the video frames in parallel on the two links according to the arrangement sequence of the video frames, the basic frame and the extended frame of the video frames can be quickly sent to the video receiving end, and for the video receiving end, the basic frame and the extended frame of the video frames can be quickly aggregated to obtain the decoded video frames.
Based on the description of the above embodiment, in the process that the video sending end transmits the video frames to be transmitted in parallel on the multiple links, one or more video frames in the video frames to be transmitted may be lost on the multiple links at the same time due to some factors (for example, the receiving end restarts or the receiving end does not consume the message sent by the sending end in time, so that the message is overtime), and then the video receiving end cannot acquire the lost video frames from the multiple links.
Optionally, with reference to fig. 10, as shown in fig. 12, in a case that one or more video frames are lost simultaneously on multiple links, the method for transmitting video data according to the embodiment of the present application further includes S1210-S1220.
S1210, the video sending end determines a retransmission video frame set.
The retransmission video frame set is the intersection of the video frames to be retransmitted corresponding to the N links.
For example, assuming that the value of N is 2, the set of video frames to be retransmitted in link 1 (i.e., video frames lost on link 1) includes a base frame of a P1 frame, an extended frame of a P3 frame, and an extended frame of a P4 frame; the set of video frames to be retransmitted in link 2 (i.e., video frames lost on link 2) includes a base frame of P1 frames, an extension frame of P2 frames, and an extension frame of P4 frames. Taking the intersection of the video frame set to be retransmitted of the link 1 and the video frame set to be retransmitted of the link 2, and taking the intersection as a retransmitted video frame set, wherein the retransmitted video frame set comprises: a base frame of a P1 frame and an extension frame of a P4 frame.
And S1220, the video sending end transmits the video frames in the retransmission video frame set in the jth link in a queue-insertion transmission mode.
The transmission quality of the j link is higher than that of other links except the j link in the N links, and the j link is the link with the highest transmission quality in the N links.
The uploading queue-inserting transmission mode is to insert the video frames in the retransmitted video frame set into the video frame queue which is being transmitted by the current link so as to preferentially transmit the video frames in the retransmitted video frame set.
Optionally, in this embodiment of the present application, the video sending end may insert a video frame in the retransmitted video frame set into a head of a video frame queue being transmitted by a current link, that is, the video sending end temporarily stops transmitting the video frame in the current video frame queue, preferentially transmits the video frame in the retransmitted video frame set, and after the transmission of the video frame in the video frame set to be retransmitted is finished, the video sending end continues to transmit the video frame in the video frame queue.
Optionally, the method for transmitting the video frame in the retransmission video frame set in the jth link by the video sending end specifically includes: and the video sending end transmits the video frames in the retransmission video frame set in the jth link according to the sequence of the priorities of the video frames in the retransmission video frame set from high to low.
In the embodiment of the present application, the priority of the video frames sequentially from low to high is: the video data in the video frame is more important when the priority is higher.
Based on the above example, the retransmission video frame set determined by the video sending end includes the base frame of the P1 frame and the extension frame of the P4 frame, and in 2 links (for example, link 1 and link 2) for transmitting the video frame to be transmitted, the transmission quality of link 1 is better than that of link 2. For link 1, it is assumed that the video frames included in the current video frame queue are a base frame of a P20 frame, an extended frame of the P20 frame, a base frame of a P21 frame, and an extended frame of the P21 frame in this order, that a 100 base frame of the P frame, and an extended frame of the P100 frame, the video transmitting end suspends transmission of the video frames in the current video frame queue (i.e., suspends transmission of the base frame of the P20 frame, etc.), and the video transmitting end sequentially transmits the base frame of the P1 frame, and the extended frame of the P4 frame in the order from top to bottom according to the priority of the video frames in the retransmission video frame set (the priority of the base frame of the P1 frame is higher than the priority of the extended frame of the P4 frame).
In the embodiment of the application, on one hand, the video sending end transmits the video frames in the retransmission video frame set in an inserting queue mode, and preferentially transmits the video frames in the retransmission video frame set, so that the video receiving end can acquire lost video frames as soon as possible, the speed of the video receiving end for aggregating the video frames is improved, and the rapid aggregation of the video frames is realized; on the other hand, when the video sending end transmits the video frames in the retransmission video frame set, the video sending end transmits the video frames in the retransmission video frame set according to the sequence from top to bottom of the priority of the video frames in the retransmission video set, and since the video frames with higher priority include more important data information, the retransmission of the video frames according to the priority can ensure that the important information in the first video data is not lost to a certain extent, so as to ensure the playing quality of the video data.
It should be noted that, in this embodiment of the present application, a method for a video sending end to transmit video data on each link is similar, and a detailed description is given below by taking a kth link as an example of a method for transmitting video data, as shown in fig. 13, the method for transmitting video data on the kth link includes: S1301-S1308.
S1301, the video sending end carries out frequency domain layered coding on the first video data to obtain a plurality of video frames of the first video data.
It should be understood that, in the above method for performing frequency domain layered coding on the first video data by the video sending end to obtain a plurality of video frames of the first video data, reference is made to the related description of S1010 in the above embodiment, and details are not repeated here.
S1302, the video sending end obtains feedback information of the kth link.
In this embodiment, the feedback information of the link may include transmission delay and/or retransmission times. Optionally, the transmission delay may be any one of an average transmission delay of video frames, a sum of transmission delays of a predetermined number of video frames, or a transmission delay of a current video frame.
And S1303, the video sending end determines the transmission quality of the kth link according to the feedback information of the kth link.
In the embodiment of the application, the video sending end determines whether the transmission quality of the kth link meets the preset condition according to the feedback information of the kth link. Specifically, the method comprises the following steps:
when the feedback information of the kth link is transmission delay, if the transmission delay of the kth link is less than a preset delay, the transmission quality of the kth link meets a preset condition; and if the transmission delay of the kth link is greater than or equal to the preset delay, the transmission quality of the kth link does not meet the preset condition.
When the feedback information of the kth link is the retransmission times, if the retransmission times of the kth link is less than the preset times, the transmission quality of the kth link meets a preset condition; and if the retransmission times of the kth link are more than or equal to the preset times, the transmission quality of the kth link does not meet the preset condition.
When the feedback information of the kth link is transmission delay and retransmission times, if the transmission delay of the kth link is smaller than preset delay and the retransmission times of the kth link are smaller than preset times, the transmission quality of the kth link meets preset conditions; and if the transmission delay of the kth link is greater than or equal to a preset threshold value and/or the retransmission times of the kth link are greater than or equal to preset times, the transmission quality of the kth link does not meet a preset condition.
It should be understood that, in the embodiment of the present application, that the transmission quality of the kth link satisfies the preset condition indicates that the transmission quality of the kth link is better, and that the transmission quality of the kth link does not satisfy the preset condition indicates that the transmission quality of the kth link is poorer.
And S1304, the video sending end determines a video frame to be transmitted corresponding to the kth link according to the transmission quality of the kth link.
Referring to the description of S920, the video frames to be transmitted corresponding to the kth link may be all video frames of the first video data, or may be partial video frames of the first video data, and specifically, S1304 includes S1304a or S1304b.
S1304a, when the transmission quality of the kth link meets a preset condition, the video frames to be transmitted corresponding to the kth link are all video frames of the first video data.
And S1304b, under the condition that the transmission quality of the kth link does not meet a preset condition, the video sending end selects part of the video frames of the first video data as the to-be-transmitted video frames corresponding to the kth link according to a preset proportion corresponding to the kth link and the sequence from high priority to low priority of the video frames of the first video data.
The preset proportion meets the following requirements:
wherein eta represents a predetermined ratio, num 1 Representing the number of non-key video frames, num, of video frames of the first video data 2 Representing a total number of video frames of the first video data; wherein a non-key video frame refers to a video frame comprising non-key data.
Optionally, the non-key video frames in the first video data include extended frames of key frames, base frames of forward frames, and extended frames of forward frames. When the transmission quality of a certain link is poor, the video sending end can choose to discard non-key video frames in the video frames of the first video data, and the influence of the non-key video frames on the video quality is small.
It should be understood that, since the base frame of the key frame contains important information in the key frame, the base frame of the key frame must be transmitted in the link as the video frame to be transmitted (i.e., the base frame of the key frame is a non-discardable video frame) in the embodiment of the present application. The priority of the video frames of the first video data is, from high to low: a key frame base frame, a forward frame base frame, a key frame extension frame, and a forward frame extension frame.
Illustratively, it is assumed that the video frames of the first video data include 100 video frames, and the preset proportion is 40% (i.e., when the link quality is poor, 60 video frames need to be discarded), and the 100 video frames include an extended frame of 48 forward frames, an extended frame of 2 key frames, a base frame of 48 forward frames, and a base frame of 2 key frames. And when the transmission quality of the kth link does not meet the preset condition, the video sending end selects the basic frames of the 38 forward frames and the basic frames of the 2 key frames as the video frames to be transmitted corresponding to the kth link according to the sequence of the priority of the video frames of the first video data from high to low.
S1305, the video sending end transmits the video frame to be transmitted corresponding to the k link on the k link.
In the method for transmitting video data provided by the embodiment of the application, a video sending end detects the transmission quality of a k link used for transmitting first video data, and determines a video frame to be transmitted corresponding to the k link according to the transmission quality of the k link, wherein under the condition that the transmission quality of the k link meets a preset condition, all video frames in the first video data are used as the video frames to be transmitted of the k link, and under the condition that the transmission quality of the k link does not meet the preset condition, part of the video frames in the first video data are used as the video frames to be transmitted of the k link; and the determined video frame to be transmitted comprises all key data in the first video data. Therefore, when the transmission quality of the kth link is poor, the part of video frames in the first video data transmitted on the kth link comprises all the key data in the first video data, so that smooth transmission of the key data in the first video data can be ensured to a certain extent, and the problems of blockage, screen splash, high time delay and the like when a video receiving end plays the received video data are solved.
And S1306, the video sending end detects the transmission quality of the kth link.
It should be noted that the above S1306 is executed after the video sender finishes transmitting the video frame to be transmitted corresponding to the k-th link on the k-th link.
In this embodiment of the present application, for a detailed description of a method for detecting transmission quality of a kth link by a video sending end, reference may be made to the related description of the foregoing embodiment, and details are not repeated here.
S1307, the video sending end adjusts the preset proportion corresponding to the kth link according to the transmission quality of the kth link.
In this embodiment of the application, the video sending end uses the adjusted preset proportion for subsequent video data transmission according to the transmission quality of the kth link, for example, the adjusted preset proportion is used for transmitting second video data, where the second video data and the first video data are derived from two adjacent video segments of the same video to be transmitted.
It should be understood that, in the embodiment of the present application, the transmission quality of the kth link is continuously detected in the above manner, and the preset ratio is dynamically adjusted according to the transmission quality of the kth link, so that smooth transmission of a video frame to be transmitted can be ensured to a certain extent, and the utilization rate of the kth link can be improved.
Referring to fig. 13, S1307 includes S1401 to S1403, as shown in fig. 14.
S1401, the video sending end judges whether the transmission quality of the kth link meets a preset condition.
In this embodiment of the application, for a detailed description of a method for the video sending end to determine whether the transmission quality of the kth link meets the preset condition, reference may be made to the related description of S1303 described above, and details are not repeated here.
And S1402, increasing the preset proportion by the video sending end under the condition that the transmission quality of the kth link meets the preset condition.
Under the condition that the transmission quality of k links meets a preset condition, the video sending end increases a preset proportion; that is to say, the transmission quality of the kth link is better, and the video sending end increases the preset proportion, so that the video sending end sends more video frames in the video data to the video receiving end, and the playing quality of the video data decoded by the video receiving end can be improved.
And S1403, under the condition that the transmission quality of the kth link does not meet the preset condition, the video sending end reduces the preset proportion.
Under the condition that the transmission quality of the kth link does not meet the preset condition, the preset proportion is reduced by the video sending end, namely, when the transmission quality of the kth link is poor, the occupation ratio of non-key video frames in the video frames to be transmitted is reduced, so that the number of video data frames sent to the video receiving end by the video sending end is reduced, the transmission pressure of the kth link is further reduced, and the transmission efficiency of the kth link is improved.
And S1308, the video sending end selects part of the video frames of the second video data as the video frames to be transmitted corresponding to the kth link according to the adjusted preset proportion corresponding to the kth link and the sequence from high priority to low priority of the video frames of the second video data.
In the embodiment of the application, after the video sending end determines the video frame to be transmitted corresponding to the kth link, the video sending end transmits the video frame to be transmitted corresponding to the kth link on the kth link.
Accordingly, the present application provides a video data transmission device, where the video data transmission device is configured to execute each step in the above-mentioned video data transmission method, and the present application may divide the video data transmission device into functional modules according to the above-mentioned method example, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The division of the modules in the embodiment of the present application is schematic, and is only a logic function division, and there may be another division manner in actual implementation.
In the case of adopting a division of the respective functional modules corresponding to the respective functions, fig. 15 shows a schematic diagram of a possible configuration of the video data transmission apparatus relating to the above-described embodiment. As shown in fig. 15, the video data transmission apparatus includes a detection module 1501, a determination module 1502, and a transmission module 1503.
The detecting module 1501 is configured to detect the transmission quality of the N links, for example, to execute S910 in the foregoing method embodiment.
The determining module 1502 is configured to determine, according to the transmission quality of the N links, video frames to be transmitted corresponding to the N links, for example, to execute S920 in the foregoing method embodiment.
The transmission module 1503 is configured to transmit the video frames to be transmitted respectively corresponding to the N links on the N links, for example, execute S930 in the foregoing method embodiment.
Optionally, the video data transmission apparatus provided in the embodiment of the present application further includes: an encoding module 1507; the encoding module 1507 is configured to perform frequency-domain layered encoding on the first video data to obtain a plurality of video frames of the first video data, for example, to perform S1010 and S1301 in the above method embodiments.
Optionally, the determining module 1502 is also configured to determine a set of retransmitted video frames, for example, to perform S1210 in the above method embodiment.
Optionally, the video data transmission apparatus provided in the embodiment of the present application further includes: a queue-insertion module 1508; the enqueueing module 1508 is configured to transmit the video frames in the retransmitted video frame set in the jth link in an enqueueing transmission manner, for example, to perform S1220 in the above-described method embodiment.
Optionally, the video data transmission apparatus provided in the embodiment of the present application further includes: an acquisition module 1504; the obtaining module 1504 is configured to obtain feedback information of the kth link, for example, to execute S1302 in the foregoing method embodiment.
Optionally, the determining module 1502 is further configured to determine the transmission quality of the kth link according to the feedback information of the kth link, for example, to execute S1303 in the foregoing method embodiment.
Optionally, the determining module 1502 is further configured to determine, according to the transmission quality of the kth link, a video frame to be transmitted corresponding to the kth link. For example, S1304 in the above-described method embodiment is performed.
Optionally, the video data transmission apparatus provided in the embodiment of the present application further includes: the control module 1505; the control module 1505 is configured to, when the transmission quality of the kth link does not meet a preset condition, select, by the video sending end, a part of video frames in the video frames of the first video data as to-be-transmitted video frames corresponding to the kth link according to a preset proportion corresponding to the kth link and according to a sequence from high to low of priorities of the video frames of the first video data; for example, S1304b in the above-described method embodiment is performed.
Optionally, the transmission module 1503 is further configured to transmit a video frame to be transmitted, corresponding to the kth link, on the kth link; for example, S1305 in the above-described method embodiment is performed.
Optionally, the detecting module 1501 is further configured to detect the transmission quality of the kth link, for example, to execute S1306 in the foregoing method embodiment.
Optionally, the video data transmission apparatus provided in the embodiment of the present application further includes: an adjustment module 1506; the adjusting module 1506 is configured to adjust, by the video sending end, the preset ratio corresponding to the kth link according to the transmission quality of the kth link, for example, to perform S1307, S1402, and S1403 in the foregoing method embodiment.
Optionally, the control module 1505 is further configured to select, according to the adjusted preset proportion corresponding to the kth link, a part of video frames in the video frames of the second video data as to-be-transmitted video frames corresponding to the kth link in an order from high priority to low priority of the video frames of the second video data; for example, S1308 in the above-described method embodiment is performed.
Each module of the video data transmission apparatus may also be configured to perform other actions in the method embodiment, and all relevant contents of each step related to the method embodiment may be referred to the functional description of the corresponding functional module, which is not described herein again.
In the case of using an integrated unit, a schematic structural diagram of a video data transmission device provided in an embodiment of the present application is shown in fig. 16. In fig. 16, the video data transmission apparatus includes: a processing module 1601 and a communication module 1602. The processing module 1601 is used for controlling and managing the actions of the video data transmission apparatus, such as performing the steps performed by the detection module 1501, the determination module 1502, the acquisition module 1504, the control module 1505, the encoding module 1507, and the queue-insertion module 1508, and/or other processes for performing the techniques described herein. The communication module 1602 is used to support interaction between the video data transmission apparatus and other devices, etc., for example, to perform the steps performed by the transmission module 1503 described above, and/or to perform other processes for the techniques described herein. As shown in fig. 16, the video data transmission apparatus may further include a storage module 1603, where the storage module 1603 is used to store program codes of the video data transmission apparatus, video frames to be transmitted, and the like.
The processing module 1601 may be a processor or a controller, such as the processor 810 in fig. 8, among others. The communication module 1602 may be a transceiver, an RF circuit or a communication interface, etc., such as the mobile communication module 850 and/or the wireless communication module 860 of fig. 8. The storage module 1603 may be a memory, such as the internal memory 821 in fig. 8.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented using a software program, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produce, in whole or in part, the processes or functions described in the embodiments of the application. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) means or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device including one or more available media integrated servers, data centers, and the like. The usable medium may be a magnetic medium (e.g., floppy disk, magnetic tape), an optical medium (e.g., digital Video Disk (DVD)), or a semiconductor medium (e.g., solid State Drive (SSD)), among others.
Through the description of the foregoing embodiments, it will be clear to those skilled in the art that, for convenience and simplicity of description, only the division of the functional modules is illustrated, and in practical applications, the above function distribution may be completed by different functional modules as needed, that is, the internal structure of the apparatus may be divided into different functional modules to complete all or part of the above described functions. For the specific working processes of the system, the apparatus and the unit described above, reference may be made to the corresponding processes in the foregoing method embodiments, and details are not described here again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules or units is only one type of logical functional division, and other divisions may be realized in practice, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit may be implemented in the form of hardware, or may also be implemented in the form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) or a processor to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: flash memory, removable hard drive, read only memory, random access memory, magnetic or optical disk, and the like.
The above description is only an embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions within the technical scope of the present disclosure should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (16)
1. A method for transmitting video data, comprising:
detecting the transmission quality of N links; the N links are used for transmitting first video data, the N links are links in different networks, the first video data comprise a plurality of video frames, and N is a positive integer greater than or equal to 2;
determining video frames to be transmitted corresponding to the N links respectively according to the transmission quality of the N links; under the condition that the transmission quality of a kth link meets a preset condition, video frames to be transmitted corresponding to the kth link are all video frames of the first video data; under the condition that the transmission quality of the kth link does not meet a preset condition, the video frames to be transmitted corresponding to the kth link are partial video frames in all the first video data; the value of k is any one of 1,2, \8230, and the value of \8230, and the video frame to be transmitted comprises all key data of the first video data;
and transmitting the video frames to be transmitted respectively corresponding to the N links in the N links.
2. The method of claim 1, wherein detecting the transmission quality of the k-th link comprises:
acquiring feedback information of the kth link, wherein the feedback information comprises transmission delay and/or retransmission times;
determining the transmission quality of the kth link according to the feedback information of the kth link;
when the feedback information is transmission delay, if the transmission delay of the kth link is smaller than preset delay, the transmission quality of the kth link meets the preset condition; if the transmission delay of the kth link is greater than or equal to a preset delay, the transmission quality of the kth link does not meet the preset condition;
when the feedback information is retransmission times, if the retransmission times of the kth link are less than preset times, the transmission quality of the kth link meets the preset condition; if the retransmission times of the kth link are greater than or equal to preset times, the transmission quality of the kth link does not meet the preset condition;
when the feedback information is transmission delay and retransmission times, if the transmission delay of the kth link is less than a preset delay and the retransmission times of the kth link is less than the preset times, the transmission quality of the kth link meets the preset condition; and if the transmission delay of the kth link is greater than or equal to a preset threshold value and/or the retransmission times of the kth link are greater than or equal to preset times, the transmission quality of the kth link does not meet the preset condition.
3. The method according to claim 1 or 2, wherein the determining a video frame to be transmitted corresponding to the kth link if the transmission quality of the kth link does not meet a preset condition comprises:
according to the preset proportion corresponding to the k link, selecting a part of video frames in the video frames of the first video data as video frames to be transmitted corresponding to the k link from high to low priority of the video frames of the first video data;
the preset proportion meets the following requirements:
wherein eta represents the preset ratio, num 1 Representing the number of non-key video frames, num, of the video frames of the first video data 2 Representing a total number of video frames of the first video data; wherein the non-key video frames refer to video frames comprising non-key data.
4. The method according to claim 3, wherein after the video frames to be transmitted respectively corresponding to the N links are transmitted in the N links, the method further comprises:
detecting the transmission quality of the k link;
and adjusting the preset proportion corresponding to the kth link according to the transmission quality of the kth link.
5. The method according to claim 4, wherein adjusting the preset ratio corresponding to the kth link according to the transmission quality of the kth link includes:
increasing the preset proportion under the condition that the transmission quality of the kth link meets the preset condition;
and reducing the preset proportion under the condition that the transmission quality of the kth link does not meet the preset condition.
6. The method according to claim 4 or 5, characterized in that the method further comprises:
determining a video frame to be transmitted corresponding to second video data on the K link according to the adjusted preset proportion corresponding to the K link;
and transmitting the video frame to be transmitted corresponding to the k link in the k link.
7. The method according to any one of claims 3-6, further comprising:
performing frequency domain layered coding on first video data to obtain a video frame of the first video data, wherein the video frame of the first video data comprises a basic frame and an extended frame of at least one key frame and a plurality of basic frames and extended frames of a forward frame; the non-key video frames comprise extended frames of key frames, basic frames of forward frames and extended frames of forward frames, and the key data of the first video data are the basic frames of the key frames.
8. The method of claim 7,
the priority of the video frame of the first video data is the basic frame of the key frame, the basic frame of the forward frame, the extended frame of the key frame and the extended frame of the forward frame in sequence from high to low.
9. The method of any one of claims 1 to 7,
the ith video frame in the video frames of the first video data corresponding to the first link is a basic frame of the first video frame, and the ith video frame in the video frames of the first video data corresponding to the second link is an extended frame of the first video frame; the first link and the second link are two different links of the N links.
10. The method according to any one of claims 1-8, further comprising:
determining a retransmission video frame set, wherein the retransmission video frame set is an intersection of video frames to be retransmitted corresponding to the N links respectively;
transmitting the video frames in the retransmission video frame set in a jth link in a queue-insertion transmission mode; wherein the transmission quality of the j link is higher than that of other links, and the other links are links of the N links except for the j link.
11. The method of claim 10, wherein transmitting a video frame in the retransmitted set of video frames in a jth link comprises:
and transmitting the video frames in the jth link according to the sequence of the priorities of the video frames in the retransmission video frame set from high to low.
12. The method according to any of claims 1-11, wherein transmitting the video frames in the retransmitted set of video frames in a j-th link in a queue-insertion transmission manner comprises:
and inserting the retransmission video frame set into the head bit of the video frame queue to be transmitted corresponding to the j link.
13. The method according to any one of claims 1-12, wherein before transmitting the video frames to be transmitted respectively corresponding to the N links in the N links, the method further comprises:
and encrypting the video frames to be transmitted respectively corresponding to the N links.
14. A video data transmission apparatus comprising a memory and a processor, the memory coupled to the processor; the memory for storing computer program code, the computer program code comprising computer instructions; the computer instructions, when executed by the processor, cause the video data transmission apparatus to perform the method of any of claims 1 to 13.
15. A computer storage medium comprising computer instructions which, when run on a video data transmission apparatus, cause the video data transmission apparatus to perform the method of any one of claims 1 to 13.
16. A computer program product, characterized in that, when run on a computer, causes the computer to perform the method according to any one of claims 1 to 13.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110729347.XA CN115550683A (en) | 2021-06-29 | 2021-06-29 | Video data transmission method and device |
| PCT/CN2022/096020 WO2023273763A1 (en) | 2021-06-29 | 2022-05-30 | Video data transmission method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110729347.XA CN115550683A (en) | 2021-06-29 | 2021-06-29 | Video data transmission method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115550683A true CN115550683A (en) | 2022-12-30 |
Family
ID=84692518
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110729347.XA Pending CN115550683A (en) | 2021-06-29 | 2021-06-29 | Video data transmission method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN115550683A (en) |
| WO (1) | WO2023273763A1 (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116405741B (en) * | 2023-04-21 | 2024-01-16 | 深圳大学 | Video transmission scheduling method, device and medium based on multiple transmission paths |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103945281A (en) * | 2014-04-29 | 2014-07-23 | 中国联合网络通信集团有限公司 | Method, device and system for video transmission processing |
| CN104320669A (en) * | 2014-10-24 | 2015-01-28 | 北京有恒斯康通信技术有限公司 | Video transmission method and apparatus |
| CN105553868A (en) * | 2015-09-21 | 2016-05-04 | 宇龙计算机通信科技(深圳)有限公司 | Data transmission method and device and terminal |
| CN105656774A (en) * | 2016-01-12 | 2016-06-08 | 东北大学 | Multipath hybrid redundant transmission method for real-time network video session media data |
| CN107210827A (en) * | 2014-10-13 | 2017-09-26 | 大力系统公司 | Replace marred video frame |
| CN109412756A (en) * | 2018-11-09 | 2019-03-01 | 广州美凯信息技术股份有限公司 | A kind of double-link backup method, system and storage medium based on optical fiber 4KHDR ultra high-definition transmission of video |
| CN109951722A (en) * | 2019-02-27 | 2019-06-28 | 腾讯科技（深圳）有限公司 | The transmission method and transmitting device of video data |
| CN110225347A (en) * | 2019-06-24 | 2019-09-10 | 北京大米科技有限公司 | Method of transmitting video data, device, electronic equipment and storage medium |
| CN111294664A (en) * | 2019-05-28 | 2020-06-16 | 杭州萤石软件有限公司 | Audio and video data transmission method, electronic equipment and storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106301733B (en) * | 2015-06-26 | 2020-11-20 | 中兴通讯股份有限公司 | Data transmission method and device |
| US10505597B2 (en) * | 2016-05-09 | 2019-12-10 | Qualcomm Incorporated | Reference signals and link adaptation for massive MIMO |
| EP3858084A1 (en) * | 2018-09-27 | 2021-08-04 | FRAUNHOFER-GESELLSCHAFT zur Förderung der angewandten Forschung e.V. | Sidelink feedback |
-
2021
- 2021-06-29 CN CN202110729347.XA patent/CN115550683A/en active Pending
-
2022
- 2022-05-30 WO PCT/CN2022/096020 patent/WO2023273763A1/en active Application Filing
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103945281A (en) * | 2014-04-29 | 2014-07-23 | 中国联合网络通信集团有限公司 | Method, device and system for video transmission processing |
| CN107210827A (en) * | 2014-10-13 | 2017-09-26 | 大力系统公司 | Replace marred video frame |
| CN104320669A (en) * | 2014-10-24 | 2015-01-28 | 北京有恒斯康通信技术有限公司 | Video transmission method and apparatus |
| CN105553868A (en) * | 2015-09-21 | 2016-05-04 | 宇龙计算机通信科技(深圳)有限公司 | Data transmission method and device and terminal |
| CN105656774A (en) * | 2016-01-12 | 2016-06-08 | 东北大学 | Multipath hybrid redundant transmission method for real-time network video session media data |
| CN109412756A (en) * | 2018-11-09 | 2019-03-01 | 广州美凯信息技术股份有限公司 | A kind of double-link backup method, system and storage medium based on optical fiber 4KHDR ultra high-definition transmission of video |
| CN109951722A (en) * | 2019-02-27 | 2019-06-28 | 腾讯科技（深圳）有限公司 | The transmission method and transmitting device of video data |
| CN111294664A (en) * | 2019-05-28 | 2020-06-16 | 杭州萤石软件有限公司 | Audio and video data transmission method, electronic equipment and storage medium |
| CN110225347A (en) * | 2019-06-24 | 2019-09-10 | 北京大米科技有限公司 | Method of transmitting video data, device, electronic equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 邓中亮，段大高，崔岩松等: "《基于H.264的视频编/解码与控制技术》", 31 July 2010, 北京邮电大学出版社, pages: 9 - 12 * |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2023273763A1 (en) | 2023-01-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110381345B (en) | Screen projection display method and electronic equipment | |
| CN110072070B (en) | Multi-channel video recording method, equipment and medium | |
| EP4050900A1 (en) | Screen projection audio and video playback method and electronic device | |
| WO2020244623A1 (en) | Air-mouse mode implementation method and related device | |
| JP7324311B2 (en) | Audio and video playback method, terminal, and audio and video playback device | |
| WO2020133183A1 (en) | Audio data synchronization method and device | |
| CN110519632B (en) | Screen projection method and equipment | |
| CN114726946A (en) | Method for automatically switching Bluetooth audio coding modes and electronic equipment | |
| CN113542839A (en) | Screen projection method of electronic equipment and electronic equipment | |
| WO2022121775A1 (en) | Screen projection method, and device | |
| CN114489533A (en) | Screen projection method and device, electronic equipment and computer readable storage medium | |
| CN112954251B (en) | Video processing method, video processing device, storage medium and electronic equipment | |
| CN112771828B (en) | Audio data communication method and electronic equipment | |
| CN113497851B (en) | Control display method and electronic equipment | |
| WO2023273763A1 (en) | Video data transmission method and device | |
| WO2024012345A1 (en) | Mirroring picture processing method and related apparatus | |
| CN114339429A (en) | Audio and video playing control method, electronic equipment and storage medium | |
| CN114205336A (en) | Cross-device audio playing method, mobile terminal, electronic device and storage medium | |
| CN112469012A (en) | Bluetooth communication method and related device | |
| US20230319217A1 (en) | Recording Method and Device | |
| CN113596320B (en) | Video shooting variable speed recording method, device and storage medium | |
| CN113923528B (en) | Screen sharing method, terminal and storage medium | |
| CN112153404B (en) | Code rate adjusting method, code rate detecting method, code rate adjusting device, code rate detecting device, code rate adjusting equipment and storage medium | |
| CN115525366A (en) | Screen projection method and related device | |
| CN114125352A (en) | Screen sharing method, electronic equipment and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |