CN108769726B

CN108769726B - Multimedia data pushing method and device, storage medium and equipment

Info

Publication number: CN108769726B
Application number: CN201810614487.0A
Authority: CN
Inventors: 余学亮
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2018-06-14
Filing date: 2018-06-14
Publication date: 2022-05-31
Anticipated expiration: 2038-06-14
Also published as: CN108769726A

Abstract

The invention discloses a multimedia data pushing method, a multimedia data pushing device, a multimedia data storage medium and multimedia data pushing equipment, and belongs to the technical field of internet. The method comprises the following steps: receiving multimedia data sent by a first terminal through at least two network channels, wherein the first terminal is used for sending the multimedia data generated in the live broadcast process out through the at least two network channels; carrying out data recombination on the received multimedia data; and pushing the recombined multimedia data to the second terminal. The first terminal sends the multimedia data through the plurality of network channels, and then the server recombines the multimedia data transmitted by the plurality of network channels, so that the integrity of the multimedia data output to the second terminal can be effectively guaranteed, the problem of poor live broadcast effect caused by serious data loss due to poor network state when the multimedia data are transmitted through a single network channel is greatly reduced, in addition, the image definition cannot be reduced by the multimedia data pushing mode, and the video image quality is better.

Description

Multimedia data pushing method and device, storage medium and equipment

Technical Field

The present invention relates to the field of internet technologies, and in particular, to a multimedia data pushing method, apparatus, storage medium, and device.

Background

The method of playing the multimedia data by using the terminal has become one of the important ways for people to show themselves and play entertainment, wherein the multimedia data includes picture data and audio data. Taking the live broadcast scene of the mobile terminal as an example, since the location where the terminal of the anchor user is located is not fixed, the network accessed by the terminal is generally a shared network or a mobile network. For the shared network, the number of access people, the signal coverage range, the bandwidth upper limit and other factors restrict, and when the number of terminals accessing the shared network is more and the terminals deviate from the signal coverage range is farther, the terminals are unsmooth when the multimedia data are sent through the network channel of the shared network; for the mobile network, the mobile network signal is greatly reduced when the location is remote or the number of people around is too large due to the restriction of factors such as geographical location and the number of people around, and the phenomenon that the data transmission is not smooth can also occur when the terminal transmits multimedia data through a network channel of the mobile network, and the live broadcast quality is undoubtedly seriously influenced by the occurrence of the phenomenon.

In the related art, in order to ensure the live broadcast quality in the poor network environment, when a terminal sends multimedia data through a single network channel, the following processing manner is generally adopted: according to the current data uploading speed and the occupation ratio of the cached multimedia data, a video encoder is called to adjust a coding compression strategy, and the coding data amount of each frame in the live broadcasting process is reduced by adopting a mode of reducing the code rate of the video encoder; namely, the terminal reduces the data volume of the picture data generated in the live broadcast process to relieve the data uploading pressure of a single network channel.

Aiming at the data pushing mode, when a single network channel uploads data in a poor network state, a large amount of data is likely to be lost, which shows that the phenomena of abnormal playing pictures, pause playing or frame skipping and the like occur at a user watching side, and the live broadcasting effect is poor; in addition, since the code rate of video coding is related to the picture definition, the code rate of video coding is reduced to reduce the amount of coded data, and the picture definition is also used as a cost, which may cause a problem of poor video quality and further cause a poor live broadcast effect.

Disclosure of Invention

The embodiment of the invention provides a multimedia data pushing method, a multimedia data pushing device, a storage medium and equipment, wherein the multimedia data is pushed in parallel by adopting a plurality of network channels, the phenomena of abnormal playing pictures, pause playing or frame skipping and the like at a user watching side can be effectively solved, the video picture quality is good, and the live broadcast effect is better. The technical scheme is as follows:

in one aspect, a multimedia data pushing method is provided, which is applied to a server, and the method includes:

receiving multimedia data sent by a first terminal through at least two network channels, wherein the first terminal is used for sending the multimedia data generated in the live broadcast process out through the at least two network channels;

carrying out data recombination on the received multimedia data to obtain recombined multimedia data;

and pushing the recombined multimedia data to a second terminal.

In another aspect, a multimedia data push method is provided, and is applied to a terminal, where the method includes:

acquiring multimedia data generated in a live broadcast process;

and sending the multimedia data to a server through at least two network channels so that the server performs data recombination on the multimedia data received through the at least two network channels and pushes the obtained recombined multimedia data to a second terminal.

In another aspect, a multimedia data pushing apparatus is provided, which is applied to a server, and includes:

the system comprises a receiving module, a processing module and a processing module, wherein the receiving module is used for receiving multimedia data sent by a first terminal through at least two network channels, and the first terminal is used for sending the multimedia data generated in the live broadcasting process out through the at least two network channels;

the processing module is used for carrying out data recombination on the received multimedia data to obtain recombined multimedia data;

and the first sending module is used for pushing the recombined multimedia data to the second terminal.

In another aspect, a multimedia data push apparatus is provided, which is applied to a terminal, and includes:

the acquisition module is used for acquiring multimedia data generated in a live broadcasting process;

and the second sending module is used for sending the multimedia data to a server through at least two network channels so that the server performs data recombination on the multimedia data received through the at least two network channels and pushes the obtained recombined multimedia data to a second terminal.

In another aspect, a storage medium is provided, where at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement the multimedia data pushing method described above.

In another aspect, a multimedia data pushing device is provided, where the device includes a processor and a memory, where the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the multimedia data pushing method described above.

The technical scheme provided by the embodiment of the invention has the following beneficial effects:

the first terminal sends multimedia data through a plurality of network channels, then, the server recombines the multimedia data transmitted through the plurality of network channels, the integrity of the multimedia data finally output to the second terminal can be effectively guaranteed, the problem that the live broadcast effect is poor due to serious data loss caused by poor network state when the multimedia data are transmitted through a single network channel is greatly reduced, an expression mode is changed, the abnormal decoding phenomenon is greatly reduced on the side of a watching user due to the fact that the consistency and the continuity of the multimedia data output to the second terminal are guaranteed, the problem of picture playing such as green screen, flower screen, mosaic and the like of a screen can be effectively solved, the problems of smoothness such as frame skipping, blocking, incoherence and the like are solved, and the live broadcast effect is good. In addition, the first terminal sends the multimedia data through a plurality of network channels, so that dynamic code rate adaptation is not needed to be carried out to reduce the packet loss rate in order to relieve network congestion during front-end coding, the image definition cannot be reduced by the multimedia data pushing mode, the video quality is better, and the live broadcast effect is further ensured.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic structural diagram of a multimedia data pushing method according to an embodiment of the present invention;

fig. 2 is a schematic diagram of data uploading related to a multimedia data pushing method according to an embodiment of the present invention;

fig. 3 is a flowchart of a multimedia data pushing method according to an embodiment of the present invention;

FIG. 4 is a block diagram illustrating an embodiment of a data block structure;

fig. 5 is a flowchart of a multimedia data pushing method according to an embodiment of the present invention;

FIG. 6 is a schematic flow chart of data reassembly according to an embodiment of the present invention;

FIG. 7 is a schematic flow chart of another data reorganization provided by the embodiment of the present invention;

fig. 8 is a flowchart of a multimedia data pushing method according to an embodiment of the present invention;

fig. 9 is a schematic diagram of data uploading related to a multimedia data pushing method according to an embodiment of the present invention;

fig. 10 is a flowchart illustrating an overall implementation of a multimedia data pushing method according to an embodiment of the present invention;

fig. 11 is a flowchart illustrating an overall implementation of a multimedia data pushing method according to an embodiment of the present invention;

fig. 12 is a schematic flowchart of transmitting multimedia data according to an embodiment of the present invention;

fig. 13 is a schematic structural diagram of a multimedia data pushing apparatus according to an embodiment of the present invention;

fig. 14 is a schematic structural diagram of a multimedia data pushing apparatus according to an embodiment of the present invention;

fig. 15 is a schematic structural diagram of a multimedia data pushing apparatus according to an embodiment of the present invention;

fig. 16 is a schematic structural diagram of a multimedia data pushing device according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Before explaining the embodiments of the present invention in detail, some terms related to the embodiments of the present invention are explained.

Multimedia data: and the multimedia data which are collected and coded by the terminal of the anchor user in real time when the anchor user initiates the live broadcasting. In the embodiment of the present invention, the multimedia data is generally pushed to the server by the terminal of the anchor user in the form of a live stream.

Sharing the network: instead of being shared by one terminal, the shared network is usually accessed by a plurality of terminals, and the larger the number of terminals accessing the shared network is, the worse the stability of the shared network is. Illustratively, a WiFi (Wireless Fidelity) network provided by an offline shop is one of the shared networks.

FLV (FLASH VIDEO, streaming media format) Tag packet: FLV is a general Audio and Video data encapsulation format, and for a live stream adopting the FLV data encapsulation format, an FLV Tag packet is a basic unit constituting an FLV data stream, which may be divided into a Video (Video) Tag and an Audio (Audio) Tag, and exemplarily, the composition of the FLV data stream may be as follows:

FLV Header+Header Size、Video Tag+Tag Size、Audio Tag+Tag Size、Audio Tag+Tag Size、Video Tag+Tag Size、Audio Tag+Tag Size、Audio Tag+Tag Size…。

wherein, the above-mentioned from left to right is live streaming head + head size, video label + label size, audio label + label size … in proper order.

Note that video mentioned here refers to picture data only, and for distinguishing it from audio data, it will be referred to as picture data hereinafter.

FLV Chunk: the FLV Tag packets are combined and packed according to the video frame rate or the video frame interval duration to form FLV chunks, where the FLV chunks may also be referred to as FLV data blocks, or simply referred to as data blocks.

Assuming that the Video frame rate of a live stream is 25, the Video frame interval duration is 1/25 ═ 0.04s, and then 1 Video Tag and 2 to 3 Audio tags are contained in a FLV data block, i.e. the duration of the combined multimedia data is about 0.04 s. In another expression, one FLV data block comprises Video Tag + Tag Size, Audio Tag + Tag Size, and Audio Tag + Tag Size; according to the above combination mode, the live stream combined by taking FLV data blocks as units is: FLV Header + Header Size, FLV Chunk, … ….

Code rate: used to measure the amount of data per second of the live stream in kbps. The code rate is divided into a video code rate and an audio code rate, and generally, the higher the code rate is, the better the definition of a corresponding video picture is and the better the tone quality of audio is; in addition, the code rate also affects data uploading, and generally, the higher the code rate is, the higher the requirement is correspondingly when data uploading is performed, for example, the required bandwidth and the network quality are improved therewith.

CDN (Content Delivery Network): the method can redirect the request initiated by the user to the service node closest to the user in real time according to the network flow and the comprehensive information such as the connection, load condition, distance to the user, response time and the like of each service node. The method aims to enable the user to obtain the required content nearby so as to solve the situation of network congestion and improve the response speed of the user for accessing the website.

The following briefly introduces an implementation environment related to the multimedia data pushing method provided by the embodiment of the present invention. Referring to fig. 1, the implementation environment includes a terminal 101 of a host user, a live stream receiving server 102, a CDN server 103, a terminal 104 of a viewing user, and a data reorganization server 105.

The multimedia data pushing method provided by the embodiment of the invention adopts a plurality of network channels to push the multimedia data, and accordingly, the number of the live stream receiving servers 102 is usually multiple. Exemplarily, only 2 network channels, i.e. 2 live stream receiving servers 102, are illustrated in fig. 1.

The types of the terminal 101 of the anchor user and the terminal 104 of the watching user include, but are not limited to, a smart phone, a tablet computer, and the like, and fig. 1 only illustrates that both are smart phones. The live stream receiving server 102, the CDN server 103 and the data restructuring server 105 may be independent servers or may be integrated into one server, which is not specifically limited in this embodiment of the present invention.

The network channels may be WiFi networks, 2G (2nd Generation mobile communication technology, second Generation mobile communication technology), 3G (3rd Generation mobile communication technology, third Generation mobile communication technology), 4G (the 4th Generation mobile communication technology, fourth Generation mobile communication technology), and the like, which is not specifically limited in the embodiment of the present invention.

The first point to be noted is that the network channels exist independently and do not interfere with each other, and for the two network channels in fig. 1, if one of the network channels is a 4G network channel of an a operator, the other network channel may be a 4G network channel of a B operator or a WiFi network.

The second point to be noted is that the data amount transmitted by each network channel may be the whole data amount generated by the terminal 101 of the anchor user or may be a partial data amount. Taking the data volume of the multimedia data generated by the terminal 101 of the anchor user as M as an example, the first push mode may be that each network channel transmits the multimedia data with the data volume of M; the second push method may be that each network channel transmits a part of the data volume M, and taking N network channels as an example, the data volume transmitted by each network channel may be M/N.

Illustratively, for example, when the terminal 101 of the anchor user pushes the multimedia data, it may select to copy a copy of the multimedia data generated in the live broadcast process as redundant data to upload the multimedia data through 2 network channels, respectively. The 2 network channels may correspond to a dual-card dual-standby situation of the terminal, for example, the terminal 101 of the anchor user is configured with 2 4G cards, which are respectively sourced from an operator a and an operator B, as shown in fig. 1, when the terminal 101 of the anchor user pushes multimedia data, a copy of the multimedia data may be uploaded to the live stream receiving server 102 of the operator a through the network channel of the operator a, and similarly, the copied redundant data may be uploaded to the live stream receiving server 102 of the operator B through the network channel of the operator B; then, the two live stream receiving servers 102 forward the received multimedia data to the data restructuring server 105, and the data restructuring server 105 summarizes the two received paths of multimedia data; after the two paths of multimedia data are recombined, screened and arranged by the data recombination server 105, a reliable piece of multimedia data is output to the CDN server 103, and the CDN server 103 pushes the reliable piece of multimedia data to the terminal 104 of the viewing user.

Taking the first push method as an example, a detailed example will be described below about a process of pushing multimedia data in parallel through multiple network channels. Referring to fig. 2, assuming that the data amount of multimedia data generated by the terminal 101 of the anchor user is M, including a total of N +1 data blocks numbered 1/2/3/4 … … to N +1, if the multimedia data of data amount M is uploaded through the network channel 1, the data block 2 and the data block 4 are lost, when multimedia data with data volume M is uploaded through the network channel 2, the data block 1 and the data block 3 are lost, and then after the live stream receiving server 102 corresponding to the two network channels respectively sends the received multimedia data to the data reorganization server 105, the data reorganization server 105 recombines into a complete multimedia data, the multimedia data includes a total of N +1 data blocks numbered 1/2/3/4 … … to N +1, which ensures the integrity of the multimedia data.

In summary, the method for pushing multimedia data through multiple network channels in parallel according to the embodiments of the present invention can effectively solve the following problems in the related art when multimedia data is pushed through a single network channel:

when the state of the network accessed by the terminal of the anchor user is not good, for example, the accessed network is in a weak network state, a packet loss phenomenon is likely to occur due to network congestion when multimedia data is pushed, and on the side of the viewing user, a situation that decoding of other frames which are subsequently decoded by the frame fails due to loss of part of reference frames may exist, and accordingly, local green screen, flower screen, mosaic, frame skipping, pause, play discontinuity and the like may occur on the screen of the terminal of the viewing user. Although the dynamic code rate adaptation scheme is adopted to relieve the packet loss phenomenon caused by network congestion to a certain extent and relieve the data transmission pressure, the picture definition is required to be used as a cost, namely the video image quality is reduced, and the effect is poor.

In the embodiment of the invention, the integrity of the multimedia data finally output to the user side of the audience can be effectively ensured by adopting the plurality of network channels to push the multimedia data and carrying out screening and recombination on the multimedia data transmitted by the plurality of network channels, the problem of poor live broadcast effect caused by serious data loss due to poor network state when the multimedia data is transmitted through a single network channel is greatly reduced, namely, the problem of poor live broadcast effect caused by serious data loss due to poor network state is ensured, namely, due to the fact that the consistency and continuity of the multimedia data output to the user side of the audience are ensured, abnormal decoding phenomena on the user side of the audience are greatly reduced, the problems of picture playing such as green screen, flower screen, mosaic and the like, frame skipping, blocking, incoherence and the like of the screen can be effectively solved, and the live broadcast effect is better. In addition, the terminal of the anchor user sends the multimedia data through a plurality of network channels, so that dynamic code rate adaptation is not needed to be carried out to reduce the packet loss rate in order to relieve network congestion during front-end coding, the image definition cannot be reduced by the multimedia data pushing mode, the video quality is better, and the live broadcast effect is further ensured.

The following explains the method for pushing multimedia data in parallel through multiple network channels according to an embodiment of the present invention by using a detailed implementation manner.

Fig. 3 is a flowchart of a multimedia data pushing method according to an embodiment of the present invention. An interactive main body of the method is a first terminal, a server and a second terminal, where the first terminal is a terminal 101 of a main user in fig. 1, the second terminal is a terminal 104 of a watching user in fig. 1, and the server at least includes a live stream receiving server 102 and a data reorganizing server 105 in fig. 1, where the live stream receiving server 102 and the data reorganizing server 105 may be independent servers, or may be integrated into one server, which is not specifically limited in this embodiment of the present invention. Referring to fig. 3, a method flow provided by the embodiment of the present invention includes:

301. the first terminal acquires multimedia data generated in a live broadcast process and sends the multimedia data to the server through at least two network channels.

As described above, the multimedia data pushed by the first terminal is specifically a data block (Chunk) unit, and each data block includes a header and a body, and for any data block, the header of the data block includes a sequence number of the data block, and the body of the data block includes picture data and audio data.

Illustratively, as shown in fig. 4, for a data block of FLV Chunk1, the Chunk Number field of the FLV Chunk Header at the head thereof includes the sequence Number of the data block, and the body part of the data block may include 1 Video Tag + Tag Size, and 2 to 3 Audio tags + Tag Size, where the body part includes picture data and how much Audio data, which relate to the Video frame rate.

The first point to be noted is that the internal structure of the other data blocks shown in fig. 4 is similar to the data block named FLV Chunk 1.

In addition, for each data block, an increasing sequence number may be allocated to each data block according to the sequence of each data block, for example, sequence number1 of FLV Chunk1, sequence number2 of FLV Chunk2, sequence number 3 of FLVChunk3, sequence number … …, and sequence number N of FLV ChunkN.

The second point to be described is that, for the above serial numbers, each data block needs to be carried from the live stream receiving server to the data reassembly server, and then, the data reassembly server can verify and reassemble and sort the multimedia data uploaded through multiple network channels according to the serial number of each data block; in another expression, after receiving the multimedia data forwarded by the live stream receiving server, the data reassembly server specifically restores the multimedia data generated by the first terminal according to the sequence of increasing sequence numbers.

The third point to be described is that the first terminal can send the multimedia data through two or more network channels, and each network channel is independent from each other and does not interfere with each other. The number of the network channels is based on the number that the first terminal can support on hardware, which is not specifically limited in the embodiment of the present invention.

302. After receiving the multimedia data sent by the first terminal through the at least two network channels, the server performs data recombination on the received multimedia data and pushes the obtained recombined multimedia data to the second terminal.

In the embodiment of the present invention, the first terminal specifically pushes multimedia data to a live stream receiving server corresponding to each network channel. For example, if one network channel is from an operator a, the first terminal pushes multimedia data to a live streaming receiving server of the operator a through the network channel, and if the other network channel is from an operator B, the first terminal pushes multimedia data to a live streaming receiving server of the operator B through the network channel, and after receiving the multimedia data sent by the first terminal, each live streaming receiving server forwards the received multimedia data to a data reassembly server, so that the data reassembly server completes the screening and reassembly of the data.

It should be noted that, although there is a risk of packet loss when the multimedia data is uploaded through each network channel, after the multimedia data transmitted through the plurality of network channels is screened and recombined by the data recombination server, the data recombination server may restore a reliable piece of multimedia data as much as possible, so as to reduce packet loss as much as possible, thereby maintaining the integrity of the data.

In the method provided by the embodiment of the invention, the terminal of the anchor user sends the multimedia data through a plurality of network channels, and then, the server can effectively ensure the integrity of the multimedia data which is finally output to the audience user side by screening and recombining the multimedia data transmitted by a plurality of network channels, greatly reduce the problem of poor live broadcast effect caused by serious data loss due to poor network state when the multimedia data is transmitted by a single network channel, and change an expression mode, since the consistency and continuity of the multimedia data output to the viewing user side are secured, therefore, the abnormal decoding phenomenon on the user watching side is greatly reduced, the problems of picture playing such as green screen, flower screen, mosaic and the like on the screen and fluency such as frame skipping, blockage, incoherence and the like can be effectively solved, and the live broadcast effect is better.

In addition, the terminal of the anchor user sends the multimedia data through a plurality of network channels, so that dynamic code rate adaptation is not needed to be carried out to reduce the packet loss rate in order to relieve network congestion during front-end coding, the image definition cannot be reduced by the multimedia data pushing mode, the video quality is better, and the live broadcast effect is further ensured.

In another embodiment, when the first terminal sends the multimedia data through each network channel, the data filtering and recombining server takes different measures when screening and recombining the data according to different data amounts transmitted by each network channel. Assuming that the data volume sent by the first terminal through each network channel is consistent with the data volume generated in the live broadcast process, that is, if a redundant data transmission manner of multiple network channels is adopted, as shown in fig. 5, the data reassembly server includes the following steps when performing data reassembly on the received multimedia data:

501. and respectively caching the multimedia data received through different network channels into different input caches, wherein the sequence numbers of the data blocks stored at the cache initial positions in each input cache are consistent.

The number of the input caches is consistent with the number of the network channels, and if the number of the network channels is M, the number of the input caches is also M. That is, the input buffer is used to store the live stream originally pushed by the anchor user and forwarded by the live stream receiving server, and further, the data reassembly server stores the multimedia data transmitted in the same network channel in the same input buffer.

In addition, the data restructuring server further provides an output cache for storing the filtered and restructured multimedia data, so that the CDN server distributes the multimedia data to downstream viewing users, that is, the live stream that is finally pushed to the viewing users is the multimedia data stored by the output cache.

It should be noted that, when each input buffer stores multimedia data transmitted by each network channel, the following principle is also needed: and ensuring that the sequence numbers of the data blocks stored at the initial cache position in each input cache are consistent to be used as a basis for subsequently judging whether the data blocks are lost.

The above storage principle is described in detail below with reference to fig. 6 as an example. Assuming that the number of the network channels is 2, the data reassembly server needs 2 input buffers, which are respectively named as input buffer 1 and input buffer 2, and for the two input buffers, when the multimedia data is cached, it is further required to ensure that the sequence number of the data block stored at the cache start position in the input buffer 1 is consistent with the sequence number of the data block stored at the cache start position in the input buffer 2. Illustratively, in fig. 6, the sequence number of the first data Chunk FLVChunk1 stored in the input buffer 1 is consistent with the sequence number of the first data Chunk FLV Chunk2 stored in the input buffer 2, for example, the sequence numbers of FLV Chunk1 and FLV Chunk2 are both 1.

502. When the data amount stored in each input buffer is located in the first target interval, data reading is sequentially performed from the buffer starting position of each input buffer.

In the embodiment of the present invention, the amount of data stored in each input buffer is usually measured by the number of data blocks. Referring to fig. 6, taking the symbol N as an example of the number of data blocks of the cache, when the number N of data blocks stored in each input cache is in the first target interval, the data reassembly server starts to read data blocks sequentially from the cache start position of each input cache in parallel. The value of N is crucial, if the value of N is small, the accuracy of the data reorganization server in data screening and reorganization may be affected due to insufficient data cached in the input cache, and if the value of N is too large, the live broadcast real-time performance may be affected and the delay may be large. Generally, the value of N is based on the amount of data that can be played in the input buffer for 3 to 5 seconds, and assuming that the current live broadcast is at a frame rate of 15, the value of N is approximately 45 to 75, so that the data reorganization server can start to read data when 45 to 75 data blocks are stored in each input buffer. In addition, step 502 may be started to be executed when the input buffer with the data amount located in the first target interval is present, which is not specifically limited in the embodiment of the present invention.

503. And performing data reorganization based on the data blocks read in each input buffer.

In the embodiment of the present invention, when the data restructuring server performs data restructuring based on the data block read in each input cache, the following method may be specifically adopted:

and a, sequentially judging whether the sequence numbers of the data blocks stored at the same cache position are consistent or not from the cache initial position of each input cache.

Continuing to take the Number of network channels as 2 as an example, referring to fig. 7, the data reassembly server sequentially reads the data blocks stored in each input buffer from the first data block stored, and parses the Chunk Number field in the header of each data block to obtain the sequence Number of each data block.

And step b, when the serial numbers of the data blocks stored in the same cache position are consistent, storing the data block stored in any input cache to the output cache.

As shown in fig. 7, if the sequence numbers of the data blocks stored in the same cache positions in the input cache 1 and the input cache 2 are consistent, it indicates that none of the 2 network channels is lost when transmitting the data block, and for this case, the data reassembly server may select to store the data block stored in any one of the input caches to the output cache.

For example, in fig. 7, the sequence number of the first data block stored in the input buffer 1 and the sequence number of the first data block stored in the input buffer 2 are both 1, so that the data block stored in the input buffer 1 may be stored to the buffer start position of the output buffer, or the data block stored in the input buffer 2 may also be stored to the buffer start position of the output buffer.

And c, when the serial numbers of the data blocks stored at the same cache position are inconsistent, determining a target data block with the minimum serial number, and storing the target data block to an output cache.

Under the condition that the sequence numbers of the data blocks stored in the same cache positions in the input cache 1 and the input cache 2 are inconsistent, if the sequence number of the second data block stored in the input cache 1 is 2, but the sequence number of the second data block stored in the input cache 2 is 3, it indicates that packet loss occurs in a network channel corresponding to the input cache 2 when data is transmitted, so that the sequence numbers of the data blocks in the input cache 2 are out of order; if the sequence number of the second data block stored in the input buffer 1 is 3, but the sequence number of the second data block stored in the input buffer 2 is 2, it indicates that packet loss occurs in the network channel corresponding to the input buffer 1 during data transmission, resulting in disorder of the sequence numbers of the data blocks in the input buffer 1.

For this situation, the sequence number with the smallest value needs to be selected, and the target data block with the smallest sequence number needs to be stored in the output buffer, and continuing to take fig. 7 as an example, the second data block stored in the input buffer 1 needs to be written into the output buffer. Taking the example that Chunk Number1 refers to the sequence Number of a data block at any position in the input buffer 1, and Chunk Number2 refers to the sequence Number of a data block at the same position in the input buffer 2, Min (Chunk Number1, Chunk Number2) needs to be selected, and then the data block corresponding to Min (Chunk Number1, Chunk Number2) is stored in the output buffer.

In the embodiment of the disclosure, the terminal of the anchor user sends multimedia data through a plurality of network channels, and the data volume sent by each network channel is consistent with the data volume generated in the live broadcast process, then the server screens and recombines the multimedia data transmitted by the plurality of network channels, and each network channel is responsible for transmitting a complete data, so the server can effectively recover the original data pushed by the anchor user according to the data transmitted by each network channel, effectively ensure the integrity of the multimedia data finally output to the audience user side, greatly reduce the problem that the data is seriously lost due to poor network state when the multimedia data is transmitted through a single network channel, further cause poor live broadcast effect, change an expression mode, and ensure the consistency and continuity of the multimedia data output to the watching user side, therefore, the abnormal decoding phenomenon on the watching user side is greatly reduced, the problems of picture playing such as green screen, flower screen and mosaic and the smoothness problems of frame skipping, blocking, incoherence and the like of the screen can be effectively solved, and the live broadcasting effect is better.

In addition, as the terminal of the anchor user sends a plurality of complete multimedia data through a plurality of network channels, the server can effectively recover the original data pushed by the anchor user according to the data transmitted by each network channel, and therefore, the dynamic code rate adaptation is not needed to be carried out to reduce the packet loss rate in order to relieve network congestion during front-end coding, so that the picture definition cannot be reduced by the multimedia data pushing mode, the video image quality is better, and the live broadcast effect is further ensured.

In another embodiment, it is assumed that the data volume sent by each network channel is a part of the data volume generated in the live broadcast process, that is, if a distributed data transmission manner of multiple network channels is adopted, as shown in fig. 8, the data reassembly server, when performing data reassembly on the received multimedia data, includes the following steps:

801. and buffering the multimedia data received through different network channels to different input buffers.

This step is the same as step 501 described above.

802. When the data amount stored in each input buffer is located in the first target interval, data reading is sequentially performed from the buffer starting position of each input buffer.

This step is the same as step 502 described above.

803. And sequentially storing the read data blocks to an output buffer according to the sequence of the sequence numbers from small to large.

Continuing with 2 network channels as an example, referring to fig. 12, assuming that the amount of data generated in the live broadcast process is M, each network channel is responsible for transmitting M/2, for example, the multimedia data uploaded by the network channel 1 includes data blocks with

sequence numbers

1, 3, 5, 7, 9, … …, N-1, N +1, and … …, and the multimedia data uploaded by the network channel 2 includes data blocks with

sequence numbers

2, 4, 6, 8, 10, … …, N, and … …, after the multimedia data transmitted by the network channel 1 is buffered to the input buffer 1 and the multimedia data transmitted by the network channel 2 is buffered to the input buffer 2, data reassembly can be performed according to the sequence numbers from small to large.

For example, data blocks read from the same buffer position of each input buffer can be sequentially stored to the output buffer in the sequence of sequence numbers from small to large. Taking fig. 12 as an example, assuming that the sequence number of the data block stored in the cache start position of the input cache 1 is 1, and the sequence number of the data block stored in the cache start position of the input cache 2 is 2, the data block with the sequence number of 1 is first stored in the output cache, and then the data block with the sequence number of 2 is stored in the output cache.

In the embodiment of the disclosure, the terminal of the anchor user sends multimedia data through a plurality of network channels, and the data volume sent by each network channel is a part of the data volume generated in the live broadcast process, thereby effectively relieving the data uploading pressure of each network channel, greatly reducing the packet loss rate, then, the server screens and recombines the multimedia data transmitted by the plurality of network channels, effectively ensuring the integrity of the multimedia data finally output to the audience user side, greatly reducing the problem of bad live broadcast effect caused by serious data loss due to bad network state when the multimedia data is transmitted through a single network channel, and changing an expression mode, because the consistency and continuity of the multimedia data output to the watching user side are ensured, the abnormal decoding phenomenon at the watching user side is greatly reduced, the problems of picture playing such as green screen, flower screen and mosaic and the problems of smoothness such as frame skipping, blockage and incoherence of the screen can be effectively solved, and the live broadcast effect is better.

In addition, as the terminal of the anchor user sends the multimedia data through the plurality of network channels, and the data volume sent by each network channel is a part of the data volume generated in the live broadcast process, the data uploading pressure of each network channel is effectively relieved, the packet loss rate is greatly reduced, and the dynamic code rate adaptation for relieving network congestion is not needed to reduce the packet loss rate during front-end coding, so the picture definition cannot be reduced by the multimedia data pushing mode, the video image quality is better, and the live broadcast effect is further ensured.

In addition, aiming at a distributed data transmission mode of a plurality of network channels, the bandwidth of the plurality of network channels is utilized, the data volume generated in the live broadcast process is distributed to the plurality of network channels, and the subsequent data recombination servers are utilized to uniformly combine the distributed data into live broadcast streams for transmission again, so that the live broadcast streams with higher quality and higher definition can be transmitted.

In addition, the more the number of network channels used for transmitting multimedia data is, the higher the value of the scheme can be exerted, that is, the more the number of network channels is, the less the data volume averagely allocated to a single network channel for transmission is, so that the scheme is more suitable for data uploading, and the success rate of recovery for packet loss is higher. If the bandwidth of the single channel network is a constant value, which reflects that the data volume that can be carried by the anchor user side can be continuously increased along with the increase of the number of the channels of the network, thereby better expanding the definition of the picture and allowing the video image quality to be shown by more code rates.

Fig. 10 is a flowchart of a multimedia data pushing method according to an embodiment of the present invention. The interactive main body of the method is a first terminal, a live stream receiving server, a data recombination server and a second terminal, wherein the first terminal is a terminal of a main broadcasting user in figure 1, the second terminal is a terminal of a watching user in figure 1, and the live stream receiving server and the data recombination server can be independent servers respectively or can be integrated into one server. When multi-channel data redundancy uploading is adopted, referring to fig. 10, the sorting process of the multimedia data pushing method provided by the embodiment of the present invention may be combed as follows:

A. the first terminal obtains multimedia data generated in a live broadcasting process, and sends the multimedia data to different live broadcasting stream receiving servers through at least two network channels, and the data volume sent by each network channel is consistent with the data volume generated in the live broadcasting process.

The network channels of different operators correspond to the live stream receiving servers of different operators.

B. After receiving the multimedia data, different live stream receiving servers respectively forward the received multimedia data to a data recombination server.

C. After receiving the multimedia data forwarded by different live stream receiving servers, the data recombination server caches the multimedia data received by the different live stream receiving servers to different input caches, wherein the sequence numbers of the data blocks stored at the cache initial positions in each input cache are consistent.

D. And when the data quantity stored in each input buffer is positioned in the first target interval, the data reorganization server sequentially reads data from the buffer starting position of each input buffer.

E. And the data recombination server sequentially judges whether the sequence numbers of the data blocks stored at the same cache position are consistent or not from the cache initial position of each input cache.

F. And when the serial numbers of the data blocks stored at the same cache position are consistent, the data recombination server stores the data block stored in any input cache to the output cache again.

G. And when the sequence numbers of the data blocks stored at the same cache position are inconsistent, determining a target data block with the minimum sequence number, and storing the target data block to an output cache.

H. And the data recombination server pushes the obtained recombined multimedia data to the second terminal.

Fig. 11 is a flowchart of a multimedia data pushing method according to an embodiment of the present invention. The interactive main body of the method is a first terminal, a live stream receiving server, a data recombination server and a second terminal, wherein the first terminal is a terminal of a main broadcasting user in figure 1, the second terminal is a terminal of a watching user in figure 1, and the live stream receiving server and the data recombination server can be independent servers respectively or can be integrated into one server. When multi-channel distributed data uploading is adopted, referring to fig. 11, the sorting process of the multimedia data pushing method provided by the embodiment of the present invention may be combed as follows:

A. the first terminal acquires multimedia data generated in the live broadcasting process, and sends the multimedia data to different live broadcasting stream receiving servers through at least two network channels, and the data volume sent by each network channel is a part of the data volume generated in the live broadcasting process.

E. And the data recombination server sequentially stores the read data blocks to an output cache according to the sequence of the sequence numbers from small to large.

F. And the data recombination server pushes the obtained recombined multimedia data to the second terminal.

In another embodiment, for example, the second terminal refers to a terminal of a viewing user, and since the data reassembly server needs to perform verification and data reassembly on the data blocks stored in the input buffers, the flow rate cannot be real-time, and accordingly, the data reassembly server cannot perform data push based on the data blocks stored in the output buffers, at this time, if the output buffers are used to send data, the problem that the flow rate cannot follow up is encountered, and since the flow rate is not consistent with the play frame rate of the second terminal, video buffering is caused, that is, the data transmission speed is slower than the play speed, and the player of the second terminal can play data without pause.

In order to solve the problem, the data reorganization server does not immediately send the reorganized multimedia data in the output cache, but sends the multimedia data after the data amount of the cache reaches a certain size. For example, after buffering the data amount of the playing time duration 1 to 3s, data transmission is started according to the flow rate, so that the problem of the playing progress of the second terminal is solved. In another expression mode, when the data reassembly server pushes the reassembled multimedia data to the second terminal, the following method is specifically adopted:

and when the data volume stored in the output cache is located in a second target interval, the data reorganization server pushes the data blocks stored in the output cache to the second terminal according to the current live video frame rate.

The first point to be described is that the data reassembly server specifically pushes the multimedia data in the output cache to the CDN server, and then the CDN server pushes the multimedia data to the second terminal.

The second point to be noted is that when the data reassembly server pushes the data block stored in the output buffer to the second terminal according to the video frame rate, the method includes, but is not limited to, taking the following steps to implement:

step 1, acquiring time intervals for sending each stored data block based on a video frame rate.

And 2, after the last data block is sent, sending the next data block after an acquired time interval, wherein the sequence number of the last data block is continuous with the sequence number of the next data block.

The following describes a data push method of the data reorganization server by using a specific example.

As shown in fig. 12, it is assumed that the data reassembly server sets the system clock to start timing from 0 before executing the data transmission task, and for example, the symbol fps refers to the video frame rate, 1/fps is the time interval of one frame, and N/fps is the time interval of N frames; then, the data recombination server sends the first data block stored in the output buffer at the time point of 0, the second data block is sent again after waiting for the system clock to reach the time point of 1/fps, the third data block is sent again after waiting for the system clock to reach the time point of 2/fps, and so on, the Nth data block is sent again after waiting for the system clock to reach the time point of N-1/fps.

It should be noted that, the above method for transmitting the data stored in the output buffer needs to use a constant video frame rate as a premise, and in other words, if the frame rate of the currently live video is a fixed value, for example, the frame rate is constant at 25, the above method can be used to transmit the data. If the frame rate of the currently live video is not fixed, for example, a certain period of time is 25 frame rates, and another period of time is 15 frame rates, the sending time point of each data block stored in the output buffer needs to be determined in another manner, specifically, for this case, the timestamp information may be obtained from the header of each data block, and then each data block is sent according to the corresponding timestamp information, that is, when the system clock reaches the time point matching the timestamp information, the corresponding data block is sent.

Fig. 13 is a schematic structural diagram of a multimedia data pushing apparatus according to an embodiment of the present invention. Referring to fig. 13, the apparatus includes:

a receiving module 1301, configured to receive multimedia data sent by a first terminal through at least two network channels, where the first terminal is configured to send multimedia data generated in a live broadcast process through the at least two network channels;

a processing module 1302, configured to perform data reassembly on the received multimedia data to obtain reassembled multimedia data;

and the first sending module 1303 is configured to push the reassembled multimedia data to a second terminal.

According to the device provided by the embodiment of the invention, after the terminal of the anchor user sends the multimedia data through the plurality of network channels, the integrity of the multimedia data finally output to the audience user side can be effectively ensured by screening and recombining the multimedia data transmitted by the plurality of network channels, the problem of poor live broadcast effect caused by serious data loss due to poor network state when the multimedia data is transmitted through a single network channel is greatly reduced, an expression mode is changed, the abnormal decoding phenomenon at the viewing user side is greatly reduced due to the fact that the consistency and continuity of the multimedia data output to the viewing user side are ensured, the problems of picture playing such as green screen, flower screen and mosaic and the like of a screen can be effectively solved, the problems of frame skipping, blocking, discontinuous smoothness and the like can be effectively solved, and the live broadcast effect is good.

In another embodiment, the data volume sent by each network channel is consistent with the data volume generated in the live broadcast process; or the like, or, alternatively,

the data volume sent by each network channel is a part of the data volume generated in the live broadcast process.

In another embodiment, the multimedia data comprises a plurality of data blocks, each data block comprising a header and a body, the header comprising a sequence number of the data block, and the body comprising picture data and audio data.

In another embodiment, when the data volume sent by each network channel is consistent with the data volume generated in the live broadcasting process, the processing module is further configured to buffer the multimedia data received through different network channels to different input buffers respectively, and the sequence numbers of the data blocks stored at the starting positions of the buffers in each input buffer are consistent; when the data volume stored in the input buffer is positioned in a first target interval, sequentially reading data from the buffer initial position of the input buffer; and performing data reorganization based on the data blocks read in the input buffer.

In another embodiment, the processing module is further configured to sequentially determine, starting from a cache start position of each input cache, whether sequence numbers of data blocks stored at the same cache position are consistent; and when the serial numbers of the data blocks stored at the same cache position are consistent, storing the data block stored in any input cache to an output cache.

In another embodiment, the processing module is further configured to determine a target data block with a smallest sequence number when the sequence numbers of the data blocks stored at the same cache location are inconsistent, and store the target data block to the output cache.

In another embodiment, when the data volume sent by each network channel is a part of the data volume generated in the live broadcasting process, the processing module is further configured to buffer the multimedia data received through different network channels to different input buffers respectively; when the data size stored in the input buffers is located in a first target interval, sequentially reading data from the buffer starting position of each input buffer; and sequentially storing the read data blocks to an output cache according to the sequence of the sequence numbers from small to large.

In another embodiment, the first sending module is further configured to, when the data amount stored in the output buffer is located in a second target interval, push the data block stored in the output buffer to the second terminal according to the frame rate of the current live video.

In another embodiment, the first sending module is further configured to obtain a time interval for sending each stored data block based on the frame rate of the currently live video; after the last data block is sent, sending the next data block after the time interval; wherein the sequence number of the previous data block is consecutive to the sequence number of the next data block.

All the above alternative solutions can be adopted in any combination to form the alternative embodiments of the present disclosure.

Fig. 14 is a schematic structural diagram of a multimedia data pushing apparatus according to an embodiment of the present invention. Referring to fig. 14, the apparatus includes:

an obtaining module 1401, configured to obtain multimedia data generated in a live broadcast process;

the second sending module 1402 is configured to send the multimedia data to a server through at least two network channels, so that the server performs data reassembly on the multimedia data received through the at least two network channels, and pushes the obtained reassembled multimedia data to a second terminal.

The device provided by the embodiment of the invention sends the multimedia data through the plurality of network channels, then the server screens and recombines the multimedia data transmitted by the plurality of network channels, can effectively ensure the integrity of the multimedia data finally output to the user side of audiences, greatly reduces the problem of poor live broadcast effect caused by serious data loss due to poor network state when the multimedia data is transmitted through a single network channel, and changes an expression mode.

In addition, the multimedia data is sent through a plurality of network channels, so that dynamic code rate adaptation is not needed to be carried out to reduce the packet loss rate in order to relieve network congestion during front-end coding, the image definition cannot be reduced by the multimedia data pushing mode, the video quality is better, and the live broadcast effect is further ensured.

In another embodiment, the second sending module is further configured to send, through each network channel, multimedia data that is consistent with a data amount generated in a live broadcast process; or, sending part of multimedia data generated in the live broadcast process through each network channel.

It should be noted that: in the foregoing embodiment, when the multimedia data sending apparatus pushes multimedia data, only the division of the functional modules is used for illustration, and in practical applications, the function distribution may be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above. In addition, the multimedia data sending apparatus provided in the above embodiment and the multimedia data pushing method embodiment belong to the same concept, and the specific implementation process thereof is described in detail in the method embodiment.

Fig. 15 shows a block diagram of a multimedia data pushing apparatus 1500 according to an exemplary embodiment of the present invention. The device 1500 may be a portable mobile terminal such as: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, motion video Experts compression standard Audio Layer 3), an MP4 player (Moving Picture Experts Group Audio Layer IV, motion video Experts compression standard Audio Layer 4), a notebook computer, or a desktop computer. Device 1500 may also be referred to as a user equipment, portable terminal, laptop terminal, desktop terminal, or the like by other names.

In general, the apparatus 1500 includes: a processor 1501 and memory 1502.

Processor 1501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, or the like. The processor 1501 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). Processor 1501 may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in an awake state, and is also referred to as a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 1501 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, processor 1501 may also include an AI (Artificial Intelligence) processor for processing computational operations related to machine learning.

The memory 1502 may include one or more computer-readable storage media, which may be non-transitory. The memory 1502 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in the memory 1502 is used to store at least one instruction for execution by the processor 1501 to implement the multimedia data pushing method provided by the method embodiments of the present application.

In some embodiments, the apparatus 1500 may further optionally include: a peripheral interface 1503 and at least one peripheral. The processor 1501, memory 1502, and peripheral interface 1503 may be connected by buses or signal lines. Various peripheral devices may be connected to peripheral interface 1503 via buses, signal lines, or circuit boards. Specifically, the peripheral device includes: at least one of radio frequency circuitry 1504, touch screen display 1505, camera 1506, audio circuitry 1507, positioning assembly 1508, and power supply 1509.

The peripheral interface 1503 may be used to connect at least one peripheral related to I/O (Input/Output) to the processor 1501 and the memory 1502. In some embodiments, the processor 1501, memory 1502, and peripheral interface 1503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1501, the memory 1502, and the peripheral interface 1503 may be implemented on separate chips or circuit boards, which is not limited in this embodiment.

The Radio Frequency circuit 1504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuitry 1504 communicates with communication networks and other communication devices via electromagnetic signals. The radio frequency circuit 1504 converts an electrical signal into an electromagnetic signal to transmit, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1504 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and so forth. The radio frequency circuit 1504 can communicate with other terminals via at least one wireless communication protocol. The wireless communication protocols include, but are not limited to: the world wide web, metropolitan area networks, intranets, generations of mobile communication networks (2G, 3G, 4G, and 5G), Wireless local area networks, and/or WiFi (Wireless Fidelity) networks. In some embodiments, the radio frequency circuit 1504 may also include NFC (Near Field Communication) related circuits, which are not limited in this application.

The display screen 1505 is used to display a UI (User Interface). The UI may include graphics, text, icons, video, and any combination thereof. When the display screen 1505 is a touch display screen, the display screen 1505 also has the ability to capture touch signals on or over the surface of the display screen 1505. The touch signal may be input to the processor 1501 as a control signal for processing. In this case, the display screen 1505 may also be used to provide virtual buttons and/or a virtual keyboard, also referred to as soft buttons and/or a soft keyboard. In some embodiments, the display 1505 may be one, providing the front panel of the device 1500; in other embodiments, the display 1505 may be at least two, each disposed on a different surface of the device 1500 or in a folded design; in still other embodiments, the display 1505 may be a flexible display disposed on a curved surface or a folded surface of the device 1500. Even further, the display 1505 may be configured in a non-rectangular irregular pattern, i.e., a shaped screen. The Display 1505 can be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), and other materials.

The camera assembly 1506 is used to capture images or video. Optionally, the camera assembly 1506 includes a front camera and a rear camera. Generally, a front camera is disposed at a front panel of the terminal, and a rear camera is disposed at a rear surface of the terminal. In some embodiments, the number of the rear cameras is at least two, and each rear camera is any one of a main camera, a depth-of-field camera, a wide-angle camera and a telephoto camera, so that the main camera and the depth-of-field camera are fused to realize a background blurring function, and the main camera and the wide-angle camera are fused to realize panoramic shooting and VR (Virtual Reality) shooting functions or other fusion shooting functions. In some embodiments, camera assembly 1506 may also include a flash. The flash lamp can be a monochrome temperature flash lamp or a bicolor temperature flash lamp. The double-color-temperature flash lamp is a combination of a warm-light flash lamp and a cold-light flash lamp, and can be used for light compensation at different color temperatures.

The audio circuitry 1507 may include a microphone and speaker. The microphone is used for collecting sound waves of a user and the environment, converting the sound waves into electric signals, and inputting the electric signals to the processor 1501 for processing or inputting the electric signals to the radio frequency circuit 1504 to realize voice communication. The microphones may be multiple and placed at different locations on the device 1500 for stereo sound capture or noise reduction purposes. The microphone may also be an array microphone or an omni-directional pick-up microphone. The speaker is then used to convert electrical signals from the processor 1501 or the radio frequency circuitry 1504 into sound waves. The loudspeaker can be a traditional film loudspeaker or a piezoelectric ceramic loudspeaker. When the speaker is a piezoelectric ceramic speaker, the speaker can be used for purposes such as converting an electric signal into a sound wave audible to a human being, or converting an electric signal into a sound wave inaudible to a human being to measure a distance. In some embodiments, the audio circuitry 1507 may also include a headphone jack.

The positioning component 1508 is operable to locate a current geographic Location of the device 1500 to implement navigation or LBS (Location Based Service). The Positioning component 1508 can be a Positioning component based on the Global Positioning System (GPS) in the united states, the beidou System in china, or the galileo System in russia.

The power supply 1509 is used to power the various components in the device 1500. The power supply 1509 may be alternating current, direct current, disposable or rechargeable. When the power supply 1509 includes a rechargeable battery, the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery. The wired rechargeable battery is a battery charged through a wired line, and the wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery may also be used to support fast charge technology.

In some embodiments, the device 1500 also includes one or more sensors 1510. The one or more sensors 1510 include, but are not limited to: acceleration sensor 1511, gyro sensor 1512, pressure sensor 1513, fingerprint sensor 1514, optical sensor 1515, and proximity sensor 1516.

The acceleration sensor 1511 can detect the magnitude of acceleration on three coordinate axes of the coordinate system established with the apparatus 1500. For example, the acceleration sensor 1511 may be used to detect components of the gravitational acceleration in three coordinate axes. The processor 1501 may control the touch screen display 1505 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 1511. The acceleration sensor 1511 may also be used for acquisition of motion data of a game or a user.

The gyro sensor 1512 may detect a body direction and a rotation angle of the device 1500, and the gyro sensor 1512 and the acceleration sensor 1511 may cooperate to collect a 3D motion of the user on the device 1500. The processor 1501 may implement the following functions according to the data collected by the gyro sensor 1512: motion sensing (such as changing the UI according to a user's tilting operation), image stabilization at the time of photographing, game control, and inertial navigation.

Pressure sensor 1513 may be disposed on a side bezel of device 1500 and/or underneath touch screen 1505. When the pressure sensor 1513 is disposed on the side frame of the device 1500, the holding signal of the user to the device 1500 may be detected, and the processor 1501 performs left-right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 1513. When the pressure sensor 1513 is disposed at a lower layer of the touch display 1505, the processor 1501 controls the operability control on the UI interface according to the pressure operation of the user on the touch display 1505. The operability control comprises at least one of a button control, a scroll bar control, an icon control and a menu control.

The fingerprint sensor 1514 is configured to capture a fingerprint of the user, and the processor 1501 identifies the user based on the fingerprint captured by the fingerprint sensor 1514, or the fingerprint sensor 1514 identifies the user based on the captured fingerprint. Upon recognizing that the user's identity is a trusted identity, the processor 1501 authorizes the user to perform relevant sensitive operations including unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings, etc. The fingerprint sensor 1514 may be disposed on the front, back, or side of the device 1500. When a physical key or vendor Logo is provided on device 1500, fingerprint sensor 1514 may be integrated with the physical key or vendor Logo.

The optical sensor 1515 is used to collect ambient light intensity. In one embodiment, processor 1501 may control the brightness of the display on touch screen 1505 based on the intensity of ambient light collected by optical sensor 1515. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 1505 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 1505 is turned down. In another embodiment, the processor 1501 may also dynamically adjust the shooting parameters of the camera assembly 1506 according to the ambient light intensity collected by the optical sensor 1515.

A proximity sensor 1516, also known as a distance sensor, is typically provided on the front panel of the device 1500. The proximity sensor 1516 is used to capture the distance between the user and the front of the device 1500. In one embodiment, the processor 1501 controls the touch display 1505 to switch from a bright screen state to a dark screen state when the proximity sensor 1516 detects that the distance between the user and the front of the device 1500 is gradually decreasing; when the proximity sensor 1516 detects that the distance between the user and the front of the device 1500 is gradually increasing, the processor 1501 controls the touch display 1505 to switch from a breath screen state to a bright screen state.

Those skilled in the art will appreciate that the configuration shown in FIG. 15 is not limiting of the apparatus 1500, and may include more or fewer components than shown, or some components may be combined, or a different arrangement of components may be used.

Fig. 16 is a schematic structural diagram of a multimedia data pushing apparatus according to an embodiment of the present invention, where the server 1600 may generate a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 1601 and one or more memories 1602, where the memory 1602 stores at least one instruction, and the at least one instruction is loaded and executed by the processors 1601 to implement the multimedia data pushing method provided by the foregoing method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input/output interface for inputting and outputting, and the server may also include other components for implementing the functions of the device.

In an exemplary embodiment, a computer-readable storage medium, such as a memory, including instructions executable by a processor in a terminal to perform the multimedia data pushing method in the above-described embodiments is also provided. For example, the computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.

Claims

1. A multimedia data pushing method applied to a server, the method comprising:

receiving multimedia data which is sent by a first terminal through at least two network channels and comprises a plurality of data blocks, wherein the first terminal is used for sending the multimedia data generated in the live broadcasting process through the at least two network channels, the data blocks comprise headers and bodies, the headers comprise serial numbers of the data blocks, the bodies comprise picture data and audio data, and the quantity of the picture data and the audio data is determined according to the video frame rate of the current live broadcasting;

when the data volume sent by each network channel is consistent with the data volume generated in the live broadcast process, respectively caching the multimedia data received through different network channels into different input caches, wherein the serial numbers of the data blocks stored at the initial cache positions in each input cache are consistent;

when the data size stored in the input buffer is located in a first target interval, sequentially reading data from the buffer starting position of the input buffer, wherein the data reading is performed on different input buffers in parallel;

sequentially judging whether the sequence numbers of the data blocks stored at the same cache position are consistent or not from the cache initial position of each input cache; when the serial numbers of the data blocks stored at the same cache position are inconsistent, determining a target data block with the minimum serial number, and storing the target data block to an output cache to obtain recombined multimedia data, wherein the data block stored at the cache initial position of the output cache is the data block stored at the cache initial position of any input cache;

when the current live video frame rate is a fixed value and the data volume stored in the output buffer is located in a second target interval, acquiring a time interval for sending each stored data block based on the current live video frame rate; after the time interval elapses after the last data block is sent to the second terminal, sending the next data block to the second terminal; wherein the sequence number of the previous data block is consecutive to the sequence number of the next data block;

when the current live video frame rate is not a fixed value and the data volume stored in the output buffer is located in the second target interval, acquiring timestamp information from the head of each data block; and when the system clock reaches a time point matched with any time stamp information, sending a data block corresponding to any time stamp information.

2. The method of claim 1, further comprising:

and when the serial numbers of the data blocks stored at the same cache position are consistent, storing the data block stored in any input cache to the output cache.

3. The method of claim 1, wherein when the amount of data transmitted by each network channel is a portion of the amount of data generated during the live broadcast, the method further comprises:

respectively caching the multimedia data received through different network channels to different input caches;

when the data size stored in the input buffers is located in the first target interval, sequentially reading data from the buffer starting position of each input buffer;

and sequentially storing the read data blocks to the output cache according to the sequence of the sequence numbers from small to large.

4. A multimedia data pushing method is applied to a terminal, and the method comprises the following steps:

the method comprises the steps of obtaining multimedia data which are generated in a live broadcast process and comprise a plurality of data blocks, wherein the data blocks comprise headers and bodies, the headers comprise serial numbers of the data blocks, the bodies comprise picture data and audio data, and the quantity of the picture data and the audio data is determined according to the video frame rate of the current live broadcast;

transmitting the multimedia data to a server through each of at least two network channels;

the server is used for caching the multimedia data received through different network channels to different input caches respectively when the data volume sent by each network channel is consistent with the data volume generated in the live broadcast process, and the serial numbers of the data blocks stored at the cache initial positions in each input cache are consistent; when the data size stored in the input buffer is located in a first target interval, sequentially reading data from the buffer starting position of the input buffer, wherein the data reading is performed on different input buffers in parallel; sequentially judging whether the serial numbers of the data blocks stored at the same cache position are consistent or not from the cache initial position of each input cache; when the serial numbers of the data blocks stored at the same cache position are inconsistent, determining a target data block with the minimum serial number, and storing the target data block to an output cache to obtain recombined multimedia data, wherein the data block stored at the cache initial position of the output cache is the data block stored at the cache initial position of any input cache;

the server is further configured to acquire a time interval for sending each stored data block based on the current live video frame rate when the current live video frame rate is a fixed value and the data amount stored in the output cache is located in a second target interval; after the time interval elapses after the last data block is sent to the second terminal, sending the next data block to the second terminal; wherein the sequence number of the previous data block is consecutive to the sequence number of the next data block;

the server is further used for acquiring timestamp information from the head of each data block when the current live video frame rate is not a fixed value and the data volume stored in the output cache is located in the second target interval; and when the system clock reaches a time point matched with any time stamp information, sending a data block corresponding to any time stamp information.

5. A multimedia data pushing apparatus, applied to a server, the apparatus comprising:

the system comprises a receiving module, a processing module and a processing module, wherein the receiving module is used for receiving multimedia data which is sent by a first terminal through at least two network channels and comprises a plurality of data blocks, the first terminal is used for sending out the multimedia data generated in a live broadcast process through the at least two network channels, the data blocks comprise a head part and a main body, the head part comprises a serial number of the data block, the main body comprises picture data and audio data, and the quantity of the picture data and the audio data is determined according to the video frame rate of the current live broadcast;

the processing module is used for respectively caching the multimedia data received through different network channels to different input caches when the data volume sent by each network channel is consistent with the data volume generated in the live broadcast process, and the serial numbers of the data blocks stored at the initial cache positions in each input cache are consistent; when the data size stored in the input buffer is located in a first target interval, sequentially reading data from the buffer starting position of the input buffer, wherein the data reading is performed on different input buffers in parallel; sequentially judging whether the serial numbers of the data blocks stored at the same cache position are consistent or not from the cache initial position of each input cache; when the serial numbers of the data blocks stored at the same cache position are inconsistent, determining a target data block with the minimum serial number, and storing the target data block to an output cache to obtain recombined multimedia data, wherein the data block stored at the cache initial position of the output cache is the data block stored at the cache initial position of any input cache;

a first sending module, configured to, when the current live video frame rate is a fixed value and the data amount stored in the output buffer is located in a second target interval, obtain, based on the current live video frame rate, a time interval for sending each stored data block; after the time interval elapses after the last data block is sent to the second terminal, sending the next data block to the second terminal; wherein the sequence number of the previous data block is consecutive to the sequence number of the next data block;

the first sending module is further configured to obtain timestamp information from a header of each data block when the current live video frame rate is not a fixed value and the data amount stored in the output buffer is located in the second target interval; and when the system clock reaches a time point matched with any one piece of time stamp information, sending a data block corresponding to the any one piece of time stamp information.

6. A multimedia data pushing apparatus, applied to a terminal, the apparatus comprising:

the device comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring multimedia data which comprises a plurality of data blocks and is generated in a live broadcast process, the data blocks comprise headers and bodies, the headers comprise serial numbers of the data blocks, and the bodies comprise picture data and audio data, wherein the number of the picture data and the audio data is determined according to the video frame rate of the current live broadcast;

the second sending module is used for sending the multimedia data to the server through each of at least two network channels;

the server is further used for acquiring timestamp information from the head of each data block when the current live video frame rate is not a fixed value and the data volume stored in the output cache is located in the second target interval; and when the system clock reaches a time point matched with any one piece of time stamp information, sending a data block corresponding to the any one piece of time stamp information.

7. A storage medium having stored therein at least one instruction, which is loaded and executed by a processor to implement the multimedia data pushing method according to any one of claims 1 to 3 or the multimedia data pushing method according to claim 4.

8. A multimedia data push device, characterized in that the device comprises a processor and a memory, wherein at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the multimedia data push method according to any one of claims 1 to 3, or the multimedia data push method according to claim 4.