CN111510720A - Real-time streaming media data transmission method, electronic device and server - Google Patents
Real-time streaming media data transmission method, electronic device and server Download PDFInfo
- Publication number
- CN111510720A CN111510720A CN202010334661.3A CN202010334661A CN111510720A CN 111510720 A CN111510720 A CN 111510720A CN 202010334661 A CN202010334661 A CN 202010334661A CN 111510720 A CN111510720 A CN 111510720A
- Authority
- CN
- China
- Prior art keywords
- code stream
- layer code
- stream data
- data
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/70—Media network packetisation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
- H04L65/762—Media network packet handling at the source
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
- H04L65/765—Media network packet handling intermediate
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440227—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by decomposing into layers, e.g. base layer and one or more enhancement layers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
- H04N21/462—Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities
- H04N21/4621—Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen
Abstract
The invention provides a transmission method of real-time streaming media data, which comprises the steps of carrying out optimization processing on the collected real-time streaming media data; performing layered coding on the optimized real-time streaming media data to generate basic layer code stream data and extended layer code stream data; packaging the generated basic layer code stream data and the generated extended layer code stream data and pushing the packaged basic layer code stream data and the generated extended layer code stream data to a server; the server receives basic layer code stream data and extended layer code stream data which are packaged and pushed by the electronic device; decapsulating the encapsulated base layer code stream data and the encapsulated extension layer code stream data; and sending the basic layer code stream data or the expansion layer code stream data to the user side for decoding and playing according to the network environment when the user side pulls the stream. The invention also provides an electronic device and a server. The invention adopts layered coding to code audio and video frame data, and ensures low delay of network-based live video playing and fluency under different network environments.
Description
Technical Field
The present invention relates to the field of streaming media data processing technologies based on networks, and in particular, to a method, an electronic device, and a server for transmitting real-time streaming media data.
Background
With the development of science and technology, the application of the internet-based streaming media live broadcast is more and more extensive. Streaming media refers to media formats such as audio, video, or multimedia files that are played continuously in real-time over a network using streaming technology. The live streaming of the streaming media refers to the transmission of the real-time streaming media, and the live streaming media is characterized in that a main broadcasting end locally encodes and encapsulates the acquired audio and video original data, then the audio and video original data is pushed to a server end in real time, and a spectator end performs stream pulling, decapsulation, decoding and rendering from the server in real time and then watches the audio and video original data. Today, many video playing platforms transcode video files into videos with different definitions by using a transcoding technology, so that video playing options with different definitions are provided. However, since video transcoding takes a long time, live broadcast delay is increased, which affects live broadcast effects of many live broadcast scenes, such as live broadcast effect of live broadcast interaction, and low delay of live broadcast video broadcast cannot be guaranteed while video broadcast options with different definitions are provided.
Disclosure of Invention
In view of the above, there is a need to provide a method, an electronic device and a server for transmitting real-time streaming media data, in which the real-time streaming media data is layered encoded by layered encoding, streaming media data with different definitions can be provided according to a network environment of a user end, and the streaming media data is decoded and played at the user end.
A first aspect of the present invention provides a method for transmitting real-time streaming media data, where the method includes:
optimizing real-time streaming media data acquired by an electronic device;
performing layered coding on the optimized real-time streaming media data to generate basic layer code stream data and extended layer code stream data;
packaging the generated basic layer code stream data and the generated extended layer code stream data, and pushing the basic layer code stream data and the extended layer code stream data to a server;
the server receives basic layer code stream data and extended layer code stream data which are packaged and pushed by the electronic device;
decapsulating the encapsulated base layer code stream data and the encapsulated extension layer code stream data; and
and sending the basic layer code stream data or the expansion layer code stream data to the user side for decoding and playing according to the network environment of the user side when the user side pulls the stream.
Preferably, the step of "performing optimization processing on the real-time streaming media data collected by the electronic device" includes:
detecting whether a video in the real-time streaming media data contains a human face;
when the video in the real-time streaming media data contains a face, performing facial beautification on the face;
responding to user operation to carry out special effect processing on the video of the real-time streaming media data; and
and carrying out noise removal and volume adjustment processing on the audio frequency in the real-time streaming media data.
Preferably, the step of performing layered coding on the real-time streaming media data collected by the electronic device to generate the base layer code stream data and the extended layer code stream data specifically includes:
extracting high-quality audio and/or video frame data and low-quality audio and/or video frame data from the collected real-time streaming media data;
coding high-quality audio and/or video frame data to generate the base layer code stream data; and
and coding the low-quality audio and/or video frame data to generate the extended layer code stream data.
Preferably, the base layer code stream data includes high-quality audio and/or video frame data, the extension layer code stream data includes low-quality audio and/or video frame data, and the network bandwidth required for transmitting the extension layer code stream data to the user side is a preset percentage of the network bandwidth required for transmitting the base layer code stream data to the user side.
Preferably, the step of sending the base layer code stream data or the extension layer code stream data to the user side for decoding and playing according to the network environment of the user side when the user side pulls the stream includes:
judging whether the downlink network bandwidth of the user side is greater than or equal to a preset value;
when the downlink network bandwidth of the user side is judged to be larger than or equal to the preset value, the basic layer code stream data is sent to the user side for decoding and playing when the user side pulls the stream; and
and when the downlink network bandwidth of the user side is judged to be smaller than the preset value, the extended layer code stream data is sent to the user side for decoding and playing when the user side pulls the stream.
A second aspect of the present invention provides an electronic device communicatively connected to at least one server, the electronic device comprising:
the acquisition module is used for acquiring streaming media data in real time;
the optimization module is used for optimizing the acquired real-time streaming media data;
the encoding module is used for carrying out layered encoding on the real-time streaming media data subjected to the optimization processing to generate basic layer code stream data and extended layer code stream data;
the packaging module is used for packaging the generated basic layer code stream data and the generated extended layer code stream data; and
and the flow pushing module is used for pushing the packaged basic layer code stream data and the packaged expansion layer code stream data to the server, and the server decapsulates the basic layer code stream data and the expansion layer code stream data which are pushed by the flow pushing module and sends the basic layer code stream data or the expansion layer code stream data to the user side for decoding and playing according to the network environment of the user side when the user side pulls the flow.
Preferably, the encoding module is configured to extract high-quality audio and/or video frame data and low-quality audio and/or video frame data from the collected real-time streaming media data, encode the high-quality audio and/or video frame data to generate the base layer stream data, and encode the low-quality audio and/or video frame data to generate the extension layer stream data.
A third aspect of the present invention provides a server for connecting to an electronic device and at least one client, the server comprising:
the receiving module is used for receiving the basic layer code stream data and the extended layer code stream data which are packaged and pushed by the electronic device, wherein the basic layer code stream data and the extended layer code stream data are produced by layering and coding the acquired and optimized real-time streaming media data through the electronic device;
the decapsulation module is used for decapsulating the base layer code stream data and the extension layer code stream data; and
and the sending module is used for sending the basic layer code stream data or the expansion layer code stream data to the user side for decoding and playing according to the network environment of the user side when the user side pulls the stream.
Preferably, the base layer code stream data includes high-quality audio and/or video frame data, the extension layer code stream data includes low-quality audio and/or video frame data, and the network bandwidth required by the sending module to send the extension layer code stream data to the user side is a preset percentage of the network bandwidth required by the sending module to send the base layer code stream data to the user side.
According to the transmission method, the electronic device and the server of the real-time streaming media data, the real-time streaming media data are layered and coded, the streaming media data with different qualities are provided for a user side to decode and play, video playing selection with different definitions is provided, low delay of live video playing and fluency under different network environments are guaranteed, and user experience is effectively improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a schematic diagram of an application environment architecture of a transmission method of real-time streaming media data according to a preferred embodiment of the invention.
Fig. 2 is a schematic structural diagram of an electronic device according to a preferred embodiment of the invention.
Fig. 3 is a schematic structural diagram of a server according to a preferred embodiment of the present invention.
Fig. 4 is a schematic structural diagram of a transmission system of real-time streaming media data according to a preferred embodiment of the invention.
Fig. 5 is a flowchart of a method for transmitting real-time streaming media data according to a preferred embodiment of the invention.
Description of the main elements
Microphone 15
Encoding module 103
Receiving module 106
Sending module 108
Judging module 109
Steps S501-S507
The following detailed description will further illustrate the invention in conjunction with the above-described figures.
Detailed Description
In order that the above objects, features and advantages of the present invention can be more clearly understood, a detailed description of the present invention will be given below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict.
In the following description, numerous specific details are set forth to provide a thorough understanding of the present invention, and the described embodiments are merely a subset of the embodiments of the present invention, rather than a complete embodiment. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
Fig. 1 is a schematic diagram of an application environment architecture of a transmission method of real-time streaming media data according to a preferred embodiment of the invention.
The transmission method of the real-time streaming media data is applied to an operating environment formed by an electronic device 1, a server 2 and at least one user terminal 3, wherein the electronic device 1 is in communication connection with the server 2 through a network, and the server 2 is in communication connection with the at least one user terminal 3 through the network. The network may be a wired network or a wireless network, such as the internet, WI-FI, cellular, satellite, etc.
The electronic device 1 and the user side 3 may be a smart phone, a personal computer, or the like, and the server 2 may be a single server, a server cluster, a cloud server, or the like.
Fig. 2 is a schematic structural diagram of an electronic device according to a preferred embodiment of the invention.
The electronic device 1 includes, but is not limited to, a first processor 11, a first memory 12, a first computer program 13 stored in the first memory 12 and executable on the first processor 11, a camera 14, a microphone 15, an audio encoder 16, and a video encoder 17. Wherein the first computer program 13 may be a real-time streaming media data transmission program. The first processor 11 implements steps in a transmission method of real-time streaming media data, such as steps S501-S505 shown in fig. 5, when executing the first computer program 13. Alternatively, the first processor 11 implements the functions of each module/unit in the transmission system of real-time streaming media when executing the first computer program 13, for example, the module 101 and 105 in fig. 4.
Illustratively, the first computer program 13 may be divided into one or more modules/units, which are stored in the first memory 12 and executed by the first processor 11. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution of the first computer program 13 in the electronic device 1. For example, the first computer program 13 may be divided into an acquisition module 101, an optimization module 102, an encoding module 103, an encapsulation module 104, a plug flow module 105 in fig. 4. The specific functions of each module refer to the functions of each module in the embodiment of the transmission system of the real-time streaming media data.
Fig. 3 is a schematic structural diagram of a server according to a preferred embodiment of the present invention.
The server 2 includes, but is not limited to, a second processor 21, a second memory 22, and a second computer program 23 stored in the second memory 22 and executable on the second processor 21, such as a real-time streaming media data transmission program. The second processor 21 implements steps in the transmission method of real-time streaming media data, such as steps S506-S507 shown in fig. 5, when executing the second computer program 23. Alternatively, the second processor 21 implements the functions of each module/unit in the transmission system of real-time streaming media data when executing the second computer program 23, such as the module 106 and 109 in fig. 4.
Illustratively, the second computer program 23 may be divided into one or more modules/units, which are stored in the second memory 22 and executed by the second processor 21. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the second computer program 23 in the server 2. For example, the second computer program 23 may be divided into the receiving module 106, the decapsulating module 107, the sending module 108 and the determining module 109 in fig. 4. The specific functions of each module refer to the functions of each module in the embodiment of the transmission system of the real-time streaming media data.
It will be understood by those skilled in the art that the schematic diagrams are merely examples of the electronic apparatus 1 and the server 2, and do not constitute a limitation to the electronic apparatus 1 and the server 2, and may include more or less components than those shown, or combine some components, or different components, for example, the electronic apparatus 1 and the server 2 may further include an input and output device, a network access device, a bus, etc.
The first Processor 11 and the second Processor 21 may be a Central Processing Unit (CPU), other general-purpose processors, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. The general purpose processor may be a microprocessor, or the first processor 11 and the second processor 21 may be any conventional processors, etc., the first processor 11 is a control center of the electronic apparatus 1 and is connected to each part of the whole electronic apparatus 1 by various interfaces and lines, and the second processor 21 is a control center of the server 2 and is connected to each part of the whole server 2 by various interfaces and lines.
The first memory 12 may be used for storing the first computer program 13 and/or modules/units, and the first processor 11 may implement various functions of the electronic device 1 by running or executing the computer program and/or modules/units stored in the first memory 12 and calling data stored in the first memory 12. The second memory 22 may be used to store the second computer program 23 and/or modules/units, and the second processor 21 implements various functions of the server 2 by running or executing the computer program and/or modules/units stored in the second memory 22 and calling data stored in the second memory 22.
The first memory 12 and the second memory 22 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required by at least one function, and the like; the storage data area may store data created according to the use, and the like. In addition, the first memory 12 and the second memory 22 may include high speed random access memory, and may also include nonvolatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.
Please refer to fig. 4, which is a functional block diagram of a preferred embodiment of a system for transmitting real-time streaming media data according to the present invention.
In some embodiments, the transmission system 100 for real-time streaming media data operates in an operating environment formed by the electronic device 1, the server 2 and the client 3. The transmission system 100 of real-time streaming media data may comprise a plurality of functional modules consisting of program code segments. The program codes of the program segments in the transmission system 100 for real-time streaming media data may be stored in the first memory 12 and the second memory 22, and executed by the at least one first processor 11 and the at least one second processor 21 to implement the transmission function of real-time streaming media.
In this embodiment, the transmission system 100 of real-time streaming media data may be divided into a plurality of functional modules according to the functions performed by the transmission system. Referring to fig. 3, the functional modules may include an acquisition module 101, an optimization module 102, an encoding module 103, an encapsulation module 104, a stream pushing module 105, a receiving module 106, a decapsulation module 107, a sending module 108, and a determining module 109. The module referred to in the present invention is a series of computer program segments capable of being executed by at least one of the first processor 11 and the second processor 21 and performing a fixed function, and is stored in the first memory 12 and the second memory 22. It is understood that, in other embodiments, the modules may also be program instructions or firmware (firmware) solidified in the first processor 11 and the second processor 21.
The acquisition module 101 is configured to acquire streaming media data in real time.
In this embodiment, the electronic device 1 is installed with a live application, and when a user of the electronic device 1 has a live request, the live application can be opened to establish a communication connection with the server 2.
In this embodiment, the streaming media data includes video and/or audio. In the live broadcasting process, the electronic device 1 acquires a video of a user or a scene where the electronic device 1 is located in real time through the camera 14, and acquires an audio of the user or the scene where the electronic device 1 is located in real time through the microphone 15.
The optimization module 102 is configured to perform optimization processing on the real-time streaming media data acquired by the electronic device 1.
In this embodiment, the optimization module 102 detects whether a video in the real-time streaming media data includes a face, and performs a facial beautification process on the face when the video in the real-time streaming media data includes the face.
Further, the optimization module 102 performs special effect processing on the video of the real-time streaming media data in response to a user operation. Specifically, the live application program at least provides a plurality of character special effects and a plurality of environment special effects, and a user of the electronic device 1 can change the character image by selecting the character special effects and change the environment characteristics by selecting the environment special effects. In this way, the optimization module 102 performs corresponding special effect processing on the video of the real-time streaming media data in response to the character special effect and/or the environment special effect selected by the user.
Further, the optimization module 102 performs noise removal and volume adjustment on the audio in the real-time streaming media data. Specifically, a volume range is prestored in the live broadcast application program. The optimization module 102 determines whether the volume of the audio in the real-time streaming media data is within the volume range, and when it is determined that the volume of the audio is not within the volume range, the volume of the audio is made to fall within the volume range by increasing the volume or decreasing the volume.
The encoding module 103 is configured to perform layered encoding on the acquired real-time streaming media data to generate base layer code stream data and extension layer code stream data.
In this embodiment, the encoding module 103 extracts high-quality audio and/or video frame data and low-quality audio and/or video frame data from the collected streaming media data. The high-quality audio and/or video frame data has a higher bit rate, frame rate and resolution, and the low-quality audio and/or video frame data is complementary to the high-quality audio and/or video frame data and has a lower bit rate, frame rate and resolution.
The encoding module 103 controls the audio encoder 16 to encode high-quality audio frame data, and/or controls the video encoder 17 to encode high-quality video frame data, so as to generate the base layer code stream data. The encoding module 103 further controls the audio encoder 16 to encode low-quality audio frame data, and/or controls the video encoder 17 to encode low-quality video frame data, so as to generate the extended layer code stream data. In the present embodiment, the audio encoder 16 performs audio encoding using a faac encoding method, and the video encoder 17 performs video encoding using an H264 encoding method.
In this embodiment, the base layer bitstream data includes high-quality audio and/or video frame data, for example, a video resolution of 640 × 480. The extended layer bitstream data includes low quality audio and/or video frame data, e.g., video resolution of 320 × 240.
In this embodiment, the network bandwidth required for transmitting the extended layer code stream data is a preset percentage of the network bandwidth required for transmitting the base layer code stream data. Wherein said preset percentage is between 10% and 20%, preferably 20%. For example, the network bandwidth required for transmitting the base layer code stream data is 4Mb/s, and the network bandwidth required for transmitting the extended layer code stream data is 20% of the network bandwidth required for transmitting the base layer code stream data and is 0.8 Mb/s.
The encapsulation module 104 is configured to encapsulate the generated base layer code stream data and extended layer code stream data.
In this embodiment, the encapsulation module 104 encapsulates the generated base layer code stream data and extended layer code stream data to form a data packet of an encapsulation protocol. Preferably, the encapsulation module 104 encapsulates the audio frame data and the video frame data in the base layer code stream data into a base layer code stream data packet in mp4 format, encapsulates the audio frame data and the video frame data in the extended layer code stream data into an extended layer code stream data packet in mp4 format, and encapsulates the base layer code stream data packet and the extended layer code stream data packet into a data packet in mp4 format.
The plug-flow module 105 is configured to plug-flow the encapsulated base layer code stream data and extended layer code stream data to the server 2.
In this embodiment, the stream pushing module 105 uploads the encapsulated base layer code stream data and extended layer code stream data to the electronic device 1 based on a UDP (User datagram protocol) private protocol, so as to implement real-time transmission of streaming media data.
The receiving module 106 is configured to receive the basic layer code stream data and the extended layer code stream data that are encapsulated and pushed by the electronic device 1.
The decapsulation module 107 is configured to decapsulate the encapsulated base layer code stream data and extended layer code stream data.
In this embodiment, the decapsulation module 107 disassembles the data packet with the encapsulation protocol, and processes information in the data packet header, thereby obtaining the base layer code stream data and the extended layer code stream data therein.
The sending module 108 is configured to send the base layer code stream data or the extension layer code stream data to the user side 3 for decoding and playing according to the network environment of the user side 3 when the user side 3 pulls the stream.
In this embodiment, the user side 3 is installed with a live client application. When the user of the user side 3 has a requirement for watching live broadcast, the live broadcast client application program can be started, and a live broadcast channel corresponding to the electronic device 1 can be selected. At this time, the user side 3 establishes a communication connection with the server 2 to pull a stream, and requests a code stream from the server 2 through the live client application program. In other embodiments, the user terminal 3 may also access a live website to request a code stream from the server 2 in a web-based manner.
In this embodiment, when the user side 3 pulls a stream, the current downlink network bandwidth is sent to the server 2. When the server 2 receives the downlink network bandwidth of the user side 3, the determining module 109 determines whether the downlink network bandwidth of the user side 3 is greater than or equal to a preset value. When the determining module 109 determines that the downlink network bandwidth of the user side 3 is greater than or equal to the preset value, the sending module 108 sends the base layer code stream data to the user side 3 for decoding and playing when the user side 3 pulls the stream. When the determining module 109 determines that the downlink network bandwidth of the user side 3 is smaller than the preset value, the sending module 108 sends the extended layer code stream data to the user side 3 for decoding and playing when the user side 3 pulls the stream. In the present embodiment, the preset value is 4 Mb/s. In other embodiments, the preset value may also be set to other suitable values according to requirements.
In another embodiment, the live client application of the client 3 provides a high-quality play option and a low-quality play option, and a user can select a play option in a self-line manner according to the downstream network bandwidth of the network environment where the client 3 is currently located to view live streaming media with corresponding quality. The sending module 108 sends the base layer code stream data or the extended layer code stream data to the user side 3 for decoding and playing according to the selection of the user.
Specifically, when the user selects a high-quality playing option when playing the live streaming media at the user terminal 3, the sending module 108 sends the base layer code stream data to the user terminal 3 for decoding and playing. When the user selects a low-quality playing option when playing the live streaming media at the user terminal 3, the sending module 108 sends the extended layer code stream data to the user terminal 3 for decoding and playing.
And the user end 3 decodes the received base layer code stream data or the received expansion layer code stream data to obtain high-quality real-time audio and video or low-quality real-time audio and video, and plays the high-quality real-time audio and video. And the user side 3 receives the base layer code stream data and performs decoding and playing when the downlink network bandwidth is higher, so that a user can watch a video with higher definition through the user side 3. And the user side 3 receives the extended layer code stream data and performs decoding playing when the downlink network bandwidth is low, so that a user can watch a video with low definition but smooth playing through the user side 3. Therefore, the user side 3 can play the live broadcast streaming media suitable for the current network environment through a web browser or a live broadcast client application program, so that network blockage and delay are avoided, and user experience is effectively improved.
In this embodiment, the sending module 108 sends the base layer code stream data or the extended layer code stream data to the user side 3 based on the UDP private protocol, so as to implement real-time transmission of streaming media data.
Further, when the electronic device 1 acquires real-time streaming media data of a live broadcast scene, the acquisition module 101 controls the camera 14 to shoot images of the live broadcast scene at preset intervals, the stream pushing module 105 further transmits the shot images of the live broadcast scene to the server 2 along with the base layer code stream data and the extended layer code stream data, and the sending module 108 further sets the images of the live broadcast scene as cover pictures of live broadcast channels corresponding to the live broadcast scene at preset intervals. In this manner, a user may learn about the live content of the live channel prior to entering the live channel. In this embodiment, the predetermined time is half an hour.
Further, the transmission system 100 for real-time streaming media data can also be applied to an application scenario of multi-person and multi-microphone interaction. The server 2 may simultaneously receive streaming media data streamed in real time by the plurality of electronic devices 1 based on the UDP private protocol. The data pushed by each electronic device 1 includes base layer code stream data and extended layer code stream data generated by layered coding. The server 2 performs mixed flow processing on the received streaming media data from the plurality of electronic devices 1, including mixing audio frame data and synthesizing video pictures, so as to generate integrated base layer code stream data and extended layer code stream data.
Fig. 5 is a flowchart illustrating a method for transmitting real-time streaming media data according to the present invention. The order of the steps in the flow chart may be changed and some steps may be omitted according to different needs.
Step S501, collecting streaming media data in real time.
And step S502, optimizing the collected real-time streaming media data.
Step S503, the collected real-time streaming media data is layered coded to generate basic layer code stream data and extended layer code stream data.
Step S504, the generated base layer code stream data and extended layer code stream data are encapsulated and pushed to the server 2.
Step S505 is to receive the basic layer code stream data and the extended layer code stream data that are encapsulated and pushed by the electronic device 1.
Step S506, decapsulate the encapsulated base layer code stream data and extended layer code stream data.
Step S507, when the user side 3 pulls the stream, the base layer code stream data or the extension layer code stream data is sent to the user side 3 for decoding and playing according to the network environment of the user side.
Further, the method may further comprise the steps of: when the electronic device 1 collects real-time streaming media data of a live broadcast scene, controlling the camera 14 to shoot images of the live broadcast scene at preset time intervals; transmitting the shot image of the live broadcast scene to the server 2 along with the basic layer code stream data and the extended layer code stream data; and setting the image of the live broadcast scene as a cover picture of a live broadcast channel corresponding to the live broadcast scene every the preset time. In this embodiment, the predetermined time is half an hour.
In a live broadcast scenario, a video stream with a fixed bitrate and resolution is usually pushed by the anchor, that is, the user of the electronic device 1, but because there may be a great difference in the environment where the audience, that is, the user of the user terminal 3, is located, for example, at home, at an office, on the road, etc., the difference in the environment causes the network and the network quality used by the user terminal 3 to be different, in order to improve the user experience of watching the live broadcast audience under a poor network environment, it is necessary to reduce the streaming bitrate of the user terminal 3 under the poor network environment, and meanwhile, the low delay of the live broadcast is ensured.
The real-time streaming media data playing method, the electronic device and the server provided by the invention transmit the streaming media data according to the UDP private protocol, and the coding and decoding work of the streaming media data is synchronously carried out, thereby effectively reducing the time delay in the live broadcasting process. In addition, two code streams are generated through layered coding, so that the user terminal 3 can receive streaming media data which is suitable for the network environment to carry out smooth playing. Under the condition that the user side 3 is in a better network environment, the user can experience a clearer image quality effect, and under the condition that the user side 3 is in a poorer network environment, the user can be guaranteed to watch smoothly in a priority mode, so that the watching experience of the user is effectively improved.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. Several units or means recited in the apparatus claims may also be embodied by one and the same item or means in software or hardware. The terms first, second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A transmission method of real-time streaming media data is characterized by comprising the following steps:
optimizing real-time streaming media data acquired by an electronic device;
performing layered coding on the optimized real-time streaming media data to generate basic layer code stream data and extended layer code stream data;
packaging the generated basic layer code stream data and the generated extended layer code stream data, and pushing the basic layer code stream data and the extended layer code stream data to a server;
the server receives basic layer code stream data and extended layer code stream data which are packaged and pushed by the electronic device;
decapsulating the encapsulated base layer code stream data and the encapsulated extension layer code stream data; and
and sending the basic layer code stream data or the expansion layer code stream data to the user side for decoding and playing according to the network environment of the user side when the user side pulls the stream.
2. The method for transmitting real-time streaming media data according to claim 1, wherein the step of "performing optimization processing on the real-time streaming media data collected by the electronic device" includes:
detecting whether a video in the real-time streaming media data contains a human face;
when the video in the real-time streaming media data contains a face, performing facial beautification on the face;
responding to user operation to carry out special effect processing on the video of the real-time streaming media data; and
and carrying out noise removal and volume adjustment processing on the audio frequency in the real-time streaming media data.
3. The method for transmitting real-time streaming media data according to claim 1, wherein the step of performing layered coding on the real-time streaming media data collected by the electronic device to generate the base layer code stream data and the extended layer code stream data specifically comprises:
extracting high-quality audio and/or video frame data and low-quality audio and/or video frame data from the collected real-time streaming media data;
coding high-quality audio and/or video frame data to generate the base layer code stream data; and
and coding the low-quality audio and/or video frame data to generate the extended layer code stream data.
4. The transmission method of real-time streaming media data according to claim 3, wherein: the basic layer code stream data comprises high-quality audio and/or video frame data, the expansion layer code stream data comprises low-quality audio and/or video frame data, and the network bandwidth required for transmitting the expansion layer code stream data to the user side is a preset percentage of the network bandwidth required for transmitting the basic layer code stream data to the user side.
5. The method for transmitting real-time streaming media data according to claim 1, wherein the step of sending the base layer bitstream data or the extended layer bitstream data to the user terminal for decoding and playing according to the network environment of the user terminal when the user terminal pulls the stream comprises:
judging whether the downlink network bandwidth of the user side is greater than or equal to a preset value;
when the downlink network bandwidth of the user side is judged to be larger than or equal to the preset value, the basic layer code stream data is sent to the user side for decoding and playing when the user side pulls the stream; and
and when the downlink network bandwidth of the user side is judged to be smaller than the preset value, the extended layer code stream data is sent to the user side for decoding and playing when the user side pulls the stream.
6. The method for transmitting real-time streaming media data according to claim 1, wherein the method further comprises:
when the electronic device collects real-time streaming media data of a live broadcast scene, shooting images of the live broadcast scene at preset time intervals;
transmitting the shot image of the live broadcast scene to the server along with the basic layer code stream data and the extended layer code stream data; and
and setting the image of the live broadcast scene as a cover picture of a live broadcast channel corresponding to the live broadcast scene every the preset time.
7. An electronic device communicatively coupled to at least one server, the electronic device comprising:
the acquisition module is used for acquiring streaming media data in real time;
the optimization module is used for optimizing the acquired real-time streaming media data;
the encoding module is used for carrying out layered encoding on the real-time streaming media data subjected to the optimization processing to generate basic layer code stream data and extended layer code stream data;
the packaging module is used for packaging the generated basic layer code stream data and the generated extended layer code stream data; and
and the flow pushing module is used for pushing the packaged basic layer code stream data and the packaged expansion layer code stream data to the server, and the server decapsulates the basic layer code stream data and the expansion layer code stream data which are pushed by the flow pushing module and sends the basic layer code stream data or the expansion layer code stream data to the user side for decoding and playing according to the network environment of the user side when the user side pulls the flow.
8. The electronic device of claim 7, wherein the encoding module is configured to extract high quality audio and/or video frame data and low quality audio and/or video frame data from the captured real-time streaming data, encode the high quality audio and/or video frame data to generate the base layer bitstream data, and encode the low quality audio and/or video frame data to generate the extended layer bitstream data.
9. A server for communicating with an electronic device and at least one client, the server comprising:
the receiving module is used for receiving the basic layer code stream data and the extended layer code stream data which are packaged and pushed by the electronic device, wherein the basic layer code stream data and the extended layer code stream data are produced by layering and coding the acquired and optimized real-time streaming media data through the electronic device;
the decapsulation module is used for decapsulating the base layer code stream data and the extension layer code stream data; and
and the sending module is used for sending the basic layer code stream data or the expansion layer code stream data to the user side for decoding and playing according to the network environment of the user side when the user side pulls the stream.
10. The server of claim 9, wherein: the basic layer code stream data comprises high-quality audio and/or video frame data, the expansion layer code stream data comprises low-quality audio and/or video frame data, and the network bandwidth required by the sending module for sending the expansion layer code stream data to the user side is a preset percentage of the network bandwidth required by the sending module for sending the basic layer code stream data to the user side.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010334661.3A CN111510720A (en) | 2020-04-24 | 2020-04-24 | Real-time streaming media data transmission method, electronic device and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010334661.3A CN111510720A (en) | 2020-04-24 | 2020-04-24 | Real-time streaming media data transmission method, electronic device and server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111510720A true CN111510720A (en) | 2020-08-07 |
Family
ID=71871325
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010334661.3A Pending CN111510720A (en) | 2020-04-24 | 2020-04-24 | Real-time streaming media data transmission method, electronic device and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111510720A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112600858A (en) * | 2021-01-06 | 2021-04-02 | 智道网联科技(北京)有限公司 | Automatic driving data processing method, vehicle-mounted equipment and vehicle |
| CN113038128A (en) * | 2021-01-25 | 2021-06-25 | 腾讯科技（深圳）有限公司 | Data transmission method and device, electronic equipment and storage medium |
| WO2022148302A1 (en) * | 2021-01-07 | 2022-07-14 | 苏州浪潮智能科技有限公司 | Information exchange method, system and device, and computer-readable storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1643875A (en) * | 2002-03-27 | 2005-07-20 | 英国电讯有限公司 | Data streaming system and method |
| CN101800895A (en) * | 2008-09-19 | 2010-08-11 | 美国博通公司 | Method and system for vidoe encoding |
| CN103200399A (en) * | 2012-01-04 | 2013-07-10 | 北京大学 | Video quality fluctuation control method and device based on scalable video coding |
| CN104754366A (en) * | 2015-03-03 | 2015-07-01 | 腾讯科技（深圳）有限公司 | Audio and video file live broadcasting method, device and system |
| CN105392029A (en) * | 2015-11-09 | 2016-03-09 | Tcl集团股份有限公司 | Realization method and system of HLS on-demand playing and direct broadcasting |
| CN107454416A (en) * | 2017-09-12 | 2017-12-08 | 广州酷狗计算机科技有限公司 | Video flow sending method and device |
| CN110730356A (en) * | 2019-09-05 | 2020-01-24 | 天脉聚源(杭州)传媒科技有限公司 | Real-time refreshing method, system and device for live video and storage medium |
-
2020
- 2020-04-24 CN CN202010334661.3A patent/CN111510720A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1643875A (en) * | 2002-03-27 | 2005-07-20 | 英国电讯有限公司 | Data streaming system and method |
| CN101800895A (en) * | 2008-09-19 | 2010-08-11 | 美国博通公司 | Method and system for vidoe encoding |
| CN103200399A (en) * | 2012-01-04 | 2013-07-10 | 北京大学 | Video quality fluctuation control method and device based on scalable video coding |
| CN104754366A (en) * | 2015-03-03 | 2015-07-01 | 腾讯科技（深圳）有限公司 | Audio and video file live broadcasting method, device and system |
| CN105392029A (en) * | 2015-11-09 | 2016-03-09 | Tcl集团股份有限公司 | Realization method and system of HLS on-demand playing and direct broadcasting |
| CN107454416A (en) * | 2017-09-12 | 2017-12-08 | 广州酷狗计算机科技有限公司 | Video flow sending method and device |
| CN110730356A (en) * | 2019-09-05 | 2020-01-24 | 天脉聚源(杭州)传媒科技有限公司 | Real-time refreshing method, system and device for live video and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 王德: "《学校田径运动会的策划与组织》", vol. 37, 西安交通大学出版社, article 孙恒, pages: 235 - 243 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112600858A (en) * | 2021-01-06 | 2021-04-02 | 智道网联科技(北京)有限公司 | Automatic driving data processing method, vehicle-mounted equipment and vehicle |
| WO2022148302A1 (en) * | 2021-01-07 | 2022-07-14 | 苏州浪潮智能科技有限公司 | Information exchange method, system and device, and computer-readable storage medium |
| CN113038128A (en) * | 2021-01-25 | 2021-06-25 | 腾讯科技（深圳）有限公司 | Data transmission method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210377575A1 (en) | Live streaming method and system, server, and storage medium | |
| EP3562163B1 (en) | Audio-video synthesis method and system | |
| US8687114B2 (en) | Video quality adaptation based upon scenery | |
| RU2510908C2 (en) | Description of aggregated units of media data with backward compatibility | |
| CN111510720A (en) | Real-time streaming media data transmission method, electronic device and server | |
| US20110274180A1 (en) | Method and apparatus for transmitting and receiving layered coded video | |
| CN111416989A (en) | Video live broadcast method and system and electronic equipment | |
| CN107819809B (en) | Method and device for synchronizing content | |
| CN112019927B (en) | Video live broadcast method, microphone connecting equipment, live broadcast system and storage medium | |
| CN101917613A (en) | Acquiring and coding service system of streaming media | |
| WO2018185778A1 (en) | Systems and methods for achieving optimal network bitrate | |
| US11438645B2 (en) | Media information processing method, related device, and computer storage medium | |
| KR20080086262A (en) | Method and apparatus for sharing digital contents, and digital contents sharing system using the method | |
| CN111182342A (en) | Media data playing method, device, equipment and storage medium based on DLNA | |
| WO2016193052A1 (en) | Method for initiating a transmission of a streaming content delivered to a client device and access point for implementing this method | |
| CN113301359A (en) | Audio and video processing method and device and electronic equipment | |
| EP3316546B1 (en) | Multimedia information live method and system, collecting device and standardization server | |
| CN113630576A (en) | Adaptive video streaming system and method | |
| CN104639979A (en) | Video sharing method and system | |
| CN116233490A (en) | Video synthesis method, system, device, electronic equipment and storage medium | |
| KR100747664B1 (en) | Method for process multimedia data adaptive to bandwidth and host apparatus | |
| TW201441935A (en) | System and method of video screenshot | |
| KR101440131B1 (en) | Qos Image Processing System to control Multi-CCTV by Using Mobile Client Terminal and Qos Image Processing Method thereof | |
| CN113747181A (en) | Network live broadcast method, live broadcast system and electronic equipment based on remote desktop | |
| CN107005745B (en) | Method and apparatus for encapsulating a stream of audiovisual content |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |