WO2020124725A1 - 基于WebRTC协议的音视频推送方法和推流客户端 - Google Patents

基于WebRTC协议的音视频推送方法和推流客户端 Download PDF

Info

Publication number
WO2020124725A1
WO2020124725A1 PCT/CN2019/072181 CN2019072181W WO2020124725A1 WO 2020124725 A1 WO2020124725 A1 WO 2020124725A1 CN 2019072181 W CN2019072181 W CN 2019072181W WO 2020124725 A1 WO2020124725 A1 WO 2020124725A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
streaming client
additional
audio
input unit
Prior art date
Application number
PCT/CN2019/072181
Other languages
English (en)
French (fr)
Inventor
林新建
Original Assignee
网宿科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 网宿科技股份有限公司 filed Critical 网宿科技股份有限公司
Priority to EP19897732.4A priority Critical patent/EP3902272A4/en
Priority to US17/288,992 priority patent/US20210409476A1/en
Publication of WO2020124725A1 publication Critical patent/WO2020124725A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/435—Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/2187—Live feed
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G06F3/165—Management of the audio stream, e.g. setting of volume, audio stream path
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/61—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio
    • H04L65/611—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio for multicast or broadcast
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/65—Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/75—Media network packet handling
    • H04L65/762—Media network packet handling at the source 
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/14—Session management
    • H04L67/141—Setup of application sessions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/55—Push-based network services
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/27—Server based end-user applications
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/439—Processing of audio elementary streams
    • H04N21/4398—Processing of audio elementary streams involving reformatting operations of audio signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
    • H04N21/4788—Supplemental services, e.g. displaying phone caller identification, shopping application communicating with other users, e.g. chatting
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/643—Communication protocols
    • H04N21/6437—Real-time Transport Protocol [RTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/8106—Monomedia components thereof involving special audio data, e.g. different tracks for different languages
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/8126—Monomedia components thereof involving additional data, e.g. news, sports, stocks, weather forecasts
    • H04N21/8133—Monomedia components thereof involving additional data, e.g. news, sports, stocks, weather forecasts specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/85—Assembly of content; Generation of multimedia applications
    • H04N21/854—Content authoring

Definitions

  • the present invention relates to the field of streaming media technology, and in particular to an audio and video pushing method and streaming client based on WebRTC protocol.
  • WebRTC Web Real-Time Communication, derived from webpage real-time communication
  • the WebRTC protocol can support web browsers for real-time voice conversations or video conversations. It can realize real-time transmission of video and audio based on browsers, and is widely used in the above business scenarios.
  • the streaming client can first establish a WebRTC connection with the resource server.
  • the streaming client can record live video, it can record the live audio and video data (which can be called source data) based on the WebRTC connection. ) Push to the resource server.
  • the pull stream client that has established a WebRTC connection with the resource server can obtain the source data of the live broadcast from the resource server, so as to play the live video.
  • the embodiments of the present invention provide an audio and video pushing method and a streaming client based on WebRTC protocol.
  • the technical solution is as follows:
  • a method for pushing audio and video based on the WebRTC protocol includes:
  • the streaming client calls the audio interface of the browser to create an audio processing module, and sets a mixed output unit on the audio processing module;
  • the streaming client inputs source data and additional data into the audio processing module, and generates processing data through the hybrid output unit;
  • the streaming client combines the source data and the processed data to generate mixing data
  • the streaming client pushes the mixing data to the resource server, so that the streaming client obtains the mixing data from the resource server.
  • the method further includes:
  • the streaming client sets a source data input unit and an additional data input unit on the audio processing module
  • the push streaming client builds a source data processing line through the source data input unit and the mixed output unit;
  • the push streaming client builds an additional data processing line through the additional data input unit and the mixed output unit.
  • the push streaming client inputs source data and additional data to the audio processing module, and generates processing data through the mixed output unit, including:
  • the streaming client inputs the source data into the source data input unit, and imports the output data of the source data input unit into the hybrid output unit through the source data processing line;
  • the streaming client inputs the additional data into the additional data input unit, and through the additional data processing line, imports the output data of the additional data input unit into the hybrid output unit;
  • the streaming client processes the output data of the source data input unit and the output data of the additional data input unit through the hybrid output unit to generate the processed data.
  • an active volume adjustment unit is provided between the source data input unit and the hybrid output unit on the source data processing line;
  • the streaming client imports the output data of the source data input unit into the mixed output unit through the source data processing line, including:
  • the streaming client imports the output data of the source data input unit into the source volume adjustment unit through the source data processing line, and imports the output data of the source volume adjustment unit into the hybrid output unit.
  • an additional volume adjustment unit is provided between the additional data input unit and the hybrid output unit on the additional data processing line;
  • the streaming client imports the output data of the additional data input unit into the mixed output unit through the additional data processing line, including:
  • the streaming client imports the output data of the additional data input unit into the additional volume adjustment unit through the additional data processing line, and imports the output data of the additional volume adjustment unit into the hybrid output unit.
  • inputting additional data into the additional data input unit by the push streaming client includes:
  • the streaming client calls the decodeAudioData function of the audio processing module, decodes the additional data, and inputs the decoded additional data into the additional data input unit.
  • the push streaming client combines the source data and the processed data to generate mixing data, including:
  • the streaming client calls the data processing interface of the browser to extract video data from the source data
  • the streaming client calls the data processing interface of the browser to extract audio data in the processed data
  • the streaming client calls the data processing interface of the browser to merge the video data and the audio data to generate the mixing data.
  • the method further includes:
  • the streaming client calls the data collection interface of the browser to collect the source data, and inputs the source data to the audio processing module.
  • the method further includes:
  • the streaming client calls the additional data interface of the browser to obtain the additional data, and inputs the additional data to the audio processing module.
  • the pushing the streaming client to push the mixing data to the resource server includes:
  • the push streaming client calls the connection interface of the browser to create a push module
  • the push streaming client establishes a connection with the resource server through the push module
  • the push streaming client inputs the mixing data into the push module, and pushes it to the resource server through the push module.
  • an audio and video streaming client based on the WebRTC protocol is provided.
  • the streaming client is used for:
  • streaming client is also used to:
  • a source data input unit and an additional data input unit are provided on the audio processing module
  • An additional data processing line is constructed through the additional data input unit and the hybrid output unit.
  • the push streaming client is specifically used for:
  • the output data of the source data input unit and the output data of the additional data input unit are processed by the hybrid output unit to generate the processed data.
  • an active volume adjustment unit is provided between the source data input unit and the hybrid output unit on the source data processing line;
  • the streaming client is specifically used to: through the source data processing line, import the output data of the source data input unit into the source volume adjustment unit, and import the output data of the source volume adjustment unit into the Mixed output unit.
  • an additional volume adjustment unit is provided between the additional data input unit and the hybrid output unit on the additional data processing line;
  • the streaming client is specifically used to: through the additional data processing line, import the output data of the additional data input unit into the additional volume adjustment unit, and import the output data of the additional volume adjustment unit into the Mixed output unit.
  • the push streaming client is specifically used for:
  • the push streaming client is specifically used for:
  • the mixing data is generated.
  • streaming client is also used to:
  • streaming client is also used to:
  • the push streaming client is specifically used for:
  • the mixing data is input to the pushing module, and pushed to the resource server through the pushing module.
  • the push streaming client calls an audio interface of the browser to create an audio processing module, and a mixed output unit is provided on the audio processing module; the push streaming client inputs source data and additional data into the audio A processing module, and generating processing data through the mixing output unit; the push streaming client merges the source data and the processing data to generate mixing data; and the push streaming client converts the mixing data Push to the resource server, so that the pull stream client obtains the mixing data from the resource server.
  • the source data and additional data are merged into one piece of mixed data, which reduces the transmission of one piece of data during transmission and reduces the bandwidth used during transmission; only one piece of data needs to be processed during playback and recording, which can be supported More types of browsers are used for playback, which is helpful for later viewing.
  • FIG. 1 is a flowchart of a method for pushing audio and video based on the WebRTC protocol provided by an embodiment of the present invention
  • FIG. 2 is a processing flowchart of generating processing data of source data and additional data in an audio and video pushing method based on WebRTC protocol provided by an embodiment of the present invention
  • FIG. 3 is a processing flowchart of combining source data and processing data to obtain mixed audio data in a method for pushing audio and video based on the WebRTC protocol provided by an embodiment of the present invention
  • FIG. 4 is a schematic diagram of a streaming client pushing a mixed audio data to a resource server in an audio and video pushing method based on WebRTC protocol provided by an embodiment of the present invention
  • FIG. 5 is a schematic structural diagram of an audio and video streaming client based on the WebRTC protocol provided by an embodiment of the present invention
  • FIG. 6 is a schematic structural diagram of an audio and video streaming client based on the WebRTC protocol provided by an embodiment of the present invention
  • FIG. 7 is a schematic structural diagram of an audio and video streaming client based on the WebRTC protocol provided by an embodiment of the present invention.
  • FIG. 8 is a schematic structural diagram of an audio and video streaming client based on the WebRTC protocol provided by an embodiment of the present invention.
  • An embodiment of the present invention provides a method for pushing audio and video based on the WebRTC protocol.
  • the method can be applied to a streaming client that uses the WebRTC protocol to transmit live audio and video data.
  • the streaming module is implemented.
  • the push streaming client establishes a connection with the resource server through the WebRTC protocol, and pushes the live broadcast data stream to the resource server based on the connection. All the processing of the push streaming client described below can be understood as actually performed by the push streaming module in the push streaming client.
  • the application scenario of this embodiment may be: the streaming client collects live audio and video data through a device such as a camera or microphone as source data, and at the same time, obtains audio data that you want to add to the source data as additional data through network download or other methods.
  • the push-stream client processes the source data and the additional data to generate processed data; the push-stream client then merges the source data and the processed data to obtain the mixing data after adding the additional data to the source data, and pushes the streaming client
  • the mixing data can be pushed to the resource server through the WebRTC connection established with the resource server, so that the pull stream client can obtain the mixing data from the resource server.
  • Step 101 The streaming client calls the audio interface of the browser to create an audio processing module, and sets a mixed output unit on the audio processing module.
  • the streaming client when used for audio and video live broadcasting, a browser using the AudioContext interface as an audio interface is installed on the streaming client, and the streaming client calls the audio context interface of the browser to create an audio processing module AC (AudioContext ), and then set the mixed output unit MDN (MediaStreamAudioDestinationNode) used to mix the source data OS (OriginalStream) and additional data FD (FileData) in the audio processing module AC through the createMediaStreamDestination function.
  • the setting method can use the following formula:
  • the audio processing module AC is also provided with an input unit, and the corresponding processing can be as follows: the push streaming client sets the source data input unit and the additional data input unit on the audio processing module; the push streaming client passes the source data input unit and the mixed output unit Build source data processing line; push streaming client builds additional data processing line through additional data input unit and mixed output unit.
  • the push streaming client can also set the source data input unit MSN (MediaStreamAudioSourceNode) for inputting the source data OS through the createMediaStreamSource function in the audio processing module AC, and then push the streaming client
  • the end connects the source data input unit MSN and the mixed output unit MDN through the connect function, so that the source data OS can sequentially pass through the source data input unit MSN and the mixed output unit MDN, and perform corresponding processing in each unit, so that it is formed
  • a one-way line for processing source data OS that is, a source data processing line built by source data input unit MSN and mixed output unit MDN.
  • the method of setting the source data input unit MSN through the createMediaStreamSource function can use the following formula:
  • the method of connecting the source data input unit MSN and the mixed output unit MDN through the connect function can use the following formula:
  • the push-stream client sets the additional data input unit BSN (AudioBufferSourceNode) for inputting additional data FD through the createBufferSource function in the audio processing module AC. Then, the push-stream client also inputs the additional data into the unit BSN and through the connect function
  • the hybrid output unit MDN is connected, so that the additional data FD can sequentially pass through the additional data input unit BSN and the hybrid output unit MDN, and perform corresponding processing in each unit, so that a unidirectional for processing additional data FD is formed Line, that is, an additional data processing line constructed by an additional data input unit BSN and a hybrid output unit MDN
  • the method of setting the additional data input unit BSN through the createBufferSource function can use the following formula:
  • the method of connecting the additional data input unit BSN and the mixed output unit MDN through the connect function can use the following formula:
  • Step 102 The streaming client inputs source data and additional data into the audio processing module, and generates processing data through the mixed output unit.
  • the push streaming client after setting the mixed output unit MDN, the push streaming client obtains the source data OS and the additional data FD, and inputs it to the audio processing module AC. Then, the push streaming client passes the mixed output unit MDN, according to the source data OS and The additional data FD generates processing data DS (DisposeStream).
  • the processing in step 102 may be specifically as follows: the push stream client inputs the source data into the source data input unit, and through the source data processing line, the source data input unit outputs The data is imported into the hybrid output unit; the push-stream client inputs the additional data into the additional data-input unit, and the output data of the additional data-input unit is imported into the hybrid output unit through the additional data processing line; The output data of the input unit and the output data of the additional data input unit are processed to generate processed data.
  • the push-stream client after building the source data processing line and the additional data processing line, the push-stream client inputs the source data OS into the source data input unit MSN. According to the source data processing line, the push-stream client can input the source data into the unit The output data of MSN is imported into the mixed output unit MDN. The push-stream client inputs the additional data FD into the additional data input unit BSN. According to the additional data processing line, the push-stream client can import the output data of the additional data input unit BSN into the hybrid output unit MDN. After being imported into the output data of the source data input unit MSN and the output data of the additional data input unit BSN, the hybrid output unit MDN processes the two output data to generate processing data DS.
  • the volume of the source data OS can be adjusted.
  • the corresponding processing can be as follows: an active volume adjustment unit is set between the source data input unit and the mixed output unit on the source data processing line; push The stream client imports the output data of the source data input unit into the source volume adjustment unit through the source data processing line, and imports the output data of the source volume adjustment unit into the mixed output unit.
  • the streaming client can also set the source volume adjustment unit GN (GainNode) in the audio processing module AC through the createGain function.
  • the method of setting the source volume adjustment unit GN through the createGain function can use the following formula:
  • the streaming client sets the source volume adjustment unit GN between the source data input unit MSN and the hybrid output unit MDN.
  • the push client connects the source data input unit MSN, the source volume adjustment unit GN, and the hybrid output unit MDN sequentially through the connect function to form a source data processing line, so that the source data OS is input to the source data output unit MSN
  • the output data of the source data input unit MSN can be imported into the source volume adjustment unit GN
  • the output data processed by the source volume adjustment unit GN can be imported into the hybrid output unit MDN.
  • the method of sequentially connecting the source data input unit MSN, the source volume adjustment unit GN, and the hybrid output unit MDN through the connect function can use the following formula:
  • the volume of the additional data FD can be adjusted.
  • the corresponding processing can be as follows: an additional volume adjustment unit is provided between the additional data input unit and the hybrid output unit on the additional data processing line; push The streaming client imports the output data of the additional data input unit into the additional volume adjustment unit through the additional data processing line, and imports the output data of the additional volume adjustment unit into the hybrid output unit.
  • the streaming client can also set an additional volume adjustment unit GN1 through the createGain function in the audio processing module AC.
  • the method of setting the additional volume adjustment unit GN1 through the createGain function can adopt the following formula:
  • the streaming client sets the additional volume adjustment unit GN1 between the additional data input unit BSN and the hybrid output unit MDN.
  • the push client connects the additional data input unit BSN, the additional volume adjustment unit GN1, and the hybrid output unit MDN in sequence through the connect function to form an additional data processing line, so that the additional data FD is input to the additional data output unit BSN
  • the output data of the additional data input unit BSN can be imported into the additional volume adjustment unit GN1
  • the output data processed by the additional volume adjustment unit GN1 can be imported into the hybrid output unit MDN.
  • the method of sequentially connecting the additional data input unit BSN, the additional volume adjustment unit GN and the hybrid output unit MDN through the connect function can use the following formula:
  • the additional data FD obtained by the streaming server also needs to be decoded accordingly to facilitate mixing with the source data OS.
  • the corresponding processing can be as follows:
  • the streaming client calls the decodeAudioData function of the audio processing module to decode the additional data and decode it. After the additional data is input to the additional data input unit.
  • the streaming client calls the decodeAudioData function of the audio processing module AC to decode the additional data FD to obtain audio PCM (Pulse Code Modulation) data DFD, and then decodes The additional data DFD is input to the additional data input unit BSN.
  • PCM Pulse Code Modulation
  • the method to decode the additional data FD by calling the decodeAudioData function can use the following formula:
  • the above process for generating the processing data DS based on the source data OS and the additional data FD can refer to FIG. 2.
  • the live streaming audio and video data can be collected as the source data OS through the push streaming client.
  • the corresponding processing can be as follows: the push streaming client calls the browser's data collection interface to collect the source data and the source Data input audio processing module.
  • the streaming client is built-in or externally connected with audio and video collection devices such as cameras and/or microphones.
  • the streaming client can call the data collection interface provided by the browser, such as the getUserMedia interface
  • the acquisition equipment collects audio and video data.
  • the streaming client uses the collected audio and video data as the source data OS and inputs it to the audio processing module AC.
  • the method of collecting source data OS through the getUserMedia interface can be as follows:
  • the streaming client calls the additional data of the browser
  • the interface acquires additional data and inputs the additional data into the audio processing module.
  • the user may need to add some background music, special sound effects and other audio and video during the live broadcast process to increase the live broadcast effect. Therefore, these audio and video data can be obtained through the network.
  • the streaming client After obtaining the link address of the additional data FD, the streaming client calls the additional data interface provided by the browser, such as the XMLHttpRequest interface, downloads audio and video data from the network as the additional data FD, and inputs it to the audio processing module AC.
  • the additional data interface provided by the browser, such as the XMLHttpRequest interface
  • Step 103 The streaming client combines source data and processed data to generate mixing data.
  • the streaming client after obtaining the processing data DS through the mixing data output unit MDN, the streaming client combines the source data OS and the processing data DS to finally generate mixing data MS (MixingStream).
  • the page of the streaming client includes a streaming area and a display area.
  • the push flow area provides some common webpage elements such as buttons and text boxes.
  • the push flow users can perform corresponding operations to push the mixing data MS.
  • the display area provides a page external container (page div, page division), which can interact with the streaming client to display the mixing data MS pushed to the resource server, which is convenient for the streaming client to preview the mixing locally The effect of data.
  • the streaming client After the streaming client generates the mixing data MS, the mixing data MS can be displayed in the display area in real time. Specifically, after generating the mixing data MS, the streaming client may create a display video (Video) tag and load the mixing data MS into the display video tag.
  • Video display video
  • the playback attribute of the displayed video tag can be set to be visible, so that the streaming user can directly view the screen of the mixing data MS from the outside.
  • the streaming client can create an internal container (internal div, internal division) and display the mixing data MS loaded in the display video tag in the internal container.
  • the browser page may display the internal container of the mixing data MS through callback, so that the internal container displaying the mixing data MS is filled in the external container of the page set in the browser page in advance.
  • the outer container of the page can customize the effect style displayed on the screen in the display area.
  • the process of generating the mixing data MS can be as follows: the push streaming client calls the browser's data processing interface to extract the video data in the source data; the push streaming client calls the browser's data processing interface to extract the audio data in the processed data; push streaming The client calls the browser's data processing interface to merge the video data and audio data to generate mixed audio data.
  • the data processing interface of the browser may be a MediaStream interface or other interfaces.
  • the streaming client calls the MediaStream interface of the browser, and the video data VT is extracted from the source data OS through the getVideoTracks function. VideoTracks).
  • the method of extracting video data VT from the source data OS through the getVideoTracks function can use the following formula:
  • the push streaming client calls the MediaStream interface of the browser and extracts audio data AT (AudioTracks) from the processed data DS through the getAudioTracks function.
  • the method of extracting audio data AT from the processed data DS through the getVideoTracks function can use the following formula:
  • the streaming client continues to call the MediaStream interface of the browser to merge the extracted video data VT and audio data AT to generate the mixing data MS.
  • the method of combining the extracted video data VT and audio data AT to generate the mixing data MS can adopt the following formula:
  • the resource server will re-encode and decode the audio data and video data of the mixing data MS. Since the combined mixing data MS has only one audio data and one Video data, in this way, can improve the compatibility of the resource server codec.
  • Step 104 The streaming client pushes the mixing data to the resource server, so that the streaming client obtains the mixing data from the resource server.
  • the streaming client after the streaming client generates the mixing data MS, it can push the mixing data MS to the resource server, so that after the streaming client establishes a connection with the resource server, the mixing data can be obtained from the resource server MS.
  • the page of the pull stream client includes a pull stream area and a display area.
  • the pull stream area provides some common webpage elements such as buttons and text boxes.
  • the pull stream user can perform corresponding operations to pull the mixing data MS. Specifically, after setting the configuration information of the related stream in the pull-stream area, the pull-stream client starts to pull the stream; the pull-stream user checks the MS image of the mixing data displayed in the display area and listens to whether the audio output device such as headphones or computer speakers There are mixing effects (such as hearing the voice of the streamer and the background music at the same time).
  • the streaming client Before the streaming client sends the mixing data MS to the resource server, it also needs to establish a connection with the resource server.
  • the corresponding processing can be as follows: the streaming client calls the connection interface of the browser to create a push module; the streaming client passes The push module establishes a connection with the resource server; the push streaming client inputs the mixing data into the push module and pushes it to the resource server through the push module.
  • the push streaming client calls the connection interface provided by the browser to create a push module.
  • the push streaming client can use the RTCPeerConnection interface as the connection interface.
  • the push streaming client establishes a connection with the resource server based on the WebRTC protocol through the push module. After that, the push streaming client can input the mixing data MS into the pushing module, and then push the mixing data MS to the resource server through the pushing module.
  • the pull-stream client can also establish a connection with the resource server based on the WebRTC protocol. After the resource server receives the mixing data MS, the pull-stream client that establishes a connection with the resource server can pull the mixing data MS from the resource server.
  • the push streaming client calls an audio interface of the browser to create an audio processing module, and a mixed output unit is provided on the audio processing module; the push streaming client inputs source data and additional data into the audio A processing module, and generating processing data through the mixing output unit; the push streaming client merges the source data and the processing data to generate mixing data; and the push streaming client converts the mixing data Push to the resource server, so that the pull stream client obtains the mixing data from the resource server.
  • the source data and additional data are merged into one piece of mixed data, which reduces the transmission of one piece of data during transmission and reduces the bandwidth used during transmission; only one piece of data needs to be processed during playback and recording, which can be supported More types of browsers are used for playback, which is helpful for later viewing.
  • an embodiment of the present invention also provides an audio and video streaming client based on the WebRTC protocol.
  • the streaming client 500 is used to:
  • the streaming client 500 is also used to:
  • a source data input unit 522 and an additional data input unit 523 are provided on the audio processing module 511;
  • An additional data processing line is constructed through the additional data input unit 523 and the hybrid output unit 521.
  • the streaming client 500 is specifically used to:
  • the output data of the source data input unit 522 and the output data of the additional data input unit 523 are processed by the hybrid output unit 521 to generate the processed data.
  • an active volume adjustment unit 524 is provided between the source data input unit 522 and the hybrid output unit 521 on the source data processing line;
  • the streaming client 500 is specifically configured to: through the source data processing line, import the output data of the source data input unit 522 into the source volume adjustment unit 524, and output the source volume adjustment unit 524 Data is imported into the hybrid output unit 521.
  • an additional volume adjustment unit 525 is provided between the additional data input unit 523 and the hybrid output unit 521 on the additional data processing line;
  • the streaming client 500 is specifically configured to: through the additional data processing line, import the output data of the additional data input unit 523 into the additional volume adjustment unit 525, and output the additional volume adjustment unit 525 Data is imported into the hybrid output unit 521.
  • the streaming client 500 is specifically used to:
  • the streaming client 500 is specifically used to:
  • the mixing data is generated.
  • the streaming client 500 is also used to:
  • the streaming client 500 is also used to:
  • the streaming client 500 is specifically used to:
  • the mixing data is input to the pushing module 512, and pushed to the resource server through the pushing module 512.
  • the WebRTC protocol-based audio and video streaming client provided in the above embodiment only uses the above-mentioned division of each functional module as an example to illustrate the implementation of the WebRTC protocol-based audio and video push.
  • you can The above function allocation is performed by different function modules, that is, the internal structure of the push client is divided into different function modules to complete all or part of the functions described above.
  • the audio and video streaming client based on the WebRTC protocol provided by the above embodiment and the WebRTC protocol based audio and video pushing method embodiment belong to the same concept.
  • the specific implementation process refer to the method embodiment for details, which will not be repeated here.
  • each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware.
  • the above technical solution can be embodied in the form of a software product in essence or part of the contribution to the existing technology, and the computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic Discs, optical discs, etc., include several instructions to enable a computer device (which may be a personal computer, server, or network device, etc.) to perform the methods described in the various embodiments or some parts of the embodiments.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • General Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Security & Cryptography (AREA)
  • Databases & Information Systems (AREA)
  • Information Transfer Between Computers (AREA)
  • Telephonic Communication Services (AREA)
  • Stored Programmes (AREA)

Abstract

本发明公开了一种基于WebRTC协议的音视频推送方法和推流客户端，属于流媒体技术领域。所述方法包括：推流客户端调用浏览器的音频接口创建音频处理模块，并在所述音频处理模块上设置混合输出单元；所述推流客户端将源数据和附加数据输入所述音频处理模块，并通过所述混合输出单元生成处理数据；所述推流客户端将所述源数据和所述处理数据进行合并，生成混音数据；所述推流客户端将所述混音数据推送给资源服务器，以使拉流客户端从所述资源服务器获取所述混音数据。本发明可以降低传输过程中使用的带宽。

Description

基于WebRTC协议的音视频推送方法和推流客户端 技术领域
本发明涉及流媒体技术领域，尤其涉及一种基于WebRTC协议的音视频推送方法和推流客户端。
背景技术
随着互联网技术的不断发展，实时音视频通信日益受到人们的青睐，越来越多的人通过网络收听音频直播或者收看视频直播。WebRTC(Web Real-Time Communication，源自网页实时通信)协议可以支持网页浏览器进行实时语音对话或视频对话，能够基于浏览器实现视频、音频的实时传输，被广泛应用在上述业务场景中。
通过WebRTC协议进行音视频直播时，推流客户端可以先与资源服务器建立WebRTC连接，推流客户端在录制直播视频时，可以基于WebRTC连接将录制的直播的音视频数据(可称为源数据)推送至资源服务器。之后，与资源服务器建立了WebRTC连接的拉流客户端就可以从资源服务器获取该直播的源数据，从而播放上述直播视频。
在实现本发明的过程中，发明人发现现有技术中至少存在以下问题：
在使用WebRTC协议传输音视频的过程中，如果想在源数据中添加背景音乐、特效音等音频数据，只能将要加入的音频数据(可称为附加数据)与源数据分别传输，然后同时播放。然而有些浏览器不支持同时播放多个音视频数据，无法同时播放源数据与附加数据，并且这样会造成传输过程中占用带宽较高；源数据与附加数据只能被分别录制，不方便回看。
发明内容
为了解决现有技术的问题，本发明实施例提供了一种基于WebRTC协议的音视频推送方法和推流客户端。所述技术方案如下：
第一方面，提供了一种基于WebRTC协议的音视频推送方法，所述方法包 括：
推流客户端调用浏览器的音频接口创建音频处理模块，并在所述音频处理模块上设置混合输出单元；
所述推流客户端将源数据和附加数据输入所述音频处理模块，并通过所述混合输出单元生成处理数据；
所述推流客户端将所述源数据和所述处理数据进行合并，生成混音数据；
所述推流客户端将所述混音数据推送给资源服务器，以使拉流客户端从所述资源服务器获取所述混音数据。
进一步的，所述方法还包括：
所述推流客户端在所述音频处理模块上设置源数据输入单元和附加数据输入单元；
所述推流客户端通过所述源数据输入单元和所述混合输出单元搭建源数据处理线；
所述推流客户端通过所述附加数据输入单元和所述混合输出单元搭建附加数据处理线。
进一步的，所述推流客户端将源数据和附加数据输入所述音频处理模块，并通过所述混合输出单元生成处理数据，包括：
所述推流客户端将所述源数据输入所述源数据输入单元，并通过所述源数据处理线，将所述源数据输入单元的输出数据导入所述混合输出单元；
所述推流客户端将所述附加数据输入所述附加数据输入单元，并通过所述附加数据处理线，将所述附加数据输入单元的输出数据导入所述混合输出单元；
所述推流客户端通过所述混合输出单元对所述源数据输入单元的输出数据和附加数据输入单元的输出数据进行处理，生成所述处理数据。
进一步的，所述源数据处理线上所述源数据输入单元和所述混合输出单元间设置有源音量调节单元；
所述推流客户端通过所述源数据处理线，将所述源数据输入单元的输出数据导入所述混合输出单元，包括：
所述推流客户端通过所述源数据处理线，将所述源数据输入单元的输出数据导入所述源音量调节单元，并将所述源音量调节单元的输出数据导入所述混合输出单元。
进一步的，所述附加数据处理线上所述附加数据输入单元和所述混合输出单元间设置有附加音量调节单元；
所述推流客户端通过所述附加数据处理线，将所述附加数据输入单元的输出数据导入所述混合输出单元，包括：
所述推流客户端通过所述附加数据处理线，将所述附加数据输入单元的输出数据导入所述附加音量调节单元，并将所述附加音量调节单元的输出数据导入所述混合输出单元。
进一步的，所述推流客户端将附加数据输入所述附加数据输入单元，包括：
所述推流客户端调用所述音频处理模块的decodeAudioData函数，对所述附加数据进行解码，将解码后的附加数据输入所述附加数据输入单元。
进一步的，所述推流客户端将所述源数据和所述处理数据进行合并，生成混音数据，包括：
所述推流客户端调用所述浏览器的数据处理接口提取所述源数据中的视频数据；
所述推流客户端调用所述浏览器的数据处理接口提取所述处理数据中的音频数据；
所述推流客户端调用所述浏览器的数据处理接口将所述视频数据和所述音频数据合并后，生成所述混音数据。
进一步的，所述方法还包括：
所述推流客户端调用所述浏览器的数据采集接口采集所述源数据，并将所述源数据输入所述音频处理模块。
进一步的，所述方法还包括：
所述推流客户端调用所述浏览器的附加数据接口获取所述附加数据，并将所述附加数据输入所述音频处理模块。
进一步的，所述推流客户端将所述混音数据推送给资源服务器，包括：
所述推流客户端调用所述浏览器的连接接口创建推送模块；
所述推流客户端通过所述推送模块与所述资源服务器建立连接；
所述推流客户端将所述混音数据输入所述推送模块，并通过所述推送模块推送给所述资源服务器。
第二方面，提供了一种基于WebRTC协议的音视频推流客户端，所述推流客户端用于：
调用浏览器的音频接口创建音频处理模块，并在所述音频处理模块上设置混合输出单元；
将源数据和附加数据输入所述音频处理模块，并通过所述混合输出单元生成处理数据；
将所述源数据和所述处理数据进行合并，生成混音数据；
将所述混音数据推送给资源服务器，以使拉流客户端从所述资源服务器获取所述混音数据。
进一步的，所述推流客户端还用于：
在所述音频处理模块上设置源数据输入单元和附加数据输入单元；
通过所述源数据输入单元和所述混合输出单元搭建源数据处理线；
通过所述附加数据输入单元和所述混合输出单元搭建附加数据处理线。
进一步的，所述推流客户端具体用于：
将所述源数据输入所述源数据输入单元，并通过所述源数据处理线，将所述源数据输入单元的输出数据导入所述混合输出单元；
将所述附加数据输入所述附加数据输入单元，并通过所述附加数据处理线，将所述附加数据输入单元的输出数据导入所述混合输出单元；
通过所述混合输出单元对所述源数据输入单元的输出数据和附加数据输入单元的输出数据进行处理，生成所述处理数据。
进一步的，所述源数据处理线上所述源数据输入单元和所述混合输出单元间设置有源音量调节单元；
所述推流客户端具体用于：通过所述源数据处理线，将所述源数据输入单元的输出数据导入所述源音量调节单元，并将所述源音量调节单元的输出数据导入所述混合输出单元。
进一步的，所述附加数据处理线上所述附加数据输入单元和所述混合输出单元间设置有附加音量调节单元；
所述推流客户端具体用于：通过所述附加数据处理线，将所述附加数据输入单元的输出数据导入所述附加音量调节单元，并将所述附加音量调节单元的输出数据导入所述混合输出单元。
进一步的，所述推流客户端具体用于：
调用所述音频处理模块的decodeAudioData函数，对所述附加数据进行解码，将解码后的附加数据输入所述附加数据输入单元。
进一步的，所述推流客户端具体用于：
调用所述浏览器的数据处理接口提取所述源数据中的视频数据；
调用所述浏览器的数据处理接口提取所述处理数据中的音频数据；
调用所述浏览器的数据处理接口将所述视频数据和所述音频数据合并后，生成所述混音数据。
进一步的，所述推流客户端还用于：
调用所述浏览器的数据采集接口采集所述源数据，并将所述源数据输入所述音频处理模块。
进一步的，所述推流客户端还用于：
调用所述浏览器的附加数据接口获取所述附加数据，并将所述附加数据输入所述音频处理模块。
进一步的，所述推流客户端具体用于：
调用所述浏览器的连接接口创建推送模块；
通过所述推送模块与所述资源服务器建立连接；
将所述混音数据输入所述推送模块，并通过所述推送模块推送给所述资源服务器。
本发明实施例提供的技术方案带来的有益效果是：
本发明实施例中，推流客户端调用浏览器的音频接口创建音频处理模块，并在所述音频处理模块上设置混合输出单元；所述推流客户端将源数据和附加数据输入所述音频处理模块，并通过所述混合输出单元生成处理数据；所述推流客户端将所述源数据和所述处理数据进行合并，生成混音数据；所述推流客户端将所述混音数据推送给资源服务器，以使拉流客户端从所述资源服务器获取所述混音数据。这样，将源数据和附加数据合并成为一份混音数据，在传输过程中减少了一份数据的传输，降低传输过程中使用的带宽；播放和录制时也只需要处理一份数据，可以支持更多种类的浏览器进行播放，有利于后期回看。
附图说明
为了更清楚地说明本发明实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本发明实施例提供的一种基于WebRTC协议的音视频推送方法的流程图；
图2是本发明实施例提供的一种基于WebRTC协议的音视频推送方法中源数据和附加数据生成处理数据的处理流程图；
图3是本发明实施例提供的一种基于WebRTC协议的音视频推送方法中源数据和处理数据合并得到混音数据的处理流程图；
图4是本发明实施例提供的一种基于WebRTC协议的音视频推送方法中推流客户端向资源服务器推送混音数据的原理图；
图5是本发明实施例提供的一种基于WebRTC协议的音视频推流客户端的结构示意图；
图6是本发明实施例提供的一种基于WebRTC协议的音视频推流客户端的结构示意图；
图7是本发明实施例提供的一种基于WebRTC协议的音视频推流客户端的结构示意图；
图8是本发明实施例提供的一种基于WebRTC协议的音视频推流客户端的结构示意图。
具体实施方式
为使本发明的目的、技术方案和优点更加清楚，下面将结合附图对本发明实施方式作进一步地详细描述。
本发明实施例提供了一种基于WebRTC协议的音视频推送方法，该方法可以适用于采用WebRTC协议传输直播的音视频数据的推流客户端上，并可以具体由推流客户端中设置的推流模块来实现，推流客户端通过WebRTC协议与资源服务器建立连接，并基于该连接向资源服务器推送直播数据流。下述所有推流客户端的处理，均可以理解为实际由推流客户端中的推流模块执行。本实施 例的应用场景可以是：推流客户端通过摄像头或麦克风等设备采集直播的音视频数据作为源数据，同时，通过网络下载或其他方法获取想要加入源数据的音频数据作为附加数据，之后，推流客户端对源数据和附加数据处理后生成处理数据；推流客户端再将源数据和处理数据合并，就得到了将附加数据加入源数据后的混音数据，推流客户端可以通过和资源服务器建立的WebRTC连接，将该混音数据推送给资源服务器，这样，拉流客户端就可以从资源服务器上获取该混音数据。
下面将结合具体实施方式，对图1所示的一种基于WebRTC协议的音视频推送流程进行详细的说明，内容可以如下：
步骤101：推流客户端调用浏览器的音频接口创建音频处理模块，并在音频处理模块上设置混合输出单元。
在实施中，当利用推流客户端进行音视频直播时，推流客户端上安装有以AudioContext接口作为音频接口的浏览器，推流客户端调用浏览器的AudioContext接口创建音频处理模块AC(AudioContext)，然后在音频处理模块AC中通过createMediaStreamDestination函数设置用于将源数据OS(OriginalStream)和附加数据FD(FileData)混合后输出的混合输出单元MDN(MediaStreamAudioDestinationNode)，设置方法可以采用如下公式：
MDN＝AC.createMediaStreamDestination()
音频处理模块AC上还设置有输入单元，相应的处理可以如下：推流客户端在音频处理模块上设置源数据输入单元和附加数据输入单元；推流客户端通过源数据输入单元和混合输出单元搭建源数据处理线；推流客户端通过附加数据输入单元和混合输出单元搭建附加数据处理线。
在实施中，在设置混合输出单元MDN的同时，推流客户端还可以在音频处理模块AC中通过createMediaStreamSource函数设置用于输入源数据OS的源数据输入单元MSN(MediaStreamAudioSourceNode)，然后，推流客户端通过connect函数将源数据输入单元MSN和混合输出单元MDN连接起来，使得源数据OS可以顺次经过源数据输入单元MSN和混合输出单元MDN，并在各单元进行相应的处理，这样，就形成了一个用于处理源数据OS的单向线路，即通过源数据输入单元MSN和混合输出单元MDN搭建的源数据处理线。
通过createMediaStreamSource函数设置源数据输入单元MSN的方法可以采 用如下公式：
MSN＝AC.createMediaStreamSource(OS)
通过connect函数将源数据输入单元MSN和混合输出单元MDN连接起来的方法可以采用如下公式：
MSN.connect(MDN)
同理，推流客户端在音频处理模块AC中通过createBufferSource函数设置用于输入附加数据FD的附加数据输入单元BSN(AudioBufferSourceNode)，然后，推流客户端同样通过connect函数将附加数据输入单元BSN和混合输出单元MDN连接起来，使得附加数据FD可以顺次经过附加数据输入单元BSN和混合输出单元MDN，并在各单元进行相应的处理，这样，就形成了一个用于处理附加数据FD的单向线路，即通过附加数据输入单元BSN和混合输出单元MDN搭建的附加数据处理线。
通过createBufferSource函数设置附加数据输入单元BSN的方法可以采用如下公式：
BSN＝AC.createBufferSource()
通过connect函数将附加数据输入单元BSN和混合输出单元MDN连接起来的方法可以采用如下公式：
BSN.connect(MDN)
步骤102：推流客户端将源数据和附加数据输入音频处理模块，并通过混合输出单元生成处理数据。
在实施中，在设置完成混合输出单元MDN后，推流客户端获取源数据OS和附加数据FD，并输入音频处理模块AC，然后，推流客户端通过混合输出单元MDN，根据源数据OS和附加数据FD生成处理数据DS(DisposeStream)。
基于上文搭建的源数据处理线和附加数据处理线，步骤102的处理具体可以如下：推流客户端将源数据输入源数据输入单元，并通过源数据处理线，将源数据输入单元的输出数据导入混合输出单元；推流客户端将附加数据输入附加数据输入单元，并通过附加数据处理线，将附加数据输入单元的输出数据导入混合输出单元；推流客户端通过混合输出单元对源数据输入单元的输出数据和附加数据输入单元的输出数据进行处理，生成处理数据。
在实施中，在搭建好源数据处理线和附加数据处理线之后，推流客户端将 源数据OS输入源数据输入单元MSN，根据源数据处理线，推流客户端可以就将源数据输入单元MSN的输出数据，导入混合输出单元MDN。推流客户端将附加数据FD输入附加数据输入单元BSN，根据附加数据处理线，推流客户端就可以将附加数据输入单元BSN的输出数据，导入混合输出单元MDN。混合输出单元MDN在被导入源数据输入单元MSN的输出数据和附加数据输入单元BSN的输出数据后，对两份输出数据进行处理生成处理数据DS。
在对源数据OS进行处理的过程中，可以对源数据OS的音量大小进行调节，相应的处理可以如下：源数据处理线上源数据输入单元和混合输出单元间设置有源音量调节单元；推流客户端通过源数据处理线，将源数据输入单元的输出数据导入源音量调节单元，并将源音量调节单元的输出数据导入混合输出单元。
在实施中，推流客户端还可以在音频处理模块AC中通过createGain函数设置源音量调节单元GN(GainNode)。
通过createGain函数设置源音量调节单元GN的方法可以采用如下公式：
GN＝AC.createGain()
然后，推流客户端将源音量调节单元GN设置在源数据输入单元MSN和混合输出单元MDN之间。具体的，推流客户端通过connect函数将源数据输入单元MSN、源音量调节单元GN和混合输出单元MDN顺次连接起来，构成源数据处理线，从而使得源数据OS被输入源数据输出单元MSN后，源数据输入单元MSN的输出数据，可以被导入源音量调节单元GN，经过源音量调节单元GN处理后的输出数据，可以被导入混合输出单元MDN。
通过connect函数将源数据输入单元MSN、源音量调节单元GN和混合输出单元MDN顺次连接起来的方法可以采用如下公式：
MSN.connect(GN)；
GN.connect(MDN)
在对附加数据FD进行处理的过程中，可以对附加数据FD的音量大小进行调节，相应的处理可以如下：附加数据处理线上附加数据输入单元和混合输出单元间设置有附加音量调节单元；推流客户端通过附加数据处理线，将附加数据输入单元的输出数据导入附加音量调节单元，并将附加音量调节单元的输出数据导入混合输出单元。
在实施中，推流客户端还可以在音频处理模块AC中通过createGain函数设 置附加音量调节单元GN1。
通过createGain函数设置附加音量调节单元GN1的方法可以采用如下公式：
GN1＝AC.createGain()
然后，推流客户端将附加音量调节单元GN1设置在附加数据输入单元BSN和混合输出单元MDN之间。具体的，推流客户端通过connect函数将附加数据输入单元BSN、附加音量调节单元GN1和混合输出单元MDN顺次连接起来，构成附加数据处理线，从而使得附加数据FD被输入附加数据输出单元BSN后，附加数据输入单元BSN的输出数据，可以被导入附加音量调节单元GN1，经过附加音量调节单元GN1处理后的输出数据，可以被导入混合输出单元MDN。
通过connect函数将附加数据输入单元BSN、附加音量调节单元GN和混合输出单元MDN顺次连接起来的方法可以采用如下公式：
BSN.connect(GN1)；
GN1.connect(MDN)
推流服务器得到的附加数据FD还需要进行相应的解码处理，以便于和源数据OS混合，相应的处理可以如下：推流客户端调用音频处理模块的decodeAudioData函数，对附加数据进行解码，将解码后的附加数据输入附加数据输入单元。
在实施中，推流客户端在得到附加数据FD后，调用音频处理模块AC的decodeAudioData函数，对附加数据FD进行解码，得到音频PCM(Pulse Code Modulation，脉冲编码调制)数据DFD，然后将解码后的附加数据DFD输入附加数据输入单元BSN。
调用decodeAudioData函数对附加数据FD进行解码的方法可以采用如下公式：
DFD＝AC.decodeAudioData(FD)
上述根据源数据OS和附加数据FD生成处理数据DS的处理过程可以参考图2所示。
在进行音视频直播的过程中，可以通过推流客户端采集直播的音视频数据作为源数据OS，相应的处理可以如下：推流客户端调用浏览器的数据采集接口采集源数据，并将源数据输入音频处理模块。
在实施中，推流客户端上内置或外接有摄像头和/或麦克风等音视频采集设 备，当用户进行直播时，推流客户端可以调用浏览器提供的数据采集接口，例如getUserMedia接口，启动这些采集设备进行音视频数据采集。推流客户端将采集到的音视频数据作为源数据OS，输入音频处理模块AC。
通过getUserMedia接口采集源数据OS的方法可以如下：
getUserMedia(config)
.then((stream)＝>{
OS＝stream；
})
为了在音视频直播的过程中加入其他音视频，可以通过推流客户端从网络上下载所需的音视频数据作为附加数据FD，相应的处理可以如下：推流客户端调用浏览器的附加数据接口获取附加数据，并将附加数据输入音频处理模块。
在实施中，用户在直播过程中，可能需要增加一些背景音乐、特殊音效等音视频来增加直播效果，因此，可以通过网络获取这些音视频数据。在得到附加数据FD的链接地址后，推流客户端调用浏览器提供的附加数据接口，例如XMLHttpRequest接口，从网络中下载音视频数据作为附加数据FD，并输入音频处理模块AC。
步骤103：推流客户端将源数据和处理数据进行合并，生成混音数据。
在实施中，在通过混音数据输出单元MDN得到处理数据DS后，推流客户端将源数据OS和处理数据DS进行合并，最终生成混音数据MS(MixingStream)。
在本发明的一实施方式中，推流客户端的页面包括推流区域和显示区域。其中推流区域提供一些按钮及文本框等常见网页元素，推流用户可以进行相应操作，以实现混音数据MS的推送。显示区域提供一个页面外部容器(页面div，页面division)，该页面外部容器可以与推流客户端进行交互，用于显示推送给资源服务器的混音数据MS，便于推流客户端本地预览混音数据的效果。推流客户端在生成混音数据MS之后，可以将该混音数据MS实时展示在显示区域中。具体地，在生成混音数据MS后，推流客户端可以创建显示视频(Video)标签，并将混音数据MS加载至显示视频标签中。其中，该显示视频标签的播放属性可以设置为可见，这样，推流用户在外部便可以直接观看到混音数据MS的画面。然后，推流客户端可以创建内部容器(内部div，内部division)，并将显示视频标签中加载的混音数据MS展示于内部容器中。然后，浏览器页面可以通 过回调展示混音数据MS的内部容器，从而将展示混音数据MS的内部容器填充于预先在浏览器页面中设置的页面外部容器内。页面外部容器可以在显示区域中自定义画面展示的效果样式。
由于源数据OS自身包含有音频数据，直接将源数据OS与处理数据DS合并，会导致混音数据MS中含有多个音频数据，进而影响传输过程中对混音数据MS的编解码，故而，生成混音数据MS的处理可以如下：推流客户端调用浏览器的数据处理接口提取源数据中的视频数据；推流客户端调用浏览器的数据处理接口提取处理数据中的音频数据；推流客户端调用浏览器的数据处理接口将视频数据和音频数据合并后，生成混音数据。
在实施中，浏览器的数据处理接口可以是MediaStream接口或者其他接口，在得到处理数据DS后，推流客户端调用浏览器的MediaStream接口，通过getVideoTracks函数，从源数据OS中提取视频数据VT(VideoTracks)。
通过getVideoTracks函数从源数据OS中提取视频数据VT的方法可以采用如下公式：
VT＝OS.getVideoTracks()[0]
同时，推流客户端调用浏览器的MediaStream接口，通过getAudioTracks函数，从处理数据DS中提取音频数据AT(AudioTracks)。
通过getVideoTracks函数从处理数据DS中提取音频数据AT的方法可以采用如下公式：
AT＝DS.getAudioTracks()[0]
之后，推流客户端继续调用浏览器的MediaStream接口将提取得到的视频数据VT和音频数据AT合并，生成混音数据MS。
将提取得到的视频数据VT和音频数据AT合并生成混音数据MS的方法可以采用如下公式：
MS＝MediaStream([AT,VT])
上述根据源数据OS和处理数据DS合并得到混音数据MS的处理过程可以参考图3所示。当混音数据MS被推流客户端推送至资源服务器后，资源服务器会对混音数据MS的音频数据和视频数据重新编解码，合并后得到的混音数据MS中由于只有一个音频数据和一个视频数据，这样，可以提高资源服务器编解码的兼容性。
步骤104：推流客户端将混音数据推送给资源服务器，以使拉流客户端从资源服务器获取混音数据。
在实施中，推流客户端在生成混音数据MS后，可以将混音数据MS推送给资源服务器，这样，拉流客户端与资源服务器建立连接后，就可以从资源服务器上获取混音数据MS。
在本发明的一实施方式中，拉流客户端的页面包括拉流区域和显示区域。其中拉流区域提供一些按钮及文本框等常见网页元素，拉流用户可以进行相应操作，以实现混音数据MS的拉取。具体的，在拉流区域设置好相关流的配置信息后，拉流客户端开始拉流；拉流用户查看显示区域所显示的混音数据MS画面，用音频输出设备如耳机或者电脑扬声器收听是否有混音效果(如同时听到推流端主播说话声和背景音乐声)。
推流客户端在将混音数据MS发送给资源服务器之前，还需要先与资源服务器建立连接，相应的处理可以如下：推流客户端调用浏览器的连接接口创建推送模块；推流客户端通过推送模块与资源服务器建立连接；推流客户端将混音数据输入推送模块，并通过推送模块推送给资源服务器。
在实施中，当需要向资源服务器推送音视频数据时，推流客户端调用浏览器提供的连接接口创建推送模块，通常，推流客户端可以使用RTCPeerConnection接口作为连接接口。推流客户端通过推送模块，基于WebRTC协议与资源服务器建立连接，之后，推流客户端就可以将混音数据MS输入推送模块，然后再通过推送模块将混音数据MS推送给资源服务器，上述过程可以参考图4所示。拉流客户端可以同样基于WebRTC协议与资源服务器建立连接，在资源服务器接收到混音数据MS之后，与资源服务器建立连接的拉流客户端就可以从资源服务器上拉取混音数据MS。
本发明实施例中，推流客户端调用浏览器的音频接口创建音频处理模块，并在所述音频处理模块上设置混合输出单元；所述推流客户端将源数据和附加数据输入所述音频处理模块，并通过所述混合输出单元生成处理数据；所述推流客户端将所述源数据和所述处理数据进行合并，生成混音数据；所述推流客户端将所述混音数据推送给资源服务器，以使拉流客户端从所述资源服务器获取所述混音数据。这样，将源数据和附加数据合并成为一份混音数据，在传输过程中减少了一份数据的传输，降低传输过程中使用的带宽；播放和录制时也 只需要处理一份数据，可以支持更多种类的浏览器进行播放，有利于后期回看。
基于相同的技术构思，本发明实施例还提供了一种基于WebRTC协议的音视频推流客户端，如图5所示，所述推流客户端500用于：
调用浏览器510的音频接口创建音频处理模块511，并在所述音频处理模块511上设置混合输出单元521；
将源数据和附加数据输入所述音频处理模块511，并通过所述混合输出单元521生成处理数据；
将所述源数据和所述处理数据进行合并，生成混音数据；
将所述混音数据推送给资源服务器，以使拉流客户端从所述资源服务器获取所述混音数据。
可选的，如图6所示，所述推流客户端500还用于：
在所述音频处理模块511上设置源数据输入单元522和附加数据输入单元523；
通过所述源数据输入单元522和所述混合输出单元521搭建源数据处理线；
通过所述附加数据输入单元523和所述混合输出单元521搭建附加数据处理线。
可选的，所述推流客户端500具体用于：
将所述源数据输入所述源数据输入单元522，并通过所述源数据处理线，将所述源数据输入单元的输出数据导入所述混合输出单元521；
将所述附加数据输入所述附加数据输入单元523，并通过所述附加数据处理线，将所述附加数据输入单元的输出数据导入所述混合输出单元521；
通过所述混合输出单元521对所述源数据输入单元522的输出数据和附加数据输入单元523的输出数据进行处理，生成所述处理数据。
可选的，如图7所示，所述源数据处理线上所述源数据输入单元522和所述混合输出单元521间设置有源音量调节单元524；
所述推流客户端500具体用于：通过所述源数据处理线，将所述源数据输入单元522的输出数据导入所述源音量调节单元524，并将所述源音量调节单元524的输出数据导入所述混合输出单元521。
可选的，如图7所示，所述附加数据处理线上所述附加数据输入单元523 和所述混合输出单元521间设置有附加音量调节单元525；
所述推流客户端500具体用于：通过所述附加数据处理线，将所述附加数据输入单元523的输出数据导入所述附加音量调节单元525，并将所述附加音量调节单元525的输出数据导入所述混合输出单元521。
可选的，所述推流客户端500具体用于：
调用所述音频处理模块511的decodeAudioData函数，对所述附加数据进行解码，将解码后的附加数据输入所述附加数据输入单元521。
可选的，所述推流客户端500具体用于：
调用所述浏览器510的数据处理接口提取所述源数据中的视频数据；
调用所述浏览器510的数据处理接口提取所述处理数据中的音频数据；
调用所述浏览器510的数据处理接口将所述视频数据和所述音频数据合并后，生成所述混音数据。
可选的，所述推流客户端500还用于：
调用所述浏览器510的数据采集接口采集所述源数据，并将所述源数据输入所述音频处理模块511。
可选的，所述推流客户端500还用于：
调用所述浏览器510的附加数据接口获取所述附加数据，并将所述附加数据输入所述音频处理模块511。
可选的，如图8所示，所述推流客户端500具体用于：
调用所述浏览器510的连接接口创建推送模块512；
通过所述推送模块512与所述资源服务器建立连接；
将所述混音数据输入所述推送模块512，并通过所述推送模块512推送给所述资源服务器。
需要说明的是：上述实施例提供的基于WebRTC协议的音视频推流客户端在实现基于WebRTC协议的音视频推送时，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将推流客户端的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。同时，上述实施例提供的基于WebRTC协议的音视频推流客户端与基于WebRTC协议的音视频推送方法实施例属于同一构思，其具体实现过程 详见方法实施例，这里不再赘述。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到各实施方式可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件。基于这样的理解，上述技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品可以存储在计算机可读存储介质中，如ROM/RAM、磁碟、光盘等，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务端，或者网络设备等)执行各个实施例或者实施例的某些部分所述的方法。
以上所述仅为本发明的较佳实施例，并不用以限制本发明，凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。

Claims (20)

  1. 基于WebRTC协议的音视频推送方法，其特征在于，所述方法包括：
    推流客户端调用浏览器的音频接口创建音频处理模块，并在所述音频处理模块上设置混合输出单元；
    所述推流客户端将源数据和附加数据输入所述音频处理模块，并通过所述混合输出单元生成处理数据；
    所述推流客户端将所述源数据和所述处理数据进行合并，生成混音数据；
    所述推流客户端将所述混音数据推送给资源服务器，以使拉流客户端从所述资源服务器获取所述混音数据。
  2. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    所述推流客户端在所述音频处理模块上设置源数据输入单元和附加数据输入单元；
    所述推流客户端通过所述源数据输入单元和所述混合输出单元搭建源数据处理线；
    所述推流客户端通过所述附加数据输入单元和所述混合输出单元搭建附加数据处理线。
  3. 根据权利要求2所述的方法，其特征在于，所述推流客户端将源数据和附加数据输入所述音频处理模块，并通过所述混合输出单元生成处理数据，包括：
    所述推流客户端将所述源数据输入所述源数据输入单元，并通过所述源数据处理线，将所述源数据输入单元的输出数据导入所述混合输出单元；
    所述推流客户端将所述附加数据输入所述附加数据输入单元，并通过所述附加数据处理线，将所述附加数据输入单元的输出数据导入所述混合输出单元；
    所述推流客户端通过所述混合输出单元对所述源数据输入单元的输出数据和附加数据输入单元的输出数据进行处理，生成所述处理数据。
  4. 根据权利要求3所述的方法，其特征在于，所述源数据处理线上所述源 数据输入单元和所述混合输出单元间设置有源音量调节单元；
    所述推流客户端通过所述源数据处理线，将所述源数据输入单元的输出数据导入所述混合输出单元，包括：
    所述推流客户端通过所述源数据处理线，将所述源数据输入单元的输出数据导入所述源音量调节单元，并将所述源音量调节单元的输出数据导入所述混合输出单元。
  5. 根据权利要求3所述的方法，其特征在于，所述附加数据处理线上所述附加数据输入单元和所述混合输出单元间设置有附加音量调节单元；
    所述推流客户端通过所述附加数据处理线，将所述附加数据输入单元的输出数据导入所述混合输出单元，包括：
    所述推流客户端通过所述附加数据处理线，将所述附加数据输入单元的输出数据导入所述附加音量调节单元，并将所述附加音量调节单元的输出数据导入所述混合输出单元。
  6. 根据权利要求3所述的方法，其特征在于，所述推流客户端将附加数据输入所述附加数据输入单元，包括：
    所述推流客户端调用所述音频处理模块的decodeAudioData函数，对所述附加数据进行解码，将解码后的附加数据输入所述附加数据输入单元。
  7. 根据权利要求1所述的方法，其特征在于，所述推流客户端将所述源数据和所述处理数据进行合并，生成混音数据，包括：
    所述推流客户端调用所述浏览器的数据处理接口提取所述源数据中的视频数据；
    所述推流客户端调用所述浏览器的数据处理接口提取所述处理数据中的音频数据；
    所述推流客户端调用所述浏览器的数据处理接口将所述视频数据和所述音频数据合并后，生成所述混音数据。
  8. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    所述推流客户端调用所述浏览器的数据采集接口采集所述源数据，并将所述源数据输入所述音频处理模块。
  9. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    所述推流客户端调用所述浏览器的附加数据接口获取所述附加数据，并将所述附加数据输入所述音频处理模块。
  10. 根据权利要求1所述的方法，其特征在于，所述推流客户端将所述混音数据推送给资源服务器，包括：
    所述推流客户端调用所述浏览器的连接接口创建推送模块；
    所述推流客户端通过所述推送模块与所述资源服务器建立连接；
    所述推流客户端将所述混音数据输入所述推送模块，并通过所述推送模块推送给所述资源服务器。
  11. 基于WebRTC协议的音视频推流客户端，其特征在于，所述推流客户端用于：
    调用浏览器的音频接口创建音频处理模块，并在所述音频处理模块上设置混合输出单元；
    将源数据和附加数据输入所述音频处理模块，并通过所述混合输出单元生成处理数据；
    将所述源数据和所述处理数据进行合并，生成混音数据；
    将所述混音数据推送给资源服务器，以使拉流客户端从所述资源服务器获取所述混音数据。
  12. 根据权利要求11所述的推流客户端，其特征在于，所述推流客户端还用于：
    在所述音频处理模块上设置源数据输入单元和附加数据输入单元；
    通过所述源数据输入单元和所述混合输出单元搭建源数据处理线；
    通过所述附加数据输入单元和所述混合输出单元搭建附加数据处理线。
  13. 根据权利要求12所述的推流客户端，其特征在于，所述推流客户端具体用于：
    将所述源数据输入所述源数据输入单元，并通过所述源数据处理线，将所述源数据输入单元的输出数据导入所述混合输出单元；
    将所述附加数据输入所述附加数据输入单元，并通过所述附加数据处理线，将所述附加数据输入单元的输出数据导入所述混合输出单元；
    通过所述混合输出单元对所述源数据输入单元的输出数据和附加数据输入单元的输出数据进行处理，生成所述处理数据。
  14. 根据权利要求13所述的推流客户端，其特征在于，所述源数据处理线上所述源数据输入单元和所述混合输出单元间设置有源音量调节单元；
    所述推流客户端具体用于：通过所述源数据处理线，将所述源数据输入单元的输出数据导入所述源音量调节单元，并将所述源音量调节单元的输出数据导入所述混合输出单元。
  15. 根据权利要求13所述的推流客户端，其特征在于，所述附加数据处理线上所述附加数据输入单元和所述混合输出单元间设置有附加音量调节单元；
    所述推流客户端具体用于：通过所述附加数据处理线，将所述附加数据输入单元的输出数据导入所述附加音量调节单元，并将所述附加音量调节单元的输出数据导入所述混合输出单元。
  16. 根据权利要求13所述的推流客户端，其特征在于，所述推流客户端具体用于：
    调用所述音频处理模块的decodeAudioData函数，对所述附加数据进行解码，将解码后的附加数据输入所述附加数据输入单元。
  17. 根据权利要求11所述的推流客户端，其特征在于，所述推流客户端具体用于：
    调用所述浏览器的数据处理接口提取所述源数据中的视频数据；
    调用所述浏览器的数据处理接口提取所述处理数据中的音频数据；
    调用所述浏览器的数据处理接口将所述视频数据和所述音频数据合并后，生成所述混音数据。
  18. 根据权利要求11所述的推流客户端，其特征在于，所述推流客户端还用于：
    调用所述浏览器的数据采集接口采集所述源数据，并将所述源数据输入所述音频处理模块。
  19. 根据权利要求11所述的推流客户端，其特征在于，所述推流客户端还用于：
    调用所述浏览器的附加数据接口获取所述附加数据，并将所述附加数据输入所述音频处理模块。
  20. 根据权利要求11所述的推流客户端，其特征在于，所述推流客户端具体用于：
    调用所述浏览器的连接接口创建推送模块；
    通过所述推送模块与所述资源服务器建立连接；
    将所述混音数据输入所述推送模块，并通过所述推送模块推送给所述资源服务器。
PCT/CN2019/072181 2018-12-19 2019-01-17 基于WebRTC协议的音视频推送方法和推流客户端 WO2020124725A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP19897732.4A EP3902272A4 (en) 2018-12-19 2019-01-17 AUDIO AND VIDEO PUSHING METHOD AND AUDIO AND VIDEO DATA STREAM PUSHING CLIENT BASED ON A WEBRTC PROTOCOL
US17/288,992 US20210409476A1 (en) 2018-12-19 2019-01-17 Method and stream-pushing client terminal for pushing audio and video based on webrtc protocol

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811559034.9 2018-12-19
CN201811559034.9A CN109547844A (zh) 2018-12-19 2018-12-19 基于WebRTC协议的音视频推送方法和推流客户端

Publications (1)

Publication Number Publication Date
WO2020124725A1 true WO2020124725A1 (zh) 2020-06-25

Family

ID=65856146

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/072181 WO2020124725A1 (zh) 2018-12-19 2019-01-17 基于WebRTC协议的音视频推送方法和推流客户端

Country Status (4)

Country Link
US (1) US20210409476A1 (zh)
EP (1) EP3902272A4 (zh)
CN (1) CN109547844A (zh)
WO (1) WO2020124725A1 (zh)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110536143A (zh) * 2019-07-11 2019-12-03 网宿科技股份有限公司 推流方法与电子设备
CN110602522B (zh) * 2019-10-11 2021-08-03 西南民族大学 一种多路实时直播WebRTC流合成方法
CN111314786A (zh) * 2019-12-10 2020-06-19 杭州当虹科技股份有限公司 一种基于WebRTC的PC屏幕分享方法
CN111246253B (zh) * 2020-01-21 2022-04-15 北京世纪好未来教育科技有限公司 视频流传输方法和装置
CN111510738B (zh) * 2020-04-26 2023-08-11 北京字节跳动网络技术有限公司 一种直播中音频的传输方法及装置
CN113766251A (zh) * 2020-06-22 2021-12-07 北京沃东天骏信息技术有限公司 直播连麦的处理方法、系统、服务器及存储介质
CN111770182B (zh) * 2020-06-30 2022-05-31 北京百度网讯科技有限公司 数据推送方法和装置
CN113434110A (zh) * 2021-06-29 2021-09-24 稿定（厦门）科技有限公司 一种基于浏览器的实时音频处理方法、系统、储存装置
CN113542906A (zh) * 2021-07-20 2021-10-22 山东新一代信息产业技术研究院有限公司 一种基于rtsp视频的网页无插件播放方法
CN115002558A (zh) * 2022-05-18 2022-09-02 武汉斗鱼鱼乐网络科技有限公司 一种直播过程中的音视频录制方法及相关设备
CN117459771B (zh) * 2023-12-26 2024-04-19 杭州睿盯科技有限公司 一种网络摄像机录像存储、预览、回放的方法及系统

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106331736A (zh) * 2016-08-24 2017-01-11 武汉斗鱼网络科技有限公司 直播客户端语音处理系统及其处理方法
CN106412663A (zh) * 2016-09-30 2017-02-15 乐视控股（北京）有限公司 直播方法、直播装置以及终端
CN107027050A (zh) * 2017-04-13 2017-08-08 广州华多网络科技有限公司 辅助直播的音视频处理方法及装置

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2833595B1 (en) * 2013-07-31 2018-10-31 Alcatel Lucent Multimedia streams in web applications
US20150350428A1 (en) * 2014-05-29 2015-12-03 CafeX Communications, Ltd. Pushing web and application pages during video/audio calls
CN105872253B (zh) * 2016-05-31 2020-07-07 腾讯科技（深圳）有限公司 一种直播声音处理方法及移动终端
CN106453289B (zh) * 2016-09-29 2020-05-22 广州市千钧网络科技有限公司 一种视频直播的数据传输方法及传输装置

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106331736A (zh) * 2016-08-24 2017-01-11 武汉斗鱼网络科技有限公司 直播客户端语音处理系统及其处理方法
CN106412663A (zh) * 2016-09-30 2017-02-15 乐视控股（北京）有限公司 直播方法、直播装置以及终端
CN107027050A (zh) * 2017-04-13 2017-08-08 广州华多网络科技有限公司 辅助直播的音视频处理方法及装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3902272A4 *

Also Published As

Publication number Publication date
US20210409476A1 (en) 2021-12-30
EP3902272A4 (en) 2022-03-09
EP3902272A1 (en) 2021-10-27
CN109547844A (zh) 2019-03-29

Similar Documents

Publication Publication Date Title
WO2020124725A1 (zh) 基于WebRTC协议的音视频推送方法和推流客户端
CN104780335B (zh) 一种WebRTC P2P音视频通话的方法及装置
TWI440346B (zh) 基於開放架構之域相依即時多語系通信服務
WO2016124101A1 (zh) 一种信息展示方法、装置及系统
CA2592809C (en) Pseudo-rich hybrid phone/browser
EP3525403A1 (en) Video sms message sending and receiving methods and apparatuses thereof, and handheld electronic device
CN106921873A (zh) 直播控制方法及装置
WO2017101298A1 (zh) 音视频信息的输入方法、装置、网络电视及用户设备
CN111092898B (zh) 报文传输方法及相关设备
WO2019062667A1 (zh) 会议内容的传输方法及装置
US20170092274A1 (en) Captioning system and/or method
WO2023125350A1 (zh) 音频数据推送方法、装置、系统、电子设备及存储介质
WO2015117373A1 (zh) 一种语音消息可视化服务的实现方法及装置
WO2015007137A1 (zh) 一种会议电视终端、辅流数据接入方法及计算机存储介质
CN108337556B (zh) 用于播放影音文件的方法和装置
CN102306430B (zh) 实现音视频一体化的自助系统和设备
CN112839192A (zh) 基于浏览器的音视频通讯系统及方法
CN105760126B (zh) 一种多屏文件共享方法及系统
CN114979545A (zh) 多终端的通话方法和存储介质及电子设备
CN102891977A (zh) 基于iPhone平台下实现的语音视频通话方法
JP2007158685A (ja) 動画配信システム
US20140137148A1 (en) System for Managing the Streaming and Recording of Audiovisual Data
CN116566963B (zh) 一种音频处理方法、装置、电子设备和存储介质
KR102546532B1 (ko) 발화 영상 제공 방법 및 이를 수행하기 위한 컴퓨팅 장치
CN108831475A (zh) 一种文本消息提取方法及系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19897732

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2019897732

Country of ref document: EP

Effective date: 20210719