WO2020134791A1 - 流媒体数据的混流方法、装置及存储介质、计算机设备 - Google Patents

流媒体数据的混流方法、装置及存储介质、计算机设备 Download PDF

Info

Publication number
WO2020134791A1
WO2020134791A1 PCT/CN2019/120714 CN2019120714W WO2020134791A1 WO 2020134791 A1 WO2020134791 A1 WO 2020134791A1 CN 2019120714 W CN2019120714 W CN 2019120714W WO 2020134791 A1 WO2020134791 A1 WO 2020134791A1
Authority
WO
WIPO (PCT)
Prior art keywords
streaming media
media data
data
timestamp
mixed
Prior art date
Application number
PCT/CN2019/120714
Other languages
English (en)
French (fr)
Inventor
陈宇辉
Original Assignee
广州华多网络科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 广州华多网络科技有限公司 filed Critical 广州华多网络科技有限公司
Publication of WO2020134791A1 publication Critical patent/WO2020134791A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/2187—Live feed
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/242—Synchronization processes, e.g. processing of PCR [Program Clock References]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/85—Assembly of content; Generation of multimedia applications
    • H04N21/854—Content authoring
    • H04N21/8547—Content authoring involving timestamps for synchronizing content

Definitions

  • the present application relates to the technical field of streaming media data processing. Specifically, the present application relates to a method and apparatus for mixing streaming media data, computer-readable storage media, and computer equipment.
  • the back-end server mixed flow is becoming more and more important in the weak live interactive service due to the advantages of delay, bandwidth, and scalability.
  • the server-side mixed flow technology determines the quality of the mixed picture of the front-end anchor, which directly affects the live broadcast users.
  • the time of the audio and video streams received by the server is different. Each audio and video stream comes from a different time axis.
  • mixing it is easy to cause the mixing to be out of synchronization, which affects the quality of the mixed picture in the live broadcast.
  • the present application proposes a method, device, computer-readable storage medium, and computer device for mixing multiple streaming media data, which can adapt to different frame rates of each source stream according to the output frame rate after mixing, which not only simplifies the logic of traditional mixing, and Improve the synchronization effect of mixed flow.
  • a method for mixing multiple pieces of streaming media data including: acquiring frame data of each piece of streaming media data in multiple pieces of streaming media data; determining an output frame rate after mixing the multiple pieces of streaming media data, and according to the output
  • the frame rate is used to calculate the frame interval of the streaming media data after mixing; according to the current mixing timestamp and the frame interval, the mixing timestamp of the next mixing is determined; when the next mixing is performed, according to the determined mixing timestamp From the streaming media data, the frame data corresponding to each streaming media data is extracted and mixed.
  • the acquiring frame data of each piece of streaming media data in the plurality of pieces of streaming media data includes receiving multiple pieces of streaming media data and dividing each piece of streaming media data in the plurality of pieces of streaming media data into multiple pieces.
  • Frame data determine the jitter buffer length of the corresponding streaming media data according to the jitter of each piece of streaming media data, and set the buffer area of the corresponding streaming media data according to the jitter buffer length; convert each piece of streaming media data
  • the frame data of each is buffered into the corresponding buffer area, and the buffering time of the frame data in each of the buffer areas is controlled to be the same; the frame data of each piece of streaming media data is obtained from each of the buffer areas separately.
  • the method further includes: mapping the display timestamp of the frame data of each piece of streaming media data to the same time coordinate system; the next time mixing the stream, according to the determined mixing timestamp
  • the mixing time stamp extracts frame data corresponding to each piece of streaming media data from the multiple pieces of streaming media data for mixing, including: displaying a time stamp of the frame data of each piece of streaming media data in the same time coordinate system , Respectively obtaining frame data within a preset time range from the mixed stream timestamp and displaying a time stamp less than or equal to the mixed stream timestamp from the buffer area corresponding to each piece of streaming media data for mixed stream.
  • the mixed streaming of the frame data includes: if the buffer corresponding to any one of the multiple streaming media data does not exist in the preset time range from the mixed streaming timestamp and the display timestamp is less than or Frame data equal to the mixed stream timestamp, then wait for the preset duration to obtain the next mixed stream timestamp, and when the next mixed stream timestamp is reached, respectively from the cache area corresponding to each piece of streaming media data Obtaining frame data within a preset time range from the next mixed flow timestamp and displaying a timestamp less than or equal to the next mixed flow timestamp for mixed flow.
  • the preset waiting time does not exceed the time when the buffer area in the multiple pieces of streaming media data reaches the maximum input stream.
  • the method further includes: acquiring the first header of each piece of streaming media data from the frame data of each piece of streaming media data Frame data; the display timestamp of the first frame data with the largest display timestamp is taken as the mixed stream start timestamp.
  • the multiple pieces of streaming media data include multiple pieces of video streaming data; in the next mixed streaming, according to the determined mixed streaming timestamp, each corresponding piece of streaming media data is extracted from the multiple pieces of streaming media data respectively
  • the mixed frame data includes: separately extracting frame data corresponding to each piece of video stream data from the multiple pieces of video stream data; according to the parameters displayed after the multiple pieces of video stream data are mixed and streaming, the corresponding frame data Perform layout; use the mixed flow timestamp each time as the display timestamp output after the mixed flow, and mix the frame data after the layout.
  • a mixed streaming device for multiple pieces of streaming media data including: a first acquisition module for acquiring frame data of each piece of streaming media data in multiple pieces of streaming media data; and a calculation module for determining the plurality of pieces of streaming media data The output frame rate after the mixed flow is calculated, and the frame interval of the streamed media data after the mixed flow is calculated according to the output frame rate; the second acquisition module is used to determine the mixed flow of the next mixed flow according to the mixed flow timestamp of the current mixed flow and the frame interval Timestamp; mixed flow module, used for extracting frame data corresponding to each piece of streaming media data from the multiple pieces of streaming media data according to the determined mixed flow timestamp for mixed flow in the next mixed flow.
  • a computer-readable storage medium having computer instructions stored thereon; the instructions are used to perform a method for mixing multiple streams of media data according to any of the foregoing embodiments.
  • a computer device including: one or more processors; a memory; one or more application programs, wherein the one or more application programs are stored in the memory and configured to be configured by the one or more The processor executes, and the one or more application programs are configured to: execute the method for mixing multiple pieces of streaming media data according to any one of the foregoing embodiments.
  • the frame interval of the streaming media data output by the mixed streams is calculated.
  • the time stamp of the next mixed flow of the system can be determined by the frame interval and the mixed flow time stamp of the current mixed flow. Determine the time to reach the next mixing time, and extract the corresponding frame data from multiple pieces of streaming media data to mix according to the determined mixing time stamp. Therefore, the above method can adapt different frame rates of each source stream according to the output frame rate after mixing, which not only simplifies the logic of traditional mixing, but also improves the synchronization effect of mixing.
  • FIG. 1 is a structural block diagram of a mixed-flow system in an embodiment of a mixed-flow method for multiple pieces of streaming media data provided by this application;
  • FIG. 2 is a flowchart of a method in an embodiment of a method for mixing multiple streams of media data provided by this application;
  • FIG. 3 is a flowchart of a method in an embodiment of step S100 provided by the present application.
  • FIG. 4 is a schematic flowchart of a specific embodiment of a method for mixing multiple streams of media data provided by this application;
  • FIG. 5 is a schematic structural diagram of an embodiment of a device for mixing streaming media data provided by the present application.
  • FIG. 6 is a schematic structural diagram of an embodiment of a computer device provided by this application.
  • the present application provides a method for mixing streams of multiple pieces of streaming media data.
  • the mixed streaming method of multiple pieces of streaming media data can be applied to multiple anchor audio and video links in a live streaming media scenario.
  • a mixed flow system for a mixed flow method of multiple pieces of streaming media data will be first described in advance. See Figure 1:
  • multiple anchors upload the collected streaming media data to the streaming media upload server through the anchor client.
  • Each streaming media upload server then transmits the streaming media data of the corresponding anchor to the streaming media mixing server for mixing.
  • the streaming media mixing server sends the mixed streaming media data to a CDN (Content Distribution Network) for download and viewing by live viewers through the viewer client.
  • the streaming data uploaded by multiple anchors to the streaming upload server may correspond to different timestamps.
  • the streaming media mixing server performs mixing, it may cause the phenomenon that multiple streaming media after mixing are not synchronized.
  • the mixing method of multiple streaming media data provided by the present application can adapt to different frame rates of each source stream according to the output frame rate after mixing, which not only simplifies the logic of traditional mixing, but also improves the synchronization effect of mixing.
  • the method for mixing streams of multiple pieces of streaming media data includes the following steps:
  • the streaming media mixing server reads multiple pieces of streaming media data for mixing from each streaming media upload server.
  • the streaming media data may be audio streaming data, video streaming data, or audio and video streaming data.
  • each piece of streaming media data is divided into multiple frame data.
  • each piece of streaming media data can be divided into multiple frame data related to time series according to the time axis coordinate system.
  • the frame data of each piece of streaming media data is separately stored in the buffer area of the server. The length of the buffer area and the size of the streaming media data in the buffer area may be determined according to the jitter of each piece of streaming media data.
  • the method further includes: acquiring the first frame data of each stream media data from the frame data of each stream media data; displaying the display time of the first frame data with the largest time stamp The stamp is used as the mixing start timestamp of mixing.
  • the mixing start timestamp of the first mixing of the streaming media mixing server is determined by the display timestamp of the first frame of each piece of streaming media data among the multiple pieces of streaming media data being mixed.
  • the streaming media mixing server obtains the first frame data of each piece of streaming media data, and obtains the display timestamp of each first frame data, and uses the display timestamp of the first frame data with the maximum display timestamp as the streaming media mixing server for the first mixing flow mixing Start timestamp.
  • S200 Determine an output frame rate after mixing the multiple pieces of streaming media data, and calculate a frame interval of the mixed streaming media data according to the output frame rate.
  • the streaming media mixing server is set in advance according to the current state of the system.
  • the frame interval of the streaming media data output by the streaming media mixing server after the mixing is calculated by the output frame rate after the mixing.
  • the streaming media mixing server sets the output frame rate of the mixed streaming media data to output 20 frames of data per second. At this time, it can be calculated that the frame interval of streaming media data after mixing is 50 milliseconds.
  • S300 Determine a mixed flow timestamp of the next mixed flow according to the mixed flow timestamp of the current mixed flow and the frame interval.
  • the streaming media mixing server when it obtains the frame interval of the streaming media data after the mixing, it can determine the mixing timestamp for the next mixing according to the mixing timestamp of the current mixing. Specifically, the current timestamp and the frame interval of the streaming media data after mixing can be accumulated, and the accumulated timestamp can be used as the mixing timestamp of the next mixing. For example, if the time stamp of the current mixed stream is 100 milliseconds and the frame interval of the streaming media data after the mixed stream is 50 milliseconds, the mixed stream timestamp of the next mixed stream is 150 milliseconds.
  • the streaming media mixing server determines the mixing timestamp of each next mixing flow according to the current timestamp and the frame interval of the streaming media data after mixing, and when the time reaches the next mixing timestamp, according to the determined
  • the next mixing timestamp extracts frame data corresponding to each piece of streaming media data from multiple pieces of streaming media data used for mixing and mixing, respectively. Specifically, it may be first determined whether the frame data corresponding to each piece of streaming media data is cached in the cache area, and when it is determined that the frame data corresponding to each piece of streaming media data is cached in the cache area, from multiple streams for mixing In the media data, the frame data corresponding to each piece of streaming media data is extracted and mixed. Otherwise, the mixed streaming data is not mixed at the determined mixed streaming timestamp.
  • the multiple pieces of streaming media data include multiple pieces of video stream data; step S400 includes: extracting frame data corresponding to each piece of video stream data from the multiple pieces of video stream data; according to the multiple The parameters of the video stream data are displayed after being mixed, and the corresponding frame data is laid out; each time the mixed time stamp is used as the display time stamp output after the mixed flow, the mixed frame data is mixed and flowed.
  • the streaming media mixing server mixes multiple pieces of video streaming data.
  • the streaming media mixing server separately extracts the frame data corresponding to each video stream data from multiple video stream data for mixing, the frame data corresponding to each video stream data is adjusted according to each display parameter displayed to the audience client after mixing the stream Layout to mix flow at each time of mixing flow timestamp.
  • the step S100 includes:
  • S110 Receive multiple pieces of streaming media data, and divide each piece of streaming media data into multiple pieces of frame data.
  • S120 Determine a jitter buffer length of the corresponding streaming media data according to the jitter situation of each piece of streaming media data, and set a buffer area of the corresponding stream media data according to the jitter buffer length.
  • S130 Cache the frame data of each piece of streaming media data into a corresponding buffer area respectively, and control the frame data buffer time in each of the buffer areas to be the same.
  • S140 Obtain frame data of each piece of streaming media data from each of the buffer areas.
  • each streaming media data is divided into multiple pieces of frame data.
  • the jitter buffer length of the buffer area in the system that buffers the streaming media data is determined according to the jitter of each piece of streaming media data.
  • each stream media data received is buffered into a corresponding buffer area.
  • the buffer time of each frame data in the buffer area is controlled in each buffer area.
  • streaming media data A and streaming media data B The frame data of the streaming media data A cached in the buffer area includes frame data a1 and frame data a2.
  • the frame data of the streaming media data B cached in the buffer area includes frame data b1 and frame data b2.
  • the frame data a1, frame data a2, frame data b1, and frame data b2 have the same buffering time in the buffer area.
  • the streaming media mixing server extracts the frame data of each piece of streaming media data from each buffer area to mix the extracted frame data.
  • the method for mixing streams of multiple pieces of streaming media data further includes: mapping the display time stamp of the frame data of each piece of streaming media data to the same time coordinate system.
  • step S400 includes: separately obtaining the distance from the mixed stream from the buffer area corresponding to each piece of streaming media data according to the display time stamp of the frame data of each piece of streaming media data in the same time coordinate system Frame data within the preset time range of the time stamp and displaying the time stamp less than or equal to the mixed time stamp are mixed.
  • the streaming media mixing server maps the display time stamps of the frame data of multiple pieces of streaming media data to the same time coordinate system to set the same time reference standard for all the frame data, so that the display time stamps of the frame data are removed from the cache area
  • the extracted frame data is more accurate, thereby improving the quality of the mixed picture displayed to the audience client after the mixed flow.
  • the acquiring the buffer area corresponding to each piece of streaming media data separately within a preset time range from the mixed stream timestamp and the display timestamp is less than Or mixing frame data equal to the mixed stream timestamp, including: if the buffer area corresponding to any one of the multiple stream media data does not exist within the preset time range from the mixed stream timestamp Frame data whose timestamp is less than or equal to the mixed stream timestamp, then wait for the preset duration to obtain the next mixed stream timestamp, and when the next mixed stream timestamp is reached, each stream media data In the corresponding buffer area, frame data within a preset time range from the next mixed flow timestamp and having a display timestamp less than or equal to the next mixed flow timestamp is acquired for mixed flow. Wherein, the waiting preset duration does not exceed the duration when the buffer area in the multiple pieces of streaming media data reaches the maximum input stream.
  • the streaming media mixing server first determines whether each stream media data used for mixing in the cache module is cached within the preset time range of the current mixing timestamp and the display time when the confirmation time reaches the mixing timestamp Frame data with a time stamp less than or equal to the current mixed stream timestamp. If yes, the frame data within the preset time range of the current mixed stream timestamp and the display timestamp is less than or equal to the current mixed stream timestamp is directly extracted from the buffer area corresponding to each piece of streaming media data for mixed flow. If not, the streaming media mixing server does not perform the mixing operation at the current mixing timestamp, and obtains the next mixing timestamp to perform the mixing operation.
  • the jitter buffer length of the buffer area set by the streaming media mixing server for each piece of streaming media data is limited, so the time that the streaming media mixing server waits for mixing should not exceed the time when any buffer reaches the maximum input stream.
  • the frame interval of the streaming media data output by the mixed streams is calculated.
  • the time stamp of the next mixed flow of the system can be determined by the frame interval and the mixed flow time stamp of the current mixed flow. Determine the time to reach the next mixing time, and extract the corresponding frame data from multiple pieces of streaming media data for mixing according to the determined mixing time stamp. Therefore, the above method can adapt different frame rates of each source stream according to the output frame rate after mixing, which not only simplifies the logic of traditional mixing, but also improves the synchronization effect of mixing.
  • the server receives the video data of each stream, and each video data is assembled into video frame data, and the global jitter buffer length of each data stream is calculated according to the reception jitter of each data stream. Unify the jitter buffer length of each data stream.
  • the video frame data is extracted from the jitter buffer, it is decoded sequentially.
  • the PTS Presentation Time Stamp display time stamp
  • the error is finely adjusted and corrected according to the received PTS of the video frame data of each data stream. Extract the first frame data of each data stream, and the maximum PTS is used as the mixed stream start timestamp.
  • the frame interval is calculated according to the output frame rate after the mixed flow, and the current frame mixed flow time and frame interval are used to calculate the next frame mixed flow timestamp.
  • the frame with the PTS less than or equal to the mixed flow timestamp and closest to the timestamp is taken from each data stream as the frame to be mixed. At this time, the same frame may be reused. Further, it is determined whether the number of frames to be mixed matches the number of input streams. If there is no frame and the maximum input stream length in the buffer area does not exceed the maximum waiting time, wait for a short period of time and perform the mixing stream timestamp calculation step.
  • the frames to be mixed are laid out according to the parameters of the client client's mixed picture, and the mixed flow timestamp is re-encoded as the output PTS to output. At the same time update the current mixed picture timestamp. If the current time does not reach the end of the mixed flow, the mixed flow is continued. If the current time has reached the end of the mixed flow, the mixed flow operation is ended.
  • the mixing method of multiple streaming media data described in the above specific embodiments mixes multiple data streams with simple and unified logic, and automatically complements or deletes each source stream according to the target frame rate (output frame rate after mixing) Frame, which greatly simplifies the mixed flow logic and improves the synchronization effect of the mixed picture of the audience client.
  • the present application also provides a mixing device for multiple pieces of streaming media data.
  • the multiple streaming media data mixing device includes a first acquisition module 10, a calculation module 20, a second acquisition module 30, and a mixing module 40.
  • the first obtaining module 10 is used to obtain the frame data of each piece of streaming media data in the plurality of pieces of streaming media data.
  • the streaming media mixing server reads multiple pieces of streaming media data for mixing from each streaming media upload server.
  • the streaming media data may be audio streaming data, video streaming data, or audio and video streaming data.
  • each piece of streaming media data is divided into multiple frame data.
  • each piece of streaming media data can be divided into multiple frame data related to time series according to the time axis coordinate system.
  • the frame data of each piece of streaming media data is stored separately in the cache area of the server.
  • the length of the buffer area and the size of the streaming media data in the buffer area may be determined according to the jitter of each piece of streaming media data.
  • the calculation module 20 is used to determine the output frame rate of the mixed streaming media data, and calculate the frame interval of the mixed streaming media data according to the output frame rate.
  • the streaming media mixing server is set in advance according to the current state of the system.
  • the frame interval of the streaming media data output by the streaming media mixing server after the mixing is calculated by the output frame rate after the mixing.
  • the streaming media mixing server sets the output frame rate of the mixed streaming media data to output 20 frames of data per second. At this time, it can be calculated that the frame interval of streaming media data after mixing is 50 milliseconds.
  • the second obtaining module 30 is configured to determine the mixing timestamp of the next mixing flow according to the mixing timestamp of the current mixing flow and the frame interval.
  • the streaming media mixing server when it obtains the frame interval of the streaming media data after the mixing, it can determine the mixing timestamp for the next mixing according to the mixing timestamp of the current mixing.
  • the current timestamp and the frame interval of the streaming media data after mixing can be accumulated, and the accumulated timestamp can be used as the mixing timestamp of the next mixing. For example, if the time stamp of the current mixed stream is 100 milliseconds and the frame interval of the streaming media data after the mixed stream is 50 milliseconds, the mixed stream timestamp of the next mixed stream is 150 milliseconds.
  • the mixing module 40 is configured to extract frame data corresponding to each piece of streaming media data from the plurality of pieces of streaming media data according to the determined mixing timestamp for mixing during the next mixing.
  • the streaming media mixing server determines the mixing timestamp of each next mixing flow according to the current timestamp and the frame interval of the streaming media data after mixing, and when the time reaches the next mixing timestamp, according to the determined
  • the next mixing timestamp extracts frame data corresponding to each piece of streaming media data from multiple pieces of streaming media data used for mixing and mixing, respectively.
  • the frame data corresponding to each piece of streaming media data may be first determined whether the frame data corresponding to each piece of streaming media data is cached in the cache area, and when it is determined that the frame data corresponding to each piece of streaming media data is cached in the cache area, from multiple streams for mixing In the media data, the frame data corresponding to each piece of streaming media data is extracted and mixed. Otherwise, the mixed streaming data is not mixed at the determined mixed streaming timestamp.
  • each module in the multiple streaming media data mixing device provided by this application is also used to perform the operations corresponding to each step in the multiple streaming media data mixing method described in this application. No more detailed explanation.
  • the present application also provides a computer-readable storage medium.
  • a computer program is stored on the storage medium; when the computer program is executed by the processor, the method for mixing multiple streams of media data according to any of the foregoing embodiments is implemented.
  • the storage medium may be a memory.
  • internal memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or random access memory.
  • the external memory may include hard disks, floppy disks, ZIP disks, U disks, magnetic tapes, etc.
  • the storage media disclosed in this application include but are not limited to these types of memories.
  • the memory disclosed in this application is only an example and not a limitation.
  • a computer device includes: one or more processors; memory; one or more application programs. Wherein the one or more application programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more application programs are configured to execute any one of the foregoing embodiments Mixed streaming method of multiple streaming media data.
  • the computer device in this embodiment may be a server, a personal computer, and a network device.
  • the device includes devices such as a processor 603, a memory 605, an input unit 607, and a display unit 609.
  • the memory 605 may be used to store application programs 601 and various functional modules.
  • the processor 603 runs the application programs 601 stored in the memory 605 to execute various functional applications and data processing of the device.
  • the memory may be internal memory or external memory, or include both internal memory and external memory.
  • the internal memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or random access memory.
  • ROM read-only memory
  • PROM programmable ROM
  • EPROM electrically programmable ROM
  • EEPROM electrically erasable programmable ROM
  • flash memory or random access memory.
  • the external memory may include hard disks, floppy disks, ZIP disks, U disks, magnetic tapes, etc.
  • the memories disclosed in this application include but are not limited to these types of memories.
  • the memory disclosed in this application is only an example and not a limitation.
  • the input unit 607 is used to receive an input of a signal and a keyword input by a user.
  • the input unit 607 may include a touch panel and other input devices.
  • the touch panel can collect the user's touch operations on or near it (such as the user's operation with any suitable objects or accessories such as fingers, stylus, etc. on or near the touch panel), and according to the preset
  • the program drives the corresponding connection device; other input devices may include but are not limited to one or more of a physical keyboard, function keys (such as playback control keys, switch keys, etc.), trackball, mouse, joystick, etc.
  • the display unit 609 can be used to display information input by the user or information provided to the user and various menus of the computer device.
  • the display unit 609 may take the form of a liquid crystal display, an organic light emitting diode, or the like.
  • the processor 603 is the control center of the computer equipment. It uses various interfaces and lines to connect the various parts of the entire computer. By running or executing the software programs and/or modules stored in the memory 605, and calling the data stored in the memory, the execution Various functions and processing data.
  • the device includes one or more processors 603, and one or more memories 605, and one or more applications 601.
  • the one or more application programs 601 are stored in the memory 605 and configured to be executed by the one or more processors 603, and the one or more application programs 601 are configured to perform the operations described in the above embodiments Mixed streaming method of multiple streaming media data.
  • each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module.
  • the above integrated modules can be implemented in the form of hardware or software function modules. If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
  • a person of ordinary skill in the art may understand that all or part of the steps for implementing the above-mentioned embodiments may be completed by hardware, or by a program instructing related hardware.
  • the program may be stored in a computer-readable storage medium, and the storage medium may include Memory, magnetic disk or optical disk, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Databases & Information Systems (AREA)
  • Information Transfer Between Computers (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

本申请提供一种多条流媒体数据的混流方法、装置及计算机可读存储介质、计算机设备，所述方法包括：获取多条流媒体数据中每条流媒体数据的帧数据；确定将所述多条流媒体数据进行混流后的输出帧率，根据所述输出帧率计算混流后流媒体数据的帧间隔；根据当前混流的混流时间戳和所述帧间隔，确定下一次混流的混流时间戳；在下一次混流时，根据确定的混流时间戳分别从所述多条流媒体数据中提取每条流媒体数据对应的帧数据进行混流。上述方法能够根据混流后输出帧率自适应各条源流的不同帧率，不仅简化传统混流的逻辑，并且提高了混流的同步效果。

Description

流媒体数据的混流方法、装置及存储介质、计算机设备 技术领域
本申请涉及流媒体数据处理技术领域，具体而言，本申请涉及一种多条流媒体数据的混流方法、装置及计算机可读存储介质、计算机设备。
背景技术
随着流媒体分发技术的发展，直播场景下音视频连麦得到广泛应用。现下多个主播连麦进行直播的视频互动模式得到广大观众的喜爱。多个主播建立同一直播间，通过连麦方式进行互动，观众在同一直播间可观看到多个主播的直播互动。要实现多个主播的连麦直播，后台必须将多个主播的音视频流进行混流从而实现直播间的混画。
后台服务端混流由于延时、带宽、扩展性方面的优势在直播弱交互业务中的作用越来越重要，服务端混流技术决定了前端主播的混画质量，混画质量直接影响到直播的用户体验。然而，服务器端接收到的音视频流时间不一样，每条音视频流来自不同的时间轴，在进行混流时，容易造成混流不同步，从而影响直播间的混画质量。
发明内容
本申请提出一种多条流媒体数据的混流方法、装置及计算机可读存储介质、计算机设备，能够根据混流后输出帧率自适应各条源流的不同帧率，不仅简化传统混流的逻辑，并且提高了混流的同步效果。
本申请提供以下方案：
一种多条流媒体数据的混流方法，包括：获取多条流媒体数据中每条流媒体数据的帧数据；确定将所述多条流媒体数据进行混流后的输出帧率，根据所述输出帧率计算混流后流媒体数据的帧间隔；根据当前混流的混流时间戳和所述帧间隔，确定下一次混流的混流时间戳；在下一次混流时，根据确定的混流时间戳分别从所述多条流媒体数据中提取每条流媒体数据对应的帧数据进行混流。
在一实施例中，所述获取多条流媒体数据中每条流媒体数据的帧数据，包括：接收多条流媒体数据，将所述多条流媒体数据中每条流媒体数据划分为多条帧数据； 根据所述每条流媒体数据的抖动情况确定对应流媒体数据的抖动缓存长度，根据所述抖动缓存长度设置所述对应流媒体数据的缓存区；将所述每条流媒体数据的帧数据分别缓存进对应的缓存区，控制每个所述缓存区中帧数据缓存时间相同；分别从每个所述缓存区中获取所述每条流媒体数据的帧数据。
在一实施例中，所述方法还包括：将所述每条流媒体数据的帧数据的显示时间戳映射到同一时间坐标系；所述在下一次混流时，根据确定的混流时间戳根据每次所述混流时间戳分别从所述多条流媒体数据中提取每条流媒体数据对应的帧数据进行混流，包括：根据所述同一时间坐标系中每条流媒体数据的帧数据的显示时间戳，分别从所述每条流媒体数据对应的所述缓存区中获取距离所述混流时间戳的预设时间范围内的且显示时间戳小于或等于所述混流时间戳的帧数据进行混流。
在一实施例中，所述分别从所述每条流媒体数据对应的所述缓存区中获取距离所述混流时间戳的预设时间范围内的且显示时间戳小于或等于所述混流时间戳的帧数据进行混流，包括：若所述多条流媒体数据中任意一条流媒体数据对应的缓存区中不存在所述距离所述混流时间戳的预设时间范围内的且显示时间戳小于或等于所述混流时间戳的帧数据，则等待预设时长后获取下一次混流时间戳，在到达所述下一次混流时间戳时，分别从所述每条流媒体数据对应的所述缓存区中获取距离所述下一次混流时间戳的预设时间范围内的且显示时间戳小于或等于所述下一次混流时间戳的帧数据进行混流。
在一实施例中，所述等待预设时长不超过所述多条流媒体数据中缓存区达到最大输入流时的时长。
在一实施例中，所述获取多条流媒体数据中每条流媒体数据的帧数据之后，还包括：从所述每条流媒体数据的帧数据中获取所述每条流媒体数据的首帧数据；将显示时间戳最大的首帧数据的显示时间戳作为混流的混流起始时间戳。
在一实施例中，所述多条流媒体数据包括多条视频流数据；所述在下一次混流时，根据确定的混流时间戳分别从所述多条流媒体数据中提取每条流媒体数据对应的帧数据进行混流，包括：分别从所述多条视频流数据中提取每条视频流数据对应的帧数据；根据所述多条视频流数据混流后进行展示的参数对所述对应的帧数据进行布局；将每次所述混流时间戳作为混流后输出的显示时间戳，对布局后的帧数据进行混流。
一种多条流媒体数据的混流装置，包括：第一获取模块，用于获取多条流媒体数据中每条流媒体数据的帧数据；计算模块，用于确定将所述多条流媒体数据进行 混流后的输出帧率，根据所述输出帧率计算混流后流媒体数据的帧间隔；第二获取模块，用于根据当前混流的混流时间戳和所述帧间隔，确定下一次混流的混流时间戳；混流模块，用于在下一次混流时，根据确定的混流时间戳分别从所述多条流媒体数据中提取每条流媒体数据对应的帧数据进行混流。
一种计算机可读存储介质，其上存储有计算机指令；所述指令用于执行上述任一实施例所述的多条流媒体数据的混流方法。
一种计算机设备，包括：一个或多个处理器；存储器；一个或多个应用程序，其中所述一个或多个应用程序被存储在所述存储器中并被配置为由所述一个或多个处理器执行，所述一个或多个应用程序配置用于：执行根据上述任一实施例所述的多条流媒体数据的混流方法。
上述多条流媒体数据的混流方法，在确定多条流媒体数据混流后的输出帧率之后，计算混流输出的流媒体数据的帧间隔。通过帧间隔和当前混流的混流时间戳可确定系统下一次混流的时间戳。确定时间到达下一次混流时间，根据确定出的混流时间戳分别从多条流媒体数据中提取对应的帧数据进行混流。因此，上述方法能够根据混流后输出帧率自适应各条源流的不同帧率，不仅简化传统混流的逻辑，并且提高了混流的同步效果。
本申请附加的方面和优点将在下面的描述中部分给出，这些将从下面的描述中变得明显，或通过本申请的实践了解到。
附图说明
本申请上述的和/或附加的方面和优点从下面结合附图对实施例的描述中将变得明显和容易理解，其中：
图1为本申请提供的一种多条流媒体数据的混流方法的一实施例中的混流系统结构框图；
图2为本申请提供的一种多条流媒体数据的混流方法的一实施例中的方法流程图；
图3为本申请提供的步骤S100的一实施例中的方法流程图；
图4为本申请提供的一种多条流媒体数据的混流方法的一具体实施例中的流程示意图；
图5为本申请提供的一种多条流媒体数据的混流装置的一实施例中的结构示意图；
图6为本申请提供的一种计算机设备的一实施例中的结构示意图。
具体实施方式
下面详细描述本申请的实施例，所述实施例的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的，仅用于解释本申请，而不能解释为对本申请的限制。
本技术领域技术人员可以理解，除非特意声明，这里使用的单数形式“一”、“一个”、“所述”和“该”也可包括复数形式，这里使用的“第一”、“第二”仅用于区别同一技术特征，并不对该技术特征的顺序和数量等加以限定。应该进一步理解的是，本申请的说明书中使用的措辞“包括”是指存在所述特征、整数、步骤、操作、元件和/或组件，但是并不排除存在或添加一个或多个其他特征、整数、步骤、操作、元件、组件和/或它们的组。
本技术领域技术人员可以理解，除非另外定义，这里使用的所有术语(包括技术术语和科学术语)，具有与本申请所属领域中的普通技术人员的一般理解相同的意义。还应该理解的是，诸如通用字典中定义的那些术语，应该被理解为具有与现有技术的上下文中的意义一致的意义，并且除非像这里一样被特定定义，否则不会用理想化或过于正式的含义来解释。
本申请提供一种多条流媒体数据的混流方法。该多条流媒体数据的混流方法可应用在流媒体直播场景下的多个主播音视频连麦。以下先对多条流媒体数据的混流方法的混流系统进行先导性说明。参见图1所示：
在进行多个主播连麦混画时，多个主播通过主播客户端将采集到的流媒体数据上传到流媒体上传服务器。每个流媒体上传服务器再将对应主播的流媒体数据分别传送到流媒体混流服务器进行混流。进一步地，流媒体混流服务器将混流后的流媒体数据发送到CDN(内容分发网络)，以由直播观众通过观众客户端下载观看。其中，多个主播分别上传到流媒体上传服务器的流媒体数据可能对应不同的时间戳。由流媒体混流服务器进行混流时，有可能导致混流后的多个流媒体不同步的现象。
本申请提供的多条流媒体数据的混流方法，能够根据混流后输出帧率自适应各条源流的不同帧率，不仅简化传统混流的逻辑，并且提高了混流的同步效果。在一实施例中，如图2所示，该多条流媒体数据的混流方法，包括以下步骤：
S100，获取多条流媒体数据中每条流媒体数据的帧数据。
在本实施例中，流媒体混流服务器从每个流媒体上传服务器中读取出多条用于进行混流的流媒体数据。流媒体数据可以是音频流数据，视频流数据，或者音视频流数据。进一步地，将每条流媒体数据划分为多个帧数据。其中，可以按照时间轴坐标系将每条流媒体数据划分为时序相关的多个帧数据。每条流媒体数据的帧数据单独存储在服务器的缓存区。缓存区的长度以及缓存区中的流媒体数据的大小，可根据每条流媒体数据的抖动情况确定。
在一实施例中，步骤S100之后，还包括：从所述每条流媒体数据的帧数据中获取所述每条流媒体数据的首帧数据；将显示时间戳最大的首帧数据的显示时间戳作为混流的混流起始时间戳。
在该实施例中，流媒体混流服务器进行初次混流的混流起始时间戳由进行混流的多条流媒体数据中，每条流媒体数据中的首帧数据的显示时间戳决定。流媒体混流服务器获取每条流媒体数据的首帧数据，并获取每个首帧数据的显示时间戳，将最大显示时间戳的首帧数据的显示时间戳作为流媒体混流服务器进行初次混流的混流起始时间戳。
S200，确定将所述多条流媒体数据进行混流后的输出帧率，根据所述输出帧率计算混流后流媒体数据的帧间隔。
在本实施例中，对于每个混流后的流媒体数据的输出帧率，流媒体混流服务器根据系统当前状态提前设置。通过混流后的输出帧率计算混流后流媒体混流服务器输出的流媒体数据的帧间隔。例如，流媒体混流服务器设置混流后的流媒体数据的输出帧率为1秒输出20帧数据。此时，可计算得到混流后流媒体数据的帧间隔为50毫秒。
S300，根据当前混流的混流时间戳和所述帧间隔，确定下一次混流的混流时间戳。
在本实施例中，流媒体混流服务器获取到混流后流媒体数据的帧间隔时，可根据当前混流的混流时间戳，确定下一次进行混流的混流时间戳。具体地，可将当前时间戳与混流后流媒体数据的帧间隔进行累加，将累加得到的时间戳作为下一次混流的混流时间戳。例如，当前混流的时间戳为100毫秒，混流后流媒体数据的帧间隔为50毫秒，则下次混流的混流时间戳为150毫秒。
S400，在下一次混流时，根据确定的混流时间戳分别从所述多条流媒体数据中提取每条流媒体数据对应的帧数据进行混流。
在本实施例中，流媒体混流服务器根据当前时间戳和混流后流媒体数据的帧间 隔确定出每一个下一次混流的混流时间戳之后，当时间到达下一次混流时间戳时，根据确定出的下一次混流时间戳分别从多条用于进行混流的流媒体数据中提取每条流媒体数据对应的帧数据进行混流。具体地，可先判断缓存区中是否缓存有每条流媒体数据对应的帧数据，在确定缓存区中均缓存有每条流媒体数据对应的帧数据时，从多条用于进行混流的流媒体数据中提取每条流媒体数据对应的帧数据进行混流。否则，在该确定的混流时间戳不进行流媒体数据的混流。
在一实施例中，所述多条流媒体数据包括多条视频流数据；步骤S400，包括：分别从所述多条视频流数据中提取每条视频流数据对应的帧数据；根据所述多条视频流数据混流后进行展示的参数对所述对应的帧数据进行布局；将每次所述混流时间戳作为混流后输出的显示时间戳，对布局后的帧数据进行混流。
在该实施例中，流媒体混流服务器对多条视频流数据进行混流。当流媒体混流服务器分别从多条视频流数据中提取每条视频流数据对应的帧数据进行混流时，根据混流后展示到观众客户端的各个展示参数对各条视频流数据对应的帧数据进行调整布局，以在每次混流时间戳时进行混流。
在一实施例中，如图3所示，所述步骤S100，包括：
S110，接收多条流媒体数据，将所述多条流媒体数据中每条流媒体数据划分为多条帧数据。
S120，根据所述每条流媒体数据的抖动情况确定对应流媒体数据的抖动缓存长度，根据所述抖动缓存长度设置所述对应流媒体数据的缓存区。
S130，将所述每条流媒体数据的帧数据分别缓存进对应的缓存区，控制每个所述缓存区中帧数据缓存时间相同。
S140，分别从每个所述缓存区中获取所述每条流媒体数据的帧数据。
在该实施例中，流媒体混流服务器接收多条用于混流的流媒体数据之后，将每条流媒体数据划分为多条帧数据。并且，根据每条流媒体数据的抖动性确定系统中缓存该流媒体数据的缓存区的抖动缓存长度。进一步地，将接收到的每条流媒体数据分别缓存进对应的缓存区。同时，控制每个缓存区中，每个帧数据在缓存区的缓存时间相同。例如，流媒体数据A和流媒体数据B。流媒体数据A缓存在缓存区的帧数据包括帧数据a1和帧数据a2。流媒体数据B缓存在缓存区的帧数据包括帧数据b1和帧数据b2。帧数据a1、帧数据a2、帧数据b1和帧数据b2在缓存区中的缓存时间均相同。进而，流媒体混流服务器从每个缓存区中提取每条流媒体数据的帧数据，以将提取的帧数据进行混流。
进一步地，在该实施例的一个实施方式中，所述多条流媒体数据的混流方法还包括：将所述每条流媒体数据的帧数据的显示时间戳映射到同一时间坐标系。此时，步骤S400，包括：根据所述同一时间坐标系中每条流媒体数据的帧数据的显示时间戳，分别从所述每条流媒体数据对应的所述缓存区中获取距离所述混流时间戳的预设时间范围内的且显示时间戳小于或等于所述混流时间戳的帧数据进行混流。
在该实施方式中，由于不同的流媒体数据来自不同的数据上传通道，流媒体混流服务器接收到的多条流媒体数据的显示时间戳来自不同的时间坐标系。因此，流媒体混流服务器将多条流媒体数据的帧数据的显示时间戳均映射到同一时间坐标系，以为所有的帧数据设置同一时间参考标准，从而使得根据帧数据的显示时间戳从缓存区中获取对应的帧数据进行混流时，提取的帧数据更加精准，从而提高了混流后显示到观众客户端的混画质量。
进一步地，在该实施方式的一个实施例中，所述分别从所述每条流媒体数据对应的所述缓存区中获取距离所述混流时间戳的预设时间范围内的且显示时间戳小于或等于所述混流时间戳的帧数据进行混流，包括：若所述多条流媒体数据中任意一条流媒体数据对应的缓存区中不存在所述距离所述混流时间戳的预设时间范围内的且显示时间戳小于或等于所述混流时间戳的帧数据，则等待预设时长后获取下一次混流时间戳，在到达所述下一次混流时间戳时，分别从所述每条流媒体数据对应的所述缓存区中获取距离所述下一次混流时间戳的预设时间范围内的且显示时间戳小于或等于所述下一次混流时间戳的帧数据进行混流。其中，所述等待预设时长不超过所述多条流媒体数据中缓存区达到最大输入流时的时长。
在该实施例中，流媒体混流服务器在确认时间达到混流时间戳时，先判断缓存模块中用于混流的每条流媒体数据是否缓存有距离当前混流时间戳预设时间范围内的且显示时间戳小于或等于当前混流时间戳的帧数据。若是，则直接从每条流媒体数据对应的缓存区中提取距离当前混流时间戳预设时间范围内的且显示时间戳小于或等于当前混流时间戳的帧数据进行混流。若否，则流媒体混流服务器在当前的混流时间戳不进行混流操作，得到下一个混流的混流时间戳再进行混流操作。然而，流媒体混流服务器给每条流媒体数据设置的缓存区的抖动缓存长度有限，因此流媒体混流服务器等待混流的时间不宜超过任一缓存区达到最大输入流时的时长。
上述多条流媒体数据的混流方法，在确定多条流媒体数据混流后的输出帧率之后，计算混流输出的流媒体数据的帧间隔。通过帧间隔和当前混流的混流时间戳可确定系统下一次混流的时间戳。确定时间到达下一次混流时间，根据确定出的混流 时间戳分别从多条流媒体数据中提取对应的帧数据进行混流。因此，上述方法能够根据混流后输出帧率自适应各条源流的不同帧率，不仅简化传统混流的逻辑，并且提高了混流的同步效果。
为了更进一步地说明上述多条流媒体数据的混流方法，以下提供一具体的实施例：
如图4所示，服务端接收各条流的视频数据，各条视频数据分别组装成视频帧数据，并且根据各条数据流的接收抖动情况计算缓存每条数据流的全局抖动缓冲长度，进一步统一各条数据流的抖动缓冲长度。视频帧数据从抖动缓冲区被提取出之后依次进行解码。进一步地，将各条数据流的视频帧数据的PTS(Presentation Time Stamp显示时间戳)映射到同一坐标系，并根据接收到的各条数据流的视频帧数据的PTS进行微调修正误差。提取取各条数据流的首帧数据中，最大PTS作为混流起始时间戳。在进行每次混流时，根据混流后输出帧率计算帧间隔，结合当前混流时间和帧间隔算出下一帧混流时间戳。在每个混流时间戳时间到达时，从各条数据流中取出PTS小于等于混流时间戳且最接近该时间戳的帧作为待混帧，此时相同帧可能被重复使用。进一步地，判断待混帧与输入流数是否匹配，如若缺帧且缓存区中最大输入流长度未超过最大等待时间，则等待一小段时间并执行混流时间戳计算步骤。在进行混流时，将待混帧根据观众客户端混画参数进行布局，将混流时间戳作为输出PTS进行重新编码以输出。同时更新当前混画时间戳。若当前时间未达到混流结束之间，则继续进行混流。若当前时间已达到混流结束时间，则结束混流操作。
上述具体实施例所述的多条流媒体数据的混流方法，以简单统一的逻辑对多条数据流进行混流，根据目标帧率(混流后的输出帧率)对各源流自动进行补帧或删帧，从而大大简化混流逻辑，提高观众客户端的混画同步效果。
本申请还提供一种多条流媒体数据的混流装置。在一实施例中，如图5所示，该多条流媒体数据的混流装置包括第一获取模10、计算模块20、第二获取模块30以及混流模块40。
第一获取模块10用于获取多条流媒体数据中每条流媒体数据的帧数据。在本实施例中，流媒体混流服务器从每个流媒体上传服务器中读取出多条用于进行混流的流媒体数据。流媒体数据可以是音频流数据，视频流数据，或者音视频流数据。进一步地，将每条流媒体数据划分为多个帧数据。其中，可以按照时间轴坐标系将每条流媒体数据划分为时序相关的多个帧数据。每条流媒体数据的帧数据单独存储 在服务器的缓存区。缓存区的长度以及缓存区中的流媒体数据的大小，可根据每条流媒体数据的抖动情况确定。
计算模块20用于确定将所述多条流媒体数据进行混流后的输出帧率，根据所述输出帧率计算混流后流媒体数据的帧间隔。在本实施例中，对于每个混流后的流媒体数据的输出帧率，流媒体混流服务器根据系统当前状态提前设置。通过混流后的输出帧率计算混流后流媒体混流服务器输出的流媒体数据的帧间隔。例如，流媒体混流服务器设置混流后的流媒体数据的输出帧率为1秒输出20帧数据。此时，可计算得到混流后流媒体数据的帧间隔为50毫秒。
第二获取模块30用于根据当前混流的混流时间戳和所述帧间隔，确定下一次混流的混流时间戳。在本实施例中，流媒体混流服务器获取到混流后流媒体数据的帧间隔时，可根据当前混流的混流时间戳，确定下一次进行混流的混流时间戳。具体地，可将当前时间戳与混流后流媒体数据的帧间隔进行累加，将累加得到的时间戳作为下一次混流的混流时间戳。例如，当前混流的时间戳为100毫秒，混流后流媒体数据的帧间隔为50毫秒，则下次混流的混流时间戳为150毫秒。
混流模块40用于在下一次混流时，根据确定的混流时间戳分别从所述多条流媒体数据中提取每条流媒体数据对应的帧数据进行混流。在本实施例中，流媒体混流服务器根据当前时间戳和混流后流媒体数据的帧间隔确定出每一个下一次混流的混流时间戳之后，当时间到达下一次混流时间戳时，根据确定出的下一次混流时间戳分别从多条用于进行混流的流媒体数据中提取每条流媒体数据对应的帧数据进行混流。具体地，可先判断缓存区中是否缓存有每条流媒体数据对应的帧数据，在确定缓存区中均缓存有每条流媒体数据对应的帧数据时，从多条用于进行混流的流媒体数据中提取每条流媒体数据对应的帧数据进行混流。否则，在该确定的混流时间戳不进行流媒体数据的混流。
在其他实施例中，本申请提供的多条流媒体数据的混流装置中的各个模块还用于执行本申请所述的多条流媒体数据的混流方法中，对应各个步骤执行的操作，在此不再做详细的说明。
本申请还提供一种计算机可读存储介质。该存储介质上存储有计算机程序；所述计算机程序被处理器执行时，实现上述任一实施例所述的多条流媒体数据的混流方法。该存储介质可以是存储器。例如，内存储器或外存储器，或者包括内存储器和外存储器两者。内存储器可以包括只读存储器(ROM)、可编程ROM(PROM)、电可编程ROM(EPROM)、电可擦写可编程ROM(EEPROM)、快闪存储器、或者随机存储器。外 存储器可以包括硬盘、软盘、ZIP盘、U盘、磁带等。本申请所公开的存储介质包括但不限于这些类型的存储器。本申请所公开的存储器只作为例子而非作为限定。
本申请还提供一种计算机设备。一种计算机设备包括：一个或多个处理器；存储器；一个或多个应用程序。其中所述一个或多个应用程序被存储在所述存储器中并被配置为由所述一个或多个处理器执行，所述一个或多个应用程序配置用于执行上述任一实施例所述的多条流媒体数据的混流方法。
图6为本申请一实施例中的计算机设备的结构示意图。本实施例所述计算机设备可以是服务器、个人计算机以及网络设备。如图6所示，设备包括处理器603、存储器605、输入单元607以及显示单元609等器件。本领域技术人员可以理解，图6示出的设备结构器件并不构成对所有设备的限定，可以包括比图示更多或更少的部件，或者组合某些部件。存储器605可用于存储应用程序601以及各功能模块，处理器603运行存储在存储器605的应用程序601，从而执行设备的各种功能应用以及数据处理。存储器可以是内存储器或外存储器，或者包括内存储器和外存储器两者。内存储器可以包括只读存储器(ROM)、可编程ROM(PROM)、电可编程ROM(EPROM)、电可擦写可编程ROM(EEPROM)、快闪存储器、或者随机存储器。外存储器可以包括硬盘、软盘、ZIP盘、U盘、磁带等。本申请所公开的存储器包括但不限于这些类型的存储器。本申请所公开的存储器只作为例子而非作为限定。
输入单元607用于接收信号的输入，以及接收用户输入的关键字。输入单元607可包括触控面板以及其它输入设备。触控面板可收集用户在其上或附近的触摸操作(比如用户使用手指、触笔等任何适合的物体或附件在触控面板上或在触控面板附近的操作)，并根据预先设定的程序驱动相应的连接装置；其它输入设备可以包括但不限于物理键盘、功能键(比如播放控制按键、开关按键等)、轨迹球、鼠标、操作杆等中的一种或多种。显示单元609可用于显示用户输入的信息或提供给用户的信息以及计算机设备的各种菜单。显示单元609可采用液晶显示器、有机发光二极管等形式。处理器603是计算机设备的控制中心，利用各种接口和线路连接整个电脑的各个部分，通过运行或执行存储在存储器605内的软件程序和/或模块，以及调用存储在存储器内的数据，执行各种功能和处理数据。
在一实施方式中，设备包括一个或多个处理器603，以及一个或多个存储器605，一个或多个应用程序601。其中所述一个或多个应用程序601被存储在存储器605中并被配置为由所述一个或多个处理器603执行，所述一个或多个应用程序601配置用于执行以上实施例所述的多条流媒体数据的混流方法。
此外，在本申请各个实施例中的各功能单元可以集成在一个处理模块中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。所述集成的模块如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。
本领域普通技术人员可以理解实现上述实施例的全部或部分步骤可以通过硬件来完成，也可以通过程序来指令相关的硬件完成，该程序可以存储于一计算机可读存储介质中，存储介质可以包括存储器、磁盘或光盘等。
以上所述仅是本申请的部分实施方式，应当指出，对于本技术领域的普通技术人员来说，在不脱离本申请原理的前提下，还可以做出若干改进和润饰，这些改进和润饰也应视为本申请的保护范围。
应该理解的是，在本申请各实施例中的各功能单元可集成在一个处理模块中，也可以各个单元单独物理存在，也可以两个或两个以上单元集成于一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。
以上所述仅是本申请的部分实施方式，应当指出，对于本技术领域的普通技术人员来说，在不脱离本申请原理的前提下，还可以做出若干改进和润饰，这些改进和润饰也应视为本申请的保护范围。

Claims (10)

  1. 一种多条流媒体数据的混流方法，包括：
    获取多条流媒体数据中每条流媒体数据的帧数据；
    确定将所述多条流媒体数据进行混流后的输出帧率，根据所述输出帧率计算混流后流媒体数据的帧间隔；
    根据当前混流的混流时间戳和所述帧间隔，确定下一次混流的混流时间戳；
    在下一次混流时，根据确定的所述混流时间戳分别从所述多条流媒体数据中提取每条流媒体数据对应的帧数据进行混流。
  2. 根据权利要求1所述的方法，所述获取多条流媒体数据中每条流媒体数据的帧数据，包括：
    接收多条流媒体数据，将所述多条流媒体数据中每条流媒体数据划分为多条帧数据；
    根据所述每条流媒体数据的抖动情况确定对应流媒体数据的抖动缓存长度，根据所述抖动缓存长度设置所述对应流媒体数据的缓存区；
    将所述每条流媒体数据的帧数据分别缓存进对应的缓存区，控制每个所述缓存区中帧数据缓存时间相同；
    分别从每个所述缓存区中获取所述每条流媒体数据的帧数据。
  3. 根据权利要求2所述的方法，所述方法还包括：将所述每条流媒体数据的帧数据的显示时间戳映射到同一时间坐标系；所述在下一次混流时，根据确定的所述混流时间戳根据每次所述混流时间戳分别从所述多条流媒体数据中提取每条流媒体数据对应的帧数据进行混流，包括：
    根据所述同一时间坐标系中每条流媒体数据的帧数据的显示时间戳，分别从所述每条流媒体数据对应的所述缓存区中获取距离所述混流时间戳的预设时间范围内的且显示时间戳小于或等于所述混流时间戳的帧数据进行混流。
  4. 根据权利要求3所述的方法，所述分别从所述每条流媒体数据对应的所述缓存区中获取距离所述混流时间戳的预设时间范围内的且显示时间戳小于或等于所述混流时间戳的帧数据进行混流，包括：
    若所述多条流媒体数据中任意一条流媒体数据对应的缓存区中不存在所述距离所述混流时间戳的预设时间范围内的且显示时间戳小于或等于所述混流时间戳的帧数据，则等待预设时长后获取下一次混流时间戳，在到达所述下一次混流时间戳时，分别从所述每条流媒体数据对应的所述缓存区中获取距离所述下一次混流时 间戳的预设时间范围内的且显示时间戳小于或等于所述下一次混流时间戳的帧数据进行混流。
  5. 根据权利要求4所述的方法，所述等待预设时长不超过所述多条流媒体数据中缓存区达到最大输入流时的时长。
  6. 根据权利要求1所述的方法，所述获取多条流媒体数据中每条流媒体数据的帧数据之后，还包括：
    从所述每条流媒体数据的帧数据中获取所述每条流媒体数据的首帧数据；
    将显示时间戳最大的首帧数据的显示时间戳作为混流的混流起始时间戳。
  7. 根据权利要求1所述的方法，所述多条流媒体数据包括多条视频流数据；所述在下一次混流时，根据确定的所述混流时间戳分别从所述多条流媒体数据中提取每条流媒体数据对应的帧数据进行混流，包括：
    分别从所述多条视频流数据中提取每条视频流数据对应的帧数据；
    根据所述多条视频流数据混流后进行展示的参数对所述对应的帧数据进行布局；
    将每次所述混流时间戳作为混流后输出的显示时间戳，对布局后的帧数据进行混流。
  8. 一种多条流媒体数据的混流装置，包括：
    第一获取模块，用于获取多条流媒体数据中每条流媒体数据的帧数据；
    计算模块，用于确定将所述多条流媒体数据进行混流后的输出帧率，根据所述输出帧率计算混流后流媒体数据的帧间隔；
    第二获取模块，用于根据当前混流的混流时间戳和所述帧间隔，确定下一次混流的混流时间戳；
    混流模块，用于在下一次混流时，根据确定的所述混流时间戳分别从所述多条流媒体数据中提取每条流媒体数据对应的帧数据进行混流。
  9. 一种计算机可读存储介质，其上存储有计算机指令；所述指令用于执行上述权利要求1-7中任一项所述的多条流媒体数据的混流方法。
  10. 一种计算机设备，包括：
    一个或多个处理器；
    存储器；
    一个或多个应用程序，其中所述一个或多个应用程序被存储在所述存储器中并被配置为由所述一个或多个处理器执行，所述一个或多个应用程序配置用于：执行 根据权利要求1～7任一项所述的多条流媒体数据的混流方法。
PCT/CN2019/120714 2018-12-27 2019-11-25 流媒体数据的混流方法、装置及存储介质、计算机设备 WO2020134791A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811613917.3A CN109729373B (zh) 2018-12-27 2018-12-27 流媒体数据的混流方法、装置及存储介质、计算机设备
CN201811613917.3 2018-12-27

Publications (1)

Publication Number Publication Date
WO2020134791A1 true WO2020134791A1 (zh) 2020-07-02

Family

ID=66297358

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/120714 WO2020134791A1 (zh) 2018-12-27 2019-11-25 流媒体数据的混流方法、装置及存储介质、计算机设备

Country Status (2)

Country Link
CN (1) CN109729373B (zh)
WO (1) WO2020134791A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111107299A (zh) * 2019-12-05 2020-05-05 视联动力信息技术股份有限公司 一种多路视频合成的方法及装置

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109729373B (zh) * 2018-12-27 2020-12-08 广州华多网络科技有限公司 流媒体数据的混流方法、装置及存储介质、计算机设备
CN110290421A (zh) * 2019-08-01 2019-09-27 北京达佳互联信息技术有限公司 帧率调整方法、装置、计算机设备及存储介质
CN110475137B (zh) * 2019-09-06 2021-05-14 北京市博汇科技股份有限公司 一种高精度分布式显控帧同步方法和系统
CN110650353B (zh) * 2019-09-25 2020-12-04 广州华多网络科技有限公司 多人连麦混画方法及装置、存储介质及电子设备
CN111277885B (zh) * 2020-03-09 2023-01-10 北京世纪好未来教育科技有限公司 音视频同步方法、装置、服务器和计算机可读存储介质
CN112511768B (zh) * 2020-11-27 2024-01-02 上海网达软件股份有限公司 多画面合成方法、装置、设备及存储介质
CN114007108B (zh) * 2021-10-28 2023-09-19 广州华多网络科技有限公司 音频流混音控制方法及其装置、设备、介质、产品
CN114679589B (zh) * 2022-02-07 2023-11-17 上海哔哩哔哩科技有限公司 直播混流服务动态调整方法及装置

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101374231A (zh) * 2007-04-30 2009-02-25 Vixs系统公司 合并多个视频流的系统以及相应使用的方法
EP2713609A1 (en) * 2012-09-28 2014-04-02 Stockholms Universitet Holding AB Dynamic delay handling in mobile live video production systems
CN104994278A (zh) * 2015-06-30 2015-10-21 北京竞业达数码科技有限公司 多路视频同步处理方法及装置
CN105430537A (zh) * 2015-11-27 2016-03-23 刘军 对多路数据进行合成的方法、服务器及音乐教学系统
CN108881957A (zh) * 2017-11-02 2018-11-23 北京视联动力国际信息技术有限公司 一种多媒体文件的混合方法和装置
CN109729373A (zh) * 2018-12-27 2019-05-07 广州华多网络科技有限公司 流媒体数据的混流方法、装置及存储介质、计算机设备

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104602095A (zh) * 2014-12-26 2015-05-06 广东威创视讯科技股份有限公司 一种组合桌面采集与同步显示方法及系统
CN105491393A (zh) * 2015-12-02 2016-04-13 北京暴风科技股份有限公司 多人视频直播业务的实现方法
CN106921866B (zh) * 2017-05-03 2021-12-28 广州方硅信息技术有限公司 辅助直播的多视频导播方法和设备
CN108881927B (zh) * 2017-11-30 2020-06-26 视联动力信息技术股份有限公司 一种视频数据合成方法和装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101374231A (zh) * 2007-04-30 2009-02-25 Vixs系统公司 合并多个视频流的系统以及相应使用的方法
EP2713609A1 (en) * 2012-09-28 2014-04-02 Stockholms Universitet Holding AB Dynamic delay handling in mobile live video production systems
CN104994278A (zh) * 2015-06-30 2015-10-21 北京竞业达数码科技有限公司 多路视频同步处理方法及装置
CN105430537A (zh) * 2015-11-27 2016-03-23 刘军 对多路数据进行合成的方法、服务器及音乐教学系统
CN108881957A (zh) * 2017-11-02 2018-11-23 北京视联动力国际信息技术有限公司 一种多媒体文件的混合方法和装置
CN109729373A (zh) * 2018-12-27 2019-05-07 广州华多网络科技有限公司 流媒体数据的混流方法、装置及存储介质、计算机设备

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111107299A (zh) * 2019-12-05 2020-05-05 视联动力信息技术股份有限公司 一种多路视频合成的方法及装置

Also Published As

Publication number Publication date
CN109729373A (zh) 2019-05-07
CN109729373B (zh) 2020-12-08

Similar Documents

Publication Publication Date Title
WO2020134791A1 (zh) 流媒体数据的混流方法、装置及存储介质、计算机设备
WO2023024834A9 (zh) 一种游戏数据处理方法、装置及存储介质
US11805291B2 (en) Synchronizing media content tag data
US20210337250A1 (en) A method and device for combining audio and video data streams
US20190387263A1 (en) Synchronously displaying and matching streaming media and subtitles
US11051050B2 (en) Live streaming with live video production and commentary
US10341672B2 (en) Method and system for media synchronization
WO2020056877A1 (zh) 一种调整直播视频的时间戳的方法和装置
US11503366B2 (en) Dynamic playout of transition frames while transitioning between play out of media streams
US11553215B1 (en) Providing alternative live media content
US20080127272A1 (en) Aggregation of Multiple Media Streams to a User
WO2019134293A1 (zh) 一种直播方法、装置、服务器及介质
US11201903B1 (en) Time synchronization between live video streaming and live metadata
US10887646B2 (en) Live streaming with multiple remote commentators
US20150113576A1 (en) Method and apparatus for ip video signal synchronization
WO2017107516A1 (zh) 网络视频播放方法及装置
WO2022033494A1 (zh) 交互类直播连麦的控制方法、装置、系统、设备及介质
TW201424347A (zh) 多視角畫面之同步顯示系統以及多視角畫面之同步顯示方法
JP2020174378A (ja) 異種ネットワーキング環境におけるメディアレンダリングの同期化
CN102413335A (zh) 一种手动调节节目音画同步的装置及方法
CN110650353B (zh) 多人连麦混画方法及装置、存储介质及电子设备
KR102150685B1 (ko) 미디어 서비스 채널 전환 방법 및 장치
US11317035B1 (en) Method and system for synchronized playback of multiple video streams over a computer network
US9584572B2 (en) Cloud service device, multi-image preview method and cloud service system
KR102012580B1 (ko) 멀티미디어 콘텐츠 처리를 위한 장치 및 그 제어방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19902572

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19902572

Country of ref document: EP

Kind code of ref document: A1