WO2020098455A1 - Method for real-time delivery of media stream and server - Google Patents

Abstract

Disclosed by the present invention are a method for real-time delivery of a media stream and a server. The method comprises the following steps: receiving a media segment request sent by a client terminal, wherein the media segment request does not carry any control parameter or carry at least one control parameter, and the control parameter includes a first category of parameters indicating a target media stream to be transmitted, a second category of parameters indicating candidate media units to be transmitted, and a unit sorting manner; generating a media segment according to the media segment request, wherein the target media stream to be transmitted is determined according to the first category of parameters, the candidate media units to be transmitted are determined according to the second category of parameters, and the candidate media units to be transmitted are sorted and encapsulated into the media segment in an order specified by the unit sorting manner; and sending the media segment to the client terminal. The method can realize real-time and segmented delivery of media stream according to the requirements of the client terminal, thereby effectively reducing the delay and overhead of media stream transmission, and supporting the transmission priority of the latest and high-priority media units.

Description

Media stream real-time delivery method and server

Cross-reference of related applications

This application requires the priority of the Chinese patent application number "201811351129.1" filed by Beijing Kaiguang Information Technology Co., Ltd. and filed on November 14, 2018, with the invention titled "media streaming real-time delivery method and server".

Technical field

The invention relates to the technical field of digital information transmission, in particular to a method and server for real-time delivery of media streams.

Background technique

With the rapid development of the Internet, especially the mobile Internet, the real-time transmission of audio, video, image and other multimedia data through the Internet has become the basic demand of many applications. To meet this demand, various real-time streaming media transmission technologies have been proposed, which are currently available Three types are widely used: real-time transport protocol (RTP (Real-time Transport Protocol) / RTSP (Real Time Streaming Protocol)), RTMP (Real Time Time Messaging Protocol), real-time messaging protocol ) And HTTP (HyperTextTransferProtocol, Hypertext Transfer Protocol) adaptive streaming HAS (HTTPAdaptiveStreaming). Among them, HTTP adaptive streaming includes multiple solutions: HLS (HTTP Live Streaming) proposed by Apple, Smooth Streaming proposed by Microsoft, HDS (HTTP Dynamic Streaming) proposed by Adobe, and DASH (Dynamic Adaptive) proposed by MPEG. Streaming over HTTP, dynamic adaptive streaming based on HTTP).

The common feature of the HTTP adaptive streaming solution in the related art is that the media stream is cut into media fragments in a short time (2s ～ 10s), and an index file or a manifest file describing these media fragments is generated at the same time (such as m3u8 in HLS) Playlists and MPD files in DASH), and then save them to each web server, the client obtains the URL (Uniform Resource Locator) access address of these media fragments by accessing the playlist or manifest file, and then You can use standard HTTP protocol to download and play these media segments one by one. In short, the main difference between the above schemes is the difference between the packaging format adopted by the media fragments and the format of the manifest file.

Compared with RTP / RTSP and RTMP, HTTP adaptive streaming can make full use of the existing Internet Web cache facilities (such as CDN and various Web cache servers), and can support large-scale user access. At the same time, by providing media segments with multiple bit rates, it can also support clients to select segments with appropriate bit rates according to network conditions and terminal capabilities, to achieve bit rate adaptation. Therefore, HTTP adaptive streaming has become the mainstream method of real-time streaming media delivery on the Internet.

However, the HTTP adaptive streaming solutions in related technologies all have the following problems:

First, the length of media fragments cannot adapt to dynamically changing network transmissions. The current HAS solutions all use a pre-segmentation method, that is, the server generates media fragments and their manifest files according to a preset time and submits them to the web server. When the network transmission bandwidth is sufficient and the delay is small, setting a larger fragment length means increasing the real-time transmission delay; when the network transmission bandwidth is insufficient and the delay is large, setting a smaller fragment length means frequent file requests , Increase the burden on the server and network transmission overhead. Since the transmission bandwidth and transmission delay on the Internet are dynamically changing, using a fixed-length pre-segmentation method cannot achieve optimal transmission.

Second, the manifest file adds transmission delay and overhead. The client needs to obtain the manifest file before it can obtain the URL address of the media segment. However, because the manifest file takes a period of time to be transmitted to the client, the manifest file obtained by the client does not reflect the current generation of the latest media fragments. In addition, when the manifest file encounters blocking or transmission errors, it will block the user Fast access to media fragments reduces the transmission performance of real-time streaming media.

Third, priority transmission of the latest media content is not supported. In many application scenarios, such as real-time monitoring and live event live broadcasting, users often expect to see the latest media content in time, especially when the transmission network bandwidth is insufficient to transmit all media content, and the existing HTTP adaptive streaming solution This requirement cannot be met in the following, which is reflected in: first, from the packaging and generation of media fragments on the server, to the update of the manifest file, to the client to obtain the manifest file and send the request, and finally to the server to receive the request after a delay At this time, the media segment requested by the client according to the manifest file may not be the latest media content generated; secondly, when the media segment is encapsulated in chronological order and transmitted using HTTP / TCP protocol, once congestion occurs, it can be transmitted to The client is only the first half of these media clips, while the second half containing newer media content cannot be delivered in time.

Fourth, priority transmission of high-priority content is not supported. In some application scenarios, the media stream includes various types of media units. The importance or priority of these media units is different. For example, the audio unit usually has a higher priority than the video unit. In addition, even the same type of media Units may also have different priorities due to encoding. For example, in video encoding, the I frame is more important than the P and B frames. When the network transmission conditions are poor and all media units cannot be transmitted, in order to ensure the user's experience, high-priority content should be transmitted first. However, for the existing HTTP adaptive streaming solution, the media fragments are generated in advance The content of media clips cannot be adjusted, so it is impossible to support the preferential transmission of high-priority content, which needs to be resolved urgently.

Summary of the invention

The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

To this end, the first object of the present invention is to propose a real-time delivery method of media streams, which can effectively reduce the media stream transmission delay and overhead, and support the preferential transmission of the latest media units and high-priority media units.

The second object of the present invention is to propose a real-time delivery server for media streams.

The third object of the present invention is to propose a computer device.

The fourth object of the present invention is to propose a non-transitory computer-readable storage medium.

The fifth object of the present invention is to propose a computer program product.

To achieve the above objective, an embodiment of the present invention provides a method for real-time delivery of media streams, characterized in that the media stream is a sequence of media units generated in real time, where each media unit is associated with a generation time And / or a sequence number indicating the generation sequence, the method includes: receiving a media segment request sent by a client, wherein the media segment request does not carry or carries at least one control parameter, and the control parameter includes indicating the target media to be transmitted The first type parameter of the stream, the second type parameter indicating the candidate media unit to be transmitted and the unit sorting method; generating a media segment according to the media segment request, wherein the target to be transmitted is determined according to the first type parameter Media stream, determining the candidate media units to be transferred according to the second type of parameters, sorting and encapsulating the candidate media units to be transferred in the order specified by the unit sorting method, and packaging the media segments; sending the Media segment to the client.

The real-time delivery method of the media stream according to the embodiment of the present invention generates a media segment in real time according to the request of the client and returns it to the client, so as to realize the real-time media stream delivery according to the client's needs, and the duration of the media segment will be automatically adapted When the network transmission bandwidth changes, the client can control the length of media segmentation through active requests. Since each media segment is triggered by the client's request, the manifest file is no longer needed, and the client does not need to request and parse the manifest file. On the one hand, the client can obtain the latest media stream more quickly, reducing the transmission delay of the real-time media stream. On the other hand, it also reduces the transmission overhead and processing overhead caused by the manifest file. Finally, the client can also pass Request to control the generation time and arrangement order of the media units in the media segment. When the network transmission conditions are poor, ensure that the latest generated media unit or high-priority media unit is sent in a timely manner, thereby effectively reducing the media stream transmission delay and overhead , And supports priority transmission of the latest media units and high-priority media units.

In addition, the method for real-time delivery of media streams according to the above embodiments of the present invention may also have the following additional technical features:

Further, in an embodiment of the present invention, the generating a media segment according to the media segment request further includes: if the media segment request does not carry the first type parameter, the target media to be transmitted The stream is a media stream designated by default; if the second segment parameter is not carried in the media segment request, the candidate media unit includes a media unit designated by default, and the media unit designated by default is the A media unit in which the sequence number interval of all and latest media units in the target media stream is less than the first preset value, or a media unit in which the generation time interval of all and latest media units in the target media stream is less than the second preset value; if The media segment request does not carry the unit sorting mode, and then the candidate media unit is encapsulated into the media segment according to the unit sorting mode specified by default.

Further, in an embodiment of the present invention, the generating the media segment according to the media segment request further includes: if the media segment request carries at least one parameter of the second category, wherein each of the second category The parameter corresponds to at least one constraint condition of the candidate media unit, and the candidate media unit to be transmitted includes all media units in the target media stream that simultaneously satisfy all the constraint conditions corresponding to the second-type parameter.

Further, in an embodiment of the present invention, each media unit is associated with a sequence offset, where the sequence offset refers to the interval between the sequence number of the media unit and the latest media unit, and the second type of parameter includes A start sequence number and / or a maximum sequence offset, wherein the constraint condition corresponding to the start sequence number is: if the start sequence number is valid, the sequence number of the candidate media unit is after the start sequence number; the maximum sequence The constraint condition corresponding to the bias is: if the maximum order bias is valid, the order bias of the candidate media unit is less than or equal to the maximum order bias.

Further, in an embodiment of the present invention, each media unit is associated with a time offset, the time offset refers to a time interval between the generation of the media unit and the latest media unit, and the second type of parameters includes A start time and / or a maximum time offset, wherein the constraint condition corresponding to the start time is: if the start time is valid, the generation time of the candidate media unit is after the start time; The constraint condition corresponding to the maximum time offset is: if the maximum time offset is valid, the time offset of the candidate media unit is less than or equal to the maximum time offset.

Further, in an embodiment of the present invention, each media unit is associated with a priority, the second type parameter includes a minimum priority, and the constraint corresponding to the minimum priority includes: if the minimum If the priority is valid, the priority of the candidate media unit is greater than or equal to the minimum priority; if the other second-type parameters carried in the media segment request do not define the scope of the candidate media unit, the scope of the candidate media unit is Specify by default.

Optionally, in an embodiment of the present invention, the unit sorting mode is one of the following basic sorting modes: sequence number forward, sequence number reverse, generation time forward, and generation time reverse.

Optionally, in an embodiment of the present invention, each media unit is associated with a priority, and the unit sorting mode is one of the following basic sorting modes: sequence number forward, sequence number reverse, and generation time forward , Reverse time, high priority and low priority.

Optionally, in an embodiment of the present invention, the unit sorting mode is a cascade of multiple basic sorting modes, and sorting the candidate media units in the order specified by the unit sorting mode includes: sorting the candidate media units Sort according to the first basic sorting manner, and sort the candidate media units with the same position after sorting according to the second basic sorting manner, and so on until the sorting is completed.

To achieve the above objective, another embodiment of the present invention provides a real-time delivery server for a media stream, where the media stream is a sequence of media units generated in real time, where each media unit is associated with a generation time and / or A sequence number indicating the generation sequence, the server includes: a client interface component, configured to receive a media segment request sent by the client and return the corresponding media segment, wherein the media segment request does not carry or carries at least one control parameter, And the control parameters include a first type parameter indicating the target media stream to be transmitted, a second type parameter indicating the candidate media unit to be transmitted, and a unit sorting method; a media segment generation component generates a media segment according to the media segment request, wherein , Determine the target media stream to be transmitted according to the first type of parameter, determine the candidate media unit to be transmitted according to the second type parameter, and specify the candidate media unit to be transmitted in the unit sorting manner The media segments are sequentially sorted and packaged, and the media segments are sent to the client through the client interface component.

The real-time delivery server of the media stream according to the embodiment of the present invention generates media segments in real time according to the client's request and returns them to the client, so as to realize the real-time media stream delivery according to the client's needs, and the duration of the media segment will be automatically adapted When the network transmission bandwidth changes, the client can control the length of media segmentation through active requests. Since each media segment is triggered by the client's request, the manifest file is no longer needed, and the client does not need to request and parse the manifest file. On the one hand, the client can obtain the latest media stream more quickly, reducing the transmission delay of the real-time media stream. On the other hand, it also reduces the transmission overhead and processing overhead caused by the manifest file. Finally, the client can also pass Request to control the generation time and arrangement order of the media units in the media segment. When the network transmission conditions are poor, ensure that the latest generated media unit or high-priority media unit is sent in a timely manner, thereby effectively reducing the media stream transmission delay and overhead , And supports priority transmission of the latest media units and high-priority media units.

In addition, the real-time delivery server for media streams according to the above embodiments of the present invention may also have the following additional technical features:

Further, in an embodiment of the present invention, the media segment generation component is further configured to: when the media segment request does not carry the first type parameter, the target media stream to be transmitted is specified by default A media stream, and when the media segment request does not carry the second type parameter, the candidate media unit includes a media unit specified by default, and the media unit specified by default is all of the target media stream A media unit with a sequence number interval less than the first preset value from the latest media unit, or all media units in the target media stream whose generation time interval is less than the second preset value, and the media segment When the request does not carry the unit sorting mode, the candidate media unit is encapsulated into the media segment according to the unit sorting mode specified by default.

Further, in an embodiment of the present invention, the media segment generating component is further configured to request to carry at least one parameter of the second type in the media segment, wherein each parameter of the second type corresponds to a candidate When the media unit has at least one constraint condition, the candidate media unit to be transmitted includes all media units in the target media stream that simultaneously satisfy all the constraint conditions corresponding to the second type parameter.

Further, in an embodiment of the present invention, each media unit is associated with a sequence offset, where the sequence offset refers to the interval between the sequence number of the media unit and the latest media unit, and the second type of parameter includes A start sequence number and / or a maximum sequence offset, wherein the constraint condition corresponding to the start sequence number is: if the start sequence number is valid, the sequence number of the candidate media unit is after the start sequence number; the maximum sequence The constraint condition corresponding to the bias is: if the maximum order bias is valid, the order bias of the candidate media unit is less than or equal to the maximum order bias.

Further, in an embodiment of the present invention, each media unit is associated with a time offset, the time offset refers to a time interval between the generation of the media unit and the latest media unit, and the second type of parameters includes A start time and / or a maximum time offset, wherein the constraint condition corresponding to the start time is: if the start time is valid, the generation time of the candidate media unit is after the start time; The constraint condition corresponding to the maximum time offset is: if the maximum time offset is valid, the time offset of the candidate media unit is less than or equal to the maximum time offset.

Further, in an embodiment of the present invention, each media unit is associated with a priority, the second type parameter includes a minimum priority, and the constraint corresponding to the minimum priority includes: if the minimum If the priority is valid, the priority of the candidate media unit is greater than or equal to the minimum priority; if the other second-type parameters carried in the media segment request do not define the scope of the candidate media unit, the scope of the candidate media unit is Specify by default.

Optionally, in an embodiment of the present invention, the unit sorting mode is one of the following basic sorting modes: sequence number forward, sequence number reverse, generation time forward, and generation time reverse.

Optionally, in an embodiment of the present invention, each media unit is associated with a priority, and the unit sorting mode is one of the following basic sorting modes: sequence number forward, sequence number reverse, and generation time forward , Reverse time, high priority and low priority.

Optionally, in an embodiment of the present invention, the unit sorting mode is a cascade of multiple basic sorting modes, and sorting the candidate media units in the order specified by the unit sorting mode includes: sorting the candidate media units Sort according to the first basic sorting manner, and sort the candidate media units with the same position after sorting according to the second basic sorting manner, and so on until the sorting is completed.

To achieve the above objective, an embodiment of the third aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, Realize the real-time delivery method of the media stream as described in the above embodiments.

In order to achieve the above object, an embodiment of the fourth aspect of the present invention provides a non-transitory computer-readable storage medium. When the program is executed by a processor, a method for real-time delivery of a media stream as described in the foregoing embodiment is implemented.

In order to achieve the above object, an embodiment of the fifth aspect of the present invention provides a computer program product. When instructions in the computer program product are executed by a processor, a method for real-time delivery of media streams as described in the foregoing embodiment is performed.

Additional aspects and advantages of the present invention will be partially given in the following description, and some will become apparent from the following description, or be learned through the practice of the present invention.

BRIEF DESCRIPTION

The above-mentioned and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

1 is a flowchart of a method for real-time delivery of media streams according to an embodiment of the present invention;

2 is a schematic diagram of a real-time transmission process of a client continuously submitting a media segment request according to an embodiment of the present invention;

3 is a schematic diagram of sorting and encapsulating candidate media units into media segments according to an embodiment of the present invention (unit sorting method is sequence number reverse);

4 is a schematic diagram of sorting and encapsulating candidate media units into media segments according to another embodiment of the present invention (unit sorting method is time reverse);

FIG. 5 is a schematic diagram of sorting and packaging candidate media units according to yet another embodiment of the present invention (packet sorting method is high priority first + time reverse); and

6 is a schematic structural diagram of a real-time delivery server for media streams according to an embodiment of the present invention.

detailed description

Hereinafter, embodiments of the present invention will be described in detail. Examples of the embodiments are shown in the drawings, in which the same or similar reference numerals indicate the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the drawings are exemplary, and are intended to explain the present invention, and should not be construed as limiting the present invention.

In the Internet, it is often necessary to transfer various real-time generated audio streams, video streams or data streams from one network node to another network node. These network nodes include not only various terminals, such as PCs, mobile phones, tablets, but also various Various application servers, such as video servers and audio servers, collectively refer to these audio streams, video streams, or data streams as media streams. The transmission process of the media stream can be described by a general client-server model: the media stream generated in real time is delivered by the server to the client. The server and the client here refer to logical functional entities, where the server is a functional entity that sends media streams and the client is a functional entity that receives media streams. The server and client can exist on any network node.

Each transmitted media stream is a sequence of media units generated in real time. For different media streams, the corresponding media units can be selected by themselves. When the media stream is a byte stream generated in real time, one byte can be selected as the media unit; when the media stream is an audio stream or video stream obtained through real-time sampling, the original audio frame or video frame can be selected as the media Unit; when the media stream is a real-time sampled and encoded audio stream or video stream, you can select the encoded audio frame, encoded video frame or access unit (Access Unit) as the media unit; when the media stream is a When real-time sampling, encoding and encapsulating audio or video streams, you can select the encapsulated transmission package (such as RTP packet, PES / PS / TS packet, etc.) as the media unit; when the media stream is a real-time sampling, encoding, encapsulation With pre-segmented audio or video streams, you can select a segmented media segment (such as the TS format segment used in the HLS protocol and the fMP4 format segment used in the DASH protocol) as the media unit.

Each media unit may be associated with a generation time, which is usually a time stamp. Each media unit can also be associated with a serial number, which can be used to indicate the order in which the media units are generated. When the serial number is used to indicate the order in which the media unit is generated, the meaning of the serial number needs to be defined according to the specific media unit. When the media unit is a byte, the serial number of the media unit is the byte serial number; when the media unit is an audio frame or video frame, the serial number of the media unit is the frame serial number; when the media unit is a transmission packet, the serial number of the media unit Is the packet sequence number; when the media unit is a stream segment, the sequence number of the media unit is the segment sequence number (such as the Media Sequence of each TS segment in HLS).

For a media stream, you can associate a sequence number and a generation time, for example, when the real-time media stream is an RTP packet stream, the RTP header has a Sequence Number field to indicate the RTP packet In order, there is a Timestramp field to indicate the generation time of the media data encapsulated in RTP. In this case, multiple consecutive RTP packets may correspond to the same generation time, but the sequence number is unique.

The method of the embodiment of the present invention may be implemented for any real-time media stream. In the following embodiments, the embodiments of the present invention will respectively select RTP real-time media streams or MPEG2-TS real-time media streams to illustrate the implementation method of the embodiments of the present invention. For RTP real-time streaming, the media unit is an RTP packet, and the sequence number of the RTP packet is selected as the sequence number of the media unit. The packet sequence number of the RTP packet is a 16-bit field with a maximum value of 65535. For continuously generated RTP The sequence number of the packet is cyclically counted. If the current packet sequence number is Seq, the sequence number of the next packet is (Seq + 1)% 65536. Therefore, the sequence number is subject to its bit length in implementation, and the sequence number size may not reflect its In the case of sequence, at this time, it can be judged by the generation time of the media unit whether the serial number has a cycle count, so as to accurately determine the sequence relationship and interval between the sequence numbers of the two media units. For the MPEG2-TS real-time stream, you can use a method similar to HLS / DASH to divide the TS stream into TS fragments of a fixed duration, such as about 1 second, each TS fragment can include multiple media frames, and then press these fragments The generation sequence is numbered, and as a media unit, the timestamp of the first media frame contained in each segment indicates the generation time of the segment.

In real-time streaming media protocols such as RTP or RTMP in related technologies, the server push method is adopted: once a new media unit is available on the server, it is actively sent to the client. The method of the embodiment of the present invention may be similar to various HTTP adaptive streams (such as HLS, smooth stream, MPEG-DASH), and adopts the client-side pull method, but the difference is that the existing various HTTP adaptive streams In the client, the client requests or pulls the divided segments according to the manifest file, and each segment can be identified by a URL. In the embodiment of the present invention, the media segment may not be pre-divided, but The server generates it immediately according to the client's request, and the client can control the content and length of the media segment.

The method and server for real-time delivery of media streams according to embodiments of the present invention will be described below with reference to the drawings. First, the method for real-time delivery of media streams according to embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a flowchart of a method for real-time delivery of media streams according to an embodiment of the present invention.

As shown in FIG. 1, in the real-time delivery method of the media stream, the media stream is a sequence of media units generated in real time, where each media unit is associated with a generation time and / or a sequence number indicating the generation order. The method includes:

In step S101, a media segment request sent by a client is received, where the media segment request does not carry or carries at least one control parameter, and the control parameter includes a first type parameter indicating a target media stream to be transmitted, and a candidate indicating to be transmitted The second type of media unit parameters and unit ordering method.

Specifically, the media segment request may not carry any first-type parameters and second-type parameters, and new parameters may be defined according to the needs of further implementation. For example, control parameters that may be used as the first-type parameters include: media stream identification, Media stream name, etc .; control parameters that can be used as the second type of parameters include: start sequence number, start time, maximum sequence offset, maximum time offset, minimum priority, maximum priority, etc.

Media segment requests can be submitted using any protocol, such as the common HTTP protocol, TCP protocol, UDP protocol, etc. When HTTP protocol is used to submit media segment request, HTTP-GET or HTTP-POST can also be used.

When the media segment request carries control parameters, the control parameters need to be encapsulated into a string or byte stream in a certain way and sent to the server. For example, when HTTP-GET is used to send a media segment request, the control parameters can be encapsulated as a string in the URL. Examples of media segment requests using HTTP-GET are as follows:

Media segment request without control parameters:

GET "http://www.xxx-server.com/msreq" [req1]

Media segment request carrying a control parameter:

GET "http://www.xxx-server.com/msreq?streamID=601"[req2]

GET "http://www.xxx-server.com/msreq?seqBegin = 1005" [req3]

GET "http://www.xxx-server.com/msreq?timeBegin=31000"[req4]

GET "http://www.xxx-server.com/msreq?maxSeqOffset=10"[req5]

GET "http://www.xxx-server.com/msreq?maxTimeOffset=2000"[req6]

GET "http://www.xxx-server.com/msreq?minPriority=4"[req7]

GET "http://www.xxx-server.com/msreq?maxPriority=3"[req8]

GET "http://www.xxx-server.com/msreq?unitSortMode=SEQ_BACKWARD"[req9]

Media segment request carrying two control parameters:

GET "http://www.xxx-server.com/msreq?seqBegin = 1010 & maxSeqOffset = 5" [req10]

GET "http://www.xxx-server.com/msreq?timeBegin=31000&maxTimeOffset=3000"[req11]

GET "http://www.xxx-server.com/msreq?seqBegin = 1010 & minPriority = 3" [req12]

GET "http://www.xxx-server.com/msreq?timeBegin=31000&

unitSortMode ＝ TIME_BACKWARD ”[req13]

GET "http://www.xxx-server.com/msreq?timeBegin=31000&

unitSortMode = HIGH_PRIORITY_FIRST "[req14]

GET "http://www.xxx-server.com/msreq?maxTimeOffset=2000&

unitSortMode = HIGH_PRIORITY_FIRST + TIME_BACKWARD "[req15]

Media segment request carrying three control parameters:

GET "http://www.xxx-server.com/msreq?seqBegin=1010&maxSeqOffset=5&unitSortMode=TIME_BACKWARD"[req16]

GET "http://www.xxx-server.com/msreq?timeBegin=33000&

maxTimeOffset = 3000 & unitSortMode = TIME_BACKWARD ”[req17]

In the URL of the above request, the parameter names streamID, seqBegin, timeBegin, maxSeqOffset, maxTimeOffset, minPriority, maxPriority and unitSortMode represent the media stream ID, start sequence number, start time, maximum sequence offset, maximum time offset, minimum priority, maximum Priority and unit sorting method.

The server can use a Web server to receive the client ’s media segment request, extract the corresponding control parameters from the requested URL, and classify the control parameters: if it is a media stream identifier, this parameter is the first type of parameter; If it is the start sequence number, start time, maximum sequence offset, maximum time offset, minimum priority and maximum priority, it is the second type of parameter.

In step S102, a media segment is generated according to the media segment request, wherein the target media stream to be transmitted is determined according to the first type of parameter, the candidate media unit to be transmitted is determined according to the second type of parameter, and the candidate media unit to be transmitted is The order specified by the cell sorting method is sorted and encapsulated into media segments.

Specifically, after receiving the media segment request, the server can obtain the control parameters carried in the media segment request, and then can determine the target media stream to be transmitted according to the first type parameters therein, and determine the second type parameters carried The candidate media units to be transmitted are finally sorted and packaged into media segments in the order specified by the unit sorting manner. Among them, you can use a custom encapsulation protocol to encapsulate one or more media units into a media segment. For example, a simple encapsulation protocol is as follows: The media segment is composed of a segment header and a segment payload. The segment payload is formed by cascading several media units. The header indicates the starting position and length of each media unit. When the candidate units are packaged into the media side, the order of the media units in the media segment should be consistent with the order specified by the unit sorting method.

In step S103, the media segment is sent to the client.

Specifically, the server can select the appropriate method to send the media segment to the client according to the protocol used by the client's media segment request. For example, when the received media segment request uses HTTP GET, it can be sent through the HTTP GET response message. Generated media segment: Put the media segment into the entity body of the HTTP response message.

When the server receives continuous media segment requests from the client, the server will continuously generate new media segments according to the client's request. These new media segments encapsulate several recently generated media units, and the client parses these media segments. The media units of the real-time media stream can be recovered, and the real-time transmission of the media stream from the server to the client is realized. This process is shown in FIG. 2.

Due to the method of generating media segments on the fly, the method of the embodiment of the present invention no longer requires a manifest file, thereby reducing transmission delay and saving overhead. In addition, the client can adjust the frequency of sending requests to control the length of the media segment to better adapt to changes in network bandwidth.

The foregoing is a detailed explanation of Embodiment 1, and Embodiment 2 will be described in detail below. In the following embodiments, the server will be described how to generate a media segment according to a media segment request.

Further, in an embodiment of the present invention, generating the media segment according to the media segment request further includes: if the media segment request does not carry the first type of parameters, the target media stream to be transmitted is the media stream specified by default; if If the media segment request does not carry the second type of parameters, the candidate media unit includes the media unit specified by default, and the default designated media unit is the sequence number interval of all and latest media units in the target media stream is less than the first pre A set media unit, or a media unit whose generation time interval of all and latest media units in the target media stream is less than a second preset value; if the media segment request does not carry the unit sorting method, then The candidate media units are encapsulated into media segments according to the order of units specified by default.

Taking RTP real-time streaming as an example, the media unit is an RTP packet, and each RTP packet carries a packet sequence number. Assume that the newly generated RTP packet has a packet sequence number of 1020, the first preset value is 20, and the default specified unit sorting method is sequence number forward. When the server receives a media segment request that does not carry any parameters, such as [req1] , The server can select one of the existing real-time media streams as the target media stream, then the candidate media units to be sent include the 20 most recently generated RTP packets in the target media stream (packet sequence numbers from 1001 to 1020), and then the 20 The RTP packets are encapsulated into media segments in sequence.

Taking the TS real-time stream as an example, the media unit is a TS segment, and each TS segment is associated with a generation time, and the generation time is the time stamp of the first media frame in the TS segment. Assume that the generation time of the newly generated TS segment is 33000 (unit is microseconds), the second preset value is 3000, and the default cell sorting method is generation time forward. When the server receives a media that does not carry any parameters When the segment request is [req1], the server can select one of the existing real-time media streams as the target media stream, and the candidate media unit to be sent includes the TS segment generated in the target media stream in the last 3 seconds, that is, the generation time Tp is TS segments within the range (30000 <Tp <= 33000), and then encapsulate these TS segments into media segments in the order of generation time.

With the above embodiment, each media segment request sent by the user will return several media units that have been generated recently. When the server continues to receive the media segment request, it will continue to deliver the recently generated media unit to the client.

Embodiment 3 In the following embodiments, it will be described how the server determines candidate media units to be transmitted according to the second type of parameters.

Further, in an embodiment of the present invention, if the media segment request carries at least one parameter of the second type, wherein each parameter of the second type corresponds to at least one constraint condition of the candidate media unit, Then, the candidate media unit to be transmitted includes all media units in the target media stream that simultaneously satisfy all the constraint conditions corresponding to the second type parameters.

The following will further give six types of second-type parameters, and the constraints corresponding to each type of second-type parameters. In specific implementation, one or more of these types of parameters can be used as needed, and it is not limited to define other types of self-defined parameter:

1) Starting serial number

The constraint condition corresponding to the starting sequence number is: if the starting sequence number is valid, the sequence number of the candidate media unit is after the starting sequence number.

2) Maximum order deviation

The sequence deviation of a media unit refers to the interval between the sequence number of the media unit and the latest media unit, and the constraint condition corresponding to the maximum sequence deviation is: if the maximum sequence deviation is valid, the sequence of the candidate media unit is too small Is equal to or equal to the maximum order deviation.

3) Start time

The constraint condition corresponding to the start time is: if the start time is valid, the generation time of the candidate unit is after the start time.

4) Maximum time deviation

The time offset of a media unit refers to the time interval between the generation of the media unit and the latest media unit, and the constraint condition of the maximum time offset is: if the maximum time offset is valid, the time offset of the candidate media unit is small At or equal to the maximum time offset;

5) Minimum priority

Each media unit is associated with a priority, and the constraint condition corresponding to the minimum priority is: if the minimum priority is valid, the priority of the candidate media unit is greater than or equal to the minimum priority; if the media segment request carries The other parameters of the second type do not limit the range of the candidate media unit, and the range of the candidate media unit is specified by default.

6) Maximum priority

Each media unit is associated with a priority, and the constraint condition corresponding to the maximum priority is: if the maximum priority is valid, the priority of the candidate media unit is less than or equal to the maximum priority; if the media segment request carries The other parameters of the second type do not limit the range of the candidate media unit, and the range of the candidate media unit is specified by default.

The validity and invalidity of the second type of parameters mentioned above refers to whether the value of the parameter is within a specified range. Taking the starting serial number as an example, the value of the starting serial number cannot exceed the serial number of the current latest media unit. On the other hand, to ensure real-time performance, the value of the starting serial number cannot be earlier than the serial number of an existing media unit. The starting number within the above range is valid. If a second type parameter is invalid, it is equivalent to not carrying this second type parameter. When all the parameters of the second type are invalid, the candidate media unit is the media unit specified by default.

It should be pointed out that in other embodiments, a new second type of parameter can be defined, but, if under certain mapping rules, parameter A can be mapped to parameter B, and the constraint conditions corresponding to parameter A can be converted into parameters B's constraint, then these two parameters are called equivalent parameters. Equivalent parameters should not be regarded as multiple second-type parameters, but should still be regarded as different implementations of the same parameter. For example, a second type of parameter can be defined-the minimum sequence number, and the constraint condition corresponding to the minimum sequence number is: if the minimum sequence number is valid, the sequence number of the candidate media unit should be greater than or equal to the minimum sequence number, the large So it means that the serial number is after the minimum serial number. Comparing the constraints of the minimum sequence number and the start sequence number, we can find that if a mapping rule is established: minimum sequence number = start sequence number + 1, then the candidate media units included in the constraints corresponding to the start sequence number and the minimum sequence number will be exactly the same. Therefore, the minimum sequence number and the starting sequence number are actually equivalent parameters, which should be regarded as different implementation methods of the same second type parameter, and should not be regarded as a new second type parameter.

The following implementation process takes the RTP real-time stream as an example. The media unit is an RTP packet, and each RTP packet carries a packet sequence number, which is used to indicate the order in which the RTP packets are generated. Assuming that the packet sequence number of the newly generated RTP packet is 1020, when the server receives the following media segment request:

1) GET "http://www.xxx-server.com/msreq?seqBegin = 1005" [req3]

The request only carries a second type of parameter: the start sequence number, there are 15 RTPs that satisfy the constraint conditions corresponding to the start sequence number (packet sequence numbers from 1006 to 1020), then the candidate media units to be sent include these 15 RTPs package;

2) GET "http://www.xxx-server.com/msreq?maxSeqOffset=10" [req5]

The request only carries one parameter of the second type: maximum sequence bias, there are 11 RTP packets that satisfy the constraints corresponding to the maximum sequence bias (packet sequence numbers from 1010 to 1020, where the sequence bias of the RTP packet with the packet sequence number 1010 is 10, the sequence bias of the RTP packet whose packet sequence number is 1020 is 0), then the candidate media unit to be sent includes 11 RTP packets (the packet sequence number is from 1010 to 1020);

3) GET "http://www.xxx-server.com/msreq?seqBegin = 1010 &

maxSeqOffset = 5 ”[req10]

The request carries two parameters of the second type: the starting sequence number and the maximum sequence offset. The candidate unit needs to meet two constraints. The first constraint is that the sequence number of the candidate media unit should be greater than 1010, which satisfies the constraint There are 10 candidate media units (packet numbers from 1011 to 1020). The second condition is that the sequence bias of the candidate media units does not exceed 5. The candidate media units satisfying this constraint condition have 6 RTP packets (packet numbers from 1015 to 1020). Therefore, the candidate media unit that simultaneously satisfies the above two constraints includes 6 RTP packets (packet numbers from 1015 to 1020).

The following implementation process takes a TS real-time stream as an example. The media unit is a TS segment, and each TS segment is associated with a generation time, and the generation time is a time stamp of the first media frame in the TS segment. Assuming that the generation time of the newly generated TS segment is 33000 (in microseconds), when the server receives the following media segment request:

1) GET "http://www.xxx-server.com/msreq?timeBegin=31000"[req4]

The request only carries a second type of parameter: the start time, the constraint condition corresponding to the start sequence number is that the generation time of the TS segment should be after the start time, then the candidate media unit includes the generation time Tp in the range (31000 <Tp <= 33000) all TS fragments;

2) GET "http://www.xxx-server.com/msreq?

maxTimeOffset = 2000 "[req6]

The request only carries a second type of parameter: maximum time offset. The constraint condition that satisfies the maximum time deviation is: the generation time interval between the candidate media unit and the latest media unit is less than or equal to 2000, that is, the candidate media unit includes all TS segments within the generation time Tp range (31000 <= Tp <= 33000);

3) GET "http://www.xxx-server.com/msreq?timeBegin=31000&

maxTimeOffset = 3000 ”[req11]

The request carries two parameters of the second type: start time and maximum time offset. The constraint condition corresponding to the start time is that the generation time of the TS segment should be greater than 31000, then the candidate media unit includes all TS segments whose generation time Tp is within the range (31000 <Tp <= 33000); the constraint condition corresponding to the maximum time offset Is that the generation time interval between the candidate media unit and the latest media unit is less than or equal to 3000, and the candidate media unit that satisfies the constraint includes all TS segments whose generation time Tp is within the range (30000 <= Tp <= 33000), so the The candidate media units to be transmitted that satisfy the above two constraints are all TS segments with a generation time Tp in the range (31000 <Tp <= 33000).

The following implementation process takes the RTP stream with priority as an example, the media unit is an RTP packet, each RTP packet carries a packet sequence number, and a priority is associated according to the payload content carried by the RTP packet. The priority of each media unit can be defined according to specific circumstances. For example, when the RTP packet is a multimedia data stream, the RTP packet can be divided into three priorities:

Priority 3: The audio information encapsulated in the RTP packet;

Priority 2: The key video information is encapsulated in the RTP package;

Priority 1: The non-critical video information encapsulated in the RTP packet;

Assume that the packet sequence number of the newly generated RTP packet is 1020. Starting from the RTP packet with the packet sequence number 1000, the associated priority of each RTP packet is shown in Table 1. Among them, Table 1 is a priority table associated with each RTP packet.

Table 1

Package number	Generation time	priority	Package number	Generation time	priority
1000	29500	3	1011	32000	2
1001	31000	1	1012	32000	3
1002	31000	2	1013	32500	1
1003	31000	2	1014	32500	2
1004	31000	3	1015	32500	2
1005	31500	1	1016	32500	3
1006	31500	2	1017	33000	1
1007	31500	2	1018	33000	2
1008	31500	3	1019	33000	2
1009	32000	1	1020	33000	3
1010	32000	2	A	A	A

When the server receives the following media segment request:

1) GET "http://www.xxx-server.com/msreq?minPriority=3" [req7]

The request only carries a second type parameter: minimum priority. Since there is no other second type parameter to limit the scope of the media unit, the constraint condition that the candidate media unit needs to satisfy first is that its serial number is within a range specified by default . Here, if the first preset value is 20, the serial number range specified by default is: 20 RTP packets from 1001 to 1020; at the same time, the constraint condition that the candidate media unit needs to meet is that its priority is greater than or equal to the minimum priority Therefore, there are 5 RTP packets that satisfy the above constraints at the same time (packet sequence numbers are 1004, 1008, 1012, 1016, and 1020, respectively).

2) GET "http://www.xxx-server.com/msreq?maxPriority=1" [req8]

The request only carries a second type parameter: maximum priority. Since there is no other second type parameter to limit the scope of the media unit, the first constraint that the candidate media unit needs to satisfy is that its serial number is within a range specified by default . Here, if the first preset value is 20, then the default designated serial number range is: 20 RTP packets from 1001 to 1020; at the same time, the constraint condition that the candidate media unit needs to meet is that its priority is less than or equal to the maximum priority Therefore, there are 5 RTP packets that satisfy the above constraints at the same time (packet numbers are 1001, 1005, 1009, 1013, and 1017, respectively).

In this embodiment, the client can obtain the most recently generated media unit by continuously submitting the media segment request and carrying the second type parameters such as the start sequence number or the start time to realize the real-time transmission of the media stream. When the client determines that the network cannot transmit all the media units in time, it can carry the second type of parameters in the media segment request-the maximum sequence offset and the maximum time offset. The server will only transmit the most recently generated media units, discarding the earlier generation The media unit that has not been delivered in time is implemented so as to realize the preferential delivery of the latest media content by the client. In addition, when the client judges that the network cannot transmit all the media units in time, it can also carry the second type of parameters in the media segment request-the minimum priority, and the server will only transmit media units with a priority higher than the minimum priority. Realize the priority transmission of high-priority media units by the client.

Embodiment 4 In the following embodiments, it will be described how the server sorts the candidate media units according to the unit sorting method and encapsulates them into media segments when generating media segments.

Further, in an embodiment of the present invention, the unit sorting method is one of the following basic sorting methods: sequence number forward, sequence number reverse, generation time forward, and generation time reverse. When a priority is associated with a media unit, the basic sorting method also includes two types: high priority priority and low priority priority.

It can be understood that, when the unit sorting mode is the sequence number forward, the candidate media units are encapsulated into the media segment according to the sequence indicated by the sequence number, that is, the media unit with the earlier serial number is located in the media segment in the front; the unit is sorted When the serial number is reversed, the media unit with the later serial number is positioned higher in the media segment; when the unit sorting method is positive generation time, the media unit with the earlier generation time is positioned higher in the media segment; When the unit sorting method is reversed in generation time, the later the generation time, the higher the position of the media unit in the media segment; when the unit sorting method is high priority, the high priority media unit is ranked in front of the low priority media unit. ; When the unit sorting method is low priority first, the low priority media unit is ranked in front of the high priority media unit.

For some media streams, there are cases where the generation time and even priority of multiple media units with consecutive serial numbers may be the same. Therefore, if only one basic sorting method is used (such as reverse time or high priority), there are still multiple media units with the same sorting position. For these media units, a default basic sorting can be selected. The method (such as the serial number is positive) and then sort. In another implementation method, the unit sorting method may be a cascade of multiple basic sorting methods: first basic sorting method + second basic sorting method + third basic sorting method, and so on. At this time, the sorting of the candidate media units in the order specified by the unit sorting method includes: first sorting the candidate media units according to the first basic sorting method, and then sorting the candidate media units with the same position after sorting according to the second basic Sort by sorting, and so on. If there are still media units in the same position after sorting according to the specified multiple basic sorting methods, the media units are sorted according to a default basic sorting method (sequence number forward).

The following implementation process takes the RTP stream with priority as an example, the media unit is an RTP packet, each RTP packet carries a packet sequence number, and a priority is associated according to the payload content carried by the RTP packet. The priority of each media unit can be defined according to the specific situation. For example, when the RTP packet is a multimedia data stream, the RTP packet can be divided into three priorities: priority 1, priority 2, and priority 3. Assuming that the packet sequence number of the newly generated RTP packet is 1020, starting from the RTP packet with the packet sequence number 1000, the associated priority of each RTP packet is shown in Table 1. When the server receives the following media segment request:

1) GET "http://www.xxx-server.com/msreq?

unitSortMode = SEQ_BACKWARD ”[req9]

The request does not carry parameters of the first type and parameters of the second type. Therefore, the target media stream is a media stream designated by default; the candidate media unit is a media unit designated by default, and a first preset is set If the value is 20, the media unit specified by default includes the 20 most recently generated RTP packets in the target media stream (packet sequence numbers from 1001 to 1020). The ordering of the units carried in this request is the reverse sequence number. Therefore, when the 20 RTP packets are encapsulated into the media segment, the media unit with the later sequence number is positioned higher in the media segment, that is, the candidate media unit is encapsulated into the media The sequence in the segment is shown in Figure 3.

2) GET "http://www.xxx-server.com/msreq?timeBegin=31000&

unitSortMode ＝ TIME_BACKWARD ”[req13]

The request carries a second type of parameter: the start time. Therefore, the constraint condition that the candidate media unit needs to meet is that the generation time of the RTP packet is after the start time. There are 16 RTP packets that satisfy the above constraints (packet sequence numbers from 1005 to 1020). The ordering of the units carried in the request is reversed in time. Therefore, the later the generation time (ie, the latest generation) of the RTP packet is in the media segment, the more advanced, in addition, for RTP packets with the same generation time, the default is The province sorting method is that the sequence number is positive, and the order in which the final candidate media units are packed into the media segment is shown in FIG. 4.

3) GET "http://www.xxx-server.com/msreq?maxTimeOffset=2000&

unitSortMode = HIGH_PRIORITY_FIRST + TIME_BACKWARD "[req15]

The request carries a second type of parameter, the maximum time offset. Therefore, the constraint condition that the candidate media unit needs to meet is that the time offset of the RTP packet should be less than or equal to the maximum time offset. Since the generation time of the latest media unit is 33000 and the maximum time offset is 2000, the generation time range of the candidate media unit should be all RTP packets whose generation time Tp is within the range (31000 <= Tp <= 33000). The unit sorting method carried in the request is a concatenation of two basic sorting modes: the first basic sorting mode is high priority first, and the second basic sorting mode is time reversal, therefore, the priority is Candidate units are sorted. Then, the candidate units with the same position after sorting are sorted in reverse time. Finally, if there are candidate units with the same position, they are sorted according to the default sequence number. The final candidate media unit is packaged. The sequence into the media segment is shown in Figure 5.

When the network transmission conditions are poor, you can set the unit sorting method in the request to reverse the serial number or reverse the generation time, so that the newly generated media unit can be delivered to the client first, ensuring priority for the latest media unit Send. On the other hand, it is also possible to encapsulate the high-priority media unit in front of the media segment by setting the unit sorting mode to high-priority priority to ensure the client's priority delivery of the high-priority media unit.

According to the real-time delivery method of the media stream provided by the embodiment of the present invention, the media segment is generated in real time according to the request of the client and returned to the client, so as to realize the real-time media stream delivery segmented according to the client's needs. The duration of the media segment will be Automatically adapt to changes in network transmission bandwidth, the client can control the length of the media segmentation through active requests. When the transmission bandwidth between the client and the server is sufficient and the delay is small, the client can make a request more quickly, thereby obtaining Shorter media segments reduce real-time transmission delay; when the transmission bandwidth between the client and the server is insufficient and the delay is large, the client can extend the interval for submitting requests, thereby obtaining a longer media segment and reducing requests Times to reduce the transmission overhead; because each media segment is triggered by the request of the client, the manifest file is no longer needed, and the client does not need to request and parse the manifest file. On the one hand, the client can get the latest media stream more quickly , Reducing the transmission delay of real-time media streams, on the other hand, it also reduces the transmission overhead and processing overhead caused by the manifest file. Finally, the client can also control the generation time and arrangement order of the media units in the media segment by request , When the network transmission conditions are poor, ensure that the newly generated media unit or the media unit with higher priority is sent in time as much as possible, thereby effectively reducing the media stream transmission delay and overhead, and supporting the latest media unit and high priority media unit Priority transmission.

Next, a real-time delivery server for media streams according to an embodiment of the present invention will be described with reference to the drawings.

6 is a schematic structural diagram of a media stream real-time delivery server according to an embodiment of the present invention.

As shown in FIG. 6, the real-time delivery server 10 of the media stream includes: a client interface component 100 and a media segment generation component 200.

The client interface component 100 is used to receive the media segment request sent by the client and return the corresponding media segment, where the media segment request does not carry or carries at least one control parameter, and the control parameter includes a target media stream to be transmitted. The first type of parameter, the second type of parameter indicating the candidate media unit to be transmitted and the unit sorting manner. The media segment generation component 200 generates a media segment according to the media segment request, wherein the target media stream to be transmitted is determined according to the first type of parameter, the candidate media unit to be transmitted is determined according to the second type of parameter, and the candidate media unit to be transmitted is The order specified by the unit sorting method is sorted and packaged into media segments, and the media segments are sent to the client through the client interface component 100. The server 10 according to the embodiment of the present invention can generate media segments in real time according to the request of the client and return them to the client, so as to realize real-time media stream delivery segmented according to the needs of the client, thereby effectively reducing the media stream transmission delay and overhead, and Support the priority transmission of the latest media unit and high priority media unit.

Specifically, the client interface component 100 is used to receive the client's media segment request and send the generated media segment to the client; the media segment request can carry 0, 1 or more control parameters; the control parameters include the following categories : First-type parameters, second-type parameters, and unit sorting method; first-type parameters are used to indicate the target media stream to be transmitted; second-type parameters are used to indicate the candidate media unit to be transmitted. The client interface component can use any specified protocol to receive media segment requests. For example, when the HTTP protocol is used, the client interface component can be a Web server that can receive any media segment request using the http protocol and return it through an HTTP response. Media segment; when the TCP protocol is used, the client interface component is a TCP server and provides a fixed service port.

The media segment generation component 200 is used to generate the required media segment according to the client's media segment request. Obtain the media segment request and its control parameters from the client interface component, determine the target media stream to be transmitted according to the first type of parameters, determine the candidate media unit to be transmitted according to the second type of parameters, and extract it from the media stream storage unit The candidate media units to be transmitted are encapsulated into media segments in the order specified by the unit sorting method, and then directly sent to the client interface component for sending.

Further, in an embodiment of the present invention, the media segment generation component 200 is further configured to: when the media segment request does not carry the first type of parameters, the target media stream to be transmitted is the media stream specified by default, and in the media segment When the request does not carry the second type of parameters, the candidate media unit includes a media unit designated by default, and the media unit designated by default is a media unit in which the sequence number interval of all and latest media units in the target media stream is less than the first preset value, Or for all media units in the target media stream whose generation time interval is less than the second preset value, and when the media segment request does not carry the unit sorting mode, the candidate media units are sorted according to the default specified unit sorting mode To encapsulate into media segments.

Further, in an embodiment of the present invention, the media segment generation component 200 is further configured to carry at least one second type parameter in the media segment request, where each second type parameter corresponds to at least one constraint condition of the candidate media unit At this time, the candidate media units to be transmitted include all media units in the target media stream that simultaneously satisfy all the constraint conditions corresponding to the parameters of the second type.

Further, in an embodiment of the present invention, each media unit is associated with a sequence offset. The sequence offset refers to the interval between the sequence number of the media unit and the latest media unit. The second type of parameters includes the starting sequence number and / or the maximum sequence offset. , Where the constraint condition corresponding to the starting sequence number is: if the starting sequence number is valid, the sequence number of the candidate media unit is after the starting sequence number; the constraint condition corresponding to the maximum sequence deviation is: if the maximum sequence deviation is valid, the candidate media unit's The order deviation is less than or equal to the maximum order deviation.

Further, in an embodiment of the present invention, each media unit is associated with a time offset. Time offset refers to the time interval between the generation of the media unit and the latest media unit. The second type of parameters includes the start time and / or the maximum time. Where the constraint condition corresponding to the start time is: if the start time is valid, the generation time of the candidate media unit is after the start time; the constraint condition corresponding to the maximum time offset is: if the maximum time offset is valid, the candidate media The time offset of the unit is less than or equal to the maximum time offset.

Further, in an embodiment of the present invention, each media unit is associated with a priority. The second type of parameter includes a minimum priority, and the constraint conditions corresponding to the minimum priority include: if the minimum priority is valid, the candidate media unit The priority of is greater than or equal to the minimum priority; if the other second-type parameters carried in the media segment request do not define the range of candidate media units, the range of candidate media units is specified by default.

Optionally, in an embodiment of the present invention, the unit sorting mode is one of the following basic sorting modes: serial number forward, serial number reverse, generation time forward, and generation time reverse.

Optionally, in an embodiment of the present invention, each media unit is associated with a priority, and the unit sorting mode is one of the following basic sorting modes: sequence number forward, sequence number reverse, generation time forward, and generation time reverse Direction, high priority first, low priority first.

Optionally, in an embodiment of the present invention, the unit sorting mode is a cascade of multiple basic sorting modes, and sorting the candidate media units in the order specified by the unit sorting mode includes: sorting the candidate media units according to the first basic sorting mode Sort, and sort the candidate media units in the same position after sorting according to the second basic sorting manner, and so on until sorting is completed.

It should be noted that the foregoing explanation of the embodiment of the method for real-time delivery of media streams of media streams is also applicable to the real-time delivery server for media streams of this embodiment, and details are not described herein.

According to the real-time delivery server of the media stream provided by the embodiment of the present invention, the media segment is generated in real time according to the request of the client and returned to the client, so as to realize the real-time media stream delivery according to the client's needs. The length of the media segment will be Automatically adapt to changes in network transmission bandwidth, the client can control the length of the media segmentation through active requests. When the transmission bandwidth between the client and the server is sufficient and the delay is small, the client can make a request more quickly, thereby obtaining Shorter media segments reduce real-time transmission delay; when the transmission bandwidth between the client and the server is insufficient and the delay is large, the client can extend the interval for submitting requests, thereby obtaining a longer media segment and reducing requests Times to reduce the transmission overhead; because each media segment is triggered by the request of the client, the manifest file is no longer needed, and the client does not need to request and parse the manifest file. On the one hand, the client can get the latest media stream more quickly , Reducing the transmission delay of real-time media streams, on the other hand, it also reduces the transmission overhead and processing overhead caused by the manifest file. Finally, the client can also control the generation time and arrangement order of the media units in the media segment by request , When the network transmission conditions are poor, ensure that the newly generated media unit or the media unit with higher priority is sent in time as much as possible, thereby effectively reducing the media stream transmission delay and overhead, and supporting the latest media unit and high priority media unit Priority transmission.

In order to implement the above embodiments, the embodiments of the present invention also provide a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the above embodiment is implemented. Describes the real-time delivery method of media streams.

In order to implement the above-mentioned embodiments, an embodiment of the present invention also proposes a non-transitory computer-readable storage medium. When the program is executed by a processor, the method for real-time delivery of media streams as described in the above embodiments is implemented.

In order to implement the above embodiments, an embodiment of the present invention also proposes a computer program product. When instructions in the computer program product are executed by a processor, a real-time delivery method of media streams as described in the above embodiments is executed.

In addition, the terms “first” and “second” are used for description purposes only, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined with "first" and "second" may include at least one of the features either explicitly or implicitly. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless specifically defined otherwise.

In the description of this specification, the description referring to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means specific features described in conjunction with the embodiment or examples , Structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. In addition, without contradicting each other, those skilled in the art may combine and combine different embodiments or examples and features of the different embodiments or examples described in this specification.

Although the embodiments of the present invention have been shown and described above, it can be understood that the above-mentioned embodiments are exemplary and cannot be construed as limitations to the present invention, and those of ordinary skill in the art can The embodiments are changed, modified, replaced, and modified.

Claims (21)

1. A method for real-time delivery of media streams, characterized in that the media stream is a sequence of media units generated in real time, wherein each media unit is associated with a generation time and / or a sequence number indicating the generation order, the method include:

   Receiving a media segment request sent by a client, where the media segment request does not carry or carries at least one control parameter, and the control parameter includes a first type parameter indicating a target media stream to be transferred, and a candidate media unit indicating to be transferred The second type of parameters and unit sorting method;

   Generate a media segment according to the media segment request, wherein the target media stream to be transmitted is determined according to the first-type parameter, the candidate media unit to be transmitted is determined according to the second-type parameter, and the The transmitted candidate media units are sorted and packaged into the media segments in the order specified by the unit sorting manner; and

   Send the media segment to the client.
2. The real-time delivery method of the media stream according to claim 1, wherein the generating a media segment according to the media segment request further comprises:

   If the media segment request does not carry the first type of parameters, the target media stream to be transmitted is a media stream specified by default;

   If the second segment parameter is not carried in the media segment request, the candidate media unit includes a media unit designated by default, and the media unit designated by default is all and latest media units in the target media stream A media unit whose sequence number interval is less than the first preset value, or a media unit whose generation time interval of all and latest media units in the target media stream is less than the second preset value;

   If the media segment request does not carry the unit sorting mode, the candidate media unit is encapsulated into the media segment according to the unit sorting mode specified by default.
3. The real-time delivery method of the media stream according to claim 1, wherein the generating the media segment according to the media segment request further comprises:

   If the media segment request carries at least one parameter of the second type, wherein each parameter of the second type corresponds to at least one constraint condition of the candidate media unit, the candidate media unit to be transmitted includes the target All media units in the media stream that simultaneously satisfy all the constraint conditions corresponding to the parameters of the second type.
4. The method for real-time delivery of media streams according to claim 3, characterized in that each media unit is associated with a sequence offset, and the sequence offset refers to the interval between the sequence number of the media unit and the latest media unit. The second type of parameters include the starting sequence number and / or maximum sequence deviation, where,

   The constraint condition corresponding to the starting sequence number is: if the starting sequence number is valid, the sequence number of the candidate media unit is after the starting sequence number;

   The constraint condition corresponding to the maximum order offset is: if the maximum order offset is valid, the order offset of the candidate media unit is less than or equal to the maximum order offset.
5. The method for real-time delivery of media streams according to claim 3, wherein each media unit is associated with a time offset, and the time offset refers to a time interval between the generation of the media unit and the latest media unit. The second type of parameters includes the start time and / or the maximum time offset, where,

   The constraint condition corresponding to the start time is: if the start time is valid, the generation time of the candidate media unit is after the start time;

   The constraint condition corresponding to the maximum time offset is: if the maximum time offset is valid, the time offset of the candidate media unit is less than or equal to the maximum time offset.
6. The method for real-time delivery of media streams according to claim 3, wherein each media unit is associated with a priority level, and the second type of parameters includes a minimum priority level, and a constraint condition corresponding to the minimum priority level include:

   If the minimum priority is valid, the priority of the candidate media unit is greater than or equal to the minimum priority;

   If the other second-type parameters carried in the media segment request do not define the range of candidate media units, the range of the candidate media units is specified by default.
7. The real-time delivery method of the media stream according to claim 1, wherein the unit sorting method is one of the following basic sorting methods: sequence number forward, sequence number reverse, generation time forward, and generation time reverse.
8. The method for real-time delivery of media streams according to claim 1, wherein each media unit is associated with a priority, and the unit sorting method is one of the following basic sorting methods: serial number forward, serial number reverse , Forward time, reverse time, high priority, low priority.
9. The method for real-time delivery of media streams according to claim 7 or 8, wherein the unit sorting method is a cascade of multiple basic sorting methods, and the sorting of candidate media units in the order specified by the unit sorting method includes : Sort the candidate media units according to the first basic sorting method, and sort the candidate media units with the same position after sorting according to the second basic sorting method, and so on until the sorting is completed.
10. A media stream real-time delivery server, characterized in that the media stream is a sequence of media units generated in real time, wherein each media unit is associated with a generation time and / or a sequence number indicating the generation order, the server include:

    The client interface component is used to receive the media segment request sent by the client and return the corresponding media segment, where the media segment request does not carry or carries at least one control parameter, and the control parameter includes a message indicating the target media stream to be transmitted The first type of parameters, the second type of parameters indicating the candidate media units to be transmitted and the unit sorting manner;

    A media segment generation component generates a media segment according to the media segment request, wherein the target media stream to be transmitted is determined according to the first-type parameter, and the candidate media unit to be transmitted is determined according to the second-type parameter , Sort and encapsulate the candidate media units to be transmitted in the order specified by the unit sorting manner into the media segments, and send the media segments to the client through the client interface component.
11. The real-time delivery server for media streams according to claim 10, wherein the media segment generation component is further configured to: when the media segment request does not carry the first type of parameters, the target media to be transmitted The stream is a media stream designated by default, and when the media segment request does not carry the second-type parameter, the candidate media unit includes a media unit designated by default, and the media unit designated by default is A media unit whose serial number interval between all and latest media units in the target media stream is less than a first preset value, or a media unit whose generation time interval between all and latest media units in the target media stream is less than a second preset value, And when the media segment request does not carry the unit sorting mode, the candidate media unit is encapsulated into the media segment according to the unit sorting mode specified by default.
12. The real-time delivery server for media streams according to claim 10, wherein the media segment generation component is further configured to carry at least one parameter of the second type in the media segment request, wherein, each of the first When the second-type parameter corresponds to at least one constraint condition of the candidate media unit, the candidate media unit to be transmitted includes all media units in the target media stream that simultaneously satisfy all the constraint conditions corresponding to the second-type parameter.
13. The real-time delivery server of the media stream according to claim 12, wherein each media unit is associated with a sequence offset, and the sequence offset refers to a sequence number interval between the media unit and the latest media unit, the The second type of parameters include the starting sequence number and / or maximum sequence deviation, where,

    The constraint condition corresponding to the starting sequence number is: if the starting sequence number is valid, the sequence number of the candidate media unit is after the starting sequence number;

    The constraint condition corresponding to the maximum order offset is: if the maximum order offset is valid, the order offset of the candidate media unit is less than or equal to the maximum order offset.
14. The real-time delivery server for media streams according to claim 12, wherein each media unit is associated with a time offset, and the time offset refers to a time interval between the generation of the media unit and the latest media unit. The second type of parameters includes the start time and / or the maximum time offset, where,

    The constraint condition corresponding to the start time is: if the start time is valid, the generation time of the candidate media unit is after the start time;

    The constraint condition corresponding to the maximum time offset is: if the maximum time offset is valid, the time offset of the candidate media unit is less than or equal to the maximum time offset.
15. The real-time delivery server for media streams according to claim 12, wherein each media unit is associated with a priority, and the second type of parameter includes a minimum priority, and the minimum priority corresponds to a constraint condition include:

    If the minimum priority is valid, the priority of the candidate media unit is greater than or equal to the minimum priority;

    If the other second-type parameters carried in the media segment request do not define the range of candidate media units, the range of the candidate media units is specified by default.
16. The real-time delivery server for media streams according to claim 10, wherein the unit sorting method is one of the following basic sorting methods: sequence number forward, sequence number reverse, generation time forward, and generation time reverse.
17. The real-time delivery server for media streams according to claim 10, wherein each media unit is associated with a priority, and the unit sorting mode is one of the following basic sorting modes: serial number forward, serial number reverse , Forward time, reverse time, high priority, low priority.
18. The real-time delivery server for media streams according to claim 16 or 17, wherein the unit sorting mode is a cascade of multiple basic sorting modes, and the sorting of candidate media units in the order specified by the unit sorting mode includes : Sort the candidate media units according to the first basic sorting method, and sort the candidate media units with the same position after sorting according to the second basic sorting method, and so on until the sorting is completed.
19. A computer device, characterized in that it includes a memory, a processor, and a computer program stored on the memory and that can be run on the processor, wherein when the processor executes the program, it is implemented as claimed in claim 1- The method of any one of 9.
20. A non-transitory computer-readable storage medium on which a computer program is stored, characterized in that when the program is executed by a processor, the method according to any one of claims 1-9 is implemented.
21. A computer program product, when instructions in the computer program product are executed by a processor, execute the method according to any one of claims 1-9.

Images