CN113179262B

CN113179262B - Method and system for transmitting media in heterogeneous network

Info

Publication number: CN113179262B
Application number: CN202110449973.3A
Authority: CN
Inventors: 张文军; 徐异凌; 柳宁; 孙军
Original assignee: Shanghai Jiaotong University
Current assignee: Shanghai Jiaotong University
Priority date: 2018-07-11
Filing date: 2018-07-11
Publication date: 2022-12-06
Anticipated expiration: 2038-07-11
Also published as: CN110719244A; CN113179262A; WO2020011208A1; CN110719244B

Abstract

The invention provides a method and a system for transmitting media in a heterogeneous network, wherein the method comprises the following steps: performing data mapping and corresponding indication point insertion on the media data encapsulated by the first protocol; packing the media data which is subjected to mapping and indication point insertion into a TS stream; signaling information is inserted in the TS stream. The invention can make MMT compatible with TS stream when implementing on broadcast network, and make a large number of users only able to receive TS stream continue to enjoy normal media service.

Description

Method and system for transmitting media in heterogeneous network

Technical Field

The present invention relates to the field of multimedia transmission, and in particular, to a method and system for media transmission and service in a heterogeneous network.

Background

With the accelerated fusion of the traditional media and the emerging media, the overall processes of the media, such as the business form, the transmission and distribution, the receiving service, and the like, are changed greatly. The video media content format is intelligently developed towards higher definition and higher dimensionality such as 4K/VR and the like; the transmission mode is from a wired/wireless/satellite single direct broadcast mode to interactive collaboration, particularly accelerated integration with the Internet; the terminal service is changed from a traditional single fixed receiving device to a high-quality and personalized multi-screen synchronous interactive service mode supporting any time and any place. However, these changes bring new opportunities for multimedia services to develop, and also bring great challenges to intelligent organization, transmission and consumption of media services. In order to solve the problem, the broadcasting network and the broadband network need to complement the advantages and perform deep fusion, so as to construct an intelligent media fusion network which can be managed, controlled and trusted, effectively organize mass, multivariate and dynamic media in a representable, processable and reliable manner, and perform efficient content distribution in a multi-network cooperative manner.

The MMT (MPEG Media Transport) protocol is an application layer protocol developed by the international standard organization MPEG (Moving Picture Experts Group) and oriented to packet switching, and is intended to provide a transmission and distribution service for Media data in heterogeneous networks, and includes a hybrid network composed of a unidirectional physical network (such as a digital broadcast network) and a bidirectional physical network (such as an IP network), which can dynamically adapt to different network conditions in the heterogeneous networks. The MMT protocol supported technologies such as multi-network fusion and multi-screen interaction provide a perfect solution for solving the problems, the service field of the traditional television is expanded, and the watching experience of the Internet video is improved. The MMT can expand and promote the traditional video service, including the association, the index and the synchronization among the multi-element contents, and the coupling presentation of the multi-element contents and multiple users and multiple terminals, thereby realizing the interaction and the cooperation among the multiple users and the multiple terminals. In China, the working group of standards for digital audio and video coding and decoding technology has also established a corresponding standard, namely, the Smart Media Transport (SMT) protocol.

In terms of content delivery technology, an MPEG-2TS (Transport Stream) Stream transmitted through a broadcast network is still an important content transmission method. The TS supports various basic media streams and various media coding standards, is widely applied and becomes a universal standard in the media industry, and at present, a large number of existing devices can only support the TS streams. Under the condition that broadcasting and other transmission modes such as the internet are continuously fused, how to upgrade the broadcasting transmission mode and enable a large number of users who do not update the equipment to continue to use the existing equipment to enjoy normal media service becomes a problem which needs to be solved. Therefore, when a new media transport protocol represented by MMT, SMT, etc. is implemented on a broadcast network, if TS streams are compatible, new users can enjoy richer and personalized media services, and users who can only receive TS streams without equipment upgrade can still enjoy normal media services.

Disclosure of Invention

Aiming at the defects in the prior art, the invention aims to provide a method and a system for transmitting media in a heterogeneous network.

The invention provides a method for transmitting media under a heterogeneous network, which comprises the following steps:

mapping and indicating point inserting step: performing data mapping and corresponding indication point insertion on the media data encapsulated by the first protocol;

and TS stream packing step: packing the media data after mapping and indication point insertion into TS stream;

a signaling insertion step: signaling information is inserted in the TS stream.

Preferably, the first protocol is an MMT protocol.

Preferably, the mapping and indication point inserting step includes: the method comprises the steps that a sample in an MMT packaging structure is placed in a load of a PES packet in a TS packaging structure, a PES packet data mapping indication point is inserted into a PES packet header, and a TS packet data mapping indication point is inserted into a TS packet header;

the PES packet data mapping indication point indicates the data structure of the MMTP packet corresponding to the PES packet;

the sample has at least one MFU, and the TS packet data mapping indication point indicates to which MFU the data carried by the TS packet belongs.

Preferably, the TS stream packing step includes: and mapping the information in the MMTP packet header to the TS packet header.

Preferably, the signaling information includes: time synchronization information, stream mapping information, main stream presentation information and auxiliary stream related information.

The invention provides a transmission system of media under a heterogeneous network, which comprises:

the mapping and indication point insertion module: performing data mapping and corresponding indication point insertion on the media data encapsulated by the first protocol;

TS stream packaging module: packing the media data which is subjected to mapping and indication point insertion into a TS stream;

a signaling insertion module: signaling information is inserted in the TS stream.

Preferably, the first protocol is an MMT protocol.

Preferably, the mapping and indication point inserting module comprises: the sample in the MMT packaging structure is put into the load of a PES packet in the TS packaging structure, a PES packet data mapping indication point is inserted into a PES packet head, and a TS packet data mapping indication point is inserted into a TS packet head;

Preferably, the TS stream packetizing module includes: and mapping the information in the MMTP packet header to the TS packet header.

Preferably, the signaling information includes: time synchronization information, stream mapping information, main stream presentation information, and auxiliary stream related information.

Compared with the prior art, the invention has the following beneficial effects:

the invention can make MMT compatible with TS stream when implementing on broadcast network, and make a large number of users only able to receive TS stream continue to enjoy normal media service.

Drawings

Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:

FIG. 1 is a system framework diagram of an embodiment of the present invention;

FIG. 2 is a schematic diagram of data mapping and pointer insertion according to an embodiment of the present invention.

Detailed Description

The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications can be made by persons skilled in the art without departing from the concept of the invention. All falling within the scope of the present invention.

The MMT protocol is taken as an example for the description in the embodiment, but the invention is not limited thereto, and the invention may also be applied to other novel protocols such as SMT.

As shown in fig. 1, the media resources of the same scene in the server include multiple correlated media resources such as a main view and an auxiliary view, where the media resources of the main view are packed into a TS stream, and meanwhile, a signaling related to multiple views in an MMT protocol is inserted, and the TS stream is sent out from a broadcast channel through a DTMB exciter. For a common set top box, as only TS stream can be analyzed, MMT related signaling is automatically ignored, and a single main visual angle media service is presented at a playing terminal; for a new generation of set top boxes capable of analyzing the MMT protocol, the TS stream is compatible, information related to multi-view presentation is analyzed, and then a request is sent to a server through a broadband network according to the information. After receiving the request, the server packs the auxiliary visual angle media resources into an MMTP packet, and sends the MMTP packet to the user terminal through the broadband network, and finally realizes the synchronous presentation of the multi-visual angle media resources at the playing terminal.

The invention provides a method for transmitting media in a heterogeneous network, which comprises the following steps:

1. mapping and indicating point inserting step: and carrying out data mapping and corresponding pointer insertion on the media data encapsulated by the first protocol.

In the MPEG-2TS encapsulation structure, the largest partition unit in each video stream is a PES packet, and the PES packet is selected to correspond to the MMT in order to reduce overhead. In the MMT package structure, a sample under the movie fragment corresponds to the PES packet. Therefore, in order to recover the data structure required for MMT presentation from the TS stream at the receiving end, it is necessary to add the encapsulation information above the sample layer, i.e., MPU _ sequence _ number, movie _ fragment _ sequence _ number, and sample _ number in the MMTP packet payload, to the PES packet. Meanwhile, in the TS packet, a field MFU _ number needs to be inserted for indicating a corresponding MFU. In addition, packet _ id in the MMTP packet header, i.e., a field indicating different media resources, should correspond to PID in the TS packet.

The specific implementation is two steps, firstly, the sample in the MMT packaging structure is put into the load (payload) of the PES packet in the TS packaging structure. For example, in FIG. 2, sample # n is placed in the payload of PES packet # n. Meanwhile, a PES packet data mapping indication point is inserted in the PES packet header to indicate the data structure of the MMTP packet corresponding to the PES packet. Next, a TS packet data mapping indication point is inserted in the TS packet header for indicating to which MFU the data carried by the TS packet belongs. In this case, one TS packet can carry only one MFU data at most, but one MFU can be carried by a plurality of TS packets. For example, in fig. 2, MFU #1 is carried by TS packets #1 and #2, and MFU #2 is carried by TS packet # 3.

Specifically, when the data mapping indication point is inserted, in the PES header, the MPU _ sequence _ number, movie _ fragment _ sequence _ number, and sample _ number are carried using the PES _ extension _ field; in the TS header, the adaptation _ field is used to carry the MFU _ number.

2. And TS stream packing step: and packing the media data subjected to mapping and indication point insertion into a TS stream.

After data mapping and corresponding indication point insertion are completed, the MMT-encapsulated main view media resources need to be packaged into TS (transport stream) for transmission. When packing the TS stream, it is necessary to map the relevant information in the MMTP header to the TS header. Table 1 shows the mapping relationship between TS and corresponding fields of MMTP header portions.

TABLE 1

MPEG-2TS packet	MMTP bag
		payload_unit_start_indicator	f_i
transport_priority	priority
		PID	packet_id
continuity_counter	packet_sequence_number
		random_access_indicator	R(RAP_flag)

The meaning of each field is as follows:

payload _ unit _ start _ indicator: a payload data unit start indicator which is set to "1" when the MMTP packet f _ i field is "00" or "01"; otherwise set to "0".

transport _ priority: and the transmission priority indicator is set according to the MMTP packet priority field indication.

PID: indicates the type of data stored in the payload, and is the same as the MMTP packet _ id field value.

continuity _ counter: a continuity counter, incremented with each packet having the same PID value, with the PID corresponding to packet _ id, and so this field corresponding to packet _ sequence _ number.

random _ access _ indicator: the random access indicator, which is located in the Application Field (AF) of the TS header, corresponds to the R (RAP _ flag) Field of the MMTP header, where R is "1" and is set to "1".

3. A signaling insertion step: signaling information is inserted in the TS stream.

To implement services such as multi-view synchronous presentation, a device implementing the MMT protocol should not only be able to parse TS streams, but also include information related to multi-view presentation in TS streams. As shown in the signaling insertion module in fig. 1, the information related to the multi-view presentation that should be inserted in the main TS stream mainly includes the following four aspects:

1) Time synchronization information: for TS analysis equipment, directly analyzing and presenting TS stream; for the MMT parsing device, the TS stream needs to be parsed, the UTC Time in the MMT main view media resource needs to be converted into a PTS (Presentation Time Stamp) or DTS (Decode Time Stamp) timestamp, and the UTC Time is recovered at the terminal. The conversion equation is as follows (where PCR is the MPEG-2TS 90kHz system clock):

2) stream mapping information: the asset _ id (PA message) in the MMT protocol is converted into stream _ id (PMT table) in the TS packet, and the MMT main view stream contains only one program, so that the PAT table is not required to be used.

3) Mainstream presentation information: including the position and layout of the main view picture.

4) Auxiliary stream related information: the TS stream corresponding to the main view media resource should include an acquisition mode (such as a URL, etc.) of the auxiliary view media resource, a timestamp, and spatial layout and play indication information of each auxiliary view picture.

For the above signaling Information, the present invention embeds it in the TS stream in the form of External Media Information (TEMI). The "outer" in The External Media Information (TEMI) is relative to the TS stream, i.e. the MMT stream, etc. that needs to be presented in synchronization with the TS stream. TEMI is expressed in AF _ descriptor form, and the allocation of AF descriptor tag in TS protocol is shown in table 2.

TABLE 2

AF Descriptor Tag	Identification
		0x00-0x03	Rec.ITU-T H.222.0\|ISO/IEC 13818-1Reserved
0x04	Timeline Descriptor
		0x05	Location Descriptor
0x06	BaseURL Descriptor
		0x07	Cets_byte_range_descriptor(see note 1)
0x08-0x7F	Rec.ITU-T H.222.0\|ISO/IEC 13818-1Reserved
		0x80-0xFF	User Private

As can be seen from Table 2, the AF Descriptor contains various newly defined descriptors such as Timeline Descriptor, location Descriptor, etc. The Timeline Descriptor may be used to carry time synchronization information of the auxiliary view media resource, the Location Descriptor may be used to carry information such as the type and URL of the external media resource, and the User Private may carry other information required for presentation.

Specifically, the syntax structure of the Timeline Descriptor is shown in table 3:

TABLE 3

The main fields related to the MMT stream are as follows:

has _ timestamp: indicating whether the external media asset contains a timestamp. '0' means no time stamp, '1' means 32 bit time stamp, '2' means 64 bit time stamp. Obviously, MMT stream is UTC based 32-bit NTP timestamp, and the value of has _ timestamp should be equal to '1'.

has _ ntp: indicating whether it is an NTP timestamp. '1' denotes an NTP timestamp. The MMT stream uses NTP time, has _ NTP value of '1'.

time _ id: the time information carried by the Timeline Descriptor indicates which external media resources are valid, and corresponds to the Timeline _ id in the Location Descriptor. That is, if both values are the same, the external media resource indicated by the location _ descriptor uses the time information indicated by the Timeline _ descriptor. The field value range is [0,0x7F ].

timescale: the time scale of the external media is indicated together with the media _ timestamp. The conversion relation between the timestamp media _ timestamp and the TS timestamp is as follows:

MTi＝timescale*(PTSi-PTS0)/90000+MT0。

the timestamp of the MMT stream is 32-bit NTP time, timescale =1/232, so the time scaling relationship is as follows:

therefore, the Timeline Descriptor may be used to carry the time information of the MMT secondary view stream. Since these MMT streams all use NTP timestamps, they have the same timeline _ id. the value range of timeline _ id is [0,0x7f ], and MMT auxiliary view stream can be allocated as required.

The Location Descriptor syntax structure is shown in Table 4.

TABLE 4

The main fields related to the MMT stream are as follows:

and the time _ id corresponds to the time _ id in the Timeline Descriptor. If the two values are the same, the external media resource indicated by the Location Descriptor uses the corresponding time information.

url _ scheme: a URL scheme indicating the location use of the external media resource.

url _ path _ length: along with URL _ path, base URL length and value are indicated. Where the Base URL refers to a common part of URLs of respective external media.

nb _ addons: indicating the number of external media resources, i.e. the number of MMT secondary view streams.

service _ type: an external media resource service type. As shown in Table 5, support for MMT streams can be defined by its User private corresponding to 0x80-0 xFF.

TABLE 5

TEMI Service Type	Add-on type
		0	Specified with Mime Type
1	MPEG-DASH
		2	ISO/IEC 14496-12file
3	Rec.ITU-T H.222.0\|ISO/IEC 13818-1 Transport Stream
		0x04-0x7E	Rec.ITU-T H.222.0\|ISO/IEC 13818-1 Reserved
0x7F	Unknown service type
		0x80–0xFF	User Private

url _ subpath _ length: along with the addon _ location, the length and value of the private portion of the URL corresponding to each external media resource is indicated.

Therefore, the Location Descriptor can provide the URL scheme and the specific URL value corresponding to each MMT auxiliary view stream. The MMT receiving end can obtain the URL information of all MMT auxiliary view angle streams by analyzing the Location Descriptor contained in the TS stream, and then pull through a broadband network.

In addition, to realize multi-view synchronous rendering, rendering information, such as layout, corresponding to each view is also required. This information can be embedded in the TS stream by defining a new AF Descriptor. The method selects 0x80 from User Private corresponding to 0x80-0xFF in AF Descriptor Tag, defines a presentation information Descriptor MMT PI Descriptor indicating a main visual angle and an auxiliary visual angle MMT stream, and is used for bearing multi-visual angle presentation information (namely MPI table) contained in the MMT main visual angle stream. The MMT PI Descriptor is shown in Table 6.

TABLE 6

In summary, by carrying the Timeline Descriptor and the Location Descriptor in the TEMI information in the TS stream, the time (synchronization) information and the URL information of the MMT secondary view stream can be indicated. For MMT receiving equipment, firstly converting TS (transport stream) corresponding to a main view stream into an MMT stream; meanwhile, through the analysis of the two descriptors, the URL information of the MMT auxiliary view streams can be obtained, and further the media resources of the auxiliary view can be requested through a broadband network. The MMT parsing device may synchronize the MMT streams corresponding to the main view and the auxiliary view by combining the time information in the Timeline Descriptor. In addition, the related presentation and layout information of each media resource in the MMT streams of the main view angle and the auxiliary view angle is obtained through the MMT PI Descriptor defined by the method. So far, the definition and the use of all signaling for multi-view presentation are realized by transmitting the main view media resource through the TS stream, and requesting the auxiliary view media resource from the broadband network after the receiving terminal analyzes the main view media resource.

The above-mentioned various TEMI information, i.e. AF Descriptor, can be pre-embedded into the TS stream in two ways:

(1) TEMI can be added as an extension to the Adaptation Field in the TS packet header

Table 7 is the Adaptation Field syntax format with the AF Descriptor extension added. Setting adaptation _ field _ extension _ flag to "1" indicates that an extension is present;

let AF _ Descriptor _ not _ present _ flag be "0" to indicate that the Adaptation Field exists as AF Descriptor.

TABLE 7

(2) TEMI can also be transmitted as a single PES packet, called TEMI stream.

In this mode, the allocation of stream _ id and stream _ type is shown in table 8 and table 9, respectively. Wherein TEMI is transmitted by using private _ stream _1 (stream _ id is 10111101), and the stream _ type is 0x26. At this time, TEMI constitutes payload of the PES packet in the form of TEMI Access Unit (TEMI _ AU). One AF Descriptor may be carried by one or more TEMI _ AUs, and one TEMI _ AU may also carry one or more AF dsdescriptors. The syntax structure of TEMI _ AU is shown in table 10.

TABLE 8

TABLE 9

Watch 10

Based on the above method for transmitting media in heterogeneous network, the present invention further provides a system for transmitting media in heterogeneous network, including:

In this embodiment, the first protocol is the MMT protocol.

The mapping and indication point insertion module comprises: the sample in the MMT packaging structure is put into the load of a PES packet in the TS packaging structure, a PES packet data mapping indication point is inserted into a PES packet head, and a TS packet data mapping indication point is inserted into a TS packet head;

at least one MFU is in the sample, and the TS packet data mapping indication point indicates to which MFU the data carried by the TS packet belongs.

The TS stream packaging module comprises: and mapping the information in the MMTP packet header to the TS packet header.

The signaling information includes: time synchronization information, stream mapping information, main stream presentation information, and auxiliary stream related information.

Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, units provided by the present invention as pure computer readable program code, the system and its various devices, modules, units provided by the present invention can be fully implemented by logically programming method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system and various devices, modules and units thereof provided by the present invention can be regarded as a hardware component, and the devices, modules and units included therein for implementing various functions can also be regarded as structures within the hardware component; means, modules, units for performing the various functions may also be regarded as structures within both software modules and hardware components for performing the method.

The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The embodiments and features of the embodiments of the present application may be combined with each other arbitrarily without conflict.

Claims

1. A method for transmitting media content, characterized in that,

converting the MMT stream into a TS stream, comprising:

classifying the multi-view media content, including at least main view media content;

performing TS packaging on the main view media content;

inserting signaling information into the TS package to form a package packet and transmitting the package packet;

the method for inserting the signaling information in the TS encapsulation comprises the following steps: embedding the signaling information into a TS stream in the form of External media information Timeline and External media information, wherein the External media information is expressed in the form of adaptive domain descriptor AF _ descriptor;

the adaptive domain descriptor comprises a presentation information descriptor, and the presentation information descriptor comprises multi-view presentation information;

the signaling information comprises any one or more of the following: time synchronization information, stream mapping information, main stream presentation information and auxiliary stream related information;

the time synchronization information includes a conversion formula that converts a value based on a program clock reference PCR and a value based on a coordinated universal time UTC:

2. the method of claim 1, wherein the media content is transmitted from the server,

the classifying the multi-view media content further comprises: secondary perspective media content.

3. The method for transmitting media contents according to claim 1,

the auxiliary stream related information at least comprises any one or more of the following information: URL information, timestamp information, spatial layout information of the auxiliary view picture, and playing indication information of the auxiliary view picture.

4. The method for transmitting media contents according to claim 1 or 2,

the presentation information descriptor comprises one or more of the following syntactic structures: descriptor tag, descriptor length, descriptor payload data.

5. The method for transmitting media contents according to claim 1 or 2,

the adaptive domain descriptor further comprises any one or more of the following: timeline descriptors, address descriptors, user-defined information.

6. The method of claim 5, wherein the media content is transmitted from the server,

the timeline descriptor allocation field 0x04 is used for carrying time synchronization information;

the address descriptor allocation field 0x05 includes any one or more of the following information: address information, URL information, auxiliary view stream layout information;

and the user-defined information distribution field is 0x80-0xFF and comprises other information required by presentation.

7. The method for transmitting media contents according to claim 1 or 2,

the presentation information descriptor is allocated with a 0x80 field.

8. A media content receiving method applied to the media content transmission method according to claim 1, characterized by comprising:

receiving a TS encapsulation packet to obtain an adaptive domain descriptor;

obtaining a presentation information descriptor comprising multi-view presentation information according to the adaptive domain descriptor;

obtaining multi-view media content according to the multi-view presentation information;

the adaptive domain descriptor further comprises any one or more of the following: a timeline descriptor, an address descriptor, and user-defined information;

the timeline descriptor allocation field 0x04 is used for carrying time synchronization information, and the time synchronization information includes a conversion formula for converting a value based on a program clock reference PCR and a value based on a coordinated universal time UTC:

9. The method of receiving media contents according to claim 8,

the presentation information descriptor comprises one or more of the following syntactic structures: descriptor tag, descriptor length, descriptor payload data;

the presentation information descriptor is allocated with a 0x80 field.

10. A media content transmission apparatus, characterized in that,

converting the MMT stream into a TS stream, comprising:

a view angle division module: classifying the multi-view media content, including at least main view media content;

and a TS encapsulation module: performing TS packaging on the main view media content;

a signaling insertion module: inserting signaling information in the TS encapsulation;

a transmission module: transmitting the TS encapsulated packet inserted by the signaling information;

the method for inserting the signaling information in the TS encapsulation comprises the following steps: embedding the signaling information into a TS stream in the form of external media information, wherein the external media information is expressed in the form of adaptive domain descriptor;

including a presence information descriptor in the adaptation domain descriptor, the presence information descriptor including multi-view presence information;

the time synchronization information includes a conversion formula that converts a value based on a program clock reference PCR and a value based on coordinated universal time UTC:

11. the media content transmission apparatus according to claim 10,

12. The media content transmission apparatus according to claim 10 or 11,

13. The media content transmission apparatus according to claim 10 or 11,

the presence information descriptor is assigned a 0x80 field;

14. A media content receiving apparatus applied to the media content transmitting apparatus according to claim 10, comprising:

a package analysis module: analyzing the received TS encapsulation packet to obtain an adaptive domain descriptor;

the descriptor parsing module: analyzing the adaptive domain descriptor to obtain a presentation information descriptor for carrying multi-view presentation information;

a media content request module: obtaining multi-view media content according to a multi-view presentation information request

15. The media content receiving apparatus according to claim 14,

the presentation information descriptor includes one or more of the following syntax structures: descriptor tag, descriptor length, descriptor payload data;

the presentation information descriptor is allocated with a 0x80 field.