CN110224793B

CN110224793B - Self-adaptive FEC method based on media content

Info

Publication number: CN110224793B
Application number: CN201910549405.3A
Authority: CN
Inventors: 徐异凌; 张文军; 黄巍; 汤旭国; 李博; 孙军; 管云峰; 柳宁
Original assignee: Shanghai Jiaotong University
Current assignee: Shanghai Jiaotong University
Priority date: 2015-10-16
Filing date: 2015-10-16
Publication date: 2021-03-09
Anticipated expiration: 2035-10-16
Also published as: CN110224793A; CN106603192A; CN106603192B

Abstract

The invention provides a self-adaptive FEC method based on media content, which classifies the media content and gives different importance, dynamically adjusts the importance degree of frames contained in a media data packet and a corresponding coding scheme according to the current channel condition under the condition of not shunting the original media data stream, transmits the data packet to a corresponding FEC coder for protection in different degrees, and finally, a source data stream is only coded into an FEC code stream, thereby reducing the maximum data volume caused by FEC while ensuring the quality of the media content to the maximum; the source data stream does not need to be shunted, so that the complexity of FEC encoding at a sending end is reduced, and the efficiency of FEC encoding is improved; the increase of data volume caused by FEC coding is greatly reduced; the coding scheme can be dynamically adjusted according to the change of the current network state, so that the method has stronger adaptability to time-varying networks.

Description

Self-adaptive FEC method based on media content

The present application is a parent application 201510673091.X, which is a divisional application with the title "an adaptive FEC mechanism based on media content", filed 2015, 10, month 16.

Technical Field

The present invention relates to the technical field of multimedia transmission, and more particularly, to an adaptive FEC (forward error correction) method based on media content.

Background

In the heterogeneous network media service system, contents are distributed to terminals through an internet protocol in which media data is transmitted using IP/TCP or UDP messages or a broadcast protocol in which contents are transmitted through MPEG 2-TS. The UDP packet may be lost after passing through a plurality of network devices, and the broadcast TS stream may generate an error code due to the influence of the transmission environment, thereby causing a picture breakage or a voice pause at the terminal side.

FEC (Forward Error Correction) is a coding technique widely used in communication systems. The media data is subjected to error correction coding through the server side, redundant information is added and transmitted together, the terminal side carries out reverse FEC decoding, and the lost message is recovered. Taking a typical block code as an example, the basic principle is as follows: at a transmitting end, coding is carried out by taking kbit information as a packet, and redundancy check information of (n-k) bits is added to form a code word with the length of n bits. After the code word reaches the receiving end through the channel, if the error is in the correctable range, the error bit can be checked and corrected through decoding, so that the interference caused by the channel is resisted, the error rate of the system is effectively reduced, and the reliability of the communication system is improved.

However, the FEC processing reduces the bit error rate of the system at the cost of redundancy overhead, and excessive FEC coding also stresses the real-time performance and the network status of the system.

In the ISO-23008-1/10/13 standard, the conventional FEC structure encodes all information uniformly, with no distinction for user information. For mixed content or distribution under a mixed network, they propose a two-layer structure. Two-layer structure as shown in figure 1.

The first layer divides the source packet block into a plurality of small blocks for FEC protection respectively, and the second layer is a whole block for FEC protection. The first layer is divided more carefully to provide smaller time delay, and the second layer ensures the recovery performance and smaller redundancy.

For mixed content transmission, the content may be divided into a timed and non-timed, so that such a two-layer structure may be adopted. The time sequence content uses the mode 1 to ensure the time delay, and the non-time sequence content simultaneously uses the mode 1 and the mode 2 to ensure the accuracy.

For users with different performance channels, users with good channel performance only need FEC1 to guarantee time delay and power consumption, and for users with poor channel performance, FEC1 and FEC2 are simultaneously carried out to guarantee accuracy.

This solves the problem to a certain extent, and for users with poor channel performance (GroupB), the two-layer structure does improve the recovery performance, but introduces a significant delay. For users with better channel performance (GroupA), FEC1 is not necessarily made, and smaller packetization brings smaller delay. The unequal importance of the information is not taken into account. The classification of information or users according to a two-layer structure and the specific strategy of splitting a large block into small blcoks is a complex problem.

Disclosure of Invention

Aiming at the defects in the prior art, the invention aims to provide a media content-based adaptive FEC method, so as to solve the problems of good instantaneity, poor accuracy, good accuracy and poor instantaneity in the current FEC system and data congestion caused by excessive FEC coding.

In order to achieve the purpose, the invention adopts the following technical scheme:

a self-adapting FEC method based on media content grades the media content and gives different importance, under the condition of not shunting the original media data stream, according to the current channel condition, the importance degree of the frame contained in the media data packet and the corresponding coding scheme are dynamically adjusted, the data packet is transmitted to the corresponding FEC coder for protection of different degrees, finally, a source data stream is only coded into an FEC code stream.

Further, the method includes adding an unequal error protection flag bit to signaling information transmitted with a Media Processing Unit (MPU).

Furthermore, the method adds a special indication field, so that more personalized protection schemes can be made. Further, the special indication field is added before the MFU packet header.

Further, the method ranks the media content and assigns different importance, and then codes with different FEC coding strength using signaling and indication field control.

Further, the method can dynamically change the coding scheme according to the current network condition, and provides self-adaptive protection strength.

Furthermore, the method adds unequal error protection zone bits in signaling information transmitted together with a Media Processing Unit (MPU), and adds a special indication field at the same time to form a more personalized protection scheme.

Preferably, the method ranks media content and assigns different importance, and then encodes with different FEC coding strengths using signaling and indication field control.

Further, for the MMT AL-FEC transmitting end architecture, the transmitting end flow is:

a) the server side generates an MMTP stream and a signaling according to the media resources;

b) transmitting the MMT packet to an FEC encoder according to the difference of the importance of each frame of data in the MMT stream, and carrying out FEC encoding on the MMT packet by adopting different FEC encoding matrixes to generate corresponding FEC codes according to different priorities;

c) integrating FEC codes formed after FEC encoding is carried out on MMT packets with different priorities of the same data stream in the previous step to generate an FEC code stream;

d) after FEC encoding is finished, corresponding repair characters, FEC data load identifications and original data load identifications are returned;

e) all repair characters are packed into an FEC repair packet and sent to the transport layer.

Further, the invention identifies the FEC stream and the adopted FEC coding structure and FEC code through signaling, and in order to support this method, the FEC _ flow _ descriptor field in the signaling is modified: adding a new FEC _ coding _ structure on the basis of three existing FEC _ coding _ structures in the AL-FEC message, wherein the function of the FEC _ coding _ structure is used for describing the currently adopted FEC coding scheme, including information such as a selected coding algorithm, whether a private coding scheme is adopted, and the maximum protection time window time and value, and the field is located in the AL-FEC signaling and is transmitted to a receiving end; selecting the value of the flag bit of the newly added fec _ coding _ structure in the range of the existing reserved;

the original signaling only supports the repair information of multiplexing a plurality of media resources in one FEC flow, and the signaling control of FEC is added for different importance parts of one media resource, so that the FEC method can be more refined, the service end can dynamically adjust the FEC intensity of different parts of the media resource according to the network state of a user, and a balance point is obtained between the network bandwidth and the user experience.

Compared with the prior art, the invention has the following beneficial effects:

by adopting the technical scheme of the invention, different importance can be given to the media contents by grading aiming at the data congestion caused by excessive coding in the current FEC system, and different FEC coding strengths are adopted by utilizing signaling and indication bit control; the maximum quality of the media content is guaranteed, and meanwhile, the extremely large data volume caused by FEC is reduced; the source data stream does not need to be shunted, so the complexity of FEC encoding at a sending end is reduced, and the efficiency of FEC encoding is improved; because only one FEC code stream is generated by encoding one source data stream, the increase of data volume caused by FEC encoding can be greatly reduced; meanwhile, the coding scheme can be dynamically adjusted according to the change of the current network state, so that the method has stronger adaptability to time-varying networks.

Drawings

Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:

fig. 1 is a FEC two-layer structure for media resources in MMT;

FIGS. 2 a-2 b are graphs of the dependency of each frame in a group of pictures;

FIG. 3 is a schematic diagram of the components and importance of portions of a general MPU;

fig. 4 is a diagram of a modified MMT AL-FEC transmit end architecture.

Detailed Description

The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications can be made by persons skilled in the art without departing from the spirit of the invention. All falling within the scope of the present invention.

As shown in fig. 1: and the MMT has a two-layer structure aiming at the FEC of the media resources, the first layer divides the source packet block into a plurality of small blocks to be respectively subjected to FEC protection, and the second layer is a whole block to be subjected to FEC protection. The first layer is divided more finely to provide less delay, and the second layer ensures recovery performance and less redundancy, but this flexibility is not sufficient.

As shown in fig. 2 a-2 b: the dependency of each frame in a group of pictures illustrates that the dependency and importance of different frames in a group of pictures are different, I-frame is the most important, the former P-frame is more important than the latter P-frame, and B-frame is the least important, so that the hierarchical FEC can be performed according to the difference in importance. This is also the basis for FEC.

The first embodiment is as follows:

the present embodiment takes the MMT transmission protocol as an example

In the MMT scheme, MFU packets under the MPU packet have different importance, and unequal error protection is lacking, so that a personalized transmission scheme cannot be set. In the packet of MMT, frames of different importance (e.g., I-frames and B-frames) are distinguishable (indicated) as in fig. 2 a-2B, fig. 3. However, unequal error protection is not involved in coding, and although the Two-stage structure scheme and the LA-FEC can achieve unequal error protection to a certain extent, the flexibility is low and the complexity is high. Especially for receiving users with varying channel conditions and different characteristics (such as different caches), the existing two ways have the problems of large redundancy, poor adaptability, only aiming at a single condition, incapability of considering from user experience and the like.

The solution is as follows:

1, according to packets belonging to frames with different importance degrees, the importance degrees are distinguished, and simultaneously, the coding scheme is adaptively changed by combining the channel condition and the experience feeling of a user. For example, when the channel condition is poor or the storage capacity of the user is limited, the transmitting end protects the data of the frame I more, so that the user receives the frame I with a higher probability, and the frame B and the frame P are discarded to a greater extent during reception, so that the problem that the frame B and the frame P are discarded in the subsequent processing of the receiving end is solved from the transmitting end, the bandwidth resource is saved, and more important frames can be protected by using the resource. In the time-varying channel, the FEC coding scheme may be adaptively changed according to the change of the current network condition, for example, if the network condition is degraded, the coding matrix of the FEC coding may be changed by changing the FEC seed, so as to increase the protection strength of the transmission data.

2 if no special indication field is added, then only coding can be carried out according to the existing condition, and unequal error protection can be arranged according to the importance degree of the frame. And adding an unequal error protection zone bit into signaling which is transmitted with the MPU.

But if a more personalized scheme is desired, the protection scheme is changed as the channel quality changes, customized according to the user characteristics, then a special indication field needs to be added. Because different MFUs have different importance, an indication is added in front of the MFU packet head, and more personalized protection schemes can be made.

As shown in fig. 4, the modified MMT AL-FEC transmitting end architecture mainly modifies the output after streaming in the MMT protocol, transmits MMT data packets to FEC encoders according to priorities to perform FEC encoding, selects different encoding matrices according to different priorities of the MMT data packets, and integrates FEC codes generated after the priorities are processed by the FEC encoders into one FEC code stream.

The architecture of the transmitting end is shown in figure 4.

A sending end flow:

a) and the server generates an MMTP stream and a signaling according to the media resources.

b) And transmitting the MMT packet to an FEC encoder according to the difference of the importance of each frame of data in the MMT stream, and carrying out FEC encoding on the MMT packet by adopting different FEC encoding matrixes to generate corresponding FEC codes according to different priorities.

c) And integrating the FEC codes formed after the FEC encoding is carried out on the MMT packets with different priorities of the same data stream in the previous step to generate an FEC code stream.

d) And after FEC encoding is finished, returning corresponding repair characters, FEC data load identification and original data load identification.

To support this method, the FEC _ flow _ descriptor field in the signaling is modified in the present embodiment to identify the FEC stream and the FEC coding structure and FEC code used by the signaling.

TABLE 1 description of fec _ coding _ Structure

The original signaling only supports the repair information of multiplexing multiple media resources in one FEC stream, in this embodiment, a new FEC _ coding _ structure is added, and the function of the FEC _ coding _ structure is to describe the currently used FEC coding scheme, including information such as a selected coding algorithm, whether a private coding scheme is used, and a maximum protection time window time and value, and this field is located in the AL-FEC signaling and is transmitted to the receiving end. The value of the flag bit of the new fec _ coding _ structure may be selected within the existing reserved range, and in the present scheme, it is recommended to select 0110.

The field semantics in the table are as follows:

seed: and the random seed is used as an initial condition for generating a pseudo-random number and is used for generating a pseudo-random sequence to construct a generation matrix and a check matrix of the FEC coding. There are many algorithms for selecting seed, and commonly used methods are, for example, obtaining the current time of the system, obtaining the current process ID, and the like.

num _ of _ priority _ for _ mmtps: the number of priorities in a media resource.

priority _ mapping: priority mapping for indicating the mapping relation between different data packets in media resources and the priority of the resources.

private _ fec _ flag: an indication bit indicating whether a proprietary FEC coding scheme is used.

private _ flag: an indication bit indicating whether there is a private _ field used to describe the private FEC coding scheme used.

private _ field _ length: a length field for describing the length of the field of the private FEC coding scheme.

private _ field: detailed information for describing the private FEC scheme.

priority _ id: and the priority id is used for indicating the priority of the MMT packet.

fec _ code _ id _ for _ repair _ flow: for describing the FEC coding scheme used.

repair _ flow _ id: and the 8-bit integer is used for indicating the generated FEC repair flow, and has a corresponding relation with the packet id in the packet header of the FEC repair packet.

maximum _ k _ for _ repair _ flow: a 24-bit integer describing the maximum number of source symbols in a source sysmbol block.

maximum _ p _ for _ repair _ flow: a 24-bit integer describing the maximum number of repair symbols in a repair sysmbol block.

protection _ window _ time: a guard window time indicating the maximum time difference in milliseconds between transmitting the first source or repair packet and the last source or repair packet in FEC coding.

protection _ window _ size: a protection window value indicating a maximum count value between a load of transmitting a first FEC packet and a load of transmitting a last FEC packet in the FEC encoded stream.

The modified signaling adopts different coding matrixes to respectively carry out FEC coding control on different priorities aiming at different importance parts of a media resource, only one FEC code stream is finally generated for one media data stream, the FEC method can be more refined, meanwhile, the increase of network flow caused by excessive FEC coding is reduced, the service end can dynamically adjust the FEC intensity of different parts of the media resource according to the network state of a user by adopting the scheme, after receiving the signaling, the receiving end can analyze seed and the priority of each resource packet, and the check matrix is generated according to the seed to carry out FEC decoding and recovery on the media resource. A balance is struck between network bandwidth and user experience.

The advantages of the above solution:

1, if the receiving end artificially discards the received frame (for example, B frame) according to the situation, the transmission resource is wasted, the above scheme solves the problem from the source end, and the unwanted packet is discarded more probably in the transmission process, thereby protecting the important packet to a greater extent.

2 personalized transmission scheme. Video transmission should be based on user experience, and the FEC protection degree of I-frames and B-frames can be designed for user conditions, such as visual experience, buffer conditions, etc., by performing more careful unequal error protection.

3, the FEC coding mode can be adaptively changed according to the current network condition, on the basis of FEC protection with different strengths for data with different priorities, if the packet loss rate is increased due to network condition deterioration, the seed of FEC can be adaptively adjusted, the generation matrix of FEC is changed, and the FEC strength is increased to resist the influence caused by network deterioration.

The existing AL-FEC scheme in 4MMT only supports different FEC code streams generated for MMT packets with different priorities, and the scheme provided by the scheme supports only one FEC code stream generated for one media resource data stream, so that the flow increase caused by FEC can be greatly reduced, and the network pressure is reduced.

The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes and modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention.

Claims

1. A self-adaptive FEC method based on media content is characterized in that media content is graded and endowed with different importance, under the condition that original media data stream is not shunted, the importance degree of frames contained in a media data packet and a corresponding coding scheme are dynamically adjusted according to the current channel condition, the data packet is transmitted to a corresponding FEC encoder for protection of different degrees, and finally a source data stream is only encoded into an FEC code stream;

aiming at an MMT AL-FEC sending end architecture, the sending end flow is as follows:

the server side generates an MMT stream and a signaling according to the media content;

transmitting the MMT packet to an FEC encoder according to the difference of the importance of each frame of data in the MMT stream, and carrying out FEC encoding on the MMT packet by adopting an FEC encoding matrix to generate corresponding FEC codes according to different priorities;

integrating FEC codes formed after FEC encoding is carried out on MMT packets with different priorities of the same data stream in the previous step to generate an FEC code stream;

after FEC encoding is finished, corresponding repair characters, FEC data load identifications and original data load identifications are returned;

all repair characters are packed into an FEC repair packet and sent to the transport layer.

2. The media content-based adaptive FEC method according to claim 1, wherein the server generated signaling is used at least to indicate repair characters.

3. The adaptive FEC method according to claim 1, wherein the MMT AL-FEC transmitting end architecture modifies output after streaming in the MMT protocol, transmits the MMT data packets to FEC encoders according to priorities to perform FEC encoding, selects different encoding matrices according to different priorities of the MMT data packets, and integrates FEC codes generated after the priorities are processed by the FEC encoders into one FEC code stream.

4. The media content-based adaptive FEC method as claimed in claim 1, wherein the method includes adding unequal error protection flag bits to signaling information transmitted with the media processing unit.

5. The adaptive FEC method based on media content according to claim 1, wherein the method adds a special indication field to form a more personalized protection scheme.

6. The media content-based adaptive FEC method as claimed in claim 5, wherein the dedicated indication field is added before the MFU packet header.

7. The adaptive FEC method for media content based on claim 1, wherein the method adds unequal error protection flag bit in the signaling information transmitted together with the media processing unit, and adds special indication field to form more personalized protection scheme.

8. The adaptive media-content-based FEC method as recited in claim 1, wherein the method is capable of adaptively changing the coding scheme according to the change of network conditions.

9. The media content-based adaptive FEC method according to claim 1, wherein the method identifies FEC stream and FEC coding structure and FEC code used by signaling, and in order to support this mechanism, FEC _ flow _ descriptor field in signaling is modified: adding a new FEC _ coding _ structure on the basis of three existing FEC _ coding _ structures in the AL-FEC message, wherein the function of the FEC _ coding _ structure is used for describing the currently adopted FEC coding scheme, including information such as a selected coding algorithm, whether a private coding scheme is adopted, and the maximum protection time window time and value, and the field is located in the AL-FEC signaling and is transmitted to a receiving end; selecting the value of the flag bit of the newly added fec _ coding _ structure in the range of the existing reserved;

the original signaling only supports the repair information of multiplexing a plurality of media resources in one FEC stream, and the signaling control of FEC is added for different importance parts of one media resource, so that an FEC mechanism can be more detailed, a server can dynamically adjust the FEC intensity of different parts of the media resource according to the network state of a user, and a balance point is obtained between the network bandwidth and the user experience.

10. The adaptive FEC method based on media contents according to claim 9, wherein the value of the flag bit of the new FEC _ coding _ structure is selected to be 0100.