JP2006093812A - Method and apparatus for packetizing high-efficiency coded time series information, applying real-time streaming transmission to information, and receiving and reproducing the information - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology for supplementing packet loss by means of real-time streaming. <P>SOLUTION: A transmitter side packetizes the time series information subjected to high efficiency coding to produce a first packet stream, generates a second packet stream resulting from delaying each packet of the first packet stream by a prescribed time, multiplexes the first and second packet streams and transmits the multiplexed stream, and a receiver side receives the signal resulting from multiplexing the first and second streams, demultiplexes the multiplexed streams into the first and second streams, sequentially stores the first packet stream of a prescribed amount to a buffer, applies sequential control to the second packet stream to extract the payload, sequentially decodes extracted data to reproduce and output time series information, and acquires a related packet to a missing packet from a packet group in the buffer, to supplement the acquired packet for the sequential control processing, when a packet loss is detected in the sequential control processing of the second packet stream. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a real-time streaming technology that is the basis of Internet television, Internet radio, digital television, and the like, and more particularly, to countermeasure technology for packet loss during transmission.

  As is well known, in real-time streaming, high-efficiency encoded time-series information (video, audio, etc.) such as MPEG is packetized and transmitted. The payload is extracted, the extracted data is sequentially decoded, and the time series information is sequentially reproduced.

  As is well known, in real-time streaming, a part of a packet stream transmitted for various reasons may not reach the receiving side. Based on this fact, various countermeasure technologies have been developed in order to prevent the reproduction quality of video and audio from being significantly lowered even if a small amount of packet loss occurs.

  As a typical countermeasure technique, FEC (Forward Error Correction) is known. This means that error correction code is applied to highly efficient time-series information to packetize it, and even if a packet is lost, the receiving device can recover the payload of the lost packet from the payload of the previous and subsequent packet sets. It is a redundant system.

As another countermeasure technique, a technique for identifying a lost packet and retransmitting the packet to the transmission source for recovery is known. Japanese Patent Laid-Open No. 2002-84239 discloses a technique for applying such a retransmission method to a multicast distribution system.
JP 2002-84239 A

  According to FEC to which Reed-Solomon code or interleaving is applied, considerable information recovery capability can be realized against packet loss. However, the higher the information recovery capability, the more complicated the logic of this redundant system, and the more complicated the processing on the transmission side and the processing on the reception side. There is an obvious limit to the information recovery capability that can be achieved with an economical practical system.

  In addition, the retransmission method must be capable of high-speed data communication in both directions between the transmission device and the reception device based on the principle. It is not applicable to a broadcasting system only in one direction from the transmission side to the reception side. Even in a bidirectional communication system, real-time performance cannot be ensured unless the system is capable of processing at a very high speed from the detection of packet loss until it is replenished by retransmission.

  The object of the present invention is not limited to an ultra-high speed bidirectional communication system, but can be applied to a broadcasting system only in one direction from the transmission side to the reception side. It is to realize a high information recovery ability for.

In order to achieve the above object, according to the first aspect of the present invention, on the transmission side, the time series information encoded with high efficiency is packetized to generate a first packet stream, and each packet of the first packet stream is predetermined. Generating a second packet stream delayed in time, multiplexing and transmitting the first packet stream and the second packet stream;
On the receiving side, the multiplexed signal of the first packet stream and the second packet stream is received and separated into the first packet stream and the second packet stream, and a predetermined amount of the first packet stream is sequentially stored in the buffer, When the second packet stream is subjected to sequence control to extract the payload, the extracted data is sequentially decoded to reproduce and output time-series information, and packet loss is detected in the second packet stream sequence control processing In addition, a real-time streaming method is provided in which a packet corresponding to a missing packet is acquired from the packet group stored in the buffer and supplemented to the sequence control process.

  In addition, the second invention generates a first packet stream by packetizing time-series information that has been encoded with high efficiency, and generates a second packet stream in which each packet of the first packet stream is delayed by a predetermined time. The real-time streaming transmission method is such that the first packet stream and the second packet stream are multiplexed and transmitted.

  According to a third aspect of the present invention, multiplexed signals of the first packet stream and the second packet stream having the same contents and time difference are received and separated into the first packet stream and the second packet stream, and a predetermined amount of the first packet stream is The second packet stream is stored in the sequential buffer, the second packet stream is subjected to the sequence control, the payload is extracted, the extracted data is sequentially decoded, and the time series information is reproduced and output. In the real-time streaming reception / playback method, when a packet loss is detected in the packet, a packet corresponding to the lost packet is acquired from the packet group stored in the buffer and the sequence control process is supplemented.

  According to a fourth aspect of the present invention, there is provided means for packetizing time-series information encoded with high efficiency and generating a first packet stream, and means for generating a second packet stream obtained by delaying each packet of the first packet stream for a predetermined time. And a real-time streaming transmission apparatus comprising means for multiplexing and transmitting the first packet stream and the second packet stream.

  Further, the fifth aspect of the invention relates to means for receiving multiplexed signals of the first packet stream and the second packet stream having the same contents and a time difference, and separating the signals into the first packet stream and the second packet stream; Means for sequentially storing packet streams in a buffer; means for performing order control of a second packet stream to extract a payload; sequentially decoding extracted data to reproduce and output time-series information; and order of second packet stream A real-time streaming reception / playback device comprising means for acquiring a packet corresponding to a lost packet from the packet group stored in the buffer and replenishing the sequence control process when a packet loss is detected in the control process And

  The present invention is a complete duplex system that transmits a first packet stream and a second packet stream having the same contents, and the redundancy is extremely high. However, the error correction logic of a redundant system is very simple. On the transmission side, the first packet stream and the second packet stream having the same contents are simply multiplexed and transmitted with a slight time difference, and the process is very simple. On the receiving side, the first packet stream is buffered a little, and when packet loss is detected in all stages of the reproduction processing system of the second packet stream, all that is necessary is to apply the buffered packet. Is also very simple.

  Although it is possible to process with simple logic in this way, since it is a redundant system that is almost completely duplicated, the information recovery capability against packet loss is extremely high.

=== Real-time streaming transmitter ===
A real-time streaming transmission apparatus to which the present invention is applied includes a DVD reproduction apparatus, a packet assembly circuit, a delay circuit, a multiplexing circuit, and a transmission circuit. The DVD stores data that is highly efficient encoded by the MPEG2 system, which is well known for time-series information such as image information and audio information. When this is played back by a DVD playback device, an MPEG encoded data stream is extracted from the DVD, and this is sequentially decoded, and image information and audio information are played back and output to the attached display and speakers.

  FIG. 1A shows an outline of processing executed by the real-time streaming transmission apparatus. FIG. 2 shows a processing image of a packet created / transmitted by the transmission apparatus.

  The MPEG encoded data stream output from the DVD playback device 1 is transferred to the packet assembly circuit 2. The packet assembly circuit 2 sequentially performs packetization processing. That is, the MPEG encoded data stream is divided into appropriate lengths, and a header is attached to a packet of a standardized length as a payload. The header incorporates time stamps such as time reference information and clock signals so that the packets can be processed in order in accordance with the timing when disassembling the packets. In this way, the data stream is sequentially packetized, these packets are sent out at a predetermined rate (packet rate), and transferred to the multiplexing circuit 4 as a first packet stream.

  The packet assembly circuit 2 further transfers a packet having the same payload as each packet of the first packet stream to the delay circuit 3. The delay circuit 3 delays each packet by a predetermined time and sequentially transfers it to the multiplexing circuit 4, which is used as a second packet stream.

  In each packet, information indicating which of the first packet stream and the second packet stream belongs is incorporated in the header. For example, S1 and S2 in FIG. 2 correspond to this. The first packet stream belongs to S1, and the second packet stream belongs to S2.

  The multiplexing circuit 4 multiplexes the first packet stream and the second packet stream and transfers them to the transmission circuit 5. In multiplexing, the first packet stream and the second packet stream corresponding to each other (having the same payload) are identified. For example, a packet number indicating what number packet of each packet stream is incorporated in the header. By doing so, it can be easily understood that packets having the same packet number in each packet stream correspond to each other. In the example of FIG. 2, the numbers below the hyphen correspond to the packet number. Time stamps can also be used for this purpose.

  The multiplexing circuit 4 incorporates and multiplexes the packets in the order in which they are transferred alternately from each packet stream so as to make two packet streams into one multiplexed packet stream, and transfers the packets to the transmission circuit 5. 5 transmits to the real-time streaming reception / playback apparatus.

=== Real-time streaming reception / playback apparatus ===
A real-time streaming reception / playback apparatus to which the present invention is applied includes a reception circuit, a separation circuit, a sequence control circuit, a buffer, an MPEG decoder, and a packet supplement circuit.

  FIG. 1B shows an outline of processing executed by the real-time streaming reception / playback apparatus. When the reception circuit 11 receives a packet stream (multiple packet stream) transmitted from the real-time streaming transmission apparatus, the reception circuit 11 first transfers the packet stream to the separation circuit 12. The separation circuit 12 refers to the header of each packet and separates the first packet stream and the second packet stream. For example, in the example of FIG. 2, the distribution is performed by S1 and S2 incorporated in the header of each packet. Then, the separated packet stream is transferred to each sequence control circuit 13.

  The order control circuit 13 refers to the time stamp (or packet number in the example of FIG. 2) of the header of each packet and performs an order control process for arranging the packets in order. In this way, the packets can be arranged in the same order as the first packet stream / second packet stream generated by the real-time streaming transmission apparatus.

For the first packet stream, each packet is transferred to the buffer 14 in the order of arrangement, and stored in the buffer 14 for a predetermined period. In this case, the buffer 14 is configured to hold at least the latest delayed number or more (five or more in the example of FIG. 2).
The buffer 14 is configured so that a packet can be searched by referring to the packet header and the payload of the target packet can be extracted.

  For the second packet stream, the packet is decomposed in the order of arrangement to extract the payload, and the MPEG encoded data stream is reproduced. This MPEG encoded data stream is transferred to the MPEG decoder 15. The MPEG decoder 15 sequentially decodes the MPEG encoded data stream and outputs a data stream of image information and sound information. This is transferred to a display or a speaker for playback output.

  When the sequence control circuit 13 detects a packet loss during the sequence control process of the second packet stream, it notifies the packet supplement circuit 16 of the fact. Upon receiving the notification, the packet supplementing circuit 16 refers to the buffer 14, extracts a packet corresponding to the missing packet, and supplements the second packet stream. That is, the packet 14 (or time stamp) is referred to inside the buffer 14 as key information, and the payload of the packet corresponding to the missing packet is extracted. This is returned to the sequence control circuit 13 and supplemented as a payload of the lost packet, and the decoding process of the MPEG encoded data is advanced.

  As described above, the real-time streaming reception / playback apparatus performs streaming playback while replenishing packet loss that may occur accidentally during packet transmission.

=== Other implementation requirements ===
The delay time given to the second packet stream in the delay circuit 3 is set according to the packet loss occurrence pattern depending on the packet replenishment performance. For example, if the occurrence pattern is such that one packet is lost at a time, if the packet replenishment circuit 16 is notified of the packet loss detection, the corresponding packet is transferred from the buffer 14 immediately before that. Good. For this purpose, a time corresponding to at least one packet is set as the delay time.

  Alternatively, if the occurrence pattern is such that a plurality of packets are lost in a concentrated manner in a burst, when the packet supplement circuit 16 is notified of the detection of the loss of the leading packet, the buffer 14 continues with the corresponding packet. A packet corresponding to each of the generated lost packets needs to be transferred. That is, the first packet to the last packet having a burst defect must be transferred to the inside of the buffer 14. For that purpose, the tendency about the time width of the burst-like defect is investigated, and the time more than that time is set as the delay time.

  It is also possible to use the missing packet replenishment technique by duplicating the packet stream shown in the present embodiment together with a known lost packet recovery technique such as FEC. That is, a redundant code is added to the MPEG encoded data stream input to the packet assembling circuit 2 by the FEC technique and packetized. Then, the MPEG encoded data stream output from the sequence control circuit 13 is extracted as a packet stream to which a redundant code is added by the FEC technique. The detected lost packet is recovered by the FEC technique. This further improves the ability to recover information.

  Further, the storage retention period of each packet in the buffer 14 is at least the delay time given to the second packet stream, and at least the payload of the corresponding packet of the second packet stream is retained until it is transferred to the MPEG decoder 15. Set to. Each packet is sequentially deleted from the buffer 14 after the storage holding period has elapsed.

  In this embodiment, the packet stream is duplexed, but it may be tripled or multiplexed. Each packet stream is delayed by a predetermined time, and MPEG decoding is performed based on the most delayed packet stream. In this way, the degree of dispersion of the multiplexed packets is increased, and it is possible to more reliably replenish packet loss.

  In the above embodiment, MPEG2 is adopted as the high compression encoding method, but it goes without saying that other methods such as MP3 (Mpeg1 layer 3) may be used.

=== Application Example of the Present Invention ===
For example, when using Internet radio sound for BGM between songs in a karaoke device, the present invention can be applied with the Internet radio station server as a real-time streaming transmission device and the karaoke device as a real-time streaming reception device.

In this application example, the delay time given to the second packet stream in the delay circuit 3 is 3 seconds when the packet rate is 20 packets per second, that is, 60 packets of the first packet stream are buffered. The effect is proved by sequentially storing in FIG.
This is because the effect of the present invention becomes remarkable due to the fact that the burst packet loss described above is likely to occur on the Internet.

  As shown in FIG. 3, the change in the amount of packets going and going on the Internet has a wave shape with periodicity. This is a phenomenon caused by the transmission protocol used on the Internet, and is brought about by a synergistic effect caused by each of a large number of servers trying to transmit a series of packets continuously. In some cases, the wavy peak may exceed the transmission allowance, and a management device or the like intervening in the network has a secure mechanism for discarding in units of packets at that time. When the Internet radio packet stream reaches such a wave-like peak, packet loss occurs in a concentrated manner. In other words, the majority of packet loss in Internet radio occurs in a pattern that periodically concentrates in bursts.

  Also in this case, packet loss is concentrated within 3 seconds, and there is a time zone in which packet loss does not concentrate immediately before that for 3 seconds or more. There is enough room to store one packet stream. As a result, it is well understood that the delay time given to the second packet stream in the delay circuit 3 is 3 seconds.

  In such a packet loss occurrence pattern, the well-known FEC technique does not exhibit the lost packet recovery performance. The remarkable effect is demonstrated by the combined use with the lost packet replenishment technique of the present invention.

It is the figure which showed the process outline | summary in the real-time streaming transmission apparatus and real-time streaming receiving / reproducing apparatus of a present Example. It is the figure which illustrated the processing image of the delay and multiplexing of two packet streams in the real-time streaming transmission apparatus of a present Example. It is the figure which showed the image of the time change of the packet amount which passes on the internet.

Explanation of symbols

2 packet assembly circuit 3 delay circuit 4 multiplexing circuit 12 separation circuit 13 sequence control circuit 14 buffer 16 packet supplement circuit

Claims (5)

On the transmission side, the time-series information encoded with high efficiency is packetized to generate a first packet stream, and a second packet stream is generated by delaying each packet of the first packet stream for a predetermined time, and the first packet The stream and the second packet stream are multiplexed and transmitted,
On the receiving side, the multiplexed signal of the first packet stream and the second packet stream is received and separated into the first packet stream and the second packet stream, and a predetermined amount of the first packet stream is sequentially stored in the buffer, When the second packet stream is subjected to sequence control to extract the payload, the extracted data is sequentially decoded to reproduce and output time-series information, and packet loss is detected in the second packet stream sequence control processing In addition, the real-time streaming method of acquiring packets corresponding to missing packets from the packet group stored in the buffer and supplementing the sequence control processing.   The high-efficiency encoded time-series information is packetized to generate a first packet stream, and a second packet stream is generated by delaying each packet of the first packet stream for a predetermined time. A real-time streaming transmission method that multiplexes and transmits packet streams.   Receiving multiplexed signals of the first packet stream and the second packet stream having the same contents and a time difference, separating them into the first packet stream and the second packet stream, and sequentially storing a predetermined amount of the first packet stream in the buffer At the same time, the second packet stream is subjected to the sequence control to extract the payload, the extracted data is sequentially decoded to reproduce and output the time series information, and the packet loss is detected in the second packet stream sequence control processing. A real-time streaming reception / reproduction method of acquiring a packet corresponding to a missing packet from the packet group stored in the buffer and supplementing the sequence control process.   Means for packetizing time-series information encoded with high efficiency and generating a first packet stream; means for generating a second packet stream obtained by delaying each packet of the first packet stream for a predetermined time; and first packet stream And a means for multiplexing and transmitting the second packet stream.   Means for receiving multiplexed signals of the first packet stream and the second packet stream having the same contents and time difference, and separating the signals into the first packet stream and the second packet stream; and sequentially storing a predetermined amount of the first packet stream in the buffer Means for extracting the payload by performing the sequence control of the second packet stream, sequentially decoding the extracted data to reproduce and output the time series information, and detecting packet loss in the sequence control process of the second packet stream And a means for acquiring a packet corresponding to a missing packet from the packet group stored in the buffer and replenishing the sequence control process.

Images