JP4771369B2 - Media stream switching method, system and program in seamless handover - Google Patents

Description

  The present invention relates to a media stream switching method, system, and program in seamless handover.

  Conventionally, there is a handover technique in which communication is performed by adaptively switching between a cellular communication network and a high-speed wireless access network such as a wired / wireless local area network (LAN) and a wireless MAN (Metropolitan Area Network). A terminal corresponding to the technology needs to include a plurality of media communication units according to the communication media (access network). Among such techniques, there is a seamless handover technique that can receive a stream such as a real-time moving image or audio information seamlessly (seamlessly) when switching communication media. Usually, different communication media often have different delay times, and there are many cases where a momentary interruption occurs when switching communication media.

  On the other hand, there is a technology that includes a handover server that is connected to a plurality of different access networks for VoIP (see, for example, Non-Patent Document 1). The handover server copies a stream to be transmitted to the destination terminal, and simultaneously transmits a stream having the same content to the access network before the handover and the access network after the handover. As a result, the destination terminal can receive the same content stream from the access network before and after the handover, and tries to avoid the instantaneous interruption at the time of switching the communication media as much as possible.

Independent Administrative Institution New Energy and Industrial Technology Development Organization, Japan Electronics and Information Technology Industries Association Mobile Home System Council, “Implementation Experiments for Utilizing Advanced Mobile Technology in Real Service in Minato Mirai,” [ online], press release on November 21, 2005, [search on February 1, 2006], Internet <URL: http://mhsf.jeita.or.jp/jikkenn/pressrelease/20051117.pdf>

  Since the handover server described in Non-Patent Document 1 simply copies the stream, the header information of the RTP (Realtime Transport Protocol) packet constituting the stream has the same value. A relative time from the start of communication is set in the time stamp field of the RTP packet.

  When communication media are different due to handover, there is generally a delay difference between the communication media. Therefore, the destination terminal receives a plurality of streams having the same content with a delay difference time.

  In addition, when a stream with a high codec rate is transferred as in a videophone service, a communication medium with a narrow band cannot be used. In this case, the handover server needs to transcode the stream and convert the codec rate to correspond to each communication medium.

  The problem here is that the value of the timestamp field of the RTP packet is reset by transcoding by the handover server. In this case, the destination terminal cannot completely synchronize the two received media streams, and it is extremely difficult to realize seamless handover.

  Accordingly, an object of the present invention is to provide a media stream switching method, system, and program capable of realizing seamless handover between communication media having different delay times.

According to the present invention, a system having a destination terminal having two different media communication units, and a transmission device for transmitting the same stream constituted by RTP (Realtime Transport Protocol) packets to the destination terminal via different communication media A seamless handover media stream switching method in
The transmission device copies the stream to be transmitted for each communication medium , converts each media stream to a different codec rate corresponding to the media communication unit of the destination terminal, and stores it in the time stamp field of the RTP packet of the converted media stream. A first step of setting the posting time information for synchronization in the moving image compression method, and simultaneously transmitting these media streams to the destination terminal;
For the two received media streams, the destination terminal reduces the playback speed of the media stream with a small delay or increases the playback speed of the media stream with a large delay so that the time stamps coincide with each other at the playback timing. A second step of:
The destination terminal has a third step of switching the media stream to be reproduced when the time stamps of the two media streams coincide with each other at the reproduction timing.

According to the present invention, a seamless handover media stream comprising: a destination terminal having two different media communication means; and a transmission device that transmits the same stream configured by RTP packets to the destination terminal via different communication media. A switching system,
The transmitter is
A stream copy means for copying a stream to be transmitted for each communication medium;
Transcoding means for converting each media stream copied by the stream copy means to a different codec rate corresponding to the media communication unit of the destination terminal;
Time stamp setting means for setting posting time information for synchronization in the moving image compression method in the time stamp field of the RTP packet of the transcoded media stream;
Transmission means for simultaneously transmitting these media streams to the destination terminal,
The destination terminal is
For the two received media streams, the playback speed adjustment is made so that the playback speed of the media stream with a small delay is slowed down or the playback speed of the media stream with a large delay is fast so that the time stamps coincide with each other at the playback timing. Means,
Media stream switching means for switching media streams to be reproduced when the time stamps of the two media streams coincide with each other at the reproduction timing.

  According to the present invention, a transmission apparatus for transmitting the same stream constituted by RTP packets to a terminal having two different media communication means via different communication media, and switching media streams by seamless handover. There,
  A stream copy means for copying a stream to be transmitted for each communication medium;
  Transcoding means for converting each media stream copied by the stream copying means to a different codec rate corresponding to the media communication unit of the terminal;
  Time stamp setting means for setting posting time information for synchronization in the moving image compression method in the time stamp field of the RTP packet of the transcoded media stream;
  A transmission means for simultaneously transmitting these media streams to the terminal;
It is characterized by having.

According to the present invention, two different media communication means are used, and the same stream composed of RTP packets is received from the transmission device according to claim 3 via different communication media, and seamlessly received. A terminal that switches media streams by handover,
For the two received media streams, the playback speed adjustment is made so that the playback speed of the media stream with a small delay is slowed down or the playback speed of the media stream with a large delay is fast so that the time stamps coincide with each other at the playback timing. Means,
Media stream switching means for switching a media stream to be reproduced when the time stamps of the two media streams coincide with each other at the reproduction timing;
It is characterized by having.

According to the present invention, a program for causing a computer mounted on a transmission apparatus to function, which transmits the same stream composed of RTP packets to different destination media having two different media communication units via different communication media. In a program for a transmission device that causes a computer to function as follows:
A stream copy means for copying a stream to be transmitted for each communication medium;
Transcoding means for converting each media stream copied by the stream copy means to a different codec rate corresponding to the media communication unit of the destination terminal;
Time stamp setting means for setting posting time information for synchronization in the moving image compression method in the time stamp field of the RTP packet of the transcoded media stream;
Transmitting means for simultaneously transmitting these media streams to the destination terminal;
And making the computer function .

According to the present invention, there is provided a program for causing a computer mounted on a terminal to function, which functions as two different media communication means and transmits a plurality of media streams to different communication media from a transmission device that allows the above-described program to function by the computer. A program for a terminal that simultaneously receives via
For the two received media streams, the playback speed adjustment is made so that the playback speed of the media stream with a small delay is slowed down or the playback speed of the media stream with a large delay is fast so that the time stamps coincide with each other at the playback timing. Means,
When the time stamps of two media streams coincide with each other at the reproduction timing, the computer is caused to function as a media stream switching unit that switches a media stream to be reproduced.

  According to the media stream switching method and the like of the present invention, seamless handover can be realized between communication media having different delay times. In particular, since the playback speed of both media streams having a delay difference can be adjusted only by the time stamp included in the media stream, time synchronization can be achieved before decoding, and the media stream can be processed with a light processing load. Switching without interruption is possible. As a result, real-time application handover without temporal sound jump or video jump can be realized.

  Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a system configuration diagram according to the present invention.

  According to FIG. 1, the transmission source terminal 3 transmits a stream to the destination terminal 1. The stream is continuous data such as a real-time moving image and audio information such as a video conference service. The destination terminal 1 has moved from the first area to the second area, and has an access point 4 for a wireless LAN (first access network) and a base station 5 for a cellular communication network (second access network). Assume that you are in an area where you can communicate at the same time.

  According to the system of FIG. 1, the stream transmitted from the transmission source terminal 3 is received by the bicast server 2. The stream is composed of RTP packets.

  The bicast server 2 includes a stream copy unit 201, a first transcoding unit 211, a first time stamp setting unit 212, a first media stream transmission unit 213, a second transcoding unit 221, A second time stamp setting unit 222 and a second media stream transmission unit 223 are included. These functional units can be realized by a program that executes a computer mounted on a server.

  The stream copy unit 201 copies a stream to be transmitted to the destination terminal 1 for each communication medium. According to FIG. 1, a first media stream via a first access network and a second media stream via a second access network are generated.

  The first transcoding unit 211 converts the first media stream into a codec rate corresponding to the first access network. For example, a high-speed communication medium such as a wireless LAN is converted to a high coding rate. Further, the second transcoding unit 221 converts the second media stream into a codec rate corresponding to the second access network. For example, a low-speed communication medium such as a cellular communication network is converted to a low coding rate. Here, there is a delay difference between the first access network and the second access network.

  Here, the first transcoding unit 211 or the second transcoding unit 221 does not always perform transcoding. When the codec rate of the stream received from the transmission source terminal 3 can be transmitted as it is to the first access network or the second access network, transcoding is not necessary. In this case, the stream copied by the stream copy unit 201 is directly notified to the first time stamp setting unit 212 or the second time stamp setting unit 222 as a media stream.

  The first time stamp setting unit 212 or the second time stamp setting unit 222 displays the posting time information for synchronization in the moving image compression method in the time stamp field of the RTP packet in the first media stream or the second media stream. Set. Table 1 below shows the time stamp field of the RTP header.

  In general, the RTP timestamp represents the moment when the first octet of the stream is sampled and is used to schedule the playback of the stream. The time stamp is represented by a 32-bit unsigned integer, and its value increases at a different rate depending on the clock rate of the codec.

  In contrast, the present invention sets, for example, PTS (Presentation Time Stamp) information in MPEG4 as presentation time information in the time stamp field of the RTP packet. In this way, the synchronization time information in the real-time application is mapped to the RTP header that is the lower layer. Thereby, the destination terminal 1 can measure the delay only by the time stamps of the RTP packets in the two media streams.

  The first media stream transmission unit 213 transmits the first media stream to the destination terminal 1 via the first access network. In addition, the second media stream transmission unit 223 transmits the second media stream to the destination terminal 1 via the second access network.

  The destination terminal 1 receives the first media stream from the access point 4 of the first access network (wireless LAN) and simultaneously receives the second media stream from the base station 5 of the second access network (cellular communication network). Receive.

  The destination terminal 1 illustrated in FIG. 1 includes a first media communication unit 10, a second media communication unit 11, a playback speed adjustment unit 121, a media stream switching unit 122, a decoding unit 123, and a speaker unit. 124 and a display unit 125. The destination terminal 1 includes at least two media communication units. These functional units can be realized by a program executed by a computer mounted on the terminal 1.

  The first media communication unit 10 includes a first communication interface unit 101, a first buffer unit 102, and a first buffer extraction unit 103. The first communication interface unit 101 receives the first media stream from the access point 4 of the wireless LAN, and accumulates the stream in the first buffer unit 102. The first buffer extraction unit 103 extracts the first media stream from the first buffer unit 102 and outputs the first media stream.

  The second media communication unit 11 includes a second communication interface unit 111, a second buffer unit 112, and a second buffer extraction unit 113. The second communication interface unit 111 receives the second media stream having the same content from the base station 5 of the cellular communication network, and accumulates the stream in the second buffer unit 112. The second buffer extraction unit 113 extracts the second media stream from the second buffer unit 112 and outputs it.

  The reproduction speed adjustment unit 121 adjusts the reproduction speed (buffer extraction speed) between the first buffer extraction unit 103 and the second buffer extraction unit 113. When adjusting the media stream with a small delay, the playback speed adjustment unit 121 reduces the playback speed. For example, when the delay of the media stream before the handover is small and the delay of the media stream after the handover is large, the destination terminal 1 decreases the playback speed of the media stream before the handover. On the other hand, when the delay of the media stream before the handover is large and the delay of the media stream after the handover is small, the destination terminal 1 reduces the playback speed of the media stream after the handover.

  The media stream switching unit 122 compares the time stamp of the first media stream output from the first media communication unit 10 with the time stamp of the second media stream output from the second media communication unit 11. To do. When the two time stamps coincide with each other at the reproduction timing, the media stream after the handover is switched. The media stream is notified to the decoding unit 123.

  The decoding unit 123 decodes the media stream according to the codec rate. The decoded audio information is notified to the speaker unit 124, and the moving image information is notified to the display unit 125.

  FIG. 2 is an explanatory diagram of the first delay time absorption in the present invention. The first delay time absorption relates to switching from a media stream having a small delay to a media stream having a large delay.

  In FIG. 2, the first media stream stored in the first buffer unit 102, the second media stream stored in the second buffer unit 112, and the media reproduced by the decoding unit 123 A stream is represented. Also, according to FIG. 2, it is assumed that the first media stream has a small delay and is currently being reproduced (decoded), and the second media stream has a large delay and is not currently being reproduced. . Here, the delay time difference is “2t” (t: unit time). Although the second media stream is accumulated in the second buffer unit 112, when the reproduction timing is reached, the second media stream is discarded without being reproduced.

(S201) The first media stream at time t is played, and the second media stream at time t-2 (delay difference 2t) is discarded. This is because the first media stream at time t-2 has already been reproduced. According to the present invention, here, the playback speed of the first media stream before the handover is halved.

(S202) Shows the states of the first and second buffers that have elapsed for time t from S201. Since the playback speed of the first media stream has been halved, the first stream at time t is still being played. On the other hand, as for the second media stream, the second media stream at time t-1 has entered the playback timing. However, since the time stamp of the second media stream does not match the time stamp of the first media stream at the reproduction timing, the second media stream at time t−1 (delay difference t) is Discarded.

(S203) Shows the state of the first and second buffers after time t has elapsed since S202. The reproduction of the first media stream at time t is finished, and the first stream at time t + 1 is reproduced. On the other hand, as for the second media stream, the second media stream at time t has entered the reproduction timing. However, since the time stamp of the second media stream does not match the time stamp of the first media stream at the reproduction timing, the second media stream at time t (delay difference t) is discarded. The

(S204) This represents the state of the first and second buffers after the time t has elapsed since S203. Since the playback speed of the first media stream is ½, the first stream at time t + 1 is still being played. On the other hand, as for the second media stream, the second media stream at time t + 1 has entered the reproduction timing. Here, the time stamp of the first media stream before the handover matches the time stamp of the second media stream after the handover at the playback timing.

  Here, the media stream to be reproduced is switched from the first media stream to the second media stream. Also, the reception of the first media stream is terminated. Thereby, seamless handover from the first media stream to the second media stream can be realized.

  FIG. 3 is an explanatory diagram of second delay time absorption in the present invention. The second delay time absorption relates to switching from a media stream having a large delay to a media stream having a small delay.

  In FIG. 3, it is assumed that the first media stream has a large delay and is currently being reproduced (decoded), and the second media stream has a small delay and is not currently being reproduced. Here, the delay time difference is “2t” (t: unit time). Although the second media stream is stored in the second buffer 112, when the playback timing is reached, the second media stream is discarded without being played back.

(S301) The first media stream at time t is played, and the second media stream at time t + 2 (delay difference 2t) is discarded. This is because the first media stream at time t + 2 is played back subsequently. According to the present invention, here, the playback speed of the second media stream after the handover is halved.

(S302) Shows the state of the first and second buffers after the time t from S301. As the first media stream, the first media stream at time t + 1 is reproduced. On the other hand, since the playback speed of the second media stream is halved, the first stream at time t + 3 has not yet entered the playback timing. However, the time stamp of the second media stream still does not match the time stamp of the first media stream at the playback timing.

(S303) Shows the state of the first and second buffers after the time t has passed since S302. As the first media stream, the first media stream at time t + 2 is reproduced. On the other hand, the second stream at the next time t + 3 has entered the reproduction timing. However, since the time stamp of the second media stream does not match the time stamp of the first media stream at the reproduction timing, the second media stream at time t + 3 (delay difference t) is discarded. The

(S304) Shows the state of the first and second buffers after the time t has elapsed since S303. Since the playback speed of the second media stream is ½, the second media stream at the time t + 3 has entered the playback timing. Here, the time stamp of the first media stream before the handover matches the time stamp of the second media stream after the handover at the playback timing.

  Here, the media stream to be reproduced is switched from the first media stream to the second media stream. Also, the reception of the first media stream is terminated. Thereby, seamless handover from the first media stream to the second media stream can be realized.

  2 and 3 described above, the playback speed of a media stream with a small delay is reduced. As another embodiment, the playback speed of a media stream having a large delay may be increased.

Further, according to the above-described embodiments of FIGS. 2 and 3, the playback speed is halved in order to adjust the delay between the two media streams. As another embodiment, for example, it is also preferable to adjust the reproduction speed as follows depending on the time taken for delayed absorption (delayed absorption time).
When reducing the playback speed: Playback speed = (delay absorption time-delay time) / delay absorption time When increasing the playback speed: playback speed = (delay absorption time + delay time) / delay absorption time

  FIG. 4 is a system configuration diagram in which a transmission source terminal transmits a plurality of media streams.

  According to FIG. 4, the transmission source terminal 3 has the function of the bicast server 2 in FIG. In this case, the transmission source terminal 3 directly transmits the first media stream and the second media stream.

  As another embodiment of the present invention, it is also preferable to switch the media stream before decoding the video intra refresh frame. This can be determined from the marker of the RTP header. This marker includes, for example, a flag indicating the video intra refresh frame by extending the header of the RTP packet. This marker is the first time stamp setting unit 212 and the second time stamp setting unit 222 of the bicast server 2, or the first time stamp setting unit 312 and the second time stamp setting unit of the transmission source terminal 3. 322.

  In addition, as another embodiment of the present invention, the judgment for switching the media stream is not only when the time stamps completely match at the reproduction timing but also when the delay difference becomes a predetermined threshold value or less. May be. This is because, since transcoding is performed for each communication medium, the time stamp may not completely match the reproduction timing.

  According to the media stream switching method and the like of the present invention, seamless handover can be realized between communication media having different delay times. In particular, since the playback speed of both media streams having a delay difference can be adjusted only by the time stamp included in the media stream, time synchronization can be achieved before decoding, and the media stream can be processed with a light processing load. Switching without interruption is possible. As a result, real-time application handover without temporal sound jump or video jump can be realized.

  According to the various embodiments of the present invention described above, those skilled in the art can easily make various changes, modifications and omissions within the scope of the technical idea and the viewpoint of the present invention. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

It is a system configuration diagram in the present invention. It is explanatory drawing of the 1st delay absorption in this invention. It is explanatory drawing of the 2nd delay absorption in this invention. It is a system block diagram in which a transmission source terminal transmits a plurality of media streams.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Destination terminal 101 1st communication interface part 102 1st buffer part 103 1st buffer extraction part 111 2nd communication interface part 112 2nd buffer part 113 2nd buffer extraction part 121 Playback speed adjustment part 122 Media Stream switching unit 123 Decoding unit 124 Speaker unit 125 Display unit 2 Bicast server 201 Stream copy unit 211 First transcoding unit 212 First time stamp setting unit 213 First media stream transmission unit 221 Second transcoding unit 222 Second time stamp setting unit 223 Second media stream transmission unit 3 Source terminal 4 Access point 5 Base station

Claims (6)

Media for seamless handover in a system having a destination terminal having two different media communication units and a transmission device that transmits the same stream formed by RTP (Realtime Transport Protocol) packets to the destination terminal via different communication media A stream switching method,
The transmission device copies a stream to be transmitted for each communication medium , converts each media stream to a different codec rate corresponding to the media communication unit of the destination terminal, and converts the RTP packet time of the converted media stream A first step of setting the posting time information for synchronization in the moving image compression method in the stamp field, and simultaneously transmitting these media streams to the destination terminal;
For the two media streams received by the destination terminal, the playback speed of the media stream with a small delay is reduced or the playback speed of the media stream with a large delay is set so that the time stamps coincide with each other at the playback timing. A second step to speed up ;
And a third step of switching the media stream to be played back when the destination terminal matches the time stamp of the two media streams at the playback timing. A seamless handover media stream switching system comprising: a destination terminal having two different media communication means; and a transmission device that transmits the same stream formed by RTP packets to the destination terminal via different communication media. ,
The transmitter is
A stream copy means for copying a stream to be transmitted for each communication medium;
Transcoding means for converting each media stream copied by the stream copy means to a different codec rate corresponding to the media communication unit of the destination terminal;
Time stamp setting means for setting posting time information for synchronization in the moving image compression method in the time stamp field of the RTP packet of the transcoded media stream;
Transmission means for simultaneously transmitting these media streams to the destination terminal,
The destination terminal is
For the two received media streams, the playback speed of the media stream with a small delay is slowed down or the playback speed of the media stream with a large delay is fasted so that the time stamps coincide with each other at the playback timing. Adjustment means;
And a media stream switching unit that switches a media stream to be reproduced when the time stamps of the two media streams coincide with each other at the reproduction timing.   A transmission device for transmitting the same stream constituted by RTP packets to different terminals via two different communication media to a terminal having two different media communication means, and for switching media streams by seamless handover,
  A stream copy means for copying a stream to be transmitted for each communication medium;
  Transcoding means for converting each media stream copied by the stream copy means to a different codec rate corresponding to the media communication unit of the terminal;
  Time stamp setting means for setting posting time information for synchronization in the moving image compression method in the time stamp field of the RTP packet of the transcoded media stream;
  Transmitting means for simultaneously transmitting these media streams to the terminal;
A transmission device comprising:   The same stream composed of RTP packets is received from the transmission device according to claim 3 via different communication media, and the media stream is switched by seamless handover. A terminal,
  For the two received media streams, the playback speed of the media stream with a small delay is slowed down or the playback speed of the media stream with a large delay is fasted so that the time stamps coincide with each other at the playback timing. Adjusting means;
  Media stream switching means for switching a media stream to be reproduced when the time stamps of the two media streams coincide with each other at the reproduction timing;
A terminal characterized by comprising: A program to function a computer mounted in the transmission device, the two different media communication unit to end late that Yusuke, to transmit the same stream constituted by RTP packets via different communication media Computer In the program for the transmitting device that causes
A stream copy means for copying a stream to be transmitted for each communication medium;
Transcoding means for converting each media stream copied by the stream copy means to a different codec rate corresponding to the media communication unit of the destination terminal;
Time stamp setting means for setting posting time information for synchronization in the moving image compression method in the time stamp field of the RTP packet of the transcoded media stream;
Transmission means for transmitting these media streams simultaneously Previous Stories end end
And a program for the transmission device, which causes the computer to function . A program for causing a computer mounted on a terminal to function, the same media stream configured by RTP packets from a transmission device that functions as two different media communication means and that allows the program according to claim 5 to function by the computer Is a program for a terminal that simultaneously receives media via different communication media and switches media streams by seamless handover,
For the two received media streams, the playback speed of the media stream with a small delay is slowed down or the playback speed of the media stream with a large delay is fasted so that the time stamps coincide with each other at the playback timing. Adjustment means;
A program for a terminal, which causes a computer to function as media stream switching means for switching a media stream to be reproduced when the time stamps of the two media streams coincide with each other at the reproduction timing.

Images