JP2007214985A - Media stream switching method, system and program in seamless handover - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a media stream switching method, etc. by which seamless handover is attained between communication media having different delay periods of time. <P>SOLUTION: A media stream switching system in seamless handover has: a destination terminal having a plurality of different media communication parts; and a transmitter which transmits the same stream to the destination terminal via different communication media. The transmitter copies the stream to be transmitted for each medium, sets time stamps for each frame of the copied media streams and simultaneously transmits the media streams to the destination terminal. The destination terminal adjusts reproduction speed of the media streams so that the time stamps matches in reproduction timing for both of the received media streams. Then, the destination terminal switches the media streams to be reproduced when the time stamps of both media streams matches in the reproduction timing. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a media stream switching method, system, and program in seamless handover.

  Conventionally, there is a handover technique in which communication is performed by adaptively switching between a cellular communication network and a high-speed wireless access network such as a wired / wireless local area network (LAN) and a wireless MAN (Metropolitan Area Network). A terminal corresponding to the technology needs to include a plurality of media communication units according to the communication media (access network). Among such techniques, there is a seamless handover technique that can receive a stream such as a real-time moving image or audio information seamlessly (seamlessly) when switching communication media. Usually, different communication media often have different delay times, and there are many cases where a momentary interruption occurs when switching communication media.

  On the other hand, there is a technology that includes a handover server that is connected to a plurality of different access networks for VoIP (see, for example, Non-Patent Document 1). The handover server copies a stream to be transmitted to the destination terminal, and simultaneously transmits a stream having the same content to the access network before the handover and the access network after the handover. As a result, the destination terminal can receive the same content stream from the access network before and after the handover, and tries to avoid the instantaneous interruption at the time of switching the communication media as much as possible.

Independent Administrative Institution New Energy and Industrial Technology Development Organization, Japan Electronics and Information Technology Industries Association Mobile Home System Council, “Implementation Experiments for Utilizing Advanced Mobile Technology in Real Service in Minato Mirai,” [ online], press release on November 21, 2005, [search on February 1, 2006], Internet <URL: http://mhsf.jeita.or.jp/jikkenn/pressrelease/20051117.pdf>

  Since the handover server described in Non-Patent Document 1 simply copies the stream, the header information of the RTP (Realtime Transport Protocol) packet constituting the stream has the same value. A relative time from the start of communication is set in the time stamp field of the RTP packet.

  When communication media are different due to handover, there is generally a delay difference between the communication media. Therefore, the destination terminal receives a plurality of streams having the same content with a delay difference time.

  In addition, when a stream with a high codec rate is transferred as in a videophone service, a communication medium with a narrow band cannot be used. In this case, the handover server needs to transcode the stream and convert the codec rate to correspond to each communication medium.

  The problem here is that the value of the timestamp field of the RTP packet is reset by transcoding by the handover server. In this case, the destination terminal cannot completely synchronize the two received media streams, and it is extremely difficult to realize seamless handover.

  Accordingly, an object of the present invention is to provide a media stream switching method, system, and program capable of realizing seamless handover between communication media having different delay times.

According to the present invention, there is provided a media stream switching method for seamless handover in a system having a destination terminal having a plurality of different media communication units and a transmission device that transmits the same stream to the destination terminal via different communication media. ,
A first step in which a transmitting device copies a stream to be transmitted for each communication medium, sets a time stamp for each frame of the copied media stream, and simultaneously transmits the media stream to a destination terminal;
A second step in which the destination terminal adjusts the playback speed of the media stream so that the time stamps match at the playback timing for both received media streams;
The destination terminal has a third step of switching the media stream to be reproduced when the time stamps of both media streams coincide with each other at the reproduction timing.

  According to another embodiment of the media stream switching method of the present invention, the media stream is composed of RTP packets, and the posting time information for synchronization in the moving image compression method is set in the time stamp field of the RTP packets. It is also preferable to do.

  According to another embodiment of the media stream switching method of the present invention, for the first step, the transmitting device copies the stream to be transmitted for each communication medium, and transmits each media stream to the media communication of the destination terminal. It is also preferable to convert to a different codec rate corresponding to each part, set a time stamp for each frame of each media stream, and simultaneously transmit these media streams to the destination terminal.

  Furthermore, according to another embodiment of the media stream switching method of the present invention, in the second step, the playback speed of a media stream with a small delay is reduced or the playback speed of a media stream with a large delay is increased. It is also preferable to make it.

According to the present invention, there is provided a seamless handover media stream switching system including a destination terminal having a plurality of different media communication means and a transmission device that transmits the same stream to the destination terminal via different communication media,
The transmitter is
A stream copy means for copying a stream to be transmitted for each communication medium;
A time stamp setting means for setting a time stamp for each frame of the copied media stream;
Transmission means for simultaneously transmitting these media streams to the destination terminal,
The destination terminal is
For both received media streams, playback speed adjustment means for adjusting the playback speed of the media stream so that the time stamps match at the playback timing;
Media stream switching means for switching a media stream to be reproduced when the time stamps of both media streams coincide with each other at the reproduction timing.

According to another embodiment of the transmission device of the present invention, the media stream is composed of RTP packets,
It is also preferable that the time stamp setting means of the transmitting device sets the posting time information for synchronization in the moving image compression method in the time stamp field of the RTP packet.

According to another embodiment of the transmission device of the present invention,
The transmission apparatus further includes transcoding means for converting each media stream copied by the stream copy means to a different codec rate corresponding to the media communication unit of the destination terminal,
It is also preferable that the time stamp setting means of the transmitting device sets a time stamp for each frame of the media stream in which the codec rate is converted.

  Furthermore, according to another embodiment of the transmission apparatus of the present invention, the playback speed adjustment means of the destination terminal reduces the playback speed of the media stream with a small delay or sets the playback speed of the media stream with a large delay. High speed is also preferable.

According to the present invention, a terminal having a plurality of different media communication means and simultaneously receiving a plurality of media streams via different communication media from the transmission device according to any one of claims 5 to 8. Because
For both received media streams, playback speed adjustment means for adjusting the playback speed of the media stream so that the time stamps match at the playback timing;
Media stream switching means for switching a media stream to be reproduced when the time stamps of both media streams coincide with each other at the reproduction timing.

According to the present invention, there is provided a program for causing a computer installed in a transmission apparatus to function, and the computer functions to transmit the same stream to different destination terminals having different media communication units via different communication media. In the program for the transmitting device to be
A stream copy means for copying a stream to be transmitted for each communication medium;
A time stamp setting means for setting a time stamp for each frame of the copied media stream;
Transmitting means for transmitting these media streams to the destination terminal at the same time.

According to the present invention, there is provided a program for causing a computer mounted on a terminal to function as a plurality of different media communication means, and a plurality of media streams from a transmission device that allows the program according to claim 10 to function by the computer. Is a program for a terminal that simultaneously receives data via different communication media,
For both received media streams, playback speed adjustment means for adjusting the playback speed of the media stream so that the time stamps match at the playback timing;
When the time stamps of both media streams coincide with each other at the reproduction timing, the computer is caused to function as media stream switching means for switching the media stream to be reproduced.

  According to the media stream switching method and the like of the present invention, seamless handover can be realized between communication media having different delay times. In particular, since the playback speed of both media streams having a delay difference can be adjusted only by the time stamp included in the media stream, time synchronization can be achieved before decoding, and the media stream can be processed with a light processing load. Switching without interruption is possible. As a result, real-time application handover without temporal sound jump or video jump can be realized.

  Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a system configuration diagram according to the present invention.

  According to FIG. 1, the transmission source terminal 3 transmits a stream to the destination terminal 1. The stream is continuous data such as a real-time moving image and audio information such as a video conference service. The destination terminal 1 has moved from the first area to the second area, and has an access point 4 for a wireless LAN (first access network) and a base station 5 for a cellular communication network (second access network). Assume that you are in an area where you can communicate at the same time.

  According to the system of FIG. 1, the stream transmitted from the transmission source terminal 3 is received by the bicast server 2. The stream is composed of RTP packets.

  The bicast server 2 includes a stream copy unit 201, a first transcoding unit 211, a first time stamp setting unit 212, a first media stream transmission unit 213, a second transcoding unit 221, A second time stamp setting unit 222 and a second media stream transmission unit 223 are included. These functional units can be realized by a program that executes a computer mounted on a server.

  The stream copy unit 201 copies a stream to be transmitted to the destination terminal 1 for each communication medium. According to FIG. 1, a first media stream via a first access network and a second media stream via a second access network are generated.

  The first transcoding unit 211 converts the first media stream into a codec rate corresponding to the first access network. For example, a high-speed communication medium such as a wireless LAN is converted to a high coding rate. Further, the second transcoding unit 221 converts the second media stream into a codec rate corresponding to the second access network. For example, a low-speed communication medium such as a cellular communication network is converted to a low coding rate. Here, there is a delay difference between the first access network and the second access network.

  Here, the first transcoding unit 211 or the second transcoding unit 221 does not always perform transcoding. When the codec rate of the stream received from the transmission source terminal 3 can be transmitted as it is to the first access network or the second access network, transcoding is not necessary. In this case, the stream copied by the stream copy unit 201 is directly notified to the first time stamp setting unit 212 or the second time stamp setting unit 222 as a media stream.

  The first time stamp setting unit 212 or the second time stamp setting unit 222 displays the posting time information for synchronization in the moving image compression method in the time stamp field of the RTP packet in the first media stream or the second media stream. Set. Table 1 below shows the time stamp field of the RTP header.

  In general, the RTP timestamp represents the moment when the first octet of the stream is sampled and is used to schedule the playback of the stream. The time stamp is represented by a 32-bit unsigned integer, and its value increases at a different rate depending on the clock rate of the codec.

  In contrast, the present invention sets, for example, PTS (Presentation Time Stamp) information in MPEG4 as presentation time information in the time stamp field of the RTP packet. In this way, the synchronization time information in the real-time application is mapped to the RTP header that is the lower layer. Thereby, the destination terminal 1 can measure the delay only by the time stamps of the RTP packets in the two media streams.

  The first media stream transmission unit 213 transmits the first media stream to the destination terminal 1 via the first access network. In addition, the second media stream transmission unit 223 transmits the second media stream to the destination terminal 1 via the second access network.

  The destination terminal 1 receives the first media stream from the access point 4 of the first access network (wireless LAN) and simultaneously receives the second media stream from the base station 5 of the second access network (cellular communication network). Receive.

  The destination terminal 1 illustrated in FIG. 1 includes a first media communication unit 10, a second media communication unit 11, a playback speed adjustment unit 121, a media stream switching unit 122, a decoding unit 123, and a speaker unit. 124 and a display unit 125. The destination terminal 1 includes at least two media communication units. These functional units can be realized by a program executed by a computer mounted on the terminal 1.

  The first media communication unit 10 includes a first communication interface unit 101, a first buffer unit 102, and a first buffer extraction unit 103. The first communication interface unit 101 receives the first media stream from the access point 4 of the wireless LAN, and accumulates the stream in the first buffer unit 102. The first buffer extraction unit 103 extracts the first media stream from the first buffer unit 102 and outputs the first media stream.

  The second media communication unit 11 includes a second communication interface unit 111, a second buffer unit 112, and a second buffer extraction unit 113. The second communication interface unit 111 receives the second media stream having the same content from the base station 5 of the cellular communication network, and accumulates the stream in the second buffer unit 112. The second buffer extraction unit 113 extracts the second media stream from the second buffer unit 112 and outputs it.

  The reproduction speed adjustment unit 121 adjusts the reproduction speed (buffer extraction speed) between the first buffer extraction unit 103 and the second buffer extraction unit 113. When adjusting the media stream with a small delay, the playback speed adjustment unit 121 reduces the playback speed. For example, when the delay of the media stream before the handover is small and the delay of the media stream after the handover is large, the destination terminal 1 decreases the playback speed of the media stream before the handover. On the other hand, when the delay of the media stream before the handover is large and the delay of the media stream after the handover is small, the destination terminal 1 reduces the playback speed of the media stream after the handover.

  The media stream switching unit 122 compares the time stamp of the first media stream output from the first media communication unit 10 with the time stamp of the second media stream output from the second media communication unit 11. To do. When the two time stamps coincide with each other at the reproduction timing, the media stream after the handover is switched. The media stream is notified to the decoding unit 123.

  The decoding unit 123 decodes the media stream according to the codec rate. The decoded audio information is notified to the speaker unit 124, and the moving image information is notified to the display unit 125.

  FIG. 2 is an explanatory diagram of the first delay time absorption in the present invention. The first delay time absorption relates to switching from a media stream having a small delay to a media stream having a large delay.

  In FIG. 2, the first media stream stored in the first buffer unit 102, the second media stream stored in the second buffer unit 112, and the media reproduced by the decoding unit 123 A stream is represented. Also, according to FIG. 2, it is assumed that the first media stream has a small delay and is currently being reproduced (decoded), and the second media stream has a large delay and is not currently being reproduced. . Here, the delay time difference is “2t” (t: unit time). Although the second media stream is accumulated in the second buffer unit 112, when the reproduction timing is reached, the second media stream is discarded without being reproduced.

(S201) The first media stream at time t is played, and the second media stream at time t-2 (delay difference 2t) is discarded. This is because the first media stream at time t-2 has already been reproduced. According to the present invention, here, the playback speed of the first media stream before the handover is halved.

(S202) Shows the states of the first and second buffers that have elapsed for time t from S201. Since the playback speed of the first media stream has been halved, the first stream at time t is still being played. On the other hand, as for the second media stream, the second media stream at time t-1 has entered the playback timing. However, since the time stamp of the second media stream does not match the time stamp of the first media stream at the reproduction timing, the second media stream at time t−1 (delay difference t) is Discarded.

(S203) Shows the state of the first and second buffers after time t has elapsed since S202. The reproduction of the first media stream at time t is finished, and the first stream at time t + 1 is reproduced. On the other hand, as for the second media stream, the second media stream at time t has entered the reproduction timing. However, since the time stamp of the second media stream does not match the time stamp of the first media stream at the reproduction timing, the second media stream at time t (delay difference t) is discarded. The

(S204) This represents the state of the first and second buffers after the time t has elapsed since S203. Since the playback speed of the first media stream is ½, the first stream at time t + 1 is still being played. On the other hand, as for the second media stream, the second media stream at time t + 1 has entered the reproduction timing. Here, the time stamp of the first media stream before the handover matches the time stamp of the second media stream after the handover at the playback timing.

  Here, the media stream to be reproduced is switched from the first media stream to the second media stream. Also, the reception of the first media stream is terminated. Thereby, seamless handover from the first media stream to the second media stream can be realized.

  FIG. 3 is an explanatory diagram of second delay time absorption in the present invention. The second delay time absorption relates to switching from a media stream having a large delay to a media stream having a small delay.

  In FIG. 3, it is assumed that the first media stream has a large delay and is currently being reproduced (decoded), and the second media stream has a small delay and is not currently being reproduced. Here, the delay time difference is “2t” (t: unit time). Although the second media stream is stored in the second buffer 112, when the playback timing is reached, the second media stream is discarded without being played back.

(S301) The first media stream at time t is played, and the second media stream at time t + 2 (delay difference 2t) is discarded. This is because the first media stream at time t + 2 is played back subsequently. According to the present invention, here, the playback speed of the second media stream after the handover is halved.

(S302) Shows the state of the first and second buffers after the time t from S301. As the first media stream, the first media stream at time t + 1 is reproduced. On the other hand, since the playback speed of the second media stream is halved, the first stream at time t + 3 has not yet entered the playback timing. However, the time stamp of the second media stream still does not match the time stamp of the first media stream at the playback timing.

(S303) Shows the state of the first and second buffers after the time t has passed since S302. As the first media stream, the first media stream at time t + 2 is reproduced. On the other hand, the second stream at the next time t + 3 has entered the reproduction timing. However, since the time stamp of the second media stream does not match the time stamp of the first media stream at the reproduction timing, the second media stream at time t + 3 (delay difference t) is discarded. The

(S304) Shows the state of the first and second buffers after the time t has elapsed since S303. Since the playback speed of the second media stream is ½, the second media stream at the time t + 3 has entered the playback timing. Here, the time stamp of the first media stream before the handover matches the time stamp of the second media stream after the handover at the playback timing.

  Here, the media stream to be reproduced is switched from the first media stream to the second media stream. Also, the reception of the first media stream is terminated. Thereby, seamless handover from the first media stream to the second media stream can be realized.

  2 and 3 described above, the playback speed of a media stream with a small delay is reduced. As another embodiment, the playback speed of a media stream having a large delay may be increased.

Further, according to the above-described embodiments of FIGS. 2 and 3, the playback speed is halved in order to adjust the delay between the two media streams. As another embodiment, for example, it is also preferable to adjust the reproduction speed as follows depending on the time taken for delayed absorption (delayed absorption time).
When reducing the playback speed: Playback speed = (delay absorption time-delay time) / delay absorption time When increasing the playback speed: playback speed = (delay absorption time + delay time) / delay absorption time

  FIG. 4 is a system configuration diagram in which a transmission source terminal transmits a plurality of media streams.

  According to FIG. 4, the transmission source terminal 3 has the function of the bicast server 2 in FIG. In this case, the transmission source terminal 3 directly transmits the first media stream and the second media stream.

  As another embodiment of the present invention, it is also preferable to switch the media stream before decoding the video intra refresh frame. This can be determined from the marker of the RTP header. This marker includes, for example, a flag indicating the video intra refresh frame by extending the header of the RTP packet. This marker is the first time stamp setting unit 212 and the second time stamp setting unit 222 of the bicast server 2, or the first time stamp setting unit 312 and the second time stamp setting unit of the transmission source terminal 3. 322.

  In addition, as another embodiment of the present invention, the judgment for switching the media stream is not only when the time stamps completely match at the reproduction timing but also when the delay difference becomes a predetermined threshold value or less. May be. This is because, since transcoding is performed for each communication medium, the time stamp may not completely match the reproduction timing.

  According to the media stream switching method and the like of the present invention, seamless handover can be realized between communication media having different delay times. In particular, since the playback speed of both media streams having a delay difference can be adjusted only by the time stamp included in the media stream, time synchronization can be achieved before decoding, and the media stream can be processed with a light processing load. Switching without interruption is possible. As a result, real-time application handover without temporal sound jump or video jump can be realized.

  According to the various embodiments of the present invention described above, those skilled in the art can easily make various changes, modifications and omissions within the scope of the technical idea and the viewpoint of the present invention. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

It is a system configuration diagram in the present invention. It is explanatory drawing of the 1st delay absorption in this invention. It is explanatory drawing of the 2nd delay absorption in this invention. It is a system block diagram in which a transmission source terminal transmits a plurality of media streams.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Destination terminal 101 1st communication interface part 102 1st buffer part 103 1st buffer extraction part 111 2nd communication interface part 112 2nd buffer part 113 2nd buffer extraction part 121 Playback speed adjustment part 122 Media Stream switching unit 123 Decoding unit 124 Speaker unit 125 Display unit 2 Bicast server 201 Stream copy unit 211 First transcoding unit 212 First time stamp setting unit 213 First media stream transmission unit 221 Second transcoding unit 222 Second time stamp setting unit 223 Second media stream transmission unit 3 Source terminal 4 Access point 5 Base station

Claims (10)

A seamless handover media stream switching method in a system having a destination terminal having a plurality of different media communication units and a transmission device that transmits the same stream to the destination terminal via different communication media,
A first step in which the transmitting device copies a stream to be transmitted for each communication medium, sets a time stamp for each frame of the copied media stream, and simultaneously transmits the media stream to the destination terminal;
A second step in which the destination terminal adjusts the playback speed of the media stream so that the time stamps match at the playback timing for both received media streams;
And a third step of switching the media stream to be played back when the destination terminal matches the time stamp of both media streams at the playback timing.   The media stream is composed of RTP (Realtime Transport Protocol) packets, and the posting time information for synchronization in the moving image compression method is set in the time stamp field of the RTP packets. The described media stream switching method.   For the first step, the transmission device copies a stream to be transmitted for each communication medium, converts each media stream to a different codec rate corresponding to the media communication unit of the destination terminal, and each media stream The media stream switching method according to claim 1 or 2, wherein a time stamp is set for each frame and the media streams are simultaneously transmitted to the destination terminal.   4. The method according to claim 1, wherein, in the second step, the playback speed of a media stream with a small delay is reduced, or the playback speed of a media stream with a large delay is increased. The media stream switching method described in 1. A seamless handover media stream switching system comprising: a destination terminal having a plurality of different media communication means; and a transmission device that transmits the same stream to the destination terminal via different communication media,
The transmitter is
A stream copy means for copying a stream to be transmitted for each communication medium;
A time stamp setting means for setting a time stamp for each frame of the copied media stream;
Transmission means for simultaneously transmitting these media streams to the destination terminal,
The destination terminal is
Reproduction speed adjusting means for adjusting the reproduction speed of the media stream so that the time stamps coincide with each other at the reproduction timing for both received media streams;
And a media stream switching unit that switches a media stream to be reproduced when the time stamps of both media streams coincide with each other at the reproduction timing. The media stream is composed of RTP (Realtime Transport Protocol) packets,
6. The system according to claim 5, wherein the time stamp setting means of the transmitting device sets posting time information for synchronization in a moving image compression method in a time stamp field of the RTP packet. The transmission device further includes transcoding means for converting each media stream copied by the stream copy means to a different codec rate corresponding to the media communication unit of the destination terminal,
The system according to claim 5 or 6, wherein the time stamp setting means of the transmitting device sets a time stamp for each frame of a media stream whose codec rate is converted.   8. The playback speed adjusting means of the destination terminal decreases the playback speed of a media stream with a small delay, or increases the playback speed of a media stream with a large delay. The system according to any one of the above. A program for causing a computer mounted on a transmission device to function, wherein the program causes the computer to function so as to transmit the same stream to different destination terminals having different media communication units via different communication media In
A stream copy means for copying a stream to be transmitted for each communication medium;
A time stamp setting means for setting a time stamp for each frame of the copied media stream;
A program for a transmission apparatus, comprising: a transmission unit that simultaneously transmits these media streams to the destination terminal. A program for causing a computer installed in a terminal to function as a plurality of different media communication means, and transmitting a plurality of media streams from a transmission device that causes the program according to claim 9 to function by the computer. A program for a terminal that simultaneously receives via
Reproduction speed adjusting means for adjusting the reproduction speed of the media stream so that the time stamps coincide with each other at the reproduction timing for both received media streams;
A program for a terminal, which causes a computer to function as media stream switching means for switching a media stream to be reproduced when the time stamps of both media streams coincide with each other at the reproduction timing.

Images