JP4911579B2 - Terminal, program and method for storing or playing back stream for analysis - Google Patents

Description

  The present invention relates to a terminal, a program, and a method for storing or reproducing a stream for analysis.

  2. Description of the Related Art Conventionally, there is a “streaming” technique that can simultaneously play multimedia data such as video or audio while receiving it via a network such as the Internet. Even if a large amount of multimedia data is received without receiving all the data, the reproduction can be started immediately. Data distributed for such streaming technology is referred to as a “stream”.

  The stream is composed of a plurality of packets including RTP (Real Time Transport Protocol) and IP (Internet Protocol) headers. The RTP header includes the generation time of the packet. The transmission device transmits a plurality of RTP packets constituting the stream to the reception device. The receiving apparatus removes the header for each RTP packet and generates a stream by combining a plurality of packets. The generated stream is reproduced by the receiving device.

  The receiving apparatus can also store the received stream as a file and play it back when necessary (see, for example, Patent Documents 1 and 2). For example, it is stored in a storage type file format such as 3 gp and matches the original stream transmitted by the transmission device.

JP 2003-348222 A JP 2003-152888 A

  When streaming is performed simultaneously while receiving a stream using the streaming technology, the reproduction is affected by network-specific delay, packet loss, transmission fluctuation, and the like. For example, when a stream is simultaneously played back while receiving a stream via a network including a large delay, the played video or audio may be skipped or jerky.

  On the other hand, the influence of delay or the like is not reflected in the stream file saved by the storage type file format. Therefore, when the stored stream is played back, it is played back in the same way as the playback of the original stream (or skipped if packets are lost).

  Therefore, an object of the present invention is to provide a terminal, a program, and a method that can analyze the influence of a network by storing the stream including the influence of delay and the like and reproducing the stream.

According to the terminal of the present invention,
A stream storage means including a stream receiving means for receiving a stream, a timer for outputting a current time, and a time giving means for giving a reception time for each packet of the stream;
Stream accumulation means for accumulating packets given reception times in the order of reception;
A stream reading means for reading the stream for each packet from the stream storage means;
Time buffer means for delaying the time according to the reception time of the read packet;
And stream reproducing means for reproducing a packet output from the time buffer means and analyzing a transition state of the decoding buffer when decoding .

According to another embodiment of the terminal of the present invention,
The stream is composed of a plurality of RTP (Real time Transport Protocol) packets in which video packets and audio packets are multiplexed,
The stream storage means further includes a call processing means for storing the SDP (Session Description Protocol) file obtained by the call processing negotiation with the counterpart communication device in the stream storage means,
Stream reproducing means, has been a packet output from the time the buffer means, and demultiplexing means for separating the video packet or audio packet, and decoding means for decoding in accordance with the video packets or audio packets in the SDP file, decoded in the decoding means It is also preferable to have analysis means for analyzing the transition state of the buffer .

According to the terminal of the present invention,
Stream accumulation means for accumulating packets given reception times for each packet of the stream in the order of reception;
A stream reading means for reading the stream for each packet from the stream storage means;
Time buffer means for delaying the time according to the reception time of the read packet;
A stream transmission unit configured to transmit the packet output from the time buffer unit to another communication device.

According to another embodiment of the terminal of the present invention,
Positioning means for acquiring position information by a GPS (Global Positioning System) function;
It is preferable to further include position information transmitting means for transmitting the position information to another communication device.

  According to another embodiment of the terminal of the present invention, the stream is preferably a test videophone stream.

According to the present invention,
A stream storage means including a stream receiving means for receiving a stream, a timer for outputting a current time, and a time giving means for giving a reception time for each packet of the stream;
Stream accumulation means for accumulating packets given reception times in the order of reception;
A stream reading means for reading the stream for each packet from the stream storage means;
Time buffer means for delaying the time according to the reception time of the read packet;
The computer is caused to function as a stream reproduction unit that reproduces a packet output from the time buffer unit and analyzes a transition state of the decode buffer at the time of decoding .

According to the present invention,
Stream accumulation means for accumulating packets given reception times for each packet of the stream in the order of reception;
A stream reading means for reading the stream for each packet from the stream storage means;
Time buffer means for delaying the time according to the reception time of the read packet;
A program that causes a computer to function as a stream transmission unit that transmits a packet output from a time buffer unit to another communication device.

According to the present invention, in a method for storing and reproducing a stream in a terminal that receives a stream via a network,
A first step of receiving a stream;
A second step of assigning a reception time to each packet of the stream;
A third step of accumulating the packets with the reception times in the stream accumulation unit in the order of reception;
A fourth step of reading the stream for each packet from the stream storage unit;
A fifth step of delaying the time according to the reception time of the read packet;
A sixth step of reproducing the packet delayed in time and analyzing the transition state of the decoding buffer at the time of decoding is provided.

According to the present invention, in a method for storing and reproducing a stream in a terminal that receives a stream via a network,
A stream accumulating unit that accumulates packets given reception times for each packet of the stream in the order of reception;
A first step of reading the stream for each packet from the stream storage unit;
A second step of delaying the time according to the reception time of the read packet;
And a third step of transmitting the time-delayed packet to another communication device.

  According to the terminal, the program, and the method of the present invention, it is possible to analyze the influence of the network by storing the stream including the influence of delay or the like and reproducing the stream. That is, the video and audio modes to be reproduced are affected by network delay and the like, and it is possible to test the influence received from the network for each application.

  Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a system configuration diagram according to the present invention.

  According to FIG. 1, the terminal 1 receives a stream from the stream distribution server 2 via the base station 5 and the Internet 4. The stream may be received from the terminal 3. The stream is received by the terminal 1 under the influence of network-specific delay, packet loss, transmission fluctuation, and the like.

  The terminal 1 in the present invention stores the received stream in a state including the influence of delay and the like. Then, the stored stream file is reproduced in a state including the influence of delay or the like. The stream may be a test videophone stream. By using the test stream, the transmission state is accurately reproduced.

  The terminal 1 can also output the saved stream file. As a result, the reproduction can be performed in a state including the influence of the delay received by the terminal 1 by another apparatus.

  Further, the terminal 1 can secure a time delay reflecting the influence of delay or the like for each packet of the stored stream file and transmit it to the stream analysis server 6. Thereby, the stream reception state in the terminal 1 is reproduced for the stream analysis server 6.

  FIG. 2 is a data configuration diagram stored according to the present invention.

  The stream is composed of a plurality of RTP packets, and is obtained by multiplexing video packets and audio packets. According to FIG. 2, the video packet and the audio packet including the RTP header are arranged in the order of reception. According to the present invention, a reception time is given to each RTP packet, and the RTP packet level is stored in the stream storage unit in the order of reception. Packets in which errors are detected are also accumulated. By accumulating at the RTP packet level, the received stream may be for RTP testing (for example, a test including CSRC in the RTP header, or a test in which a time stamp circulates). Note that a packet size may be added for simplification of implementation.

FIG. 2 shows an example of an SDP (Session Description Protocol) file. The SDP file is a file that stores session capability information of the counterpart terminal obtained by call processing negotiation. According to FIG. 2, the SDP file is composed of a session description part and one or more media description parts. The video decoding unit and the audio decoding unit are controlled based on information described in the SDP file.
v =: Protocol version o =: SDP file creator information s =: Session name c =: Network address used in session t =: Session start time and end time m =: Media type ("audio", " video ", ...)
a =: Various attribute information about the media

  FIG. 3 is a functional configuration diagram of the terminal in the present invention.

  Referring to FIG. 3, the terminal 1 includes a stream storage unit 10, a stream storage unit 11, a stream reading unit 12, a time buffer unit 13, a stream playback unit 14, a stream transmission unit 15, and a stream file output unit. 16, a positioning unit 17, and a position information transmission unit 18. These functional units can also be realized by a program executed by a computer. The computer may be a mobile phone or a personal computer or a server.

  The stream storage unit 10 stores the SDP file and the video / audio file. The stream accumulation unit 10 accumulates the RTP packets including the reception time notified from the stream storage unit 11 including the error packets in the order of reception. The received RTP packets are not rearranged. This makes it possible to accurately reproduce errors during reception and packet insertion errors. The video file and the audio file may be saved as separate files, or may be saved as a single file.

  The stream storage unit 11 includes a call processing unit 111, a call processing communication unit 112, a stream reception unit 113, a time giving unit 114, and a timer 115.

  The call processing unit 111 establishes a session by performing SIP (Session Initiation Protocol) negotiation via the call processing communication unit 112. In this call processing negotiation, SDP files are exchanged between terminals. The SDP file obtained by the call processing negotiation is notified to the stream storage unit 10.

  The stream receiving unit 113 receives the RTP packet of the stream via the network after the call processing is completed. The received RTP packet is notified to the time giving unit 114.

  The time giving unit 114 gives the current time output from the timer 115 as the reception time for each received RTP packet. The RTP packet with the reception time is notified to the stream storage unit 10. The current time output from the timer 115 is set to 10 msec or less in order to maintain sufficient accuracy with respect to the packet transmission interval.

  The stream reading unit 12 reads the video / audio file and the SDP file from the stream storage unit 10. The video / audio file is read for each packet. Here, specific reading will be described with reference to FIG.

  The time buffer unit 13 secures a time delay between the first packet previously notified from the stream reading unit 12 and the second packet currently notified. The time of the first packet is compared with the time of the second packet, a time delay is secured by the difference, and the second packet is output. The packet output from the time buffer unit 13 is input to the stream reproduction unit 14 or the stream transmission unit 15.

  The stream reproduction unit 14 includes a demultiplexing unit 141, a video decoding unit 142, an audio decoding unit 143, a video analysis unit 144, an audio analysis unit 145, a display unit 146, and a speaker unit 147.

  The demultiplexing unit 141 separates the packet output from the time buffer unit 13 into a video packet and an audio packet. The separated video packet is notified to the video decoding unit 142, and the separated audio packet is notified to the audio decoding unit 143. Note that the demultiplexing unit 141 may perform a lip sync process. The “lip sync process” is a process for correcting a deviation between a video of a person's mouth movement and a speech or a voice of a song. For this processing, a reference signal for this correction needs to be inserted into the video and audio.

  The video decoding unit 142 decodes the video data and outputs it to the display unit 146. The audio decoding unit 143 decodes the audio data and outputs it to the speaker unit 147. According to the present invention, due to the time delay effect of the time buffer unit 13, the stream can be reproduced in an actually received state, that is, with the influence of delay or the like. The video decoding unit 142 and the audio decoding unit 143 are controlled based on the SDP file notified from the stream reading unit 12.

  The video analysis unit 144 records, as a log, transition state information of the decode buffer that the video decoding unit 142 has. The voice analysis unit 145 records the transition state information of the decode buffer held by the voice decoding unit 143 as a log. Thereby, the transition state of the decoding buffer with respect to the packet arrival time can be grasped. It is also possible to test how the transition state changes when the initial accumulation amount or the maximum accumulation amount of the decode buffer is changed.

  When a delay occurs, the decode buffer becomes empty and the video data and audio data are exhausted. On the other hand, when delayed video data and audio data arrive intensively, the decode buffer temporarily overflows. In either case, the video is reproduced with deterioration such as video stop, audio delay, and sound loss.

  The stream transmission unit 15 transmits the packet output from the time buffer unit 13 to another communication device. When the other communication apparatus is, for example, the stream analysis server 6, the stream reception state at the terminal 1 is reproduced with respect to the server 6.

  The stream file output unit 16 outputs the stream file stored in the stream storage unit 10. As a result, other devices can also reproduce with the influence of the delay received by the terminal 1.

  The positioning unit 17 acquires position information by a GPS function. The position information transmission unit 18 transmits the position information to another communication device. Thereby, it is possible to notify the terminal 1 at which position the stream has been received.

  FIG. 4 is a data structure diagram showing the order of reproduction according to the present invention.

(S401) Stream file packets are played back in order from the beginning. At the start of playback, it is assumed that the pointers are at the head (V1 and A1) for both video packets and audio packets. The stream reading unit compares the reception time (12: 00: 00: 000) of V1 with the reception time (12: 00: 00: 030) of A1, and notifies the time buffer unit of V1 having an earlier reception time. Since V1 is the first notified packet, the time buffer unit notifies V1 to the demultiplexing unit with a waiting time of zero. Then, the pointer is advanced from V1 to V2.

(S402) Next, the stream reading unit compares the reception time (12: 00: 00: 133) of V2 with the reception time (12: 00: 00: 030) of A1, and sets A1 with the earlier reception time as time. Notify the buffer. The time buffer unit delays 30 milliseconds, which is the difference between the V1 reception time (12: 00: 00: 000) output earlier and the A1 reception time (12: 00: 00: 030), A1 is notified to the demultiplexing unit. Then, the pointer is advanced from A1 to A2.

(S403) Next, the stream reading unit compares the reception time (12: 00: 00: 133) of V2 with the reception time (12: 00: 00: 090) of A2, and sets A2 with the earlier reception time as time. Notify the buffer. The time buffer unit delays 60 milliseconds, which is the difference between the reception time (12: 00: 00: 030) of A1 output earlier and the reception time (12: 00: 00: 090) of A2, A2 is notified to the demultiplexing unit. Then, the pointer is advanced from A2 to A3.

(S403) Next, the stream reading unit compares the reception time (12: 00: 00: 133) of V2 with the reception time (12: 00: 00: 150) of A3, and determines V2 with the earlier reception time as time. Notify the buffer. The time buffer unit delays 43 milliseconds, which is the difference between the reception time (12: 00: 00: 090) of A2 output earlier and the reception time (12: 00: 00: 133) of V2, V2 is notified to the demultiplexing unit. Then, the pointer is advanced from V2 to V3.

  FIG. 5 is a flowchart in the present invention.

  According to FIG. 5, the storing step is represented in S501 to S508, and the reproducing step is represented in S511 to S519.

Saving step (S501) SIP negotiation is executed with the transmitting apparatus.
(S502) An SDP file is acquired and stored in the stream storage unit.
(S503) The processing up to S508 is repeated in the order of received packets.
(S504) One RTP packet is received.
(S505) The reception time is acquired from the timer.
(S506) A reception time is given to the RTP packet.
(S507) The RTP packet with the reception time is stored in the stream storage unit.
(S508) Return to S503.

Reproduction step (S511) The stream file and the SDP file are read from the stream storage unit.
(S512) The processing up to S519 is repeated for the read stream file in the order of RTP packets.
(S513) Based on the reception time given to the RTP packet, the reception time of the previous RTP packet is compared with the reception time of the current RTP packet, and the difference time is delayed.
(S514) Separation corresponding to the media of the RTP packet. Here, the video packet and the audio packet are separated.
(S515) The video packet is decoded based on the SDP file.
(S516) The decoded video signal is displayed on the display.
(S517) The voice packet is decoded based on the SDP file.
(S518) The decoded audio signal is output from the speaker.
(S519) Return to S512.

  Note that the decoding capability of the receiving apparatus may be caused by a delay or the like in stream reproduction. In addition, the coding capability of the transmission device may be caused by a delay or the like in stream reproduction at the reception device.

  As described above in detail, according to the terminal, the program, and the method of the present invention, it is possible to analyze the influence of the network by saving the stream including the influence of delay and the like. That is, the video and audio modes to be reproduced are affected by network delay and the like, and it is possible to test the influence received from the network for each application.

  Examples of application of the present invention include a videophone service (one-to-one or one-to-N), a streaming service for mobile phones, an Internet radio service, and the like. User support for improving the quality of these services can be considered.

  With regard to the videophone service, when a plurality of users are making a call via a mobile terminal, the contents of the call may be recorded and stored by operating the mobile terminal and then played back as a moving image. At this time, by applying the present invention, a recording during a call (recording of a call state, for example, a delay) is reproduced as it is, so that a realistic reproduction can be performed.

  Further, the stream file recorded and accumulated can be uploaded to the user support center operated by the service provider as it is. The service provider can grasp the quality of the videophone service for the user by playing the stream file.

  At this time, it is also preferable that the mobile terminal transmits the position information acquired by the GPS function to the user support center. The service provider can also improve the quality at the location by specifying the service provision location of the user.

  As described above, according to the present invention, the user support center can also analyze the quality of the stream file received by the mobile terminal. As a result, the service provider can provide appropriate support based on the quality.

  According to the various embodiments of the present invention described above, those skilled in the art can easily make various changes, modifications and omissions within the scope of the technical idea and the viewpoint of the present invention. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

It is a system configuration diagram in the present invention. It is a data block diagram accumulated by the present invention. It is a function block diagram of the terminal in this invention. It is a data block diagram showing the order reproduced | regenerated by this invention. It is a flowchart figure in this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Terminal, receiver 10 Stream storage part 11 Stream storage part 111 Call processing part 112 Call processing communication part 113 Stream receiving part 114 Time giving part 115 Timer 12 Stream reading part 13 Time buffer part 14 Stream reproduction part 141 Demultiplexing part 142 Video | video Decoding unit 143 Audio decoding unit 144 Video analysis unit 145 Audio analysis unit 146 Display unit 147 Speaker unit 15 Stream transmission unit 16 Stream file output unit 17 Positioning unit 18 Location information transmission unit 2 Stream distribution server 3 Terminal, transmission device 4 Internet 5 Base Station 6 Stream analysis (reference) server

Claims (9)

Stream storage means including stream receiving means for receiving a stream; a timer for outputting a current time; and a time giving means for giving a reception time for each packet of the stream;
Stream accumulating means for accumulating the packets given the reception times in the order of reception;
A stream reading means for reading the stream for each packet from the stream storage means;
Time buffer means for delaying the time according to the reception time of the read packet;
A terminal comprising: a stream reproduction unit that reproduces the packet output from the time buffer unit and analyzes a transition state of the decode buffer when decoding the packet. The stream is composed of a plurality of RTP (Real time Transport Protocol) packets in which video packets and audio packets are multiplexed,
The stream storage means further includes a call processing means for storing the SDP (Session Description Protocol) file obtained by the call processing negotiation with the counterpart communication device in the stream storage means,
The stream reproduction means includes demultiplexing means for separating the packet output from the time buffer means into the video packet or the audio packet, and decoding for decoding the video packet or the audio packet according to the SDP file. The terminal according to claim 1, further comprising: means; and analysis means for analyzing a transition state of the decoding buffer in the decoding means . Stream accumulation means for accumulating packets given reception times for each packet of the stream in the order of reception;
Stream reading means for reading the stream for each packet from the stream storage means;
Time buffer means for delaying the time according to the reception time of the read packet;
A terminal comprising: stream transmission means for transmitting the packet output from the time buffer means to another communication device. Positioning means for acquiring position information by a GPS (Global Positioning System) function;
The terminal according to the location information, in any one of claims 1 to 3, further comprising a position information transmitting means for transmitting to another communication device. The terminal according to any one of claims 1 to 4 , wherein the stream is a test videophone stream. Stream storage means including stream receiving means for receiving a stream; a timer for outputting a current time; and a time giving means for giving a reception time for each packet of the stream;
Stream accumulating means for accumulating the packets given the reception times in the order of reception;
A stream reading means for reading the stream for each packet from the stream storage means;
Time buffer means for delaying the time according to the reception time of the read packet;
A program that causes the computer to function as stream reproduction means that reproduces the packet output from the time buffer means and analyzes the transition state of the decode buffer when decoding . Stream accumulation means for accumulating packets given reception times for each packet of the stream in the order of reception;
Stream reading means for reading the stream for each packet from the stream storage means;
Time buffer means for delaying the time according to the reception time of the read packet;
A program that causes a computer to function as a stream transmission unit that transmits the packet output from the time buffer unit to another communication apparatus. In a method for storing and reproducing a stream in a terminal that receives a stream via a network,
A first step of receiving a stream;
A second step of assigning a reception time to each packet of the stream;
A third step of accumulating the packets with the reception times in a stream accumulation unit in the order of reception;
A fourth step of reading the stream for each packet from the stream storage unit;
A fifth step of delaying the time according to the reception time of the read packet;
A stream storage / reproduction method comprising: a sixth step of reproducing the packet delayed in time and analyzing a transition state of the decode buffer at the time of decoding . In a method for storing and reproducing a stream in a terminal that receives a stream via a network,
A stream accumulating unit that accumulates packets given reception times for each packet of the stream in the order of reception;
A first step of reading the stream for each packet from the stream storage unit;
A second step of delaying the time according to the reception time of the read packet;
And a third step of transmitting the time-delayed packet to another communication device.

Images