JP2006115305A - Transmission method for controlling generating timing of packet multiplexed frame in different media data, transmission program, and transmission apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a transmission method, a transmission program, and a transmission apparatus for transmitting packet multiplexed frames including a data structure for improving variations in packet arrival times in different media data and an overhead of headers and for generating the frames in optimum timing. <P>SOLUTION: Each of frames adopts the data structure wherein a serial number part, an audio media data size part, audio media data and video media data are included in succession to a UDP header, the transmission apparatus includes a first buffer for temporarily storing the audio media data, and a second buffer for temporarily storing the video media data, and the transmission apparatus monitors a buffer storage amount in the first buffer whose first threshold value is determined, monitors a buffer storage amount in the first buffer, extracts the audio media data from the first buffer and extracts the video media data from the second buffer to configure the frames when the buffer storage amount in the first buffer exceeds the first threshold value. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a transmission method, a transmission program, and a transmission device that control the generation timing of packet multiplexed frames in different media data.

  A transmission apparatus that multiplexes different media data encodes a plurality of media data generated in parallel at the same time for each medium, and configures the frames into a predetermined size and transmits them to a network. On the other hand, the receiving apparatus decodes the media data in units of received frames by using a media decoder that is different for each medium. How to determine a predetermined size for each media data is important in multiplexing multimedia data.

  As a multiplexing method, there is byte multiplexing. In byte multiplexing, different media data can be switched in units of an appropriate number of bytes. The receiving apparatus switches to a decoder suitable for the media data depending on the contents of the header added to the frame. Such packet multiplexing is highly flexible and suitable for software processing.

  FIG. 1 is a functional configuration diagram of a transmission apparatus for packet multiplexing according to the prior art.

  The transmission device 1 encodes and transmits video data and audio data in real time. The transmission device 1 is connected to a camera 4 that outputs a video signal and a microphone 5 that outputs an audio signal, and is connected to an IP network 3 to transmit a data packet.

  The transmission apparatus 1 includes a format conversion unit 101, a video encoder unit 102, a video data buffer 103, and an RTP (Real-time Transport Protocol) header addition unit 104 for video data. Further, the audio data includes an amplifier 110, an audio encoder unit 108, an audio data buffer 107, and an RTP header adding unit 106. Further, a UDP (User Datagram Protocol) / IP protocol stack unit 105 is provided corresponding to the IP (Internet Protocol) network 3.

  The video signal output from the camera 4 is converted into a video format suitable for compression encoding by the format conversion unit 101 and compressed and encoded by the video encoder 102. The method generates encoded data in units of one frame (one screen). For example, ISO / IEC (International Organization for Standardization / International Electrotechnical Commission) MPEG-4 (Moving Picture Experts Group-4) visual There is. According to the low bit rate MPEG-4, one frame is, for example, at an interval of 100 milliseconds (10 frames per second). The generated bit stream data is input to the video data buffer 103. The RTP header adding unit 104 adds an RTP header for each frame of video data extracted from the buffer 103. The UDP / IP header is added to the RTP packet by the UDP / IP protocol stack unit 105 and transmitted to the IP network 3.

  On the other hand, the audio signal output from the microphone 5 is amplified by the amplifier 110 and converted into a bit string that can be transmitted by an A / D (Analog / Digital) converter 109. Next, the audio data is compression encoded by the audio encoder 108. This method generates encoded data with a predetermined time length as one frame, and includes, for example, 3GPP (3rd Generation Partnership Project) AMR (Adaptive Multi Rate). In the AMR method, one frame is 20 milliseconds. The generated bit stream is input to the audio data buffer 107. The RTP header adding unit 106 adds an RTP header for each frame of video data extracted from the buffer 107. The UDP / IP header is added to the RTP packet by the UDP / IP protocol stack unit 105 and transmitted to the IP network 3.

  The receiving device 2 is connected to the IP network 3 to receive a data packet, and decodes it according to the media data of the received data packet. The receiving device 2 is connected to a display 6 for inputting a video signal and a speaker 7 for inputting an audio signal.

  The receiving device 2 includes a UDP / IP protocol stack unit 205 that processes IP and UDP headers for packets received from the IP network 3. The reception apparatus 2 includes an RTP header removal unit 204, a video decoder 203, a video data buffer 202, and a display control unit 201 for video data. The video signal is output from the display control unit 201 to the display 6. In addition, the audio data includes an RTP header removal unit 206, an audio decoder 207, an audio data buffer 208, a D / A conversion unit 209, and an amplifier 210. The audio signal is output from the amplifier 210 to the speaker 7.

  As prior known documents, IETF RFC3550 (refer to Non-Patent Document 1) that defines RTP, IETF RFC3016 (refer to Non-Patent Document 2) that defines a format in which MPEG-4 is mounted on RTP, and a format in which AMR is mounted on RTP. There is a defined IETF RFC3267 (see Non-Patent Document 3).

IETF (Internet Engineering Task Force) RFC (Request For Comment) 3550 IETF RFC3016 IETF RFC3267

Table 1 shows a data structure of a packet multiplexed frame in the prior art.

  According to the prior art, a frame of audio data and a frame of video data are required separately. This is because each frame is decoded by a decoder corresponding to the media data.

According to the prior art as described above, the generated RTP packet is as follows. Here, a general value at a low transmission rate (64 kbit / sec) is assumed as an encoding parameter.
Video coding bit rate: 32 kbit / s, frame rate: 10 frames / s Audio coding bit rate: 6800 bits / s, frame length: 20 ms RTP header size: 12 bytes
UDP header: 8 bytes
IP header: 20 bytes

The video data is as follows.
Video coding bit rate: 32 kbit / second ÷ 8 bit = 4 kbyte / second Number of transmission frames per second: 10 frame / second Transmission interval of video data packet: 1 second / 10 frame = 100 ms Frame rate: 4 kbyte / second ÷ 10 Frame / second = 400 bytes / frame 1 packet of video: 400 bytes + 12 bytes + 8 bytes + 20 bytes = 440 bytes
Transmission time of one video packet: 440 bytes × 8 bits ÷ 64 kbit / sec = 55 msec

The audio data is as follows.
Audio encoding bit rate: 6800 bits / second ÷ 8 bits = 850 bytes / second Number of transmission frames per second: 50 frames / second Transmission interval of voice data packets: 1 second / 50 frames = 20 milliseconds Number of bytes per frame: 850 bytes / Second x 20 milliseconds = 17 bytes / frame 1 packet of audio: 17 bytes + 12 bytes + 8 bytes + 20 bytes = 57 bytes
Transmission time of one voice packet: 57 bytes × 8 bits ÷ 64 kbit / s = 7.125 ms

  Since audio data packets are transmitted at intervals of 20 msec and video data packets are transmitted at intervals of 100 msec, transmission of video data packets is interrupted at a rate of once every five transmissions of audio data packets.

  FIG. 2 is a frame transmission sequence diagram in the prior art.

  Audio data packets are transmitted every 20 milliseconds and video data packets are transmitted every 100 milliseconds. The voice data packet includes a 40-byte header (IP, UDP, RTP) and a data portion. If the data part is 17 bytes, 7.125 msec is required for transmission. The video data packet includes a 40-byte header and a data portion. If the data portion is 400 bytes, transmission takes 55 milliseconds.

  According to FIG. 2, for example, it is assumed that the audio data packet A3 is transmitted immediately after the transmission of the video data packet V1. The video data packet V1 arrives at the receiving device 55 milliseconds after being transmitted from the transmitting device. On the other hand, the voice data packet A3 arrives at the receiving device after another 7.125 ms, that is, 62.125 ms after being transmitted from the transmitting device. For this reason, when the audio data packet interrupted by the video data packet is received by the receiving device, it is 62 ms after the transmission interval of 20 msec of the transmitting device. Thereafter, the voice data packets A4, A5, A6, A7 arrive in succession at the receiving device. The time required for transmitting the audio data packet is 7 m seconds, which is sufficiently shorter than the transmission interval of 20 msec. Therefore, the arrival interval in the receiving apparatus will be restored to the original time. Due to the interruption, the arrival delay varies.

  Unlike discrete data such as video data that is discrete in units of frames, audio data must be continuous when converted back to an analog signal. For this reason, the receiving apparatus reproduces after accumulating in the buffer in advance according to the maximum arrival delay amount of the voice data packet. Such a delay variation of the voice data packet requires a large buffer accumulation on the receiving side, and increases the entire transmission delay.

Furthermore, according to the prior art, it is necessary to add RTP, UDP, and IP headers independently to each of the video data packet and the audio data packet. Therefore,
(12 bytes + 8 bytes + 20 bytes) x 8 bits x (50 audio frames + 10 video frames)
= 19.2 kbit / s overhead is required.

Further, in order to suppress delay variation of the audio data packet, there is a method of dividing the video data packet and transmitting the video data packet and the audio data packet alternately. In this case, since both packets are transmitted at equal intervals, it is possible to avoid the arrival time fluctuation of the voice data packet. However, according to this method, the overhead due to the header is further increased, and the transmission efficiency is extremely lowered.
(12 bytes + 8 bytes + 20 bytes) x 8 bits x (50 audio frames + 50 video frames)
= 32kbit / s

  As described above, according to the prior art, different media data is transmitted in different frames, and an RTP header is required for each frame. Therefore, not only transmission efficiency is lowered, but transmission of one media data is This will affect the delay variation of media data transmission.

  Also, if the time interval of the frame generation timing is short, the media data delay is small, but the number of frames increases and the header amount increases, resulting in a decrease in transmission efficiency. On the other hand, if the time interval of the frame generation timing is long, the header amount is also relatively decreased and the transmission efficiency is increased, but the delay of the media data is increased.

  Therefore, the present invention transmits a packet multiplexed frame having a data structure capable of improving the fluctuation of the packet arrival time in different media data and the overhead due to the header, and the frame is transmitted at an optimal timing. It is an object to provide a transmission method, a transmission program, and a transmission device that are generated.

The frame transmission method according to the present invention includes:
Subsequent to the UDP header, the data structure includes a serial number portion, a first media data size portion, a first media data having a length stored in the size portion, and a second media data. A transmission method for transmitting packet multiplexed frames of different media data,
A first buffer for temporarily storing first media data, and a second buffer for temporarily storing second media data, wherein a first threshold for the first buffer is determined And
A first step of monitoring a buffer accumulation amount in the first buffer;
When the buffer accumulation amount in the first buffer exceeds the first threshold, the first media data accumulated in the first buffer is stored as the first media data of the frame, and is stored in the second buffer. Storing the accumulated second media data as the second media data of the frame, and configuring a frame.

  According to another embodiment of the method for transmitting a frame of the present invention, the second step is to transfer the first media data in an amount corresponding to the first threshold stored in the first buffer to the first of the frame. It is also preferable to store it as media data.

  According to another embodiment of the method for transmitting a frame of the present invention, a second threshold value for the second buffer is determined, and the second step sets the second threshold value stored in the second buffer to the second threshold value. It is also preferable to store a corresponding amount of the second media data as the second media data of the frame.

According to another embodiment of the frame transmission method of the present invention,
Subsequent to the UDP header, the data structure includes a serial number portion, a first media data size portion, a first media data having a length stored in the size portion, and a second media data. A transmission method for transmitting packet multiplexed frames of different media data,
A first buffer for temporarily storing the first media data; and a second buffer for temporarily storing the second media data. A second threshold for the second buffer is determined. And
A first step of monitoring a buffer accumulation amount in the second buffer;
When the buffer accumulation amount in the second buffer exceeds the second threshold, the first media data accumulated in the first buffer is stored as the first media data of the frame, and is stored in the second buffer. Storing the accumulated second media data as the second media data of the frame, and configuring a frame.

  According to another embodiment of the method for transmitting a frame of the present invention, the second step is to transfer an amount of the second media data corresponding to the second threshold stored in the second buffer to the second of the frame. It is also preferable to store it as media data.

  According to another embodiment of the frame transmission method of the present invention, the first threshold value for the first buffer is determined, and the second step is the first threshold value stored in the first buffer. It is also preferable to store a corresponding amount of the first media data as the first media data of the frame.

  According to another embodiment of the frame transmission method of the present invention, it has a clock means for defining the longest time interval, which is reset every time a frame is formed, and the second step is the first buffer. And / or when the media data is accumulated in the second buffer, but the buffer accumulation amount of the first or second buffer does not reach the first or second threshold value, it is output from the clock means. It is also preferable to construct a frame at a certain timing.

According to the frame transmission method of the present invention,
Subsequent to the UDP header, the data structure includes a serial number portion, a first media data size portion, a first media data having a length stored in the size portion, and a second media data. A transmission method for transmitting packet multiplexed frames of different media data,
A first buffer for temporarily storing first media data; a second buffer for temporarily storing second media data; and clock means for defining a time interval;
According to the timing of the clock means, the first media data stored in the first buffer is stored as the first media data of the frame, and the second media data stored in the second buffer is stored as the first media data of the frame. 2 is stored as media data to form a frame.

  According to another embodiment of the method for transmitting a frame of the present invention, a second threshold value for the second buffer is determined, and the second step sets the second threshold value stored in the second buffer to the second threshold value. It is also preferable to store a corresponding amount of the second media data as the second media data of the frame.

  According to another embodiment of the frame transmission method of the present invention, the first threshold value for the first buffer is determined, and the second step is the first threshold value stored in the first buffer. It is also preferable to store a corresponding amount of the first media data as the first media data of the frame.

  According to another embodiment of the frame transmission method of the present invention, the first media data may be audio data and the second media data may be video data. Further, the first media data may be video data, and the second media data may be audio data.

According to the frame transmission program of the present invention,
Subsequent to the UDP header, the data structure includes a serial number portion, a first media data size portion, a first media data having a length stored in the size portion, and a second media data. A transmission program for transmitting packet multiplexed frames of different media data,
A first buffer for temporarily storing first media data, and a second buffer for temporarily storing second media data, wherein a first threshold for the first buffer is determined And
A first step of monitoring a buffer accumulation amount in the first buffer;
When the buffer accumulation amount in the first buffer exceeds the first threshold, the first media data accumulated in the first buffer is stored as the first media data of the frame, and is stored in the second buffer. The accumulated second media data is stored as the second media data of the frame, and the computer is executed as the second step of configuring the frame.

According to the frame transmission program of the present invention,
Subsequent to the UDP header, the data structure includes a serial number portion, a first media data size portion, a first media data having a length stored in the size portion, and a second media data. A transmission program for transmitting packet multiplexed frames of different media data,
A first buffer for temporarily storing the first media data; and a second buffer for temporarily storing the second media data. A second threshold for the second buffer is determined. And
A first step of monitoring a buffer accumulation amount in the second buffer;
When the buffer accumulation amount in the second buffer exceeds the second threshold, the first media data accumulated in the first buffer is stored as the first media data of the frame, and is stored in the second buffer. The accumulated second media data is stored as the second media data of the frame, and the computer is executed as the second step of configuring the frame.

According to the frame transmission program of the present invention,
Subsequent to the UDP header, the data structure includes a serial number portion, a first media data size portion, a first media data having a length stored in the size portion, and a second media data. A transmission method for transmitting packet multiplexed frames of different media data,
A first buffer for temporarily storing first media data; a second buffer for temporarily storing second media data; and clock means for defining a time interval;
According to the timing of the clock means, the first media data stored in the first buffer is stored as the first media data of the frame, and the second media data stored in the second buffer is stored as the first media data of the frame. 2 is stored as media data, and the computer is executed so as to form a frame.

According to the frame transmitting apparatus of the present invention,
A first buffer for temporarily storing first media data to be transmitted;
A second buffer for temporarily storing second media data to be transmitted;
Subsequent to the UDP header, the data structure includes a serial number portion, a first media data size portion, a first media data having a length stored in the size portion, and a second media data. Multiplexing means for generating packet multiplexed frames of different media data;
A first switch connected between the first buffer and the multiplexing means;
A second switch connected between the second buffer and the multiplexing means;
The first threshold value for the first buffer is accumulated, the buffer accumulation amount in the first buffer is monitored, and when the buffer accumulation amount exceeds the first threshold value, the first media data in the first buffer is stored. And timing control means for controlling the first switch and the second switch so as to send the second media data in the second buffer to the multiplexing means.

  According to another embodiment of the frame transmitting apparatus of the present invention, the first buffer is configured to send an amount of the first media data corresponding to the first threshold value to the multiplexing means at a time. It is also preferable.

  According to another embodiment of the frame transmitting apparatus of the present invention, the second threshold value for the second buffer is determined, and the second buffer has an amount of the second medium corresponding to the second threshold value. It is also preferable that the data is sent to the multiplexing means at once.

According to the frame transmitting apparatus of the present invention,
A first buffer for temporarily storing first media data to be transmitted;
A second buffer for temporarily storing second media data to be transmitted;
Subsequent to the UDP header, the data structure includes a serial number portion, a first media data size portion, a first media data having a length stored in the size portion, and a second media data. Multiplexing means for generating packet multiplexed frames of different media data;
A first switch connected between the first buffer and the multiplexing means;
A second switch connected between the second buffer and the multiplexing means;
The second threshold value for the second buffer is accumulated, the buffer accumulation amount in the second buffer is monitored, and when the buffer accumulation amount exceeds the second threshold value, the first media data in the first buffer is stored. And timing control means for controlling the first switch and the second switch so as to send the second media data in the second buffer to the multiplexing means.

  According to another embodiment of the frame transmitting apparatus of the present invention, the second buffer is configured to send the second media data in an amount corresponding to the second threshold value to the multiplexing means at a time. It is also preferable.

  According to another embodiment of the frame transmitting apparatus of the present invention, the first threshold for the first buffer is determined, and the first buffer has an amount of the first medium corresponding to the first threshold. It is also preferable that the data is sent to the multiplexing means at once.

According to another embodiment of the frame transmission apparatus of the present invention, the frame transmission device includes a clock unit that defines a longest time interval that is reset every time a frame is configured.
The timing control means is clock means when media data is accumulated in the first buffer and / or the second buffer, but the buffer accumulation amount of the first buffer does not reach the first threshold value. It is also preferable to construct a frame at the timing output from.

According to the frame transmitting apparatus of the present invention,
A first buffer for temporarily storing first media data to be transmitted;
A second buffer for temporarily storing second media data to be transmitted;
Subsequent to the UDP header, the data structure includes a serial number portion, a first media data size portion, a first media data having a length stored in the size portion, and a second media data. Multiplexing means for generating packet multiplexed frames of different media data;
A first switch connected between the first buffer and the multiplexing means;
A second switch connected between the second buffer and the multiplexing means;
Clock means for defining the time interval;
The first and second switches send the first media data stored in the first buffer to the multiplexing means in accordance with the timing of the clock means, and the second media data stored in the second buffer is sent to the multiplexing means. The data is sent to the multiplexing means.

  According to another embodiment of the frame transmitting apparatus of the present invention, the second threshold value for the second buffer is determined, and the second switch is set to the second threshold value stored in the second buffer. It is also preferable to send a corresponding amount of second media data to the multiplexing means.

According to another embodiment of the frame transmission apparatus of the present invention,
A first threshold for the first buffer has been determined;
It is also preferable that the second switch sends the first media data in an amount corresponding to the first threshold stored in the first buffer to the multiplexing means.

According to another embodiment of the frame transmission apparatus of the present invention,
The multiplexing means is
Adding means for sending out the serial number;
A first memory for temporarily storing first media data;
A second memory for temporarily storing second media data;
The serial number output from the adding means is included in the serial number portion of the frame, and then the first media data size output from the first memory is included in the first media data size portion of the frame. Including the first media data of the length stored in the first media data size portion output from the first memory, and then the second media data output from the second memory. It is also preferable to have multiple packet generation means for repeating the generation of the included packet multiplexed frame.

  According to another embodiment of the frame transmitting apparatus of the present invention, the first media data may be audio data and the second media data may be video data. Further, the first media data may be video data, and the second media data may be audio data.

  According to the present invention, the frame generation timing is controlled according to the buffer accumulation amount without being controlled at time intervals. This control is performed according to a predetermined threshold for each buffer. This threshold value can be arbitrarily determined from the correlation between the delay of the media data and the transmission efficiency depending on the header amount of the frame, and the frame can be generated at the optimum timing. Even if the buffer accumulation amount does not reach the threshold value, frames are generated at predetermined time intervals.

  According to the present invention, video data and audio data are extracted from the buffer at the same timing, multiplexed into one frame, and transmitted. Thereby, the arrival delay of the video data and the audio data becomes constant, and the arrival fluctuation of the audio data can be avoided in the receiving device.

  Further, when generating multiple packets, 12 bytes are required for each of the video data packet and the audio data packet by RTP. According to the present invention, these different media data packets are integrated and correspond to RTP. Since the header of the portion is 2 bytes, the header size can be considerably reduced and the transmission efficiency can be increased.

  Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.

Table 2 shows the data structure of the packet multiplexed frame used by the present invention.

  According to Table 2, the frame data structure according to the present invention has a UDP header followed by a 1-byte serial number portion, a 1-byte audio data size portion, and audio data of a length stored in the size portion, Video data. Therefore, an RTP header is not required as compared with Table 1. The serial number part is an unsigned integer and has a cyclic value of 0 to 255. Since the audio data size part is 1 byte, the size of the audio data is inevitably 0 to 255 bytes.

  FIG. 3 is a diagram illustrating a media division configuration of a packet multiplexed frame according to the present invention.

  In the upper part of FIG. 3, the video data is divided into equal sizes to form packet multiplexed frames. Since video data can always be divided at a predetermined number of bytes, mounting is easy. However, in order to divide into equal sizes without being conscious of the space between the frames, the frames may be divided across the frames. In this case, if a packet including video data straddling video frames is lost, it affects two (front and rear) frames.

  The lower part of FIG. 3 shows a video frame in which a Resync marker for resynchronization defined by MPEG4 is inserted. A Resync marker is a bit string of a specific pattern, which is inserted into a stream and indicates a break of a variable-length code word. By resuming decoding from the Resync marker when an error occurs, playback from the middle of the stream becomes possible. The Resync marker is inserted at the macroblock boundary during encoding. By inserting the Resync markers between substantially equal sizes and including the spaces between the Resync markers as one video data in the multiplexed frame, the video data is not divided across the video frames. That is, the loss of one multiplexed frame does not affect two video frames.

  FIG. 4 is a functional configuration diagram of the transmission apparatus used according to the present invention. Hereinafter, functions of the transmission apparatus will be described, but each function can also be realized by a program.

  According to FIG. 4, the transmission apparatus 1 includes a multiplexing unit 111 instead of the RTP header addition units 104 and 106 in FIG. 1. The multiplexing unit 111 includes a multiplexed packet generation unit 111, a memory 1112 that temporarily stores video data, a memory 1113 that temporarily stores audio data, an adder 1114 that generates a serial number, and a delay circuit 1115. And have. Here, the audio data memory 1112 can output the audio data size (difference between the write pointer and the read pointer) in 0 to 255 bytes.

  When the audio data is stored in the memory 113 and / or the video data is stored in the memory 112, the multiplex packet generation unit 111 is a switch of 4 inputs and 1 output, and includes a serial number, audio data size, audio data, and video data. By switching in order, one frame shown in Table 2 is formed by one round. The serial number is input to the adder 1114 through the delay circuit 115 and incremented by 1 (added by 1) to become a new serial number.

  According to the present invention, video data and audio data are extracted from the buffers 103 and 107 at the same timing, shaped, and multiplexed into one frame. As a result, the arrival delay between the video data and the audio data is constant, and the arrival fluctuation of the audio data can be avoided in the receiving apparatus.

  Also, when generating multiple packets, RTP required 12 bytes for each of the video data packet and the audio data packet, whereas according to the present invention, these different media data packets are integrated and correspond to RTP. Since the header of the part is 2 bytes, the header size can be reduced.

For example, if the packet transmission interval is 20 milliseconds,
(2 bytes + 8 bytes + 20 bytes) × 8 bits × 50 = 12 kbit / second overhead.

  That is, considering that the conventional technique is 32 kbit / sec, the present invention can reduce the overhead of 20 kbit / sec. The number of bits of overhead reduced here can be assigned to the encoding bit rate of video data or audio data, and the encoding quality can be improved.

  FIG. 5 is a frame transmission sequence diagram in the transmission apparatus of FIG. Hereinafter, functions of the receiving apparatus will be described, but each function can also be realized by a program.

  FIG. 5 shows a portion of S200 of FIG. If an attempt is made to transmit the audio data packet transmitted every 20 milliseconds, a video data packet having a shorter length than the video data packet of FIG. 2 can be transmitted. According to FIG. 5, a frame composed of 17-byte audio data and 80-byte video data is received by the receiving device in 15.8 milliseconds. That is, the reception end timing of the video data V1 (400 bytes) coincides with the reception end timing of the audio data A1 to A4 (17 bytes × 4 pieces).

For example, the AMR audio data size has the following number of bytes when the packet transmission interval is 20 milliseconds. Here, typical three modes are listed.
4.75 kbit / s mode: 14 bytes
* 6.70 kbit / s mode: 19 bytes
12.2 kbit / s mode: 33 bytes

The video data size is simply expressed as “bit rate × time”. When one packet is 20 ms apart, the number of bytes is as follows.
32kbit / s: 32kbit / 8bit x 20msec = 80byte

  When the AMR is in the 6.70 kbit / second mode, the multiplexed frame requires a total of 101 bytes including a serial number of 1 byte, audio data size of 1 byte, audio data of 19 bytes, and video data of 80 bytes.

  FIG. 6 is a first functional configuration diagram of the transmission apparatus according to the present invention. Hereinafter, functions of the transmission apparatus will be described, but each function can also be implemented by a program, and each function is realized by the computer executing the program.

  6 further includes a timing control unit 112, switches 113 and 114, and a clock 116, as compared with FIG. The switch 113 is provided between the video data buffer 103 and the multiplexing unit 111 and limits the amount of video data for generating one frame. The switch 114 is provided between the audio data buffer 107 and the multiplexing unit 111, and limits the amount of audio data for generating one frame.

  The timing control unit 112 includes a comparator 1121 and a threshold storage unit 1122. The threshold storage unit 1122 stores a first threshold for the audio data buffer 107. The comparator 1121 compares the buffer accumulation amount received from the audio data buffer 107 with the first threshold value. When the buffer accumulation amount exceeds the first threshold, the switches 113 and 114 are controlled to be ON.

  By the ON control of the switch 114, the audio data buffer 107 sends an amount of audio data corresponding to the first threshold value to the multiplexing unit 111. The audio data buffer 107 can transmit only an amount of audio data corresponding to the first threshold value at a time. On the other hand, the second threshold value is also determined for the video data buffer 103, and the video data buffer 103 sends an amount of video data corresponding to the second threshold value to the multiplexing unit 111 by the ON control of the switch 113. Send it out. The video data buffer 103 can also send only video data of an amount corresponding to the second threshold value at a time.

  For example, when the voice data is AMR, the packet transmission interval is 20 milliseconds, so the voice data size in the 12.2 kbit / second mode is 33 bytes. By setting the first threshold value to 33 bytes, a frame is formed when 33 bytes are stored in the audio data buffer 107. In addition, even if there is a slight delay in the audio data, if priority is given to frame transmission efficiency, for example, two frames of audio data can be generated in one frame. At this time, by specifying the first threshold value to 66 bytes (33 bytes × 2), a frame is formed when 66 bytes are stored in the audio data buffer 107.

  Note that the audio data buffer 107 may send all the accumulated audio data to the multiplexing unit 111 by ON control of the switch 114. On the other hand, the video data buffer 103 may transmit all the stored video data to the multiplexing unit 111 by ON control of the switch 113.

  FIG. 7 is a second functional configuration diagram of the transmission apparatus according to the present invention.

  7, compared with FIG. 6, the comparator 1121 controls the switches 113 and 114 based on the buffer occupation amount of the video data buffer 103. The threshold storage unit 1122 of the timing control unit 112 stores a second threshold for the video data buffer 103. The comparator 1121 compares the buffer accumulation amount received from the video data buffer 103 with the second threshold value. When the buffer accumulation amount exceeds the second threshold, the switches 113 and 114 are controlled to be ON.

  As described above, the buffer monitored by the timing control unit 112 may be only the audio data buffer 107 or only the video data buffer 103. Further, both buffers may be monitored. That is, when either the first threshold value or the second threshold value is exceeded, the switches 113 and 114 are controlled to be ON.

  The clock 116 defines the longest time interval for transmitting a frame and is reset every time a frame is configured. This clock instructs the comparator 1121 to time out for a predetermined time. Basically, the comparator 1121 turns on the switches 113 and 114 until the media data stored in the audio data buffer 107 and / or the video data buffer 103 reaches the first and / or second threshold. Do not control. Therefore, media data may remain in the buffer. Upon receiving a timeout notification from the clock 116, the comparator 1121 forcibly controls the switches 113 and 114 to input the media data to the multiplexing unit 111. Specifically, the clock cycle takes a time longer than the 20 msec packet transmission interval in consideration of AMR audio data.

  FIG. 8 is a third functional configuration diagram of the transmission apparatus according to the present invention.

  In the transmission device 1 of FIG. 8, the clock 116 controls ON / OFF of the switches 113 and 114 instead of the timing control unit 112 as compared with FIG. 6. If the audio data is an AMR system, the clock 116 controls the switches 113 and 114 with a period of, for example, 20 milliseconds. Thus, in the normal state, 17-byte audio data and 80-byte video data are multiplexed and transmitted in one frame. As described above, controlling the switches 113 and 114 using the clock 116 having a fixed period is extremely simple in terms of mounting and can realize the frame transmission timing similar to that in FIG.

  Here, the audio data buffer 107 can also define a first threshold value that is a buffer amount that can be transmitted by one-time ON control of the switch 114. Further, the video data buffer 103 can also define a second threshold value that is a buffer amount that can be transmitted by one-time ON control of the switch 113. By defining the first threshold value to 17 bytes and the second threshold value to 80 bytes, it is possible to reliably realize the same frame transmission timing as in FIG.

  FIG. 9 is a fourth functional configuration diagram of the transmission apparatus according to the present invention.

  The transmission apparatus in FIG. 9 includes a clock 1116 in the multiplexing unit 111, as compared with FIG. The clock 1116 controls the multiplex packet generator 1111 with a period of, for example, 20 milliseconds. The multiplexed packet generator 1111 configures one multiplexed frame according to the timing of the clock 1116. Thus, in the normal state, 17-byte audio data and 80-byte video data are multiplexed and transmitted in one frame.

  FIG. 10 is a fifth functional configuration diagram of the transmission apparatus according to the present invention.

  The transmission apparatus 1 in FIG. 10 does not include the switch 113 and the switch 114 as compared with FIG. Instead, the timing control signal output from the timing controller 112 is input to the multiplexed packet generator 1111 of the multiplexer 111. The multiplex packet generator 1111 controls the generation timing of one frame based on the timing control signal. That is, when the comparator 1121 determines that the buffer accumulation amount of the audio data buffer 107 exceeds the first threshold, the multiplex packet generation unit 1111 operates to generate one frame.

  According to the above-described various embodiments of the transmission method, the transmission program, and the transmission apparatus for controlling the generation timing of the packet multiplexed frame in the present invention, various changes, modifications, and omissions in the technical idea and scope of the present invention are possible. Those skilled in the art can easily do this. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

It is a function block diagram of the transmitter for packet multiplexing by a prior art. It is the transmission sequence figure of the frame in a prior art. It is a media division | segmentation block diagram of the packet multiplexing frame in this invention. It is a function block diagram of the transmitter used by this invention. FIG. 5 is a frame transmission sequence diagram in the transmission device of FIG. It is a 1st function block diagram of the transmitter in this invention. It is a 2nd function block diagram of the transmitter in this invention. It is a 3rd function block diagram of the transmitter in this invention. It is a 4th function block diagram of the transmitter in this invention. It is a 5th function block diagram of the transmitter in this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Transmission apparatus 101 Format conversion part 102 Video encoder part 103 Video data buffer 104, 106 RTP header addition part 105 UDP / IP protocol stack part 107 Audio data buffer 108 Audio encoder part 109 A / D conversion part 110 Amplifier 111 Multiplexing Unit 116 Clock 1111 Multiplex packet generator 1112 Video data memory 1113 Audio data memory 1114 Adder 1115 Delay circuit 1116 Clock 112 Timing control unit 1121 Comparator 1122 Threshold accumulation unit 113, 114 Switch 116 Clock 2 Receiver 201 Display control unit 202 Video data buffer 203 Video decoder 204, 206 RTP header removal unit 205 UDP / IP protocol stack unit 2 07 Audio decoder 208 Audio data buffer 209 D / A converter 210 Amplifier 3 IP network 4 Camera 5 Microphone 6 Display 7 Speaker

Claims (28)

Subsequent to the UDP header, the data structure includes a serial number portion, a first media data size portion, a first media data having a length stored in the size portion, and a second media data. A transmission method for transmitting packet multiplexed frames of different media data,
A first buffer that temporarily stores the first media data; and a second buffer that temporarily stores the second media data, wherein a first threshold for the first buffer is Has been determined,
A first step of monitoring a buffer accumulation amount in the first buffer;
When the buffer accumulation amount in the first buffer exceeds the first threshold, the first media data accumulated in the first buffer is stored as first media data of the frame. And a second step of storing the second media data stored in the second buffer as second media data of the frame and configuring the frame. .   The second step stores the amount of the first media data corresponding to the first threshold accumulated in the first buffer as the first media data of the frame. The transmission method according to claim 1. A second threshold for the second buffer is determined;
The second step stores the amount of the second media data corresponding to the second threshold accumulated in the second buffer as second media data of the frame. The transmission method according to claim 1 or 2. Subsequent to the UDP header, the data structure includes a serial number portion, a first media data size portion, a first media data having a length stored in the size portion, and a second media data. A transmission method for transmitting packet multiplexed frames of different media data,
A first buffer for temporarily storing the first media data; and a second buffer for temporarily storing the second media data, wherein a second threshold for the second buffer is Has been determined,
A first step of monitoring a buffer accumulation amount in the second buffer;
When the buffer accumulation amount in the second buffer exceeds the second threshold, the first media data accumulated in the first buffer is stored as the first media data of the frame. And a second step of storing the second media data stored in the second buffer as second media data of the frame and configuring the frame. .   The second step stores the amount of the second media data corresponding to the second threshold accumulated in the second buffer as second media data of the frame. The transmission method according to claim 4. A first threshold for the first buffer is determined;
The second step stores the amount of the first media data corresponding to the first threshold accumulated in the first buffer as the first media data of the frame. The transmission method according to claim 4 or 5. A clock means for defining a longest time interval, which is reset every time the frame is configured;
In the second step, although the media data is accumulated in the first buffer and / or the second buffer, the buffer accumulation amount of the first or second buffer is the first or second buffer. 7. The transmission method according to claim 1, wherein, when the second threshold value is not reached, the frame is configured at a timing output from the clock unit. 8. Subsequent to the UDP header, the data structure includes a serial number portion, a first media data size portion, a first media data having a length stored in the size portion, and a second media data. A transmission method for transmitting packet multiplexed frames of different media data,
A first buffer for temporarily storing the first media data; a second buffer for temporarily storing the second media data; and a clock means for defining a time interval.
According to the timing of the clock means, the first media data stored in the first buffer is stored as first media data of the frame, and the second media stored in the second buffer is stored. A transmission method comprising: storing data as second media data of the frame to configure the frame. A second threshold for the second buffer is determined;
The second step stores the amount of the second media data corresponding to the second threshold accumulated in the second buffer as second media data of the frame. The transmission method according to claim 8. A first threshold for the first buffer is determined;
The second step stores the amount of the first media data corresponding to the first threshold accumulated in the first buffer as the first media data of the frame. The transmission method according to claim 8 or 9.   The transmission method according to any one of claims 1 to 10, wherein the first media data is audio data, and the second media data is video data.   The transmission method according to any one of claims 1 to 10, wherein the first media data is video data, and the second media data is audio data. Subsequent to the UDP header, the data structure includes a serial number portion, a first media data size portion, a first media data having a length stored in the size portion, and a second media data. A transmission program for transmitting packet multiplexed frames of different media data,
A first buffer that temporarily stores the first media data; and a second buffer that temporarily stores the second media data, wherein a first threshold for the first buffer is Has been determined,
A first step of monitoring a buffer accumulation amount in the first buffer;
When the buffer accumulation amount in the first buffer exceeds the first threshold, the first media data accumulated in the first buffer is stored as first media data of the frame. The second media data stored in the second buffer is stored as the second media data of the frame, and the computer is executed as a second step of configuring the frame. Send program. Subsequent to the UDP header, the data structure includes a serial number portion, a first media data size portion, a first media data having a length stored in the size portion, and a second media data. A transmission program for transmitting packet multiplexed frames of different media data,
A first buffer for temporarily storing the first media data; and a second buffer for temporarily storing the second media data, wherein a second threshold for the second buffer is Has been determined,
A first step of monitoring a buffer accumulation amount in the second buffer;
When the buffer accumulation amount in the second buffer exceeds the second threshold, the first media data accumulated in the first buffer is stored as the first media data of the frame. The second media data stored in the second buffer is stored as the second media data of the frame, and the computer is executed as a second step of configuring the frame. Send program. Subsequent to the UDP header, the data structure includes a serial number portion, a first media data size portion, a first media data having a length stored in the size portion, and a second media data. A transmission method for transmitting packet multiplexed frames of different media data,
A first buffer for temporarily storing the first media data; a second buffer for temporarily storing the second media data; and a clock means for defining a time interval.
According to the timing of the clock means, the first media data stored in the first buffer is stored as first media data of the frame, and the second media stored in the second buffer is stored. A transmission program that stores data as second media data of the frame and causes a computer to execute the frame. A first buffer for temporarily storing first media data to be transmitted;
A second buffer for temporarily storing second media data to be transmitted;
Subsequent to the UDP header, the data structure includes a serial number portion, a first media data size portion, a first media data having a length stored in the size portion, and a second media data. Multiplexing means for generating packet multiplexed frames of different media data;
A first switch connected between the first buffer and the multiplexing means;
A second switch connected between the second buffer and the multiplexing means;
Storing a first threshold value for the first buffer; monitoring a buffer storage amount in the first buffer; and when the buffer storage amount exceeds the first threshold value, Timing control means for controlling the first switch and the second switch so as to send the first media data and the second media data in the second buffer to the multiplexing means. A transmitter characterized by the above.   The said 1st buffer is comprised so that the said 1st media data of the quantity equivalent to the said 1st threshold value may be sent to the said multiplexing means at once. Transmitter device. A second threshold for the second buffer is determined;
The said 2nd buffer is comprised so that the said 2nd media data of the quantity equivalent to the said 2nd threshold value may be sent to the said multiplexing means at once, The Claim 16 or 17 characterized by the above-mentioned. The transmitting device described. A first buffer for temporarily storing first media data to be transmitted;
A second buffer for temporarily storing second media data to be transmitted;
Subsequent to the UDP header, the data structure includes a serial number portion, a first media data size portion, a first media data having a length stored in the size portion, and a second media data. Multiplexing means for generating packet multiplexed frames of different media data;
A first switch connected between the first buffer and the multiplexing means;
A second switch connected between the second buffer and the multiplexing means;
Accumulating a second threshold for the second buffer, monitoring a buffer accumulation amount in the second buffer, and when the buffer accumulation amount exceeds the second threshold, the first buffer Timing control means for controlling the first switch and the second switch so as to send the first media data and the second media data in the second buffer to the multiplexing means. A transmitter characterized by the above.   The said 2nd buffer is comprised so that the amount of said 2nd media data equivalent to the said 2nd threshold value may be sent to the said multiplexing means at once. Transmitter device. A first threshold for the first buffer is determined;
The said 1st buffer is comprised so that the said 1st media data of the quantity corresponding to the said 1st threshold value may be sent to the said multiplexing means at once. The transmitting device described. A clock means for defining a longest time interval, which is reset every time the frame is configured;
The timing control means has the buffer accumulation amount of the first buffer reaching the first threshold value even though media data is accumulated in the first buffer and / or the second buffer. The transmission apparatus according to any one of claims 16 to 21, wherein if there is not, the frame is configured at a timing output from the clock means. A first buffer for temporarily storing first media data to be transmitted;
A second buffer for temporarily storing second media data to be transmitted;
Subsequent to the UDP header, the data structure includes a serial number portion, a first media data size portion, a first media data having a length stored in the size portion, and a second media data. Multiplexing means for generating packet multiplexed frames of different media data;
A first switch connected between the first buffer and the multiplexing means;
A second switch connected between the second buffer and the multiplexing means;
Clock means for defining the time interval;
The first and second switches send the first media data stored in the first buffer to the multiplexing means according to the timing of the clock means, and are stored in the second buffer. A transmitting apparatus, wherein the second media data is sent to the multiplexing means. A second threshold for the second buffer is determined;
The said 2nd switch sends out the said 2nd media data of the quantity corresponded to the said 2nd threshold value accumulate | stored in the said 2nd buffer to the said multiplexing means. Transmitter. A first threshold for the first buffer is determined;
25. The second switch sends out the first media data in an amount corresponding to the first threshold stored in the first buffer to the multiplexing unit. The transmitting device according to 1. The multiplexing means includes
Adding means for sending the serial number;
A first memory for temporarily storing the first media data;
A second memory for temporarily storing the second media data;
The serial number output from the adding means is included in the serial number portion of the frame, and then the first media data size output from the first memory is set to the first media of the frame. Including the first media data of the length stored in the first media data size portion, which is output from the first memory and then stored in the first media data size portion; The transmission apparatus according to any one of claims 16 to 25, further comprising: multiple packet generation means for repeatedly generating a packet multiplexed frame including the second media data output from the memory.   27. The transmission apparatus according to claim 16, wherein the first media data is audio data, and the second media data is video data.   27. The transmission apparatus according to claim 16, wherein the first media data is video data, and the second media data is audio data.

Images