JP4457841B2 - Transmission apparatus, transmission program, and transmission method for controlling coding rate - Google Patents

Description

  The present invention relates to a transmission apparatus, a transmission program, and a transmission method for controlling a coding rate.

  A transmission apparatus that multiplexes different media data encodes a plurality of media data generated in parallel at the same time for each medium, and configures the frames into a predetermined size and transmits them to a network. On the other hand, the receiving apparatus decodes the media data in units of received frames by using a media decoder that is different for each medium. How to determine a predetermined size for each media data is important in multiplexing multimedia data.

  As a multiplexing method, there is byte multiplexing. In byte multiplexing, different media data can be switched in units of an appropriate number of bytes. The receiving apparatus switches to a decoder suitable for the media data depending on the contents of the header added to the frame. Such packet multiplexing is highly flexible and suitable for software processing.

  FIG. 1 is a functional configuration diagram of a transmission apparatus for packet multiplexing according to the prior art.

  The transmission device 1 encodes and transmits video data and audio data in real time. The transmission device 1 is connected to a camera 4 that outputs a video signal and a microphone 5 that outputs an audio signal, and is connected to an IP network 3 to transmit a data packet.

  The transmission apparatus 1 includes a format conversion unit 101, a video encoder unit 102, a video data buffer 103, and an RTP (Real-time Transport Protocol) header addition unit 104 for video data. Further, the audio data includes an amplifier 110, an audio encoder unit 108, an audio data buffer 107, and an RTP header adding unit 106. Further, a UDP (User Datagram Protocol) / IP protocol stack unit 105 is provided corresponding to the IP (Internet Protocol) network 3.

  The video signal output from the camera 4 is converted into a video format suitable for compression encoding by the format conversion unit 101 and compressed and encoded by the video encoder 102. The method generates encoded data in units of one frame (one screen). For example, ISO / IEC (International Organization for Standardization / International Electrotechnical Commission) MPEG-4 (Moving Picture Experts Group-4) visual There is. According to the low bit rate MPEG-4, one frame is, for example, at an interval of 100 milliseconds (10 frames per second). The generated bit stream data is input to the video data buffer 103. The RTP header adding unit 104 adds an RTP header for each frame of video data extracted from the buffer 103. The UDP / IP header is added to the RTP packet by the UDP / IP protocol stack unit 105 and transmitted to the IP network 3.

  On the other hand, the audio signal output from the microphone 5 is amplified by the amplifier 110 and converted into a bit string that can be transmitted by an A / D (Analog / Digital) converter 109. Next, the audio data is compression encoded by the audio encoder 108. This method generates encoded data with a predetermined time length as one frame, and includes, for example, 3GPP (3rd Generation Partnership Project) AMR (Adaptive Multi Rate). In the AMR method, one frame is 20 milliseconds. The generated bit stream is input to the audio data buffer 107. The RTP header adding unit 106 adds an RTP header for each frame of video data extracted from the buffer 107. The UDP / IP header is added to the RTP packet by the UDP / IP protocol stack unit 105 and transmitted to the IP network 3.

  The receiving device 2 is connected to the IP network 3 to receive a data packet, and decodes it according to the media data of the received data packet. The receiving device 2 is connected to a display 6 for inputting a video signal and a speaker 7 for inputting an audio signal.

  The receiving device 2 includes a UDP / IP protocol stack unit 205 that processes IP and UDP headers for packets received from the IP network 3. The reception apparatus 2 includes an RTP header removal unit 204, a video decoder 203, a video data buffer 202, and a display control unit 201 for video data. The video signal is output from the display control unit 201 to the display 6. In addition, the audio data includes an RTP header removal unit 206, an audio decoder 207, an audio data buffer 208, a D / A conversion unit 209, and an amplifier 210. The audio signal is output from the amplifier 210 to the speaker 7.

  As prior known documents, IETF RFC3550 (refer to Non-Patent Document 1) that defines RTP, IETF RFC3016 (refer to Non-Patent Document 2) that defines a format in which MPEG-4 is mounted on RTP, and a format in which AMR is mounted on RTP. There is a defined IETF RFC3267 (see Non-Patent Document 3).

IETF (Internet Engineering Task Force) RFC (Request For Comment) 3550 IETF RFC3016 IETF RFC3267

Table 1 shows a data structure of a packet multiplexed frame in the prior art.

  According to the prior art, a frame of audio data and a frame of video data are required separately. This is because each frame is decoded by a decoder corresponding to the media data.

According to the prior art as described above, the generated RTP packet is as follows. Here, a general value at a low transmission rate (64 kbit / sec) is assumed as an encoding parameter.
Video coding bit rate: 32 kbit / s, frame rate: 10 frames / s Audio coding bit rate: 6800 bits / s, frame length: 20 ms RTP header size: 12 bytes
UDP header: 8 bytes
IP header: 20 bytes

The video data is as follows.
Video coding bit rate: 32 kbit / second ÷ 8 bit = 4 kbyte / second Number of transmission frames per second: 10 frame / second Transmission interval of video data packet: 1 second / 10 frame = 100 ms Frame rate: 4 kbyte / second ÷ 10 Frame / second = 400 bytes / frame 1 packet of video: 400 bytes + 12 bytes + 8 bytes + 20 bytes = 440 bytes
Transmission time of one video packet: 440 bytes × 8 bits ÷ 64 kbit / sec = 55 msec

The audio data is as follows.
Audio encoding bit rate: 6800 bits / second ÷ 8 bits = 850 bytes / second Number of transmission frames per second: 50 frames / second Transmission interval of voice data packets: 1 second / 50 frames = 20 milliseconds Number of bytes per frame: 850 bytes / Second x 20 milliseconds = 17 bytes / frame 1 packet of audio: 17 bytes + 12 bytes + 8 bytes + 20 bytes = 57 bytes
Transmission time of one voice packet: 57 bytes × 8 bits ÷ 64 kbit / s = 7.125 ms

  Since audio data packets are transmitted at intervals of 20 msec and video data packets are transmitted at intervals of 100 msec, transmission of video data packets is interrupted at a rate of once every five transmissions of audio data packets.

  FIG. 2 is a frame transmission sequence diagram in the prior art.

  Audio data packets are transmitted every 20 milliseconds and video data packets are transmitted every 100 milliseconds. The voice data packet includes a 40-byte header (IP, UDP, RTP) and a data portion. If the data part is 17 bytes, 7.125 msec is required for transmission. The video data packet includes a 40-byte header and a data portion. If the data portion is 400 bytes, transmission takes 55 milliseconds.

  According to FIG. 2, for example, it is assumed that the audio data packet A3 is transmitted immediately after the transmission of the video data packet V1. The video data packet V1 arrives at the receiving device 55 milliseconds after being transmitted from the transmitting device. On the other hand, the voice data packet A3 arrives at the receiving device after another 7.125 ms, that is, 62.125 ms after being transmitted from the transmitting device. For this reason, when the audio data packet interrupted by the video data packet is received by the receiving device, it is 62 ms after the transmission interval of 20 msec of the transmitting device. Thereafter, the voice data packets A4, A5, A6, A7 arrive in succession at the receiving device. The time required for transmitting the audio data packet is 7 m seconds, which is sufficiently shorter than the transmission interval of 20 msec. Therefore, the arrival interval in the receiving apparatus will be restored to the original time. Due to the interruption, the arrival delay varies.

  Unlike discrete data such as video data that is discrete in units of frames, audio data must be continuous when converted back to an analog signal. For this reason, the receiving apparatus reproduces after accumulating in the buffer in advance according to the maximum arrival delay amount of the voice data packet. Such a delay variation of the voice data packet requires a large buffer accumulation on the receiving side, and increases the entire transmission delay.

Furthermore, according to the prior art, it is necessary to add RTP, UDP, and IP headers independently to each of the video data packet and the audio data packet. Therefore,
(12 bytes + 8 bytes + 20 bytes) x 8 bits x (50 audio frames + 10 video frames)
= 19.2 kbit / s overhead is required.

Further, in order to suppress delay variation of the audio data packet, there is a method of dividing the video data packet and transmitting the video data packet and the audio data packet alternately. In this case, since both packets are transmitted at equal intervals, it is possible to avoid the arrival time fluctuation of the voice data packet. However, according to this method, the overhead due to the header is further increased, and the transmission efficiency is extremely lowered.
(12 bytes + 8 bytes + 20 bytes) x 8 bits x (50 audio frames + 50 video frames)
= 32kbit / s

  As described above, according to the prior art, different media data is transmitted in different frames, and an RTP header is required for each frame. Therefore, not only transmission efficiency is lowered, but transmission of one media data is This will affect the delay variation of media data transmission.

  However, when the transmission path rate, that is, the buffer output rate is lower than the buffer input rate for a buffer that temporarily stores multiplexed frame packets, the number of packets stored in the buffer increases with time. In the Internet without bandwidth guarantee and the mobile network in which the wireless environment is likely to change, the transmission path rate fluctuates significantly. For voice data in real-time communication, sound interruption occurs due to voice data not arriving at a desired time interval. In addition, the video data is perceived by the user as fluctuations in the frame interval and display delay.

  In particular, in real-time communication for conversational purposes such as IP videophone, at least audio data needs to be transmitted without delay in order to establish a conversation. For this reason, in an environment where the transmission path rate fluctuates, control for reliably transmitting audio data is required.

  Therefore, the present invention provides a transmission apparatus, a transmission program, and a transmission method for controlling the coding rate so as to be suitable for real-time communication for conversational purposes such as IP videophone.

The transmission device in the present invention is
Encoder means for encoding media data to be transmitted;
A buffer for temporarily storing the encoded media data;
Memory means for measuring the number of bytes of media data accumulated in the buffer every predetermined time dT, and recording a history of the measured number of bytes (B1 to BN);
Buffer input rate calculation means for measuring a buffer input rate BRin to the buffer;
When the buffer accumulation fluctuation amount (B1-BN) increases, the buffer output rate BR of media data from the buffer is
Buffer output rate BR =
Buffer input rate BRin−buffer accumulation variation (B1−BN) / variation time interval (dT × N)
And coding rate determining means for inputting the determined buffer output rate BR to the encoder means as a coding rate.

Further, the transmission device in the present invention is
First encoder means for encoding first media data to be transmitted;
Second encoder means for encoding second media data to be transmitted;
A buffer for temporarily storing packets of encoded first media data and second media data;
Memory means for measuring the number of bytes of media data accumulated in the buffer every predetermined time dT, and recording a history of the measured number of bytes (B1 to BN);
Buffer input rate calculation means for measuring a buffer input rate BRin to the buffer;
When the buffer accumulation fluctuation amount (B1-BN) increases, the buffer output rate BR of media data from the buffer is
Buffer output rate BR =
Buffer input rate BRin−buffer accumulation variation (B1−BN) / variation time interval (dT × N)
And coding rate determining means for inputting the determined buffer output rate BR as a coding rate to the first encoder means.

  According to another embodiment of the transmission apparatus of the present invention, the first media data may be video data and the second media data may be audio data. Further, the first media data may be audio data, and the second media data may be video data.

According to another embodiment of the transmission apparatus of the present invention, the coding rate determining means sets the coding rate to coding rate ≦ buffer output rate BR.
It is also preferable to determine so that.

Furthermore, according to another embodiment of the transmission device of the present invention,
Buffer input rate BRin ≤ Buffer output rate BR
And a stable state determination means for detecting that the state where the buffer accumulation amount of the buffer is equal to or less than a predetermined threshold value is a stable state when a predetermined time has elapsed,
When the stable state determining means determines that it is in a stable state, it further comprises transmission line rate measuring means for measuring a transmission line rate at which media data should be transmitted,
The encoding rate determining means preferably determines the encoding rate using the transmission path rate measured by the transmission path rate measuring means as the buffer output rate BR.

  Furthermore, according to another embodiment of the transmission apparatus of the present invention, it is preferable to further include an averaging means for averaging with respect to the buffer accumulation fluctuation amount (B1-BN) / variation time interval (dT × N).

The transmission program in the present invention is:
Encoder means for encoding media data to be transmitted;
A buffer for temporarily storing the encoded media data;
Memory means for measuring the number of bytes of media data accumulated in the buffer every predetermined time dT, and recording a history of the measured number of bytes (B1 to BN);
Buffer input rate calculation means for measuring a buffer input rate BRin to the buffer;
When the buffer accumulation fluctuation amount (B1-BN) increases, the buffer output rate BR of media data from the buffer is
Buffer output rate BR =
Buffer input rate BRin−buffer accumulation variation (B1−BN) / variation time interval (dT × N)
And the computer functions as an encoding rate determining means for inputting the determined buffer output rate BR to the encoder means as an encoding rate.

The transmission program in the present invention is:
First encoder means for encoding first media data to be transmitted;
Second encoder means for encoding second media data to be transmitted;
A buffer for temporarily storing packets of encoded first media data and second media data;
Memory means for measuring the number of bytes of media data accumulated in the buffer every predetermined time dT, and recording a history of the measured number of bytes (B1 to BN);
Buffer input rate calculation means for measuring a buffer input rate BRin to the buffer;
When the buffer accumulation fluctuation amount (B1-BN) increases, the buffer output rate BR of media data from the buffer is
Buffer output rate BR =
Buffer input rate BRin−buffer accumulation variation (B1−BN) / variation time interval (dT × N)
And the computer functions as an encoding rate determining means for inputting the determined buffer output rate BR into the first encoder means.

  According to another embodiment of the transmission program of the present invention, the first media data may be video data and the second media data may be audio data. Further, the first media data may be audio data, and the second media data may be video data.

According to another embodiment of the transmission program of the present invention, the encoding rate determining means sets the encoding rate to encoding rate ≦ buffer output rate BR.
It is also preferable to make it function so that it may become.

Furthermore, according to another embodiment of the transmission program of the present invention,
Buffer input rate BRin ≤ Buffer output rate BR
And a stable state determination means for detecting that the state where the buffer accumulation amount of the buffer is equal to or less than a predetermined threshold value is a stable state when a predetermined time has elapsed,
A transmission path rate measuring means for measuring a transmission path rate at which the media data should be transmitted when the stable state determination means determines that the stable state has been reached;
The encoding rate determining means preferably functions to determine the encoding rate with the transmission path rate measured by the transmission path rate measuring means as the buffer output rate BR.

  Furthermore, according to another embodiment of the transmission program of the present invention, the transmission program may further include an averaging unit that averages the buffer accumulation fluctuation amount (B1-BN) / the fluctuation time interval (dT × N). preferable.

The transmission method in the present invention is:
An encoder that encodes media data to be transmitted, and a buffer that temporarily stores the encoded media data;
A first step of measuring the number of bytes of media data accumulated in the buffer every predetermined time dT and recording a history of the measured number of bytes (B1 to BN) in a memory;
A second step of measuring a buffer input rate BRin to the buffer;
When the buffer accumulation fluctuation amount (B1-BN) increases, the buffer output rate BR of media data from the buffer is
Buffer output rate BR =
Buffer input rate BRin−buffer accumulation variation (B1−BN) / variation time interval (dT × N)
And a third step of inputting the determined buffer output rate BR to the encoder as an encoding rate.

The transmission method in the present invention is as follows.
A first encoder that encodes first media data to be transmitted, a second encoder that encodes second media data to be transmitted, the encoded first media data and second media A buffer for temporarily storing each packet of data,
A first step of measuring the number of bytes of media data accumulated in the buffer every predetermined time dT and recording a history of the measured number of bytes (B1 to BN) in a memory;
A second step of measuring a buffer input rate BRin to the buffer;
When the buffer accumulation fluctuation amount (B1-BN) increases, the buffer output rate BR of media data from the buffer is
Buffer output rate BR =
Buffer input rate BRin−buffer accumulation variation (B1−BN) / variation time interval (dT × N)
And a third step of inputting the determined buffer output rate BR as a coding rate to the first encoder means.

  According to another embodiment of the transmission method of the present invention, the first media data may be video data and the second media data may be audio data. Further, the first media data may be audio data, and the second media data may be video data.

According to another embodiment of the transmission method of the present invention, the third step is the step of: encoding rate encoding rate ≦ buffer output rate BR
It is also preferable to determine so that.

Furthermore, according to another embodiment of the transmission method of the present invention,
Buffer input rate BRin ≤ Buffer output rate BR
And detecting that the state where the buffer accumulation amount of the buffer is equal to or less than a predetermined threshold value is a stable state when a predetermined time has elapsed, and
Measuring the transmission line rate at which the media data should be transmitted when the stable state determining means determines that the stable state is in a stable state;
In the third step, it is also preferable to determine the encoding rate with the transmission channel rate measured by the transmission channel rate measuring means as the buffer output rate BR.

  Furthermore, according to another embodiment of the transmission method of the present invention, it is preferable that the method further includes the step of averaging with respect to the buffer accumulation fluctuation amount (B1-BN) / fluctuation time interval (dT × N).

  According to the present invention, the buffer output rate is calculated based on the buffer input rate and the buffer accumulation fluctuation amount with time, and an encoding rate somewhat lower than the buffer output rate is input to the encoder. The information generation amount of the encoder can be controlled so that the buffer input rate does not exceed the buffer output rate (transmission path rate).

  In particular, in a transmission apparatus or the like having an audio encoder and a video encoder, by controlling the encoding rate of only the video encoder, the audio data can be maintained without changing its bit rate, and stable audio Conversation can be realized.

  In addition, when the buffer accumulation amount is equal to or less than a predetermined threshold and the state in which the coding rate determination based on the buffer accumulation amount does not fluctuate has passed for a certain period of time, the transmission line rate measurement unit determines the recovery of the transmission line rate, The information generation amount of the media encoder can be controlled according to the measured transmission line rate.

  Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.

Table 2 shows the data structure of the packet multiplexed frame used by the present invention.

  According to Table 2, the frame data structure according to the present invention has a UDP header followed by a 1-byte serial number portion, a 1-byte audio data size portion, and audio data of a length stored in the size portion, Video data. Therefore, an RTP header is not required as compared with Table 1. The serial number part is an unsigned integer and has a cyclic value of 0 to 255. Since the audio data size part is 1 byte, the size of the audio data is inevitably 0 to 255 bytes.

  FIG. 3 is a diagram illustrating a media division configuration of a packet multiplexed frame according to the present invention.

  In the upper part of FIG. 3, the video data is divided into equal sizes to form packet multiplexed frames. Since video data can always be divided at a predetermined number of bytes, mounting is easy. However, in order to divide into equal sizes without being conscious of the space between the frames, the frames may be divided across the frames. In this case, if a packet including video data straddling video frames is lost, it affects two (front and rear) frames.

  The lower part of FIG. 3 shows a video frame with a Resync marker inserted. The Resync marker is inserted at the macroblock boundary during encoding. By inserting the Resync markers between substantially equal sizes and including the spaces between the Resync markers as one video data in the multiplexed frame, the video data is not divided across the video frames. That is, the loss of one multiplexed frame does not affect two video frames.

  FIG. 4 is a functional configuration diagram of the transmission apparatus used according to the present invention. In the following, functions of the audio playback device will be described. Each function is implemented by a program, and each function is realized by the computer executing the program.

  According to FIG. 4, the transmission apparatus 1 includes a multiplexing unit 111 instead of the RTP header addition units 104 and 106 in FIG. 1. The multiplexing unit 111 includes a multiplexed packet generation unit 111, a memory 1112 that temporarily stores video data, a memory 1113 that temporarily stores audio data, a subtracter 1114 that generates a serial number, and a delay circuit 1115. And have. Here, the audio data memory 1112 can output the audio data size (difference between the write pointer and the read pointer) in 0 to 255 bytes.

  When the audio data is stored in the memory 113 and / or the video data is stored in the memory 112, the multiplex packet generation unit 111 is a switch of 4 inputs and 1 output, and includes a serial number, audio data size, audio data, and video data. By switching in order, one frame shown in Table 2 is formed by one round. The serial number is input to the subtracter 1114 through the delay circuit 115 and incremented by 1 (added by 1) to become a new serial number.

  According to the present invention, video data and audio data are extracted from the buffers 103 and 107 at the same timing, shaped, and multiplexed into one frame. As a result, the arrival delay between the video data and the audio data is constant, and the arrival fluctuation of the audio data can be avoided in the receiving apparatus.

  Also, when generating multiple packets, RTP required 12 bytes for each of the video data packet and the audio data packet, whereas according to the present invention, these different media data packets are integrated and correspond to RTP. Since the header of the part is 2 bytes, the header size can be reduced.

For example, if the packet transmission interval is 20 milliseconds,
(2 bytes + 8 bytes + 20 bytes) × 8 bits × 50 = 12 kbit / second overhead.

  That is, considering that the conventional technique is 32 kbit / sec, the present invention can reduce the overhead of 20 kbit / sec. The number of bits of overhead reduced here can be assigned to the encoding bit rate of video data or audio data, and the encoding quality can be improved.

  FIG. 5 is a frame transmission sequence diagram in the transmission apparatus of FIG. Hereinafter, functions of the receiving apparatus will be described, but each function can also be realized by a program.

  FIG. 5 shows a portion of S200 of FIG. If an attempt is made to transmit the audio data packet transmitted every 20 milliseconds, a video data packet having a shorter length than the video data packet of FIG. 2 can be transmitted. According to FIG. 5, a frame composed of 17-byte audio data and 80-byte video data is received by the receiving device in 15.8 milliseconds. That is, the reception end timing of the video data V1 (400 bytes) coincides with the reception end timing of the audio data A1 to A4 (17 bytes × 4 pieces).

For example, the AMR audio data size has the following number of bytes when the packet transmission interval is 20 milliseconds. Here, typical three modes are listed.
4.75 kbit / s mode: 14 bytes
* 6.70 kbit / s mode: 19 bytes
12.2 kbit / s mode: 33 bytes

The video data size is simply expressed as “bit rate × time”. When one packet is 20 ms apart, the number of bytes is as follows.
32kbit / s: 32kbit / 8bit x 20msec = 80byte

  When the AMR is in the 6.70 kbit / second mode, the multiplexed frame requires a total of 101 bytes including a serial number of 1 byte, audio data size of 1 byte, audio data of 19 bytes, and video data of 80 bytes.

  FIG. 6 is a basic functional configuration diagram of the transmission apparatus according to the present invention. This represents a functional configuration for one piece of media data, and the encoder may be a video encoder or an audio encoder. Hereinafter, functions of the transmission apparatus will be described. Each function is implemented by a program, and each function is realized by executing the program by a computer.

  6 includes an encoder unit 102 that encodes media data, a packet configuration unit 111 that configures the encoded data into packets, a buffer 103 that temporarily stores the encoded data, and a UDP. A UDP / IP protocol stack unit 105 which adds a / IP header and transmits the IP network. Further, N shift registers 122, which are memories, a subtractor 123, an averaging unit 124, a stability determination unit 125, a transmission path rate measurement unit 126, a timer 127, a clock 128, and a subtractor 130 are provided. A buffer input rate calculation unit 131 and a coding rate determination unit 132.

  The shift register 122 measures the number of bytes of the packet accumulated in the buffer 121 at a constant time interval dT, and stores the number of bytes in the register B1 as the number of bytes accumulated in the buffer at the current time. The time interval dT is supplied by the clock 128. For example, N registers (B1 to BN) are provided, and the number of bytes stored in each register is sequentially shifted in a FIFO (First In First Out) format.

Each variable is defined as follows.
B1: Number of bytes stored in buffer at current time BN: Number of bytes stored in buffer at past time (= current time-variable time interval (dT × N)) dT: Time interval for measuring byte count of buffer dT × N: Current time from past time Time interval BR: Buffer output rate (transmission path rate)
BRin: Buffer input rate (information generation speed)
CR: Coding rate input to the encoder

The buffer accumulation fluctuation amount dB is expressed as follows. This is performed by the subtractor 123.
dB = B1-BN Formula (1)

On the other hand, the difference between the buffer input rate BRin and the buffer output rate BR is reflected in the buffer accumulation fluctuation amount dB.
dB = (BRin−BR) × dT × N Formula (2)

From Expression (1) = Expression (2), the following relational expression is obtained.
B1-BN = (BRin-BR) * dT * N Formula (3)

From the expression (3), the following relational expression is obtained. According to the present invention, the buffer output rate BR can be determined from the buffer accumulation amount by the equation (4). This is performed by the coding rate determination unit 132.
BR = BRin− (B1−BN) / (dT × N) Formula (4)

Furthermore, the transition of the buffer amount is averaged to avoid the influence of local fluctuations. The averaging unit 124 accumulates a predetermined number of values of (B1−BN) / (dT × N), and calculates the average, thereby mitigating the actual fluctuation.
BR = BRin-average {(B1-BN) / (dT × N)} Formula (5)

Normally, if the buffer input rate BRin is not smaller than the buffer output rate BR, the buffer accumulation fluctuation amount will only increase. Therefore, in order to stabilize communication, it is necessary to satisfy the following expression.
BRin ≦ BR Formula (6)

  When determining the coding rate CR to be input to the encoder 102 from the BR estimated by the equation (5), for example, the coding rate used in operation so that the measured BRin satisfies the equation (6). Among these, CR that does not exceed BR is defined as CR (CR <≈BR). Assume that the video data coding rate CR can be changed, for example, 32 kbps → 28 kbps → 24 kbps → 4 kbps → 0 kbps. At this time, if BR = 25 kbps, CR is set to 24 kbps. That is, a settable value that is smaller than the determined BR and is actually used by the encoder is set. The determined coding rate CR is input to the encoder 102. The encoder 102 is controlled by the input code rate CR.

The stability determination unit 125 determines the timing for measuring the transmission line rate when the buffer input rate is kept low despite the recovery of the transmission line rate.
When the state of the expression (6) “buffer input rate BRin ≦ buffer output rate BR” and the buffer accumulation amount of the buffer is equal to or less than the predetermined threshold Th, Detect as there is. The predetermined threshold Th can be set to the number of bytes for one packet. For example, according to FIG. 5, 1 byte + 1 byte + 17 bytes + 80 bytes = 99 bytes (not including 28 bytes of the UDP / IP header).

  The transmission line rate measuring unit 126 actually measures the transmission line rate when the stability determination unit 125 determines that the state is in a stable state. As a specific method for measuring the transmission line rate, there is, for example, Japanese Patent Application No. 2003-080134 by the same applicant. In this method, a transmission line rate is determined based on a round trip delay time obtained by exchanging two types of report packets having different sizes between a transmission device and a reception device.

  FIG. 7 is a flowchart executed at every measurement time interval dT.

The processing in FIG. 7 is performed at regular time intervals dT, that is, whenever the buffer accumulation amount is measured, the register is shifted, and the buffer accumulation amount is stored in the shift register B1.
(S701) It is determined whether or not the buffer accumulation amount B1 at the current time accumulated in the register B1 is larger than the buffer accumulation amount BN at the past time accumulated in the register BN. If B1 is not larger than BN, the process is terminated.
(S702) If B1 is larger than BN, the equation (4) is calculated. Thereby, the buffer output rate BR is calculated.
(S703) Next, an encoding rate CR for the encoder is obtained based on the buffer output rate BR. Here, CR = BR−α.
(S704) The calculated coding rate CR is input to the encoder. The encoder is controlled according to the input coding rate CR.

  FIG. 8 is a flowchart for recovering the transmission line rate in the stability determination unit.

The process of FIG. 8 may be executed (every dT) following the process of FIG. 7 or may be executed at another timing.
(S801) It is determined whether or not the buffer input rate BRin is equal to or lower than the buffer output rate BR. If BRin is larger than BR, the current time now is substituted for variable t (S807), and the process is terminated.
(S802) If BRin is equal to or less than BR, it is determined whether or not the buffer storage amount of the buffer is smaller than a predetermined threshold Th. If the buffer accumulation amount is equal to or greater than the threshold Th, the current time now is substituted for the variable t (S807), and the process is terminated.
(S803) If the buffer accumulation amount is smaller than the threshold value Th, it is determined whether or not the time interval obtained by subtracting the past time t from the current time now is longer than the predetermined time interval TM. If it is shorter than TM, the process is terminated. The time interval TM is a transmission line rate recovery determination time, and is counted by the timer 127. For example, if the actual transmission line rate is about 80 kbps, the time interval TM is set to about 5 seconds.
(S804) If it is longer than the predetermined time interval TM, it is determined that it is in a stable state, and transmission line rate measurement processing is performed.
(S805) The transmission line rate TR measured by the transmission rate measurement process is substituted into the buffer output rate BR, and the coding rate CR (= BR-α) is determined.
(S806) Finally, the coding rate CR is input to the encoder. The encoder controls the coding rate based on the CR.

  FIG. 9 is a graph showing the time variation of the buffer accumulation amount. The vertical axis represents the buffer accumulation amount at the current time, and the horizontal axis represents time.

Assume that the relationship of the expression (6) BRin ≦ BR is satisfied at the beginning of transmission.
(S1) N = 5, the buffer accumulation amount BN = 2 at the past time, and the buffer accumulation amount B1 = 7 at the current time. This is because the buffer accumulation fluctuation amount dB (B1-BN) is increased due to the decrease in the transmission line rate. At this time, the buffer output rate BR is calculated by Equation (4). Then, the coding rate CR (= BR−α) is input to the encoder.

(S2) Next, when dT time elapses, the register is shifted, and the buffer accumulation amount at the current time is newly stored in the register B1. At this time, it is assumed that the buffer accumulation amount BN = 2 at the past time and the buffer accumulation amount B1 = 7 at the current time. At this time, the buffer output rate BR is calculated by Equation (4). Then, the coding rate CR (= BR−α) is input to the encoder.

(S3, S4) Next, when the dT time has elapsed, the register is shifted, and the buffer accumulation amount at the current time is newly stored in the register B1. At this time, the buffer accumulation amount BN = 4 at the past time and the buffer accumulation amount B1 = 5 at the current time are set. At this time, the buffer output rate BR is calculated by Equation (4). Then, the coding rate CR (= BR−α) is input to the encoder.

(S5) Next, when dT time elapses, the register is shifted, and the buffer accumulation amount at the current time is newly stored in the register B1. At this time, the buffer accumulation amount BN = 7 at the past time and the buffer accumulation amount B1 = 2 at the current time are set. Here, the buffer accumulation fluctuation amount dB (B1-BN) decreases. Therefore, a new buffer output rate is not calculated by equation (4).

(S6) Next, when dT time elapses, the register is shifted, and the buffer accumulation amount at the current time is newly stored in the register B1. At this time, the buffer accumulation amount BN = 7 at the past time and the buffer accumulation amount B1 = 2 at the current time are set. Here, the buffer accumulation fluctuation amount dB (B1-BN) decreases. Therefore, a new buffer output rate is not calculated by equation (4).

  After S4 in FIG. 3, the state of the expression (6) “buffer input rate BRin ≦ buffer output rate BR” continues with the newly determined buffer input rate BRin. During this time, the number of data bytes stored in the buffer is maintained at a small value. However, even when the transmission line rate is recovered, the buffer input rate BRin cannot be increased easily. This is because it cannot be determined from equation (4) that the transmission path rate is restored.

  After S4 in FIG. 3, is the stable state in which the state of the equation (6) “buffer input rate BRin ≦ buffer output rate BR” and the threshold value Th or less continues for a certain time TM or more? Determine whether or not. When the stable state continues, measurement of the transmission line rate is started. The actually measured transmission line rate is newly reflected as the buffer output rate BR. Accordingly, a buffer input rate BRin satisfying “buffer input rate BRin ≦ buffer output rate BR” is newly determined, and the buffer input rate BRin is input to the video encoder 102.

  The stability determination unit 125 transmits and receives report packets to measure the transmission path rate. When a report packet is transmitted, if a decrease in the transmission line rate according to the equation (4) is detected, the transmission buffer is affected by the report packet. In this case, the recovery state of the transmission line is determined to be insufficient, and the determination of the stable state is cancelled.

  FIG. 10 is a functional configuration diagram of the transmission apparatus according to the present invention.

  The transmission apparatus of FIG. 10 includes the functional configuration of FIG. 6 in the transmission apparatus of FIG. The buffer input rate calculation unit 131 acquires the buffer input rate from the video encoder unit 102 and the audio encoder unit 108. The sum of these input rates is calculated as the buffer input rate BRin. The buffer 121 temporarily stores multiplexed packets in which video data and audio data are configured in one frame by the multiplexing unit 111, and the shift register 122 stores the buffer storage amount of the buffer 121. The encoding rate determination unit 132 inputs the determined encoding rate CR to the video encoder unit 102. Therefore, the encoding rate of the video data is dynamically changed.

  However, the coding rate output by the coding rate determination unit 132 may be input to the speech encoder. Further, it may be input to both the video encoder and the audio encoder.

  For example, it is assumed that the coding rate determination unit 132 has to reduce the previous coding rate by 10,000 bits / second. According to the transmission apparatus of FIG. 10, the video encoder 102 controls the decrease in the coding rate of 10,000 bits / second. However, the encoding rate of the video encoder 102 can be decreased by 9000 bits / second and the encoding rate of the audio encoder 108 can be decreased by 1000 bits / second.

  Actually, a conversational voice encoder used by an IP videophone has a narrow bit rate setting range. For example, in AMR, it is 4.75 kbit / sec to 12.2 kbit / sec. On the other hand, the video encoder has a setting range of 32 kbit / sec. Therefore, even if the rate of the speech encoder is changed, the range is limited, and the rate of the speech encoder needs to be maintained in order to give priority to conversation. On the other hand, since the video encoder has a wide rate change range and does not depend on the bit rate as compared with the audio encoder, it is not necessary to transmit the bit rate information to the counterpart communication device. Therefore, according to the transmission apparatus of FIG. 10, the encoding rate of only the video encoder is changed.

  According to the above-described various embodiments of the transmission apparatus, transmission program, and transmission method for controlling the coding rate in the present invention, those skilled in the art can make various changes, modifications and omissions in the technical idea and scope of the present invention. It can be done easily. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

It is a function block diagram of the transmitter for packet multiplexing by a prior art. It is a sequence diagram of data packet transmission in the prior art. It is a media division | segmentation block diagram of the packet multiplexing frame in this invention. It is a function block diagram of the transmitter used by this invention. FIG. 5 is a sequence diagram of data packet transmission in the transmission device of FIG. 4. It is a basic functional block diagram of the transmission apparatus in this invention. It is a flowchart performed for every measurement time interval dT. It is a flowchart for the recovery of the transmission line rate in a stability determination part. It is a graph showing the time fluctuation | variation of the buffer accumulation amount. It is a function block diagram of the transmitter in this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Transmission apparatus 101 Format conversion part 102 Video encoder part, Encoder part 103 Video data buffer, buffer 104, 106 RTP header addition part 105 UDP / IP protocol stack part 107 Audio data buffer 108 Audio encoder part 109 A / D conversion part DESCRIPTION OF SYMBOLS 110 Amplifier 111 Packet structure part, Multiplexing part 121 Buffer 122 Shift register 123, 130 Subtractor 124 Averaging part 125 Stability determination part 126 Transmission path rate measurement part 127 Timer 128 Clock 129 Switch 131 Buffer input rate calculation part 132 Encoding Rate determining unit 3 IP network 4 Camera 5 Microphone

Claims (21)

Encoder means for encoding media data to be transmitted;
A buffer for temporarily storing the encoded media data;
Memory means for measuring the number of bytes of media data stored in the buffer every predetermined time dT, and recording a history of the measured number of bytes (B1 to BN);
Buffer input rate calculating means for measuring a buffer input rate BRin to the buffer;
When the buffer accumulation fluctuation amount (B1-BN) increases, the buffer output rate BR of media data from the buffer is
Buffer output rate BR =
Buffer input rate BRin−buffer accumulation variation (B1−BN) / variation time interval (dT × N)
And a coding rate determining unit that inputs the determined buffer output rate BR to the encoder unit as a coding rate. First encoder means for encoding first media data to be transmitted;
Second encoder means for encoding second media data to be transmitted;
A buffer for temporarily storing packets of the encoded first media data and second media data;
Memory means for measuring the number of bytes of media data stored in the buffer every predetermined time dT, and recording a history of the measured number of bytes (B1 to BN);
Buffer input rate calculating means for measuring a buffer input rate BRin to the buffer;
When the buffer accumulation fluctuation amount (B1-BN) increases, the buffer output rate BR of media data from the buffer is
Buffer output rate BR =
Buffer input rate BRin−buffer accumulation variation (B1−BN) / variation time interval (dT × N)
And a coding rate determining unit that inputs the determined buffer output rate BR to the first encoder unit as a coding rate.   The transmission device according to claim 2, wherein the first media data is video data, and the second media data is audio data.   The transmission apparatus according to claim 2, wherein the first media data is audio data, and the second media data is video data. The encoding rate determining means sets the encoding rate to encoding rate ≦ buffer output rate BR.
The transmission device according to claim 1, wherein the transmission device is determined so as to satisfy. Buffer input rate BRin ≤ Buffer output rate BR
And a stable state determination means for detecting that the state where the buffer accumulation amount of the buffer is equal to or less than a predetermined threshold value is a stable state when a predetermined time has elapsed,
When the stable state determining means determines that it is in the stable state, it further comprises transmission line rate measuring means for measuring a transmission line rate at which the media data is to be transmitted,
6. The coding rate determination unit according to claim 1, wherein the coding rate determination unit determines the coding rate using the transmission channel rate measured by the transmission channel rate measurement unit as the buffer output rate BR. The transmitter according to the item.   7. The transmission apparatus according to claim 1, further comprising averaging means for averaging the buffer accumulation fluctuation amount (B1-BN) / fluctuation time interval (dT × N). Encoder means for encoding media data to be transmitted;
A buffer for temporarily storing the encoded media data;
Memory means for measuring the number of bytes of media data stored in the buffer every predetermined time dT, and recording a history of the measured number of bytes (B1 to BN);
Buffer input rate calculating means for measuring a buffer input rate BRin to the buffer;
When the buffer accumulation fluctuation amount (B1-BN) increases, the buffer output rate BR of media data from the buffer is
Buffer output rate BR =
Buffer input rate BRin−buffer accumulation variation (B1−BN) / variation time interval (dT × N)
A transmission program for controlling a coding rate, characterized by causing a computer to function as a coding rate determining means for inputting the determined buffer output rate BR to the encoder means as a coding rate. First encoder means for encoding first media data to be transmitted;
Second encoder means for encoding second media data to be transmitted;
A buffer for temporarily storing packets of the encoded first media data and second media data;
Memory means for measuring the number of bytes of media data stored in the buffer every predetermined time dT, and recording a history of the measured number of bytes (B1 to BN);
Buffer input rate calculating means for measuring a buffer input rate BRin to the buffer;
When the buffer accumulation fluctuation amount (B1-BN) increases, the buffer output rate BR of media data from the buffer is
Buffer output rate BR =
Buffer input rate BRin−buffer accumulation variation (B1−BN) / variation time interval (dT × N)
Determined by the determined the buffer output rate BR for controlling the encoding rate for causing a computer to function as the coding rate determination unit as the coding rate is input to the first encoder means sends program.   The transmission program according to claim 9, wherein the first media data is video data, and the second media data is audio data.   The transmission program according to claim 9, wherein the first media data is audio data, and the second media data is video data. The encoding rate determining means sets the encoding rate to encoding rate ≦ buffer output rate BR.
The transmission program according to claim 8, wherein the computer is caused to function so as to be determined as follows. Buffer input rate BRin ≤ Buffer output rate BR
And a stable state determination means for detecting that the state where the buffer accumulation amount of the buffer is equal to or less than a predetermined threshold value is a stable state when a predetermined time has elapsed,
When the stable state determining means determines that the stable state has been reached, it further comprises transmission line rate measuring means for measuring a transmission line rate at which the media data is to be transmitted,
9. The coding rate determination means causes a computer to function so as to determine the coding rate with the transmission path rate measured by the transmission path rate measurement means as the buffer output rate BR. The transmission program according to any one of 1 to 12.   14. The apparatus according to claim 8, further comprising an averaging unit that averages the buffer accumulation fluctuation amount (B1−BN) / variation time interval (dT × N). Send program. An encoder that encodes media data to be transmitted, and a buffer that temporarily stores the encoded media data;
A first step of measuring the number of bytes of media data accumulated in the buffer every predetermined time dT, and recording a history of the measured number of bytes (B1 to BN) in a memory;
A second step of measuring a buffer input rate BRin to the buffer;
When the buffer accumulation fluctuation amount (B1-BN) increases, the buffer output rate BR of media data from the buffer is
Buffer output rate BR =
Buffer input rate BRin−buffer accumulation variation (B1−BN) / variation time interval (dT × N)
And a third step of inputting the determined buffer output rate BR to the encoder as an encoding rate, and a transmission method for controlling the encoding rate. A first encoder that encodes first media data to be transmitted; a second encoder that encodes second media data to be transmitted; and the encoded first media data and second A buffer for temporarily storing packets of media data,
A first step of measuring the number of bytes of media data accumulated in the buffer every predetermined time dT, and recording a history of the measured number of bytes (B1 to BN) in a memory;
A second step of measuring a buffer input rate BRin to the buffer;
When the buffer accumulation fluctuation amount (B1-BN) increases, the buffer output rate BR of media data from the buffer is
Buffer output rate BR =
Buffer input rate BRin−buffer accumulation variation (B1−BN) / variation time interval (dT × N)
And a third step of inputting the determined buffer output rate BR as an encoding rate to the first encoder means, and a transmission method for controlling the encoding rate.   The transmission method according to claim 16, wherein the first media data is video data, and the second media data is audio data.   The transmission method according to claim 16, wherein the first media data is audio data, and the second media data is video data. In the third step, the encoding rate is set to encoding rate ≦ buffer output rate BR.
The transmission method according to any one of claims 15 to 18, wherein the transmission method is determined so that Buffer input rate BRin ≤ Buffer output rate BR
And detecting that the state where the buffer accumulation amount of the buffer is equal to or less than a predetermined threshold is a stable state when a predetermined time has elapsed, and
Measuring the transmission line rate at which the media data should be transmitted when the stable state determining means determines that the stable state is present;
19. The encoding step according to any one of claims 15 to 18, wherein the third step determines the encoding rate using the transmission path rate measured by the transmission path rate measuring means as the buffer output rate BR. The transmission method described in.   The transmission method according to any one of claims 15 to 20, further comprising the step of averaging the buffer accumulation fluctuation amount (B1-BN) / fluctuation time interval (dT x N).

Images