KR100924309B1 - Quality adaptive streaming method using temporal scalability and system thereof - Google Patents

Abstract

PURPOSE: A quality-adaptive multimedia data streaming method and a system thereof are provided to off the improved service quality to a user as improving the stability of a network. CONSTITUTION: A butter state determining module(130) calculates a presumptive value of the data amount buffered in a buffer(210) of a client terminal(200), and determines the buffer state of a client terminal at a specific time. A transfer rate calculating module(150) calculates the average transfer rate of TCP traffic through feedbag message information, and calculates the transfer rate of a multimedia data stream. A network state determining module(140) determines the network state through the feedbag message information. A quality-adaptive module(160) controls the number of activated multimedia data frames.

Description

Quality Adaptive Multimedia Data Streaming Method and System through Time Scalability and Buffer Status Judgment TECHNICAL TECHNOLOGY

The present invention relates to a method and system for quality adaptive multimedia data streaming through temporal scalability and buffer state determination. More particularly, the present invention relates to a TCP-friendly data rate based on the current network state, and to encoding based on the calculated rate. The present invention relates to a method and system for providing a seamless and smooth quality adaptive streaming service to a user by adjusting a data stream transmission rate and supplementing a quality adaptation technique by determining a client buffer state.

Today, video streaming technology over the Internet is being applied in various forms of Internet content such as broadcasting and education, and its use is increasing day by day. In particular, the demand for efficient video streaming technology on the Internet is gradually increasing due to the recent increase in the demand for VOD and the activation of UCC.

The main traffics of the existing Internet such as WWW and FTP use TCP as a transport layer protocol because of the importance of reliable transmission of data.However, unlike other communication technologies, multimedia traffics such as video streaming are more sensitive to delay than reliable delivery of data. It is real-time and sensitive to video quality, but allows some packet loss and operates based on data rate. Therefore, these multimedia applications have mainly used UDP as a transport protocol to avoid the additional delay caused by retransmission required for reliable data transmission. However, when UDP is used as a transport protocol without a congestion control mechanism, there is a problem of deepening network congestion and rapidly degrading the performance of competing TCP-based traffic due to the feature of occupying the available bandwidth of the network.

In order to improve the disadvantages of the conventional multimedia transmission protocol without the congestion control mechanism described above, a method of transmitting a multimedia stream that adjusts the transmission rate in a friendly manner with TCP has been studied. When TCP-based traffic and non-TCP-based traffic contend for bandwidth in the same network environment, TCP-Friendly can be referred to as adjusting the transmission rate to use less than the average bandwidth occupied by the competing TCP traffic. .

TCP-friendly data rate control in streaming video over the Internet has the advantage of improving network stability and improving protocol equity. Conventional rate control schemes include techniques such as RAP and SQRT, which perform congestion control mechanisms similar to TCP and at the same time reduce the rate change to suit video stream transmission. RAP uses a congestion control mechanism similar to TCP, but uses a rate control rather than an ACK-based window control to reduce the rate change. On the other hand, SQRT reduces the change of the transmission rate to be suitable for video stream transmission by changing the parameter value of TCP's Additive Increase and Multiplicative Decrease (AIMD) technique.

Padhye's proposed TCP-friendly rate control method using the TCP rate modeling formula adjusts the rate of video streams to TCP-friendly rate using the TCP average rate calculated by the following equation.

<Equation 1>

Where S is the segment size, p is the packet loss rate, t _RTT is the end-to-end delay, t _RTO is the retransmission timeout, and T is the average rate of TCP connections (traffic).

However, even if TCP-controlled data rate control, there is still a problem that network information cannot be known in advance in the Internet environment that provides the best service, and that a change in data rate can still occur. In addition, there is a problem that improving the stability of the network and equity between protocols does not lead to an improvement in the streaming service quality provided to the user. Therefore, in order to improve the stability of the network and the quality of the service provided to the user in video streaming, a method of adaptively adjusting the quality of the serviced video stream based on the network condition should be applied.

Meanwhile, conventional network state based quality adaptation techniques can be classified into several types. First, in order to transmit data having a data rate suitable for a network state, there is a method of decoding an already encoded video stream and then modifying and transmitting the quantization parameter again. This method has a problem in that the CPU load is large and the service for a large number of clients is difficult in performing decoding and encoding again. Another method is to prepare a video stream encoded in several versions, and select and transmit a video stream having an appropriate data rate according to a change in network available bandwidth.

In the case of using hierarchical encoding, the server adjusts the quality of the video stream by activating the next higher layer of the hierarchically encoded video stream or by removing the active upper layer.

For efficient video streaming in the Internet environment, TCP-friendly rate control schemes are limited. In order to overcome this limitation, a quality adaptation scheme has been applied based on a hierarchical encoding scheme for video streams serviced at TCP-friendly rates.

In reality, however, few encoders have such a hierarchical encoding function, and a practical hierarchical encoding method is not applied to a streaming application.

The present invention has been made to solve the above problems, and the present invention, unlike network quality-based adaptive multimedia data streaming service quality adaptation technique, the temporal scalability (ie, frame rate control) of the multimedia data stream It aims to provide seamless and smooth streaming service to users through quality adaptation techniques.

In addition, by calculating the TCP-friendly data rate using the network state information and determining the quality of the encoded video stream based on this, it improves the stability of the network and the affinity of TCP, and determines the buffer status of the client to determine the multimedia data stream. Its purpose is to provide a higher quality streaming service by gently adjusting the quality of service.

Also, it is not a quality adaptive video stream control method based on hierarchically encoded video streams, but quality adaptive streaming method through temporal scalability and buffer status judgment of compressed multimedia data streams generally introduced in MPEG compression standard. By providing

In addition, it is an object of the present invention to accurately determine the buffer state while avoiding additional overhead by eliminating a separate control message for determining the buffer state.

In order to achieve the above object, a quality adaptive multimedia data streaming server of the present invention includes an encoder for encoding multimedia data to be transmitted into a multimedia data frame having any one of at least two frame types, and a buffer of a client terminal. A buffer state determination module that calculates an estimated value of the amount of data buffered at &lt; RTI ID = 0.0 &gt; and &lt; / RTI &gt; A transmission rate calculation module for calculating a transmission rate of a multimedia data stream to be transmitted to a client terminal based on an average transmission rate of traffic, a network status determination module for determining a network state through feedback message information, and among the encoded multimedia data frames Activate some multimedia data frames to be transmitted to the client terminal, adjust the number of activated multimedia data frames according to the buffer state or the network state of the client terminal, and use the available multimedia data frames to adjust the available bandwidth for each frame type. It can be made by including a quality adaptation module to calculate.

Also, in the above-described configuration, the quality adaptation module reduces the number of activated multimedia data frames when the buffer status determination module determines that the buffer of the client terminal is underflowed as a result of determining the buffer status of the client terminal. When it is determined that the buffer of the client terminal is to be overflowed, the number of activated multimedia data frames may be increased by additionally activating a part of the encoded multimedia data frames.

In addition, in the above-described configuration, when the network state determination module determines that the network state is congested, the quality adaptation module transmits the transmission rate of the multimedia data frame activated for each frame type according to the available bandwidth calculated for each frame type. Can be controlled.

In addition, in the above-described configuration, when the network state determination module determines that the network state is stable, the quality adaptation module may increase the transmission rate of the multimedia data stream by a predetermined size to contend for bandwidth with contention traffic.

In addition, in the above-described configuration, when the network state determination module determines that the network state is stable, the quality adaptation module determines whether there is a margin of available bandwidth, and when there is a margin, a part of the encoded multimedia data frame is discarded. By further activating, the number of activated multimedia data frames can be increased.

In addition, in the above-described configuration, the size of the transmission rate of the increased multimedia data stream may be characterized in that it does not exceed the transmission rate increase of TCP traffic competing under the same network conditions.

In addition, in the above-described configuration, when there is a change in the number of activated multimedia data frames, the buffer state determination module may determine the buffer state of the client terminal again in the next period based on the changed number of multimedia data frames.

In addition, in the above-described configuration, the quality adaptation module calculates the number of group of pictures ( _GOP ) buffered in the buffer of the client terminal (Δ _GOP ), and respectively determines the number of GOPs buffered in the buffer of the client terminal (Δ _GOP ). By comparing the minimum threshold value Δ _MIN and the maximum threshold value Δ _MAX with, it is possible to determine whether the buffer is likely to underflow and overflow.

In addition, in the above-described configuration, the quality adaptation module may determine whether to adjust the number of activated multimedia data frames according to the conditions expressed in Equations 7 to 9 below.

<Equation 7>

,

,

<Equation 8>

,

,

<Equation 9>

In addition, in the above-described configuration, the buffer status determination module may determine the buffer status of the client terminal using the value calculated by Equation 3 below.

<Equation 3>

In addition, the quality adaptive multimedia data streaming method of the present invention comprises the steps of periodically receiving a feedback message including network state information from the client terminal, and determines the amount of data buffered in the buffer of the client terminal client at a specific time Determining the buffer state of the terminal, and determining the possibility of occurrence of the underflow (overflow) or overflow (Overflow) of the buffer, and as a result of the determination of the possibility of underflow or overflow is determined that the overflow or overflow will occur In this case, the method may include adjusting the number of activated multimedia data frames.

In addition, in the above configuration, after performing the step of adjusting the number of activated multimedia data frames, it may further comprise a network state determination step of determining whether the network state is congested or stable state using a feedback message. .

In addition, in the above-described configuration, when it is determined that the network state is stable, the transmission rate may be increased by a predetermined size for bandwidth competition with contention traffic.

In addition, in the above-described configuration, when it is determined that the network state is stable, it is determined whether there is a margin in the available bandwidth, and when the available bandwidth is available, the number of activated multimedia data frames can be increased.

In addition, in the above-described configuration, when it is determined that the network state is congested, the average transmission rate of TCP traffic is calculated using the feedback message information, and the transmission rate of the multimedia data stream is calculated and transmitted according to the average transmission rate of TCP traffic. For a multimedia data frame, an available bandwidth may be allocated for each frame type and the transmission rate for each frame type may be adjusted by using a data rate of the multimedia data stream.

In addition, in the above-described configuration, it is possible to reduce the number of activated multimedia data frames when it is determined that an underflow of the buffer occurs, and to increase the number of activated multimedia data frames when it is determined that an overflow of the buffer occurs. .

In addition, in the above-described configuration, the average transmission rate of the TCP traffic may be calculated using the packet loss rate calculated by Equation 10 below.

<Equation 10>

In addition, in the above-described configuration, when there is a change in the number of activated multimedia data frames, the method may further include determining a buffer state.

According to the present invention, the stability of the network is improved, and the QoS can be provided to the user through the network state adaptive quality change using temporal scalability, thereby providing a seamless smooth streaming service.

In addition, by calculating the TCP-friendly data rate using the network state information and determining the quality of the encoded video stream based on this, it improves the stability of the network and the affinity of TCP, and determines the buffer status of the client to determine the multimedia data stream. By smoothly adjusting the service quality of the service, it is possible to provide a higher quality streaming service.

In addition, quality adaptation through temporal scalability and buffer status determination using the compressed multimedia data stream itself, which is generally introduced in the MPEG compression standard, rather than a quality adaptive video stream control method based on hierarchically encoded video streams. There is an effect that can provide an enemy streaming method.

In addition, it is not necessary to receive a separate control message for determining the buffer state, so that an additional overhead does not occur and system resources can be used more efficiently, but the buffer state can be accurately determined.

1 illustrates the structure of a quality adaptive streaming server and a client terminal according to an embodiment of the present invention.

As illustrated in FIG. 1, the quality adaptive streaming system may include a streaming server 100 and a client terminal 200, and a communication network between the streaming server 100, such as the Internet, and the client terminal 200.

The streaming server 100 may include an encoder 110 for encoding multimedia data, a memory 120 for storing the encoded multimedia data, a buffer state determination module 130 for determining a state of the buffer 210 of the client terminal 200, The network state determination module 140 that determines whether the network state is congested, calculates an average transmission rate of TCP traffic through feedback message information periodically transmitted from a client terminal, and multimedia data to be equal to or less than the calculated average transmission rate of TCP traffic. The rate calculation module 150 for calculating the transmission rate of the stream, the quality adaptation module 160 for controlling the transmission rate by adjusting the active frame number of the encoded multimedia data according to the network state and the buffer state of the client terminal and allocating available bandwidth It can be made, including.

In addition, the client terminal 200 includes a buffer 210 for buffering the received multimedia data for the streaming service, a decoder 220 for decoding the encoded multimedia data, and an output unit 230 for outputting the decoded multimedia data. It can be done by.

The multimedia data stream to be serviced from the user is encoded to have a certain frame rate through a compression technique such as MPEG-based scheme and stored in the streaming server 100. According to the congestion control mechanism of the transport protocol, the streaming server calculates the TCP-friendly transmission rate based on the current network state, and the data frames that are active for the appropriate amount according to the TCP-friendly transmission rate are the client sessions (transmission sessions). 200).

Meanwhile, the client receives the data frame transmitted by the streaming server and buffers the data in buffers 210 classified by the type, and the decoder 220 decodes and outputs the buffered data to consume the finally received frame. do.

Types of encoded multimedia data streams are classified into three types: I, P, and B frames. A frame is a basic structural unit of an image data stream. Frames existing between two adjacent I frames constitute a group of pictures (GOPs), and these GOPs correspond to the number of P frames existing between two I frames. The parameter has the number of B frames existing between two P frames. Each frame type has a different data size.

 GOP permutation  I-B-B-P-B-B-P-B-B-P  Frame rate  25 fps  Encoding data rate  1.4 Mbps  Frame type  I frame  P frame  B frame  Frames per GOP  One  3  6  Frame size (bytes)  25K  10K  2.5K

Table 1 is for explaining the characteristics of the multimedia data stream by way of example. The characteristics of the multimedia data stream will be described with reference to Table 1 above. One GOP includes one I frame, three P frames, and six B frames. Here, three P frames are included between I frames, and two B frames are configured to include two P frames. One GOP consists of a total of 10 frames and has a frame rate of 25 fps, so 25 frames per second. Thus, it can be seen that one GOP has 10 frames and thus has a 2.5 GOP rate per second.

The data size of the frame is different for each frame type. If the size of one GOP is obtained, it can be seen that it is (25 + 30 + 15) K, that is, 70K bytes. Since the GOP rate per second is 2.5, calculating the encoding data rate shows that 70K * 2.5 is 175K (byte / sec) and has an encoding data rate of 175K * 8 = 1.4Mbps.

Next, term definitions necessary to describe a quality adaptive multimedia data stream transmission control method are as follows.

                             next

B _k ; Number of bytes in one K frame (size of K frames), where

Originally encoded data stream:

NF _k ; Number of K frames

The number of frames in one GOP; NF _GOP = NF _I + NF _P + NF _B

The number of bytes in one GOP; B _GOP = NF _I * B _I + NF _P * B _P + NF _B * B _B

Frame rate per second of encoded frames; FR _GOP

Adaptive data stream:

α, β, γ; Number of I, P, and B frames each active

The number of frames in one active GOP; NF _ACTIVE = α + β + γ

Number of bytes of one GOP activated; B _ACTIVE = α * B _I + β * B _P + γ * B _B

Frame rate per second of active frames; FR _ACTIVE

)Here, the original encoded data stream refers to a multimedia data stream encoded and immediately stored in the memory 120 of the streaming server 100, and the adaptive data stream is activated to be transmitted to the client terminal 200 by applying a quality adaptation technique. Means a data stream. Therefore, the number of active frames of the adaptive data stream is less than or equal to the number of frames of the original encoded data stream.

)

The mechanism for transmitting the multimedia data stream and performing congestion control includes TCP friendly rate control. The TCP friendly rate control may use a TCP friendly rate control method based on the RTP protocol.

The network state information is notified to the streaming server 100 through the Real-time Transport Control Protocol (RTCP), which is a control protocol periodically transmitted from the client terminal 200. The transmission rate calculation module 150 of the streaming server 100 may calculate a TCP-friendly transmission rate based on the current network state by using the intermittent delay (RTT) and packet loss rate that are periodically fed back. The TCP friendly transfer rate calculated by the rate calculation module 150 is transmitted to the quality adaptation module 160.

In order to provide a high quality streaming service, buffer status information of the client terminal 200 is required. The buffer status information may be transmitted to the streaming server 100 by the client terminal 200 using an additional control message. However, this method transmits information on the past buffer status and the problem that there is overhead due to the periodic transmission of the control message. There is a problem.

Accordingly, a method of determining the buffer status of the client terminal 200 may be used at the streaming server 100 through the buffer status determination module 130. Details of the buffer state determination will be described later.

The quality adaptation module 160 determines and activates the number of frames to be transmitted in one GOP based on the received TCP friendly transmission rate and the buffer status determination of the client terminal received from the buffer status determination module 130. Allocate appropriate bandwidth for dropped frames. The buffer status of the client is determined based on the transmission rate in the next period and bandwidth allocation information for each frame, and the buffer status determination method is as described above.

The quality adaptation module 160 operates when the data rate of the encoded data stream is greater than or less than the available bandwidth of the current network. The quality adaptation module 160 adjusts the quality and transmission rate of the streaming service while increasing or decreasing the number of active frames in the GOP to suit the TCP friendly transmission rate received from the rate calculation module 150.

2 illustrates bandwidth allocation for each frame type and data buffering state information at the client side in a streaming server included in a quality adaptive streaming system according to an embodiment of the present invention.

As shown in FIG. 2, the streaming server activates and transmits a frame to be transmitted for each frame type according to the calculated TCP friendly data rate and network state information. X (t) denotes a TCP-friendly data rate calculated at any time t, and the streaming server transmits I, P, and B frames constituting the multimedia data stream at a given data rate. If the network conditions are congested and the available bandwidth cannot accommodate the entire frame of the encoded data stream, then the quality adaptation module determines the number of active frames in one GOP based on the TCP friendly transfer rate. Actual bandwidth) Allocates bandwidth for each frame within X _GOP . Here, X _I , X _P , and X _B denote bandwidths allocated for I, P, and B frames, respectively, and the number of active frames and the allocation of available bandwidth are performed in one GOP unit.

Table 2 shows the equations for calculating the expected bandwidth of the encoded data stream and the available bandwidth of the adaptive data stream.

Encoded data stream Adaptive data stream

Expected bandwidth,

Available bandwidth,

Expected bandwidth in Table 2 EX _GOP refers to the bandwidth required to transmit the entire frame of the original encoded data stream. Actual bandwidth X _GOP refers to the actual bandwidth that should be allocated to the adaptive data stream applied with quality adaptation technique according to TCP friendly transmission rate.

For original encoded data streams, the expected bandwidth is allocated as the total sum EX _GOP of the bandwidth (EX _I , EX _P , EX _B ) allocated to each frame type if the current available bandwidth can transmit the entire frame of the encoded data stream. And the expected transfer rate can be calculated. For an adaptive data stream, the available bandwidth at any time t has a value less than the expected bandwidth and is determined by the number of active frames (α, β, γ) suitable for the calculated TCP friendly transfer rate. The total available bandwidth of the network at any time t is proportionally allocated according to the data size of the selected active frames, and the available bandwidth X _GOP allocated to one _GOP is the bandwidth allocated for each frame type (X _I , X _P ,). X _B )

Next, a buffer state determination method performed by the buffer state determination module will be described.

As shown in Fig. 2, active frames transmitted over network available bandwidth are accumulated in the client buffer and the total buffered data amount Y _T (t) at any time t is represented by Equation 1 below. Can be.

<Equation 1>

의 합으로 구해질 수 있다. 여기서,

는 버퍼에 저장된 데이터 스트림의 소비율(Consumption Rate)를 의미한다. 또한, 시점 (t-1)은 시점 t보다 1초전의 의미로 반드시 해석되는 것이 아니고, 바로 시점 t 바로 이 전에 버퍼상태를 판단한 시점으로도 해석될 수 있다. 즉, t와 t-1 사이의 시간 간격은 단위시간 간격을 어떻게 설정하느냐에 따라 달라질 수 있다.Total buffered data amount (Y _T (t)) at a point t as shown in Equation (1) while the amount of data buffered in the previous time _{(t-1) (Y T} (t-1)) and the unit time The difference between the amount of data sent from the network and the amount of data consumed by the decoder during the unit time

It can be found as the sum of. here,

Is the consumption rate of the data stream stored in the buffer. In addition, the time point t-1 is not necessarily interpreted as meaning one second before the time point t, but may also be interpreted as a time point where the buffer state is determined immediately before the time point t. That is, the time interval between t and t-1 may vary depending on how the unit time interval is set.

In addition, the data amount buffered for each I, P, B frame at any time t may be expressed as Equation 2 below.

<Equation 2>

Here, the left side of Equation 2 means the amount of data buffered for each I, P, B frame at an arbitrary time point t from the top, and the first term of the right side indicates each I, at time point t-1 from the top. The amount of data buffered for each P or B frame. In addition, the second term of the right side in parentheses indicates the amount of data transmitted for each I, P, and B frame from the time point t-1 to the point in time from the top, and the third term of the right side in parenthesis represents the time point t from the top in order. The amount of data consumed by the decoder for each I, P, B frame from -1 to time t.

However, since the values in parentheses of Equations 1 and 2 are not known in real time, Equation 3 below is used to calculate an estimated value of the amount of data buffered for each frame type.

<Equation 3>

Estimates of the buffered data amount calculated in Equation (3) are used to determine the buffer status. The number of GOPs (FR _ACTIVE / NF _ACTIVE ) transmitted during the process was substituted.

The number of buffered frames (Δ _I , Δ _P , Δ _B ) for each frame type and the number of GOPs buffered in the client buffer are calculated based on the amount of buffered data for each frame type calculated from Equation 3 below. It can be calculated as in Equation 4.

<Equation 4>

That is, the number of buffered GOPs may be determined by dividing the number of buffered frames for each frame type by α, β, and γ, which are currently activated frames in one GOP. This is because I, P, and B frames constitute one GOP, and each frame has a mutual dependency in one GOP.

On the other hand, the buffer of the client terminal is used in video streaming applications because it serves to mitigate jitter between data transmission and provide a seamless playback service to the user. If there is no buffered data in the buffer of the client terminal, it means that there is no data that can be reproduced by the decoder, and as a result, serious streaming quality degradation such as interruption of playback can be brought about. On the contrary, since the inflow rate of the data flowing into the buffer is higher than the consumption rate of the buffered data, the quality of the streaming service may be degraded even when the capacity of the buffer is exceeded. Therefore, such an overflow or overflow of the buffer must be prevented to improve streaming service quality.

Quality adaptation module may prevent the determination of the underflow, based on the number of buffered GOP _(GOP Δ) or buffer overflow been. When the number of GOPs transmitted through the network during the unit time is θ and the number of GOPs consumed by the decoder during the unit time is φ, the sum of the number of transmitted GOPs and the number of buffered GOPs (Δ _GOP + θ) is consumed. Keeping it larger than the number of GOPs allowed will prevent underflow of the buffer. In addition, when the number of buffered GOPs is smaller than the buffer size of the client terminal, overflow of the buffer can be prevented. So that within a range that does not significantly affect the quality of service can prevent underflow and overflow can set two threshold minimum threshold (Δ _MIN) and maximum threshold values _{(Δ MAX) (Minimum and Maximum} Threshold). This is summarized as follows.

                              next

1. Underflow prevention condition

(클라이언트 단말버퍼로 전송된 GOP수 + 버퍼링된 GOP 수)(Number of GOP consumed + Δ _MIN )

(Number of GOPs sent to client terminal buffer + number of buffered GOPs)

<Equation 5>

2. Overflow Prevention Conditions

Δ_MAX Buffered GOP Count

Δ _MAX

<Equation 6>

Here, φ and θ can be obtained as follows, respectively.

,

,

Next, a description will be given of a quality adaptation scheme according to network conditions. If the available bandwidth of the network is higher than the consumption rate of current active frames, the streaming server may consider a method of additionally activating and transmitting one frame to the client terminal. However, such a method cannot predict a change in the available bandwidth of the network, and thus there is a problem that a change in the quality of service may occur frequently.

Therefore, to make the quality change more frequent and smoother, we use the method of activating additional new frames only if the amount of available bandwidth is larger than the total consumption of adding one frame to the currently active frame. Can be. The type of newly added active frame depends on the currently active frames. For example, if all P frames are not activated in the GOP, the next additional frame is determined as the next P frame. If all P frames are already activated, the B frame is additionally activated. That is, the B frame is activated only after all P frames are activated first.

If the current available bandwidth is less than the total consumption rate of adding one frame, the current active frame number is maintained, and the remaining bandwidth is used to buffer the data of the currently active frame. The condition for adding a new active frame can be expressed by Equation 7 below.

<Equation 7>

Condition 1:

Here, the left side means a difference obtained by subtracting the available bandwidth required for transmitting the n-1th GOP from the available bandwidth required for transmitting the nth GOP. In other words, it means how much bandwidth difference (extra margin) exists when transmitting the current GOP compared to the previous GOP. The right side is the bandwidth required to send one I, B or P frame.

Therefore, Equation (7) means that one frame is activated and transmitted when the free bandwidth of the left side (the free bandwidth generated this time than before) is greater than or equal to the bandwidth required to send one I, P or B frame.

However, there is a problem in the quality adaptation technique with only condition 1 of Equation 7 without considering the client terminal buffer state. The surplus bandwidth remaining after transmitting the currently active frames is used for data transmission and buffering of active frames, because the buffer overflow may occur if the buffering time becomes long.

Therefore, if a buffer overflow is expected, such as condition 2 below, to prevent the overflow of the buffer, the streaming server prevents the buffer overflow by providing one additional frame to activate the buffer. Increase Increasing the frame rate increases the number of frames consumed per unit time. As a result, the occupancy of the client buffer is reduced, thereby preventing the overflow of the buffer.

<Equation 8>

Condition 2:

Conversely, if the total consumption rate of the current active frame is higher than the available bandwidth, the number of active frames should be reduced. However, it is desirable to reduce the number of active frames as slowly as possible to provide high quality service for as long as possible. Therefore, if the client terminal buffer underflow is expected under the condition 3 below, the streaming server reduces the quality of service by reducing the number of currently active frames. Since the number of active frames transmitted is reduced while maintaining the current rate, the number of GOPs transmitted during a unit time increases while the number of frames consumed decreases, thereby preventing underflow of the buffer. Although the quality of service is reduced due to the decrease in the number of active frames, it can cope with the network condition adaptively and can provide a seamless service by preventing more problems such as service interruption due to underflow.

<Equation 9>

Condition 3:

If the three conditions described above are not satisfied, the streaming server maintains the current number of active frames. The data rate is controlled by the TCP-friendly data rate calculated for each rate adjustment period, and the streaming server distributes the calculated available bandwidth evenly for each activated frame type as shown in Table 2, and transmits it to the client terminal.

Hereinafter, a quality adaptive multimedia data streaming method according to an embodiment of the present invention will be described.

3 illustrates a quality adaptive multimedia data streaming system according to an embodiment of the present invention.

The quality-adaptive multimedia data streaming method operates on the basis of TCP friendly data rate change and buffer state change information of the client terminal. The client terminal periodically notifies the streaming server side of the network status information such as inter-transmission delay (RTT), retransmission timeout (RTO), and packet loss rate through an RTCP message (feedback message). Calculate the available bandwidth, or TCP friendly rate. The minimum and maximum thresholds for underflow and overflow prediction of the client terminal buffer may be set to the number of GOPs consumed during a predetermined time interval (for example, for 1 second each) minus the minimum threshold from the maximum number of buffered GOPs.

4 illustrates a quality adaptive multimedia data streaming method according to an embodiment of the present invention.

As shown in FIG. 4, when the streaming server periodically receives an RTCP control message (feedback message) from the client terminal, the streaming server determines a buffer state of the client terminal (S410). Next, the streaming server checks the possibility of underflow or overflow of the buffer using the above-described conditions 2 and 3 (S420). Next, when underflow of the buffer is expected, the occurrence of underflow is prevented by reducing the number of active frames, and when overflow is expected, overflow is prevented by increasing the number of active frames (S430). Next, the network state is determined as a congested state or a stable state based on the network state information transmitted from the client (S440). If it is determined that the network state is stable, the streaming server increases the transmission rate by R _INC in order to contend for bandwidth with contention traffic (S450). The rate increase R _INC is preferably set within a range not exceeding the rate increase of TCP traffic competing under the same network conditions. In the network stable state, the condition 1 described above is additionally checked to increase the number of active frames if there is room in the available bandwidth (S460). If it is determined that the network condition is congested, the streaming server calculates the TCP friendly transmission rate through the RTCP control message information and the low band filtering technique (S455). In the network congestion state, the available bandwidth is allocated for each frame type and the transmission rate is adjusted according to the calculated TCP friendly transmission rate (S465).

Next, after completing the above steps, the streaming server re-determines the buffer state of the client terminal based on the updated frame rate for the accuracy of the client terminal buffer state determination in the next RTCP cycle (S470).

On the other hand, the TCP average rate calculation method according to another embodiment of the present invention can be obtained by the following equation in order to improve the problem of large changes in the conventional rate.

<Equation 10>

Here, P _n means the packet loss rate calculated after the n th RTCP cycle.

NP _i refers to the number of packets transmitted by the streaming server in the i th RTCP period. L _i means loss event in the i th RTCP cycle, L _i = 1 if there is a loss event in the i th RTCP cycle, and L _i = 0 if there is no loss event in the i th RTCP cycle. It means. If there is a packet loss event, at least one packet is lost within the corresponding RTCP period, and if there is no packet loss event, it means that none of the packets are lost within the RTCP period.

Therefore, the denominator of Equation 10 is a value obtained by dividing the number of packets transmitted by the streaming server from the first RTCP cycle to the nth RTCP cycle by the number of lost event occurrences accumulated from the first RTCP cycle to the nth RTCP cycle. it means.

Also look at the case of L _n, L _n value if the packet loss events in the n th RTCP interval is L _n = 1, _n the absence of packet loss events in the second period is the RTCP L _n = 0.

When calculating the average TCP rate according to Equation 10, since the change in the rate is not large but gradually changes, it is possible to stably change the rate of the data stream and improve the fairness with competing TCPs.

While various embodiments of the present invention have been described above with reference to the accompanying drawings, the above embodiments are not intended to limit the scope of the present invention and are not limited to the above-described embodiments, but are within the scope of the technical spirit of the present invention. It will be said that the scope of the present invention extends to the embodiment to be carried out in various modifications.

1 illustrates the structure of a quality adaptive streaming server and a client terminal according to an embodiment of the present invention.

2 illustrates bandwidth allocation for each frame type and data buffering state information at the client side in a streaming server included in a quality adaptive streaming system according to an embodiment of the present invention.

3 illustrates a quality adaptive multimedia data streaming system according to an embodiment of the present invention.

4 illustrates a quality adaptive multimedia data streaming method according to an embodiment of the present invention.

<Description of the symbols for the main parts of the drawings>

100. Streaming Server 110. Encoder

120. Memory 130. Buffer Status Judgment Module

140. Network status judgment module 150. Rate calculation module

160. Quality adaptation module 200. Client terminal

210. Buffer 220. Decoder

230. Output

Claims (18)

A streaming server for transmitting a multimedia data stream to a client terminal and periodically receiving a feedback message transmitted by the client terminal. An encoder for encoding multimedia data to be transmitted into a multimedia data frame having any one of at least two frame types; A buffer state determination module that calculates an estimated value of the amount of data buffered in the buffer of the client terminal, and determines the buffer state of the client terminal at a specific time point based on the calculated estimated value; A rate calculation module for calculating an average rate of TCP traffic based on the feedback message information and calculating a rate of multimedia data stream to be transmitted to the client terminal based on the average rate of TCP traffic; A network status determination module for determining a network status based on the feedback message information; And Activate some multimedia data frames to be transmitted to the client terminal among the encoded multimedia data frames, adjust the number of activated multimedia data frames according to the buffer state or the network state of the client terminal, and adjust the number of the activated multimedia data frames. Quality adaptive multimedia data streaming server comprising a quality adaptation module for calculating the available bandwidth for each frame type by using. The method of claim 1, wherein the quality adaptation module, If the buffer status determining module determines the buffer status of the client terminal to determine that the buffer of the client terminal is to be underflowed, the number of activated multimedia data frames is reduced, and the buffer of the client terminal overflows. And if it is determined to be overflowed, further activates some of the encoded multimedia data frames to increase the number of activated multimedia data frames. The method of claim 2, When the network state determination module determines that the network state is congested, the quality adaptation module controls the transmission rate of the activated multimedia data frame for each frame type according to the available bandwidth calculated for each frame type. Quality adaptive multimedia data streaming server, characterized in that. The method of claim 2, If the network state determination module determines that the network state is stable, the quality adaptation module increases the transmission rate of the multimedia data stream by a predetermined size in order to contend for bandwidth with the competing traffic. Ever multimedia data streaming server. The method of claim 4, wherein If the network state determination module determines that the network state is stable, the quality adaptation module determines whether there is a margin of available bandwidth, and if there is a margin, additionally activates some of the encoded multimedia data frames. A quality adaptive multimedia data streaming server, characterized by increasing the number of active multimedia data frames. The method of claim 5, wherein The size of the increased data rate of the multimedia data stream is a quality adaptive multimedia data streaming server, characterized in that does not exceed the rate increase of the TCP traffic competing under the same network conditions. The method of claim 5, wherein The buffer status determination module, if there is a change in the number of the active multimedia data frame, the quality adaptive multimedia, characterized in that to determine the buffer status of the client terminal in the next period based on the changed number of multimedia data frame Data streaming server. The method of claim 7, wherein The quality adaptation module calculates the number of group of pictures ( _GOP ) buffered in the buffer of the client terminal (Δ _GOP ), and the predetermined minimum threshold value of the number of GOPs buffered in the buffer of the client terminal (Δ _GOP ), respectively. (Δ _MIN ) and a maximum threshold (Δ _MAX ) to determine the possibility of underflow and overflow of the buffer. The method of claim 8, The quality adaptation module determines whether to adjust the number of activated multimedia data frames according to the conditions represented by Equations 7 to 9, where X ^nGOP is available bandwidth required for transmitting the ^{nth GOP} , X ^{n-1 GOP} Is the available bandwidth required to send the n-1th GOP,

,

,

Is the bandwidth required to send one I, B, P frame, where X _I , X _P , and X _B are the bandwidths allocated to the active I, P, B frames, respectively, and α, β, and γ are Δ _GOP is the number of GOPs buffered in the buffer of the client terminal, Δ _MIN and Δ _MAX are the minimum and maximum thresholds, respectively. Data streaming server. <Equation 7>

, <Equation 8>

, <Equation 9>

The method of claim 9, The buffer state determination module determines the buffer state of the client terminal by using the value calculated by Equation 3 below,

Is the data amount of i frame determined to be buffered at time t,

Is the data amount of i frame determined to be buffered at time t-1,

Is the available bandwidth of i frames, calculated at time t, α, β, γ are the number of active I, P, B frames, B _i is the number of bytes in one i frame, and NF _ACTIVE = α + β + γ is The number of frames in one GOP, FR _ACTIVE is a quality adaptive multimedia data streaming server, characterized in that the frame rate per second of active frames. <Equation 3>

In the multimedia data streaming method using a streaming server, Periodically receiving a feedback message including network state information from a client terminal; Determining a buffer state of the client terminal at a specific time by determining the amount of data buffered in the buffer of the client terminal, and determining a possibility of underflow or overflow of the buffer; And And adjusting the number of activated multimedia data frames when it is determined that underflow or overflow occurs as a result of the determination of the possibility of the occurrence of underflow or overflow. The method of claim 11, After performing the step of adjusting the number of the active multimedia data frame, the quality-adaptive multimedia characterized in that it further comprises a network status determination step of determining whether the network state is congested or stable state using the feedback message How to stream data. The method of claim 12, If the network state is determined to be stable, the quality adaptive multimedia data streaming method characterized in that for increasing the bandwidth for contention competition with contention traffic. The method of claim 13, If the network state is determined to be stable, the quality adaptive multimedia data streaming method characterized in that it is determined whether there is room in the available bandwidth to increase the number of active multimedia data frames when there is room in the available bandwidth. The method of claim 14, When the network state is determined to be a congestion state, the average transmission rate of TCP traffic is calculated using the feedback message information, the transmission rate of the multimedia data stream is calculated according to the average transmission rate of the TCP traffic, and the multimedia data frame to be transmitted Quality adaptive multimedia data streaming method characterized in that the allocation of available bandwidth for each frame type by using the transmission rate of the multimedia data stream and to adjust the transmission rate for each frame type. The method of claim 15, Quality adaptive, characterized in that to reduce the number of activated multimedia data frames when it is determined that the buffer overflow occurs, and to increase the number of activated multimedia data frames when it is determined that the buffer overflow occurs. How to stream multimedia data. The method of claim 16, The average transmission rate of the TCP traffic is calculated using the packet loss rate calculated by Equation 10, where P _n is the packet loss rate calculated after the nth RTCP cycle, and NP _i is the streaming server in the i th RTCP cycle. Number of Packets, L _i is L _i = 1 when there is a loss event in the i th RTCP cycle, and L _i = 0 when there is no loss event in the i th RTCP cycle Adaptive Multimedia Data Streaming Method. <Equation 10>

The method according to any one of claims 11 to 17, And re-determining the buffer status when there is a change in the number of activated multimedia data frames.

Images