TWI736180B - System and method for stabilizing network video delivery delay - Google Patents

Abstract

The invention discloses system and method for stabilizing network video delivery delay. A server sends frame packets and a multi-quality encoding list to a client through a network, a profiler explores a network status between the client and the server, and a communicator sends a request to the server to dynamically adjust frame quality of the frame packet based on the network status. When a maximum delay group of pictures index used to determine the delay between the client and the server exceeds a threshold, times of a delay counter is increased, and when the times of the delay counter exceeds a maximum delay times, the client performs a delay event to reduce the frame quality. When detection of the client at a cycle time has no delay event occurred treat as a nice judgement, times of a nice counter is increased whenever the nice judgement occurred, and when the times of the nice counter exceeds a maximum nice times, the client performs a nice event to increase the frame quality.

Description

System and method for stabilizing network video and audio delivery time delay

The present invention relates to a network video and audio delivery technology, in particular to a system and method for stabilizing network video and audio delivery delay.

With the rapid development of fifth-generation (5G) communications or more advanced networks, smart phones, smart watches, or tablets have become increasingly popular, making the client's requirements for stable network audio and video delivery delays higher and higher.

In an existing technology, a low-latency streaming access method in a video distribution scene is proposed, which includes a video live scene processing module to realize the low-latency streaming access function in a live video scene, and is processed by the video forwarding scene. The module realizes the low-latency streaming function in seconds under video forwarding, and corrects the presentation timestamp (PTS) and decoding timestamp (DTS), and then the video-on-demand scene processing module Realize the low-latency streaming function in seconds under video-on-demand. However, this prior art does not take into account the delivery efficiency of the group of pictures (GOP), the decoding performance of the client, and the large switching of the network status. Therefore, it is not easy to achieve the stability of the client’s ability to deliver online video and audio. Extension of the request.

Therefore, how to provide a novel and innovative technology for stabilizing the network audio and video delivery delay to meet the requirements of the client for stable network audio and video delivery delay has become a major research topic for those skilled in the art.

The present invention provides a novel and innovative system and method for stabilizing the network video and audio delivery delay, for example, it can meet the requirements of the client to stabilize the network video and audio delivery delay, or to stabilize the playback delay between the server and the client. Or to ensure that the user terminal is in a stable state of delay during the viewing of real-time video and audio.

The system for stabilizing the delay of network video and audio delivery in the present invention includes: a server, which delivers multiple frame packets and a multi-quality code list through the network; and a client, which receives frame packets and packets from the server through the network Multi-quality encoding list. At the same time, the client has: a prospecting device to survey the network status between the client and the server; a communicator, based on the network status between the client and the server surveyed by the prospecting device, to the server Send a request to dynamically adjust the frame quality of the frame package according to the multi-quality code list; a delay accumulator is used to determine the delay between the client and the server when the maximum delay video group index exceeds the threshold, Increase the number of times calculated by the delay accumulator so that when the number of times calculated by the delay accumulator exceeds the maximum number of delays, the client initiates or executes the delay event in the quality change event to reduce the frame quality of the frame packet; and A good accumulator means that when a delay event in the quality change event does not occur during the cycle time detection of the user terminal, it is deemed that a good judgment has occurred, and every time a good judgment occurs, the number of times the good accumulator is calculated is increased. When the number of times calculated by the good accumulator exceeds the maximum number of good times, the client initiates or executes the good things in the quality change event To improve the frame quality of the frame package.

The method for stabilizing the delay of network video and audio delivery in the present invention includes: a server sends multiple frame packets and a multi-quality code list through the network; and a client receives a frame packet and multi-quality code from the server through the network List; a surveyor surveys the network status between the client and the server; a communicator sends a multi-quality coding list to the server based on the network status between the client and the server surveyed by the surveyor To dynamically adjust the frame quality of the frame package; when the maximum delay video group index used to determine the delay between the client and the server exceeds the threshold, add a delay accumulator to calculate the number of times When the number of times calculated by the delay accumulator exceeds the maximum number of delays, the client initiates or executes the delay event in the quality change event to reduce the frame quality of the frame packet; and when the client detects the cycle time without the relevant quality When a delay event in an abnormal event occurs, it is considered that a good judgment has occurred, and every time a good judgment occurs, the number of times calculated by a good accumulator is increased, so that when the number of times calculated by the good accumulator exceeds the maximum number of good times, the client starts Or implement good events in the quality change events to improve the frame quality of the frame package.

In order to make the above-mentioned features and advantages of the present invention more comprehensible, embodiments are specifically described below in conjunction with the accompanying drawings. In the following description, the additional features and advantages of the present invention will be partially explained, and these features and advantages will be partly known from the description, or can be learned by practicing the present invention. It should be understood that both the foregoing general description and the following detailed description are only illustrative and explanatory, and are not intended to limit the scope of the present invention.

1: A system that stabilizes the delay of network audio and video delivery

10: Server

11: Source device

12: encoder

13: transmitter

14: Communicator

20: client

21: renderer

22: Decoder

23: receiver

24: Communicator

25: Prospector

26: Player

A, B, C, D, D1, D2, P: program

A1 to A3, B1 to B3, P1 to P6: steps

DC: Delay accumulator

DE1: The first delayed event tagger

DE2: The second delayed event tagger

DGIdx: Latency between the client and the server, and the index of the latency video group

F1 to F4: Numbered Frame Pack

FC: good accumulator

GIdx: Video group index

MDC: Maximum delay accumulator

MDGIdx: Maximum Delay Video Group Index

MFC: Maximum good accumulator

MNBC: The largest accumulator without slowing

MPC: Maximum pull accumulator

NBC: No slow accumulator

NGIdx: latest video group index

PC: pull accumulator

PGIdx: Index of the video group being played

Q: Frame quality

QMax: The highest frame quality

RGIdx: Index of video groups that have been completely received

S11 to S13, S21 to S23, S31 to S41: steps

Figure 1 is a schematic diagram of the system for stabilizing network video and audio delivery delay and its operation mode in the present invention;

Figure 2 is a detailed flow chart of the server establishment service in the procedure A of Figure 1 of the present invention;

Figure 3 is a detailed flow chart of the client connected to the service in Procedure B of Figure 1 of the present invention;

Figures 4 to 5 are schematic diagrams of the concept of actively pulling the lost packets in the program C of Figure 1 of the present invention. Figure 4 shows that multiple numbered frame packets are filled into the user terminal according to their numbers. In the buffer, Figure 5 shows the packet loss judgment and the lost frame of the frame that is actively pulled by the client;

Figure 6 is a flow chart of the method for dynamic quality adjustment in the present invention to illustrate the quality change caused by actively pulling packets (frame packets);

Figure 7 is a flowchart of the mechanism of resetting the pull accumulator in the present invention;

Figure 8 is a schematic diagram of the definition of the group of movies (GOP) index in the buffer of the client in the present invention;

Figure 9 is a schematic diagram of the time delay between the client and the server in the present invention;

Figures 10 and 11 are schematic diagrams of the first delay determination and the second delay determination of the present invention, respectively;

Fig. 12 is a schematic diagram of quality degradation related to delay events in the procedure D1 of Fig. 1 of the present invention;

Figure 13 is a schematic diagram of the improvement of quality related to good events in the procedure D2 of Figure 1 of the present invention;

Figure 14 is the details of adjusting quality for quality changes in procedure D of Figure 1 of the present invention Flow chart; and

Figure 15 is a schematic flow chart of the method for stabilizing network video and audio delivery delay in the present invention.

The following describes the implementation of the present invention with specific specific embodiments. Those familiar with this technology can understand the other advantages and effects of the present invention from the contents disclosed in this specification, and can also implement other different specific equivalent embodiments. Or apply.

The system and method for stabilizing the delay of network video and audio delivery in the present invention can support video and audio compression content of group of videos (GOP), and establish a delivery method based on User Datagram Protocol (UDP). A communication channel based on Transmission Control Protocol (TCP) or a survey channel based on Internet Control Message Protocol (ICMP) can be additionally established; moreover, the above-mentioned one can communicate with each other properly according to the present invention , In order to stabilize the playback delay between the server and the client.

The present invention can actively check the delivery delay situation, judge whether the network packet is lost or not, dynamically adjust the frame quality and the buffer content of the client to ensure that the client is in a stable delay during the viewing of real-time video and audio. state. At the same time, the frame quality adjustment of the present invention can be completed in the same playback stage, so that the player on the user side can interrupt the audio-visual service without additional reconnection. Therefore, the present invention can improve the quality of the real-time service of the audio-visual network.

The present invention can clear the buffer of the client terminal according to the network status, so that the player of the client terminal no longer presents the buffered frame with a long delay. In addition, the present invention can dynamically communicate with the server to adjust the frame quality (such as resolution, bit rate, (Bitrate), etc.) in order to achieve a stable video and audio delivery delay.

The present invention defines the delay of network video and audio delivery, which can be the time gap between the server obtaining a frame and sending the frame, and then presenting the frame at the user end. The demand for stable delay comes from the player of the user end during the playback process. In this case, the delay will gradually increase, which is often caused by the adaptive adjustment of the buffer space on the client side due to network quality jitter or delay variation (Jitter).

Fig. 1 is a schematic diagram of the system 1 for stabilizing network audio and video delivery delay and its operation mode in the present invention. As shown in the figure, the system 1 for stabilizing the delay of network audio and video delivery includes a server 10 and at least one (such as one or more) client 20. The server 10 can deliver multiple frame packets and a multi-quality encoding list via the network ( See Table 1), and the client 20 can receive the frame packet (video packet) and the multi-quality code list from the server 10 via the network. For example, the server 10 is a server (such as cloud server/web server/remote server), server host, computer, etc. that can provide services, and the client 20 is a user equipment (UE), personal Computers (such as desktop computers/notebooks), portable devices (such as smartphones/tablets), wearable devices (such as smart watches), etc. The network can be a wireless network, a wired network, a 5G network, a more advanced network (such as a 6G network), or various communication networks. However, the present invention is not limited to this.

The server 10 may include a source 11, an encoder 12, a transmitter 13 and a communicator 14. For example, the source device 11 may be a hardware video camera or camera, etc., or may be a software photographing program, a photographing program, or a frame (audio and video) data acquisition program, etc., to provide or obtain raw frame data. The encoder 12 can be composed of a hardware encoding chip (circuit), a software encoding program, or a combination thereof. The transmitter 13 can be composed of a hardware transmission chip (circuit), a software transmission program, or a combination thereof. The communicator 14 can be a communication chip (circuit) of hardware or a channel of software Consists of general programs or their combinations.

The user terminal 20 may include a presenter 21, a decoder 22, a receiver 23, a communicator 24, a prospector 25, and a player 26. For example, the presenter 21 may be a hardware display or a projector, etc., for presenting continuous images. The decoder 22 may be composed of a hardware decoding chip (circuit), a software decoding program, or a combination thereof. The receiver 23 can be composed of a receiving chip (circuit) of hardware, a receiving program of software, or a combination thereof. The communicator 24 may be composed of a hardware communication chip (circuit), a software communication program, or a combination thereof. The exploration device 25 can be formed by a hardware exploration chip (circuit), a software exploration program, or a combination thereof. The player 26 can be a hardware player, a player chip (circuit), a software player program, or any combination thereof.

The client terminal 20 may further include a pull accumulator (PullCounter) PC, a maximum pull accumulator (MaxPullCounter) MPC, a delay accumulator (DelayCounter) DC, a maximum delay accumulator (MaxDelayCounter) MDC, a Slow accumulator (NoBuffCounter) NBC, a maximum no buffer accumulator (MaxNoBufferCounter) MNBC, a good accumulator (FineCounter) FC, a maximum good accumulator (MaxFineCounter) MFC, a first delay (Delay) event tagger DE1 and A second delay (Delay) event tagger DE2.

For example, the pull accumulator PC can be a hardware pull accumulator chip (circuit), a software pull accumulator program, or a combination thereof. The maximum pull accumulator MPC can be composed of the maximum pull accumulation chip (circuit) of hardware, the maximum pull accumulation program of software, or a combination thereof. The delay accumulator DC can be composed of a hardware delay accumulation chip (circuit), a software delay accumulation program, or a combination thereof. The maximum delay accumulator MDC can be composed of a hardware maximum delay accumulation chip (circuit), a software maximum delay accumulation program, or a combination thereof. The non-slow accumulator NBC can be the non-slow accumulation of the hardware It is composed of chip (circuit), software's non-slow accumulation program or a combination thereof. The largest non-deferred accumulator MNBC is composed of the largest non-deferred accumulation chip (circuit) of hardware, the largest non-deferred accumulation program of software, or a combination thereof. The good accumulator FC can be a good accumulation chip (circuit) of hardware, a good accumulation program of software, or a combination thereof. The maximum good accumulator MFC can be composed of the maximum good accumulation chip (circuit) of hardware, the maximum good accumulation program of software, or a combination thereof. The first delayed event tagger DE1 or the second delayed event tagger DE2 can be a hardware delayed event tagging chip (circuit), a software delayed event tagging program, or a combination thereof.

After the source device 11 of the server 10 provides or obtains the original frame data, the encoder 12 of the server 10 can encode the frame data into a frame packet, and compress the same frame content of the frame data into multiple frame qualities, such as the lowest frame quality It is Q0 and requires the lowest network bandwidth for delivery, while the highest frame quality is QN and requires the highest network bandwidth for delivery. When the server 10 serves the client 20, the sender 13 of the server 10 will deliver the encoded frame packet from the encoder 12 to the receiver 23 of the client 20 via the network. The communicator 14 of the server 10 can communicate with the communicator 14 of the client 20 to allow the client 20 to dynamically adjust the frame quality of the frame packet or to actively pull the missing person of the plural frame packet from the server 10.

After the receiver 23 of the user end 20 receives or obtains the encoded frame packet from the transmitter 13 of the server 10, the decoder 22 of the user end 20 can decode the encoded frame packet to obtain the decoded frame packet (e.g. The continuous frame picture), and then the decoded frame packet (such as the continuous frame picture) is displayed by the renderer 21 of the user terminal 20. The prospector 25 of the client 20 can periodically survey the network status between the client 20 and the server 10 through the Internet Control Message Protocol (ICMP) to obtain the round trip time (RTT) of the network status. Or delay change (Jitter) information. The communicator 24 of the client 20 can cooperate with the communicator of the server 10 14. Through the transmission control protocol (TCP), based on the network status surveyed by the explorer 25, it sends a request to the server 10 to dynamically adjust the frame quality of the frame packet according to the multi-quality code list, and the communicator 24 can also actively send a request to the server 10 The server 10 pulls the missing person of the plural frame packets.

As shown in Figure 1, the operation mode of the system 1 for stabilizing the delay of network video and audio delivery mainly includes the following procedures A to D.

In the procedure A in Figure 1, the server 10 creates a service, including connecting to the source 11 to obtain raw frame data, and creating an encoder 12 to obtain multi-quality coded frames and provide (create) a multi-quality code list (Such as frame quality Q0-QN), and a communicator 14 is established to allow the client 20 to adjust the frame quality (video quality) or pull packets (frame packets). Then, wait for the client 20 to connect to the service of the server 10 (instant audiovisual service). After the client 20 connects to the service of the server 10, the server 10 establishes a sender 13 to deliver the frame packet to the client 20.

In the procedure B of Figure 1, various functional elements of the client 20 are established, including the establishment of a receiver 23 to obtain the encoded frame packet from the server 10, and the establishment of a decoder 22 to decode the encoded frame packet to obtain the Decode the frame packet (such as a continuous frame picture), create a renderer 21 to display the decoded frame packet (such as a continuous frame picture), and establish a communicator 24 to send a request to adjust the frame quality (video quality) of the frame packet, or Actively pull packets (frame packets) from the server 10, and establish a prospector 25 to obtain the client 20 and the server 10 through the Internet Control Message Protocol (ICMP) echo check (Ping) (ie ICMP-Ping) function The status of the network (such as RTT or Jitter).

After the client 20 is connected to the service of the server 10, the client 20 can perform the following procedures C to D. In the procedure C in Figure 1, the event of packet loss is identified, and the lost packet is actively pulled. In the procedure D of Figure 1, identify the quality change event and reset it actively Buffer or adjust the required frame quality (video and audio quality) of the client 20.

At the same time, the program D in Figure 1 may further include or determine the delay events described in the following program D1 to reduce the quality (frame quality of the frame packet) and the good events described in the program D2 to improve the quality (frame quality of the frame packet). In the procedure D1 in Figure 1, a delay event occurs, which reduces the frame quality (video quality) or increases the buffer size. In addition, in the procedure D2 in Figure 1, the buffering situation is full (good events), and the frame quality (video quality) is improved.

Figure 2 is a detailed flow chart of the service establishment of the server 10 in the procedure A of Figure 1 of the present invention, and the procedure A of Figure 1 may include the following steps A1 to A3 of Figure 2 below.

In step A1 in Figure 2, the server 10 obtains the content of the source device 11, such as the original video frame data (Video Frames).

In step A2 of Figure 2, the server 10 creates a multi-quality code list, such as Q0, Q1...QN, where Q0 is the lowest frame quality and the lowest required bandwidth, and QN is the highest frame quality and all The required bandwidth is the highest.

In step A3 of Figure 2, the server 10 selects a method (such as H.264) that has the characteristics of a group of movies (GOP) to encode the frame data to generate a frame packet, and performs only I frame (I-frame) data on the frame data. Frame) and P-Frame (P-Frame) compression, and select or set a low (lower) video group (GOP) number (such as 4) for frame data to speed up the efficiency of buffering into the client 20.

Fig. 3 is a detailed flow chart of the connection service of the client 20 in the procedure B in Fig. 1 of the present invention, and the procedure B in Fig. 1 may include the following steps B1 to B3 in Fig. 3.

In step B1 of FIG. 3, the communicator 24 of the client 20 is established, so that the communicator 24 obtains the multi-quality code list from the server 10.

In step B2 of Figure 3, the prospector 25 of the client 20 is established to obtain the network status between the client 20 and the server 10 in the cycle time by the prospector 25, for example, the round trip time shown in the following formula 1 ( RTT) and the delay variation (Jitter) information shown in Equation 2.

Formula 1: RTT=[new_RTT+RTT x(n-1)]/n. The RTT (round trip time) value of each cycle will be updated, new_RTT is the RTT (round trip time) value obtained by the new round of measurement, n is a positive integer and can be determined by yourself; when the value of n is larger, it means the past The value of is taken into consideration to stabilize the average of RTT (round trip time) values.

Formula 2: Jitter=[new_Jitter+Jitter x(m-1)]/m. The jitter value of each cycle will be updated, new_Jitter is the jitter value obtained by the new round of measurement, m is a positive integer and can be determined by oneself; when the value of m is larger, it means that the past The value of is taken into consideration in order to stabilize the average of the delay variation (Jitter).

In step B3 of Figure 3, the buffers between the receiver 23 and the player 26 of the client 20 are established. The receiver 23 obtains the video content such as frame quality Q0 (the lowest frame quality and the lowest required bandwidth) by default, and The player 26 plays the first complete video group (GOP) after the receiver 23 has received it.

Figures 4 to 5 are schematic diagrams of the concept of actively pulling the lost packets in the program C of Figure 1 of the present invention. Figure 4 shows the numbered frame packets F1 to F4 being filled in sequence according to the numbers. Entered into the buffer of the client 20, Figure 5 shows the packet loss judgment and the lost frame of the packet being actively pulled by the client 20.

As shown in Figure 4, the present invention uses User Datagram Protocol (UDP) to transmit frame packets to improve delivery efficiency. This method may have packet loss or disorder problems, which must be adaptively adjusted. The specific method can be to number each frame packet for the client 20 to obtain When each frame is packaged, multiple numbered frame packages F1 to F4 are filled into the buffer of the player 26 of the client 20 in sequence according to the number.

In the judgment method of packet loss, the client 20 can be restricted to judge within a limited time that can be calculated, and can be dynamically adjusted according to the network status. For example, after the receiver 23 of the client 20 receives a frame packet (time point marked as T1), and waits for the subsequent packet to exceed the specified time point, then this event can be judged as a packet loss event, where waiting The time limit for subsequent packets is T1+(RTT+Jitter)/2.

When the packet loss event occurs, as shown on the right side of Figure 5, the client 20 in Figure 1 actively pulls the specified frame packet to the server 10 to quickly fill the missing frame, and fill the specified frame packet Into the buffer of the user end 20.

Figure 6 is a flow chart of the method for dynamic quality adjustment in the present invention, to illustrate the quality change caused by actively pulling packets (frame packets). As shown in FIG. 6, the lost frame packet that is actively pulled by the client 20 shown in FIG. 1 can be regarded as a basis for poor network quality, so a method for dynamic quality adjustment can be designed. For example, the present invention can design the pull accumulator PC of the client 20 to pull packets or frame packets (see step S11 in Figure 6), if the pull accumulator PC is within the pull accumulator time (PullCounterSec; PCS) If the number of times that triggers the active pull of the missing multiple frame packets to the server 10 exceeds the maximum pull of the accumulator MPC (see step S12 in Figure 6), the client 20 will start or execute the program D1 in Figure 1 The delay event in the quality change event reduces the quality (frame quality of the frame package), otherwise nothing is done (see step S13 in Figure 6).

Figure 7 is a flowchart of the mechanism of resetting the pull accumulator PC in the present invention. As shown in the figure, the present invention can open a thread to periodically check the rule shown in step S21 in Figure 7: Is it (last pull time-current time)> pull accumulator time (PCS)? If yes, proceed to step S22 to reset the pull accumulator PC to zero (0); otherwise, if not, then return to step S21 to continue the detection.

In addition, the procedure D in Figure 1 above is mainly to determine the delayed events described in the procedure D1 to reduce the quality (frame quality of the frame packet) and the good events described in the procedure D2 to improve the quality (frame quality of the frame packet), so the following explains how Identify delayed events and good events.

FIG. 8 is a schematic diagram of the definition of the GOPIndex (GIdx) in the buffer of the client 20 in the present invention, and refer to FIG. 1 for description. As shown in Figure 8, the leftmost X is the frame being played, and the buffer of the client 20 is separated by a complete video group index (GIdx), for example, the size of the video group (GOP) is 4. PlayingGOPIndex (abbreviated as PGIdx) represents the index of the movie group being played; ReadyGOPIndex (abbreviated as RGIdx) represents the index of the movie group that has been completely received; and NewestGOPIndex (abbreviated as NGIdx) represents the latest movie group index, in the present invention As the latest video group index of the server 10.

FIG. 9 is a schematic diagram of the time delay between the user end 20 and the server end 10 in the present invention, and reference is made to FIG. 1 for description.

As shown in FIG. 9, when the player 26 of the client 20 establishes the buffer of the client 20 and starts playing the video related to the frame packet, a thread is created to calculate DiffGOPIndex=NGIdx-PGIdx in the cycle time through the thread. The meaning of DiffGOPIndex (abbreviated as DGIdx) represents the delay between the client 20 and the server 10 (delayed video group index), and is equal to the latest video group index (NGIdx) minus the video group index being played (PGIdx).

At the same time, the thread of the client 20 (player 26) is checked once in the cycle time, and the following three accumulators and their times are initially set to zero (0), including: (1) delay accumulator DC, maximum time Delay times (MaxDelayTimes; MDT); (2) No-buffer accumulator NBC, maximum no-buffer times (MaxNoBuffTimes; MNBT); and (3) good accumulator FC, maximum good times (MaxFineTimes; MFT).

FIG. 10 and FIG. 11 are schematic diagrams of the first delay determination and the second delay determination of the present invention, respectively, and refer to FIG. 1 for description. That is, when the user terminal 20 judges whether it is a delay event, there are two situations as follows: the first delay judgment and the second delay judgment.

First delay judgment: as shown in Figure 10, the client 20 sets a maximum delay video group index (MaxDelayGOPIndex; MDGIdx) and an initial maximum delay video group index (MaxDelayGOPIndexINIT; MDGIdxINIT), and the maximum delay The value of the video group index (MDGIdx) is the same as the value of the initial maximum delay video group index (MDGIdxINIT). The maximum delay video group index (MDGIdx) is used to determine whether the delay between the client 20 and the server 10 (delay video group index DGIdx) exceeds the threshold. If it exceeds the threshold, the client 20 will be The delay accumulator DC plus one (ie DC++), and plus one means "plus one or a multiple of one". When the number of times calculated by the delay accumulator DC of the client 20 exceeds the maximum delay times (MDT), it is regarded as a delay event, and the client 20 will start or execute the delay event in the related quality change event in the procedure D1 in Figure 1 And reduce the quality of the frame package (frame quality).

Second delay judgment: as shown in Figure 11, the client 20 detects whether the group index of the movie being played (PGIdx) and the group index of the movie that has been completely received (RGIdx) are equal? If both PGIdx and RGIdx are equal, the non-buffered accumulator NBC of the client 20 is incremented by one (ie, NBC++), and the increment by one means "plus one or a multiple of one". When the user end 20 When the number of times calculated by the non-slugging accumulator NBC exceeds the maximum number of non-sliding (MNBT), it is regarded as a delay event, and the client 20 will start or execute the delay event in the quality change event shown in the procedure D1 of Figure 1 to reduce the quality ( The frame quality of the frame package).

When the user terminal 20 does not occur during the cycle time detection of the first delay determination event and the second delay determination event, it can be considered that a good determination has occurred. Whenever a good judgment occurs, the good accumulator FC of the client 20 is incremented by one (ie, FC++), and the increment by one means "plus one or a multiple of one". When the number of times calculated by the good accumulator FC of the client 20 exceeds the maximum good number (MFT), it is a good event, and the user 20 will start the good event in the related quality change event in the procedure D2 of Figure 1 to increase The quality of the frame package (frame quality).

FIG. 12 is a schematic diagram of the quality reduction (frame quality of the frame packet) related to the delay event in the procedure D1 of FIG. 1 of the present invention, and refer to FIG. 1 for description. As shown in Figure 12, considering the frame quality currently selected by the user terminal 20 (see symbol C), there are two cases: Q>0 on the left and Q<=0 on the right.

When Q>0 (that is, the frame quality Q of the frame packet is greater than 0) as shown on the left side of Figure 12, on the communication channel between the user end 20 (see symbol C) and the server 10 (see symbol S), the user end 20. Initiate a request to reduce the quality (frame quality of the frame packet) to the server 10 through the communicator 24, and send it to the server 10 in a way that the current frame quality Q of the frame packet is reduced by one (ie Q--) (see symbol S) , And down by one means "decrease by one or a multiple of one." Subsequently, the server 10 receives the frame quality Q decreased by one (ie Q--), and will reply Yes or Complete (OK) to the client 20. Then, after the client 20 receives the reply from the server 10, it resets the delay accumulator DC, the non-delay accumulator NBC, and the good accumulator FC to zero (0). Then, on the streaming channel between the client 20 (see symbol C) and the server 10 (see symbol S), the server 10 transmits to the first of the client 20 The content of a frame can be an Instantaneous Decoding Refresh (IDR) frame (IDR-Frame), which will force the buffer of the client 20 to empty the buffered content (such as all buffered content), and the decoder 22 of the client 20 will use The new frame quality decodes the subsequent frame content.

When Q<=0 (that is, the frame quality Q of the frame package is less than or equal to 0) as shown on the right side of Figure 12, the quality selected by the user terminal 20 (see symbol C) is already the lowest frame quality and cannot continue to reduce the quality ( The frame quality of the frame packet), so the client 20 only adds one to the maximum delay video group index (MDGIdx) (ie MDGIdx++), and adds one to indicate "plus one or a multiple of one", and resets the delay accumulator DC , No slow accumulator NBC and good accumulator FC are zero (0).

FIG. 13 is a schematic diagram of improving the quality (frame quality of the frame package) related to good events in the procedure D2 in FIG. 1 of the present invention, and refer to FIG. 1 for description. As shown in Figure 13, considering that the currently selected frame of the user terminal 20 (see symbol C) is divided into two situations: Q<QMax shown on the left and Q>=QMax shown on the right.

When Q<QMax (that is, the frame quality Q of the frame packet is less than the highest frame quality QMax) as shown on the left side of Figure 13, on the communication channel between the client 20 (see symbol C) and the server 10 (see symbol S) , The client 20 initiates a request to improve the quality (frame quality of the frame packet) to the server 10 through the communicator 24, and sends it to the server 10 in the manner of the current frame quality Q plus one (ie Q++), and the plus one means "plus One or a multiple of one". Subsequently, after the server 10 receives the frame quality Q plus one (ie Q++), it will reply Yes or Complete (OK) to the client 20. Then, after the client 20 receives the reply from the server 10, it will set the maximum delay video group index (MDGIdx) to the initial maximum delay video group index (MDGIdxINIT), and reset the time Delay accumulator DC, non-delay accumulator NBC and good accumulator FC are all zero (0). Then, on the streaming channel between the client 20 (see symbol C) and the server 10 (see symbol S), the first frame content sent by the server 10 to the client 20 can be an IDR (Instant Decode Refresh) frame (IDR-Frame), will force the buffer of the client 20 to empty the buffer content (such as all buffered content), and the decoder 22 of the client 20 will use the new frame quality to decode the subsequent frame content.

When Q>=QMax (that is, the frame quality Q of the frame package is greater than or equal to the highest frame quality QMax) as shown on the right side of Figure 13, the quality selected by the client 20 (see symbol C) is already the highest frame quality and cannot continue Improve the quality (frame quality of the frame packet), so the client 20 only sets the maximum delay video group index (MDGIdx) to the initial maximum delay video group index (MDGIdxINIT), and resets the delay accumulator DC, no The slow accumulator NBC and the good accumulator FC are all zero (0).

For example, the method of stabilizing the delay of network video and audio delivery in the present invention includes: the server 10 shown in the procedure A in Figure 1 establishes the service, the client 20 shown in the procedure B connects to the service, and the packet loss and the packet loss shown in the procedure C Take the initiative to pull, the quality shown in procedure D changes and the quality is adjusted.

At the same time, the service establishment of the server 10 shown in the procedure A in Fig. 1 includes steps A1 to A3 in Fig. 2. In step A1 in Figure 2, the server 10 obtains the content of the source device 11 (original frame data), for example, live broadcast is used as an example, and through a full HD Full HD (such as 1080p-1920 x 1080 yuv format) camera , 30 FPS (Frame per Second; display frame per second) to obtain the original frame data, that is, to obtain a frame every 33ms (milliseconds). Then, in step A2 of Figure 2, the server 10 creates a multi-quality coding list (such as frame quality Q0-QN), as shown in Table 1 below.

Table 1: Multi-quality coding list, including frame quality Q0-QN (such as Q0-Q5).

In step A3 of Figure 2, the encoder 12 of the server 10 encodes the frame data in a way that has the characteristics of a group of movies (GOP) to generate a frame packet, but does not perform B-Frame on the frame data Compression, and select or set a low (lower) video group (GOP) number for frame data. For example, the server 10 selects H.264 to encode the video format of the frame data, but does not perform B-Frame compression, and selects the number of video groups (GOP) as 4.

Then, waiting for the client 20 shown in Figure 1 to connect to the server 10, and the client 20 to connect to the service includes the following steps B1 to B3 in Figure 3 below.

In step B1 of FIG. 3, the communicator 24 of the client 20 is established, so that the communicator 24 obtains the multi-quality code list from the server 10. The client 20 receives the multi-quality coding list (see Table 1) given by the server 10, and obtains a list of 6 frame qualities Q0-Q5.

In step B2 of Fig. 3, the prospector 25 of the client 20 is established to obtain the round trip time (RTT) and delay variation (Jitter) information of the network state from the prospector 25 in the cycle time. For example, the client 20 creates a thread, and executes the thread once in a cycle time (such as every 1 second), and sends the Internet Control Message Protocol (ICMP) echo check to the server 10 Check (Ping) (ie ICMP-Ping) command to obtain information about the round trip time (RTT) and delay variation (Jitter) of the network status, and set the value of n in the above formula 1 and the value of m in formula 2 to be 3. For example, the round-trip time (RTT) acquisition information shown in Table 2 below, and the delay variation (Jitter) acquisition information shown in Table 3 below.

Table 2: Obtained information of round trip time (RTT), RTT=[new_RTT+RTT x(n-1)]/n.

Table 3: Obtained information of delay variation (Jitter), Jitter=[new_Jitter+Jitter x(m-1)]/m).

From the test results of the 10th cycle in Table 2 and Table 3 above, the round-trip time (RTT) of settlement is about 100.17ms, and the jitter of settlement is about 16.16ms.

In step B3 of Figure 3, the buffers of the receiver 23 and the player 26 of the client 20 are created, and the video content such as frame quality Q0 (the lowest frame quality and the lowest required bandwidth) is obtained by default. For example, obtain the multi-quality code list in Table 1 through step B1 in Figure 3, and select the frame quality Q0 from the multi-quality code list. The defined resolution content is [360p, H.264, AAC, bit rate 1.5Mbps ] To decode the frame content in the buffer of the client 20.

In the procedure C of Figure 1, the client 20 can actively pull the packet (frame packet) when the packet (frame packet) is lost. For example, by obtaining information about the round trip time (RTT) and delay variation (Jitter) of the network status through step B2 in Figure 3, it can be determined whether the packet is lost after waiting for the next packet. Specifically, a certain time point T1 is assumed to be AM 11: 00: 01.105 (11 o'clock in the morning, 0 minutes, 1 second, 105 milliseconds), after receiving the previous packet (IDX=1), wait until the following time is exceeded (E.g. 58.165ms) If the next packet (IDX=2) is not received, it is a packet loss lose. For example, T1+(RTT+Jitter)/2=T1+(100.17+16.16)/2=T1+58.165ms=AM 11:00:01.105+59.165ms=AM 11:00:01.163.165.

If the receiver 23 of the user end 20 does not receive the next frame packet before AM 11: 00: 01.163165 (11 o'clock in the morning, 0 minutes, 1 second, 163.165 milliseconds), the packet is lost, and the user end 20 needs to proactively report to the server 10 Pull the packet (IDX=2). After the pull packet (frame packet) is sent, it will pull the accumulator PC once and check whether the pull accumulator PC exceeds the maximum pull accumulator MPC. If the maximum pull accumulator MPC is set to 5, and assume the situation in Table 4 below. For example, the frame packet number 10 shown in Table 4 will cause a poor network quality event, so the client 20 will perform the procedure D1 in Figure 1 to reduce the quality (frame quality of the frame packet).

Table 4: Poor network quality caused by packet loss.

The present invention considers the resetting of the pull accumulator PC of the user terminal 20. For example, the client 20 generates a thread after the first active pull event of a lost packet occurs. Assuming that the cycle time is 1 second, execute a scan of the reset pull accumulator PC through the thread, and set the pull accumulation Threshold value of the device time (PCS) (for example, 10 seconds). As shown in the method in Table 5 below, observe or determine whether to pull the accumulator PC (last pull time-current time)> pull the accumulator time (take the absolute value) for further resetting the pull accumulator PC . From Table 5, it can be observed or judged that the current time has reached [11:00:10], and the user terminal 20 judges that the interval between the last time of the pull accumulator and the current time has reached the pull accumulator time ( PCS) set the threshold (10 seconds), so reset the pull accumulator PC to zero (0).

Table 5: Reset the event of pulling the accumulator PC.

Figure 14 shows the product adjusted for quality change in procedure D of Figure 1 of the present invention The qualitative detailed flow chart includes the following steps S31 to S41, and is described with reference to Figure 1. As shown in FIG. 14, the client 20 starts a thread after receiving the frame content. Here, it is assumed that the cycle time (for example, every 100 ms) is confirmed by the thread execution.

In step S31 in Figure 14, the thread of the client 20 waits for the cycle time, and sets both the first delayed event marker DE1 and the second delayed event marker DE2 to be zero (0), that is, DE1=0 and DE2 =0. In addition, when DE1=1 (the first delayed event tagger is 1), it means that the first delayed event (for example, the latest frame of the server 10 is too far away) occurred; and when DE2=1 (the second delayed event tagger is 1), it means that the second delay event (such as the unbuffered time is too long) has occurred.

In step S32 in Fig. 14, from the buffer of the player 26 of the client 20, obtain the group index of the movie being played (PGIdx), the group index of the movie that has been completely received (RGIdx), and the latest movie group Index (NGIdx).

In step S33 in Figure 14, the time delay between the client 20 and the server 10 (delayed movie group index DGIdx) is calculated, that is, DGIdx=NGIdx-PGIDx is calculated, which represents the difference between the client 20 and the server 10 The time delay (delayed video group index DGIdx) is equal to the latest video group index (NGIdx) minus the currently playing video group index (PGIdx).

In step S34 to step S35 in Figure 14, it is determined whether the first delay event (DE1=1) occurs (for example, the latest frame of the server 10 is too far away), that is, DGIdx>=MDGIdx (delayed video group index) Greater than or equal to the maximum delay video group index)? For example, the maximum delay video group index (MDGIdx) is preset to 5. In the example shown in Table 6 below, when 1000 milliseconds (ms) is reached, DGIdx=NGIdx-PGIDx=9-4=5, that is, the delay between the client 20 and the server 10 (time delay video group Index DGIdx) is already equal to or greater At the maximum delay video group index (MDGIdx=5), the delay accumulator DC will be accumulated (DC++) once, and it means that the first delay event (DE1=1) has occurred.

Table 6: The first delay event (DE1=1), for example, an embodiment where the latest frame of the server 10 is too far away. PGIdx is the video group index being played, RGIdx is the video group index that has been completely received, NGIdx is the latest video group index, and DC is the delay accumulator.

In step S36 to step S37 in Figure 14, it is judged whether the second delay event (DE2=1) occurs (if no buffering time is too long), that is, PGIdx==RGIdx (the group index of the movie being played is equal to the Group index of videos received in full)? In the example shown in Table 7 below, in 500 to 1000 milliseconds (ms), a total of 6 occurrences of unbuffered time is too long, so the total NBC of the non-slow accumulator will be accumulated (NBC++) six times to become 6, and it means the first Two delayed events (DE1=2) occurs.

Table 7: The second delay event, such as an embodiment with no buffering time too long. PGIdx is the video group index that is currently playing, RGIdx is the video group index that has been completely received, NGIdx is the latest video group index, and NBC is the non-slow accumulator.

In step S38 to step S39 in Figure 14, it is determined whether a good event has occurred, that is, DE1==0 && DE2==0 (the first delayed event tagger is equal to 0 and the second delayed event tagger is equal to 0)? If it is, the good accumulator FC will be accumulated (FC++) once. In the embodiment shown in Table 8 below, when DE1==0 && DE2==0, the good accumulator FC will be accumulated once (that is, FC++), and it indicates that a good event has occurred.

Table 8: Good events. DE1 is the first delayed event tagger, DE2 is the second delayed event tagger, and FC is the good accumulator.

After completing steps S34 to S39 in Figure 14 in the cycle time, according to the delay accumulator DC, the non-slow accumulator NBC and the good accumulator FC of the client 20, it is possible to determine whether to adjust the frame quality, as shown in the following section Figure 14 shows steps S40 to S41.

In step S40 in Fig. 14, the client 20 determines whether the frame quality should be reduced? In the embodiment shown in Table 9 below, the delay accumulator DC presents a delay event that reduces the frame quality due to delay (Delay). Assume that the maximum delay accumulator MDC is 5.

Table 9: Frame quality degradation caused by delay. DC is a delay accumulator, NBC is a non-delay accumulator, and FC is a good accumulator.

In the embodiment shown in Table 10 below, the non-buffering accumulator NBC presents a delayed event that causes the frame quality to be reduced due to no buffering. It is assumed that the maximum non-buffering accumulator MNBC is 5.

Table 10: Frame quality degradation caused by no buffering. DC is a delay accumulator, NBC is a non-delay accumulator, and FC is a good accumulator.

The delay events caused by the above Table 9 and Table 10 are followed by the process D1 in Figure 1, and the client 20 and the server 10 actively communicate to perform actions to reduce the frame quality.

In step S41 in Figure 14, the client 20 determines whether it is time to improve the frame quality? As shown in the embodiment shown in Table 11 below, the good accumulator FC shows that the first delay event (DE1) or the second delay event (DE2) does not occur, and the player 26 is given to adjust the frame quality upward For good events, assume that the maximum good accumulator MFC is 5. The good events caused by Table 11 are followed by the process D2 in Fig. 1 through the active communication between the client 20 and the server 10 to perform actions to improve the frame quality.

Table 11: Good event judgment. DC is a delay accumulator, NBC is a non-delay accumulator, and FC is a good accumulator.

The above-mentioned procedure D1 in Figure 1 is a delay event that reduces the frame quality, and Figure 12 is a schematic diagram of reducing the frame quality. In the multi-quality coding list shown in Table 1, it is assumed that the frame quality Q0-QN is Q0-Q5, a total of 6 frame qualities, the maximum delay video group index (MDGIdx) and the initial maximum delay video group index (MDGIdxINIT) ) Are all 5, and are divided into two scenarios [Q<=0] and [Q>0].

As shown on the right side of Figure 12, if Q<=0 (that is, the frame quality Q is less than or equal to 0), assuming that the current frame quality Q is zero (0), the frame quality can no longer be reduced. The extended movie group index (MDGIdx) plus one (ie MDGIdx++), it becomes 6. The purpose of this is to reduce the number of retransmissions by the client 20 (see No. C) when the network quality continues to be poor. Set the maximum delay video group index (MDGIdx) back to the initial maximum delay video group index (MDGIdxINIT) and reset the delay accumulator DC until the network quality returns to normal. The slow accumulator NBC and the good accumulator FC are all zero (0).

As shown on the left side of Figure 12, if Q>0 (i.e. frame quality Q is greater than 0), assuming that the current frame quality Q is 3 (i.e. Q=3), then the client 20 (see number C) will send to the server 10 (See No. S) Request to reduce frame quality Q (ie Q--) is equal to 2 (ie Q=2); when the server 10 responds OK or complete (OK), then reset the delay accumulator DC, no slow accumulation NBC and good accumulator FC are all zero (0), and the streaming channel between client 20 (see symbol C) and server 10 (see symbol S) is in the specified video group index (GIdx) ) At the beginning, the decoder 22 uses the new (New) frame quality (ie Q=N=2) to decode the specified video group index (GIdx).

The above-mentioned procedure D2 in Fig. 1 is for the occurrence of a good event to improve the frame quality, and Fig. 13 is a schematic diagram of the improvement of the frame quality. In the multi-quality coding list shown in Table 1, it is assumed that the frame quality Q0-QN is Q0-Q5, a total of 6 frame qualities, the maximum delay video group index (MDGIdx) and the initial maximum delay video group index (MDGIdxINIT) ) Are both 5. At this time, it is divided into two scenarios, [Q>=QMax] and [Q<QMax], and set the highest frame quality QMax to 5.

As shown on the right side of Figure 13, if Q>=QMax (that is, the frame quality Q is greater than or equal to the highest frame quality QMax), assuming the current frame quality Q is 5 (equal to the highest frame quality QMax being 5), then the client is 20 (See No. C) Frame quality can no longer be improved. At this time, only the maximum delay video group index (MDGIdx) is set back to the initial maximum delay video group index (MDGIdxINIT), and the delay accumulator DC is reset. The slow accumulator NBC and the good accumulator FC are all zero (0).

As shown on the left side of Figure 13, if Q<QMax (that is, the frame quality Q is less than the highest When frame quality QMax), assuming that the current frame quality Q is 3, the client 10 (see number C) requests the server 10 (see number S) to increase the frame quality Q equals 4 (ie Q=N=4); Respond to the server 10 to be OK or complete (OK), then set the maximum delay video group index (MDGIdx) back to the initial maximum delay video group index (MDGIdxINIT), and reset the delay accumulator DC, no delay The accumulator NBC and the good accumulator FC are all zero (0), and the streaming channel between the client 20 (see symbol C) and the server 10 (see symbol S) is in the specified video group index ( At the beginning of GIdx), the decoder 22 uses the new (New) frame quality (ie Q=N=4) to decode the specified video group index (GIdx).

Figure 15 is a schematic flow chart of the method for stabilizing network audio and video delivery delays in the present invention, and mainly includes the following procedures A, the server 10 establishes a service, the procedure B the client 20 connects to the service, and the procedure C encloses the packet Loss and active pull, delayed or good event detection described in procedure P, quality change described in procedure D, and refer to Figs. 1 to 3 for description.

The establishment of the service by the server 10 described in the procedure A in Fig. 15 may include the following steps A1 to A3 (see Figs. 1 and 2).

In step A1 of Figure 15, the server 10 obtains the content (frame data) of the source 11, for example, obtains the original frame data through an executable solution (such as a camera, a network video stream outside the system, etc.).

In step A2 of Figure 15, the server 10 creates a multi-quality coding list (Q0-QN), for example, represented by frame quality Q0, Q1...QN, where Q0 is the lowest frame quality and the required bandwidth is the lowest, and QN is the highest frame quality and the highest required bandwidth.

In step A3 of Figure 15, the encoder 12 of the server 10 encodes the frame data in a way with the characteristics of a group of movies (GOP) to generate a frame packet, but for the frame The data is not compressed by B-Frame, and a low (lower) video group (GOP) number is selected or set for the frame data. For example, the server 10 selects a method (such as H.264) with the characteristics of a group of videos (GOP) to encode the frame data to generate a frame packet, and only performs I-Frame and P-frame data on the frame data. -Frame) compression, and select or set a low (lower) video group (GOP) number (such as 4) for frame data to speed up the efficiency of buffering into the client 20.

The connection of the client 20 to the service described in the procedure B of Fig. 15 may include the following steps B1 to B3 (see Figs. 1 and 3).

In step B1 of FIG. 15, the communicator 24 of the client 20 is established, so that the communicator 24 obtains the multi-quality code list from the server 10.

In step B2 in Figure 15, the prospector 25 of the client 20 is established to obtain the round trip time (RTT) and delay variation (Jitter) information of the network state from the prospector 25 in the cycle time.

In step B3 of FIG. 15, the buffer between the receiver 23 and the player 26 of the client 20 is established. For example, the receiver 23 presets to obtain the video content of the frame quality Q0 (the lowest frame quality and the lowest required bandwidth), and the player 26 does not receive until the receiver 23 has received the first complete video group (GOP). Play it.

The packet loss and active pulling of the program C in FIG. 15 may include the following steps C1 to C3.

In step C1 in Figure 15, the client 20 determines the packet loss and actively pulls the specified packet. For example, in the determination method of packet loss, the client 20 can be restricted to make a determination within a limited time that can be calculated, and it can be dynamically adjusted according to the network status. After the client 20 receives a frame packet (the time point is marked as T1), it waits for the time point of subsequent packets to exceed the specified time At the point in time, the event can be judged as a packet loss event, and the waiting time for subsequent packets is limited to T1+(RTT+Jitter)/2. When the packet loss event occurs, the client 20 can actively pull the specified frame packet from the server 10 to quickly fill the missing frame, and fill the specified frame packet into the buffer of the client 20.

In step C2 in Figure 15, the client 20 actively pulls packets (frame packets) to cause quality changes. For example, the method for dynamically adjusting the quality of the client 20 can design a pull accumulator PC to pull packets (frame packets), if the pull accumulator PC triggers the active pull of the missing packet within the pull accumulator time (PCS) If the number of times exceeds the maximum number of times that the accumulator MPC is pulled, the client initiates or executes the delay event in the quality change event to reduce the quality (frame quality of the frame packet) (see procedure D1 in Figure 15 or Figure 1) ).

In step C3 in Figure 15, the pull accumulator PC of the client 20 is reset. For example, the user terminal 20 can detect whether to pull the accumulator PC at (last pull time-current time)> pull accumulator time (PCS)? If it is, the client 20 resets the pull accumulator PC to zero (0); otherwise, if not, the client 20 can continue to detect.

The delayed or good event detection described in the procedure P in FIG. 15 may include the following steps P1 to P6.

In step P1 in Figure 15, the thread cycle (ie cycle time) of the client 20 is detected. For example, the thread of the client 20 performs the first delay event every cycle time (for example, the delay with the server 10 is too large or the latest frame of the server 10 is too far away), and the second delay event (for example, the buffer of the client 20 has no data) ) Of the judgment.

In step P2 of FIG. 15, the first delay event is detected, such as the delay between the client 20 and the server 10 is too large or the latest frame of the server 10 is too far away. For example, the most Does the video group index (GIdx) of the newly received server 10 packet and the video group index (GIdx) of the client 20 exceed the threshold? If yes, add one to the delay accumulator DC of the first delay event (ie, DC++), and mark that the first delay event is triggered in the cycle (cycle time).

In step P3 of FIG. 15, a second delay event is detected, such as a lack of buffered data of the client 20 or an excessively long non-buffering time. For example, it is detected whether the video group index (GIdx) being played by the client 20 is the same as the video group index (GIdx) prepared by the client 20? If yes, add one to the non-slow accumulator NBC of the second delay event (ie NBC++), and mark that the second delay event is triggered in the cycle (cycle time).

In step P4 in Figure 15, a good event is detected. For example, if neither the first delay event nor the second delay event is triggered, the good accumulator FC at the user end 20 is increased by one (ie, FC++).

In step P5 of Figure 15, check whether the delay accumulator DC related to the first delay event or the non-delay accumulator NBC related to the second delay event exceeds the threshold? If yes, skip to the process of reducing the quality (frame quality of the frame package) described in the process D1 of Fig. 15 (or Fig. 1).

In step P6 in Figure 15, check whether the good accumulator FC of the good event exceeds the threshold? If yes, skip to the process of improving the quality (frame quality of the frame package) described in the process D2 of Fig. 15 (or Fig. 1).

The quality change described in the procedure D in Fig. 15 may include the following procedures D1 to D2 (see Fig. 1).

In the procedure D1 in Figure 15, the frame quality is reduced. As shown on the left side of Figure 12, if the frame quality can be reduced, the client 20 communicates with the server 10 to reduce the frame quality Quality (Q--), and reset the delay accumulator DC, the non-delay accumulator NBC, and the good accumulator FC are all zero (0), and the decoder 22 of the client 20 adopts the specified frame quality (new Frame quality) decode the specified video group index (GIdx). Conversely, as shown on the right side of Figure 12, if the frame quality cannot be reduced, the client 20 will only increase the maximum delay video group index (MDGIdx) by one (ie MDGIdx++), and reset the delay accumulator DC without delay accumulation NBC and good accumulator FC are all zero (0).

In the procedure D2 of Figure 15, improve the frame quality. As shown on the left side of Figure 13, if the frame quality can be improved, the client 20 communicates with the server 10 to improve the frame quality (Q++), and the maximum delay video group index (MDGIdx) is set as the initial maximum delay Video group index (MDGIdxINIT), and reset the delay accumulator DC, non-delay accumulator NBC, and good accumulator FC to zero (0), and the decoder 22 of the client 20 adopts the specified frame quality (new The frame quality) decodes the specified video group index (GIdx). On the contrary, as shown on the right side of Figure 13, if the frame quality cannot be improved, only the maximum delay video group index (MDGIdx) is set to the initial maximum delay video group index (MDGIdxINIT), and the delay accumulator is reset DC, no slow accumulator NBC, and good accumulator FC are all zero (0).

In summary, the system and method for stabilizing network video and audio delivery delay in the present invention has at least the following features, advantages, or technical effects.

1. The present invention can meet the requirements of the client for stable network video delivery delay, or stabilize the playback delay between the server and the client, or ensure that the client is in a stable delay during the real-time video viewing process state.

2. The present invention can be constructed on a video and audio delivery service solution (such as RTSP) based on User Datagram Protocol (UDP), and can select a compression solution that can support Group of Videos (GOP) (such as H.264).

3. The present invention can construct a prospecting device (exploring channel) on the client side to actively detect the network status of the server from the perspective of the client side, such as the round trip time (RTT) and delay change (Jitter) related to the network status, as The basis for judging the loss of packets and then adjusting the video and audio compression resolution.

4. In order to meet the needs of low-latency audio and video services, the present invention can adopt a transmission mechanism based on User Datagram Protocol (UDP). Therefore, it needs to consider its characteristics and the possibility of packet loss or packet disorder, so the client is constructed to actively detect The packet loss mechanism can identify the event of network packet loss or poor network status within a set time.

5. The present invention establishes a communicator (communication channel) between the server and the client, so that the client can actively communicate with the server to dynamically adjust the frame quality, while ensuring that the client can be selected when the network status is poor or degraded Lower frame quality, and better or higher frame quality can be selected when the network status is better or improved.

6. Through the communicator (communication channel), the present invention allows the client to actively pull the required packet (frame packet) from the server immediately after the packet is lost, so as to reduce the buffer reread time after the delay occurs.

7. The present invention can actively clear the buffered content that is too long according to the network status, so as to reduce the time delay between the client and the server, and thereby improve the service quality of video and audio delivery.

8. When the frame quality is adjusted, the present invention gradually removes the old buffer content, and also eliminates the frame content with a longer delay, so that the demand for stable delay can be achieved without reconnecting to the server.

9. The present invention may be applied to, for example, the telecommunications industry, network service provision industry, etc., and may also be applied to, for example, network audio-visual service platforms, network delivery and maintenance service products, etc.

The above embodiments are only illustrative of the principles, features and effects of the present invention, and are not intended to limit the scope of implementation of the present invention. Anyone familiar with the art can comment on the above without departing from the spirit and scope of the present invention. Modifications and changes to the implementation form. Any equivalent changes and modifications made using the content disclosed in the present invention should still be covered by the scope of the patent application. Therefore, the protection scope of the present invention should be as listed in the scope of the patent application.

1: A system that stabilizes the delay of network audio and video delivery

10: Server

11: Source device

12: encoder

13: transmitter

14: Communicator

20: client

21: renderer

22: Decoder

23: receiver

24: Communicator

25: Prospector

26: Player

DC: Delay accumulator

DE1: The first delayed event tagger

DE2: The second delayed event tagger

FC: good accumulator

MDC: Maximum delay accumulator

MFC: Maximum good accumulator

MNBC: The largest accumulator without slowing

MPC: Maximum pull accumulator

NBC: No slow accumulator

PC: pull accumulator

A, B, C, D, D1, D2: program

Claims (19)

A system for stabilizing the delay of network video and audio delivery, including: a server, which transmits multiple frame packets and a multi-quality code list through the network; and a client, which receives the frame from the server through the network Packet and the multi-quality code list, and the client has: a prospector to survey the network state between the client and the server; a communicator based on the client and the server surveyed by the prospector The network status between the server sends a request to the server to dynamically adjust the frame quality of the frame packet according to the multi-quality code list; a delay accumulator is used to distinguish the client from the service When the maximum delay video group index of the delay between terminals exceeds the threshold, the number of times calculated by the delay accumulator is increased, so that when the number of times calculated by the delay accumulator exceeds the maximum number of delays, the The client initiates or executes the delay event in the quality change event to reduce the frame quality of the frame packet; and a good accumulator, when the client detects the cycle time and the delay event in the quality change event does not occur When the good judgment occurs, and every time the good judgment occurs, increase the number of times calculated by the good accumulator, so that when the number of times calculated by the good accumulator exceeds the maximum good times, the client initiates or executes the relevant A good event in the quality change event improves the frame quality of the frame packet, wherein when the frame quality of the frame packet is greater than zero, the client sends the service to the service through the communication channel between the client and the server. The end initiates to reduce the frame of the frame packet Quality request, and the server sends the real-time decoding refresh (IDR) frame to the client through the streaming channel between the client and the server to clear the buffered content of the client. For example, the system described in item 1 of the scope of patent application, wherein the server selects a group of movies (GOP) feature to encode the frame data to generate the frame packet, and performs I frame and P frame on the frame data Compression, and set a low number of video groups (GOP) for the frame data. For example, the system described in item 1 of the scope of patent application, wherein the server has: a source device that provides or obtains frame data; an encoder that encodes the frame data from the source device into the frame packet, And the same frame content of the frame data is compressed into multiple frame qualities; a transmitter, when the server serves the client, sends the encoded frame packet from the encoder to the client through the network ; And a communicator, which communicates with the communicator of the client to allow the client to dynamically adjust the frame quality of the frame packet, or to actively pull the missing person of the plural frame packet to the server. Such as the system described in item 1 of the scope of patent application, wherein the prospector of the client obtains the communication between the client and the server through the echo check (Ping) function of the Internet Control Message Protocol (ICMP) Network status, and the client executes a thread in the cycle time to send the Internet Control Message Protocol (ICMP) echo check (Ping) command to the server to obtain the round-trip time related to the network status ( RTT) and delay change (Jitter) information. For the system described in item 1 of the scope of patent application, the user end further has: a receiver, which receives the encoded frame packet from the server end; A decoder, which decodes the encoded frame packet from the receiver to obtain the decoded frame packet; a renderer, which displays the decoded frame packet from the decoder; and a player , Is to establish the buffer of the client and play the video related to the frame package. For example, in the system described in item 1 of the scope of patent application, the user end has a pull accumulator and a maximum pull accumulator. When the pull accumulator triggers the pull accumulator within the pull accumulator time, it will actively pull to the server. When the number of missing persons of the plural frame packets exceeds the maximum number of accumulator pulls, the client initiates or executes the delay event in the quality change event to reduce the frame quality of the frame packet. For example, the system described in item 6 of the scope of patent application, wherein the client generates a thread after the first active pull event of the lost packet occurs, so that the thread is executed once in the cycle time to reset the pull Take the scan of the accumulator, and reset the pull accumulator to zero when the client determines that the interval between the last pull time of the pull accumulator and the current time has reached the pull accumulator time . For the system described in item 1 of the scope of patent application, the client has a slow accumulator, which is added when the client detects that the group index of the video being played is equal to the index of the video group that has been completely received The number of times calculated by the non-slowed accumulator, and when the calculated number of the non-slowed accumulator exceeds the maximum number of non-slowed, the client initiates or executes the delay event in the quality change event to reduce the frame quality of the frame packet . For example, the system described in item 1 of the scope of patent application is to establish a delivery method based on the User Datagram Protocol (UDP), a communication channel based on the Transmission Control Protocol (TCP), or a communication channel based on the User Datagram Protocol (UDP) between the client and the server. Internet Control Message Protocol (ICMP) exploration channel. A method for stabilizing the delay of network audio and video delivery includes: a server sends a plurality of frame packets and a multi-quality code list through the network; a client receives the frame packet and the server from the server through the network A list of multi-quality codes; a surveyor surveys the network status between the client and the server; a communicator surveys the network status between the client and the server based on the surveyor, Send a request to the server to dynamically adjust the frame quality of the frame package based on the multi-quality code list; when the maximum delay video group index used to determine the delay between the client and the server exceeds the threshold Increase the number of times calculated by a delay accumulator so that when the number of times calculated by the delay accumulator exceeds the maximum number of delays, the client initiates or executes the delay event in the quality change event to reduce the frame packet The quality of the frame; when the delay event in the quality change event does not occur in the cycle time detection of the client, it is deemed that a good judgment has occurred, and whenever the good judgment occurs, the number of times calculated by a good accumulator is added , When the number of times calculated by the good accumulator exceeds the maximum number of good times, the client initiates or executes the good event in the quality change event to improve the frame quality of the frame packet; and when the frame quality of the frame packet When it is greater than zero, the client initiates a request to reduce the frame quality of the frame packet to the server through the communication channel between the client and the server, and the server sends a request to reduce the frame quality of the frame packet through the communication channel between the client and the server. The inter-streaming channel sends real-time decoding refresh (IDR) frames to the client to clear the buffered content of the client. For example, the method described in item 10 of the scope of patent application further includes that the server selects a method with the characteristics of a group of movies (GOP) to encode the frame data to generate the frame packet, and perform I frame and P on the frame data Frame compression, and set a low number of video groups (GOP) for the frame data. For example, the method described in item 10 of the scope of patent application further includes obtaining the network status between the client and the server through the echo check (Ping) function of the Internet Control Message Protocol (ICMP) by the explorer , And the client executes a thread in the cycle time to send the Internet Control Message Protocol (ICMP) echo check (Ping) command to the server to obtain the round trip time (RTT) and the state of the network Jitter information. For example, the method described in item 10 of the scope of the patent application further includes when a pull accumulator triggers the active pull of the complex frame packet to the server during the pull accumulator time when the number of missing persons exceeds a maximum pull accumulator When the number of devices is set, the client initiates or executes the delay event in the quality change event to reduce the frame quality of the frame package. For example, the method described in item 13 of the scope of patent application further includes that the client generates a thread after the first active pull event of the lost packet occurs, so that the thread is executed once in the cycle time to reset the The scan of the pull accumulator, and when the client determines that the interval between the last pull time of the pull accumulator and the current time has reached the pull accumulator time, reset the pull accumulator to zero. For example, the method described in item 10 of the scope of the patent application further includes adding a non-slow accumulator to calculate the number of times when the client detects that the group index of the video being played is equal to the index of the video group that has been completely received, and The number of calculations in the non-slow accumulator exceeds the maximum When there is no delay, the client initiates or executes the delay event in the quality change event to reduce the frame quality of the frame package. A method for stabilizing the delay of network audio and video delivery includes: a server sends a plurality of frame packets and a multi-quality code list through the network; a client receives the frame packet and the server from the server through the network A list of multi-quality codes; a surveyor surveys the network status between the client and the server; a communicator surveys the network status between the client and the server based on the surveyor, Send a request to the server to dynamically adjust the frame quality of the frame package based on the multi-quality code list; when the maximum delay video group index used to determine the delay between the client and the server exceeds the threshold Increase the number of times calculated by a delay accumulator so that when the number of times calculated by the delay accumulator exceeds the maximum number of delays, the client initiates or executes the delay event in the quality change event to reduce the frame packet The quality of the frame; when the delay event in the quality change event does not occur in the cycle time detection of the client, it is deemed that a good judgment has occurred, and whenever the good judgment occurs, the number of times calculated by a good accumulator is added , When the number of times calculated by the good accumulator exceeds the maximum number of good times, the client initiates or executes the good event in the quality change event to improve the frame quality of the frame packet; and when the frame quality of the frame packet When it is less than or equal to zero, the client adds the maximum delay video group index, and resets the delay accumulator, the non-delay accumulator and the good accumulator to zero. A method for stabilizing the delay of network video and audio delivery, including: A server sends a plurality of frame packets and a multi-quality code list via the network; a client receives the frame packet and the multi-quality code list from the server via the network; a prospector explores the client The network status with the server; based on the network status between the client and the server explored by the explorer, a communicator sends dynamics based on the multi-quality code list to the server A request to adjust the frame quality of the frame packet; when the maximum delay video group index used to determine the delay between the client and the server exceeds the threshold, increase the number of times calculated by a delay accumulator to When the number of times calculated by the delay accumulator exceeds the maximum number of delay times, the client initiates or executes the delay event in the quality change event to reduce the frame quality of the frame packet; when the client detects the cycle time When the delay event in the quality change event does not occur, it is considered that a good judgment has occurred, and whenever the good judgment occurs, the number of times calculated by a good accumulator is increased, so that the number of times calculated by the good accumulator exceeds the maximum When the number of times is good, the client initiates or executes the good event related to the quality change event to improve the frame quality of the frame package; and when the frame quality of the frame package is less than the highest frame quality, the client passes the user through The communication channel between the client and the server initiates a request to the server to reduce the frame quality of the frame packet, and the server sends a real-time request to the client through the streaming channel between the client and the server Decode and refresh (IDR) frames to clear the contents of the buffer on the client side. A method for stabilizing the delay of network video and audio delivery, including: a server sends multiple frame packets and a multi-quality code list through the network; A client receives the frame packet and the multi-quality code list from the server through the network; a surveyor surveys the network status between the client and the server; a communicator based on the survey The network status between the client and the server detected by the device sends a request to the server to dynamically adjust the frame quality of the frame packet according to the multi-quality code list; when it is used to distinguish between the client and the server When the maximum delay video group index of the delay between the servers exceeds the threshold value, the number of times calculated by the delay accumulator is increased, so that when the number of times calculated by the delay accumulator exceeds the maximum number of delays, The client initiates or executes the delay event in the quality change event to reduce the frame quality of the frame package; when the client detects the cycle time that the delay event in the quality change event does not occur, it is deemed to be good When the number of times calculated by a good accumulator exceeds the maximum number of good times, the client initiates or executes the quality change event when the number of times calculated by the good accumulator exceeds the Good event to improve the frame quality of the frame package; and when the frame quality of the frame package is greater than or equal to the highest frame quality, the client sets the maximum delay video group index to the initial maximum delay video group index , And reset the delay accumulator, no slow accumulator and good accumulator to zero. For example, the method described in item 10, item 16, item 17 or item 18 of the scope of patent application is to establish a delivery based on the User Datagram Protocol (UDP) between the client and the server Method, communication channel based on Transmission Control Protocol (TCP), or exploration channel based on Internet Control Message Protocol (ICMP).

Images