WO2023024469A1

WO2023024469A1 - Video quality problem determination method and apparatus, storage medium, and program product

Info

Publication number: WO2023024469A1
Application number: PCT/CN2022/078279
Authority: WO
Inventors: 杨辉; 姜军庆; 徐斌斌; 高鹏; 庞宏超
Original assignee: 华为技术有限公司
Priority date: 2021-08-27
Filing date: 2022-02-28
Publication date: 2023-03-02
Also published as: CN115734043A

Abstract

The present application relates to a video quality problem determination method and apparatus, a storage medium, and a program product. The video quality problem determination method comprises: receiving protocol information on a video player and a network device; determining an abnormal frame quality parameter and an abnormal packet quality parameter according to the protocol information; and determining, according to the abnormal frame quality parameter and/or the abnormal packet quality parameter, a cause and a device resulting in abnormal video quality of the video player. According to the video quality problem determination method in the embodiments of the present application, the cause and device resulting in a video quality problem can be quickly and accurately determined.

Description

Video quality problem determination method, device, storage medium and program product

This application claims the priority of the Chinese patent application with the application number 202110993951.3 and the application name "Video Quality Problem Determining Method, Device, Storage Medium, and Program Product" submitted to the China Patent Office on August 27, 2021, the entire content of which is passed References are incorporated in this application.

technical field

The present application relates to the field of video quality detection, and in particular to a video quality problem determination method, device, storage medium and program product.

Background technique

Compared with 4G, 5G technology has the advantages of large bandwidth and low latency, so it can provide technical support for high-definition video live broadcast, real-time camera monitoring, drone shooting and other video return scenarios that require real-time transmission of video streaming media.

Video backhaul is a real-time business scenario based on streaming media. Taking high-definition video live broadcast as an example, video frames are obtained after video capture by cameras and other equipment, and video frames are encoded and compressed by encoders/backpack servers, etc., compressed video frames are encapsulated by streaming media layer protocols to generate live streams, live streams It is transmitted to the video server through the network equipment, and after decapsulation and decoding, the video player renders and displays the picture.

With the popularization of the application of video backhaul scenarios, there is a need for location analysis of video quality problems in this scenario. Among them, video quality problems include: freezing, blurred screen and other problems that may occur when users watch videos and lead to poor user experience. Among them, the above-mentioned problems such as freezing and blurring are caused by the abnormality of the video frame, and there may be multiple devices involved in the process of video frame collection, encoding, packaging, transmission, decapsulation, and decoding. In order to improve user experience, it is necessary to locate the device that causes the video quality problem and the specific reason why the device causes the video quality problem. Existing technologies are still unable to quickly and accurately complete the determination of the causes and equipment that lead to video quality problems.

Contents of the invention

In view of this, a method, device, storage medium, and program product for determining video quality problems are proposed. According to the method, device, storage medium, and program product for determining video quality problems in the embodiments of the present application, it is possible to quickly and accurately complete the video quality problem. Causes of quality problems and determination of equipment.

In the first aspect, the embodiments of the present application provide a method for determining video quality problems, the method comprising: receiving protocol information on the video player and the network device, and the network device is used to transmit video data to the video player , the protocol information indicates the transmission mode of the video frame, and the video frame includes a plurality of data packets; an abnormal frame quality parameter and an abnormal packet quality parameter are determined according to the protocol information, and the abnormal frame quality parameter indicates the The quality of the video frame to which the abnormal frame quality parameter belongs is abnormal, and the abnormal packet quality parameter indicates that the quality of the data packet to which the abnormal packet quality parameter belongs is abnormal; according to the abnormal frame quality parameter and/or the abnormal Determine the cause and device of the abnormal video quality at the video playback end.

According to the method for determining video quality problems of the present application, by receiving the protocol information sent by the video playback end and the protocol information sent by the network device used to transmit video data to the video playback end, the video frame and the content included in the video frame can be determined according to the protocol information. The transmission mode of multiple data packets can further determine abnormal frame quality parameters and abnormal packet quality parameters. Because the abnormal frame quality parameter indicates that the quality of the video frame to which it belongs is abnormal, and the abnormal packet quality parameter indicates that the quality of the data packet to which it belongs is abnormal, and the video quality is associated with the video frame and the data packet, so the abnormal frame quality parameter and the abnormal The packet quality parameter is associated with the abnormal video quality, so that according to the abnormal frame quality parameter and the abnormal packet quality parameter, the cause and equipment that cause the abnormal video quality of the video player can be determined. The video quality problem determination method of the present application jointly determines the cause and equipment that cause the video quality abnormality at the video playback end through abnormal frame quality parameters at the frame level and abnormal packet quality parameters at the packet level, which can reduce the cost of completing the determination of the cause and equipment once. Time cost, and improve the accuracy of determining the cause and equipment.

According to the method for determining the video quality problem of the present application, when only the payload header can be obtained on some devices, the protocol information can also be extracted according to the payload header, and the cause and device that cause the video quality abnormality at the video playback end can be accurately determined. Expand the applicability of methods for determining video quality issues.

According to the first aspect, in the first possible implementation of the method for determining video quality problems, the protocol information includes streaming media layer protocol information, network layer protocol information, and transport layer protocol information of data packets, and the stream The media layer protocol information includes the streaming media layer protocol information of the data packet on the video playback end, the network layer protocol information includes the network layer protocol information of the data packet on the network device and the video playback end, and the transport layer protocol information includes The transport layer protocol information of the data packet on the network device and on the video playback end, said determining abnormal frame quality parameters and abnormal packet quality parameters according to the protocol information includes: determining at least one according to the streaming media layer protocol information The video frame and each data packet corresponding to each frame in the at least one video frame; according to the streaming media layer protocol information and the network layer protocol information, determine the frame quality parameters of each frame in the at least one video frame; according to The network layer protocol information and the transport layer protocol information determine the packet quality parameters of each data packet corresponding to each frame in the at least one video frame; according to the frame quality parameters corresponding to each frame in the at least one video frame and The frame quality parameter threshold determines the abnormal frame quality parameter, and determines the abnormal packet quality parameter according to the packet quality parameter of each data packet corresponding to each frame in the at least one video frame and the packet quality parameter threshold.

In this way, the frame quality parameter and the packet quality parameter can be determined through the combination of different protocol information, so as to reflect the quality of the video frame and the quality of the data packets under the video frame. Whether the frame quality parameter and the packet quality parameter are abnormal can be determined by the frame quality parameter threshold and the packet quality parameter threshold respectively, making it more convenient to determine the abnormal frame quality parameter and the abnormal packet quality parameter.

According to the first possible implementation of the first aspect, in the second possible implementation of the method for determining video quality problems, according to the frame quality parameters corresponding to each frame in the at least one video frame and the frame A quality parameter threshold, determining the abnormal frame quality parameter, determining the abnormal packet quality parameter according to the packet quality parameter and the packet quality parameter threshold of each data packet corresponding to each frame in the at least one video frame, including: For each frame of the at least one video frame, among the frame quality parameters of the video frame, the frame quality parameter whose value is greater than the frame quality parameter threshold is determined as the abnormal frame quality parameter; for the at least one video frame For each frame, among the packet quality parameters of the data packets corresponding to the video frame, the packet quality parameter whose value is greater than the packet quality parameter threshold is determined as the abnormal packet quality parameter.

In this way, abnormal frame quality parameters and packet quality parameters can be determined. Abnormal parameters are determined by comparing parameter values with threshold values, and the determination of abnormal parameters can be completed accurately and quickly.

According to the first or second possible implementation of the first aspect, in the third possible implementation of the method for determining video quality problems, the streaming media layer protocol information of the data packet includes The frame time stamp of the video frame, the network layer protocol information of the data packet includes the packet time indicating the time of the data packet, the frame quality parameter includes a frame header jitter parameter and a frame tail packet jitter parameter, and the frame header packet The jitter parameter represents the delay jitter situation of the first data packet corresponding to the video frame, and the frame tail packet jitter parameter represents the delay jitter situation of the last data packet corresponding to the video frame, and the described stream media layer protocol information according to the and the network layer protocol information, determining the frame quality parameters of each frame in the at least one video frame includes: for each frame of the at least one video frame, determining the frame quality parameter according to the packet time of the data packet corresponding to the video frame The first packet arrival time and the last packet arrival time of the video frame, the first packet arrival time represents the arrival time of the first data packet corresponding to the video frame, and the tail packet arrival time represents the arrival of the last data packet corresponding to the video frame Time; according to the arrival time of the first packet and the arrival time of the tail packet, calculate the first packet interval and the tail packet interval of adjacent video frames respectively, and the first packet interval represents the arrival time difference of the first data packet corresponding to the adjacent video frame , the tail packet interval represents the arrival time difference of the last data packet corresponding to the adjacent video frame; according to the frame timestamp, calculate the frame interval of the adjacent video frame respectively; according to the first packet interval, frame interval, and previous The first frame packet jitter parameter of a video frame calculates the first frame packet jitter parameter of the current video frame; according to the tail packet interval, the frame interval, and the frame tail packet jitter parameter of the previous video frame, the frame tail of the current video frame is calculated Packet jitter parameters.

In this way, the first frame packet jitter parameter representing the delay jitter of the first data packet corresponding to the video frame and the frame end packet jitter representing the delay jitter of the last data packet corresponding to the video frame can be calculated parameter, while characterizing the quality of the video frame, it improves the closeness between the frame quality parameter of the video frame and the actual jitter of the video frame.

According to any one of the first to third possible implementations of the first aspect, in the fourth possible implementation of the method for determining video quality problems, the stream of data packets The media layer protocol information includes a stream media layer sequence number indicating the sequence of the data packet in the video frame to which it belongs, the network layer protocol information of the data packet includes a packet time indicating the time of the data packet, and the frame quality parameter includes a frame loss Packet parameter and frame out-of-sequence parameter, described frame packet loss parameter represents the loss situation of the data packet of video frame, and described frame out-of-sequence parameter represents the out-of-sequence situation of the data packet of video frame, and described according to described streaming media layer protocol Information and the network layer protocol information, determining the frame quality parameters of each frame in the at least one video frame includes: for each frame of the at least one video frame, according to the streaming media layer of the data packet corresponding to the video frame The serial number determines the frame packet loss parameter of the video frame; for each frame of the at least one video frame, sorts the data packets corresponding to the video frame according to the order of packet time of the data packets corresponding to the video frame , obtaining the sorting result corresponding to the video frame, and determining the frame out-of-sequence parameter of the video frame according to the sorting result and the streaming media layer sequence number of the data packet corresponding to the video frame.

In this way, the frame packet loss parameter representing the loss of data packets of the video frame and the frame disorder parameter representing the disorder of the data packets of the video frame can be calculated, which improves the quality of the video frame while characterizing the quality of the video frame. Flexibility in choice of frame quality parameters.

According to the fourth possible implementation manner of the first aspect, in the fifth possible implementation manner of the method for determining video quality problems, for each frame of the at least one video frame, according to the corresponding The streaming media layer sequence number of the data packet, determining the frame loss parameter of the video frame includes: for each frame of the at least one video frame, according to the streaming media layer sequence number of the data packet corresponding to the video frame, determining The maximum value of the streaming media layer sequence number of the data packet corresponding to this video frame, the minimum value and the number of streaming media layer sequence numbers, according to the maximum value and minimum value of the streaming media layer sequence number, determine the corresponding data of this video frame The theoretical value of the number of packets, according to the difference between the theoretical value of the number of data packets corresponding to the video frame and the number of streaming media layer serial numbers of the data packets corresponding to the video frame, determine the frame packet loss parameter of the video frame ; According to the sequence number of the streaming media layer of the data packet corresponding to the sorting result and the video frame, determine the frame disorder parameter of the video frame, including: the numerical value of the streaming media layer serial number of the current data packet is greater than that in the sorting result , the streaming media layer serial number of the data packet before the current data packet, according to the sorting serial number of the current data packet in the sorting result, the streaming media layer serial number of the current data packet, and the data packet corresponding to the video frame The minimum value of the serial number of the streaming media layer determines the frame disorder parameter corresponding to the video frame.

In this way, the frame packet loss parameter can reflect the information of whether the data packet in the video frame is lost or not, and when there is a packet loss, it can also reflect the information of the number of the packet loss; the frame disorder parameter can reflect the information in the video frame The information about whether the data packets are out of order can also reflect the information about the out-of-order depth of the out-of-order data packets when there is out-of-order, so that the accuracy of determining the quality of video frames according to the frame quality parameters can be improved.

According to any one of the first to fifth possible implementations of the first aspect, in the sixth possible implementation of the method for determining video quality problems, the at least one video frame The packet quality parameters of each data packet corresponding to each frame in the frame are determined according to the network layer protocol information and the transport layer protocol information of the data packet, and the packet quality parameters include: the packet representing the time interval between the current data packet and the previous data packet The interval parameter, the packet jitter parameter indicating the delay jitter between the current data packet and the previous data packet, the out-of-order parameter indicating the disorder of the data packet, the packet loss parameter indicating the packet loss, and the packet retransmission The retransmission parameter, the delay parameter indicating the delay of the data packet and the corresponding response message, the IP packet fragmentation parameter indicating whether there is IP packet fragmentation in the data packet, and whether the checksum of the data packet is One or more of the wrong checksum parameters occurred.

In this way, a variety of packet quality parameters used to represent the quality of the data packet can be obtained. By obtaining a variety of packet quality parameters, the quality of the data packet determined according to the packet quality parameters is closer to the actual data packet quality, which can improve the subsequent basis. Abnormal Packet Quality Parameters and Abnormal Frame Quality Parameters determine what is causing video quality issues and how accurate the device is.

According to the sixth possible implementation manner of the first aspect, in the seventh possible implementation manner of the method for determining video quality problems, the abnormal frame quality parameter and/or the abnormal packet quality Parameter determination causes the cause and equipment of abnormal video quality at the video playback end, including: for any video frame, when the frame quality parameter of the video frame and the packet quality parameter of the data packet corresponding to the video frame meet the first condition, determine The cause of the abnormal video quality at the video playback end is a video encoding problem, and it is determined that the device causing the abnormal video quality at the video playback end is a device used for video encoding; wherein, the first condition includes: the frame loss parameter of the video frame is abnormal And the packet loss parameter of the data packet corresponding to the video frame is normal, or the frame disorder parameter of the video frame is abnormal and the disorder parameter of the data packet corresponding to the video frame is normal.

Since the protocol information of the data packet is obtained at the network device and the video player, the frame quality parameter and the packet quality parameter can reflect the abnormal situation in the transmission process, and the frame packet loss parameter abnormality indicates that there is a data packet under the video frame being transmitted If the packet loss parameter of the data packet under the video frame is normal at this time, it means that no data packet is lost during transmission, so it can be determined that the data packet is lost before transmission, that is, during video encoding; frame disorder Abnormal parameters indicate that there are data packets under the video frame that are out of order during transmission. If the out-of-order parameters of the data packets under the video frame are normal at this time, it means that the data packets are not out of order during transmission, so it can be determined that the data packets are in the out-of-order state. Before transmission, that is, out of order during video encoding; based on the above reasons, the first condition is that the frame packet loss parameter of the video frame is abnormal and the packet loss parameter of the data packet corresponding to the video frame is normal, or the video frame The out-of-sequence parameter of the frame is abnormal and the out-of-order parameter of the data packet corresponding to the video frame is normal, which can accurately reflect the video encoding problem.

According to the sixth or seventh possible implementation manner of the first aspect, in the eighth possible implementation manner of the method for determining a video quality problem, according to the abnormal frame quality parameter and/or the The abnormal packet quality parameter determines the cause and equipment that cause the abnormal video quality of the video player, including: for any video frame, the frame quality parameter of the video frame and the packet quality parameter of the data packet corresponding to the video frame meet the second condition, determine that the cause of the abnormal video quality at the video player end is a terminal parameter problem, and determine that the device causing the abnormal video quality at the video player end is the device used for video playback; wherein, the second condition includes: the frame of the video frame Any parameter in the first packet jitter parameter and the frame tail packet jitter parameter is abnormal, and any parameter in the IP packet fragmentation parameter and the checksum error parameter of the data packet corresponding to the video frame is abnormal.

Since the protocol information of the data packet is obtained at the network device and the video player, the frame quality parameter and the packet quality parameter can reflect the abnormal situation in the transmission process, and the abnormality of the first frame packet jitter parameter/frame end packet jitter parameter indicates that the video The delay jitter of the first/last data packet in the frame is too large during transmission. If the IP packet fragmentation parameter of the data packet in the video frame is abnormal at this time, it means that the data packet has experienced fragmentation during transmission. , so it can be determined that the parameter configuration of the video player fails to meet the requirements of data packet transmission without fragmentation; if the checksum error parameter of the data packet under the video frame is abnormal at this time, it means that the data packet failed to The verification with the video playback end is successfully completed, so it can be determined that the parameter configuration of the video playback end fails to meet the requirements for successful verification with the data packet; based on the above reasons, the second condition, that is, the frame header jitter parameters of the video frame and the frame Any parameter in the tail packet jitter parameter is abnormal and any parameter in the IP packet fragmentation parameter and the checksum error parameter of the data packet corresponding to the video frame is abnormal, which can accurately reflect the terminal parameter problem.

According to any possible implementation manner of the sixth to eighth possible implementation manners of the first aspect, in the ninth possible implementation manner of the method for determining video quality problems, the The frame quality parameter of the video frame and/or the packet quality parameter of the abnormality determine the cause and equipment that cause the abnormal video quality of the video player, including: for any video frame, the frame quality parameter of the video frame and the corresponding video frame When the packet quality parameter of the data packet satisfies the third condition, it is determined that the cause of the abnormal video quality at the video playback end is a network transmission problem; wherein, the third condition includes: the frame packet loss parameter of the video frame is abnormal and the video frame The packet loss parameter of the data packet corresponding to the frame is abnormal, or the frame out-of-order parameter of the video frame is abnormal and the out-of-order parameter of the data packet corresponding to the video frame is abnormal, or the frame header jitter parameter of the video frame and the frame Any parameter in the last packet jitter parameter is abnormal and any parameter in the packet jitter parameter, packet interval parameter, retransmission parameter, and delay parameter of the data packet corresponding to the video frame is abnormal.

Since the protocol information of the data packet is obtained at the network device and the video player, the frame quality parameter and the packet quality parameter can reflect the abnormal situation in the transmission process, and the frame packet loss parameter abnormality indicates that there is a data packet under the video frame being transmitted If the packet loss parameter of the data packet under the video frame is abnormal at this time, it means that there is a data packet lost during transmission, so it can be determined that there is a problem with the network transmission; the abnormal frame disorder parameter indicates that there is a data packet under the video frame It is in an out-of-order state during transmission. If the out-of-order parameters of the data packets under the video frame are abnormal at this time, it means that some data packets are in out-of-order during transmission, so it can be determined that there is a problem in network transmission; based on the above reasons, in the third condition "The frame packet loss parameter of the video frame is abnormal and the packet loss parameter of the data packet corresponding to the video frame is abnormal, or the frame out-of-order parameter of the video frame is abnormal and the out-of-order data packet corresponding to the video frame Abnormal parameter", which can accurately reflect the network transmission problem; the abnormality of the frame first packet jitter parameter/frame end packet jitter parameter indicates that the delay jitter of the first/last data packet under the video frame is too large during transmission. If the packet interval parameter of the data packet under the frame is abnormal, it means that the time difference between the two data packets in the transmission process is too large, and it can be determined that there is a problem in the network transmission; if the packet jitter parameter of the data packet under the video frame is abnormal at this time, it means that the data packet If the delay and jitter in the transmission process is too large, it can be determined that there is a problem in network transmission; if the retransmission parameters of the data packet under the video frame are abnormal at this time, it means that the data packet has undergone multiple transmissions during the transmission process, and it can be determined that there is a network transmission problem. Problem: If the delay parameter of the data packet under the video frame is abnormal at this time, it means that the transmission time of the data packet is too long during the transmission process, and it can be determined that there is a problem with the network transmission. Based on the above reasons, in the third condition "any parameter in the frame header jitter parameter and the frame tail packet jitter parameter of the video frame is abnormal and the packet jitter parameter, the packet interval parameter, and the packet interval parameter of the data packet corresponding to the video frame are Any one of the retransmission parameters and delay parameters is abnormal" can accurately reflect the terminal parameter problem.

According to the ninth possible implementation of the first aspect, in the tenth possible implementation of the method for determining video quality problems, after determining that the cause of abnormal video quality at the video playback end is a network transmission problem, the The abnormal frame quality parameter and/or the abnormal packet quality parameter determine the cause and equipment that cause the abnormal video quality of the video player, and also include: according to the network layer protocol information of the data packet, determine the abnormal packet quality parameter The network layer sequence number of the corresponding data packet; according to the transport layer protocol information of the data packet, determine the packet time, transport layer sequence number, and checksum of the data packet corresponding to the abnormal packet quality parameter; receive the network layer on the network device Topology data, determine the transmission path of each data packet in the network according to the network topology data; according to the transmission path of the data packet corresponding to the abnormal packet quality parameter described in the network, find out the packet time, The data packet whose network layer serial number, transport layer serial number, and checksum all meet the conditions; according to the found packet quality parameters of the data packet that meets the conditions, determine from each network device on the transmission path of the data packet that causes the video to play The network device with abnormal video quality.

The packet quality parameter of the data packet on each device can indicate the impact of the device on the transmission of the data packet, and by finding the corresponding data packets in each network device on the transmission path, the statistics can be calculated using the found data packets In the process of packet transmission, the impact of different network devices on the transmission of data packets, and determine the specific device that causes the abnormal video quality of the video player.

According to the tenth possible implementation of the first aspect, in the eleventh possible implementation of the method for determining video quality problems, the packet time, network layer sequence number, transport layer sequence number, checksum The conditions to be met include: the absolute value of the difference between the packet time and the packet time of the data packet corresponding to the abnormal packet quality parameter is less than a preset threshold, and the network layer sequence number, the transport layer sequence number, the checksum and the The network layer sequence number, transport layer sequence number, and checksum of the data packets corresponding to the abnormal packet quality parameters are all the same.

In this way, the same data packet on different network devices along the transmission path can be found. For the same data packet, according to the abnormality of the packet quality parameters of the same data packet of each device on the transmission path, it can be inferred that the cause of the abnormality of the packet quality parameters on different devices is the current device or the upstream device of the current device. One, by further determining the abnormality of the packet quality parameters on the upstream device, it can be accurately determined that the cause of the abnormal video quality at the video playback end is a network transmission problem, and the specific network device that causes the abnormal video quality at the video playback end.

According to the tenth or eleventh possible implementation of the first aspect, in the twelfth possible implementation of the method for determining video quality problems, how many data packets correspond to the abnormal packet quality parameter When there are multiple groups of data packets that meet the conditions found, according to the packet quality parameters of the data packets that meet the conditions found, from each network device on the transmission path of the data packets, determine the network that causes the abnormal video quality of the video playback end. Equipment, including: according to the packet quality parameters of multiple sets of data packets that meet the conditions, from each network device on the transmission path of the data packet, repeatedly determine the network equipment that may cause abnormal video quality at the video playback end; count the multiple determinations Among the network devices that may cause abnormal video quality at the video playback end, the network device that appears most frequently is the network device that causes abnormal video quality at the video playback end.

By using the packet quality parameters of multiple data packets on each network device on the transmission path to perform multiple determinations, the accuracy of the determined network device that causes abnormal video quality can be improved, so that the determined network that causes abnormal video quality After the device implements the corresponding solution, the probability of abnormal video quality caused by the same problem on the same device is reduced.

In the second aspect, an embodiment of the present application provides an apparatus for determining a video quality problem, the apparatus including: an information receiving module, configured to receive protocol information on a video player and a network device, and the network device is used to convert the video The data is transmitted to the video player, and the protocol information indicates the sending mode of the video frame, and the video frame includes a plurality of data packets; the parameter determination module is used to determine abnormal frame quality parameters and abnormal packet quality according to the protocol information Parameters, the abnormal frame quality parameter represents the abnormal quality of the video frame to which the abnormal frame quality parameter belongs, and the abnormal packet quality parameter represents the abnormal quality of the data packet to which the abnormal packet quality parameter belongs; abnormal reason The determining module is configured to determine the cause and equipment for abnormal video quality at the video playback terminal according to the abnormal frame quality parameter and/or the abnormal packet quality parameter.

According to the second aspect, in the first possible implementation of the device for determining video quality problems, the protocol information includes streaming media layer protocol information, network layer protocol information, and transport layer protocol information of data packets, and the stream The media layer protocol information includes the streaming media layer protocol information of the data packet on the video playback end, the network layer protocol information includes the network layer protocol information of the data packet on the network device and the video playback end, and the transport layer protocol information includes The transport layer protocol information of the data packet on the network device and on the video playback end, said determining abnormal frame quality parameters and abnormal packet quality parameters according to the protocol information includes: determining at least one according to the streaming media layer protocol information The video frame and each data packet corresponding to each frame in the at least one video frame; according to the streaming media layer protocol information and the network layer protocol information, determine the frame quality parameters of each frame in the at least one video frame; according to The network layer protocol information and the transport layer protocol information determine the packet quality parameters of each data packet corresponding to each frame in the at least one video frame; according to the frame quality parameters corresponding to each frame in the at least one video frame and The frame quality parameter threshold determines the abnormal frame quality parameter, and determines the abnormal packet quality parameter according to the packet quality parameter of each data packet corresponding to each frame in the at least one video frame and the packet quality parameter threshold.

According to the first possible implementation manner of the second aspect, in the second possible implementation manner of the apparatus for determining video quality problems, according to the frame quality parameters corresponding to each frame in the at least one video frame and the frame A quality parameter threshold, determining the abnormal frame quality parameter, determining the abnormal packet quality parameter according to the packet quality parameter and the packet quality parameter threshold of each data packet corresponding to each frame in the at least one video frame, including: For each frame of the at least one video frame, among the frame quality parameters of the video frame, the frame quality parameter whose value is greater than the frame quality parameter threshold is determined as the abnormal frame quality parameter; for the at least one video frame For each frame, among the packet quality parameters of the data packets corresponding to the video frame, the packet quality parameter whose value is greater than the packet quality parameter threshold is determined as the abnormal packet quality parameter.

According to the first or second possible implementation of the second aspect, in the third possible implementation of the device for determining video quality problems, the streaming media layer protocol information of the data packet includes The frame time stamp of the video frame, the network layer protocol information of the data packet includes the packet time indicating the time of the data packet, the frame quality parameter includes a frame header jitter parameter and a frame tail packet jitter parameter, and the frame header packet The jitter parameter represents the delay jitter situation of the first data packet corresponding to the video frame, and the frame tail packet jitter parameter represents the delay jitter situation of the last data packet corresponding to the video frame, and the described stream media layer protocol information according to the and the network layer protocol information, determining the frame quality parameters of each frame in the at least one video frame includes: for each frame of the at least one video frame, determining the frame quality parameter according to the packet time of the data packet corresponding to the video frame The first packet arrival time and the last packet arrival time of the video frame, the first packet arrival time represents the arrival time of the first data packet corresponding to the video frame, and the tail packet arrival time represents the arrival of the last data packet corresponding to the video frame Time; according to the arrival time of the first packet and the arrival time of the tail packet, calculate the first packet interval and the tail packet interval of adjacent video frames respectively, and the first packet interval represents the arrival time difference of the first data packet corresponding to the adjacent video frame , the tail packet interval represents the arrival time difference of the last data packet corresponding to the adjacent video frame; according to the frame timestamp, calculate the frame interval of the adjacent video frame respectively; according to the first packet interval, frame interval, and previous The first frame packet jitter parameter of a video frame calculates the first frame packet jitter parameter of the current video frame; according to the tail packet interval, the frame interval, and the frame tail packet jitter parameter of the previous video frame, the frame tail of the current video frame is calculated Packet jitter parameters.

According to any one of the first to third possible implementations of the second aspect, in the fourth possible implementation of the device for determining video quality problems, the stream of data packets The media layer protocol information includes a stream media layer sequence number indicating the sequence of the data packet in the video frame to which it belongs, the network layer protocol information of the data packet includes a packet time indicating the time of the data packet, and the frame quality parameter includes a frame loss Packet parameter and frame out-of-sequence parameter, described frame packet loss parameter represents the loss situation of the data packet of video frame, and described frame out-of-sequence parameter represents the out-of-sequence situation of the data packet of video frame, and described according to described streaming media layer protocol Information and the network layer protocol information, determining the frame quality parameters of each frame in the at least one video frame includes: for each frame of the at least one video frame, according to the streaming media layer of the data packet corresponding to the video frame The serial number determines the frame packet loss parameter of the video frame; for each frame of the at least one video frame, sorts the data packets corresponding to the video frame according to the order of packet time of the data packets corresponding to the video frame , obtaining the sorting result corresponding to the video frame, and determining the frame out-of-sequence parameter of the video frame according to the sorting result and the streaming media layer sequence number of the data packet corresponding to the video frame.

According to the fourth possible implementation of the second aspect, in the fifth possible implementation of the device for determining video quality problems, for each frame of the at least one video frame, according to the corresponding The streaming media layer sequence number of the data packet, determining the frame loss parameter of the video frame includes: for each frame of the at least one video frame, according to the streaming media layer sequence number of the data packet corresponding to the video frame, determining The maximum value of the streaming media layer sequence number of the data packet corresponding to this video frame, the minimum value and the number of streaming media layer sequence numbers, according to the maximum value and minimum value of the streaming media layer sequence number, determine the corresponding data of this video frame The theoretical value of the number of packets, according to the difference between the theoretical value of the number of data packets corresponding to the video frame and the number of streaming media layer serial numbers of the data packets corresponding to the video frame, determine the frame packet loss parameter of the video frame ; According to the sequence number of the streaming media layer of the data packet corresponding to the sorting result and the video frame, determine the frame disorder parameter of the video frame, including: the numerical value of the streaming media layer serial number of the current data packet is greater than that in the sorting result , the streaming media layer serial number of the data packet before the current data packet, according to the sorting serial number of the current data packet in the sorting result, the streaming media layer serial number of the current data packet, and the data packet corresponding to the video frame The minimum value of the serial number of the streaming media layer determines the frame disorder parameter corresponding to the video frame.

According to any one possible implementation manner of the first to fifth possible implementation manners of the second aspect, in a sixth possible implementation manner of the apparatus for determining video quality problems, the at least one video frame The packet quality parameters of each data packet corresponding to each frame in the frame are determined according to the network layer protocol information and the transport layer protocol information of the data packet, and the packet quality parameters include: the packet representing the time interval between the current data packet and the previous data packet The interval parameter, the packet jitter parameter indicating the delay jitter between the current data packet and the previous data packet, the out-of-order parameter indicating the disorder of the data packet, the packet loss parameter indicating the packet loss, and the packet retransmission Retransmission parameters, delay parameters indicating the delay of data packets and corresponding response messages, IP packet fragmentation parameters indicating whether there is IP packet fragmentation in data packets, indicating whether there is an error in the checksum of data packets One or more of the checksum error parameters.

According to the sixth possible implementation manner of the second aspect, in the seventh possible implementation manner of the apparatus for determining video quality problems, the abnormal frame quality parameter and/or the abnormal packet quality Parameter determination causes the cause and equipment of abnormal video quality at the video playback end, including: for any video frame, when the frame quality parameter of the video frame and the packet quality parameter of the data packet corresponding to the video frame meet the first condition, determine The cause of the abnormal video quality at the video playback end is a video encoding problem, and it is determined that the device causing the abnormal video quality at the video playback end is a device used for video encoding; wherein, the first condition includes: the frame loss parameter of the video frame is abnormal And the packet loss parameter of the data packet corresponding to the video frame is normal, or the frame disorder parameter of the video frame is abnormal and the disorder parameter of the data packet corresponding to the video frame is normal.

According to the sixth or seventh possible implementation manner of the second aspect, in the eighth possible implementation manner of the apparatus for determining video quality problems, according to the abnormal frame quality parameter and/or the The abnormal packet quality parameter determines the cause and equipment that cause the abnormal video quality of the video player, including: for any video frame, the frame quality parameter of the video frame and the packet quality parameter of the data packet corresponding to the video frame meet the second condition, determine that the cause of the abnormal video quality at the video player end is a terminal parameter problem, and determine that the device causing the abnormal video quality at the video player end is the device used for video playback; wherein, the second condition includes: the frame of the video frame Any parameter in the first packet jitter parameter and the frame tail packet jitter parameter is abnormal, and any parameter in the IP packet fragmentation parameter and the checksum error parameter of the data packet corresponding to the video frame is abnormal.

According to any one possible implementation manner of the sixth to eighth possible implementation manners of the second aspect, in the ninth possible implementation manner of the apparatus for determining video quality problems, the The frame quality parameter of the video frame and/or the packet quality parameter of the abnormality determine the cause and equipment that cause the abnormal video quality of the video player, including: for any video frame, the frame quality parameter of the video frame and the corresponding video frame When the packet quality parameter of the data packet satisfies the third condition, it is determined that the cause of the abnormal video quality at the video playback end is a network transmission problem; wherein, the third condition includes: the frame packet loss parameter of the video frame is abnormal and the video frame The packet loss parameter of the data packet corresponding to the frame is abnormal, or the frame out-of-order parameter of the video frame is abnormal and the out-of-order parameter of the data packet corresponding to the video frame is abnormal, or the frame header jitter parameter of the video frame and the frame Any parameter in the last packet jitter parameter is abnormal and any parameter in the packet jitter parameter, packet interval parameter, retransmission parameter, and delay parameter of the data packet corresponding to the video frame is abnormal.

According to the ninth possible implementation manner of the second aspect, in the tenth possible implementation manner of the device for determining video quality problems, after determining that the cause of the abnormal video quality at the video playback end is a network transmission problem, the The abnormal frame quality parameter and/or the abnormal packet quality parameter determine the cause and equipment that cause the abnormal video quality of the video player, and also include: according to the network layer protocol information of the data packet, determine the abnormal packet quality parameter The network layer sequence number of the corresponding data packet; according to the transport layer protocol information of the data packet, determine the packet time, transport layer sequence number, and checksum of the data packet corresponding to the abnormal packet quality parameter; receive the network layer on the network device Topology data, determine the transmission path of each data packet in the network according to the network topology data; according to the transmission path of the data packet corresponding to the abnormal packet quality parameter described in the network, find out the packet time, The data packet whose network layer serial number, transport layer serial number, and checksum all meet the conditions; according to the found packet quality parameters of the data packet that meets the conditions, determine from each network device on the transmission path of the data packet that causes the video to play The network device with abnormal video quality.

According to the tenth possible implementation of the second aspect, in the eleventh possible implementation of the device for determining video quality problems, the packet time, network layer sequence number, transport layer sequence number, checksum The conditions to be met include: the absolute value of the difference between the packet time and the packet time of the data packet corresponding to the abnormal packet quality parameter is less than a preset threshold, and the network layer sequence number, the transport layer sequence number, the checksum and the The network layer sequence number, transport layer sequence number, and checksum of the data packets corresponding to the abnormal packet quality parameters are all the same.

According to the tenth or eleventh possible implementation manner of the second aspect, in the twelfth possible implementation manner of the apparatus for determining video quality problems, how many data packets correspond to the abnormal packet quality parameter When there are multiple groups of data packets that meet the conditions found, according to the packet quality parameters of the data packets that meet the conditions found, from each network device on the transmission path of the data packets, determine the network that causes the abnormal video quality of the video playback end. Equipment, including: according to the packet quality parameters of multiple sets of data packets that meet the conditions, from each network device on the transmission path of the data packet, repeatedly determine the network equipment that may cause abnormal video quality at the video playback end; count the multiple determinations Among the network devices that may cause abnormal video quality at the video playback end, the network device that appears most frequently is the network device that causes abnormal video quality at the video playback end.

In a third aspect, an embodiment of the present application provides an apparatus for determining a video quality problem, including: a processor; a memory for storing processor-executable instructions; wherein, the processor is configured to execute the above-mentioned first aspect or the first aspect A method for determining a video quality problem in one or several possible implementation manners of one aspect.

In a fourth aspect, the embodiments of the present application provide a non-volatile computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above-mentioned first aspect or multiple aspects of the first aspect are implemented. A method for determining a video quality problem in one or several possible implementation manners.

In the fifth aspect, the embodiments of the present application provide a computer program product, including computer readable code, or a non-volatile computer readable storage medium bearing computer readable code, when the computer readable code is stored in an electronic When running in the device, the processor in the electronic device executes the method for determining a video quality problem in the first aspect or one or more of the multiple possible implementations of the first aspect.

These and other aspects of the present application will be made more apparent in the following description of the embodiment(s).

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the application and, together with the specification, serve to explain the principles of the application.

Fig. 1 shows an exemplary application scenario of a method for determining a video quality problem according to an embodiment of the present application.

FIG. 2 shows an example of identifying data stream types in this embodiment of the present application.

Fig. 3 shows an exemplary method schematic diagram of a method for determining a video quality problem according to an embodiment of the present application.

Fig. 4 shows a schematic diagram of an exemplary method for determining an abnormal frame quality parameter and/or an abnormal packet quality parameter according to an embodiment of the present application.

Fig. 5 shows an example of a method for determining a video frame and a data packet corresponding to the video frame according to an embodiment of the present application.

FIG. 6 shows an example of a calculation method of a packet jitter parameter at the beginning of a frame and a packet jitter parameter at the end of a frame according to an embodiment of the present application.

FIG. 7 shows an example of a calculation manner of a frame packet loss parameter according to an embodiment of the present application.

FIG. 8 shows an example of a calculation manner of a frame out-of-sequence parameter according to an embodiment of the present application.

FIG. 9 shows an example of a normal distribution probability curve generated according to an embodiment of the present application.

Fig. 10 shows an example of realizing abnormal condition determination of parameters according to an embodiment of the present application.

Fig. 11 shows an exemplary schematic diagram of the correspondence between abnormal frame quality parameters, abnormal packet quality parameters, causes of abnormal video quality at the video player end, and devices according to an embodiment of the present application.

FIG. 12 shows an example of determining a specific network device that causes abnormal video quality at a video player end according to an embodiment of the present application.

Fig. 13 shows an exemplary structural diagram of an apparatus for determining a video quality problem according to an embodiment of the present application.

Fig. 14 shows an exemplary structural diagram of an apparatus for determining a video quality problem according to an embodiment of the present application.

Detailed ways

Various exemplary embodiments, features, and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures indicate functionally identical or similar elements. While various aspects of the embodiments are shown in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior or better than other embodiments.

In addition, in order to better illustrate the present application, numerous specific details are given in the following specific implementation manners. It will be understood by those skilled in the art that the present application may be practiced without certain of the specific details. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail in order to highlight the gist of the present application.

In the prior art, in order to locate the causes of video quality problems caused by abnormal video frames, such as stuttering and blurred screens, a device based on the live network (that is, the network where the abnormal video frame is located) is proposed to perform a large number of Data collection, and then use the statistical distribution of big data to analyze the collected data to find out the cause of the video quality problem and the solution of the equipment. The specific implementation of this technical solution is as follows:

Through data collection methods such as probe acquisition and device reporting, user experience data and asset management data are obtained from user devices (ie, video playback devices), and network topology data is obtained from network devices through data collection methods reported by devices. Among them, the user experience data includes the average video mean opinion score (vMOS), freeze duration, freeze frequency, and video quality switching times (such as switching between 1080P, 720P, and 360P video display formats). times), etc. The user experience data is used to represent the quality of the user's experience and perception when watching a video, and the video playback quality can be determined according to the user experience data. Asset management data includes users' network access information. Asset management data is used to represent the connection relationship between user equipment and network equipment. Network topology data is used to represent the connection relationship between network devices. According to the asset management data and network topology data, the total number of users connected to each network device can be determined.

For each network device, the percentage of the number of users with poor video playback quality in the total number of users on the network device may be determined as the poor quality rate of the network device. The poor quality rate of each network device is counted, and the distribution of the poor quality rate of network devices in the network can be determined according to the connection relationship between network devices.

Quality of experience (QoE) is determined according to the determined poor quality rate of each network device and a preset threshold. The quality of experience includes calculating the coefficient of variation of the poor quality rate, the average value of the poor quality rate, etc., and determining the problem equipment through the quality of experience, such as determining the degree of skewed distribution between the coefficient of variation of the poor quality rate and the preset threshold or the mean value of the poor quality rate The degree of skewed distribution with the preset threshold, and find the equipment that causes a large deviation through the degree of skewed distribution.

According to the connection relationship between the network equipment, determine whether the network equipment has a lower-level network equipment, if not, then the network equipment is the network equipment that causes the video quality problem determined by the solution of the prior art; if so, through the lower-level network equipment Quality of Experience further identifies the cause and device causing video quality issues.

The statistical distribution scheme using big data proposed in the prior art still has certain shortcomings. First of all, the solution determines the cause of the video quality problem and the time cost of the equipment is high, and the real-time performance of the solution is poor. Secondly, this solution requires an overall analysis of the data collected when video quality problems occur multiple times. That is to say, the causes and equipment that lead to video quality problems determined by the existing technology are also determined for multiple video quality problems. , while multiple video quality issues may have different causes and devices. The cause and equipment of the video quality problem determined by the prior art may be only one of the causes and equipment of the video quality problem many times, and the determined cause and equipment of the video quality problem are not accurate enough. Optimizing the device based on the identified cause of the video quality issue and optimizing the device will not reduce the chances of some of the same video quality issues recurring in the future.

To sum up, the existing technology is still unable to quickly and accurately complete the determination of the cause of the video quality problem and the device.

In order to solve the above technical problems, this application provides a video quality problem determination method, device, storage medium, and program product. The video quality problem determination method in the embodiment of the application can quickly and accurately determine the cause and effect of the video quality problem. equipment.

As shown in Figure 1, the scene may be, for example, a live video scene, which may include a video provider for providing video data, a plurality of network devices for transmitting video data, and a video server for displaying video images to users. player side. Wherein, the video provider may be a camera capable of collecting video data, or a device capable of receiving, storing and outputting video data. The video provider can encode the collected or received video data, and encapsulate it into a live streaming form according to the streaming media layer protocol. The encapsulated video data may be transmitted through network equipment, and the network equipment may include, for example, the wireless base station, bearer network equipment, core network equipment, switch, etc. in FIG. 1 . The wireless base station may be, for example, a 5G base station (next generation NB, gNB). The core network device may be, for example, a unified distributed gateway (unified distributed gateway, UDG), etc., and the switch may, for example, be a switch (end of row, EoR) using a certain deployment method. The video data is finally output to a video player, which can be, for example, a mobile phone, a tablet computer, a notebook, and the like. On the video player end, the video data in the form of live streaming can be decapsulated, decoded, and rendered, so that users can watch the video through the video player end.

Wherein, the application scenario in FIG. 1 also includes a terminal device (or server) for determining the cause of the video quality problem and the device, and the terminal device (or server) may be different from the above-mentioned video provider, network device, and video player. The terminal device (or server) may include the apparatus for determining a video quality problem in the embodiment of the present application, which can be used to execute the method for determining a video quality problem in the embodiment of the present application.

Wherein, the terminal device (or server) may also include a protocol information collection device for collecting protocol information from the network device and the video player, and a topology data collection device for collecting network topology data from the network device. Alternatively, the device for determining video quality problems, the device for collecting protocol information, and the device for collecting topology data can also be respectively set on a single terminal device or server, or any two of the three can be set together on the same terminal device or server.

Those skilled in the art should understand that the application scenarios of the embodiments of the present application are not limited to the foregoing devices. For example, there may be multiple video playback terminals connected to the network device, and for the video quality problem of each video playback terminal, on the terminal device (or server), the method for determining the video quality problem of the embodiment of the present application may be respectively executed.

In a possible implementation manner, taking the application scenario in FIG. 1 as an example, the protocol information collection device and the topology data collection device may collect protocol information and network topology data respectively. Wherein, when the video provider continuously outputs the live stream, the protocol information collection device can continuously collect the protocol information on the video playback terminal and network equipment. If there is no change in the connection relationship of the network equipment during this process, the topology data collection device only needs to collect the network topology data once. According to the protocol information on the video player and the network device, using the method for determining the video quality problem in the embodiment of the present application can determine the cause of the abnormal video quality of the video player and the device.

An exemplary collection method of protocol information in the embodiment of the present application is introduced below.

In a possible implementation manner, collecting protocol information may be implemented based on existing technologies. For example, the device for collecting protocol information firstly obtains data streams on the video playback terminal and network equipment. The data streams on the video playback terminal and the network device may include video streams (video), and may also include other types of data streams such as voice streams. The protocol information collection device can identify the collected data stream in combination with the streaming media protocol, and obtain data packets of the video stream type from the data stream. After determining the data whose type is video stream, the protocol information acquisition device can further obtain the streaming media layer protocol information and network layer protocol information of the data packet from the data whose type is video stream according to the streaming media layer protocol, network layer protocol, transport layer protocol, etc. Information, transport layer protocol information, so as to complete the collection of protocol information.

The collection of protocol information can be realized by deploying probes, or through network port packet capture through computers and other equipment, or through functions such as user tracking that come with network equipment. This section uses probes to collect protocol information as an example. Among them, the probe can have deep packet inspection (deep packet inspection, DPI) capability, and can automatically realize the identification of different types of data flows and the acquisition of protocol information.

A data packet may include a payload header (packet header) and a payload, wherein the payload part includes the specific content of the data. The payload header includes protocol information indicating the transmission mode of the data packet. In order to ensure data security, when obtaining the protocol information, the payload of the data packet can be encrypted, or only the payload header (packet header) of the data packet can be detected. For example, for the data packet on the video playback end, its streaming media layer payload header, network layer payload header, and transport layer payload header can be detected and the corresponding streaming media layer protocol information, network layer protocol information, and transport layer protocol information can be extracted , for data packets on other network devices, the network layer payload header and the transport layer payload header can be detected and the corresponding network layer protocol information and transport layer protocol information can be extracted.

For example, for the payload header of the streaming media layer, protocol information extraction can be realized by identifying the application type and protocol type. The application type of a certain data flow in the data can be identified by using the service awareness (SA) technology. For example, if a user experiences abnormal video quality when using an application to watch a live video at a certain time period, the user can watch the live video from the user according to the user identity information, the time information of the data packet, and/or the type of application, etc. Obtain data streams related to video quality exceptions that need to be analyzed from the data streams on the video playback end. By default, it is obtained from all data streams on the video player. For the obtained data stream, the judgment method specified in the existing protocol can be used to identify the data stream type layer by layer from the network layer, transport layer, streaming media layer and application layer, and determine the video stream existing in the data stream Furthermore, specific types of video streams are distinguished to determine the type of video streams containing video frames. FIG. 2 shows an example of identifying data stream types in this embodiment of the present application.

As shown in Figure 2, the protocol used by the network layer can include Internet Protocol (Internet Protocol, IP), and the protocol used by the transport layer can include Transmission Control Protocol (transmission control protocol, TCP) and user datagram protocol (user datagram). protocol, UDP). The protocols used by the streaming media layer may include real-time streaming protocol (real time streaming protocol, RTSP), real-time messaging protocol (real time messaging protocol, RTMP), real-time transport protocol (real-time transport protocol, RTP), real-time transmission control protocol (real-time transport control protocol, RTCP), user datagram protocol-based data transfer protocol (UDP-based data transfer protocol, UDT) or secure reliable transport protocol (secure reliable transport, SRT), and MPEG2 transport stream (MPEG -2 transport stream, MPEG2-TS). The protocol used by the application layer may include the H.264 video standard protocol or the H.265 video standard protocol.

Referring to Figure 2, according to the protocol used by the above transport layer, the protocol used by the streaming media layer and the protocol used by the application layer, the supported video stream types can include: TCP+RTMP video stream, TCP+RTSP+RTP video stream, UDP+ RTP video stream, UDP+RTP+MPEG2-TS video stream, UDP+SRT/UDT+MPEG2-TS video stream, and UDP+MPEG2-TS video stream. The above protocols can also support control streams, such as TCP+RTMP control streams, TCP+RTSP control streams, UDP+RTCP control streams, etc. in FIG. 2 , and the control streams do not contain video frames.

For different video streams, the protocol information can be obtained from the data packets under the video stream. Wherein, the protocol information may include the network layer protocol information, the transport layer protocol information, the streaming media layer protocol information of the data packets in the video stream, and the calculated transport layer checksum obtained according to the standard calculation method of checksum (CHECKSUM). and (hereinafter referred to as checksum).

An exemplary collection method of network topology data in the embodiment of the present application is introduced below.

For example, network topology data can be collected from network devices. The network topology data may include the topology connection relationship or service path of each network device in the network where the network device connected to the video playback terminal is located. The service path represents the connection relationship between network devices that the service flow passes through. It makes it possible to determine the upstream and downstream relationship of network devices according to the network topology data.

After the protocol information and network topology data are collected, the protocol information collecting device and the topology data collecting device can respectively transmit the collected information or data to the video quality problem determining device. The device for determining a video quality problem uses the method for determining a video quality problem according to an embodiment of the present application. Protocol information and network topology data can be used to determine the cause and equipment that cause abnormal video quality at the video playback end.

As shown in FIG. 3, in a possible implementation manner, the method for determining a video quality problem according to an embodiment of the present application includes:

S1, receiving protocol information on a video player and a network device, the network device is used to transmit video data to the video player, the protocol information indicates a sending method of a video frame, and the video frame includes a plurality of data packets;

S2. Determine an abnormal frame quality parameter and an abnormal packet quality parameter according to the protocol information, the abnormal frame quality parameter indicates that the quality of the video frame to which the abnormal frame quality parameter belongs is abnormal, and the abnormal packet quality parameter Indicating that the quality of the data packet to which the abnormal packet quality parameter belongs is abnormal;

S3. Determine the cause and equipment causing the abnormal video quality at the video playback terminal according to the abnormal frame quality parameter and/or the abnormal packet quality parameter.

For example, in step S1, protocol information can be obtained. In step S2, the recognition of the video frame and the extraction of the data packets under the video frame can be realized. For video quality problems such as blurred screen and freezing, frame quality parameters can be defined to characterize the quality of video frames. There may be multiple frame quality parameters, such as the first frame packet jitter parameter, frame end packet jitter parameter, frame packet loss parameter, and frame disorder parameter mentioned below. Different frame quality parameters can be preset with different calculation methods. Through the preset calculation method of frame quality parameters, the value of frame quality parameters can be calculated according to the protocol information, and the calculated value of frame quality parameters can be obtained through the frame quality parameter threshold value Anomaly detection is performed to determine frame quality parameters with abnormal values, that is, abnormal frame quality parameters. According to the abnormal frame quality parameter, the corresponding abnormal video frame and all data packets under the abnormal video frame can be further determined. Packet quality parameters can be defined to characterize the quality of data packets. There can be multiple packet quality parameters, such as packet interval parameters, packet jitter parameters, packet loss parameters, out-of-sequence parameters, retransmission parameters, delay parameters, IP packet fragmentation parameters, and checksum errors mentioned below parameter. Different packet quality parameters can be preset with different calculation methods. Through the preset calculation method of packet quality parameters, the value of the packet quality parameter can be calculated according to the protocol information, and the value of the packet quality parameter calculated by the packet quality parameter threshold value Anomaly detection is performed to determine packet quality parameters with abnormal values, that is, abnormal packet quality parameters.

In step S3, according to whether the abnormal frame quality parameters of the abnormal video frame and the abnormal packet quality parameters of the data packets under the abnormal video frame meet the corresponding conditions, determine the cause and equipment causing the abnormal video quality. There may be multiple conditions, such as the first condition, the second condition and the third condition mentioned below. Wherein, when different conditions are met, the determined causes and devices that lead to abnormal video quality may be different. If the device causing the abnormal video quality can be uniquely identified based on the determined cause of the abnormal video quality. For example, if it is determined that the cause of abnormal video quality is a video encoding problem, then the only device that can be determined to cause abnormal video quality is the device used for video encoding. For another example, if it is determined that the cause of abnormal video quality is a terminal parameter problem, then the only device that can be determined to cause abnormal video quality is the device used for video playback, that is, the video playback terminal. In this case, the video quality problem determination method according to the embodiment of the present application has been realized to determine the cause and equipment that lead to abnormal video quality. If the device that causes the abnormal video quality cannot be uniquely determined based on the determined cause of the abnormal video quality, for example, it is determined that the cause of the abnormal video quality is a network transmission problem. Since multiple network devices may participate in the network transmission process, it cannot be uniquely determined. Devices that cause abnormal video quality. This application can determine the abnormal data packets according to the abnormal packet quality parameters, and according to the protocol information of the abnormal data packets, find multiple data packets whose protocol information satisfies the conditions on multiple network devices participating in the network transmission process, and use the found multiple The packet quality parameter of the data packet is analyzed to determine the device causing the abnormal video quality.

Wherein, when performing the video quality problem determination method of the embodiment of the present application to determine the cause and equipment that caused the video quality abnormality at the video playback end last time, the frame quality parameter threshold and packet quality parameter threshold used may be determined according to the AI model. The AI model can be trained by using the frame quality parameters and packet quality parameters calculated for a period of time before determining the current cause of abnormal video quality at the video playback end and the device as samples, which can solve common samples that need to be marked and abnormal samples The modeled data inaccessibility problem reduces the complexity of determining abnormal frame quality parameters and abnormal packet quality parameters.

It can be seen from the above description that the abnormal frame quality parameter may be an abnormal parameter among the frame quality parameters, and the abnormal packet quality parameter may be an abnormal parameter among the packet quality parameters. In step S2, the frame quality parameter and the packet quality parameter may be determined first according to the protocol information, and then the abnormal frame quality parameter and the abnormal packet quality parameter are determined according to the frame quality parameter and the packet quality parameter.

As shown in Figure 4, in a possible implementation, the protocol information includes streaming media layer protocol information, network layer protocol information and transport layer protocol information of the data packet, and the streaming media layer protocol information includes the video player The streaming media layer protocol information of the data packet on the network layer protocol information includes the network layer protocol information of the data packet on the network device and the video player end, and the transport layer protocol information includes the network layer protocol information on the network device and the video player end The transport layer protocol information of the data packet, step S2 includes:

S21. Determine at least one video frame and each data packet corresponding to each frame in the at least one video frame according to the streaming media layer protocol information.

S22. Determine the frame quality parameters of each frame in the at least one video frame according to the streaming media layer protocol information and the network layer protocol information; the frame quality parameters may include frame header jitter parameters, frame tail packet jitter parameters, Frame loss parameters, frame disorder parameters.

S23. According to the network layer protocol information and the transport layer protocol information, determine the packet quality parameters of each data packet corresponding to each frame in the at least one video frame; the packet quality parameters may include packet interval parameters, packet jitter parameters, Packet loss parameters, out-of-sequence parameters, retransmission parameters, delay parameters, IP packet fragmentation parameters, and checksum error parameters.

S24. Determine the abnormal frame quality parameter according to the frame quality parameter corresponding to each frame in the at least one video frame and the frame quality parameter threshold, and according to the packet quality of each data packet corresponding to each frame in the at least one video frame parameter and the packet quality parameter threshold, and determine the abnormal packet quality parameter.

Step S21 can be implemented based on existing technologies. For example, in step S21, for the acquired protocol information, video frame identification and data packets corresponding to the video frame are extracted according to different types of video streams (see above FIG. 2 and related descriptions). A common method is to find a data packet with a streaming media type in the video stream according to the streaming media layer protocol information of the data packet (see step S211 below for an example). Extract the information of the streaming media type from the streaming media layer protocol information of the data packet with the streaming media type found (see step S212 below for an example), and obtain in the streaming media layer protocol information of the data packet of the video streaming media type The serial number of the streaming media layer, and then according to the serial number of the streaming media layer, the data packets used to calculate and obtain the packet quality parameter and the frame time stamps of these data packets are found. Count the frame time stamps of all data packets used to calculate and obtain the packet quality parameter, and use the packets with the same frame time stamp as the data packets belonging to the same video frame (see step S213 below for an example). In this case, each frame in at least one video frame may include data packets with the same frame time stamp, and the frame time stamps of the data packets in different video frames may be different, so that each video frame and each video frame corresponding to each frame may be determined. data pack.

Among them, for the video stream carrying MPEG2-TS (UDP+RTP+MPEG2-TS video stream, UDP+SRT/UDT+MPEG2-TS video stream, UDP+MPEG2-TS video stream), the streaming media type can be passed in the protocol information The value under the Stream type under the TS header is determined. Streaming media layer sequence number can be determined by the value of Stream PID when the value under Stream type in the protocol information is Video (video); for the video stream (TCP+RTSP+TCP+RTSP+ RTP video stream, UDP+RTP video stream), the streaming media type can be determined by the value under Payload type in the protocol information, and the serial number of the streaming media layer can be determined by the value of RTP SEQUENCE NUMBER when the value under Payload type is Video (video) Determine; for video streams carrying RTMP (TCP+RTMP video streams) other than video streams carrying MPEG2-TS and video streams carrying RTP, the streaming media type can be determined by the value under Type ID in the protocol information, streaming media The layer sequence number can be determined by the protocol information in the data packet whose value under Type ID is Video (video).

As shown in Figure 5, taking the video stream whose type is UDP+RTP+MPEG2-TS as an example, the exemplary process of step S21 is as follows:

S211. In the obtained protocol information, first determine the data type according to the value of the TS PID field of the TS header. For example, the TS PID field value of TS 1# in FIG. 5 is 0, indicating that the data type of the TS transmission segment to which the TS header belongs is a program association table (program association table, PAT). After determining that the data type of the TS transmission segment is the program association table, then determine the data packet with the streaming media type according to the value of the Program Map PID in the TS header. For example, the Program Map PID field value of TS 1# in Figure 5 is 0x0010f. In the protocol information, all TS transmission segments including the obtained Program Map PID value (0x0010f), such as the TS transmission segment corresponding to TS 8# in Figure 5, are data packets with streaming media type. For all TS headers in the streaming media layer protocol information, the method in the above step S211 is executed to determine all data packets with the streaming media type.

S212. From the TS header of the determined data packet with the streaming media type, first determine the streaming media type according to the Stream type in the program mapping table. For example, it can be known from the description of S211 that TS 8# in Fig. 5 is the TS header of the data packet with streaming media type. In TS 8#, the Stream type of the program mapping table (Program Map Table, PMT) is Video, indicating that the streaming media type of the packet to which TS 8# belongs is video. After determining that the streaming media type is video, then determine the streaming media layer serial number of the data packet that the streaming media type is video according to the value of the Stream PID of the program mapping table. For example, in TS 8#, the value of the Stream PID field of PMT is 0x0100, which is the serial number of the streaming media layer of the data packet to which TS 8# belongs. The protocol information includes the data packet with the obtained Stream PID value (0x0100), that is, the data packet whose streaming media type is video. For example, the data packets corresponding to TS 11#, TS 12#, TS 13#, and TS 14# in Figure 5 are data packets whose streaming media type is video. For the TS headers of all data packets with the streaming media type, the method in the above step S212 is executed to find all the data packets with the streaming media type being video.

S213. Combine or encapsulate the data packets of at least one TS transmission segment whose streaming media type is video to obtain multiple data packets. After combination or encapsulation, each data packet includes a plurality of TS transmission segments, for example, as shown in Figure 5, the data packets corresponding to TS 11# and TS 12# are combined to obtain a data packet. Combine the data packets corresponding to TS 13# and TS 14# to get a data packet. Wherein, the combination manner of the data packets may be random combination. These combined or encapsulated data packets may be used to calculate and obtain packet quality parameters. According to the protocol information, it is determined that the protocol information contains the data of the packetized elementary stream (Packetized Elementary Stream, PES) header, and the decoding time stamp (decode time stamp, DTS) is extracted from the PES header, and the unit of DTS is converted into seconds. The value obtained after unit conversion is the frame time stamp of the data packet, that is, the frame time stamp of the video frame to which the data packet belongs. Since the identifier of the video frame is the frame time stamp of the streaming media layer, it can be considered that the data packets with the same frame time stamp belong to the same video frame. For all data packets whose streaming media type is video, the method of step S213 above is executed to identify different video frames and determine the data packets corresponding to different video frames.

After step S21 realizes the identification of different video frames and the determination of the corresponding data packets, the data packet with the streaming media layer can be further determined, and the frame time stamp and the streaming media layer serial number etc. are obtained from the streaming media layer of the data packet, and the The transport layer obtains the transport layer sequence number, checksum, confirmation message sequence number (ACK Number), etc.; obtains the network layer sequence number, fragmentation flag (Don't Fragment, DF) and packet time from the network layer of the data packet etc. These information can be used in the process of calculating frame quality parameters and packet quality parameters.

Those skilled in the art should understand that at least one video frame may also include a part of all video frames identified in step S21 (frames to which data packets whose streaming media type is video belong). It can be further set to filter out at least one video frame from the identified video frames according to whether the frame timestamp of the video frame is in the same time period, or to obtain at least one video frame from the identified video frame according to the streaming layer sequence number of the data packet in the identified video frame The frame determines at least one video frame, or selects at least one video frame from identified video frames according to a preset sampling mechanism. As long as the streaming media type of the data packet in at least one video frame is video, the present application does not limit the specific manner of determining at least one video frame according to the streaming media layer protocol information.

In step S22, for the different video frames identified in step S21, the frame quality parameters of each video frame are obtained according to the streaming media layer protocol information and the network layer protocol information of the data packets corresponding to the different video frames. The frame quality parameters of the video frame may, for example, include one or more of a frame header packet jitter parameter, a frame tail packet jitter parameter, a frame packet loss parameter, and a frame disorder parameter. Combining the frame quality parameters of the video frame and the packet quality parameters of the data packets corresponding to the video frame, the cause and equipment that cause the video quality problem can be determined, for example, it is determined that the cause of the video quality problem is a video encoding problem, and at the same time, it is determined The device used for video encoding, or, it is determined that the cause of the video quality problem is a video playback problem, and at the same time, it is determined that the device causing the video quality problem is the device used for video playback, or, it is determined that the cause of the video quality problem is network transmission problem while identifying the device causing the video quality issue as a network device, etc. Exemplary descriptions are given below for the calculation manners of different frame quality parameters.

In a possible implementation manner, the streaming media layer protocol information of the data packet includes a frame time stamp indicating a video frame to which the data packet belongs. The network layer protocol information of the data packet includes a packet time indicating the time of the data packet. The frame quality parameters include frame header jitter parameters and frame tail packet jitter parameters. The first packet jitter parameter of the frame represents the delay jitter situation of the first data packet corresponding to the video frame, and the frame tail packet jitter parameter represents the delay jitter situation of the last data packet corresponding to the video frame,

Step S22, including:

For each frame of the at least one video frame, determine the first packet arrival time and the tail packet arrival time of the video frame according to the packet time of the data packet corresponding to the video frame. The first packet arrival time represents the arrival time of the first data packet corresponding to the video frame, and the tail packet arrival time represents the arrival time of the last data packet corresponding to the video frame;

According to the arrival time of the first packet and the arrival time of the last packet, the first packet interval and the last packet interval of adjacent video frames are respectively calculated. The first packet interval represents the arrival time difference of the first data packet corresponding to adjacent video frames, and the last packet interval represents the arrival time difference of the last data packet corresponding to adjacent video frames.

According to the frame time stamp, calculate the frame interval of adjacent video frames respectively;

According to the first packet interval, the frame interval, and the first frame packet jitter parameter of the previous video frame, calculate the first frame packet jitter parameter of the current video frame;

According to the trailing packet interval, the frame interval, and the frame trailing packet jitter parameter of the previous video frame, calculate the frame trailing packet jitter parameter of the current video frame.

For example, the frame header jitter parameter and frame tail packet jitter parameter in the embodiment of the present application can be used to characterize the delay jitter of the data packet corresponding to the video frame. The higher the probability of picture blurring and freezing.

After at least one video frame and at least one data packet corresponding to the video frame are obtained according to step S21, the first frame packet jitter parameter and the frame end packet jitter parameter can be calculated in the following manner:

For example, taking FIG. 6 as an example, at least one video frame may include 4 frames, for each video frame in video frames 1-4, according to the network layer protocol information of the data packet corresponding to the video frame, the For the packet time of time, among the data packets corresponding to the video frame, the data packet with the earliest packet time can be named as the first packet, and the packet time of the first packet is the arrival time of the first packet. A data packet with the latest packet time is named as the tail packet, and the packet time of the tail packet is the arrival time of the tail packet. In this way, the first packet and the first packet arrival of each video frame in at least one video frame can be determined Time, Tail Packet, Tail Packet Arrival Time. Based on this, the arrival time of the first packet of video frames 1-4 can be recorded as T1, T2, T3, T4 respectively, and the arrival time of the last packet of video frames 1-4 can be recorded as T1', T2', T3', T4'.

According to the first packet arrival time and the last packet arrival time of each video frame in video frames 1-4, calculate the first packet interval and the last packet interval of adjacent video frames, wherein, when 2≤n≤4 and it is an integer, the video frame The first packet interval Deltan of n and video frame n-1 can be calculated by the difference between the first packet arrival time Tn of video frame n and the first packet arrival time Tn-1 of video frame n-1, video frame n and video frame n- The tail packet interval Delta' of 1 can be calculated by the difference between the tail packet arrival time Tn' of video frame n and the tail packet arrival time Tn-1' of video frame n-1. When n=1, the first packet of video frame 1 The interval Delta1 and the tail packet interval Delta1' are equal to 0 by default. Taking Figure 6 as an example, according to the first packet arrival times T1, T2, T3, and T4 of video frames 1-4, the first packet interval Delta1=0 of video frame 1, and the first packet interval Delta2 of video frame 2 and video frame 1 can be obtained =T2-T1, the first packet interval Delta3 between video frame 3 and video frame 2=T3-T2, the first packet interval Delta4 between video frame 4 and video frame 3=T4-T3. According to the tail packet arrival time T1', T2', T3', T4' of video frames 1-4, the tail packet interval Delta1'=0 of video frame 1, and the tail packet interval Delta2' of video frame 2 and video frame 1 can be obtained =T2'-T1', Delta3'=T3'-T2' between video frame 3 and video frame 2, Delta4'=T4'-T3' between video frame 4 and video frame 3.

For each of the video frames 1-4, according to the frame time stamp indicating the video frame to which the data packet belongs in the network layer protocol information of the data packet corresponding to the video frame, the frame interval of adjacent video frames can be calculated. Taking Figure 6 as an example, the frame timestamps of video frames 1-4 can be recorded as TS1, TS2, TS3, and TS4 respectively. When 2≤n≤4 and is an integer, the frame interval between video frame n and video frame n-1 Intervaln can be calculated by the difference between the frame time stamp TSn of video frame n and the frame time stamp TSn-1 of video frame n−1. When n=1, the frame interval Interval1 of video frame 1 is equal to 0 by default. In this case, according to the frame time stamps TS1, TS2, TS3, and TS4 of video frames 1-4, the frame interval Interval1=0 of video 1, the frame interval Interval2=TS2-TS1 of video frame 2 and video frame 1, The frame interval of video frame 3 and video frame 2 is Interval3=TS3-TS2, and the frame interval of video frame 4 and video frame 3 is Interval4=TS4-TS3.

According to the first packet interval and the frame interval between each video frame in the video frame 1-4 and its previous video frame, and the frame first packet jitter parameter of the previous video frame, the video frame 1-4 can be calculated Frame header packet jitter parameters for each video frame. The frame header jitter parameter of video frame 1 is equal to 0 by default. Except for video frame 1, an example of the calculation method of the frame header jitter parameters of other video frames is shown in formula (1):

Jittern＝Jittern-1*15/16+(Deltan-Intervaln)*1/16 (1)

In the formula (1), 2≤n≤4 and is an integer, Jittern represents the frame header jitter parameter of the video frame n, 15/16 and 1/16 are preset control parameters, and the values of the control parameters can be different by setting The optimal parameter value determined after comparing the calculated frame first packet jitter parameter (or frame tail packet jitter parameter) with the real frame first packet jitter (or frame tail packet jitter), the control parameter can be used to represent the previous The ratio of the value of the first packet jitter parameter (or frame end packet jitter parameter) of a video frame to the value of the frame header packet jitter parameter (or frame end packet jitter parameter) of the current video frame. Taking the first packet interval and frame interval of video frames 1-4 in Figure 6 as an example, combined with formula (1), the frame first packet jitter parameters of video frames 1-4 can be obtained as follows:

The frame header jitter parameter Jitter1=0 of video frame 1;

The frame first packet jitter parameter Jitter2=Jitter1*15/16+(Delta2-Interval2)*1/16 of video frame 2;

The frame first packet jitter parameter Jitter3=Jitter2*15/16+(Delta3-Interval3)*1/16 of video frame 3;

The frame header jitter parameter of video frame 4 is Jitter4=Jitter3*15/16+(Delta4-Interval4)*1/16.

According to the tail packet interval and frame interval between each video frame in the video frame 1-4 and its previous video frame, and the frame tail packet jitter parameter of the previous video frame, the video frame 1-4 can be calculated End-of-frame packet jitter parameters for each video frame. The frame end packet jitter parameter of video frame 1 is equal to 0 by default. Except video frame 1, an example of the calculation method of the frame end packet jitter parameter of other video frames is shown in formula (2):

Jittern’＝Jittern-1’*15/16+(Deltan’-Intervaln’)*1/16 (2)

In formula (2), 2≤n≤4 is an integer, Jittern' represents the frame end packet jitter parameter of video frame n, and 15/16 and 1/16 are preset control parameters. Taking the tail packet interval and frame interval of video frames 1-4 in Figure 6 as an example, combined with formula (2), the frame tail packet jitter parameters of video frames 1-4 can be obtained as follows:

The frame tail packet jitter parameter Jitter1'=0 of video frame 1;

The frame end packet jitter parameter Jitter2'=Jitter1'*15/16+(Delta2'-Interval2')*1/16 of the frame tail of video frame 1;

The frame end packet jitter parameter Jitter3'=Jitter2'*15/16+(Delta3'-Interval3')*1/16 of the frame tail of video frame 1;

The frame end packet jitter parameter of video frame 1 Jitter4'=Jitter3'*15/16+(Delta4'-Interval4')*1/16.

In this way, the calculation of the jitter parameters of the first packet of the frame and the jitter parameters of the last packet of the frame can be completed.

FIG. 7 and FIG. 8 respectively show examples of calculation manners of a frame packet loss parameter and a frame disorder parameter according to an embodiment of the present application.

In a possible implementation manner, the streaming media layer protocol information of the data packet includes a streaming media layer sequence number indicating a sequence of the data packet in the video frame to which it belongs. The network layer protocol information of the data packet includes the packet time indicating the time of the data packet, and the frame quality parameter includes a frame loss parameter and a frame out-of-sequence parameter, and the frame loss parameter indicates the loss situation of the data packet of the video frame , the frame out-of-sequence parameter represents the out-of-order situation of the data packets of the video frame,

Step S22, including:

For each frame of the at least one video frame, determine the frame packet loss parameter of the video frame according to the streaming layer sequence number of the data packet corresponding to the video frame;

For each frame of the at least one video frame, sort the data packets corresponding to the video frame according to the packet time sequence of the data packets corresponding to the video frame, and obtain the corresponding sorting result of the video frame, according to the The sorting result and the streaming layer sequence number of the data packet corresponding to the video frame determine the frame out-of-sequence parameter of the video frame.

For example, the frame loss situation can be determined from the perspective of the streaming media layer of the data packet, and it can be realized by counting the sequence numbers of the streaming media layer and analyzing them. For example, as shown in FIG. The media layer serial numbers S _a -S _i can be 11, 12, 13, 15, 16, 17, 18, 19, 20 respectively, and the stream media layer serial numbers S _a -S _i obtained according to the statistics can be analyzed to determine the frame loss Condition. For an exemplary implementation thereof, reference may be made to the description below.

Frame out-of-sequence can be determined from the perspective of the streaming media layer and the network layer of the data packet. The sorting result is obtained by sorting the data packets according to the packet time, and combined with the sorting result and the stream media layer sequence number of the statistical data packet to analyze and determine. For example, as shown in Figure 8, the streaming media layer sequence numbers _SA - _SK of the data packet AK corresponding to the video frame 3 are 21, 22, 23, 25, 26, 27, 28, 24, 29, 30, 31 respectively , the sorting result of the data packet AK corresponding to the video frame 3 can be, for example, P={A, B, C, D, E, F, G, H, I, J, K}, according to the sorting result P and the statistically obtained flow The serial numbers S _A _-SK of the media layer can analyze and determine the packet loss of the frame. For an exemplary implementation thereof, reference may be made to the description below.

In a possible implementation manner, for each frame of the at least one video frame, determining the frame packet loss parameter of the video frame according to the streaming media layer sequence number of the data packet corresponding to the video frame includes:

For each frame of the at least one video frame, according to the streaming media layer serial number of the data packet corresponding to the video frame, determine the maximum value, the minimum value and the streaming media layer serial number of the data packet corresponding to the video frame Layer sequence number, according to the maximum value and the minimum value of the streaming media layer sequence number, determine the theoretical value of the data packet number corresponding to the video frame, according to the theoretical value of the data packet number corresponding to the video frame and the The difference in the number of streaming media layer sequence numbers of the data packets corresponding to the video frame determines the frame packet loss parameter of the video frame.

According to the streaming media layer sequence number of the data packet corresponding to the sorting result and the video frame, determine the frame out-of-sequence parameter of the video frame, including:

When the value of the streaming media layer serial number of the current data packet is greater than the streaming media layer serial number of the data packet before the current data packet in the sorting result, according to the sorting serial number of the current data packet in the sorting result, the current data The minimum value of the streaming layer sequence number of the packet and the streaming layer sequence number of the data packet corresponding to the video frame determines the frame disorder parameter corresponding to the video frame.

For example, as shown in Figure 7, the streaming layer sequence numbers S _a -S _i of the data packets ai corresponding to video frame 2 are 11, 12, 13, 15, 16, 17, 18, 19, 20 respectively, according to The streaming layer sequence number of the data packet ai corresponding to the video frame 2 can determine the maximum value max=20 of the streaming layer sequence number of the data packet ai corresponding to the video frame 2, the minimum value min=11, and the streaming layer sequence number The number is 9. According to the maximum value 20 and the minimum value 11 of the streaming media layer sequence number of the video frame 2, the theoretical value q=max-min+1=20-11+1=10 of the corresponding data packet number of the video frame 2 can be determined, and The difference between the stream media layer serial number 9 of the data packet corresponding to the video frame is equal to 1, therefore, there is a packet loss in the video frame 2, and the number of packet loss is 1.

As shown in Figure 8, the streaming media layer sequence numbers _SA - _SK of the data packet AK corresponding to the video frame 3 are 21, 22, 23, 25, 26, 27, 28, 24, 29, 30, 31 respectively, sorted The result can be, for example, P={A, B, C, D, E, F, G, H, I, J, K}, according to the streaming media layer serial number S _A -S _K of the data packet AK corresponding to the video frame 3 , it can be determined that the numerical value (24) of the streaming media layer sequence number of the data packet H is less than the numerical value (28) of the streaming media layer serial number of the data packet G before the data packet H in the sorting result P, and it can be determined that the video frame 3 is out of order In the case of order, and the data packet that causes disorder is data packet H. The ordering of the data packet H in the sorting result P can be calculated according to the value 24 of the streaming media layer serial number SH of the data packet _H and the minimum value MIN=21 of the streaming media layer serial number of the data packet AK corresponding to the video frame 3 The theoretical value of the serial number _SH -MIN+1=4, and then according to the sorting serial number 8 of the data packet H determined by the sorting result P and the theoretical value 4 of the sorting serial number of the data packet H in the sorting result P, the disordered depth is calculated 8-4=4. Wherein, when there is only one out-of-order data packet, the out-of-order depth of the out-of-order data packet can be used as the value of the frame out-of-order parameter corresponding to the video frame. When there are multiple out-of-order data packets, the sum of the out-of-order depth of each out-of-order data packet can be averaged according to the number of out-of-order data packets to obtain the average out-of-order depth, and the average out-of-order depth is used as The value of the frame out-of-order parameter corresponding to this video frame.

In this way, the frame packet loss parameter and the frame disorder parameter can be calculated.

In step S23, for the different video frames identified in step S21, according to the network layer protocol information and the transport layer protocol information of the data packets corresponding to the different video frames, the packet quality parameters of the data packets are obtained to reflect the data packet quality, Wherein, the packet quality parameters may include, for example, one or more of packet interval parameters, packet jitter parameters, out-of-sequence parameters, packet loss parameters, retransmission parameters, delay parameters, IP packet fragmentation parameters, and checksum error parameters. . This process can be implemented based on the existing technology, and the following will give exemplary descriptions for different packet quality parameters and their calculation methods.

In a possible implementation manner, the packet quality parameter of each data packet corresponding to each frame in the at least one video frame is determined according to the network layer protocol information and the transport layer protocol information of the data packet, and the packet quality parameter includes : The packet interval parameter that represents the time interval between the current data packet and the previous data packet, the packet jitter parameter that represents the delay jitter between the current data packet and the previous data packet, the out-of-order parameter that represents the disordered state of the data packet, and the data packet The packet loss parameter for packet loss, the retransmission parameter for data packet retransmission, the delay parameter for data packet delay, the IP packet fragmentation parameter for IP packet fragmentation, and the If the checksum of the packet is incorrect one or more of the checksum error parameters.

The calculation method of each package quality parameter is introduced respectively below.

The packet interval parameter can be determined according to the packet time in the network layer protocol information of the data packet, and its calculation method can refer to the example of the calculation method of the frame interval above. For example, assuming that video frame 4 includes 5 data packets, for each data packet in data packets 1-5, according to the packet time indicating the time of the data packet in the network layer protocol information of the data packet, the packet interval can be calculated parameter. Among them, when 2≤m≤5 and it is an integer, the packet interval parameters of data packet m and data packet m-1 can be calculated by the difference between the packet time of data packet m and the packet time of data packet m-1, m=1 When , the packet interval parameter of data packet 1 is equal to 0 by default.

The packet jitter parameter can be determined according to the packet time in the network layer protocol information of the data packet and the preset control parameters, and its calculation method can refer to the example of the calculation method of the first frame packet jitter parameter or the frame end packet jitter parameter above. For example, assume that at least one video frame includes video frames 1-4. For each video frame in the video frame 1-4, according to the network layer protocol information of the data packet corresponding to the video frame, the packet time indicating the time of the data packet can calculate the packet interval parameter. According to each data packet corresponding to each video frame and the packet interval parameter of the previous data packet, the packet jitter parameter of the previous data packet and the preset control parameter, the packet jitter parameter of each data packet can be calculated. For example, for video frame 4, video frame 4 may include 5 data packets, for example, when 2≤m≤5 and is an integer, the packet jitter parameters of data packet m and data packet m-1 can be passed through data packet m and data packet m The packet interval parameter of -1, the packet jitter parameter of data packet m-1, and the control parameters (for example, 15/16 and 1/16) are determined.

The out-of-order parameter can be determined according to the network layer serial number in the network layer protocol information of the current data packet and the previous data packet and the transport layer serial number in the transport layer protocol information. The value of the parameter can be equal to 1, and when the current data packet is not an out-of-order data packet, the value of the out-of-order parameter can be equal to 0. For example, the data packets are sorted according to the packet time. If the transport layer sequence number and the network layer sequence number of the current data packet are reduced compared with the transport layer sequence number and the network layer sequence number of the previous data packet, it indicates that the current data packet Packets are out-of-order packets.

The packet loss parameter can be determined according to the network layer serial number in the network layer protocol information of the data packet and the transport layer serial number, checksum, and confirmation message serial number in the transport layer protocol information. The current data packet is a lost data packet When , the value of the packet loss parameter may be equal to 1, and when the current data packet is not a lost data packet, the value of the packet loss parameter may be equal to 0. For example, to calculate the packet loss parameters of the data packets corresponding to the video stream using the TCP protocol, the data packets can be sorted according to the packet time. If compared with the transport layer sequence number and the network layer sequence number of the previous data packet, the current data packet The serial number of the transport layer decreases, the serial number of the network layer increases, and before the packet time of the current data packet, in the video buffer, the network layer serial number, the serial number of the transport layer, and the checksum of the current data packet have not appeared The same data packet, but a data packet with the same sequence number as the confirmation message of the current data packet has appeared, which indicates that the current data packet is a lost data packet. For another example, to calculate the packet loss parameter of the data packet corresponding to the video stream using the UDP protocol, the network layer sequence number of the data packet can be directly used to determine whether the packet is lost. The data packets can be sorted according to the packet time, and if the network layer sequence number of the current data packet decreases compared with the network layer sequence number of the previous data packet, it indicates that the current data packet is a lost data packet.

The retransmission parameters can be determined according to the network layer sequence number in the network layer protocol information of the data packet and the transport layer sequence number, checksum, and confirmation message sequence number in the transport layer protocol information. The current data packet is a retransmitted data packet , the value of the retransmission parameter may be equal to 1, and when the current data packet is not a retransmission data packet, the value of the retransmission parameter may be equal to 0. For example, to calculate the retransmission parameters of the data packets corresponding to the video stream using the TCP protocol, the data packets can be sorted according to the packet time, if compared with the transport layer sequence number and the network layer sequence number of the previous data packet, the current data packet The serial number of the transport layer decreases, the serial number of the network layer increases, and before the packet time of the current data packet, in the video buffer, the same as the network layer serial number, transport layer serial number, and checksum of the current data packet appear , it indicates that the current data packet is a retransmitted data packet. Another example is to calculate the retransmission parameters of the data packets corresponding to the video stream using the UDP protocol, and the data packets can be sorted according to the packet time. If the network layer sequence number and checksum of the current data packet are the same, and the packet time A data packet earlier than the packet time of the current data packet indicates that the current data packet is a retransmitted data packet.

The time delay parameter can be determined according to the packet time in the network layer protocol information of the data packet and the confirmation message sequence number in the transport layer protocol information. For example, the time delay parameter of the data packet corresponding to the video stream using the TCP protocol can be calculated according The confirmation message sequence number of the current data packet Find the data packet with the same confirmation message sequence number as the current data packet, and determine the delay according to the packet time of the data packet and the difference between the packet time of the found data packet and the current data packet The numeric value of the parameter.

The IP packet fragmentation parameter can be determined according to the fragmentation flag in the network layer protocol information of the data packet. For example, when there is a fragmentation flag in the network layer protocol information, it means that the data packet has an IP The value of the fragment parameter can be equal to 1. When there is no fragment flag in the network layer protocol information, it means that there is no IP packet fragmentation in the data packet, and the value of the IP packet fragment parameter can be equal to 0.

The checksum error parameter can be determined according to the checksum in the transport layer protocol information of the data packet and the checksum value calculated according to the prior art. If the two values are inconsistent, it means that the current data packet has a checksum error. The value of the checksum error parameter is equal to 1 when there is a checksum error. The value of the checksum error parameter is equal to zero when there are no checksum errors. Wherein, when the value of the checksum in the transport layer protocol information is an invalid value (the value is 0), the checksum in the transport layer protocol information and the calculated checksum value are not compared.

In this way, various packet quality parameters can be calculated. Those skilled in the art should understand that the types and calculation methods of the packet quality parameters should not be limited to the above examples. As long as the quality of the data packet can be reflected by the packet quality parameter, the embodiment of the present application does not limit the type and calculation method of the packet quality parameter.

In step S24, for the frame quality parameter calculated in step S22 and the packet quality parameter calculated in step S23, abnormal determination of the parameters is performed in combination with the frame quality parameter threshold and the packet quality parameter threshold respectively.

In a possible implementation, step S24 includes:

For each frame of the at least one video frame, among the frame quality parameters of the video frame, the frame quality parameter whose value is greater than the frame quality parameter threshold is determined as the abnormal frame quality parameter;

For each frame of the at least one video frame, among the packet quality parameters of the data packets corresponding to the video frame, the packet quality parameter whose value is greater than the packet quality parameter threshold is determined as the abnormal packet quality parameter.

For example, the frame quality parameter threshold and the packet quality parameter threshold can be determined in real time by using a preset fixed value or by using an AI model. An exemplary method for determining the threshold of the frame quality parameter and the threshold of the packet quality parameter by using the AI model is introduced below.

For example, each frame quality parameter may correspond to a frame quality parameter threshold; and each packet quality parameter may correspond to a packet quality parameter threshold. When the AI model determines a frame quality parameter threshold (or packet quality parameter threshold) corresponding to a frame quality parameter (or packet quality parameter), the data used can be calculated during a period of time before the current moment, and multiple video Multiple values of this type of frame quality parameter corresponding to frames (or multiple values of this type of packet quality parameter corresponding to multiple data packets). For example, using the frame header jitter parameters calculated for a period of time before the current moment (hereinafter referred to as the historical frame header jitter parameters) as the parameter samples of the AI model, the AI model can be based on the statistics of the historical frame header jitter parameters case, fitting generates a normal distribution probability curve P(x) of the historical frame first packet jitter parameter x, and uses the cumulative distribution function F(a) of the normal distribution probability curve (F(a)=P(x< = a)) Calculate the frame quality parameter threshold a for the frame header jitter parameter. FIG. 9 shows an example of a normal distribution probability curve generated according to an embodiment of the present application. As shown in FIG. 9 , the histogram shows the numerical distribution of the jitter parameter of the first packet of the historical frame, and the curve is a normal distribution probability curve P(x) fitted according to the distribution of the jitter parameter of the first packet of the historical frame.

An abnormality threshold is set in advance, and when the cumulative distribution function F(a) is equal to the abnormality threshold, the value of a can be the corresponding frame quality parameter threshold. If the frame header packet jitter parameter calculated after the current moment is greater than the threshold, it can be determined to be an abnormal frame header packet jitter parameter, that is, to be determined to be an abnormal frame quality parameter. Taking the normal distribution probability curve in Fig. 9 as an example, the abnormality threshold is preset as 75%, and calculated according to the cumulative distribution function of the normal distribution probability curve, when the value of the cumulative distribution function F(a) is equal to 75%, The corresponding value of a may be equal to 100, for example. In this case, the normal range of the value of the frame header jitter parameter is 0-100. Fig. 10 shows an example of realizing abnormal condition determination of parameters according to an embodiment of the present application. As shown in Figure 10, the value of a frame header jitter parameter that needs to be determined to be abnormal is 240, and the value is greater than 100. The frame header jitter parameter is an abnormal frame quality parameter, so that the abnormality of the parameter can be determined.

When the frame quality parameter (or packet quality parameter) that needs to be determined to be abnormal is calculated, the frame quality parameter (or packet quality parameter threshold) obtained by the AI model can be used to determine the frame quality parameter (or packet quality parameter) that needs to be determined to be abnormal. or packet quality parameter) is an abnormal frame quality parameter (or abnormal packet quality parameter). Further, the parameter samples that have been determined to be abnormal frame quality parameters (or abnormal packet quality parameters) in the parameter samples can also be cropped, so that the frame quality parameter threshold (or packet quality parameter threshold) is also updated as the parameter samples change .

For example, the method described above is used to determine the abnormality of the frame quality parameter, and the determined abnormality of the frame quality parameter can be shown in Table 1 below, for example.

Table 1

As shown in Table 1, the value of 1 for abnormal jitter at the beginning of the frame, abnormal jitter at the end of the frame, abnormal packet loss, and abnormal frame disorder indicates that the corresponding frame quality parameter of the frame is an abnormal frame quality parameter, this frame is also an abnormal frame. Then according to Table 1, it can be seen that video frame 1 has no frame quality parameter abnormality, the first frame packet jitter parameter of video frame 2 is abnormal, the frame out-of-sequence parameter is abnormal, the frame first packet jitter parameter of video frame 3 is abnormal, the frame tail packet jitter parameter is abnormal, The frame loss parameter is abnormal, and the frame quality parameter of video frame 4 is abnormal.

The method described above is used to determine the abnormal situation of the packet quality parameter, and the determined abnormal situation of the packet quality parameter can be shown in Table 2 below, for example.

Table 2

In Table 2, gNB, UDG, and EoR represent 5G base stations, unified distributed gateways, and switches using a certain deployment method, respectively. Among them, abnormal packet loss, out-of-sequence abnormality, packet jitter abnormality, packet interval abnormality, retransmission abnormality, delay abnormality, IP packet fragmentation abnormality, and checksum error abnormality have a value of 1 Indicates that the package quality parameter is an abnormal package quality parameter, and the package is also an abnormal package. According to Table 2, it can be seen that on the 5G base station gNB, the packet quality parameters of

data packets

1, 3, 4, and 5 are not abnormal, and the packet jitter parameters of data packet 2 are abnormal; on the unified distributed gateway UDG, the packet jitter parameters of data packet 1 are abnormal , the packet loss parameter and delay parameter of data packet 2 are abnormal, and the packet loss parameter and packet jitter parameter of data packet 3 are abnormal; The packet loss parameter is abnormal, and the packet loss parameter and packet jitter parameter of packet 3 are abnormal.

After the abnormal frame quality parameters and abnormal packet quality parameters are obtained in step S2, in step S3, it can be determined according to the relationship between the abnormal frame quality parameters, the abnormal packet quality parameters and the preset conditions, which causes the abnormal video quality of the video playback terminal. Causes and equipment. Fig. 11 shows an exemplary schematic diagram of the correspondence between abnormal frame quality parameters, abnormal packet quality parameters, causes of abnormal video quality at the video player end, and devices according to an embodiment of the present application.

As shown in FIG. 11 , the packet quality parameter corresponding to the frame quality parameter can be found by using the correspondence between the video frame and the data packet, so that the data packet to which the packet quality parameter belongs is the data packet under the video frame to which the frame quality parameter belongs. The frame quality parameter and its corresponding packet quality parameter usually have a certain correlation. For example, when the packet loss parameter of a certain data packet is abnormal, the situation of the frame packet loss parameter of the video frame to which the data packet belongs can be checked. When the out-of-sequence parameter of a data packet is abnormal, you can check the frame out-of-sequence parameter of the video frame to which the data packet belongs. When the packet interval parameter or packet jitter parameter of a certain data packet is abnormal, it may cause the data packet to fail to arrive at the expected time of decoding, which may cause the video frame to which the data packet belongs to have a large jitter. You can view the video frame to which the data packet belongs The case of the first frame packet jitter parameter and the frame end packet jitter parameter. When the retransmission parameters or delay parameters of a data packet are abnormal, the sending of the data packet may be slowed down, which may cause the video frame to which the data packet belongs to shake greatly. You can view the video frame to which the data packet belongs. The situation of the jitter parameters of the first packet of the frame and the jitter parameters of the last packet of the frame. The IP packet fragmentation parameter and the checksum error parameter of the data packet are also associated with the jitter of the video frame to which the data packet belongs, therefore. When the IP packet fragmentation parameters or checksum error parameters of a data packet are abnormal, you can check the frame header jitter parameters and frame tail packet jitter parameters of the video frame to which the data packet belongs.

From the above description, we can see that there are many kinds of abnormal frame quality parameters and abnormal packet quality parameters. You can first set the conditions corresponding to the cause of abnormal video quality on the video playback end and the device, and then judge the abnormal frame quality parameters. and whether the abnormal packet quality parameters meet the corresponding conditions to determine the cause and equipment that cause the abnormal video quality of the video player.

The conditions that may be set and the causes and devices that lead to abnormal video quality at the video player end determined according to the conditions will be described exemplarily below in conjunction with FIG. 11 .

In a possible implementation, step S3 includes:

For any video frame, when the frame quality parameter of the video frame and the packet quality parameter of the data packet corresponding to the video frame meet the first condition, it is determined that the cause of the abnormal video quality at the video playback end is a video encoding problem, and it is determined that the The device with abnormal video quality at the video playback end is the device used for video encoding;

Wherein, the first condition includes:

The frame packet loss parameter of the video frame is abnormal and the packet loss parameter of the data packet corresponding to the video frame is normal, or

The frame out-of-order parameter of the video frame is abnormal and the out-of-order parameter of the data packet corresponding to the video frame is normal.

For example, as shown in Figure 11, if the frame loss parameter in the frame quality parameter is abnormal and the packet loss parameter in the packet quality parameter is normal, or the frame disorder parameter in the frame quality parameter is abnormal and the disorder parameter in the packet quality parameter is normal , it can be determined that the abnormal frame quality parameter and the abnormal packet quality parameter meet the first condition, and then determine that the cause of the video quality problem occurs during the video encoding process, the video encoding process is dominated by the equipment used for video encoding, and the video encoding There is one and only one device used, so the only device that can be sure to cause the video encoding problem is the device used for video encoding.

In practical applications, it can be set so that as long as the frame quality parameter and packet quality parameter can be determined to meet any possibility in the first condition, the cause and device that cause the abnormal video quality at the video playback end can be directly determined without having to determine the frame quality There is another possibility whether the parameter and the packet quality parameter satisfy the remaining ones in the first condition. Alternatively, it may also be set to determine whether the frame quality parameter and the packet quality parameter satisfy each possibility in the first condition, and after determining that the frame quality parameter and the packet quality parameter satisfy at least one possibility in the first condition, then determine the result Reasons and equipment for abnormal video quality at the video playback end. Those skilled in the art should understand that both of the above two setting methods can be used to determine the cause and equipment that cause the abnormal video quality of the video playback terminal, and this application does not limit this.

In a possible implementation, step S3 includes:

For any video frame, when the frame quality parameter of the video frame and the packet quality parameter of the data packet corresponding to the video frame meet the second condition, it is determined that the cause of the abnormal video quality at the video playback end is a terminal parameter problem, and it is determined that the cause The device with abnormal video quality at the video playback end is the device used for video playback;

Wherein, the second condition includes:

Any parameter in the frame header jitter parameter and frame tail packet jitter parameter of the video frame is abnormal and any parameter in the IP packet fragmentation parameter and checksum error parameter of the data packet corresponding to the video frame is abnormal .

Since the protocol information of the data packet is obtained at the network device and the video player, the frame quality parameter and the packet quality parameter can reflect the abnormal situation in the transmission process, and the abnormality of the first frame packet jitter parameter/frame end packet jitter parameter indicates that the video The delay jitter of the first/last data packet in the frame is too large during transmission. If the IP packet fragmentation parameter of the data packet in the video frame is abnormal at this time, it means that the data packet has experienced fragmentation during transmission. Therefore, it can be determined that the parameter configuration of the video playback terminal fails to meet the requirements for data packets to be transmitted without fragmentation; The video playback end successfully completes the verification, so it can be determined that the parameter configuration of the video playback end fails to meet the requirements for successful verification of the data packet; based on the above reasons, the second condition, that is, the first frame packet jitter parameter and the end of the frame of the video frame Any parameter in the packet jitter parameters is abnormal and any parameter in the IP packet fragmentation parameter and the checksum error parameter of the data packet corresponding to the video frame is abnormal, which can accurately reflect the terminal parameter problem.

For example, as shown in Figure 11, if the frame first packet jitter parameter/frame end packet jitter parameter in the frame quality parameter is abnormal and the IP packet fragmentation parameter/checksum error parameter in the packet quality parameter is normal, it can be determined that the abnormal The frame quality parameter and the abnormal packet quality parameter meet the second condition, and then it is determined that the cause of the video quality problem appears on the playback parameters during the video playback process, the video playback process is dominated by the device used for video playback, and the video playback There is only one device used (that is, the video playback terminal), so it can be uniquely determined that the device that causes the terminal parameter problem is the device used for video playback.

In practical applications, it can be set so that as long as the frame quality parameter and packet quality parameter can be determined to meet any of the possibilities in the second condition, the cause and device that cause the abnormal video quality at the video playback end can be directly determined without having to determine the frame quality parameters and packet quality parameters satisfy the remaining possible cases in the second condition. Alternatively, it may also be set to determine whether the frame quality parameter and the packet quality parameter satisfy each possibility in the second condition, and after determining that the frame quality parameter and the packet quality parameter satisfy at least one possibility in the second condition, then determine the result Reasons and equipment for abnormal video quality at the video playback end. Those skilled in the art should understand that both of the above two setting methods can be used to determine the cause of the abnormal video quality of the video player and the determination of the device, which is not limited in the present application.

In a possible implementation, step S3 includes:

For any video frame, when the frame quality parameter of the video frame and the packet quality parameter of the data packet corresponding to the video frame meet the third condition, it is determined that the cause of the abnormal video quality at the video playback end is a network transmission problem;

Wherein, the third condition includes:

The frame packet loss parameter of the video frame is abnormal and the packet loss parameter of the data packet corresponding to the video frame is abnormal, or

The frame out-of-order parameter of the video frame is abnormal and the out-of-order parameter of the data packet corresponding to the video frame is abnormal, or

Any one of the frame header jitter parameters and frame tail packet jitter parameters of the video frame is abnormal and any of the packet jitter parameters, packet interval parameters, retransmission parameters, and delay parameters of the data packets corresponding to the video frame A parameter exception.

For example, as shown in Figure 11, if the frame loss parameter in the frame quality parameter is abnormal and the packet loss parameter in the packet quality parameter is abnormal, or the frame disorder parameter in the frame quality parameter is abnormal and the disorder parameter in the packet quality parameter is abnormal , or if the frame first packet jitter parameter/frame end packet jitter parameter in the frame quality parameter is abnormal and the packet jitter parameter/packet interval parameter/retransmission parameter/delay parameter in the packet quality parameter is abnormal, you can determine the abnormal frame quality parameter and all The aforementioned abnormal packet quality parameters meet the third condition, and then it is determined that the cause of the abnormal video quality occurs during network transmission, and the abnormal video quality is caused by network equipment.

Table 3 shows an example of determining that the cause of the video quality problem is a network transmission problem according to the embodiment of the present application.

table 3

Referring to Table 3, a frame timestamp identifies a unique frame. The packet information that needs to be viewed corresponding to a frame may include packet time, network layer sequence number, transport layer sequence number, checksum, and abnormality of packet quality parameters. For abnormal packet quality parameters, according to the association relationship between frame quality parameters and packet quality parameters, in the "Analysis Result" column, the analysis results that lead to video quality problems can be given. For the example of table 3, because the frame packet loss parameter of video frame 1 is abnormal, and the packet loss parameter of data packet 2 corresponding to video frame 1 is abnormal, and the packet loss parameter of data packet 3 corresponding to video frame 1 is abnormal, therefore satisfying the third conditions, it can be determined that the cause of the video quality problem is a network transmission problem.

In practical applications, it can be set so that as long as the frame quality parameter and packet quality parameter can be determined to meet any possibility in the third condition, the cause of the abnormal video quality at the video playback end can be directly determined without having to determine the frame quality parameter and Whether the packet quality parameter satisfies the remaining possibilities in the third condition. Alternatively, it may also be set to determine whether the frame quality parameter and the packet quality parameter satisfy each possibility in the third condition, and after determining that the frame quality parameter and the packet quality parameter satisfy at least one possibility in the third condition, then determine the result The cause of abnormal video quality on the video player side. Those skilled in the art should understand that both of the above two setting manners can be used to determine the cause of the abnormal video quality at the video playback end, which is not limited in the present application.

Since the network transmission process may use multiple network devices, it is also possible to further determine the specific device causing the terminal parameter problem.

In a possible implementation, after determining that the cause of the abnormal video quality at the video playback end is a network transmission problem, step S3 further includes:

According to the network layer protocol information of the data packet, determine the network layer serial number of the data packet corresponding to the abnormal packet quality parameter;

According to the transport layer protocol information of the data packet, determine the packet time, transport layer sequence number, and checksum of the data packet corresponding to the abnormal packet quality parameter;

receiving network topology data on the network device, and determining the transmission path of each data packet in the network according to the network topology data;

According to the transmission path of the data packet corresponding to the abnormal packet quality parameter described in the network, find the data packet that satisfies the conditions of packet time, network layer serial number, transport layer serial number, and checksum in each network device on the transmission path ;

According to the found packet quality parameters of the data packets satisfying the conditions, the network equipment causing the abnormal video quality of the video playback terminal is determined from among the network equipments on the transmission path of the data packets.

The packet quality parameter of the data packet on each device can indicate the impact of the device on the transmission of the data packet, and by finding the corresponding data packets in each network device on the transmission path, the statistics can be calculated using the found data packets During the transmission of the packet, the impact of different network devices on the transmission of the data packet, and determine the specific device that causes the abnormal video quality of the video player.

For example, after determining that the abnormal frame quality parameter and the abnormal packet quality parameter meet the third condition, according to the network layer protocol information, the network layer sequence number of the data packet corresponding to the abnormal packet quality parameter can be determined, and according to the transport layer The protocol information can determine the packet time, transport layer sequence number, and checksum of the data packet corresponding to the abnormal packet quality parameter. According to the network topology data collected by the topology data collection device, the transmission path of the data packet corresponding to the abnormal packet quality parameter on the network device can be obtained. For example, referring to Table 3, the abnormal packet quality parameter can be, for example, the packet loss parameter of data packet 2 and the packet loss parameter of data packet 3. According to the network topology data, it can be determined, for example, that the transmission path of data packet 2 and data packet 3 can be gNB→UDG→EoR.

According to the transmission path of the data packet corresponding to the abnormal packet quality parameter in the network, you can find the data packet that meets the conditions of the packet time, network layer serial number, transport layer serial number, and checksum in each network device on the transmission path. By comparing the abnormal packet quality parameters of the data packets that meet the conditions, the network device that causes the abnormal video quality of the video playback terminal is determined. For example, see Table 3, based on packet time (8:14:12.008), network layer sequence number (48160), transport layer sequence number (3480724955), checksum (1231221433) of packet 2, and packet 3 Packet time (8:14:12.019), network layer sequence number (48162), transport layer sequence number (3480726183), checksum (2134423523), and find a packet that meets the conditions.

In a possible implementation, the packet time, the network layer serial number, the transport layer serial number, and the checksum satisfying conditions include:

The absolute value of the difference between the packet time and the packet time of the data packet corresponding to the abnormal packet quality parameter is less than a preset threshold, and the network layer sequence number, the transport layer sequence number, the checksum and the abnormal packet quality parameter The network layer sequence number, transport layer sequence number, and checksum of the corresponding data packets are all the same.

For example, combining Table 2 and Table 3, according to the network layer sequence number (48160), transport layer sequence number (3480724955), checksum (1231221433) of data packet 2 in Table 3, it can be found in Table 2 that the network layer sequence The data packet whose serial number is equal to 48160, the serial number of the transport layer is equal to 3480724955, and the checksum is equal to 1231221433 can find the data packet 3 on the gNB, the data packet 2 on the UDG, and the data packet 2 on the EoR that meet the requirements, and then further determine to find Whether the absolute value of the difference between the packet time of the data packet and the packet time (8:14:12.008) of data packet 2 in Table 3 is less than the preset threshold, for example, the value of the preset threshold can be equal to 0.01, in this case, gNB The absolute value of the packet time difference between packet 3 on UDG, packet 2 on UDG, and packet 2 on EoR and packet 2 in Table 3 is less than the preset threshold. Packet 3 on gNB, packet 2 on UDG The data packet 2 and the data packet 2 on the EoR may be the data packets found for the data packet 2 in Table 3 that meet the conditions.

According to the network layer serial number (48162), transport layer serial number (3480726183), and checksum (2134423523) of data packet 3 in Table 3, it can be found in Table 2 that the network layer serial number is equal to 48162, and the transport layer serial number is equal to 3480726183, the data packet whose checksum is equal to 2134423523, you can find the data packet 5 on the gNB, the data packet 3 on the UDG and the data packet 3 on the EoR that meet the requirements, and then further determine the packet time of the found data packet and Table 3 Whether the absolute value of the difference between the packet time (8:14:12.019) of data packet 3 is less than the preset threshold, for example, the value of the preset threshold can be equal to 0.01, in this case, the data packet 5 on the gNB, the The absolute value of the packet time difference between data packet 3 and data packet 3 on EoR and data packet 3 in Table 3 is less than the preset threshold, data packet 5 on gNB, data packet 3 on UDG and data packet on EoR 3 may be the packet found for packet 3 in Table 3 that satisfies the condition.

According to the found packet quality parameters of the data packets satisfying the conditions, the specific network device causing the abnormal video quality at the video playback end can be determined. FIG. 12 shows an example of determining a specific network device that causes abnormal video quality at a video player end according to an embodiment of the present application.

For example, as shown in Figure 12, A, B, and C are gNB, UDG, and EoR respectively, and the data packets on gNB, UDG, and EoR that meet the conditions found for data packet 2 in Table 3 (hereinafter referred to as A , B, and C devices) as an example, according to the network topology data, it can be determined that the transmission path of the data packet is gNB→UDG→EoR. The abnormal packet quality parameter of data packet 2 in Table 3 is the packet loss parameter. When the packet loss parameters of the data packets on devices A, B, and C are all abnormal, it can be determined that the packet loss introduced by device A (gNB) leads to The video quality is abnormal; the packet loss parameters of data packets on B and C are abnormal, and when the packet loss parameters of data packets on device A are normal, it can be determined that device B (UDG) introduces packet loss, which leads to abnormal video quality; device C The packet loss parameters of the above data packets are abnormal, and when the packet loss parameters of the data packets on devices A and B are normal, it can be determined that device C (EoR) introduces packet loss, which leads to abnormal video quality.

Theoretically, according to the transmission path of gNB→UDG→EoR, if the packet loss parameters of device A are abnormal, the packet loss parameters of devices B and C must be abnormal; if the packet loss parameters of device B are abnormal, the packet loss parameters of device C must be abnormal. Must be abnormal. However, when collecting protocol information through probes, collection errors may occur, causing errors in the protocol information, which in turn may cause errors in the packet quality parameters determined according to the protocol information. Therefore, according to different network devices found on the transmission path When determining the network device that causes the video quality problem of the data packets that meet the conditions, the influence of the protocol information collection error can be further excluded. For example, if the packet loss parameters of data packets on devices A and B are abnormal, and the packet loss parameters of data packets on device C are normal, it can be considered that there is an error when collecting protocol information from device C, and it can be determined that device A has introduced packet loss, and then The video quality is abnormal; the packet loss parameters on devices A and C are abnormal, and when the packet loss parameters on device B are normal, it can be considered that there is an error when collecting protocol information from device B, and it can be determined that device A introduced Packet loss, resulting in abnormal video quality.

There are other examples of packet loss parameters on devices A, B, and C. For example, the packet loss parameters on devices A, B, and C are all normal, and the packet loss parameters on device A are abnormal. , Examples of normal packet loss parameters on devices B and C, abnormal packet loss parameters on device B, and normal packet loss parameters on devices A and C, because these examples have a very low probability of occurrence, It will not be described in detail here.

In this way, different situations of protocol information collection errors on the device and data packet loss during transmission can be distinguished, and the accuracy of determining the device that causes abnormal video quality at the video playback end can be improved.

After comparative analysis using the above method, the comparative results shown in Table 4 can be obtained.

Table 4

See Table 4, wherein each data packet (packet 2, packet 3) with abnormal packet quality parameters is uniquely identified by a network layer sequence number, a transport layer sequence number and a checksum. The abnormal packet quality parameter is a packet quality parameter that needs to be compared with the found data packets. The packet loss represents the abnormality of the abnormal packet quality parameters that need to be compared on the device. 0 means that the packet quality parameters are normal, and 1 means that the packet quality parameters are abnormal. Combining with FIG. 11 , it can be determined that the network device that causes the abnormality of data packet 2 and data packet 3 in Table 3 and then causes abnormal video quality is UDG.

Among them, in the example of Table 4, for different data packets, the determined network devices that cause abnormal video quality are the same, and in other application scenarios, due to errors, it may also occur There are many cases of network devices with abnormal video quality. In this case, the determined multiple network devices that cause abnormal video quality may be further screened to obtain a network device that is most likely to cause abnormal video quality.

In a possible implementation manner, when there are multiple data packets corresponding to the abnormal packet quality parameter, multiple groups of data packets satisfying the conditions are found,

According to the packet quality parameters of the found data packets that meet the conditions, from each network device on the transmission path of the data packet, determine the network device that causes the abnormal video quality of the video player, including:

According to the packet quality parameters of multiple sets of data packets that meet the conditions, from each network device on the transmission path of the data packet, repeatedly determine the network device that may cause abnormal video quality at the video playback end;

Among the network devices that may cause abnormal video quality at the video playback end, the network device with the most occurrences is counted as the network device that causes abnormal video quality at the video playback end.

For example, according to the statistics of the number of occurrences of all devices in the "device causing exception" column under all abnormal video frames, the network device most likely to cause abnormal video quality is determined. Combined with Table 4, there are 2 times of abnormal packet loss parameters caused by UDG, and 0 times of abnormal packet loss parameters caused by other devices, so the network device most likely to cause abnormal video quality is UDG. If in another example, there are 2 packet loss parameter exceptions caused by UDG, 5 packet loss parameter exceptions caused by EoR, and 0 packet loss parameter exceptions caused by gNB, then the network with the greatest possibility of video quality abnormality The device is EoR.

As shown in FIG. 13 , in a possible implementation, the present application also proposes an apparatus for determining video quality problems, the apparatus including: an information receiving module 130, configured to receive protocol information on the video playback terminal and network equipment , the network device is used to transmit the video data to the video player, the protocol information indicates the sending mode of the video frame, and the video frame includes a plurality of data packets; the parameter determination module 131 is used to determine according to the protocol information An abnormal frame quality parameter and an abnormal packet quality parameter, the abnormal frame quality parameter indicating that the quality of the video frame to which the abnormal frame quality parameter belongs is abnormal, and the abnormal packet quality parameter indicating the abnormal packet quality parameter The quality of the associated data packet is abnormal; the abnormal cause determination module 132 is used to determine the cause and equipment that cause the abnormal video quality of the video playback terminal according to the abnormal frame quality parameter and/or the abnormal packet quality parameter.

In a possible implementation manner, the protocol information includes streaming media layer protocol information, network layer protocol information, and transport layer protocol information of the data packet, and the streaming media layer protocol information includes the flow of the data packet on the video playback end. Media layer protocol information, the network layer protocol information includes the network layer protocol information of the data packet on the network device and the video playback end, and the transport layer protocol information includes the transport layer of the data packet on the network device and the video playback end Protocol information, the determination of abnormal frame quality parameters and abnormal packet quality parameters according to the protocol information includes: according to the streaming media layer protocol information, determining at least one video frame and the frame corresponding to each frame in the at least one video frame Each data packet; according to the streaming media layer protocol information and the network layer protocol information, determine the frame quality parameters of each frame in the at least one video frame; according to the network layer protocol information and the transport layer protocol information, Determining the packet quality parameter of each data packet corresponding to each frame in the at least one video frame; determining the abnormal frame quality parameter according to the frame quality parameter corresponding to each frame in the at least one video frame and the frame quality parameter threshold, The abnormal packet quality parameter is determined according to the packet quality parameter of each data packet corresponding to each frame in the at least one video frame and the packet quality parameter threshold.

In a possible implementation manner, the abnormal frame quality parameter is determined according to the frame quality parameter corresponding to each frame in the at least one video frame and the frame quality parameter threshold, and the abnormal frame quality parameter is determined according to each frame quality parameter in the at least one video frame. The packet quality parameter of each data packet corresponding to the frame and the packet quality parameter threshold value, determining the abnormal packet quality parameter includes: for each frame of the at least one video frame, among the frame quality parameters of the video frame, the value A frame quality parameter greater than the frame quality parameter threshold is determined as the abnormal frame quality parameter; for each frame of the at least one video frame, among the packet quality parameters of the data packet corresponding to the video frame, the value is greater than The packet quality parameter of the packet quality parameter threshold is determined as the abnormal packet quality parameter.

In a possible implementation manner, the streaming media layer protocol information of the data packet includes a frame time stamp indicating the video frame to which the data packet belongs, and the network layer protocol information of the data packet includes a packet time indicating the time of the data packet , the frame quality parameter includes a frame header jitter parameter and a frame tail packet jitter parameter, the frame header jitter parameter represents the delay jitter of the first data packet corresponding to the video frame, and the frame tail packet jitter parameter represents The delay and jitter situation of the last data packet corresponding to the video frame, according to the streaming media layer protocol information and the network layer protocol information, determine the frame quality parameters of each frame in the at least one video frame, including: for For each frame of the at least one video frame, the arrival time of the first packet and the arrival time of the last packet of the video frame are determined according to the packet time of the data packet corresponding to the video frame, and the arrival time of the first packet represents the first packet corresponding to the video frame. The arrival time of a data packet, the arrival time of the last packet corresponding to the video frame is represented by the tail packet arrival time; according to the arrival time of the first packet and the arrival time of the tail packet, the first packet interval and the first packet interval of adjacent video frames are calculated respectively. The tail packet interval, the first packet interval represents the arrival time difference of the first data packet corresponding to the adjacent video frame, and the tail packet interval represents the arrival time difference of the last data packet corresponding to the adjacent video frame; according to the frame time stamp, respectively calculate the frame interval of adjacent video frames; according to the first packet interval, frame interval, and the frame first packet jitter parameter of the previous video frame, calculate the frame first packet jitter parameter of the current video frame; according to the tail packet interval, frame interval, and the frame end packet jitter parameter of the previous video frame, and calculate the frame end packet jitter parameter of the current video frame.

In a possible implementation manner, the streaming media layer protocol information of the data packet includes a streaming media layer sequence number indicating the sequence of the data packet in the video frame to which it belongs, and the network layer protocol information of the data packet includes indicating data The packet time of the time of packet, described frame quality parameter comprises frame packet loss parameter and frame disorder parameter, and described frame packet loss parameter represents the loss situation of the data packet of video frame, and described frame disorder parameter represents the data of video frame Out-of-sequence situation of the packet, said determining the frame quality parameters of each frame in the at least one video frame according to the streaming media layer protocol information and the network layer protocol information, including: for each frame of the at least one video frame One frame, according to the streaming media layer sequence number of the data packet corresponding to the video frame, determine the frame packet loss parameter of the video frame; for each frame of the at least one video frame, according to the packet of the data packet corresponding to the video frame In order of time, the data packets corresponding to the video frame are sorted to obtain the corresponding sorting result of the video frame, and according to the streaming media layer sequence number of the data packet corresponding to the sorting result and the video frame, the video frame is determined Frame out-of-order parameters.

In a possible implementation manner, for each frame of the at least one video frame, determining the frame packet loss parameter of the video frame according to the streaming media layer sequence number of the data packet corresponding to the video frame includes: For each frame of the at least one video frame, according to the streaming media layer serial number of the data packet corresponding to the video frame, determine the maximum value, the minimum value and the streaming media layer serial number of the data packet corresponding to the video frame Layer sequence number, according to the maximum value and the minimum value of the streaming media layer sequence number, determine the theoretical value of the data packet number corresponding to the video frame, according to the theoretical value of the data packet number corresponding to the video frame and the The difference of the streaming media layer sequence number of the data packet corresponding to the video frame determines the frame packet loss parameter of the video frame; according to the streaming media layer serial number of the data packet corresponding to the sorting result and the video frame, determine The frame out-of-sequence parameters of the video frame include: when the numerical value of the streaming media layer serial number of the current data packet is greater than the streaming media layer serial number of the data packet before the current data packet in the sorting result, according to the sorting The sequence number of the current data packet in the result, the streaming media layer serial number of the current data packet, and the minimum value of the streaming media layer serial number of the data packet corresponding to the video frame determine the frame out-of-sequence parameter corresponding to the video frame.

In a possible implementation manner, the packet quality parameter of each data packet corresponding to each frame in the at least one video frame is determined according to the network layer protocol information and the transport layer protocol information of the data packet, and the packet quality parameter includes : The packet interval parameter that represents the time interval between the current data packet and the previous data packet, the packet jitter parameter that represents the delay jitter between the current data packet and the previous data packet, the out-of-order parameter that represents the disordered state of the data packet, and the data packet Packet loss parameter for packet loss, retransmission parameter for data packet retransmission, delay parameter for data packet and corresponding response message delay, IP packet fragmentation for data packet One or more of the text fragmentation parameter and the checksum error parameter indicating whether the checksum of the data packet is wrong.

In a possible implementation manner, the determining the cause and device for abnormal video quality at the video playback end according to the abnormal frame quality parameter and/or the abnormal packet quality parameter includes: for any video frame, When the frame quality parameter of the video frame and the packet quality parameter of the data packet corresponding to the video frame meet the first condition, it is determined that the cause of the abnormal video quality at the video playback end is a video encoding problem, and it is determined that the video quality at the video playback end is abnormal The device is a device used for video encoding; wherein the first condition includes: the frame loss parameter of the video frame is abnormal and the packet loss parameter of the data packet corresponding to the video frame is normal, or the frame loss parameter of the video frame The out-of-order parameter is abnormal and the out-of-order parameter of the data packet corresponding to the video frame is normal.

In a possible implementation manner, the determining the cause and device for abnormal video quality at the video playback end according to the abnormal frame quality parameter and/or the abnormal packet quality parameter includes: for any video frame, When the frame quality parameter of the video frame and the packet quality parameter of the data packet corresponding to the video frame meet the second condition, it is determined that the cause of the abnormal video quality at the video playback end is a terminal parameter problem, and it is determined that the video quality at the video playback end is abnormal The device is a device used for video playback; wherein, the second condition includes: any parameter in the first frame packet jitter parameter and the frame end packet jitter parameter of the video frame is abnormal and the data packet corresponding to the video frame Any of the IP packet fragmentation parameters and checksum error parameters is abnormal.

In a possible implementation manner, the determining the cause and device for abnormal video quality at the video playback end according to the abnormal frame quality parameter and/or the abnormal packet quality parameter includes: for any video frame, When the frame quality parameter of the video frame and the packet quality parameter of the data packet corresponding to the video frame meet the third condition, it is determined that the cause of the abnormal video quality at the video playback end is a network transmission problem; wherein the third condition includes : The frame packet loss parameter of the video frame is abnormal and the packet loss parameter of the data packet corresponding to the video frame is abnormal, or the frame out-of-order parameter of the video frame is abnormal and the out-of-order parameter of the data packet corresponding to the video frame is abnormal Abnormal, or any parameter in the frame first packet jitter parameter and frame end packet jitter parameter of the video frame is abnormal and the packet jitter parameter, packet interval parameter, retransmission parameter, and delay parameter of the data packet corresponding to the video frame Any parameter in is abnormal.

In a possible implementation manner, after it is determined that the cause of the abnormal video quality at the video player end is a network transmission problem, the determination according to the abnormal frame quality parameter and/or the abnormal packet quality parameter causes the video player end The reason and equipment of abnormal video quality also include: according to the network layer protocol information of the data packet, determine the network layer sequence number of the data packet corresponding to the abnormal packet quality parameter; according to the transport layer protocol information of the data packet, determine Packet time, transport layer sequence number, and checksum of the data packet corresponding to the abnormal packet quality parameter; receive network topology data on the network device, and determine the transmission path of each data packet in the network according to the network topology data; The transmission path of the data packet corresponding to the abnormal packet quality parameter is found in each network device on the transmission path, and the packet time, the network layer serial number, the transport layer serial number, and the checksum all meet the conditions; according to the found The packet quality parameters of the data packets satisfying the conditions are determined from the network devices on the transmission path of the data packets to determine the network device that causes the abnormal video quality of the video player.

In a possible implementation manner, the conditions for satisfying the packet time, the network layer sequence number, the transport layer sequence number, and the checksum include: the difference between the packet time and the packet time of the data packet corresponding to the abnormal packet quality parameter The absolute value of the difference is less than the preset threshold, and the network layer sequence number, transport layer sequence number, checksum and the network layer sequence number, transport layer sequence number, checksum of the data packet corresponding to the abnormal packet quality parameter are the same.

In a possible implementation manner, when there are multiple data packets corresponding to the abnormal packet quality parameter, there are multiple groups of data packets found satisfying the conditions, and according to the packet quality parameters of the found data packets satisfying the conditions, from Among the network devices on the transmission path of the data packet, determine the network device that causes the abnormal video quality of the video player, including: according to the packet quality parameters of multiple sets of data packets that meet the conditions, from each network device on the transmission path of the data packet Among the network devices that may cause abnormal video quality at the video playback end, the network device that may cause abnormal video quality at the video playback end is determined multiple times; among the network devices that may cause abnormal video quality at the video playback end, the network device with the most occurrences is counted as the network device that causes abnormal video quality at the video playback end. network equipment.

An embodiment of the present application provides an apparatus for determining a video quality problem, including: a processor and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions.

An embodiment of the present application provides a non-volatile computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the foregoing method is realized.

An embodiment of the present application provides a computer program product, including computer-readable codes, or a non-volatile computer-readable storage medium bearing computer-readable codes, when the computer-readable codes are stored in a processor of an electronic device When running in the electronic device, the processor in the electronic device executes the above method.

As shown in Figure 14, the device for determining video quality problems may include a mobile phone, a foldable electronic device, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC) ), netbooks, cellular phones, personal digital assistants (personal digital assistant, PDA), augmented reality (augmented reality, AR) devices, virtual reality (virtual reality, VR) devices, artificial intelligence (artificial intelligence, AI) devices, wearable At least one of portable devices, vehicle-mounted devices, smart home devices, smart city devices, and server devices. The embodiment of the present application does not specifically limit the specific type of the apparatus for determining a video quality problem.

The apparatus for determining a video quality problem may include a processor 110 , a memory 121 , and a communication module 160 . It can be understood that the structure shown in the embodiment of the present application does not constitute a specific limitation on the apparatus for determining a video quality problem. In some other embodiments of the present application, the device for determining a video quality problem may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The processor 110 can generate an operation control signal according to the instruction opcode and the timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 may be a cache memory. The memory may store instructions or data used by the processor 110 or used frequently. If the processor 110 needs to use the instruction or data, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.

The memory 121 may be used to store computer-executable program code including instructions. The memory 121 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, an application program required by at least one function (such as an abnormal frame quality parameter determination function, an abnormal packet quality parameter determination function) and the like. The storage data area can store data (such as frame quality parameters, packet quality parameters, etc.) created during the use of the device for determining video quality problems. In addition, the memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like. The processor 110 executes the instructions stored in the memory 121 and/or the instructions stored in the memory provided in the processor to execute various function methods of the device for determining video quality problems or the above-mentioned method for determining video quality problems.

The communication module 160 can be used to receive streaming data (such as the protocol information collection device in the embodiment of the application) from other devices or equipment (such as the protocol information collection device and the topology data collection device in the embodiment of the application) through wireless communication/wired communication. information, network topology data), and output streaming data to other devices or devices (such as the cause of the video quality problem and device information determined in the embodiment of this application). For example, it can provide WLAN (such as Wi-Fi network), Bluetooth (Bluetooth, BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication) , NFC), infrared technology (infrared, IR) and other wireless communication solutions.

A computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disk, hard disk, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), erasable Electrically Programmable Read-Only-Memory (EPROM or flash memory), Static Random-Access Memory (Static Random-Access Memory, SRAM), Portable Compression Disk Read-Only Memory (Compact Disc Read-Only Memory, CD -ROM), Digital Video Disc (DVD), memory sticks, floppy disks, mechanically encoded devices such as punched cards or raised structures in grooves with instructions stored thereon, and any suitable combination of the foregoing .

Computer readable program instructions or codes described herein may be downloaded from a computer readable storage medium to a respective computing/processing device, or downloaded to an external computer or external storage device over a network, such as the Internet, local area network, wide area network, and/or wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or a network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

Computer program instructions for performing the operations of the present application may be assembly instructions, instruction set architecture (Instruction Set Architecture, ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or in one or more source or object code written in any combination of programming languages, including object-oriented programming languages—such as Smalltalk, C++, etc., and conventional procedural programming languages—such as the “C” language or similar programming languages. Computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In cases involving a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or it can be connected to an external computer such as use an Internet service provider to connect via the Internet). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (Field-Programmable Gate Array, FPGA) or programmable logic arrays (Programmable Logic Array, PLA), the electronic circuit can execute computer-readable program instructions, thereby realizing various aspects of the present application.

Aspects of the present application are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It should be understood that each block of the flowcharts and/or block diagrams, and combinations of blocks in the flowcharts and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that when executed by the processor of the computer or other programmable data processing apparatus , producing an apparatus for realizing the functions/actions specified in one or more blocks in the flowchart and/or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause computers, programmable data processing devices and/or other devices to work in a specific way, so that the computer-readable medium storing instructions includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks in flowcharts and/or block diagrams.

It is also possible to load computer-readable program instructions into a computer, other programmable data processing device, or other equipment, so that a series of operational steps are performed on the computer, other programmable data processing device, or other equipment to produce a computer-implemented process , so that instructions executed on computers, other programmable data processing devices, or other devices implement the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

The flowchart and block diagrams in the figures show the architecture, functions and operations of possible implementations of apparatuses, systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in a flowchart or block diagram may represent a module, a portion of a program segment, or an instruction that includes one or more Executable instructions. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved.

It should also be noted that each block in the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts, can be implemented with hardware (such as circuits or ASIC (Application Specific Integrated Circuit, application-specific integrated circuit)), or can be implemented with a combination of hardware and software, such as firmware.

Although the present invention has been described in conjunction with various embodiments herein, in the process of implementing the claimed invention, those skilled in the art can understand and Other variations of the disclosed embodiments are implemented. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that these measures cannot be combined to advantage.

Having described various embodiments of the present application above, the foregoing description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and alterations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principle of each embodiment, practical application or improvement of technology in the market, or to enable other ordinary skilled in the art to understand each embodiment disclosed herein.

Claims

A method for determining a video quality problem, characterized in that the method comprises:

Receiving protocol information on the video playback terminal and network equipment, the network equipment is used to transmit video data to the video playback terminal, the protocol information indicates the transmission mode of the video frame, and the video frame includes a plurality of data packets;

Determine an abnormal frame quality parameter and an abnormal packet quality parameter according to the protocol information, the abnormal frame quality parameter indicates that the quality of the video frame to which the abnormal frame quality parameter belongs is abnormal, and the abnormal packet quality parameter indicates the abnormal quality parameter The quality of the data packet to which the abnormal packet quality parameter belongs is abnormal;

According to the abnormal frame quality parameter and/or the abnormal packet quality parameter, determine the cause and equipment causing the abnormal video quality at the video playback terminal.
The method according to claim 1, wherein the protocol information includes streaming media layer protocol information, network layer protocol information and transport layer protocol information of the data packet, and the streaming media layer protocol information includes the The streaming media layer protocol information of the data packet, the network layer protocol information includes the network layer protocol information of the data packet on the network device and the video player, and the transport layer protocol information includes the data on the network device and the video player The transport layer protocol information of the packet,

The determining the abnormal frame quality parameter and the abnormal packet quality parameter according to the protocol information includes:

According to the streaming media layer protocol information, determine at least one video frame and each data packet corresponding to each frame in the at least one video frame;

Determine the frame quality parameters of each frame in the at least one video frame according to the streaming media layer protocol information and the network layer protocol information;

According to the network layer protocol information and the transport layer protocol information, determine the packet quality parameters of each data packet corresponding to each frame in the at least one video frame;

Determine the abnormal frame quality parameter according to the frame quality parameter and the frame quality parameter threshold corresponding to each frame in the at least one video frame, and determine the abnormal frame quality parameter according to the packet quality parameter and the packet quality parameter of each data packet corresponding to each frame in the at least one video frame A packet quality parameter threshold is used to determine the abnormal packet quality parameter.
The method according to claim 2, wherein the abnormal frame quality parameter is determined according to the frame quality parameter corresponding to each frame in the at least one video frame and the frame quality parameter threshold, and according to the at least one The packet quality parameters and the packet quality parameter thresholds of each data packet corresponding to each frame in the video frame determine the abnormal packet quality parameters, including:

For each frame of the at least one video frame, among the frame quality parameters of the video frame, the frame quality parameter whose value is greater than the frame quality parameter threshold is determined as the abnormal frame quality parameter;

For each frame of the at least one video frame, among the packet quality parameters of the data packets corresponding to the video frame, the packet quality parameter whose value is greater than the packet quality parameter threshold is determined as the abnormal packet quality parameter.
The method according to claim 2 or 3, wherein the streaming media layer protocol information of the data packet includes a frame time stamp indicating the video frame to which the data packet belongs, and the network layer protocol information of the data packet includes indicating data The packet time of the time of the packet, the frame quality parameter includes a frame first packet jitter parameter and a frame tail packet jitter parameter, and the frame first packet jitter parameter represents the delay jitter situation of the first data packet corresponding to the video frame, and the The frame end packet jitter parameter indicates the delay jitter of the last data packet corresponding to the video frame,

The determining the frame quality parameters of each frame in the at least one video frame according to the streaming media layer protocol information and the network layer protocol information includes:

For each frame of the at least one video frame, the first packet arrival time and the last packet arrival time of the video frame are determined according to the packet time of the data packet corresponding to the video frame, and the first packet arrival time represents the first packet arrival time corresponding to the video frame. The arrival time of a data packet, the arrival time of the tail packet represents the arrival time of the last data packet corresponding to the video frame;

According to the arrival time of the first packet and the arrival time of the last packet, the first packet interval and the last packet interval of adjacent video frames are calculated respectively, and the first packet interval represents the arrival time difference of the first data packet corresponding to the adjacent video frame, so Said tail packet interval represents the arrival time difference of the last data packet corresponding to adjacent video frames;

According to the frame time stamp, calculate the frame interval of adjacent video frames respectively;

According to the first packet interval, the frame interval, and the first frame packet jitter parameter of the previous video frame, calculate the first frame packet jitter parameter of the current video frame;

According to the trailing packet interval, the frame interval, and the frame trailing packet jitter parameter of the previous video frame, calculate the frame trailing packet jitter parameter of the current video frame.
The method according to any one of claims 2-4, wherein the streaming media layer protocol information of the data packet includes a streaming media layer sequence number indicating the sequence of the data packet in the video frame to which it belongs, the The network layer protocol information of the data packet includes the packet time indicating the time of the data packet, and the frame quality parameter includes a frame packet loss parameter and a frame out-of-sequence parameter, and the frame packet loss parameter represents the loss situation of the data packet of the video frame, so The out-of-sequence parameter of the frame represents the out-of-order situation of the data packet of the video frame,

The determining the frame quality parameters of each frame in the at least one video frame according to the streaming media layer protocol information and the network layer protocol information includes:

For each frame of the at least one video frame, determine the frame packet loss parameter of the video frame according to the streaming layer sequence number of the data packet corresponding to the video frame;

For each frame of the at least one video frame, sort the data packets corresponding to the video frame according to the packet time sequence of the data packets corresponding to the video frame, and obtain the corresponding sorting result of the video frame, according to the The sorting result and the streaming layer sequence number of the data packet corresponding to the video frame determine the frame out-of-sequence parameter of the video frame.
The method according to claim 5, wherein, for each frame of the at least one video frame, determine the frame loss of the video frame according to the streaming layer serial number of the data packet corresponding to the video frame parameters, including:

For each frame of the at least one video frame, according to the streaming media layer serial number of the data packet corresponding to the video frame, determine the maximum value, the minimum value and the streaming media layer serial number of the data packet corresponding to the video frame Layer sequence number, according to the maximum value and the minimum value of the streaming media layer sequence number, determine the theoretical value of the data packet number corresponding to the video frame, according to the theoretical value of the data packet number corresponding to the video frame and the The difference between the streaming media layer sequence numbers of the data packets corresponding to the video frame determines the frame packet loss parameter of the video frame;

According to the streaming media layer sequence number of the data packet corresponding to the sorting result and the video frame, determine the frame out-of-sequence parameter of the video frame, including:

When the value of the streaming media layer serial number of the current data packet is greater than the streaming media layer serial number of the data packet before the current data packet in the sorting result, according to the sorting serial number of the current data packet in the sorting result, the current data The minimum value of the streaming layer sequence number of the packet and the streaming layer sequence number of the data packet corresponding to the video frame determines the frame disorder parameter corresponding to the video frame.
The method according to claims 2-6, wherein the packet quality parameters of each data packet corresponding to each frame in the at least one video frame are determined according to the network layer protocol information and the transport layer protocol information of the data packet, The packet quality parameters include: a packet interval parameter representing the time interval between the current data packet and the previous data packet, a packet jitter parameter representing the time delay jitter between the current data packet and the previous data packet, and a packet jitter parameter representing the disorder of the data packet. sequence parameter, packet loss parameter indicating packet loss, retransmission parameter indicating packet retransmission, delay parameter indicating the delay of data packet and corresponding response message, indicating whether there is an IP packet One or more of the IP packet fragmentation parameters of the text fragmentation, and the checksum error parameters indicating whether the checksum of the data packet is wrong.
The method according to claim 7, wherein the determination of the cause and the device causing the abnormal video quality at the video playback terminal according to the abnormal frame quality parameter and/or the abnormal packet quality parameter includes:

For any video frame, when the frame quality parameter of the video frame and the packet quality parameter of the data packet corresponding to the video frame meet the first condition, it is determined that the cause of the abnormal video quality at the video playback end is a video encoding problem, and it is determined that the The device with abnormal video quality at the video playback end is the device used for video encoding;

Wherein, the first condition includes:

The frame packet loss parameter of the video frame is abnormal and the packet loss parameter of the data packet corresponding to the video frame is normal, or

The frame out-of-order parameter of the video frame is abnormal and the out-of-order parameter of the data packet corresponding to the video frame is normal.
The method according to claim 7 or 8, wherein the determination of the cause and the device causing the abnormal video quality at the video playback terminal according to the abnormal frame quality parameter and/or the abnormal packet quality parameter includes:

For any video frame, when the frame quality parameter of the video frame and the packet quality parameter of the data packet corresponding to the video frame meet the second condition, it is determined that the cause of the abnormal video quality at the video playback end is a terminal parameter problem, and it is determined that the cause The device with abnormal video quality at the video playback end is the device used for video playback;

Wherein, the second condition includes:

Any parameter in the frame header jitter parameter and frame tail packet jitter parameter of the video frame is abnormal and any parameter in the IP packet fragmentation parameter and checksum error parameter of the data packet corresponding to the video frame is abnormal .
The method according to any one of claims 7-9, characterized in that, according to the abnormal frame quality parameter and/or the abnormal packet quality parameter, the cause of the abnormal video quality at the video playback terminal is determined and equipment, including:

For any video frame, when the frame quality parameter of the video frame and the packet quality parameter of the data packet corresponding to the video frame meet the third condition, it is determined that the cause of the abnormal video quality at the video playback end is a network transmission problem;

Wherein, the third condition includes:

The frame packet loss parameter of the video frame is abnormal and the packet loss parameter of the data packet corresponding to the video frame is abnormal, or

The frame out-of-order parameter of the video frame is abnormal and the out-of-order parameter of the data packet corresponding to the video frame is abnormal, or

Any one of the frame header jitter parameters and frame tail packet jitter parameters of the video frame is abnormal and any of the packet jitter parameters, packet interval parameters, retransmission parameters, and delay parameters of the data packets corresponding to the video frame A parameter exception.
The method according to claim 10, characterized in that, after determining that the cause of the abnormal video quality at the video player end is a network transmission problem, the determination is made according to the abnormal frame quality parameter and/or the abnormal packet quality parameter Reasons and equipment that lead to abnormal video quality at the video playback end, including:

According to the network layer protocol information of the data packet, determine the network layer serial number of the data packet corresponding to the abnormal packet quality parameter;

According to the transport layer protocol information of the data packet, determine the packet time, transport layer sequence number, and checksum of the data packet corresponding to the abnormal packet quality parameter;

receiving network topology data on the network device, and determining the transmission path of each data packet in the network according to the network topology data;

According to the transmission path of the data packet corresponding to the abnormal packet quality parameter described in the network, find the data packet that satisfies the conditions of packet time, network layer serial number, transport layer serial number, and checksum in each network device on the transmission path ;

According to the found packet quality parameters of the data packets satisfying the conditions, the network equipment causing the abnormal video quality of the video playback terminal is determined from among the network equipments on the transmission path of the data packets.
A device for determining video quality problems, characterized in that the device includes:

The information receiving module is used to receive the protocol information on the video player and the network device, the network device is used to transmit the video data to the video player, the protocol information indicates the sending mode of the video frame, and the video frame includes multiple packets;

A parameter determination module, configured to determine an abnormal frame quality parameter and an abnormal packet quality parameter according to the protocol information, the abnormal frame quality parameter indicates that the quality of the video frame to which the abnormal frame quality parameter belongs is abnormal, and the abnormal The packet quality parameter indicates that the quality of the data packet to which the abnormal packet quality parameter belongs is abnormal;

The abnormal cause determination module is used to determine the cause and equipment that cause the abnormal video quality of the video playback terminal according to the abnormal frame quality parameter and/or the abnormal packet quality parameter.
A device for determining video quality problems, characterized in that it includes:

processor;

memory for storing processor-executable instructions;

Wherein, the processor is configured to implement the method according to any one of claims 1-11 when executing the instructions.
A non-volatile computer-readable storage medium, on which computer program instructions are stored, wherein, when the computer program instructions are executed by a processor, the method according to any one of claims 1-11 is implemented.
A computer program product, comprising computer-readable codes, or a non-volatile computer-readable storage medium bearing computer-readable codes, characterized in that, when the computer-readable codes are run in an electronic device, the A processor in the electronic device executes the method described in any one of claims 1-11.