WO2018166162A1 - 一种音视频直播中检测客户端播放状态的系统及方法 - Google Patents

一种音视频直播中检测客户端播放状态的系统及方法 Download PDF

Info

Publication number
WO2018166162A1
WO2018166162A1 PCT/CN2017/102283 CN2017102283W WO2018166162A1 WO 2018166162 A1 WO2018166162 A1 WO 2018166162A1 CN 2017102283 W CN2017102283 W CN 2017102283W WO 2018166162 A1 WO2018166162 A1 WO 2018166162A1
Authority
WO
WIPO (PCT)
Prior art keywords
client
server
data
audio
detection information
Prior art date
Application number
PCT/CN2017/102283
Other languages
English (en)
French (fr)
Inventor
唐乐军
Original Assignee
广州视源电子科技股份有限公司
广州视睿电子科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 广州视源电子科技股份有限公司, 广州视睿电子科技有限公司 filed Critical 广州视源电子科技股份有限公司
Publication of WO2018166162A1 publication Critical patent/WO2018166162A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/258—Client or end-user data management, e.g. managing client capabilities, user preferences or demographics, processing of multiple end-users preferences to derive collaborative data
    • H04N21/25808—Management of client data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
    • H04N21/4424—Monitoring of the internal components or processes of the client device, e.g. CPU or memory load, processing speed, timer, counter or percentage of the hard disk space used

Definitions

  • the invention belongs to the technical field of webcasting, and particularly relates to a system and method for detecting a playing state of a client in an audio and video live broadcast.
  • Live broadcast as a real-time display method, is sought after by more and more people, especially young people.
  • the live broadcast is divided into live text and live video.
  • Traditional TV stations mainly focus on live video, such as the news series "News Network” and the performing arts “Spring Festival Gala”.
  • live broadcasts such as live sports events and news.
  • text, pictures, and videos can be broadcast live.
  • participants can be used as a person to initiate a live broadcast or as a person to watch or listen to live broadcasts. Whether the live broadcast status is normal or not is crucial for those who launch live broadcasts and those who watch or listen to live broadcasts.
  • the current live video and audio status detection only determines whether the live broadcast is successful by whether the data arrives or whether the signaling is passed.
  • this kind of audio and video live state detection technology does not know the actual operation of the live video broadcast for the person who initiates the live broadcast, so the live broadcast state detection technology is not very accurate.
  • the embodiment of the present invention provides a system and method for detecting a client playing state in an audio and video live broadcast.
  • a system for detecting a playing state of a client in an audio and video live broadcast comprising: a collecting end, a server, and a guest a client, wherein the collection end is connected to the server, and the server is connected to the client;
  • the collecting end adds the mark of the state detection information to the collected audio and video data, and uploads the audio and video data added with the state detection information mark to the server;
  • the server detects the received audio and video data, and if the flag of the state detection information is detected, decodes the image at this time;
  • the client obtains audio and video data from the server and plays the same; when the client plays the audio and video data, if the flag of the state detection information is detected, the flag of the state detection information is recorded in real time, and the screenshot of the playback area is performed, and the screenshot data and the screenshot data are The tag data of the status detection information is sent to the server in real time;
  • the server compares whether the decoded image of the tag corresponding to the same state detection information is consistent with the sent screenshot data. If the server does not receive the tag data of the screenshot data and the state detection information sent by the client within a limited time, the server determines The client playback status is abnormal.
  • the client sends the screenshot data and the tag data of the state detection information to the server in real time, it is also sent together with the identity information of the client.
  • the marked frame of the state detection information is added as a key frame.
  • the status detection information is marked with a time stamp and/or a barrage.
  • the server feeds back to the user who initiates the live video broadcast that the playing state of the client is abnormal, or feeds back to the server administrator the identity information of the client whose playing state is abnormal.
  • a method for detecting a playing state of a client in an audio and video live broadcast comprising:
  • Acquisition step collecting audio and video data at the collection end
  • Marking step the collecting end timing adds a flag of the state detecting information to the collected audio and video data;
  • Uploading step after the marking step, the collecting end uploads the marked audio and video data of the state detecting information to the server;
  • First detecting step the server detects the received audio and video data, if the server detects the status check Measuring the mark of the information, then decoding the image at this time;
  • the second detecting step is: the client obtains audio and video data from the server, and when the client plays the audio and video data, if the flag of the state detecting information is detected, the flag of the state detecting information is recorded in real time, and a screenshot of the playing area is simultaneously performed, and The screenshot data and the tag data of the status detection information are sent to the server in real time;
  • the determining step the server receives the tag data of the screenshot data and the state detection information sent by the client, and compares whether the decoded image corresponding to the tag of the same state detection information is consistent with the sent screenshot data, if the server is inconsistent or within a limited time If the tag data of the screenshot data and the status detection information sent by the client is not received, it is determined that the client playing status is abnormal.
  • the client when the client sends the screenshot data and the tag data of the status detection information to the server in real time, it is also sent together with the identity information of the client.
  • the marked frame of the state detection information is added as a key frame.
  • the status detection information is marked with a time stamp and/or a barrage.
  • the server feeds back to the user who initiated the live video broadcast that the playing state of the client is abnormal, or feeds back to the server administrator the identity information of the client whose playing state is abnormal.
  • the system and the method for detecting the playing state of the client in the live broadcast of the audio and video proposed by the embodiment of the present invention can be compared with the technical solutions in the prior art, such as whether the data arrives and the interface is successfully invoked. More accurate and timely detection of the current state of the client during the live broadcast.
  • FIG. 1 is a structural block diagram of a system for detecting a playing state of a client in an audio and video live broadcast according to an embodiment of the present invention
  • FIG. 2 is a flowchart of a method for detecting a playing state of a client in an audio and video live broadcast according to an embodiment of the present invention
  • FIG. 3 is a flowchart of a method for detecting a playing state of a client in an audio and video live broadcast according to another embodiment of the present invention.
  • the system includes: an acquisition end 11, a server 12, and a client 13, wherein the collection end 11 is connected to the server 12, and the server is connected to the server. 12 is connected to the client 13.
  • the collecting end 11 collects audio and video data, and periodically adds a time stamp to the key frames of the collected audio and video data, so that the audio and video data carries state detection information; the collecting end 11 uploads the audio and video data with the time stamp added to the Server 12.
  • the key frame refers to a frame that can be independently coded without requiring other frame images as a reference, and can also be referred to as an independent frame, generally referring to an I frame.
  • the key frames may be one or more.
  • the client 13 downloads audio and video data from the server 12 or transmits the audiovisual data to the client 13 by the server 12, and the client 13 plays the audio and video data from the server 12; when the client 13 plays the audio and video data, if it detects
  • a time stamp is added to a frame of audio and video data, the time stamp data is recorded, and a play area screenshot is taken, and the screenshot data and the time stamp data are sent to the server 12 together with the identity information of the client 13 in real time.
  • the identity information can be an ip address or a MAC address.
  • the server 12 detects the audio and video data uploaded by the collection end. If a timestamp is added to the data of the frame, the image corresponding to the frame data with the timestamp is decoded, and the decoded image is converted into the decoded image.
  • the RGB data associated with the RGB data of the time stamp data and the decoded image, and the identity information of the client.
  • the server 12 waits for the receiving client to send the screenshot data and the timestamp data and the identity information of the client 13. For each timestamp data saved in the server, the server 12 receives the time sent by the client 13 and the time within a limited time.
  • the screenshot data corresponding to the stamp data is RGB converted to the transmitted screenshot data, converted into RGB data of the sent screenshot, and the server 12 compares the RGB data of the screenshot sent by the client 13 with the RGB of the saved decoded image through a graphic comparison algorithm. Whether the data is consistent; if the server 12 does not receive the screenshot data corresponding to the timestamp data sent by the client 13 in an inconsistent or limited time, it is determined that the client 13 plays an abnormal audio and video state, and performs real-time alarm notification and the like.
  • the person who initiates the live broadcast has the abnormality of playing the audio and video status of the client, or feeds back to the server administrator the identity information of the client whose playback status is abnormal; if it is consistent, it determines that the audio and video status of the client 13 is normal.
  • the system can perform playback state detection on multiple clients at the same time.
  • the limited time may be set by the user (for example, a person who initiates a live broadcast), or may be automatically set by the server. For each timestamp, the time limit can be the same or different.
  • the time stamp is used as the mark of the state detection information
  • other time stamping methods may be used, or other types of marking methods may be used, for example, a barrage is added in a certain frame, and the barrage is added.
  • a mark of the state detection information; or as a mark of the state detection information together with the time stamp and the barrage the time stamp and the barrage may be set in the same frame or may be set in different frames.
  • the time stamp is used as the mark of the state detecting information.
  • the method proposed in this embodiment is shown in FIG. 2, and the method includes:
  • Step S21 the collecting step: collecting audio and video data by the collecting end;
  • Step S22 adding a step of adding: a time stamp is added to the key frame of the collected audio and video data, so that the audio and video data carries state detection information; the key frame refers to that no other frame image is needed for reference.
  • the key frames may be one or more.
  • Step S23 Uploading step: the collecting end uploads the audio and video data with the time stamp added to the server;
  • Step S24 The first detecting step: the server detects the audio and video data uploaded by the collecting end, and when the server detects that a timestamp is added to the data of the frame, the image corresponding to the frame data to which the timestamp is added is decoded, and the decoding is performed. The subsequent image is converted into RGB data of the decoded image, and the time stamp data and the RGB data of the decoded image are stored in association;
  • Step S25 The second detecting step: the client downloads audio and video data from the server or the server transfers the pronunciation video data to the client, and the client plays the audio and video data from the server; if the client detects the audio and video data, if the user detects A timestamp is added to the frame data, the timestamp data is recorded, and a screenshot of the play area is performed, and the screenshot data and the timestamp data are sent to the server together with the identification information of the client in real time; the identification information may be an ip address or a MAC. address;
  • Step S26 The determining step: the server waits to receive the screenshot data and the timestamp data sent by the client, and the identity identification information of the client. For each timestamp data saved in the server, if the server receives the sent by the client within a limited time, The screenshot data corresponding to the timestamp data is RGB converted to the transmitted screenshot data, and converted into RGB data of the sent screenshot; the server compares the RGB data of the sent screenshot and the RGB data of the saved decoded image by a graphics comparison algorithm. Whether it is consistent; if the server does not receive the screenshot data corresponding to the timestamp data sent by the client in an inconsistent or limited time, it is determined that the audio and video status of the client is abnormal, and real-time alarm notification is performed, for example, the notification is initiated. The person has the client playing the audio and video status abnormality, or feedback to the server administrator the identity information of the client whose playing status is abnormal; if they are consistent, it is determined that the audio and video state of the client is normal.
  • the barrage is used as a mark for detecting the state information, and a barrage is added to some frames of the audio and video data.
  • the method proposed in this embodiment is as shown in FIG. 3, and includes:
  • Step S31 the collecting step: collecting audio and video data by the collecting end;
  • Step S32 adding a marking step: the collecting end timing adds a barrage to some frames of the collected audio and video data, so that the audio and video data carries state detection information;
  • Step S33 uploading step: the collecting end uploads the audio and video data added to the barrage to the server;
  • Step S34 The first detecting step: the server detects the audio and video data uploaded by the collecting end. When the server detects that a certain frame data is added to the barrage, the image corresponding to the frame data of the barrage is decoded and decoded. The subsequent image is converted into RGB data of the decoded image, and the RGB data of the barrage and the decoded image are saved in association;
  • Step S35 The second detecting step: the client downloads audio and video data from the server or the server transfers the pronunciation video data to the client, and the client plays the audio and video data from the server; if the client detects the audio and video data, if the user detects When the frame data is added to the barrage, the barrage data is recorded, and the play area screenshot is taken at the same time, and the screenshot data and the barrage data are sent to the server together with the identification information of the client in real time; the identity identification information may be an ip address. Or MAC address;
  • Step S36 The determining step: the server waits to receive the screenshot data and the barrage data sent by the client, and the identity identification information of the client. For each barrage data saved in the server, if the server receives the client and the client sends the message within a limited time.
  • the screenshot data corresponding to the barrage data is RGB converted to the transmitted screenshot data, and converted into RGB data of the sent screenshot; the server compares the RGB data of the sent screenshot and the RGB data of the saved decoded image by a graphics comparison algorithm.
  • the server determines that the client plays the video status abnormally, and performs real-time alarm notification and the like, for example, notifying the person who initiated the live broadcast
  • the client plays the audio and video status abnormally, or feeds back to the server administrator the identity information of the client whose playback status is abnormal. If it is consistent, it determines that the video status of the client is normal.
  • a "computer-readable medium” can be any apparatus that can contain, store, communicate, propagate, or transport a program for use in an instruction execution system, apparatus, or device, or in conjunction with such an instruction execution system, apparatus, or device.
  • computer readable media include the following: electrical connections (electronic devices) having one or more wires, portable computer disk cartridges (magnetic devices), random access memory (RAM), Read only memory (ROM), erasable editable read only memory (EPROM or flash memory), fiber optic devices, and portable compact disk read only memory (CDROM).
  • the computer readable medium may even be a paper or other suitable medium on which the program can be printed, as it may be optically scanned, for example by paper or other medium, followed by editing, interpretation or, if appropriate, other suitable The method is processed to obtain the program electronically and then stored in computer memory.
  • portions of the invention may be implemented in hardware, software, firmware or a combination thereof.
  • multiple steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system.
  • a suitable instruction execution system For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques well known in the art: having logic gates for implementing logic functions on data signals. Discrete logic circuits, application specific integrated circuits with suitable combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computer Graphics (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

一种音视频直播中检测客户端播放状态的系统及方法，其中，系统包括：采集端、服务器和客户端，采集端与服务器连接，服务器与客户端连接；采集端能够在采集的音视频数据中加入状态检测信息的标记；服务器检测到音视频数据中的状态检测信息的标记，则将此时的图像进行解码；客户端能够从服务器获取音视频数据并播放；客户端在播放音视频数据时，如果检测到状态检测信息的标记，则实时记录该状态检测信息的标记，同时进行播放区域截图；如果对应于同一状态检测信息的标记的解码后的图像与发送的截图数据不一致或在限定时间内服务器没有接收到客户端发送的截图数据，则判定客户端播放状态异常。能够准确检测直播客户端当前的状态。

Description

一种音视频直播中检测客户端播放状态的系统及方法 技术领域
本发明属于网络直播技术领域，具体涉及一种音视频直播中检测客户端播放状态的系统及方法。
背景技术
直播，作为一种实时展示方式，受到越来越多的人，特别是年轻人的追捧。直播分为文字图片直播和视频直播，传统电视台多以视频直播为主，比如新闻类的《新闻联播》、演艺类的《春节联欢晚会》等。在网络时代，多以图文直播为主，比如直播体育赛事、新闻等。在移动互联网时代，文字、图片、视频皆可进行直播，通过网络，参与者既可以作为发起直播的人，也可以作为观看或收听直播的人。直播状态是否正常，对于发起直播的人和观看或收听直播的人来说，都是至关重要的。
目前的音视频直播状态检测只是通过数据是否到达、信令是否走通来判断是否直播成功。但这种音视频直播状态检测技术对于发起直播的人来说，他们并不知道音视频直播的实际运作情况，因此这种直播状态检测技术并不是很精确。
发明内容
为了解决上述的客户端直播状态检测不精确的技术问题，本发明实施例提出了一种音视频直播中检测客户端播放状态的系统及方法。
一种音视频直播中检测客户端播放状态的系统，包括：采集端、服务器和客 户端，其中，采集端与服务器连接，服务器与客户端连接；
采集端在采集的音视频数据中加入状态检测信息的标记，并将加入了状态检测信息标记的音视频数据上传到服务器；
服务器检测接收到的音视频数据，如果检测到状态检测信息的标记，则将此时的图像进行解码；
客户端从服务器获取音视频数据并播放；客户端在播放音视频数据时，如果检测到状态检测信息的标记，则实时记录该状态检测信息的标记，同时进行播放区域截图，并将截图数据和状态检测信息的标记数据一起实时发送到服务器；
服务器比较对应于同一状态检测信息的标记的解码后的图像与发送的截图数据是否一致，如果不一致或在限定时间内服务器没有接收到客户端发送的截图数据和状态检测信息的标记数据，则判定客户端播放状态异常。
进一步地，所述客户端将截图数据和状态检测信息的标记数据一起实时发送到服务器时，还连同客户端的身份标识信息一起发送。
进一步地，加入状态检测信息的标记的帧为关键帧。
进一步地，状态检测信息的标记为时间戳和/或弹幕。
进一步地，在判定客户端播放状态异常时，服务器向发起音视频直播的用户反馈该客户端播放状态异常，或向服务器管理员反馈播放状态异常的客户端的身份标识信息。
一种音视频直播中检测客户端播放状态的方法，该方法包括：
采集步骤：采集端采集音视频数据；
加标记步骤：采集端定时在采集到的音视频数据中加入状态检测信息的标记；
上传步骤：经过加标记步骤后，采集端将加入了状态检测信息的标记的音视频数据上传到服务器；
第一检测步骤：服务器检测接收到的音视频数据，如果服务器检测到状态检 测信息的标记，则将此时的图像进行解码；
第二检测步骤：客户端从服务器获取音视频数据，客户端在播放音视频数据时，如果检测到状态检测信息的标记，则实时记录该状态检测信息的标记，同时进行播放区域截图，并将截图数据和状态检测信息的标记数据一起实时发送到服务器；
判断步骤：服务器接收客户端发送的截图数据和状态检测信息的标记数据，并比较对应于同一状态检测信息的标记的解码后的图像与发送的截图数据是否一致，如果不一致或在限定时间内服务器没有接收到客户端发送的截图数据和状态检测信息的标记数据，则判定客户端播放状态异常。
进一步地，在第二检测步骤中，所述客户端将截图数据和状态检测信息的标记数据一起实时发送到服务器时，还连同客户端的身份标识信息一起发送。
进一步地，加入状态检测信息的标记的帧为关键帧。
进一步地，状态检测信息的标记为时间戳和/或弹幕。
进一步地，在判断步骤中，当判定客户端播放状态异常时，服务器向发起音视频直播的用户反馈该客户端播放状态异常，或向服务器管理员反馈播放状态异常的客户端的身份标识信息。
本发明实施例的有益效果：本发明实施例提出的音视频直播中检测客户端播放状态的系统及方法，相比现有技术中通过数据是否到达、接口是否调用成功等技术方案来说，能够更准确及时地检测直播过程中客户端的当前状态。
附图说明
图1为本发明实施例提出的音视频直播中检测客户端播放状态的系统的结构框图；
图2是本发明一实施例提出的音视频直播中检测客户端播放状态的方法的流程图；
图3是本发明另一实施例提出的音视频直播中检测客户端播放状态的方法的流程图。
具体实施方式
为使本发明的目的、技术方案和优点更加清楚明白，以下结合具体实施例，并参照附图，对本发明进一步详细说明。但本领域技术人员知晓，本发明并不局限于附图和以下实施例。
本发明实施例提出的音视频直播中检测客户端播放状态的系统，如图1所示，该系统包括：采集端11、服务器12和客户端13，其中，采集端11与服务器12连接，服务器12与客户端13连接。
采集端11采集音视频数据，并定时在采集到的音视频数据的关键帧中加入时间戳，从而使得音视频数据携带有状态检测信息；采集端11将加入了时间戳的音视频数据上传到服务器12。所述关键帧指的是不需要其他帧图像作参考，就可以独立进行编码的帧，亦可称为独立帧，一般是指的I帧。所述关键帧可以为一个或多个。
客户端13从服务器12下载音视频数据或由服务器12向客户端13转发音视频数据，客户端13对来自服务器12的音视频数据进行播放；客户端13在播放音视频数据时，如果检测到某帧音视频数据中加入了时间戳，则记录该时间戳数据，同时进行播放区域截图，并将截图数据和时间戳数据连同客户端13的身份标识信息一起实时发送到给服务器12，所述身份标识信息可以为ip地址或MAC地址。
服务器12检测采集端上传的音视频数据，如果检测到某帧数据中加入了时间戳时，则将加入了时间戳的该帧数据对应的图像进行解码，将解码后的图像转换为解码图像的RGB数据，关联地保存时间戳数据和解码图像的RGB数据以及客户端的身份标识信息。
服务器12等待接收客户端发送截图数据和时间戳数据以及客户端13的身份标识信息，对于服务器中保存的每个时间戳数据，服务器12如果在限定时间内接收到客户端13发送的与该时间戳数据对应的截图数据，则对该发送的截图数据进行RGB转换，转换为发送的截图的RGB数据，服务器12通过图形对比算法比较客户端13发送的截图的RGB数据和保存的解码图像的RGB数据是否一致；如果不一致或限定时间内服务器12没有接收到客户端13发送的与该时间戳数据对应的截图数据，则判定该客户端13播放音视频状态异常，并进行实时报警通知等操作，例如通知发起直播的人有客户端播放音视频状态异常，或向服务器管理员反馈播放状态异常的客户端的身份标识信息；如果一致，则判定客户端13播放音视频状态正常。
所述系统可以同时对多个客户端进行播放状态检测。
进一步地，限定时间可以由用户(例如发起直播的人)自行设定，也可以由服务器自动设定。对于每个时间戳来说，其限定时间可以相同，也可以不同。
上述实施例中，以时间戳作为状态检测信息的标记，本领域技术人员知晓，也可以采用其他时间标记的方式，或采用其他类型的标记方式，例如在某帧中加入弹幕，将弹幕作为状态检测信息的标记；或者将时间戳和弹幕一起作为状态检测信息的标记，此时时间戳和弹幕可以设置在同一帧中，也可以设置在不同帧中。
本发明实施例提出的音视频直播中检测客户端播放状态的方法，在本实施例中，以时间戳作为状态检测信息的标记。本实施例提出的方法如图2所示，该方法包括：
步骤S21、采集步骤：采集端采集音视频数据；
步骤S22、加标记步骤：采集端定时在采集到的音视频数据的关键帧中加入时间戳，从而使得音视频数据携带有状态检测信息；所述关键帧指的是不需要其他帧图像作参考，就可以独立进行编码的帧，亦可称为独立帧，一般是指的I帧。 所述关键帧可以为一个或多个。
步骤S23、上传步骤：采集端将加入了时间戳的音视频数据上传到服务器；
步骤S24、第一检测步骤：服务器检测采集端上传的音视频数据，当服务器检测到某帧数据中加入了时间戳时，则将加入了时间戳的该帧数据对应的图像进行解码，将解码后的图像转换为解码图像的RGB数据，关联地保存时间戳数据和解码图像的RGB数据；
步骤S25、第二检测步骤：客户端从服务器下载音视频数据或服务器向客户端转发音视频数据，客户端对来自服务器的音视频数据进行播放；客户端在播放音视频数据时如果检测到某帧数据中加入了时间戳，则记录该时间戳数据，同时进行播放区域截图，并将截图数据和时间戳数据连同客户端的身份识别信息一起实时发送到服务器；身份识别信息可以为ip地址或MAC地址；
步骤S26、判断步骤：服务器等待接收客户端发送的截图数据和时间戳数据以及客户端的身份识别信息，对于服务器中保存的每个时间戳数据，服务器如果在限定时间内接收到客户端发送的与该时间戳数据对应的截图数据，则对该发送的截图数据进行RGB转换，转换为发送的截图的RGB数据；服务器通过图形对比算法比较该发送的截图的RGB数据和保存的解码图像的RGB数据是否一致；如果不一致或限定时间内服务器没有接收到客户端发送的与该时间戳数据对应的截图数据，则判定该客户端播放音视频状态异常，并进行实时报警通知等操作，例如通知发起直播的人有客户端播放音视频状态异常，或向服务器管理员反馈播放状态异常的客户端的身份标识信息；如果一致，则判定客户端播放音视频状态正常。
本发明另一实施例提出的音视频直播中检测客户端播放状态的方法，在本实施例中，以弹幕作为检测状态信息的标记，在音视频数据的某些帧中加入弹幕。该实施例提出的方法如图3所示，包括：
步骤S31、采集步骤：采集端采集音视频数据；
步骤S32、加标记步骤：采集端定时在采集到的音视频数据的某些帧中加入弹幕，从而使得音视频数据携带有状态检测信息；
步骤S33、上传步骤：采集端将加入弹幕的音视频数据上传到服务器；
步骤S34、第一检测步骤：服务器检测采集端上传的音视频数据，当服务器检测到某帧数据中加入了弹幕时，则将加入了弹幕的该帧数据对应的图像进行解码，将解码后的图像转换为解码图像的RGB数据，关联地保存弹幕和解码图像的RGB数据；
步骤S35、第二检测步骤：客户端从服务器下载音视频数据或服务器向客户端转发音视频数据，客户端对来自服务器的音视频数据进行播放；客户端在播放音视频数据时如果检测到某帧数据加入了弹幕，则记录该弹幕数据，同时进行播放区域截图，并将截图数据和弹幕数据连同该客户端的身份识别信息一起实时发送到服务器；所述身份标识信息可以为ip地址或MAC地址；
步骤S36、判断步骤：服务器等待接收客户端发送的截图数据和弹幕数据以及客户端的身份标识信息，对于服务器中保存的每个弹幕数据，服务器如果在限定时间内接收到客户端发送的与该弹幕数据对应的截图数据，则对该发送的截图数据进行RGB转换，转换为发送的截图的RGB数据；服务器通过图形对比算法比较该发送的截图的RGB数据和保存的解码图像的RGB数据是否一致；如果不一致或限定时间内服务器没有接收到客户端发送的与该弹幕数据对应的截图数据，则判定客户端播放视频状态异常，并进行实时报警通知等操作，例如通知发起直播的人有客户端播放音视频状态异常，或向服务器管理员反馈播放状态异常的客户端的身份标识信息；如果一致，则判定客户端播放视频状态正常。
本领域技术人员可以理解，在流程图中表示或在此以其他方式描述的逻辑和/或步骤，例如，可以被认为是用于实现逻辑功能的可执行指令的定序列表，可 以具体实现在任何计算机可读介质中，以供指令执行系统、装置或设备(如基于计算机的系统、包括处理器的系统或其他可以从指令执行系统、装置或设备取指令并执行指令的系统)使用，或结合这些指令执行系统、装置或设备而使用。就本说明书而言，“计算机可读介质”可以是任何可以包含、存储、通信、传播或传输程序以供指令执行系统、装置或设备或结合这些指令执行系统、装置或设备而使用的装置。
计算机可读介质的更具体的示例(非穷尽性列表)包括以下：具有一个或多个布线的电连接部(电子装置)，便携式计算机盘盒(磁装置)，随机存取存储器(RAM)，只读存储器(ROM)，可擦除可编辑只读存储器(EPROM或闪速存储器)，光纤装置，以及便携式光盘只读存储器(CDROM)。另外，计算机可读介质甚至可以是可在其上打印所述程序的纸或其他合适的介质，因为可以例如通过对纸或其他介质进行光学扫描，接着进行编辑、解译或必要时以其他合适方式进行处理来以电子方式获得所述程序，然后将其存储在计算机存储器中。
应当理解，本发明的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中，多个步骤或方法可以用存储在存储器中且由合适的指令执行系统执行的软件或固件来实现。例如，如果用硬件来实现，和在另一实施方式中一样，可用本领域公知的下列技术中的任一项或它们的组合来实现：具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路，具有合适的组合逻辑门电路的专用集成电路，可编程门阵列(PGA)，现场可编程门阵列(FPGA)等。
在本说明书的描述中，参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本发明的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不一定指的是相同的实施例或示例。而且，描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施例或示例中以合适的方式结合。
以上，对本发明的实施方式进行了说明。但是，本发明不限定于上述实施方式。凡在本发明的精神和原则之内，所做的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。

Claims (10)

  1. 一种音视频直播中检测客户端播放状态的系统，其特征在于，包括：采集端、服务器和客户端，其中，采集端与服务器连接，服务器与客户端连接；
    采集端在采集的音视频数据中加入状态检测信息的标记，并将加入了状态检测信息标记的音视频数据上传到服务器；
    服务器检测接收到的音视频数据，如果检测到状态检测信息的标记，则将此时的图像进行解码；
    客户端从服务器获取音视频数据并播放；客户端在播放音视频数据时，如果检测到状态检测信息的标记，则实时记录该状态检测信息的标记，同时进行播放区域截图，并将截图数据和状态检测信息的标记数据一起实时发送到服务器；
    服务器比较对应于同一状态检测信息的标记的解码后的图像与发送的截图数据是否一致，如果不一致或在限定时间内服务器没有接收到客户端发送的截图数据和状态检测信息的标记数据，则判定客户端播放状态异常。
  2. 根据权利要求1所述的系统，其特征在于，所述客户端将截图数据和状态检测信息的标记数据一起实时发送到服务器时，还连同客户端的身份标识信息一起发送。
  3. 根据权利要求2所述的系统，其特征在于，加入状态检测信息的标记的帧为关键帧。
  4. 根据权利要求1所述的系统，其特征在于，状态检测信息的标记为时间戳和/或弹幕。
  5. 根据权利要求1至4中任一项所述的系统，其特征在于，在判定客户端播 放状态异常时，服务器向发起音视频直播的用户反馈该客户端播放状态异常，或向服务器管理员反馈播放状态异常的客户端的身份标识信息。
  6. 一种音视频直播中检测客户端播放状态的方法，其特征在于，该方法包括：
    采集步骤：采集端采集音视频数据；
    加标记步骤：采集端定时在采集到的音视频数据中加入状态检测信息的标记；
    上传步骤：经过加标记步骤后，采集端将加入了状态检测信息的标记的音视频数据上传到服务器；
    第一检测步骤：服务器检测接收到的音视频数据，如果服务器检测到状态检测信息的标记，则将此时的图像进行解码；
    第二检测步骤：客户端从服务器获取音视频数据，客户端在播放音视频数据时，如果检测到状态检测信息的标记，则实时记录该状态检测信息的标记，同时进行播放区域截图，并将截图数据和状态检测信息的标记数据一起实时发送到服务器；
    判断步骤：服务器接收客户端发送的截图数据和状态检测信息的标记数据，并比较对应于同一状态检测信息的标记的解码后的图像与发送的截图数据是否一致，如果不一致或在限定时间内服务器没有接收到客户端发送的截图数据和状态检测信息的标记数据，则判定客户端播放状态异常。
  7. 根据权利要求6所述的方法，其特征在于，在第二检测步骤中，所述客户端将截图数据和状态检测信息的标记数据一起实时发送到服务器时，还连同客户端的身份标识信息一起发送。
  8. 根据权利要求7所述的方法，其特征在于，加入状态检测信息的标记的帧 为关键帧。
  9. 根据权利要求6所述的方法，其特征在于，状态检测信息的标记为时间戳和/或弹幕。
  10. 根据权利要求6至9中任一项所述的方法，其特征在于，在判断步骤中，当判定客户端播放状态异常时，服务器向发起音视频直播的用户反馈该客户端播放状态异常，或向服务器管理员反馈播放状态异常的客户端的身份标识信息。
PCT/CN2017/102283 2017-03-14 2017-09-19 一种音视频直播中检测客户端播放状态的系统及方法 WO2018166162A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710150872.X 2017-03-14
CN201710150872.XA CN106803997B (zh) 2017-03-14 2017-03-14 一种音视频直播中检测客户端播放状态的系统及方法

Publications (1)

Publication Number Publication Date
WO2018166162A1 true WO2018166162A1 (zh) 2018-09-20

Family

ID=58987973

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/102283 WO2018166162A1 (zh) 2017-03-14 2017-09-19 一种音视频直播中检测客户端播放状态的系统及方法

Country Status (2)

Country Link
CN (1) CN106803997B (zh)
WO (1) WO2018166162A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113766306A (zh) * 2021-04-21 2021-12-07 腾讯科技（北京）有限公司 检测视频卡顿的方法、装置、计算机设备及存储介质

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106803997B (zh) * 2017-03-14 2019-12-17 广州视源电子科技股份有限公司 一种音视频直播中检测客户端播放状态的系统及方法
CN110380843B (zh) * 2018-04-13 2022-12-02 武汉斗鱼网络科技有限公司 一种信息处理方法及相关设备
CN114363063A (zh) * 2018-11-01 2022-04-15 西安万像电子科技有限公司 数据传输方法、装置及系统
CN111641847A (zh) * 2020-06-11 2020-09-08 南昌威爱教育科技有限公司 一种用于虚拟现实教学的数据传输方法
CN112218175B (zh) * 2020-12-09 2021-03-02 深圳市房多多网络科技有限公司 直播间状态的处理方法、装置及计算设备
CN112911325B (zh) * 2021-01-29 2023-07-14 百果园技术(新加坡)有限公司 一种跨直播间连线的恢复方法和装置
CN113886206B (zh) * 2021-09-30 2023-11-03 南京奥拓电子科技有限公司 一种互动传媒终端用户行为数据采集方法及系统

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103974061A (zh) * 2014-05-27 2014-08-06 合一网络技术(北京)有限公司 一种播放测试方法及系统
US20140233854A1 (en) * 2013-02-15 2014-08-21 Yahoo! Inc. Real time object scanning using a mobile phone and cloud-based visual search engine
CN105245514A (zh) * 2015-09-28 2016-01-13 珠海多玩信息技术有限公司 外挂识别方法、装置及系统
CN106028147A (zh) * 2016-06-23 2016-10-12 北京华兴宏视技术发展有限公司 视频信号监测方法及视频信号监测系统
CN106488291A (zh) * 2016-11-17 2017-03-08 百度在线网络技术（北京）有限公司 在视频直播中同步显示文件的方法和装置
CN106803997A (zh) * 2017-03-14 2017-06-06 广州视源电子科技股份有限公司 一种音视频直播中检测客户端播放状态的系统及方法

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106101825B (zh) * 2015-04-30 2019-04-02 北京视联动力国际信息技术有限公司 一种终端监控的方法和服务器
CN106302477A (zh) * 2016-08-18 2017-01-04 合网络技术（北京）有限公司 一种视频直播测试方法及系统
CN106412662B (zh) * 2016-09-20 2018-10-19 腾讯科技（深圳）有限公司 时间戳分配方法及装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140233854A1 (en) * 2013-02-15 2014-08-21 Yahoo! Inc. Real time object scanning using a mobile phone and cloud-based visual search engine
CN103974061A (zh) * 2014-05-27 2014-08-06 合一网络技术(北京)有限公司 一种播放测试方法及系统
CN105245514A (zh) * 2015-09-28 2016-01-13 珠海多玩信息技术有限公司 外挂识别方法、装置及系统
CN106028147A (zh) * 2016-06-23 2016-10-12 北京华兴宏视技术发展有限公司 视频信号监测方法及视频信号监测系统
CN106488291A (zh) * 2016-11-17 2017-03-08 百度在线网络技术（北京）有限公司 在视频直播中同步显示文件的方法和装置
CN106803997A (zh) * 2017-03-14 2017-06-06 广州视源电子科技股份有限公司 一种音视频直播中检测客户端播放状态的系统及方法

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113766306A (zh) * 2021-04-21 2021-12-07 腾讯科技（北京）有限公司 检测视频卡顿的方法、装置、计算机设备及存储介质
CN113766306B (zh) * 2021-04-21 2023-11-14 腾讯科技（北京）有限公司 检测视频卡顿的方法、装置、计算机设备及存储介质

Also Published As

Publication number Publication date
CN106803997B (zh) 2019-12-17
CN106803997A (zh) 2017-06-06

Similar Documents

Publication Publication Date Title
WO2018166162A1 (zh) 一种音视频直播中检测客户端播放状态的系统及方法
US9667920B2 (en) Hybrid active and passive people metering for audience measurement
ES2688026T3 (es) El uso de huellas digitales para asociar datos con una obra
US10305613B2 (en) Method and system for detecting image delay
CN104488277B (zh) 用于监测媒体呈现的方法和装置
US9344760B2 (en) Information processing apparatus, information processing method, and program
WO2017092360A1 (zh) 多媒体播放时的交互方法及装置
CN107231581B (zh) 用于视频播放的方法、系统及流媒体播放控制服务器
WO2015096729A1 (zh) 多媒体直播举报的方法、终端、服务器及系统
US20160154857A1 (en) Electronic data generation methods
CN111147808B (zh) 网络装置、影像处理方法及电脑可读媒体
CN110740386B (zh) 直播切换方法、装置及存储介质
US20160029053A1 (en) Method for transmitting media data and virtual desktop server
US11539985B2 (en) No reference realtime video quality assessment
WO2016050113A1 (zh) 一种业务实现方法、设备及存储介质
TW201806380A (zh) 動畫分割裝置及監視方法
CN106797327A (zh) 使用与自适应比特率流传输相关联的消息执行对移动平台的媒体监视
KR20140117470A (ko) 디지털 영화에서 광고 재생 출력 확인을 위한 방법 및 장치
EP3754998B1 (en) Streaming media quality monitoring method and system
CN112822435A (zh) 一种用户可轻松接入的安防方法、装置及系统
US20140025782A1 (en) System and method for playing and transmitting network video
JP2009194767A (ja) ビデオ評価装置及び方法、並びにビデオ提供装置
CN109194971A (zh) 一种为多媒体文件的生成方法及装置
US10885343B1 (en) Repairing missing frames in recorded video with machine learning
JP2015232916A (ja) 操作記録装置、操作記録再生システム、及びプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17900604

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17900604

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 16.03.2020)

122 Ep: pct application non-entry in european phase

Ref document number: 17900604

Country of ref document: EP

Kind code of ref document: A1