CN115134617A - Live broadcast quality inspection system, method, Web server and storage medium - Google Patents
Live broadcast quality inspection system, method, Web server and storage medium Download PDFInfo
- Publication number
- CN115134617A CN115134617A CN202210682572.7A CN202210682572A CN115134617A CN 115134617 A CN115134617 A CN 115134617A CN 202210682572 A CN202210682572 A CN 202210682572A CN 115134617 A CN115134617 A CN 115134617A
- Authority
- CN
- China
- Prior art keywords
- monitor
- quality inspection
- live broadcast
- data stream
- live
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000007689 inspection Methods 0.000 title claims abstract description 207
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000003860 storage Methods 0.000 title claims abstract description 14
- 230000002159 abnormal effect Effects 0.000 claims abstract description 96
- 238000004891 communication Methods 0.000 claims abstract description 68
- 238000012544 monitoring process Methods 0.000 claims abstract description 42
- 238000012795 verification Methods 0.000 claims abstract description 26
- 238000009826 distribution Methods 0.000 claims description 66
- 239000002131 composite material Substances 0.000 claims description 30
- 230000003111 delayed effect Effects 0.000 claims description 13
- 230000003068 static effect Effects 0.000 claims description 11
- 230000015271 coagulation Effects 0.000 claims description 8
- 238000005345 coagulation Methods 0.000 claims description 8
- 238000004064 recycling Methods 0.000 claims description 6
- 238000012552 review Methods 0.000 claims description 6
- 238000011084 recovery Methods 0.000 claims description 2
- 238000011161 development Methods 0.000 description 14
- 238000001514 detection method Methods 0.000 description 13
- 238000012545 processing Methods 0.000 description 11
- 230000008569 process Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 8
- 230000000007 visual effect Effects 0.000 description 7
- 238000012550 audit Methods 0.000 description 6
- 238000012372 quality testing Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000005538 encapsulation Methods 0.000 description 3
- 238000004806 packaging method and process Methods 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 238000007711 solidification Methods 0.000 description 3
- 230000008023 solidification Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 238000003973 irrigation Methods 0.000 description 2
- 230000002262 irrigation Effects 0.000 description 2
- 238000003058 natural language processing Methods 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000008713 feedback mechanism Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 238000003908 quality control method Methods 0.000 description 1
- 230000005236 sound signal Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000003245 working effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/04—Real-time or near real-time messaging, e.g. instant messaging [IM]
- H04L51/046—Interoperability with other network applications or services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
Abstract
The invention discloses a live broadcast quality inspection system, a live broadcast quality inspection method, a Web server and a storage medium. The system comprises: the quality inspection and verification module, the instant communication module and the data display module are arranged in the computer; the quality inspection and verification module is respectively connected with the instant communication module and the data display module, the instant communication module is connected with external instant communication software, and the data display module is connected with a Web front end; the quality inspection auditing module is used for acquiring and monitoring a live broadcast data stream; the instant communication module is used for acquiring abnormal data contained in the live broadcast data stream monitored by the quality inspection and verification module and outputting the abnormal data; and the data display module is used for acquiring abnormal data contained in the live data stream monitored by the quality inspection and auditing module and displaying the abnormal data. The invention aims to provide a set of real-time and flexible live broadcast quality inspection system, which realizes high-efficiency multidimensional quality inspection and abnormal real-time notification.
Description
Technical Field
The invention relates to the technical field of audio and video processing, in particular to a live broadcast quality inspection system, a live broadcast quality inspection method, a Web server and a storage medium.
Background
In the field of current audio and video processing, a large number of audio and video implementation libraries such as huffyuv, WebRTC (Web Real-Time Communication, Web instant messaging), live555 and the like are developed. As an outstanding development technology in the field, FFmpeg (Fast Forward Mpeg) becomes the first choice of audio and video developers by means of an active community atmosphere, sound protocol support, strong function coverage and flexible development entrance. With the help of rich command instructions provided by FFmpeg and directed acyclic Graph model design, developers can take basic operations such as encapsulation, coding and decoding, filtering and the like as nodes for business requirements, and disassemble the business requirements into DAG (directed acyclic Graph); and audio and video data is input from a DAG inlet, and a processing result is output by a node with the output degree of 0, so that live broadcast quality inspection is realized. The working environment of the mode only needs to install the FFmpeg, and developers only need to convert the requirements into a shell instruction. However, the method has the defect of insufficient flexibility, developers can only adopt some basic components provided by FFmpeg, and high-order custom logic is not easy to access; when the complexity of the service is slightly high, the requirement disassembly is complicated, and the instruction splicing is inconvenient.
Disclosure of Invention
The embodiment of the invention mainly aims to provide a live broadcast quality inspection system, a live broadcast quality inspection method, a Web server and a storage medium, and aims to provide a real-time and flexible live broadcast quality inspection system to realize efficient multi-dimensional quality inspection and abnormal real-time notification.
In order to achieve the above object, an embodiment of the present invention provides a live broadcast quality inspection system, which is developed based on JavaCV, FFmpeg, and OpenCV, and includes:
the quality inspection and verification module, the instant communication module and the data display module; the quality inspection and verification module is respectively connected with the instant communication module and the data display module, the instant communication module is connected with external instant communication software, and the data display module is connected with a Web front end;
the quality inspection auditing module is used for acquiring and monitoring a live broadcast data stream;
the instant communication module is used for acquiring abnormal data contained in the live broadcast data stream monitored by the quality inspection and verification module and outputting the abnormal data;
and the data display module is used for acquiring abnormal data contained in the live broadcast data stream monitored by the quality inspection and audit module and displaying the abnormal data.
Optionally, the quality inspection and auditing module includes a monitor abstraction layer, an asynchronous frame distribution queue, a video frame monitor abstraction layer, a composite monitor abstraction layer, an audio frame monitor abstraction layer, an image distribution queue, a composite frame distribution queue, an audio distribution queue, a resource recycling layer, a black screen monitor, a fuzzy monitor, a static field monitor, a brightness monitor, a two-dimensional code monitor, an image auditing monitor, a flower screen monitor, a solidification monitor, a silence monitor, and a voice auditing monitor;
the monitor abstraction layer is connected with the asynchronous frame distribution queue, the asynchronous frame distribution queue is respectively connected with the video frame monitor abstraction layer, the composite monitor abstraction layer and the audio frame monitor abstraction layer, the video frame abstraction layer is connected with the image distribution queue, the image distribution queue is respectively connected with the black screen monitor, the fuzzy monitor, the static field monitor, the brightness monitor, the two-dimensional code monitor and the image auditing monitor, the composite monitor abstraction layer is connected with the composite frame distribution queue, the composite frame distribution queue is connected with the flower screen monitor and the coagulation monitor, the audio frame monitor is connected with the audio distribution queue, the audio distribution queue is connected with the silent monitor and the voice auditing monitor, the black screen monitor and the audio frame monitor, The fuzzy monitor, the static field monitor, the brightness monitor, the two-dimensional code monitor, the image auditing monitor, the flower screen monitor, the coagulation monitor, the mute monitor and the voice auditing monitor are respectively connected with the resource recycling layer.
Optionally, the data presentation module is further configured to:
acquiring the live broadcast data stream and displaying a live broadcast state corresponding to the live broadcast data stream;
displaying live comments contained in the live data stream;
and summarizing abnormal data contained in the live broadcast data stream to obtain an analysis report, and displaying the analysis report.
The embodiment of the invention provides a live broadcast quality inspection method, which comprises the following steps:
acquiring a data stream through a quality inspection and auditing module and monitoring in real time;
if the quality inspection and verification module monitors that the live broadcast data stream contains abnormal data, the abnormal data is output through the instant communication module, and the abnormal data is displayed through the data display module.
Optionally, the quality inspection and review module includes a monitor abstraction layer and an asynchronous frame distribution queue, and the step of acquiring and monitoring the live data stream includes:
acquiring the live broadcast data stream through the monitor abstract layer, and writing the live broadcast data stream into the asynchronous frame distribution queue after decapsulation;
decoding the live broadcast data stream through the asynchronous frame distribution queue and then distributing the decoded live broadcast data stream to a corresponding monitor;
and monitoring the live data stream through a monitor corresponding to the live data stream.
Optionally, the step of obtaining the data stream through the quality inspection and auditing module and monitoring the data stream in real time includes:
converting the abnormal data into communication information through the instant communication module;
and outputting the communication information to external instant communication software through the instant communication module.
Optionally, the step of obtaining the data stream through the quality inspection and auditing module and monitoring the data stream in real time includes:
if the quality inspection auditing module monitors that the live broadcast corresponding to the live broadcast data stream is not broadcast on time, the quality inspection auditing module judges whether the live broadcast corresponding to the live broadcast data stream is broadcast in absence;
and if the quality inspection auditing module monitors that the live broadcast corresponding to the live broadcast data stream is not in an open broadcast state, determining that the live broadcast corresponding to the live broadcast data stream is in a delayed broadcast state, and outputting the information of the live broadcast in the delayed broadcast state corresponding to the live broadcast data stream through the instant communication module and the data display module.
Optionally, the step of obtaining the data stream through the quality inspection and auditing module and monitoring the data stream in real time includes:
judging whether the live broadcast data stream is continuous or not through the quality inspection and auditing module;
if the live data stream is continuous, the quality inspection auditing module is used for continuously monitoring the live data stream;
and if the quality inspection auditing module monitors that the live data stream stops, stopping monitoring the live data stream through the quality inspection auditing module.
In addition, to achieve the above object, the present invention also provides a Web server, including: the system comprises a memory, a processor and a live broadcast quality inspection system which is stored on the memory and can run on the processor, wherein when the processor executes a program of the live broadcast quality inspection, the steps of the live broadcast quality inspection system are realized.
In addition, to achieve the above object, the present invention further provides a computer-readable storage medium having stored thereon a program for live quality inspection, which when executed by a processor, implements the steps of the live quality inspection system as described above.
The embodiment of the invention provides a live broadcast quality inspection system, a live broadcast quality inspection method, a Web server and a storage medium, wherein the system comprises: the quality inspection and verification module, the instant communication module and the data display module; the quality inspection and verification module is respectively connected with the instant communication module and the data display module, the instant communication module is connected with external instant communication software, and the data display module is connected with a Web front end; the quality inspection auditing module is used for acquiring and monitoring a live broadcast data stream; the instant communication module is used for acquiring abnormal data contained in the live broadcast data stream monitored by the quality inspection and verification module and outputting the abnormal data; and the data display module is used for acquiring abnormal data contained in the live data stream monitored by the quality inspection and auditing module and displaying the abnormal data. According to the method and the device, the live broadcast data stream is monitored in real time through the quality inspection and auditing module, and when the quality inspection and auditing module monitors that abnormal data exist in the live broadcast data stream, the abnormal data are output through the instant communication module and the data display module, so that a user can know that the abnormal data exist in the live broadcast data stream in multiple ways, and real-time communication in the live broadcast quality inspection process is realized. Because quality inspection audits module, instant messaging module, data display module have high expansibility, can expand according to the demand, be connected with multiple equipment to this has solved the not enough problem of live quality inspection flexibility, has improved quality inspection efficiency.
Drawings
FIG. 1 is a schematic diagram of functional modules of a Web server to which a live broadcast quality inspection device of the present invention belongs;
FIG. 2 is a functional block diagram of the live broadcast quality inspection system according to the present invention;
FIG. 3 is an architecture diagram relating to a live broadcast quality inspection system of the present invention;
FIG. 4 is a schematic view of an interface display involved in the live broadcast quality inspection system of the present invention;
FIG. 5 is a schematic diagram of a live comment analysis PC side to which the live quality inspection system of the present invention relates;
fig. 6 is a schematic diagram of a live comment analysis mobile terminal to which the live quality inspection system of the present invention relates;
FIG. 7 is a schematic view of a detection life cycle involved in the live broadcast quality inspection system of the present invention;
fig. 8 is a data flow diagram of a quality inspection auditing module related to the live broadcast quality inspection system of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The main solution of the embodiment of the invention is to disclose a live broadcast quality inspection system, which comprises: the quality inspection and verification module, the instant communication module and the data display module; the quality inspection auditing module is respectively connected with the instant communication module and the data display module, the instant communication module is connected with external instant communication software, and the data display module is connected with a Web front end; the quality inspection auditing module is used for acquiring and monitoring a live broadcast data stream; the instant communication module is used for acquiring abnormal data contained in the live broadcast data stream monitored by the quality inspection and verification module and outputting the abnormal data; and the data display module is used for acquiring abnormal data contained in the live broadcast data stream monitored by the quality inspection and audit module and displaying the abnormal data.
The technical terms related to the embodiment of the invention are as follows:
robustness: robust is the transliteration of Robust, i.e., in the sense of being Robust and Robust. It also represents the ability of the system to survive abnormal and dangerous conditions. For example, whether computer software is halted or not, or whether the computer software is crashed or not, or whether the computer software is not crashed or not, is the robustness of the software in the case of input errors, disk failures, network overload or intentional attacks. By "robustness", it is also meant that the control system maintains some other characteristic under certain (structural, size) parameter perturbation. According to different definitions of performance, stable robustness and performance robustness can be divided. A fixed controller designed with the robustness of a closed loop system as a goal is called a robust controller.
The JavaCV is: JavaCV is a packaging library based on a java cpp calling mode (JNI one-layer packaging), which is composed of multiple open-source computer vision libraries, and commonly used libraries and utility classes in the computer vision field including FFmpeg, OpenCV, tenserflow, caffe, tesseract, libdc1394, OpenKinect, videoInput, ARToolKitPlus and the like are packaged.
huffyuv: huffyuv is a lossless video compression coding method invented by Ben Rudiak-Gould and used for replacing uncompressed YCbCr as an acquisition format.
WebRTC: web Real-Time Communication (Web instant messaging), which is an API supporting a Web browser to perform Real-Time voice or video conversations.
live 555: the cross-platform C + + open source project provides a solution for streaming media, and supports standard streaming media transmission protocols such as RTP/RTCP, RTSP, SIP and the like. live555 realizes streaming, receiving, processing and other support for audio and video data in a plurality of audio and video coding formats, including MPEG, h.263+, DV, JPEG video and a plurality of audio coding. Meanwhile, due to good design, Live555 is very easy to extend the support of other formats. live555 has been used for the realization of streaming media playing functions of multiple players, such as vlc (videolan), and MPlayer.
FFmpeg: (Fast Forward Mpeg) FFmpeg is a set of open source computer programs that can be used to record, convert digital audio, video, and convert them into streams. LGPL or GPL licenses are used. It provides a complete solution for recording, converting and streaming audio and video. It contains a very advanced audio/video codec library libavcodec, and many of the codes in libavcodec are developed from the beginning in order to ensure high portability and codec quality. FFmpeg was developed under the Linux platform, but it could also be compiled to run in other operating system environments as well, including Windows, Mac OS X, and the like.
A DAG: the directed acyclic Graph is a Graph structure in Graph theory, and consists of a limited number of vertexes and directed edges, and starting from any vertex, the Graph cannot return to the starting vertex through a plurality of directed edges.
A shell: in computer science, shell is commonly called shell (to distinguish it from core) and refers to software (command interpreter) that provides an operation interface for users. It is similar to command.com and later cmd.exe under DOS. It receives a user command and then invokes the corresponding application.
API: application Programming Interface (API), the primary purpose of an API is to provide applications and developers the ability to access a set of routines without accessing source code or understanding the details of the internal workings. Software that provides the functionality defined by an API is referred to as an implementation of this API. An API is an interface and thus an abstraction.
ASR: (Automatic Speech Recognition technology), which is a technology for converting human Speech into text.
IM: instant Messaging is a real-time communication system that allows two or more people to communicate text messages, files, voice and video in real time using a network.
The detection effect of the live broadcast quality inspection depends on the technical model selection in the field of audio and video. Currently, in the field of audio and video processing, audio and video protocols proposed by various manufacturers are various, a large number of audio and video implementation libraries such as huffyuv, WebRTC, live555 and the like emerge, and various schemes are continuously updated along with the development and popularization of audio and video hardware. The FFmpeg as an outstanding development technology in the field is supported by an active community atmosphere and a sound protocol, has strong functional coverage and flexible development entrance, and becomes the first choice of an audio-video developer. In the field of live broadcast quality inspection, the FFmpeg is a popular market in the field of live broadcast quality inspection by virtue of strong audio and video processing capability. At present, the solutions for live broadcast quality inspection mainly include the following three types:
(1) WebRTC-based tool development
The WebRTC integrates full-line functions of audio and video acquisition, coding and decoding, transmission, display and the like, is brought into the H5 standard, and is favored by broad developers. However, the technology also has obvious defects of unstable transmission quality, incomplete protocol compatibility and the like, and needs to be further improved and developed.
(2) FFmpeg instruction set based development
By means of rich command line instructions provided by FFmpeg and Directed Acyclic graph model (DAG) design, developers can take basic operations such as encapsulation, coding and decoding, filtering and the like as nodes to disassemble business requirements into DAG. And audio and video data is input from a DAG inlet, and a processing result is output by a node with the out degree of 0. The working environment of the mode only needs to install FFmpeg; the development is simple and convenient, and developers only need to convert the requirements into a shell instruction. However, this approach also has significant disadvantages: firstly, the flexibility is insufficient, developers can only adopt some basic components provided by FFmpeg, and high-order custom logic is not easy to access; when the service complexity is slightly high, the requirement disassembly is complicated, and the instruction splicing is inconvenient; the audio and video data streams flow in the DAG relative to the black boxes, are not easy to master and are inconvenient to interact with an external system.
(3) FFmpeg API-based development
In order to process audio-video data more flexibly, some developers choose to develop the data through the FFmpeg native API. The technical scheme gives the developer great development freedom, but because native API is too scattered, the technical details needing attention of the developer are more, the development cost is high, and the project period is long. Other audio and video advanced developers provide a high-level API Wrapper in a mode of an open source class library through further packaging on the basis of the native FFmpeg. The development threshold of a developer is reduced, and the realization of the service value is accelerated. The typical scheme is Python Wrapper integrated into the computer vision ecology, and the scheme brings a larger imagination space for the audio and video application scene. However, due to the requirement of high performance for live broadcast quality inspection, a part of performance indexes is sacrificed by means of Python Wrapper. Meanwhile, most enterprise services are constructed on Java services at present, and the development direction of enterprises is better met by means of a Java Wrapper scheme on environmental equipment and service operation and maintenance.
The invention provides a solution, and aims to provide a real-time and flexible live broadcast quality inspection system to realize efficient multidimensional quality inspection and abnormal real-time notification.
Specifically, referring to fig. 1, fig. 1 is a schematic diagram of a Web service deployment of the live broadcast quality inspection system of the present invention. The live broadcast quality inspection method is deployed on a Web server in a software package mode and provides a front-end page. The user can access the intelligent mobile terminal with data processing capability and the fixed terminal equipment for use.
In this embodiment, the live broadcast quality inspection system is composed of a presentation layer 110, a logic layer 120, a data layer 130, and a network layer 140.
The display layer 110 is mainly an interface, a page, etc. for interacting with an external system and a user; a core program of live broadcast quality inspection is packaged in the logic layer 120; the data layer 130 provides persistence, caching and the like of data generated in the operation of the system; the network layer 140 provides support for the HTTP, RTMP protocol.
Wherein the core logic code in the logic layer 120, when executed, implements the following steps:
acquiring a data stream through a quality inspection and auditing module and monitoring in real time;
if the quality inspection and verification module monitors that the live broadcast data stream contains abnormal data, the abnormal data is output through the instant communication module, and the abnormal data is displayed through the data display module.
Further, the core logic code in the logic layer 120, when executed, also implements the following steps:
converting the abnormal data into communication information through the instant communication module;
and outputting the communication information to external instant communication software through the instant communication module.
Further, the core logic code in the logic layer 120, when executed, also implements the following steps:
acquiring the live broadcast data stream through the monitor abstraction layer, and writing the live broadcast data stream into the asynchronous frame distribution queue after decapsulation;
decoding the live broadcast data stream through the asynchronous frame distribution queue and then distributing the decoded live broadcast data stream to a corresponding monitor;
and monitoring the live data stream through a monitor corresponding to the live data stream.
Further, the core logic code in the logic layer 120, when executed, also implements the following steps:
if the quality inspection auditing module monitors that the live broadcast corresponding to the live broadcast data stream is not broadcast on time, the quality inspection auditing module judges whether the live broadcast corresponding to the live broadcast data stream is broadcast in absence;
and if the quality inspection auditing module monitors that the live broadcast corresponding to the live broadcast data stream is not in an open broadcast state, determining that the live broadcast corresponding to the live broadcast data stream is in a delayed broadcast state, and outputting the information of the live broadcast in the delayed broadcast state corresponding to the live broadcast data stream through the instant communication module and the data display module.
Further, the core logic code in the logic layer 120, when executed, also implements the following steps:
judging whether the live broadcast data stream is continuous or not through the quality inspection and auditing module;
if the live data stream is continuous, the quality inspection auditing module is used for continuously monitoring the live data stream;
and if the quality inspection auditing module monitors that the live data stream stops, stopping monitoring the live data stream through the quality inspection auditing module.
According to the scheme, the data stream is acquired through the quality inspection and auditing module and is monitored in real time; if the quality inspection and verification module monitors that the live broadcast data stream contains abnormal data, the abnormal data is output through the instant communication module, and the abnormal data is displayed through the data display module. According to the method and the device, the live broadcast data stream is monitored in real time through the quality inspection and auditing module, and when the quality inspection and auditing module monitors that abnormal data exist in the live broadcast data stream, the abnormal data are output through the instant communication module and the data display module, so that a user can know that the abnormal data exist in the live broadcast data stream in multiple ways, and real-time communication in the live broadcast quality inspection process is realized. Because quality inspection audits module, instant messaging module, data display module have high expansibility, can expand according to the demand, be connected with multiple equipment to this has solved the not enough problem of live quality inspection flexibility, has improved quality inspection efficiency.
Referring to fig. 2, fig. 2 is a functional module schematic diagram of the live broadcast quality inspection system of the present invention. Live quality control system includes:
a quality inspection and verification module 10, an instant communication module 20 and a data display module 30; the quality inspection auditing module 10 is respectively connected with the instant communication module 20 and the data display module 30, the instant communication module 20 is connected with external instant communication software, and the data display module 30 is connected with a Web front end;
the quality inspection auditing module 10 is used for acquiring and monitoring live broadcast data streams;
the instant messaging module 20 is configured to obtain abnormal data included in the live broadcast data stream monitored by the quality inspection and review module, and output the abnormal data;
the data display module 30 is configured to obtain abnormal data included in the live broadcast data stream monitored by the quality inspection and review module, and display the abnormal data.
The quality inspection auditing module 10 comprises a monitor abstract layer, an asynchronous frame distribution queue, a video frame monitor abstract layer, a composite monitor abstract layer, an audio frame monitor abstract layer, an image distribution queue, a composite frame distribution queue, an audio distribution queue, a resource recycling layer, a black screen monitor, a fuzzy monitor, a static field monitor, a brightness monitor, a two-dimensional code monitor, an image auditing monitor, a flower screen monitor, a solidification monitor, a mute monitor and a voice auditing monitor;
the monitor abstraction layer is connected with the asynchronous frame distribution queue, the asynchronous frame distribution queue is respectively connected with the video frame monitor abstraction layer, the composite monitor abstraction layer and the audio frame monitor abstraction layer, the video frame abstraction layer is connected with the image distribution queue, the image distribution queue is respectively connected with the black screen monitor, the fuzzy monitor, the static field monitor, the brightness monitor, the two-dimensional code monitor and the image auditing monitor, the composite monitor abstraction layer is connected with the composite frame distribution queue, the composite frame distribution queue is connected with the flower screen monitor and the coagulation monitor, the audio frame monitor is connected with the audio distribution queue, the audio distribution queue is connected with the silent monitor and the voice auditing monitor, the black screen monitor and the audio frame monitor, The fuzzy monitor, the static field monitor, the brightness monitor, the two-dimensional code monitor, the image auditing monitor, the flower screen monitor, the coagulation monitor, the mute monitor and the voice auditing monitor are respectively connected with the resource recycling layer.
The data presentation module 30 is further configured to:
acquiring the live broadcast data stream and displaying a live broadcast state corresponding to the live broadcast data stream;
displaying live comments contained in the live data stream;
and summarizing abnormal data contained in the live broadcast data stream to obtain an analysis report, and displaying the analysis report.
Based on the above Web server architecture but not limited to the above architecture, a first embodiment of the live broadcast quality inspection method of the present invention is provided. The live broadcast quality inspection method comprises the following steps:
and S101, acquiring a data stream through a quality inspection and auditing module and monitoring in real time.
And S102, if the quality inspection and verification module monitors that the live broadcast data stream contains abnormal data, outputting the abnormal data through an instant communication module, and displaying the abnormal data through a data display module.
The execution main body of the method can be an intelligent mobile terminal with data processing capacity and a fixed Web server, and the method is deployed on the Web server in a software package form and provides a front-end page. In this embodiment, based on JavaCV as a technical scheme foundation, high-level customization of service logic is performed by using a bottom library such as FFmpeg and OpenCV, so that real-time quality inspection of live multi-modal content can be realized. Compared with the C and Python Wrapper modes, the scheme of the embodiment has better affinity for enterprise server development, and can fully utilize the capability of Java ecological service to expand the boundary of the audio and video scene. As shown in fig. 3, fig. 3 is an architecture diagram related to a live broadcast quality inspection system, which includes a quality inspection auditing module, an instant communication module, and a data display module; the quality inspection auditing module is used for acquiring and auditing the live data stream, the instant communication module is used for outputting auditing information acquired by the quality inspection auditing module to operators and a main broadcast in an information form, and the data display module is used for displaying abnormal conditions of live broadcast contained in the live data stream and confirmed by the quality inspection auditing module and the like so as to display a front-end picture. Because the live broadcast quality inspection system has high expansibility, if new algorithms such as green screen and picture shaking are developed in the later period, the new algorithms can be conveniently integrated into the framework, and other codes are hardly changed (for example, various notification and statistical codes do not need to be changed).
In order to solve the problem of insufficient flexibility of live broadcast quality inspection, after a live broadcast task is created by a main broadcast, a quality inspection auditing module allocates monitoring resources for the live broadcast task and creates a monitor, and the quality inspection auditing module integrates high-order algorithms such as real-time ASR, image auditing, text analysis and the like; the quality inspection auditing module can analyze live audio signals according to the real-time ASR, analyze live video images according to an image auditing algorithm and analyze live comment characters according to a text analysis algorithm. After the anchor is played, the live broadcast data stream of the anchor is acquired and monitored through the quality inspection and verification module so as to analyze and verify the live broadcast data stream and ensure the smooth operation of the live broadcast.
If the quality inspection auditing module monitors that the live broadcast data stream contains abnormal data, if the corresponding live broadcast of the live broadcast data stream is silent, pornographic, solidified, two-dimension codes, black fields, violent and the like, the abnormal data is output through the instant communication module and displayed through the data display module, so that the main broadcast and the operators can know the problems existing in the current live broadcast, the faults are solved in time, and the normal broadcast of the live broadcast is guaranteed.
Wherein, step S101 is followed by:
step A1, displaying the live broadcast state corresponding to the live broadcast data stream through the data display module.
Step A2, displaying the live comments contained in the live data stream through the data display module.
Step A3, summarizing abnormal data contained in the live data stream through the data display module to obtain an analysis report, and displaying the analysis report.
After the anchor broadcast is played, firstly, the quality inspection auditing module confirms the live broadcast state of the live broadcast data stream, for example: the live broadcast is delayed, the live broadcast is spacious, the live broadcast is punctual to broadcast etc., then show the live broadcast state that the live broadcast dataflow corresponds through the data display module. And extracting the barrage comments and the live comments of the live data stream according to a text analysis algorithm through a quality inspection auditing module, and then displaying the barrage comments and the live comments contained in the live data stream through a data display module. Obtaining abnormal data through a quality inspection auditing module according to the audited live data stream, then displaying the abnormal data contained in the live data stream through a data display module, and/or summarizing the abnormal data contained in the live data stream through the data display module to obtain an analysis report, and displaying the analysis report.
Specifically, as shown in fig. 4, in the page provided by the data presentation module, the quality inspector or operator interface can observe on the top side which live broadcasts are in question currently. And the currently added live broadcast task and the state of the currently added live broadcast task are conveniently checked from the live broadcast list on the right side. After clicking the mouse to select, displaying the live broadcast detailed information, the live broadcast content and the finished abnormal scene in the middle of the page. On both sides are the in-progress exception scene and the viewer's current comments, respectively. In addition to basic real-time abnormal scene data display, the data display module also provides time period abnormal data summarization. The data of all live broadcast abnormal conditions (such as data of all live broadcast mechanisms, scene occurrence frequency proportion, index trend and the like) in a period of time are measured, so that quality testing personnel can collect, return to test and summarize the data, and data support is made for improving decisions.
Wherein, step S101 is followed by:
and step B1, if it is monitored by the quality inspection auditing module that the live broadcast corresponding to the live broadcast data stream is not broadcast on time, judging whether the live broadcast corresponding to the live broadcast data stream is broadcast in absence or not by the quality inspection auditing module.
And step B2, if it is monitored by the quality inspection and verification module that the live broadcast corresponding to the live broadcast data stream is not in the open broadcast, determining that the live broadcast corresponding to the live broadcast data stream is in the delayed broadcast, and outputting the information of the live broadcast in the delayed broadcast corresponding to the live broadcast data stream through the instant communication module and the data display module.
In this embodiment, at first, whether the corresponding live broadcast of live data stream is punctual to open according to the start time of live data stream, wherein, the live broadcast is preset the start time and has set up when establishing the live broadcast task and finish, if the start time of live data stream satisfies preset the start time, then confirm the punctual broadcast of live broadcast that the live data stream corresponds to open, and pass through instant messaging module, the output of data display module with the information that the punctual broadcast of live broadcast that the live data stream corresponds, in order to supply the operator to know the live broadcast condition. And if the start time of the live data stream does not meet the theoretical start time, determining that the live data corresponding to the live data stream is not broadcast on time.
And further, whether live broadcast corresponding to the live broadcast data stream is delayed broadcast or not is judged, if the live broadcast data stream does not start within the preset time, the live broadcast corresponding to the live broadcast data stream is determined to be delayed broadcast, and the delayed broadcast information is output through the instant messaging module and the data display module so that operators can know the live broadcast condition and pay attention to the corresponding live broadcast anchor. If the live broadcast is not started within the preset delay broadcast tolerance time after the live broadcast is delayed, determining that the live broadcast corresponding to the live broadcast data stream is open broadcast, and outputting open broadcast information through an instant communication module and a data display module to enable operators to know the live broadcast condition and manage and control the corresponding live broadcast anchor; the preset delay broadcast tolerance time can be set according to actual conditions.
Wherein, step S101 is followed by:
and step C1, judging whether the live broadcast data stream is continuous or not through the quality inspection and audit module.
And step C2, if the live data stream is continuous, continuously monitoring the live data stream through the quality inspection auditing module.
And step C3, if the quality inspection auditing module monitors that the live data stream stops, stopping monitoring the live data stream by the quality inspection auditing module.
In this embodiment, after monitoring the live data stream, the quality inspection and audit module determines whether the live data stream is continuous, and if the live data stream is discontinuous, that is, the push stream is interrupted, a cut-off abnormality is detected, and information of the push stream interruption of the live data stream is output through the instant messaging module and the data display module, so that an operator can adjust the live data stream according to the push stream interruption information. And if the push flow is recovered by live broadcasting after adjustment, the data display module moves the flow break abnormity from the current abnormity to the historical abnormity, and the flow break recovery is indicated. And if the live data stream is continuous, continuously monitoring the live data stream. And if the live broadcast data stream is monitored to stop, namely the stream pushing is stopped, the live broadcast data stream is stopped to be monitored, the monitoring is finished, and the live broadcast corresponding to the live broadcast data stream is finished.
Wherein, step S102 includes:
and D1, converting the abnormal data into communication information through the instant communication module.
And D2, outputting the communication information to the instant communication software through the instant communication module.
In this embodiment, the instant messaging module is designed in an adapter mode, when a quality inspection task is started, a preset abnormal scene observer is injected into the quality inspection auditing module, and when a user configured frame number threshold is reached, abnormal data is converted into a message of an IM tool to be sent out, as shown in fig. 5 and 6, so that quality inspectors can be helped to master the live broadcast quality condition at any time and any place; the abnormal scene observer is a preset user authorized to receive the live broadcast condition, and the user configured frame number threshold may be set according to the actual condition, which is not specifically limited in this embodiment.
Wherein, step S102 further includes:
step E1, if it is monitored by the quality inspection and auditing module that the live data stream contains abnormal data, determining the duration of the abnormal data by the quality inspection and auditing module.
And E2, if the duration time meets a preset threshold value, outputting the abnormal data through the instant messaging module, and displaying the abnormal data through the data display module.
If the quality inspection and auditing module monitors that the live broadcast data stream contains abnormal data, such as a black field, the quality inspection and auditing module judges the duration of the abnormal data (the frame number of the black field); if the duration time meets a preset threshold value, outputting abnormal data through the instant communication module, and displaying abnormal data through the data display module to enable operators to know the live broadcast condition so as to manage and control the corresponding live broadcast anchor; the preset threshold value can be set according to actual conditions.
It should be noted that, for various types of anomalies, the monitor corresponding to the live broadcast data stream not only reports the monitored anomaly data and the anomaly type of the anomaly data, but also includes the time, duration, instant live broadcast screenshot, and the like of the anomaly occurrence. In order to increase the flexibility and robustness of abnormal scene detection, the judgment indexes and the duration threshold of each scene can be flexibly configured. Meanwhile, the monitor corresponding to the live data stream maintains a complete life cycle from the beginning to the end of the anomaly, and the life cycle is as shown in fig. 7 by taking a setting example that the black field lasts for 4 frames. Wherein 0 represents normal; 1 indicates that black field anomaly is monitored; 2, abnormal monitoring of a black field is performed, but the abnormal monitoring does not reach a user configured frame number (or duration) threshold, wherein the user configured frame number (or duration) threshold can be set according to the user requirements so as to enhance the robustness of scene detection; 3 means that the user configured frame number threshold is just reached; 4 represents that the user configuration frame number is exceeded; and 5 indicates that the scene has ended and is now normal. All relevant information of the abnormal data, such as time screenshots and the like, can be informed to the observer component, so that the relevant information of the abnormal data can be output to operators and/or a host through the observer component; before the live broadcast quality inspection task is initialized, all abnormal flows can be conveniently tracked and processed by injecting a custom observer component. It can be understood that, when the duration of the abnormal picture/audio playing meets the threshold of the user configured frame number (or duration), it represents that the current live broadcast segment is abnormal; but through carrying out self-defining observer subassembly external subassembly to observer subassembly, can realize the good expansibility in the system function to this affinity that improves live quality inspection system.
The invention discloses a live broadcast quality inspection system, a live broadcast quality inspection method, a Web server and a storage medium. The method comprises the following steps: acquiring a data stream through a quality inspection and auditing module and monitoring in real time; if the quality inspection and verification module monitors that the live broadcast data stream contains abnormal data, the abnormal data is output through the instant communication module, and the abnormal data is displayed through the data display module. The invention adopts the JavaCV technology to provide a high-order API which is more convenient and easy to use on the basis of keeping the programming flexibility, and the technology of the embodiment is convenient to expand and fuse with other systems based on the powerful cloud ecology of Java. The invention aims to perform full-flow multi-mode real-time accompanying monitoring on live broadcast activities. The invention takes the live broadcast stream address as input and simulates the playing process of the client. By means of technical schemes of real-time frame extraction, voice service, image algorithm, deep learning and the like, whether quality problems exist in live broadcast is analyzed, an abnormal real-time feedback mechanism is provided by combining an instant communication tool, real-time communication in the live broadcast quality inspection process is achieved, and quality inspection efficiency is improved.
Based on the first embodiment of the live broadcast quality inspection method, the second embodiment of the live broadcast quality inspection system is provided. In the present embodiment, step S101: the data stream is acquired through the quality inspection auditing module and the real-time monitoring comprises the following steps:
step S1011, obtaining the live broadcast data stream through the monitor abstraction layer, and writing the live broadcast data stream into the asynchronous frame distribution queue after decapsulation.
Step S1012, decoding and distributing the live data stream to the corresponding monitor through the asynchronous frame distribution queue.
Step S1013, monitoring the live data stream by a monitor corresponding to the live data stream.
Referring to fig. 8, the quality inspection and auditing module includes a monitor abstraction layer, an asynchronous frame distribution queue, a video frame monitor abstraction layer, a composite monitor abstraction layer, an audio frame monitor abstraction layer, an image distribution queue, a composite frame distribution queue, an audio frame distribution queue, and a resource recycling layer. Due to the limitation of space of the figure, the monitor realized by the system specifically comprises a black screen monitor, a fuzzy monitor, a static field monitor, a brightness monitor, a two-dimensional code monitor, an image auditing monitor, a flower screen monitor, a coagulation monitor, a mute monitor and a voice auditing monitor.
In this embodiment, the live broadcast quality inspection system decapsulates and decodes frames of a live broadcast data stream through a monitor abstraction layer to obtain decoded information, and writes audio frame information and video frame information included in the live broadcast data stream into an asynchronous frame distribution queue, so as to distribute each modal data stream to a respective corresponding monitor through the asynchronous frame distribution queue.
The method comprises the steps that visual frame information in a live broadcast data stream is distributed to a visual frame monitor abstract layer through an asynchronous frame distribution queue, the visual frame monitor abstract layer consumes the live broadcast data stream, the visual frame monitor abstract layer is further decoded into an image, and visual frame pixel information is distributed to each visual frame monitor through an image distribution queue, wherein each visual frame monitor comprises a black screen monitor, a fuzzy monitor, a static field monitor, a brightness monitor, a two-dimensional code monitor and an image auditing monitor; the image distribution queue is an asynchronous frame distribution queue. Different computer vision and image auditing algorithms are built in each video frame monitor to judge whether the live broadcast data stream is detected to be abnormal or not, the detection result is captured by an observer component, and the corresponding detection result is reported to quality inspection personnel and/or operation personnel and/or a main broadcast. For example, the video monitor studies whether the live data stream has the conditions of static field, blur, brightness, black screen and content violation through computer vision and image auditing algorithms, and reports the detection result to quality testing personnel and/or operators and/or anchor broadcasters.
It should be noted that, in order to improve the system performance of the live broadcast quality inspection system, all the distributions are only copied for reference, and memory data is not copied, and a concurrence component is used for managing a potential image data concurrent access scene, so that the memory utilization rate is greatly improved.
Audio frame information in the live broadcast data stream is distributed to an audio frame monitor abstract layer through an asynchronous frame distribution queue, the audio frame monitor abstract layer consumes the live broadcast data stream, further decodes the audio frequency and distributes audio frame sampling information to each audio frame monitor through the audio frame distribution queue, wherein each audio frame monitor comprises a mute monitor and a voice auditing monitor; the sound frame distribution queue is an asynchronous frame distribution queue. Computer hearing, real-time ASR and voice auditing algorithm are built in each audio frame monitor to judge whether the live broadcast data stream is detected to be abnormal or not, the detection result is captured by an observer component, and the corresponding detection result is reported to quality testing personnel and/or operators and/or anchor broadcasters. For example, the voice frame monitor studies whether the live data stream has the conditions of silence and the speaker-speaking violation through a real-time ASR algorithm, and reports the detection result to quality testing personnel and/or operators and/or the speaker.
It should be noted that, in order to improve the system performance of the live broadcast quality inspection system, all the distributions are only copied for reference, and do not copy the memory data, and the concurrency component is used to manage the potential audio data concurrency access scene, so as to greatly improve the memory utilization rate.
The method comprises the steps that audio and video are comprehensively analyzed, scenes under certain decoding information are processed by a composite monitor abstraction layer, the composite monitor abstraction layer consumes live broadcast data streams, the live broadcast data streams are further decoded, and video frame pixel information, video frame decoding information (code rate, coding mode, frame rate, key frame and the like), audio frame sampling information and audio frame decoding information (sampling rate, channel number and the like) of the live broadcast data streams acquired by the composite monitor abstraction layer are distributed to each composite monitor through a composite frame distribution queue, wherein the composite monitors comprise a screen-patterned monitor and a solidification monitor; the composite frame distribution queue is an asynchronous frame distribution queue. The composite monitor studies and judges whether the live broadcast data stream is detected abnormally through algorithms such as computer vision, computer hearing, natural language processing, audio and video coding and decoding, image auditing, voice auditing and the like to obtain a detection result, and reports the corresponding detection result to quality testing personnel and/or operators and/or a main broadcast. For example, the composite monitor judges whether the live broadcast data stream is solidified or not through computer vision and computer hearing algorithms; whether the screen splash condition occurs in the live broadcast data stream is judged by monitoring whether code rate dip, GoP elongation and the like exist in the live broadcast, and the detection result is reported to quality testing personnel and/or operating personnel and/or anchor. As long as the monitor that can not simply determine only by means of video frames and only audio frames needs to be distributed to the composite monitor for access processing.
It should be noted that, in order to improve the system performance of the live broadcast quality inspection system, all the distributions are only copied for reference, and do not copy the memory data, and the concurrency component is used to manage the potential data concurrency access scene, so that the memory utilization rate is greatly improved, and the system is allowed to carry more quality inspection tasks in unit time.
Further, in this embodiment, the quality inspection auditing module screens out invalid irrigation comments in the live broadcast data stream by means of strategies such as a natural language processing algorithm, business rules and the like, performs statistical analysis on quality-related feedback, and automatically classifies the screened audience bullet screens into five categories of violation, irrigation, pleasure, complaint and business opportunity as classification results; and finally, giving a label to the user corresponding to the bullet screen according to the classification result so as to determine a disposal strategy of the user corresponding to the bullet screen according to the label. For example, if the label of the user is illegal, the user is restricted from speaking; if the label of the user is a business opportunity, pushing corresponding business information for the user; therefore, the live broadcast watching experience of the user is improved.
Compared with solutions based on other audio and video processing tools such as WebRTC and the like, the FFmpeg-based live broadcast quality inspection system can support more live broadcast and audio and video protocol types, has good encapsulation performance and is non-invasive to a stream pushing end and a client. Compared with the FFmpeg command type, the API and other Wrapper, the JavaCV technology adopted by the embodiment provides a high-level API which is more convenient and easy to use on the basis of keeping the programming flexibility. Meanwhile, the embodiment is very compatible with most of the current Java-based ecology of the servers, and is convenient to expand and fuse with other systems based on the powerful cloud ecology of Java, for example: monitors of the live broadcast quality inspection system, such as a black screen monitor, a fuzzy detector, a flower screen monitor and a mute monitor, can be conveniently and quickly accessed into the monitoring system.
In addition, an embodiment of the present invention further provides a Web server, where the Web server includes a memory, a processor, and a live broadcast quality inspection program that is stored in the memory and can be run on the processor, and the live broadcast quality inspection program, when executed by the processor, implements the steps of the live broadcast quality inspection system described above.
Since the live broadcast quality inspection program is executed by the processor, all technical solutions of all the embodiments are adopted, so that at least all the beneficial effects brought by all the technical solutions of all the embodiments are achieved, and detailed description is omitted.
In addition, an embodiment of the present invention further provides a computer-readable storage medium, where a live broadcast quality inspection program is stored on the computer-readable storage medium, and when executed by a processor, the live broadcast quality inspection program implements the steps of the live broadcast quality inspection system described above.
Since the live broadcast quality inspection program is executed by the processor, all technical solutions of all the embodiments are adopted, so that at least all the beneficial effects brought by all the technical solutions of all the embodiments are achieved, and detailed description is omitted.
Compared with the prior art, the live broadcast quality inspection system, the live broadcast quality inspection method, the Web server and the storage medium provided by the invention pass through the quality inspection and verification module, the instant communication module and the data display module; the quality inspection and verification module is respectively connected with the instant communication module and the data display module, the instant communication module is connected with external instant communication software, and the data display module is connected with a Web front end; the quality inspection auditing module is used for acquiring and monitoring a live broadcast data stream; the instant communication module is used for acquiring abnormal data contained in the live broadcast data stream monitored by the quality inspection and verification module and outputting the abnormal data; and the data display module is used for acquiring abnormal data contained in the live data stream monitored by the quality inspection and auditing module and displaying the abnormal data. The invention aims to solve the problem of insufficient flexibility of live broadcast quality inspection, realize real-time communication in the live broadcast quality inspection process and improve the quality inspection efficiency.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or method. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or method that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk, usb disk, removable hard disk) and includes several instructions for enabling a Web server (which may be a computer, a server, a controlled terminal, or a network device) to execute the method of each embodiment of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A live broadcast quality inspection system is developed and obtained based on JavaCV, FFmpeg and OpenCV, and comprises the following components:
the quality inspection and verification module, the instant communication module and the data display module; the quality inspection and verification module is respectively connected with the instant communication module and the data display module, the instant communication module is connected with external instant communication software, and the data display module is connected with a Web front end;
the quality inspection auditing module is used for acquiring and monitoring a live broadcast data stream;
the instant communication module is used for acquiring abnormal data contained in the live broadcast data stream monitored by the quality inspection and verification module and outputting the abnormal data;
and the data display module is used for acquiring abnormal data contained in the live data stream monitored by the quality inspection and auditing module and displaying the abnormal data.
2. The live broadcast quality inspection system according to claim 1, wherein the quality inspection auditing module comprises a monitor abstraction layer, an asynchronous frame distribution queue, a video frame monitor abstraction layer, a composite monitor abstraction layer, a sound frame monitor abstraction layer, an image distribution queue, a composite frame distribution queue, an audio distribution queue, a resource recovery layer, a black screen monitor, a fuzzy monitor, a static field monitor, a brightness monitor, a two-dimensional code monitor, an image auditing monitor, a flower screen monitor, a coagulation monitor, a silence monitor, and a voice auditing monitor;
the monitor abstract layer is connected with the asynchronous frame distribution queue, the asynchronous frame distribution queue is respectively connected with the video frame monitor abstract layer, the composite monitor abstract layer and the audio frame monitor abstract layer, the video frame abstract layer is connected with the image distribution queue, the image distribution queue is respectively connected with the black screen monitor, the blur monitor, the still field monitor, the brightness monitor, the two-dimensional code monitor and the image review monitor, the composite monitor abstract layer is connected with the composite frame distribution queue, the composite frame distribution queue is connected with the flower screen monitor and the coagulation monitor, the audio frame monitor is connected with the audio distribution queue, the audio distribution queue is connected with the silence monitor and the voice review monitor, the black screen monitor and the audio review monitor, The fuzzy monitor, the static field monitor, the brightness monitor, the two-dimensional code monitor, the image auditing monitor, the flower screen monitor, the coagulation monitor, the mute monitor and the voice auditing monitor are respectively connected with the resource recycling layer.
3. The live quality inspection system of claim 1, wherein the data presentation module is further configured to:
acquiring the live broadcast data stream and displaying a live broadcast state corresponding to the live broadcast data stream;
displaying live comments contained in the live data stream;
and summarizing abnormal data contained in the live broadcast data stream to obtain an analysis report, and displaying the analysis report.
4. A live broadcast quality inspection method is characterized by comprising the following steps:
acquiring a data stream through a quality inspection and auditing module and monitoring in real time;
if the quality inspection and verification module monitors that the live broadcast data stream contains abnormal data, the abnormal data is output through the instant communication module, and the abnormal data is displayed through the data display module.
5. The live broadcast quality inspection method according to claim 4, wherein the quality inspection auditing module includes a monitor abstraction layer and an asynchronous frame distribution queue, and the step of acquiring data streams and monitoring in real time through the quality inspection auditing module includes:
acquiring the live broadcast data stream through the monitor abstract layer, and writing the live broadcast data stream into the asynchronous frame distribution queue after decapsulation;
decoding the live broadcast data stream through the asynchronous frame distribution queue and then distributing the decoded live broadcast data stream to a corresponding monitor;
and monitoring the live data stream through a monitor corresponding to the live data stream.
6. The live broadcast quality inspection method according to claim 4, wherein the step of obtaining the data stream through the quality inspection auditing module and monitoring in real time comprises the following steps:
converting the abnormal data into communication information through the instant communication module;
and outputting the communication information to external instant communication software through the instant communication module.
7. The live broadcast quality inspection method according to claim 4, wherein the step of obtaining the data stream through a quality inspection auditing module and monitoring in real time is followed by:
if the quality inspection auditing module monitors that the live broadcast corresponding to the live broadcast data stream is not broadcast on time, the quality inspection auditing module judges whether the live broadcast corresponding to the live broadcast data stream is broadcast in absence;
and if the quality inspection auditing module monitors that the live broadcast corresponding to the live broadcast data stream is not in an open broadcast state, determining that the live broadcast corresponding to the live broadcast data stream is in a delayed broadcast state, and outputting the information of the live broadcast in the delayed broadcast state corresponding to the live broadcast data stream through the instant communication module and the data display module.
8. The live broadcast quality inspection method according to claim 4, wherein the step of obtaining the data stream through the quality inspection auditing module and monitoring in real time comprises the following steps:
judging whether the live broadcast data stream is continuous or not through the quality inspection and auditing module;
if the live data stream is continuous, the quality inspection auditing module is used for continuously monitoring the live data stream;
and if the quality inspection auditing module monitors that the live data stream stops, stopping monitoring the live data stream through the quality inspection auditing module.
9. A Web server, characterized in that the Web server comprises a memory, a processor and a live quality inspection method stored on the memory and operable on the processor, and that the program of the live quality inspection, when executed by the processor, implements the steps of the live quality inspection method according to any one of claims 4 to 8.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a program for live quality inspection, which program, when executed by a processor, carries out the steps of the live quality inspection method of any one of claims 4-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210682572.7A CN115134617B (en) | 2022-06-16 | Live broadcast quality inspection system, live broadcast quality inspection method, web server and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210682572.7A CN115134617B (en) | 2022-06-16 | Live broadcast quality inspection system, live broadcast quality inspection method, web server and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115134617A true CN115134617A (en) | 2022-09-30 |
| CN115134617B CN115134617B (en) | 2024-04-26 |
Family
ID=
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116128453A (en) * | 2023-02-18 | 2023-05-16 | 广州市点易资讯科技有限公司 | Online course inspection method, system, equipment and medium |
| CN116128453B (en) * | 2023-02-18 | 2024-05-03 | 广州市点易资讯科技有限公司 | Online course inspection method, system, equipment and medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200204279A1 (en) * | 2018-12-20 | 2020-06-25 | Viamedia, Inc. | Integrating digital advertising with cable tv network and broadcast advertising |
| CN111464819A (en) * | 2020-03-30 | 2020-07-28 | 腾讯音乐娱乐科技（深圳）有限公司 | Live image detection method, device, equipment and storage medium |
| CN112055230A (en) * | 2020-09-03 | 2020-12-08 | 北京中润互联信息技术有限公司 | Live broadcast monitoring method and device, computer equipment and readable storage medium |
| CN114257828A (en) * | 2021-12-20 | 2022-03-29 | 上海哔哩哔哩科技有限公司 | Live broadcast audit content processing method and system |
| CN114466210A (en) * | 2022-01-27 | 2022-05-10 | 广州方硅信息技术有限公司 | Live broadcast quality detection processing method and device, equipment, medium and product thereof |
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200204279A1 (en) * | 2018-12-20 | 2020-06-25 | Viamedia, Inc. | Integrating digital advertising with cable tv network and broadcast advertising |
| CN111464819A (en) * | 2020-03-30 | 2020-07-28 | 腾讯音乐娱乐科技（深圳）有限公司 | Live image detection method, device, equipment and storage medium |
| CN112055230A (en) * | 2020-09-03 | 2020-12-08 | 北京中润互联信息技术有限公司 | Live broadcast monitoring method and device, computer equipment and readable storage medium |
| CN114257828A (en) * | 2021-12-20 | 2022-03-29 | 上海哔哩哔哩科技有限公司 | Live broadcast audit content processing method and system |
| CN114466210A (en) * | 2022-01-27 | 2022-05-10 | 广州方硅信息技术有限公司 | Live broadcast quality detection processing method and device, equipment, medium and product thereof |
Non-Patent Citations (2)
| Title |
|---|
| 侯忠辉;: "视频监控画面质量检测研究", 计算机安全, no. 05, 15 May 2013 (2013-05-15), pages 55 - 59 * |
| 魏崇毓;张宏琳;: "基于Live555的手机实时直播系统设计与实现", 计算机工程与设计, no. 05, 16 May 2016 (2016-05-16), pages 1157 - 1160 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116128453A (en) * | 2023-02-18 | 2023-05-16 | 广州市点易资讯科技有限公司 | Online course inspection method, system, equipment and medium |
| CN116128453B (en) * | 2023-02-18 | 2024-05-03 | 广州市点易资讯科技有限公司 | Online course inspection method, system, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10721282B2 (en) | Media acceleration for virtual computing services | |
| US20040156613A1 (en) | Method and system for computer software application execution | |
| CN101505365B (en) | Real-time video monitoring system implementing method based on network television set-top box | |
| GB2463329A (en) | A set top box emulator for Internet Protocol Television (IPTV) | |
| CN104333808A (en) | General player applied to Android and Linux smart televisiosn | |
| CN102281473A (en) | Information processing method for television terminal and television terminal | |
| US20070006268A1 (en) | Digital media player exposing operational state data | |
| CN109391843B (en) | Online video speed doubling playing method, device, medium and intelligent terminal | |
| CN1921610B (en) | Client-based video stream interactive processing method and processing system | |
| CN114095784A (en) | H.265 format video stream transcoding playing method, system, device and medium | |
| CN101719990A (en) | Method for censoring stream media live programs | |
| US20030033612A1 (en) | Software appliance method and system | |
| CN115134617A (en) | Live broadcast quality inspection system, method, Web server and storage medium | |
| CN106296283B (en) | Advertisement playing method and system based on mobile application | |
| CN115134617B (en) | Live broadcast quality inspection system, live broadcast quality inspection method, web server and storage medium | |
| US20140141875A1 (en) | Temporary modification for extending functionality of computer games and software applications. | |
| CN112040304A (en) | Hard disk video recorder system supporting wireless screen projection | |
| CN113542765B (en) | Media data jump continuous playing method and display device | |
| US9026983B2 (en) | Flexible and scalable software system architecture for implementing multimedia applications | |
| CN115955557A (en) | Kadun analysis method and device thereof | |
| CN115103215A (en) | Live broadcast quality inspection method, system, Web server and storage medium | |
| US7882510B2 (en) | Demultiplexer application programming interface | |
| CN112423100A (en) | WEB audio and video self-defined rendering method based on HTML5 | |
| CN111147928A (en) | Video processing method, server, terminal and system | |
| CN105812922A (en) | Multimedia file data processing method, system, player and client |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |