CN109151464A - IPTV set top box failure real-time detection method based on high amount of traffic processing - Google Patents
IPTV set top box failure real-time detection method based on high amount of traffic processing Download PDFInfo
- Publication number
- CN109151464A CN109151464A CN201811351493.8A CN201811351493A CN109151464A CN 109151464 A CN109151464 A CN 109151464A CN 201811351493 A CN201811351493 A CN 201811351493A CN 109151464 A CN109151464 A CN 109151464A
- Authority
- CN
- China
- Prior art keywords
- top box
- real
- set top
- daily record
- record data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
- H04N17/004—Diagnosis, testing or measuring for television systems or their details for digital television systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/4425—Monitoring of client processing errors or hardware failure
Abstract
The invention discloses a kind of IPTV set top box failure real-time detection method based on high amount of traffic processing, set-top box generates and reports log；F5 load balancing daily record data；Nginx forwards daily record data；Collection container acquires daily record data；Kafka queue is written in daily record data；Daily record data is analyzed in real time in Spark cluster；Elasticsearch cluster is written in failure analysis result；Front end shows failure detection result.The real-time processing to IPTV set top box mass data may be implemented in the present invention, and can do resilient expansion to entire frame performance according to portfolio；Processing platform is finally analyzed in real time by this log big data efficiently, elastic, realizes the real-time detection to IPTV set top box failure.
Description
Technical field
The present invention relates to a kind of failure real-time detection method, especially a kind of IPTV set top box based on high amount of traffic processing
Failure real-time detection method.
Background technique
As the technologies such as internet, DTV develop rapidly, the use of IPTV set top box is very universal, as current
The core component of home entertaining, the failure of IPTV set top box can life to family and amusement cause to seriously affect.And it is current
After the fault detection of IPTV set top box relies primarily on the telephone complaint report barrier of user, operator sends engineer to visit detection, and
The log for manually transferring the set-top box is analyzed.The troubleshooting scheme of this passive type, not only time lag is serious, and
Maintenance cost is high, and user's perception is poor.
The daily record data comprising error code can be all generated inside the IPTV set top box of mainstream at present, to these daily record datas
Analyze the fault detection of achievable set-top box.But there is no methods at present can the reality that generates of online set-top boxes all to the whole network
When massive logs data be acquired and analyze.
In the prior art, such as application No. is a kind of Chinese patent of CN201510604795.1 " failure messages of set top box
Processing method, device and set-top box ", for another example application No. is a kind of Chinese patent of CN200910188997.7 " acquisition machine tops
The method and its acquisition module of box fault message " discloses the methods of some existing set-top box failures detections and processing, but
The presence of these technologies can not detect the online set-top box of magnanimity simultaneously, and not have the problems such as real-time capacity.
Summary of the invention
Technical problem to be solved by the invention is to provide a kind of IPTV set top box failure based on high amount of traffic processing is real
When detection method, to realize to the real-time detection of IPTV set top box failure.
In order to solve the above technical problems, the technical scheme adopted by the invention is that:
A kind of IPTV set top box failure real-time detection method based on high amount of traffic processing, it is characterised in that comprise the steps of:
Step 1: set-top box generates and reports log；
Step 2: F5 load balancing daily record data；
Step 3: Nginx forwards daily record data；
Step 4: collection container acquires daily record data；
Step 5: Kafka queue is written in daily record data；
Step 6: daily record data is analyzed in real time in Spark cluster；
Step 7: Elasticsearch cluster is written in failure analysis result；
Step 8: front end shows failure detection result.
Further, the step 2 be specially in IPTV set top box system program the daily record data generated in real time is passed through
Http agreement will be transmitted to F5 hardware load balancer.
Further, the step 3 is specially that more Nginx forwarding services are respectively configured under F5 hardware load balancer
Device, F5 is by massive logs data load balance that synchronization set-top box reports to the more Nginx servers configured below.
Further, the log number that the step 4 will acquire specifically by the upstream module of Nginx server
According to the log collection program for being forwarded to multiple containers, by the docker container ip of each log collection program with polling mode
Location and port are configured in the upstream module of Nginx server, so that log be made to be forwarded to containerization in a manner of http
Log collection program in.
Further, the step 5 is specially that the daily record data that will acquire of log collection program of containerization pushes to
In Kafka message queue, for consumption later.
Further, the step 6 is specially to pass through the real-time stream process mould of Spark Streaming on Spark cluster
Type, the daily record data read in Kafka message queue carry out real-time analysis processing, statistically analyze what current IPTV set top box occurred
Error code.
Further, the step 7 is specially that the real-time stream process model of Spark Streaming will analyze result deposit
In search engine Elasticsearch, show in real time for front end.
Further, the step 8 is specially and is realized in the form of chart etc. to IPTV set top box according to real-time results data
The real-time detection of failure shows.
Compared with prior art, the present invention having the following advantages that and effect: the present invention may be implemented to IPTV set top box sea
The real-time processing of data is measured, and resilient expansion can be done to entire frame performance according to portfolio；Finally by this efficient, bullet
Property log big data analyze processing platform in real time, realize the real-time detection to IPTV set top box failure.
Detailed description of the invention
Fig. 1 is a kind of process of IPTV set top box failure real-time detection method based on high amount of traffic processing of the invention
Figure.
Fig. 2 is a kind of structural frames of IPTV set top box failure real-time detection method based on high amount of traffic processing of the invention
Frame figure.
Specific embodiment
Below by embodiment, the present invention is described in further detail, following embodiment be explanation of the invention and
The invention is not limited to following embodiments.
As illustrated in fig. 1 and 2, a kind of IPTV set top box failure real-time detection side based on high amount of traffic processing of the invention
Method comprising the steps of:
Step 1: set-top box generates and reports log；The preset program of IPTV set top box built-in system generates daily record data in real time.
Step 2: F5 load balancing daily record data；System program leads to the daily record data generated in real time in IPTV set top box
Crossing http agreement will be transmitted to F5 hardware load balancer.
Step 3: Nginx forwards daily record data；More Nginx forwarding services are respectively configured under F5 hardware load balancer
Device, F5 is by massive logs data load balance that synchronization set-top box reports to the more Nginx servers configured below.
Step 4: collection container acquires daily record data；The log that will acquire by the upstream module of Nginx server
Data forwarding to multiple containers log collection program, with polling mode by the docker container ip of each log collection program
Address and port are configured in the upstream module of Nginx server, so that log be made to be forwarded to container in a manner of http
In the log collection program of change.
Step 5: Kafka queue is written in daily record data；The daily record data push that the log collection program of containerization will acquire
Into Kafka message queue, for consumption later.
Step 6: daily record data is analyzed in real time in Spark cluster；Pass through Spark on Spark cluster
The real-time stream process model of Streaming, the daily record data read in Kafka message queue carry out real-time analysis processing, statistical
Analyse the error code that current IPTV set top box occurs.
Spark is read by the daily record data of parsing in Kafka message queue, and is patrolled according to related set-top box failure business
It collects and writes model, analyzed in real time by Spark Streaming, as counted 503 mistakes in 1 minute log period of set-top box
Whether code is compared a period frequency of occurrence and is increased more than 20%, so that it is determined that set-top box access resource pool is with the presence or absence of abnormal.
Step 7: Elasticsearch cluster is written in failure analysis result；The real-time stream process mould of Spark Streaming
Type will be analyzed in result deposit search engine Elasticsearch, show in real time for front end.
Step 8: front end shows failure detection result.The reality to IPTV set top box failure is realized according to real-time results data
When detection show.Foreground shows result data in program real-time calling Elasticsearch, by forms such as charts to set-top box
Fault condition is showed in real time.
It is collected, transmits, handles and shows by the log generated in real time to the online IPTV set top box of the whole network, to reach
To the real-time detection to set-top box failure.
Massive logs data are once shunted by F5 load balancer, until multiple Nginx forwarding servers, each
Daily record data can be forwarded to multiple collection containers again and carry out second diffluence by Nginx forwarding server, so that mass data
Acquisition problems be addressed.Furthermore Nginx and collection container can carry out the dilatation and contracting of elasticity according to the size of portfolio
Hold.
Daily record data is analyzed in real time using high amount of traffic processing technique, and by message queue cluster to magnanimity day
Will data are transmitted, and the quick search of result is realized eventually by search engine technique.
The real-time processing to IPTV set top box mass data may be implemented in the present invention, and can be according to portfolio to entire frame
Frame performance does resilient expansion；Processing platform is finally analyzed in real time by this log big data efficiently, elastic, is realized to IPTV
The real-time detection of set-top box failure.
Above content is only illustrations made for the present invention described in this specification.Technology belonging to the present invention
The technical staff in field can do various modifications or supplement or is substituted in a similar manner to described specific embodiment, only
It should belong to guarantor of the invention without departing from the content or beyond the scope defined by this claim of description of the invention
Protect range.
Claims (8)
1. a kind of IPTV set top box failure real-time detection method based on high amount of traffic processing, it is characterised in that include following step
It is rapid:
Step 1: set-top box generates and reports log；
Step 2: F5 load balancing daily record data；
Step 3: Nginx forwards daily record data；
Step 4: collection container acquires daily record data；
Step 5: Kafka queue is written in daily record data；
Step 6: daily record data is analyzed in real time in Spark cluster；
Step 7: Elasticsearch cluster is written in failure analysis result；
Step 8: front end shows failure detection result.
2. the IPTV set top box failure real-time detection method described in accordance with the claim 1 based on high amount of traffic processing, feature
Be: the step 2 be specially in IPTV set top box system program the daily record data generated in real time is transmitted by http agreement
To F5 hardware load balancer.
3. the IPTV set top box failure real-time detection method described in accordance with the claim 1 based on high amount of traffic processing, feature
Be: the step 3 is specially that more Nginx forwarding servers are respectively configured under F5 hardware load balancer, and F5 will be the same as the moment
The massive logs data load balance that reports of set-top box is carved to the more Nginx servers configured below.
4. the IPTV set top box failure real-time detection method described in accordance with the claim 1 based on high amount of traffic processing, feature
Be: the step 4 is forwarded to multiple specifically by the daily record data that the upstream module of Nginx server will acquire
The log collection program of containerization is configured the address container ip docker of each log collection program and port with polling mode
Into the upstream module of Nginx server, so that log be made to be forwarded to the log collection journey of containerization in a manner of http
In sequence.
5. the IPTV set top box failure real-time detection method described in accordance with the claim 1 based on high amount of traffic processing, feature
Be: the step 5 is specially that the daily record data that will acquire of the log collection program of containerization pushes to Kafka message queue
In, for consumption later.
6. the IPTV set top box failure real-time detection method described in accordance with the claim 1 based on high amount of traffic processing, feature
Be: the step 6 is specially to read Kafka by the real-time stream process model of Spark Streaming on Spark cluster
Daily record data in message queue carries out real-time analysis processing, statisticallys analyze the error code that current IPTV set top box occurs.
7. the IPTV set top box failure real-time detection method described in accordance with the claim 1 based on high amount of traffic processing, feature
Be: the step 7 is specially that the real-time stream process model of Spark Streaming will analyze result deposit search engine
In Elasticsearch, show in real time for front end.
8. the IPTV set top box failure real-time detection method described in accordance with the claim 1 based on high amount of traffic processing, feature
Be: the step 8 is specially the real-time inspection realized in the form of chart etc. according to real-time results data to IPTV set top box failure
Survey shows.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811351493.8A CN109151464A (en) | 2018-11-14 | 2018-11-14 | IPTV set top box failure real-time detection method based on high amount of traffic processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811351493.8A CN109151464A (en) | 2018-11-14 | 2018-11-14 | IPTV set top box failure real-time detection method based on high amount of traffic processing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109151464A true CN109151464A (en) | 2019-01-04 |
Family
ID=64805866
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811351493.8A Pending CN109151464A (en) | 2018-11-14 | 2018-11-14 | IPTV set top box failure real-time detection method based on high amount of traffic processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109151464A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111143314A (en) * | 2019-12-26 | 2020-05-12 | 厦门服云信息科技有限公司 | Log analysis method and system based on high-speed streaming processing technology |
| CN111431955A (en) * | 2019-01-10 | 2020-07-17 | 中科星图股份有限公司 | Streaming data processing system and method |
| CN112073755A (en) * | 2020-09-04 | 2020-12-11 | 中邮科通信技术股份有限公司 | Method for realizing intelligent switching of authentication modes based on IPTV service log |
| TWI732410B (en) * | 2020-01-02 | 2021-07-01 | 中華電信股份有限公司 | System and method for automatic reporting of detecting a set-top box abnormally |
| CN114996335A (en) * | 2022-08-03 | 2022-09-02 | 海看网络科技(山东)股份有限公司 | IPTV log real-time clustering analysis method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120219298A1 (en) * | 2005-09-23 | 2012-08-30 | Samsung Electronics Co., Ltd. | Remote control system and method having reduced vulnerability to noise |
| CN103051956A (en) * | 2012-12-24 | 2013-04-17 | 乐视致新电子科技（天津）有限公司 | Set-top box for realizing log report and fault diagnosis and method thereof |
| CN104036025A (en) * | 2014-06-27 | 2014-09-10 | 蓝盾信息安全技术有限公司 | Distribution-base mass log collection system |
| CN107948324A (en) * | 2017-12-29 | 2018-04-20 | 广东欧珀移动通信有限公司 | Ask Transmission system, method, apparatus and storage medium |
| CN108038207A (en) * | 2017-12-15 | 2018-05-15 | 暴风集团股份有限公司 | A kind of daily record data processing system, method and server |
| CN108365971A (en) * | 2018-01-10 | 2018-08-03 | 深圳市金立通信设备有限公司 | Daily record analytic method, equipment and computer-readable medium |
-
2018
- 2018-11-14 CN CN201811351493.8A patent/CN109151464A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120219298A1 (en) * | 2005-09-23 | 2012-08-30 | Samsung Electronics Co., Ltd. | Remote control system and method having reduced vulnerability to noise |
| CN103051956A (en) * | 2012-12-24 | 2013-04-17 | 乐视致新电子科技（天津）有限公司 | Set-top box for realizing log report and fault diagnosis and method thereof |
| CN104036025A (en) * | 2014-06-27 | 2014-09-10 | 蓝盾信息安全技术有限公司 | Distribution-base mass log collection system |
| CN108038207A (en) * | 2017-12-15 | 2018-05-15 | 暴风集团股份有限公司 | A kind of daily record data processing system, method and server |
| CN107948324A (en) * | 2017-12-29 | 2018-04-20 | 广东欧珀移动通信有限公司 | Ask Transmission system, method, apparatus and storage medium |
| CN108365971A (en) * | 2018-01-10 | 2018-08-03 | 深圳市金立通信设备有限公司 | Daily record analytic method, equipment and computer-readable medium |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111431955A (en) * | 2019-01-10 | 2020-07-17 | 中科星图股份有限公司 | Streaming data processing system and method |
| CN111431955B (en) * | 2019-01-10 | 2023-03-24 | 中科星图股份有限公司 | Streaming data processing system and method |
| CN111143314A (en) * | 2019-12-26 | 2020-05-12 | 厦门服云信息科技有限公司 | Log analysis method and system based on high-speed streaming processing technology |
| TWI732410B (en) * | 2020-01-02 | 2021-07-01 | 中華電信股份有限公司 | System and method for automatic reporting of detecting a set-top box abnormally |
| CN112073755A (en) * | 2020-09-04 | 2020-12-11 | 中邮科通信技术股份有限公司 | Method for realizing intelligent switching of authentication modes based on IPTV service log |
| CN112073755B (en) * | 2020-09-04 | 2022-09-06 | 中邮科通信技术股份有限公司 | Method for realizing intelligent switching of authentication modes based on IPTV service log |
| CN114996335A (en) * | 2022-08-03 | 2022-09-02 | 海看网络科技(山东)股份有限公司 | IPTV log real-time clustering analysis method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109151464A (en) | IPTV set top box failure real-time detection method based on high amount of traffic processing | |
| Yu et al. | {dShark}: A general, easy to program and scalable framework for analyzing in-network packet traces | |
| CN107508722B (en) | Service monitoring method and device | |
| CN104378264B (en) | A kind of virtual machine process flux monitoring method based on sFlow | |
| CN105574205A (en) | Dynamic log analyzing system for distributed computing environment | |
| CN102664789B (en) | The processing method of a kind of large-scale data and system | |
| CN109361532A (en) | The high-availability system and method and computer readable storage medium of network data analysis | |
| CN105337753B (en) | A kind of internet real quality monitoring method and device | |
| CN103067218B (en) | A kind of express network packet content analytical equipment | |
| CN106815112A (en) | A kind of mass data monitoring system and method based on deep-packet detection | |
| CN105577411B (en) | Cloud service monitoring method and device based on service origin | |
| CN103763149B (en) | Real-time statistical method for network user number | |
| CN109714648A (en) | A kind of video flow load balancing method and device | |
| CN106972985A (en) | Accelerate the method and DPI equipment of the processing of DPI device datas and forwarding | |
| CN110351238A (en) | Industry control honey pot system | |
| CN110087064A (en) | A kind of detection method of monitor terminal, system and a kind of device and storage medium | |
| CN113259467B (en) | Webpage asset fingerprint tag identification and discovery method based on big data | |
| CN104917628A (en) | Automatic diagnosis method of Ethernet router/switch packet loss fault | |
| CN105357071A (en) | Identification method and identification system for network complex traffic | |
| CN103036746B (en) | Passive measurement method and passive measurement system of web page responding time based on network intermediate point | |
| CN110572698B (en) | Method, device, equipment and medium for counting inspection and monitoring playing records | |
| CN101924769A (en) | Payload characteristic identification based method for identifying Sohu dragon oath game service | |
| CN103369403B (en) | Set Top Box program request packet analysis system and analysis method | |
| CN109361546B (en) | Program early warning method and device based on video network | |
| CN107480189A (en) | A kind of various dimensions real-time analyzer and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 210029 No. 268, Hanzhoung Road, Nanjing, Jiangsu Applicant after: CLP Hongxin Information Technology Co., Ltd Address before: 210029 No. 268, Hanzhoung Road, Nanjing, Jiangsu Applicant before: Jiangsu Hongxin System Integration Co., Ltd. |
|
| CB02 | Change of applicant information | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190104 |
|
| RJ01 | Rejection of invention patent application after publication |