CN103488774A

CN103488774A - Processing method for big data log analysis

Info

Publication number: CN103488774A
Application number: CN201310453556.1A
Authority: CN
Inventors: 曾敬鸿; 邓翔宇
Original assignee: GUIZHOU BROADCASTING&TV INFORMATION NETWORK Co Ltd
Current assignee: GUIZHOU BROADCASTING&TV INFORMATION NETWORK Co Ltd
Priority date: 2013-09-29
Filing date: 2013-09-29
Publication date: 2014-01-01

Abstract

The invention discloses a processing method for big data log analysis. The method includes the steps that first, a user inputs a user identifier, a service identifier, a query time point and an extension time period through a foreground page; second, a background system acquires the parameter information input through the foreground page and decomposes tasks with fixed time intervals as unit; third, each decomposed task is compared with completed tasks stored in a database, and the completed tasks are filtered out; fourth, the uncompleted tasks after comparison are issued to all client-sides through a server-side; fifth, an inquiry program is started after all the client-sides receive the tasks, and inquiry result data are reported to a cache; six, the inquiry result data are sent to the foreground page to be read by the user after all the inquiry tasks are completed. The processing method for big data log analysis solves the problem that because a traditional data analysis method adopted for log analysis of data of a broadcasting and TV service system has no universality, complex algorithms can exist, and an analysis result acquired on each time can not provide available data for analysis at later stages.

Description

A kind of disposal route of analyzing for large data logging

Technical field

The invention belongs to broadcasting and TV service monitoring system technology, relate in particular to a kind of disposal route of analyzing for large data logging.

Background technology

Sharply increase along with broadcasting and TV business system user quantity, system occurs that wrong and location mistake becomes increasingly complex, conventional location mistake is generally to go to location by the profiling error daily record, it is generally the time according to customer complaint generation problem, then find journal file from the journal file of magnanimity, and then the abnormal log that finds the report user to go wrong from journal file, but because the user is many, concurrent large, system all generates a large amount of daily records all the time, location and the reparation of the problem of giving have brought very large difficulty, thereby traditional data analysis generally reaches the purpose of accelerating efficiency by specific algorithm optimization for specific data structure, do not possess versatility and the algorithm of more complicated may occur, and each analysis result can not provide available data for post analysis, the while treatment effeciency is low etc.

Summary of the invention

The technical problem to be solved in the present invention: a kind of disposal route of analyzing for large data logging is provided, thereby adopt traditional data analysis generally for specific data structure, by specific algorithm optimization, to reach the purpose of accelerating efficiency to solve broadcasting and TV business system data log analysis, do not possess versatility and the algorithm of more complicated may occur, and each analysis result can not provide available data and the problem such as query processing efficiency is low for post analysis.

Technical solution of the present invention:

A kind of disposal route of analyzing for large data logging, it comprises the steps:

Step 1, user input user's sign, business sign, query time point and expansion time section by front page layout;

Step 2, background system obtain front page layout input parameter information, are spaced apart the Partition of Unity task with regular time;

The completed task of storing in step 3, each task after decomposing and database is compared, and filters out completed task;

Step 4, the uncompleted task afterwards of comparing are issued to each client by server end;

Step 5, each client, after task, start polling routine, and the Query Result data upload is arrived to buffer memory, and announcement server end corresponding task completes;

After step 6, all query tasks complete, server end obtains the Query Result data from buffer memory and delivers to database, and the Query Result data are delivered to front page layout and read for the user.

Step 2 is described with the regular time interval, and its time is spaced apart 20 minutes.

In step 5, be reported to the Query Result data of buffer memory after all query tasks complete, the unified server end of delivering to is stored.

Beneficial effect of the present invention:

By the present invention, take the time as unit carries out the task cutting, be issued to each client Agent after comparing by the multithread analyzing daily record, accelerate log analysis efficiency; By the task cutting, shorten the time cycle of each task, improve success ratio, the data that analysis completes can be to analyze next time provides data available, the task cutting time interval is made as 20 minutes, after considering that the client finds fault, the reaction time of reporting fault is generally about 20 minutes, therefore the time interval is made as 20 minutes, to improve search efficiency, reduce client's stand-by period, the result data in buffering is the unified server of delivering to after all query tasks complete, be in order to reduce the burden of server, accelerate the treatment effeciency in processing procedure; Thereby the invention solves broadcasting and TV business system data log analysis adopts traditional data analysis generally for specific data structure, by specific algorithm optimization, to reach the purpose of accelerating efficiency, do not possess versatility and the algorithm of more complicated may occur, and each analysis result can not provide data available and the problem such as query processing efficiency is low for post analysis.

embodiment:

Step 1, user input user's sign, business sign, query time point and expansion time section by front page layout; Be generally that to take the number of minutes of time point and expansion time section be that 3 integers are optional value: 20 minutes, 40 minutes, 1 hour, number of seconds was 0, and this is mainly after considering that the rating end subscriber is found fault, the reaction time of reporting fault.

Step 2, background system obtain front page layout input parameter information, are spaced apart the Partition of Unity task with regular time; Its time is spaced apart 20 minutes optimums, also can arbitrarily set interval.

The completed task of storing in each task after step 3, background system will decompose and database is compared, and filters out completed task; Having completed of task does not need to repeat inquiry, and the result that directly output is stored in server gets final product.

Step 5, each client, after task, start polling routine, and the Query Result data are reported to buffer memory, kept in, and this task of announcement server complete;

Step 6, by heartbeat, judge, after all query tasks complete, server end obtains the Query Result data from buffer memory and delivers to database, and the Query Result data are delivered to front page layout and read for the user.

Claims

1. a disposal route of analyzing for large data logging, it comprises the steps:

2. according to claim 1 a kind of for the disposal route that data logging is analyzed greatly, it is characterized in that: step 2 is described with the regular time interval, and its time is spaced apart 20 minutes.

3. according to claim 1 a kind of for the disposal route that data logging is analyzed greatly, it is characterized in that: in step 5, be reported to the Query Result data of buffer memory after all query tasks complete, the unified server end of delivering to is stored.