CN106682096A

CN106682096A - Method and device for log data management

Info

Publication number: CN106682096A
Application number: CN201611090842.6A
Authority: CN
Inventors: 李铮; 侯怀锋; 高飞龙; 郑超平; 张超; 郑扬; 张娟娜
Original assignee: Beijing Qihoo Technology Co Ltd; Qizhi Software Beijing Co Ltd
Current assignee: Beijing Qihoo Technology Co Ltd; Qizhi Software Beijing Co Ltd
Priority date: 2016-12-01
Filing date: 2016-12-01
Publication date: 2017-05-17

Abstract

The invention discloses a method and device for log data management. The method comprises the steps of performing analytical processing on at least one assigned service, and obtaining formative log data of services and storing the log data in a corresponding data mart in a data warehouse; configuring permissions for each data mart in the data warehouse according to user grouping; providing a front end page, and determining user grouping information of a user according to user login information sent by the front end page; and when data mart checking instructions sent by the front end page are received, presenting data mart information which the user has checking authority to the user through the front end page according to the user's user grouping information. According to the technical scheme, unified management and storage as well as authority distribution of the service log data are achieved, the control over the log data is achieved standardly and orderly, and the method and the device are particularly suitable for enterprises having multiple service lines, large log data size and complex organizational structures.

Description

A kind of management method and device of daily record data

Technical field

The present invention relates to field of computer technology, and in particular to a kind of management method and device of daily record data.

Background technology

Especially Internet firm of enterprise, company, routine work often produces substantial amounts of daily record data, to these data It is a task that is very great and needing Standardized Design to be managed.For example, some data are that hash needs to lose Abandon, some data need to carry out putting in storage etc. by the generation time.

And differ because these data are numerous and diverse, there is a difficult problem in unified management in different formats；And daily record data may be used also The trade secret informations such as portfolio are can relate to, how to guarantee that daily record data had both facilitated user to check, but it is also to need not betray the pot to the roses The problem to be solved.

The content of the invention

In view of the above problems, it is proposed that the present invention so as to provide one kind overcome the problems referred to above or at least in part solve on State the management method and device of the daily record data of problem.

According to one aspect of the present invention, there is provided a kind of management method of daily record data, including：

Dissection process is carried out to the daily record data of at least one specified services, the daily record data of formatting of business is obtained simultaneously It is saved in data warehouse in corresponding Data Mart；

Each Data Mart in for data warehouse configures authority by user grouping；

Front end page is provided, according to the user login information that front end page sends, the user grouping information of the user is determined； When the Data Mart for receiving front end page transmission checks instruction, according to the user grouping information of the user, by front end page Show that it has the Data Mart information for checking authority towards the user.

Alternatively, the method also includes：

The corresponding relation of configuration service and domain name, the daily record data for receiving is classified by domain name；

The daily record data at least one specified services carries out dissection process to be included：According to the corresponding domain of the business Name, to the daily record data under the domain name classification dissection process is carried out.

Alternatively, the Data Mart includes at least one tables of data；

It is described for data warehouse in each Data Mart by user grouping configuration authority also include：For each number of Data Mart Authority is checked according to table by user grouping configuration, and check authority by user grouping configuration for each row of tables of data；

The user grouping information according to the user, shows that it has the number for checking authority by front end page to the user Include according to fairground information：According to the user grouping information of the user, show that it has a tables of data for checking authority to the user, and/ Or show that it has the row in the tables of data for checking authority to the user.

Alternatively, the method also includes：

Receive and preserve the report generation task that user is submitted to by front end page；

According to the daily record data in the Data Mart specified in the report generation task, generate form and preserve to form Data base；Wherein, the authority for arranging in the tables of data in the corresponding Data Mart of the authority of each row of the form of generation is identical； The authority of the form is determined according to the authority of each row of form.

Alternatively, the Data Mart checks that instruction and/or report generation task include：

The standard queries sentence or the query argument of user input of user input；

The method also includes：Corresponding Data Mart is inquired about with the standard queries sentence of user input, obtaining the user has Daily record data in the Data Mart information for checking authority and/or the Data Mart specified；Or, according to the inquiry of user input Parameter generates standard queries sentence, and with the standard queries sentence for generating corresponding Data Mart is inquired about, and obtains the user and checks Daily record data in the Data Mart information of authority and/or the Data Mart specified.

Alternatively, the method also includes：

Sentence filtering rule is set, and the standard queries sentence of standard queries sentence or generation to user input was carried out Filter.

Alternatively, the method also includes：

The form that receive user is uploaded by front end page, and/or deposited by the form that front end page is submitted to according to user Storage path, the form specified is obtained from the path, is saved in report database.

Alternatively, the method also includes：

According to the user grouping information of user, to it the Data Mart and/or form of editable authority is shown；

The authority edit instruction that receive user is submitted to, is compiled accordingly to the authority of the Data Mart and/or form Volume.

Alternatively, the method also includes：

According to the user grouping information of the user, show that it has the report name for checking authority to the user by front end page Claim；

The report generation task includes：The form that user specifies；

The daily record data in the Data Mart specified in the report generation task, generate form and preserve to Report database includes：The configuration information of the form specified using user, according to the data specified in the report generation task Daily record data in fairground generates form.

Alternatively, the method also includes：

When the form for receiving user input checks instruction, user in the form is shown to the user by front end page Data with the row for checking authority.

Alternatively, the method also includes：

New Report in report database is checked into that number of times is initialized as zero；When the form for receiving user input is checked During instruction, corresponding report is checked into that number of times increases by one；

Cleaning cycle is set for each form, when the time point of cleaning cycle is reached, according to judging that checking for the form is secondary Whether number is less than cleaning threshold, if being less than, deletes corresponding report generation task.

Alternatively, the method also includes：

According to the user grouping information of user, show that it has the statistics graph model of access right to it, so that the front end The page generates corresponding cartogram according to user's statistics graph model selected and the data for having shown that form.

Alternatively, any one during the statistics graph model is included as follows：

According to the model of the data genaration cartogram for having shown that form cached in the front end page；

Again the data that user in the form has the row for checking authority are obtained from data source, generates the model of cartogram；

Data source to having shown that form enters the statistics graph model of edlin.

Alternatively, the method also includes：

Any one operation performed to user is recorded, and corresponding with user profile monitoring data is saved in by the operating time In storehouse.

Alternatively, the method also includes：

The warning operation for operating report to the police strategy and correspondence execution is set by action type；

When strategy is reported to the police in the operation for matching the action type of any one operation that user performs, corresponding warning is performed Operation.

According to another aspect of the present invention, there is provided a kind of managing device of daily record data, including：

Daily record data resolution unit, is suitable to carry out dissection process to the daily record data of at least one specified services, obtains industry The daily record data of the formatting of business is simultaneously saved in data warehouse in corresponding Data Mart；

Control of authority unit, each Data Mart being suitable in data warehouse configures authority by user grouping；

Display unit, is adapted to provide for front end page, according to the user login information that front end page sends, determines the user's User grouping information；When the Data Mart for receiving front end page transmission checks instruction, believed according to the user grouping of the user Breath, shows that it has the Data Mart information for checking authority by front end page to the user.

Alternatively, the daily record data resolution unit, is suitable to the corresponding relation of configuration service and domain name, by the day for receiving Will data are classified by domain name；According to the corresponding domain name of the business, the daily record data under the domain name classification is carried out at parsing Reason.

Alternatively, the Data Mart includes at least one tables of data；

The control of authority unit, each tables of data for being suitable for Data Mart checks authority by user grouping configuration, and Each row for tables of data check authority by user grouping configuration；

The display unit, is suitable to the user grouping information according to the user, shows that it has to the user and checks authority Tables of data, and/or show that it has the row in the tables of data for checking authority to the user.

Alternatively, the device also includes：

Form unit, is suitable to receive and preserve the report generation task that user is submitted to by front end page, according to the report Daily record data in the Data Mart specified in table generation task, generates form and preserves to report database；

The control of authority unit, the authority for being suitable to each row according to form determines the authority of the form, wherein, generation The authority arranged in tables of data in the corresponding Data Mart of the authority of each row of form is identical.

The device also includes：Query unit, is suitable to inquire about corresponding Data Mart with the standard queries sentence of user input, Obtain the daily record data during the user has the Data Mart information for checking authority and/or the Data Mart specified；Or, according to The query argument of family input generates standard queries sentence, and with the standard queries sentence for generating corresponding Data Mart is inquired about, and obtains The user have the Data Mart information for checking authority and/or the Data Mart specified in daily record data.

Alternatively, the query unit, is further adapted for arranging sentence filtering rule, standard queries sentence to user input or The standard queries sentence of generation is filtered.

Alternatively, the form unit, is further adapted for the form that receive user is uploaded by front end page, and/or according to The form store path that family is submitted to by front end page, the form specified is obtained from the path, is saved in report database.

Alternatively, the display unit, is further adapted for the user grouping information according to user, and to it editable authority is shown Data Mart and/or form；

The control of authority unit, is further adapted for the authority edit instruction of receive user submission, to the Data Mart and/or The authority of form is edited accordingly.

Alternatively, the display unit, is further adapted for the user grouping information according to the user, by front end page to the use Family shows that it has the report name for checking authority；

The report generation task includes：The form that user specifies；

The form unit, is adapted in use to the configuration information of the form that user specifies, according in the report generation task Daily record data in the Data Mart specified generates form.

Alternatively, the display unit, is further adapted for when the form for receiving user input checks instruction, by front end page Show data of the user with the row for checking authority in the form towards the user.

Alternatively, the device also includes：

Form checks recording unit, is suitable to for New Report in report database to check that number of times is initialized as zero；When connecing When the form for receiving user input checks instruction, corresponding report is checked into that number of times increases by one；

Task clears up unit, is suitable for each form and arranges cleaning cycle, when the time point of cleaning cycle is reached, according to sentencing Break the form checks that number of times, whether less than cleaning threshold, if being less than, deletes corresponding report generation task.

Alternatively, the display unit, is further adapted for the user grouping information according to user, shows that it has access right to it Statistics graph model, so as to the statistics graph model selected according to user of the front end page and have shown that the data of form, generate Corresponding cartogram.

Alternatively, the device also includes：

Monitoring unit, any one operation for being suitable to perform user is recorded, corresponding with user profile by the operating time In being saved in monitoring data storehouse.

Alternatively, the device also includes：Alarm unit, is suitable to arrange operation warning strategy and correspondence execution by action type Warning operation；When strategy is reported to the police in the operation for matching the action type of any one operation that user performs, perform corresponding Report to the police and operate.

From the foregoing, technical scheme, by the daily record data dissection process of business into the daily record number for formatting During the corresponding Data Mart of data warehouse is saved according to after, be each Data Mart configures authority by user grouping, when receiving When the Data Mart that front end page sends checks instruction, according to the user grouping information that the log-on message of the user determines, pass through Front end page shows that it has the Data Mart information for checking authority to the user.The technical scheme realizes the daily record number to business According to unified management warehouse-in and authority distribution, specification, it is orderly realize the management and control to daily record data, be particularly suitable for business The enterprise that line is more, daily record data amount is big, organizational structure is complicated.

Described above is only the general introduction of technical solution of the present invention, in order to better understand the technological means of the present invention, And can be practiced according to the content of description, and in order to allow the above and other objects of the present invention, feature and advantage can Become apparent, below especially exemplified by the specific embodiment of the present invention.

Description of the drawings

By the detailed description for reading hereafter preferred implementation, various other advantages and benefit is common for this area Technical staff will be clear from understanding.Accompanying drawing is only used for illustrating the purpose of preferred implementation, and is not considered as to the present invention Restriction.And in whole accompanying drawing, it is denoted by the same reference numerals identical part.In the accompanying drawings：

Fig. 1 shows the workflow schematic diagram of daily record data platform；

Fig. 2 shows a kind of schematic flow sheet of the management method of daily record data according to an embodiment of the invention；

Fig. 3 shows a kind of structural representation of the managing device of daily record data according to an embodiment of the invention.

Specific embodiment

The exemplary embodiment of the disclosure is more fully described below with reference to accompanying drawings.Although showing the disclosure in accompanying drawing Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure and should not be by embodiments set forth here Limited.On the contrary, there is provided these embodiments are able to be best understood from the disclosure, and can be by the scope of the present disclosure Complete conveys to those skilled in the art.

Technical scheme can apply to daily record data platform, and Fig. 1 shows the workflow of daily record data platform Journey schematic diagram.As shown in figure 1, the daily record data platform is by daily record data, and by ETL, (Extract-Transform-Load takes out Take-change-load) process, result data is saved in into data warehouse, while supporting the statistical computation to daily record data, generate Form is saved in report database；And front end page is provided, and it is easy to the access of user, understand the operation conditions of task and incite somebody to action Report data carries out visual presentation etc..Whole platform also provides rights management function and task scheduling function, to daily record data Process, statistics and displaying regulated and controled.The present invention will focus on introduce it is interior in terms of rights management therein and visual presentation Hold.

Fig. 2 shows a kind of schematic flow sheet of the management method of daily record data according to an embodiment of the invention, such as Shown in Fig. 2, the method includes：

Step S210, to the daily record data of at least one specified services dissection process is carried out, and obtains the formatting of business Daily record data is simultaneously saved in data warehouse in corresponding Data Mart.

Data warehouse (Data Warehouse) is particularly suitable for for scattered data carrying out standardization storage, and data bins Can be according to the particular content of business in storehouse, the fact that arrange corresponding table and dimension table, it is generally the case that daily record data can be protected In the fact that be stored to corresponding table.When business is more, multiple Data Marts (Data Mart) can be set in data warehouse, each Data Mart can correspond to a class business.

Step S220, is that each Data Mart in data warehouse configures authority by user grouping.

For example, what the sector member of product A only possessed the Data Mart to product A checks authority, without possessing to producing The Data Mart of product B checks authority.

Step S230, there is provided front end page, according to the user login information that front end page sends, determines the user of the user Grouping information.

Step S240, when the Data Mart for receiving front end page transmission checks instruction, according to the user of the user point Group information, shows that it has the Data Mart information for checking authority by front end page to the user.

In this step, if user A submits to Data Mart to check instruction by front end page, it can be viewed and is possessed Check the Data Mart information of authority, such as Data Mart title.

It can be seen that, the method shown in Fig. 2 is saved in the daily record data dissection process of business into after the daily record data for formatting It is that each Data Mart configures authority by user grouping in the corresponding Data Mart of data warehouse, sends out when front end page is received When the Data Mart for sending checks instruction, according to the log-on message of the user determine user grouping information, by front end page to The user shows that it has the Data Mart information for checking authority.The technical scheme realizes the unified pipe of the daily record data to business Reason warehouse-in and authority distribution, specification, it is orderly realize the management and control to daily record data, be particularly suitable for that service line is more, daily record Data volume is big, the enterprise that organizational structure is complicated.

Following embodiments on the basis of rights management, will introduce the ETL process of daily record data.

In one embodiment of the invention, the method shown in Fig. 2 also includes：The corresponding relation of configuration service and domain name, The daily record data for receiving is classified by domain name；Dissection process bag is carried out to the daily record data of at least one specified services Include：According to the corresponding domain name of the business, dissection process is carried out to the daily record data under the domain name classification.

In actual applications, by taking Internet firm as an example, it is often opened for the different domain name of each traffic assignments managed The development of the business of carrying out, thus produce daily record data both also be from different domain names, by the daily record data for receiving by domain Name carries out classification fast can realize exactly that daily record data is classified.Due to the extensive application of distributed type assemblies, each industry Business is often carried out on different distributed type assemblies, and for example, the functional module of same business may be in some of the whole nation Dispose on individual different clusters, similarly, there are multiple service lines of the task may be run on same cluster, then daily record data is by next The modes such as source carry out classification and are nothing like classifying fast and convenient by domain name.There can also be multiple subdomains under each domain name in this example Name, the subservice in correspondence business, such domain name compares various, can be saved as metadata with the corresponding relation of business, It is managed as data dictionary and uses.It is of course also possible to distribute not by user grouping for checking, change etc. for metadata With authority, such as manager can possess modification authority, and domestic consumer only possesses authority for checking partial content etc..

In one embodiment of the invention, said method also includes：A sample daily record to being input into is carried out at parsing Reason, the analysis result of output formatting；The analysis result of receive user confirms after instruction that record parsing sample daily record is adopted Daily record resolution rules；Carrying out dissection process to the daily record data of at least one specified services includes：According to the daily record solution for being recorded Analysis rule, the Source log data of the business specified to user carry out dissection process.

For example for sample daily record：<Id=123><Sex=male><Age=18>, daily record resolution rules can be obtained For：ID is the string number started with " id="；Sex is a string of the characters started with " sex="；Age is with " age =" start numeral.So apply the daily record resolution rules, it is possible to right<Id=1233><Sex=male><Age=8>、< Id=12332><Sex=male><Age=28>Such Source log data carry out parsing.

Specifically, multiple log contents identification engines can be preset, for recognize different-format respectively log content simultaneously It is parsed into one or more fields；Sample daily record is sequentially inputted in multiple log content identification engines；By each log content The each field of output of identification engine carries out collecting the analysis result for obtaining formatting.

The system used due to each business or server may be different, and the form of the daily record data of generation is also various many Sample.Shown below is the example of several daily records：

1、http://mbs.hao.360.cn/index.phpId=1353332＆sex=male＆age=28＆....

2、{"id":"13532232332","sex":"male","age":"28"}

3、<Id=13532232332><Sex=male><Age=28>

4、id->13532232332；sex->male；age->28

It can be seen that the form of these four daily records is diverse.Default multiple daily records can be utilized in the above-described embodiments Recognize engine to recognize the log content of different-format.For example, JSON is a kind of more conventional data form, and its content is included Structure it is typically specific, for example log content is divided into (as above by multiple fields with symbols such as braces, colon, quotation marks The example 2 in face), and be directed to the daily record of JSON forms identification engine just can be carried out log content at parsing by these separators Reason, obtains one or more data of one or more fields.Specifically, during log content identification engine can include as follows One or more：IP address recognizes engine；Time-stamp Recognition engine；ID recognizes engine；Channel recognizes engine；JSON forms Content recognition engine.The form of IP address can be estimate (for example：Xxx.xxx.xxx.xxx), ID often by NAME, USER_ID or ID etc. can arrange corresponding key values (such as channel) as key values, channel by developer, timestamp Form is usually " YYYY-MM-DD HH:mm:SS”.Especially, IP address identification engine can with after IP address is identified, IP address is further parsed, IP address analysis result includes following one or more fields：Country, province, city, operator.Certainly, According to demand more detailed address can also be extended to, IP address analysis result can also include that word is expanded in area, street etc. Section, but these are relatively low for follow-up process use under normal conditions, can waste certain resource, can carry out according to demand Arrange.

As can be seen that daily record identification engine is also configurable, the key that such as different business is used channel is probably Different.Therefore in one embodiment of the invention, there is provided log content recognizes engine edition interface, by the interface Increase/delete/modification log content recognizes the instruction of engine, according to instruction perform corresponding log content identification module increase/ Delete/modification operation.

In practice the form of daily record is very various.Luckily these journal formats all include specification, discernible knot mostly Structure, can be recognized by different daily record identification engines.But for the daily record data that a business is produced, generally can't All of daily record identification engine is used, and the daily record data quantity that business is produced is various, if calling all of daily record every time Recognize engine to be identified, not only waste of resource, efficiency is also very beneath.Therefore in one embodiment of the invention, it is above-mentioned In method, each log content for having output result is only recognized the identification resolution rules summary record of engine.So next time is right When the daily record data of the business carries out dissection process, it is not necessary to which waste actually will not recognize engine using to log content .

However, not all daily record data is generated with the form of such specification, default daily record recognizes engine Most daily record data can be processed, but can also there is the situation that analysis result does not meet daily record original meaning.Therefore in the present invention One embodiment in, when there is the content of the equal None- identified of each log content identification engine in sample daily record, by making by oneself Justice identification interface exports the content；The recognition result being input into after self-defined identification interface artificial cognition and corresponding knowledge Other resolution rules；The identification resolution rules being input into after the artificial cognition are recorded as to parse the daily record parsing that sample daily record is adopted A part for rule.For example, conventional separator includes colon, branch, big round bracket etc., if the daily record data of a class business In contain the separator being of little use, it is necessary to user is identified result and corresponding identification solution in self-defined identification interface Analysis rule input.

In one embodiment of the invention, said method also includes：By front end page receives input to parsing knot The instruction that each field in fruit is operated, and perform corresponding operation；The finger that each field in analysis result is operated Make one or more in including as follows：Adjust the instruction of the clooating sequence of each field；The instruction of the title of modification specific field； Delete the instruction of specific field.

For example, the data of a field in analysis result are calculated without any help subsequent statistical, can delete this Field；Field name in analysis result is " USERNAME ", wishes to be changed to " user name " etc. during subsequent treatment, these Can be operated in analysis result editing interface.

Whole field can be operated in a upper embodiment, and in one embodiment of the invention, said method In, the identification resolution rules of a log content identification engine include：To recognizing and in one or more fields for parsing The parameter value of specific field is arranged and limits threshold value；For parameter value carries out discard processing beyond the daily record data for limiting threshold value.This Sample can just discard the unwanted data in part when daily record is parsed, and reduce the later stage and carry out the operation of daily record data discarding.

Describing in above-described embodiment carries out the operation of dissection process to daily record data, and parses the form of the business for obtaining The daily record data of change is saved in data warehouse in corresponding Data Mart, is somebody's turn to do in particular it is required that being determined according to field attribute Row in the corresponding data warehouse of field；Data to be stored are stored in data warehouse in corresponding row by field.

Address above, table and dimension table can be included the fact that in data warehouse, what these tables of data were typically built up in advance, For storing the data for receiving.Therefore, the data for receiving can by by attribute determine its it is corresponding be which in tables of data Row.And each table is included, and row are typically different, thus only need to determine in the present embodiment its it is corresponding be data warehouse In which row.

Specifically, determine that the row in the corresponding data warehouse of the field include according to field attribute：Read data warehouse Metadata, obtains the attribute of each row in data warehouse；According to the attribute of each row in field attribute and data warehouse, set up to be stored The mapping relations of the data of each field and Ge Lie in data.Metadata includes：Service attribute and/or data that data warehouse is respectively arranged Attribute, wherein, service attribute include it is following at least one：Business Name, business domain name, business description information；Data belong to Property include it is following at least one：Row name, data form, data type.For example, it is which that " user name " this field is corresponding Individual row, such corresponding relation can be stored in the metadata of data warehouse.

In one embodiment of the invention, said method also includes：By front end page receives input to specifying row The instruction that processed of data, instruction include it is following at least one：Data deciphering, Data Format Transform, data encoding Conversion；According to the instruction for receiving, the data to specifying row are processed accordingly.

For example, unserializing process is carried out to the data of the row so as to become readable；Time in daily record is colon form Data, be converted into timestamp, etc..

Data can be screened out in daily record identification engine in previous embodiment, in one embodiment of the invention Additionally provide the method screened out to the data in data warehouse：By the train value to specifying row of front end page receives input The instruction for limiting threshold value is set；According to the instruction, train value is carried out into delete processing beyond the whole piece data for limiting threshold value.For example, Whole piece data of the access times less than 3 are deleted.

Analogously, can also by front-end interface receives input he to data warehouse in the finger that operated of each row Order, and perform corresponding operation；The instruction that each row in data warehouse are operated include it is following in one or more：Adjust The instruction of the clooating sequence of whole each row；The instruction of the row name of row is specified in modification；Delete the instruction for specifying row.

It should be noted that directly the data in data warehouse are adjusted in this embodiment, and in aforementioned enforcement Although the clooating sequence of each field, the title of modification specific field can also be adjusted in example or specific field is deleted, these numbers According to data warehouse is not also stored in, in data buffer storage.

In one embodiment of the invention, in said method, Data Mart includes at least one tables of data；For data bins Each Data Mart in storehouse is also included by user grouping configuration authority：Each tables of data for Data Mart is looked into by user grouping configuration See authority, and authority is checked by user grouping configuration for each row of tables of data；According to the user grouping information of the user, pass through Front end page shows that it has the Data Mart information for checking authority to include to the user：According to the user grouping information of the user, Show that it has the tables of data for checking authority to the user, and/or show that it there are the row in the tables of data for checking authority to the user. For example, operation personnel can check the tables of data of business flowing water, and technical staff can not check.

Following embodiments will be introduced and include report generation, i.e. the daily record data to formatting carries out the concrete reality of statistical computation It is existing.

In one embodiment of the invention, said method also includes：Receive and preserve user to submit to by front end page Report generation task；According to the daily record data in the Data Mart specified in report generation task, generate form and preserve to Report database；Wherein, the authority for arranging in the tables of data in the corresponding Data Mart of the authority of each row of the form of generation It is identical；The authority of the form is determined according to the authority of each row of form.

Daily record data that in the present embodiment can be in data warehouse, carries out statistical computation, generates statistical report form etc.. By the form thus according to the data genaration in data warehouse, its authority has also followed the authority of former data.Finally, if one Form includes four row, then the user of the user grouping only having permission to either rank in this four row can just check the form, and And it can only view it and have that string for checking authority.For example, the user grouping that user A is located can check first three of form Row, then the form that he sees just only three is arranged rather than four row.

In one embodiment of the invention, in said method, according in the Data Mart specified in report generation task Daily record data, generating form and preserving to report database includes：According to report generation task, it is determined that complete report generation appointing The data processing model of business, and specified daily record data is read from corresponding Data Mart；Using the data processing mould for determining Type treats daily record data and carries out data processing, generates form and preserves to report database.

Existing form exploitation generally requires technical staff and writes code manually, and data are carried out with statistical computation, time-consuming to take Power, and when the form quantity of demand is more, it is impossible to quickly meet the demand of user.Present embodiments provide a kind of form Automatic generation method, user only needs to specify the daily record data generated needed for form, selects the data processing model specified, so that it may To obtain corresponding form, simply save time.Wherein, report generation task includes：Input address；According to report generation task middle finger Daily record data in fixed Data Mart, generating form and preserving to report database includes：Daily record number is read from input address According to generation form is simultaneously preserved to report database.Certainly, report database can also be specified by OPADD.

Except specifying input address, can be so that more many condition be limited in report generation task by user, such as using number According to the partial data of certain tables of data in fairground.Therefore in one embodiment of the invention, report generation task includes：User The standard queries sentence or the query argument of user input of input；The method also includes：With the standard queries sentence of user input Corresponding Data Mart is inquired about, the daily record data in the Data Mart that user specifies is obtained；Or, according to the inquiry of user input Parameter generates standard queries sentence, and with the standard queries sentence for generating corresponding Data Mart is inquired about, and obtains the data set specified Daily record data in city.For example for the technical staff of enterprise data center, standard of compiling query statement is handy, but For common business personnel may can't write such query statement.Therefore provide in the present embodiment user is defeated The querying condition for entering carries out the function of assembly.The content that for convenience identifying user is input into, user is in inquiry except specifying phase Outside the Data Mart answered, can also specify for the character repertoire of identified input content, it is to avoid the content of input is not correctly validated.

Due to the data volume in Data Mart it is very huge, if user do not arrange in querying condition it is corresponding limit, Such as time conditions, the data volume asked may cause the collapse of Data Mart.Therefore in one embodiment of the present of invention In, said method also includes：Sentence filtering rule, the standard queries language of standard queries sentence or generation to user input are set Sentence is filtered.The method of above-mentioned inquiry can be not only used for the inquiry of concrete daily record data, it is also possible to for Data Mart letter The inquiry of breath.

Generate form needed for data processing model can user specify, it is also possible to user from provide data processing Select in model, in one embodiment of the invention, according to report generation task, it is determined that completing the number of report generation task Include according to model is processed：The data processing model specified is selected from data processing model storehouse, wherein, in data processing model storehouse Comprising following at least one data processing model：Newly-increased statistical model；Enliven statistical model；Retain statistical model.These moulds Type can respectively be counted in the daily record of a certain day from daily record, user's (Adding User) quantity for not occurring in history, The retention situation that the quantity of any active ues, a certain day Add User in the specified time period.Certainly, specific data model can be with root It is added according to business demand or configures, here is not limited.User is typically no longer needed when using these data processing models Write code, it is only necessary to use by parameter information is provided, for example, which row to carry out statistical computation etc. using.That is, Report generation task also includes：The parameter information of the data processing model specified；Using the data processing model for determining to daily record Data carry out data processing to be included：Application parameter information carries out with postponing to specified data processing model, using with postponing Data processing model carries out data processing to daily record.

Certainly, the model of this " foolproof " can not meet all demands of user.Therefore in a reality of the present invention In applying example, the data processing model specified in said method is streaming models；Report generation task also includes：At least one The self-defining code snippet of section；Parameter information includes：One piece of logic partitioning is right in every section of code snippet and streaming models Should be related to.

For example, user is desirable with statistical computation of the Map-Reduce frameworks execution to daily record data.So in this enforcement In example, user only needs to develop respectively Map program code segments and Reduce program code segments, and (core for namely performing calculating is patrolled Volume), without writing complete code again.When form task is submitted to, it is only necessary to by Map program code segments and Reduce program generations Code section is respectively filled in front end page code input frame corresponding with Map sections and Reduce sections, it is possible to realize on backstage overall The assembly of code, such program is easily managed and changes, and decreases the possibility of written in code mistake.

In the case where above-mentioned model can not all meet user's request, user can also select self-defining model, therefore In said method, report generation task includes：The address of self-defining data processing model；It is determined that completing report generation task Data processing model includes：Self-defining data processing model is read from the address.Or, in the case of size of code is less Can be by uploading between front end page.Self-defining data processing model can also be saved in data processing model storehouse, on Biography person can also be its distribution authority.

Except the method that daily record data above in Data Mart generates form, user can also open other modes The report upload sent out carries out unified control of authority and management to report database.Therefore in one embodiment of the invention, Said method also includes：The form that receive user is uploaded by front end page, and/or submitted to by front end page according to user Form store path, the form specified is obtained from the path, is saved in report database.

In one embodiment of the invention, said method also includes：According to the user grouping information of user, to its displaying The Data Mart and/or form of editable authority；The authority edit instruction that receive user is submitted to, to Data Mart and/or form Authority edited accordingly.

In the present embodiment there is provided the management control method of authority, possess the manager of higher-rights (such as in data The manager of the heart) authority of data that can possess it editing authority enters edlin.For example, business director can control it The member of each group can only see the related form of the group.

In one embodiment of the invention, said method also includes：According to the user grouping information of the user, by front End page shows that it has the report name for checking authority towards the user；Report generation task includes：The form that user specifies；According to Daily record data in the Data Mart specified in report generation task, generating form and preserving to report database includes：Use The configuration information of the form that user specifies, the daily record data in the Data Mart specified in report generation task generates report Table.

The method according to the new form of existing report generation is provided in the present embodiment, is properly termed as " form clone ", i.e., Using a configuration information for having developed the form for completing, generate new form, form of so new form etc. all with develop Former form it is similar.

Following embodiments will also be introduced and for report data carry out implementing for visual presentation.

In one embodiment of the invention, said method also includes：When the form for receiving user input checks instruction When, data of the user with the row for checking authority in the form are shown to the user by front end page.

Due to the data volume in form be typically it is very big (business that may include carry out since all data, such as it is several The data in year), row name is only shown in the aforementioned embodiment and does not show the concrete data in row.And check instruction in user input Afterwards, then by specific data (including restrictive condition, for example, time conditions) it is shown.

In one embodiment of the invention, said method also includes：New Report in report database is checked time Number is initialized as zero；When the form for receiving user input checks instruction, corresponding report is checked into that number of times increases by one；For each Whether form arranges cleaning cycle, when the time point of cleaning cycle is reached, check number of times less than clearly according to judge the form Reason threshold value, if being less than, deletes corresponding report generation task.

After report generation task is set up, because daily record data is being continuously generated, therefore report data is also constantly more Newly, this undoubtedly consumes substantial amounts of resource, therefore even 0 form less for the amount of checking, corresponding report generation is appointed Business carries out deletion can economize on resources.

In one embodiment of the invention, said method also includes：According to the user grouping information of user, to its displaying It has the statistics graph model of access right, so that front end page is according to user's statistics graph model selected and the number for having shown that form According to the corresponding cartogram of generation.

Form is typically the mode of form, and this is simultaneously unfavorable for viewing, and the mode of cartogram is just more directly perceived, such as pie Figure, block diagram etc..Therefore the method that cartogram is generated according to report data is provided in the present embodiment.And specifically, statistics Graph model include it is following in any one：According to the mould of the data genaration cartogram for having shown that form cached in front end page Type；Again the data that user in the form has the row for checking authority are obtained from data source, generates the model of cartogram；To opening up The data source for showing form enters the statistics graph model of edlin.

In above-mentioned model, need not according to the model of the data genaration cartogram for having shown that form cached in front end page Interact with server, even if such user is ranked up, classifies etc. operation in front end page to data, all without again Lose time to conduct interviews report database with resource；And the form more sensitive for real-time property, it is possible to use Again the data that user in the form has the row for checking authority are obtained from data source, generates the model of cartogram.Further, since In many cases user wishes to modify form, for example change row name, and access report database be it is sufficiently complex, because This additionally provides the statistics graph model that the data source to having shown that form enters edlin.

Following embodiments will introduce the scheduling of task by taking report generation task as an example.Certainly, daily record data is carried out at ETL Reason is also one of task, and the method that its flow process is introduced with following embodiments is similar.

Because report generation task can expend resource, therefore it is very necessary that rational scheduling is carried out to task.At this In one embodiment of invention, said method also includes：Corresponding task configuration text is generated and preserved according to report generation task Part；According to the multiple tasks configuration file for having preserved, the task topological diagram comprising dependence between task is generated and preserved；According to The task topological diagram of generation completes task scheduling.

Task topological diagram shows the dependence between task, and such as task A only could be transported after the completion of the operation of task B OK.So specifically, completing task scheduling according to the task topological diagram for generating includes：When any one task in task topological diagram When meeting other service conditions in addition to dependence between task of the task, according to task topological diagram judge the task whether according to Rely in other task runs；If being independent of other task runs, the task configuration file of the task is read, directly run this Business；If relying on other task runs, after its other task whole end of run for relying on, then the reading the task of the task Configuration file, runs the task.

The run time of the task of its dependence is often estimated in the scheduling of existing report generation task, and such as task B can 2 points of operations can finish in the afternoon, then the run time of task A is possibly set to afternoon two point ten minutes.It is contemplated that taking office The operation of business is relevant with the idle degree of the cluster of operation task, and when cluster is compared with busy, possible task B can be transported just half past two in the afternoon Row is finished, and so in the afternoon task A of 2 points of operations in ten minutes will operation exception；And work as cluster compared with idle, possible task B is under Half past one at noon is just run and finishes, and until ten minutes afternoons two point task A just can run, the money of this time cluster of 40 minutes Source is just wasted.And in the present embodiment, after the other conditions of task run meet, finish in its task run for relying on Afterwards the task just can be run.

Specifically, any one during report generation task also includes as follows：The basic parameter of task；The time of task run Condition；The cluster of task run；The physical resource condition of task run；The data resource condition of task run；Task and other The dependence of task.

For example, it is intended that task is run on which cluster, it is desirable to which what configuration the machine on cluster meets；Periodic task Could be arranged to be performed in daily fixed time period, and temporary duty can also increase restrictive condition, as only specified Perform in daily fixed time period in one week.When task is submitted to into cluster, can be according to load balancing principle, from task At least one machine is selected to run the task in the cluster of operation.

If the dependence comprising task and other tasks in report generation task, can directly according to task and its The dependence of his task generates task topological diagram；Task topology can also be generated according to the data resource condition of task run Figure, wherein, the data resource condition of task run includes：The input address of task run desired data, and/or task run knot The OPADD of fruit.

For example, the data needed for task A are the results that task B is obtained, then the OPADD of task B and task A Input address matching, thus obtained task A and depended on task B.

User can be viewed by front end page and check the task topological diagram of authority, and it is modified.Therefore In one embodiment of the invention, the method also includes：In response to the displaying instruction that front end page sends, by multiple tasks And/or the task topological diagram comprising dependence between multiple tasks returns to front end page and is shown.User can be with basis There are the multiple tasks for checking authority, generate new task topological diagram.For example, task A and task B are added to into new task topology In figure, and the task A of formulation depends on task B.Or, the dependence in existing task topological diagram is modified.I.e.：Connect Newly-increased/modification/deletion the instruction of dependence between the task that front end page sends is received, is correspondingly generated or modification task topology Figure.Front end page can be visually presented with task topological diagram, and for example, user is when topological diagram is changed, it is only necessary to make task Pull into for node or hauling-out figure, with arrow logo dependence between two tasks.

In one embodiment of the present of invention, corresponding appointing can also be caused by modifying to report generation task Business topological diagram sends change.The modification instruction of report generation task is received, the task configuration file of corresponding task is modified； Instruction is changed according to report generation task to judge whether to need the task topological diagram related to the task to modify, if so, then Modified according to the task topological diagram that amended task configuration file is related to the task.For example, input address are have changed, The task A of may be such that eliminates the reliance on task B, but is changed into dependence task C.

Because report generation task can be to ensure stability towards all users in enterprise, can be using such as lower section Method：Report generation task for receiving report generation task is provided and submits interface to；At least one for task scheduling server Running state parameter arranges corresponding alarm threshold value, the current operating conditions parameter of monitor task dispatch server；When monitoring Any one running state parameter when reaching corresponding alarm threshold value, perform predetermined warning corresponding with the alarm threshold value and operate, And be set to report generation task submission interface unavailable.So in task dispatch server heavy-duty service, Ke Yiting Interface is submitted to report generation task, no longer receive new report generation task, and in task dispatch server no longer high load capacity During operation, the availability that form task submits interface to is recovered again.That is, when the running state parameter for monitoring drops to warning level When under value, report generation task submission interface is set to again available；Interface is submitted to submit to by report generation task Report generation task, generate and preserve corresponding task configuration file.

Refer in previous embodiment, task needs to be submitted in corresponding cluster to be run.In one embodiment of the present of invention In, additionally provide following method：Judge whether the cluster of task run meets task and submit condition to, if meeting, will appoint accordingly Business configuration file is submitted on corresponding cluster.Wherein, at least one during task submits to condition to include as follows：Task run Cluster can be accessed；The available resources of the cluster of task run are not less than predetermined threshold；The cluster of task run is not in safeguarding State.

That is, first judging the cluster of task run, whether whether whether network is unobstructed, safeguarded, can also be transported The capable task.Otherwise task cannot normally be submitted to the cluster, even if being submitted to the cluster, can not correctly run.

User can also check the operation conditions of task by front end page.In one embodiment of the invention, the party Method also includes：In response to the task choosing instruction that front end page is submitted to, the task run state of corresponding task in each cluster is obtained Information, returns to the task run status information of corresponding task front end page and is shown.So user can check at any time It has the running state information of the checking authority of the task, such as：Task run session information；Task run progress msg；Task is remained Remaining temporal information；Task run log information.The error log of task can be read in time, and task is modified；Or, Task is divided into multiple stages, for example, need the statistical computation for carrying out multiple stages, can check that task is carried out that stage. According to the amount of work and cluster resource of task, it may also be inferred that the operation progress of task and tasks leave time.

Task tends not to enough mistake letters for always normally running on cluster, artificial Exclusion Tasks being needed in prior art Breath, wastes time and energy, and in one embodiment of the invention, said method also includes：Receive the mission failure day that each cluster is submitted to Will；Mission failure daily record is analyzed, the failure information of task is obtained.Thus mission failure daily record is manually checked, arrange The time for looking into failure cause saves.Specifically, mission failure daily record is analyzed, obtains the failure information bag of task Include：The default failure sample storehouse comprising at least one failure model；Failure model includes：Mission failure log matches are regular and appoint The failure information of business；Mission failure daily record is matched with the failure model in failure sample storehouse, according to the failure for matching Model obtains the failure information of task.

For example, it is input into without data in path, such task cannot be run, then corresponding record is just had in daily record. Corresponding failure model is unsuccessfully set if such, then by mission failure daily record and the failure model failed in sample storehouse Matched, it is possible to the quick failure information for determining task, for example, included：The failure cause of task, the error code of task, The type of error of task.The type of error of task can include that type can be retried and can not retry type.Without number in be input into path According to as a example by, even if retrying the task, still without data in the input path, then task still will not normally be run.And such as Fruit is only to cannot connect to corresponding data base, then being likely to after retrying will be successful, and such type of error is exactly can be again Examination type mistake.Therefore the failure information of task can also include：The solution of mission failure；The method also includes：Foundation is appointed The solution of business failure, the task is resubmited on corresponding cluster, or, carry out the alert process of predetermined way. For type mistake can be retried, the task is resubmited by it and is retried on corresponding cluster；It is wrong for type can not be retried By mistake, the alert process of predetermined way is carried out, for example, sends mail or note to attendant.

Following embodiments describe the monitoring management carried out to whole platform.

In one embodiment of the invention, said method also includes：Any one operation performed to user is recorded, Corresponding with user profile it is saved in monitoring data storehouse by the operating time.

Although as can be seen that the operation that user performs has strict rights management, being still very sensitive operation. Any one operation for performing to user in the present embodiment is recorded, and can facilitate exclusion in the future, and secret is occurring A suspect is quickly determined during leakage accident.

In one embodiment of the invention, said method also includes：Operation is set by action type and reports to the police tactful and right The warning operation that should be performed；When strategy is reported to the police in the operation for matching the action type of any one operation that user performs, perform Corresponding operation of reporting to the police.

For example, although certain user possesses the authority of checking to large quantities of forms, but within the shorter time period, it is visited in large quantities These forms are asked, this behavior is likely to reveal the behavior of corporate secret, it is therefore desirable to perform corresponding operation of reporting to the police, this Sample just can as far as possible reduce the loss of secret, be saved in time.

Fig. 3 shows a kind of structural representation of the managing device of daily record data according to an embodiment of the invention, such as Shown in Fig. 3, the managing device 300 of daily record data includes：

Daily record data resolution unit 310, is suitable to carry out dissection process to the daily record data of at least one specified services, obtains The daily record data of the formatting of business is simultaneously saved in data warehouse in corresponding Data Mart.

Control of authority unit 320, each Data Mart being suitable in data warehouse configures authority by user grouping.

Display unit 330, is adapted to provide for the front end page for logging in, according to the User logs in letter that front end page sends Breath, determines the user grouping information of the user；When the Data Mart for receiving front end page transmission checks instruction, according to the use The user grouping information at family, shows that it has the Data Mart information for checking authority by front end page to the user.

It can be seen that, the device shown in Fig. 3 is saved in the daily record data dissection process of business into after the daily record data for formatting It is that each Data Mart configures authority by user grouping in the corresponding Data Mart of data warehouse, sends out when front end page is received When the Data Mart for sending checks instruction, according to the log-on message of the user determine user grouping information, by front end page to The user shows that it has the Data Mart information for checking authority.The technical scheme realizes the unified pipe of the daily record data to business Reason warehouse-in and authority distribution, specification, it is orderly realize the management and control to daily record data, be particularly suitable for that service line is more, daily record Data volume is big, the enterprise that organizational structure is complicated.

In one embodiment of the invention, in said apparatus, daily record data resolution unit 310, be suitable to configuration service and The corresponding relation of domain name, the daily record data for receiving is classified by domain name；According to the corresponding domain name of the business, to the domain name Daily record data under classification carries out dissection process.

In one embodiment of the invention, in said apparatus, daily record data resolution unit 310 includes：Sample parsing is single Unit, being suitable to a sample daily record to being input into carries out dissection process, the analysis result of output formatting；Resolution rules label Unit, is suitable to the analysis result of receive user and confirms after instruction, the daily record resolution rules that record parsing sample daily record is adopted；Daily record Resolution unit, is suitable to according to the daily record resolution rules for being recorded, and the Source log data of the business specified to user are carried out at parsing Reason.

In one embodiment of the invention, in said apparatus, sample resolution unit includes：Multiple log content identifications are drawn Hold up, for recognizing the log content of different-format respectively and being parsed into one or more fields；Aggregation process unit, is suitable to sample Example daily record is sequentially inputted in multiple log content identification engines, and each field of output that each log content recognizes engine is converged The analysis result for formatting must be arrived.

In one embodiment of the invention, in said apparatus, resolution rules recording unit is suitable to will have output result The identification resolution rules summary record of each log content identification engine is to parse the daily record resolution rules that sample daily record is adopted.

In one embodiment of the invention, in said apparatus, daily record data resolution unit 310 also includes：Identification engine Administrative unit, is adapted to provide for log content identification engine edition interface, is increased/is deleted/changed in daily record by the interface Hold the instruction of identification engine, performing corresponding log content identification module according to instruction increases/delete/modification operation.

In one embodiment of the invention, in said apparatus, daily record data resolution unit 310 also includes：Make identification by oneself Unit, is suitable to when there is the content of the equal None- identified of each log content identification engine in sample daily record, by self-defined identification Interface exports the content；The recognition result being input into after self-defined identification interface artificial cognition and corresponding identification parsing Rule；The identification resolution rules being input into after the artificial cognition are recorded as parsing the daily record resolution rules that sample daily record is adopted A part.

In one embodiment of the invention, in said apparatus, multiple log contents identification engines include it is following in one Plant or various：IP address recognizes engine；Time-stamp Recognition engine；ID recognizes engine；Channel recognizes engine；In JSON forms Hold identification engine.

In one embodiment of the invention, in said apparatus, IP address recognizes that engine after IP address is identified, enters one Step parsing IP address, IP address analysis result includes following one or more fields：Country, province, city, operator.

In one embodiment of the invention, in said apparatus, sample resolution unit is further adapted for being received by front end page Input to analysis result in the instruction that operated of each field, and perform corresponding operation；Each word in analysis result One or more instructed in including as follows of Duan Jinhang operations：Adjust the instruction of the clooating sequence of each field；Modification designated word The instruction of the title of section；Delete the instruction of specific field.

In one embodiment of the invention, in said apparatus, a log content recognizes the identification resolution rules of engine Including：The parameter value of the specific field to recognizing and in one or more fields for parsing is arranged and limits threshold value；Daily record is parsed Unit, is suitable to for parameter value carries out discard processing beyond the daily record data for limiting threshold value.

In one embodiment of the invention, in said apparatus, daily record data resolution unit 310 also includes：

Memory element, is suitable to determine the row in the corresponding data warehouse of the field according to field attribute；By data to be stored In arranging accordingly in data warehouse by field storage.

In one embodiment of the invention, in said apparatus, memory element is suitable to read the metadata of data warehouse, Obtain the attribute of each row in data warehouse；According to the attribute of each row in field attribute and data warehouse, in setting up data to be stored The data of each field and the mapping relations of Ge Lie.

In one embodiment of the invention, in said apparatus, metadata includes：The service attribute that data warehouse is respectively arranged And/or data attribute, wherein, service attribute include it is following at least one：Business Name, business domain name, business description letter Breath；Data attribute include it is following at least one：Row name, data form, data type.

In one embodiment of the invention, in said apparatus, daily record data resolution unit 310 also includes：Data manipulation Unit, is suitable to according to the data processing instructions for receiving, and the data to specifying row are processed accordingly.During instruction is included as follows At least one：Data deciphering, Data Format Transform, data encoding conversion.

In one embodiment of the invention, in said apparatus, daily record data resolution unit 310 also includes：Data manipulation Unit, is suitable to arrange the instruction for limiting threshold value according to the train value for specifying row, and train value is entered beyond the whole piece data for limiting threshold value Row delete processing.

In one embodiment of the invention, in said apparatus, daily record data resolution unit 310 also includes：：Data manipulation Unit, is suitable to according to the data processing instructions for receiving, and the data to specifying row are processed accordingly.During instruction is included as follows At least one：Adjust the instruction of the clooating sequence of each row；The instruction of the row name of row is specified in modification；Delete the instruction for specifying row.

In one embodiment of the invention, in said apparatus, Data Mart includes at least one tables of data；Control of authority Unit 320, is suitable for each tables of data of Data Mart and checks authority by user grouping configuration, and for tables of data each row by using Family packet configuration checks authority；Display unit 330, is suitable to the user grouping information according to the user, shows that it has to the user The tables of data of authority is checked, and/or shows that it there are the row in the tables of data for checking authority to the user.

In one embodiment of the invention, said apparatus also include：Form unit, is suitable to receive and preserve user and passes through The report generation task that front end page is submitted to, according to the daily record data in the Data Mart specified in report generation task, generates Form is simultaneously preserved to report database；Control of authority unit 320, the authority for being suitable to each row according to form determines the power of the form Limit, wherein, the authority arranged in the tables of data in the corresponding Data Mart of the authority of each row of the form of generation is identical.

In one embodiment of the invention, in said apparatus, form unit is suitable to according to report generation task, it is determined that The data processing model of report generation task is completed, and specified daily record data is read from corresponding Data Mart；Using true Fixed data processing model treats daily record data and carries out data processing, generates form and preserves to report database.

In one embodiment of the invention, in said apparatus, report generation task includes：Input address；According to form Daily record data in the Data Mart specified in generation task, generating form and preserving to report database includes：From input ground Daily record data is read in location, generates form and preserves to report database.In one embodiment of the invention, in said apparatus, Report generation task includes：The standard queries sentence or the query argument of user input of user input；Form unit, be suitable to with The standard queries sentence of family input inquires about corresponding Data Mart, obtains the daily record data in the Data Mart that user specifies；Or Person, according to the query argument of user input standard queries sentence is generated, and with the standard queries sentence for generating corresponding data are inquired about Fairground, obtains the daily record data in the Data Mart specified.

In one embodiment of the invention, in said apparatus, Data Mart checks instruction and/or report generation task bag Include：The standard queries sentence or the query argument of user input of user input；The device also includes：Query unit, be suitable to with The standard queries sentence of family input inquires about corresponding Data Mart, obtain the user have the Data Mart information of checking authority and/ Or the daily record data in the Data Mart specified；Or, standard queries sentence is generated according to the query argument of user input, with life Into standard queries sentence inquire about corresponding Data Mart, obtaining the user has the Data Mart information for checking authority and/or refers to Daily record data in fixed Data Mart.

In one embodiment of the invention, in said apparatus, query unit is further adapted for arranging sentence filtering rule, right The standard queries sentence of user input or the standard queries sentence of generation are filtered.

In one embodiment of the invention, in said apparatus, report generation task includes：The data processing mould specified Type；Form unit, is suitable to from data processing model storehouse the data processing model for selecting to specify.

In one embodiment of the invention, in said apparatus, following at least one is included in data processing model storehouse Data processing model：Newly-increased statistical model；Enliven statistical model；Retain statistical model.

In one embodiment of the invention, in said apparatus, it is intended that data processing model be streaming models； Report generation task also includes：At least one section self-defining code snippet；Parameter information includes：Every section of code snippet with The corresponding relation of one piece of logic partitioning in streaming models.

In one embodiment of the invention, in said apparatus, report generation task includes：Self-defining data processing mould The address of type；Form unit, is suitable to read self-defining data processing model from the address.

In one embodiment of the invention, in said apparatus, form unit is further adapted for self-defining data processing mould Type is saved in data processing model storehouse.

In one embodiment of the invention, in said apparatus, form unit is further adapted for receive user by front end page The form of upload, and/or the form store path submitted to by front end page according to user, the report specified is obtained from the path Table, is saved in report database.

In one embodiment of the invention, in said apparatus, display unit 330 is further adapted for according to the user of user point Group information, to it the Data Mart and/or form of editable authority is shown；Control of authority unit 220, is further adapted for receive user and carries The authority edit instruction of friendship, is edited accordingly to the authority of Data Mart and/or form.

In one embodiment of the invention, in said apparatus, display unit 330 is further adapted for the user according to the user Grouping information, shows that it has the report name for checking authority by front end page to the user；Report generation task includes：User The form specified；Form unit, is adapted in use to the configuration information of the form that user specifies, according to what is specified in report generation task Daily record data in Data Mart generates form.

In one embodiment of the invention, in said apparatus, display unit 330 is further adapted for that user input ought be received Form when checking instruction, show that user in the form has the data of the row for checking authority to the user by front end page.

In one embodiment of the invention, said apparatus also include：Form checks recording unit, is suitable to report data New Report checks that number of times is initialized as zero in storehouse；When the form for receiving user input checks instruction, by corresponding report Check number of times increase by one；Task clears up unit, is suitable for each form and arranges cleaning cycle, in the time point for reaching cleaning cycle When, check that number of times, whether less than cleaning threshold, if being less than, deletes corresponding report generation task according to judge the form.

In one embodiment of the invention, in said apparatus, display unit 330 is further adapted for according to the user of user point Group information, shows that it has the statistics graph model of access right to it, so that the statistics graph model that front end page is selected according to user With the data for having shown that form, corresponding cartogram is generated.

In one embodiment of the invention, in said apparatus, any one during graph model is included as follows is counted：According to front What is cached in end page face has shown that the model of the data genaration cartogram of form；Again user's tool in the form is obtained from data source There are the data of the row for checking authority, generate the model of cartogram；Data source to having shown that form enters the statistics artwork of edlin Type.

In one embodiment of the invention, said apparatus also include：Task dispensing unit, is suitable to be appointed according to report generation Business generates and preserves corresponding task configuration file；Task topological diagram administrative unit, is suitable to be matched somebody with somebody according to the multiple tasks for having preserved File is put, the task topological diagram comprising dependence between task is generated and preserve；Task scheduling unit, was suitable to according to appointing for generating Business topological diagram completes task scheduling.

In one embodiment of the invention, in said apparatus, task scheduling unit was suitable to when appointing in task topological diagram When one task meets other service conditions in addition to dependence between task of the task, this is judged according to task topological diagram Whether business depends on other task runs；If being independent of other task runs, the task configuration file of the task is read, directly Run the task；If relying on other task runs, after its other task whole end of run for relying on, then this is read The task configuration file of business, runs the task.

In one embodiment of the invention, in said apparatus, task configuration information include it is following in one or more： The basic parameter of task；The time conditions of task run；The cluster of task run；The physical resource condition of task run；Task The data resource condition of operation；The dependence of task and other tasks.

In one embodiment of the invention, in said apparatus, task topological diagram administrative unit is suitable to from each task configure The dependence of the task and other tasks is read in file, task topological diagram is generated.

In one embodiment of the invention, in said apparatus, task topological diagram administrative unit is suitable to from each task configure The data resource condition of the task run is read in file, task topological diagram is generated；The data resource condition of task run includes： The input address of task run desired data, and/or the OPADD of task run result.

In one embodiment of the invention, in said apparatus, task scheduling unit is suitable to according to load balancing principle, At least one machine is selected to run the task from the cluster of task run.

In one embodiment of the invention, in said apparatus, display unit 330 is further adapted for being sent out in response to front end page Multiple tasks and/or task topological diagram comprising dependence between multiple tasks are returned to front end page by the displaying instruction sent It is shown.

In one embodiment of the invention, in said apparatus, task topological diagram administrative unit is further adapted for receiving front-end page Newly-increased/modification/deletion the instruction of dependence, correspondingly generates or changes task topological diagram between the task that face sends.

In one embodiment of the invention, in said apparatus, task dispensing unit is further adapted for receiving report generation task Modification instruction, modifies to the task configuration file of corresponding task；Task topological diagram administrative unit, is suitable to according to report generation Task modification instruction judges whether to need the task topological diagram related to the task to modify, if so, then according to amended The task configuration file task topological diagram related to the task is modified.

In one embodiment of the invention, form unit, is adapted to provide for the form for receiving report generation task and gives birth to Interface is submitted to into task；Said apparatus also include：Monitoring unit, is suitable at least one running status of task scheduling server The corresponding alarm threshold value of parameter setting, the current operating conditions parameter of monitor task dispatch server；Alert process unit, is suitable to When any one running state parameter for monitoring reaches corresponding alarm threshold value, predetermined report corresponding with the alarm threshold value is performed Alert operation, and report generation task submission interface is set to unavailable.

In one embodiment of the invention, in said apparatus, alert process unit is further adapted for when the operation shape for monitoring When state parameter drops under alarm threshold value, report generation task submission interface is set to again available；Task dispensing unit, fits In the report generation task for submitting interface submission to by report generation task, corresponding task configuration text is generated and preserved Part.

In one embodiment of the invention, in said apparatus, task dispensing unit is further adapted for being appointed according to report generation Business, determines the cluster of task run；Judge whether the cluster of task run meets task and submit condition to, if meeting, will be corresponding Task configuration file is submitted on corresponding cluster.

In one embodiment of the invention, in said apparatus, task submit to condition include it is following at least one：Appoint The cluster of business operation can be accessed；The available resources of the cluster of task run are not less than predetermined threshold；The cluster of task run is not In the state of maintenance.

In one embodiment of the invention, in said apparatus, display unit 330 is adapted for front end page submission Task choosing instruction, the task run status information of corresponding task in each cluster is obtained, by the task run shape of corresponding task State information returns to front end page and is shown.

In one embodiment of the invention, in said apparatus, task run status information include it is following at least one Kind：Task run session information；Task run progress msg；Tasks leave temporal information；Task run log information.

In one embodiment of the invention, said apparatus also include：Mission failure journal processing unit, is suitable to receive each The mission failure daily record that cluster is submitted to, is analyzed to mission failure daily record, obtains the failure information of task.

In one embodiment of the invention, in said apparatus, mission failure journal processing unit is preset with comprising at least The failure sample storehouse of one failure model, failure model includes：The failure information of mission failure log matches rule and task；Lose Lose log analysis unit to be suitable to be matched mission failure daily record with the failure model in failure sample storehouse, according to what is matched Failure model obtains the failure information of task.

In one embodiment of the invention, in said apparatus, the failure information of task include it is following at least one： The failure cause of task, the error code of task, the type of error of task.

In one embodiment of the invention, in said apparatus, the failure information of task includes：The solution party of mission failure Case；Mission failure journal processing unit, is further adapted for, according to the solution of mission failure, the task being resubmited corresponding On cluster, or, carry out the alert process of predetermined way.

In one embodiment of the invention, said apparatus also include：Monitoring unit, is suitable to any one performed to user Operation is recorded, and corresponding with user profile is saved in monitoring data storehouse by the operating time.

In one embodiment of the invention, said apparatus also include：Alarm unit, is suitable to arrange operation by action type The warning operation that strategy and the correspondence of reporting to the police is performed；When the operation warning for matching the action type of any one operation that user performs When tactful, corresponding operation of reporting to the police is performed.

It should be noted that the specific embodiment of above-mentioned each device embodiment is concrete with aforementioned corresponding method embodiment Mode is identical, will not be described here.

In sum, technical scheme, by the daily record data dissection process of business into the daily record data for formatting After be saved in the corresponding Data Mart of data warehouse, be each Data Mart configures authority by user grouping, before receiving When the Data Mart that end page face sends checks instruction, according to the user grouping information that the log-on message of the user determines, by front End page shows that it has the Data Mart information for checking authority towards the user.The technical scheme realizes the daily record data to business Unified management warehouse-in and authority distribution, specification, it is orderly realize the management and control to daily record data, be particularly suitable for service line The many, enterprise that daily record data amount is big, organizational structure is complicated.

It should be noted that：

Provided herein algorithm and display be not inherently related to any certain computer, virtual bench or miscellaneous equipment. Various fexible units can also be used together based on teaching in this.As described above, construct required by this kind of device Structure be obvious.Additionally, the present invention is also not for any certain programmed language.It is understood that, it is possible to use it is various Programming language realizes the content of invention described herein, and the description done to language-specific above is to disclose this Bright preferred forms.

In description mentioned herein, a large amount of details are illustrated.It is to be appreciated, however, that the enforcement of the present invention Example can be put into practice in the case of without these details.In some instances, known method, structure is not been shown in detail And technology, so as not to obscure the understanding of this description.

Similarly, it will be appreciated that in order to simplify the disclosure and help understand one or more in each inventive aspect, exist Above in the description of the exemplary embodiment of the present invention, each feature of the present invention is grouped together into single enforcement sometimes In example, figure or descriptions thereof.However, the method for the disclosure should be construed to reflect following intention：I.e. required guarantor The more features of feature that the application claims ratio of shield is expressly recited in each claim.More precisely, such as following Claims reflect as, inventive aspect is all features less than single embodiment disclosed above.Therefore, Thus the claims for following specific embodiment are expressly incorporated in the specific embodiment, wherein each claim itself All as the separate embodiments of the present invention.

Those skilled in the art are appreciated that can be carried out adaptively to the module in the equipment in embodiment Change and they are arranged in one or more equipment different from the embodiment.Can be the module or list in embodiment Unit or component are combined into a module or unit or component, and can be divided in addition multiple submodule or subelement or Sub-component.In addition at least some in such feature and/or process or unit is excluded each other, can adopt any Combine to all features disclosed in this specification (including adjoint claim, summary and accompanying drawing) and so disclosed Where all processes or unit of method or equipment are combined.Unless expressly stated otherwise, this specification is (including adjoint power Profit is required, summary and accompanying drawing) disclosed in each feature can it is identical by offers, be equal to or the alternative features of similar purpose carry out generation Replace.

Although additionally, it will be appreciated by those of skill in the art that some embodiments described herein include other embodiments In included some features rather than further feature, but the combination of the feature of different embodiments means in of the invention Within the scope of and form different embodiments.For example, in the following claims, embodiment required for protection appoint One of meaning can in any combination mode using.

The present invention all parts embodiment can be realized with hardware, or with one or more processor operation Software module realize, or with combinations thereof realization.It will be understood by those of skill in the art that can use in practice Microprocessor or digital signal processor (DSP) are come in the managing device for realizing daily record data according to embodiments of the present invention The some or all functions of some or all parts.The present invention is also implemented as performing method as described herein Some or all equipment or program of device (for example, computer program and computer program).Such reality The program of the existing present invention can be stored on a computer-readable medium, or can have the form of one or more signal. Such signal can be downloaded from internet website and obtained, or be provided on carrier signal, or in any other form There is provided.

It should be noted that above-described embodiment the present invention will be described rather than limits the invention, and ability Field technique personnel can design without departing from the scope of the appended claims alternative embodiment.In the claims, Any reference markss between bracket should not be configured to limitations on claims.Word "comprising" is not excluded the presence of not Element listed in the claims or step.Word "a" or "an" before element does not exclude the presence of multiple such Element.The present invention can come real by means of the hardware for including some different elements and by means of properly programmed computer It is existing.If in the unit claim for listing equipment for drying, several in these devices can be by same hardware branch To embody.The use of word first, second, and third does not indicate that any order.These words can be explained and be run after fame Claim.

Embodiment of the invention discloses that A1, a kind of management method of daily record data, wherein, the method includes：

Each Data Mart in for data warehouse configures authority by user grouping；

A2, the method as described in A1, wherein, the method also includes：

A3, the method as described in A1, wherein, the Data Mart includes at least one tables of data；

A4, the method as described in A3, wherein, the method also includes：

A5, the method as described in A4, wherein, the Data Mart checks that instruction and/or report generation task include：

A6, the method as described in A5, wherein, the method also includes：

A7, the method as described in A4, wherein, the method also includes：

A8, the method as described in A4, wherein, the method also includes：

A9, the method as described in A4, wherein, the method also includes：

The report generation task includes：The form that user specifies；

A10, the method as described in A9, wherein, the method also includes：

A11, the method as described in A10, wherein, the method also includes：

A12, the method as described in A10, wherein, the method also includes：

A13, the method as described in A12, wherein, any one counted during graph model is included as follows：

A14, the method as any one of A1-A13, wherein, the method also includes：

A15, the method as described in A14, wherein, the method also includes：

Embodiments of the invention also disclose B16, a kind of managing device of daily record data, wherein, the device includes：

B17, the device as described in B16, wherein,

The daily record data resolution unit, is suitable to the corresponding relation of configuration service and domain name, by the daily record data for receiving Classified by domain name；According to the corresponding domain name of the business, dissection process is carried out to the daily record data under the domain name classification.

B18, the device as described in B16, wherein, the Data Mart includes at least one tables of data；

B19, the device as described in B18, wherein, the device also includes：

B20, the device as described in B19, wherein, the Data Mart checks that instruction and/or report generation task include：

B21, the device as described in B20, wherein,

The query unit, is further adapted for arranging sentence filtering rule, the standard queries sentence or generation to user input Standard queries sentence is filtered.

B22, the device as described in B19, wherein,

The form unit, is further adapted for the form that receive user is uploaded by front end page, and/or according to user by front The form store path that end page face is submitted to, the form specified is obtained from the path, is saved in report database.

B23, the device as described in B19, wherein,

The display unit, is further adapted for the user grouping information according to user, and to it the data set of editable authority is shown City and/or form；

B24, the device as described in B19, wherein,

The display unit, is further adapted for the user grouping information according to the user, is shown to the user by front end page It has the report name for checking authority；

The report generation task includes：The form that user specifies；

B25, the device as described in B24, wherein,

The display unit, is further adapted for when the form for receiving user input checks instruction, by front end page to this User shows data of the user with the row for checking authority in the form.

B26, the device as described in B25, wherein, the device also includes：

B27, the device as described in B25, wherein,

The display unit, is further adapted for the user grouping information according to user, shows that it has the statistics of access right to it Graph model, so that the front end page is generated corresponding according to user's statistics graph model selected and the data for having shown that form Cartogram.

B28, the device as described in B27, wherein, any one counted during graph model is included as follows：

B29, the device as any one of B16-B28, wherein, the device also includes：

B30, the device as described in B29, wherein, the device also includes：

Alarm unit, is suitable to arrange the warning operation for operating report to the police strategy and correspondence execution by action type；When user holds When the operation for matching the action type of capable any one operation reports to the police tactful, perform corresponding warning and operate.

Claims

1. a kind of management method of daily record data, wherein, the method includes：

Dissection process is carried out to the daily record data of at least one specified services, the daily record data of the formatting of business is obtained and is preserved To in data warehouse in corresponding Data Mart；

Each Data Mart in for data warehouse configures authority by user grouping；

Front end page is provided, according to the user login information that front end page sends, the user grouping information of the user is determined；When connecing The Data Mart of front end page transmission is received when checking instruction, according to the user grouping information of the user, by front end page to The user shows that it has the Data Mart information for checking authority.

2. the method for claim 1, wherein the method also includes：

The daily record data at least one specified services carries out dissection process to be included：It is right according to the corresponding domain name of the business Daily record data under the domain name classification carries out dissection process.

3. the method for claim 1, wherein the Data Mart includes at least one tables of data；

It is described for data warehouse in each Data Mart by user grouping configuration authority also include：For each tables of data of Data Mart Authority is checked by user grouping configuration, and check authority by user grouping configuration for each row of tables of data；

The user grouping information according to the user, shows that it has the data set for checking authority by front end page to the user City's information includes：According to the user grouping information of the user, show that it has a tables of data for checking authority to the user, and/or to The user shows that it has the row in the tables of data for checking authority.

4. method as claimed in claim 3, wherein, the method also includes：

According to the daily record data in the Data Mart specified in the report generation task, generate form and preserve to report data Storehouse；Wherein, the authority for arranging in the tables of data in the corresponding Data Mart of the authority of each row of the form of generation is identical；According to The authority of each row of form determines the authority of the form.

5. method as claimed in claim 4, wherein, the Data Mart checks that instruction and/or report generation task include：

The method also includes：Corresponding Data Mart is inquired about with the standard queries sentence of user input, the user is obtained and is checked Daily record data in the Data Mart information of authority and/or the Data Mart specified；Or, according to the query argument of user input Standard queries sentence is generated, corresponding Data Mart is inquired about with the standard queries sentence for generating, obtained the user and check authority Data Mart information and/or the Data Mart specified in daily record data.

6. a kind of managing device of daily record data, wherein, the device includes：

Daily record data resolution unit, is suitable to carry out dissection process to the daily record data of at least one specified services, obtains business The daily record data of formatting is simultaneously saved in data warehouse in corresponding Data Mart；

Display unit, is adapted to provide for front end page, according to the user login information that front end page sends, determines the user of the user Grouping information；When the Data Mart for receiving front end page transmission checks instruction, according to the user grouping information of the user, lead to Cross front end page and show that it has the Data Mart information for checking authority to the user.

7. device as claimed in claim 6, wherein,

The daily record data resolution unit, is suitable to the corresponding relation of configuration service and domain name, by the daily record data for receiving by domain Name is classified；According to the corresponding domain name of the business, dissection process is carried out to the daily record data under the domain name classification.

8. device as claimed in claim 6, wherein, the Data Mart includes at least one tables of data；

The control of authority unit, is suitable for each tables of data of Data Mart and checks authority by user grouping configuration, and for number Authority is checked according to each row of table by user grouping configuration；

The display unit, is suitable to the user grouping information according to the user, shows that it has the data for checking authority to the user Table, and/or show that it has the row in the tables of data for checking authority to the user.

9. device as claimed in claim 8, wherein, the device also includes：

Form unit, is suitable to receive and preserve the report generation task that user is submitted to by front end page, is given birth to according to the form Daily record data in the Data Mart specified in task, generates form and preserves to report database；

The control of authority unit, the authority for being suitable to each row according to form determines the authority of the form, wherein, the form of generation Each row the corresponding Data Mart of authority in tables of data in the authority that arranges it is identical.

10. device as claimed in claim 9, wherein, the Data Mart checks that instruction and/or report generation task include：

The device also includes：Query unit, is suitable to inquire about corresponding Data Mart with the standard queries sentence of user input, obtains The user have the Data Mart information for checking authority and/or the Data Mart specified in daily record data；Or, it is defeated according to user The query argument for entering generates standard queries sentence, and with the standard queries sentence for generating corresponding Data Mart is inquired about, and obtains the use Family have the Data Mart information for checking authority and/or the Data Mart specified in daily record data.