CN101719139B

CN101719139B - Method for monitoring data quality based on index set

Info

Publication number: CN101719139B
Application number: CN2009102126751A
Authority: CN
Inventors: 万星明; 刘树权; 余志刚; 孙力斌; 沈鹏程; 兰清
Original assignee: NANJING CITY LINKAGE SYSTEM INTEGRATION CO Ltd
Current assignee: NANJING CITY LINKAGE SYSTEM INTEGRATION CO Ltd
Priority date: 2009-11-10
Filing date: 2009-11-10
Publication date: 2012-07-04
Anticipated expiration: 2029-11-10
Also published as: CN101719139A

Abstract

The invention provides a method for monitoring data quality based on an index set. The method comprises the following steps of: defining a plurality of index sets according to a data index; judging whether the data indexes in the index set is normal or not according to a checking point; and managing the checking point through a detachable dispatching module. All checking in the method for monitoring the data quality based on the index set all takes the data index set as a center, 60 percent of coding amount on checking codes can be reduced, and fussy manual operation with low efficiency is avoided. The dispatching module of the checking point is detachable and can be applied to any other modules for reutilization.

Description

Data quality monitoring method based on index set

Technical field

The present invention is relevant for quality of data platform field, and particularly relevant for a kind of data quality monitoring method based on index set.

Background technology

At present, existing quality of data platform does not have independent data target monitoring mechanism, just need to lean on and wait mode manually if obtain some data targets, for example to the server exectorial mode to obtain index.There is the problem of the duplication of labour and underaction in above-mentioned traditional method.

Summary of the invention

In view of this, the present invention provides a kind of data quality monitoring method based on index set, and the monitoring of data quality is had higher flexibility and transplantability.

The present invention proposes a kind of data quality monitoring method based on index set, may further comprise the steps: define some index sets according to data target; Whether the data target of concentrating through the checkpoint judge index is normal; Manage said checkpoint through separable scheduler module.

Further, the These parameters collection extracts data target through self-defining SQL and generates, and adopts XML to be configured.

Further, above-mentioned checkpoint is to be made up of one section script.

Further, above-mentioned checkpoint can be expanded and send mail and send the built-in bag of note.

Further, the general caching bag is used in above-mentioned checkpoint, and the script startup engine is carried out buffer memory.

Further, the trigger mechanism of above-mentioned scheduler module comprises that Time Triggered and condition trigger.

Beneficial effect of the present invention does; All inspections all are to be the center with the data target collection in the data quality monitoring method based on index set provided by the present invention; And all checkpoints all are one section scripts, select to use script to make a decision and have greatly improved dirigibility.The scheduler module of checkpoint is separable module, and does not aim at quality of data platform design, can be applied to other any modules, makes it obtain recycling.Utilize method for supervising of the present invention can reduce the size of code of 60% inspection code, avoided manually-operated loaded down with trivial details and inefficiency, non-professional expert also can be monitored some indexs.

Description of drawings

Shown in Figure 1 is schematic flow sheet according to the data quality monitoring method based on index set of the present invention.

Shown in Figure 2 is function block schematic diagram among Fig. 1.

Embodiment

For let above and other objects of the present invention, feature and advantage can be more obviously understandable, hereinafter is special lifts preferred embodiment, and conjunction with figs., elaborates as follows.

Shown in Figure 1 is schematic flow sheet according to the data quality monitoring method based on index set of the present invention.Shown in Figure 2 is function block schematic diagram among Fig. 1, please in the lump with reference to figure 1 and Fig. 2.As shown in Figure 1, this method comprises:

Step S10 defines some index sets according to data target.In the present embodiment, each index set all is to have a self-defining SQL to extract data target to generate.Consider transplantability, adopt XML to dispose these indexs.With each index definition is MAP, and it is exactly the index set that we need that these all MAP combine.And these indexs are not overlap each other, and are separate.Concrete XML is following:

<maps>

< map id=" report 1 " db=" ods " name=" inspection 1 " >

<！[CDATA[

SELECT?KEY1?FROM?TABLE_A?WHERE....

]]>

</map>

</maps>

Step S11 judges through the checkpoint whether the data target in the said index set is normal.In the present embodiment, whether normally be to have a checkpoint to judge for data target.Wherein, the checkpoint is one section RUBY script.As shown in Figure 2, if the checkpoint judged result is normal, then circulate; If the checkpoint judged result is unusual, then alarm.

Object lesson is following:

< checkpoint id=" 1 " name=" inspection of XXX " >

<！[CDATA[

$result.error()if$report1＜$report2

]]>

</checkpoint>

The checkpoint uses script to judge the extendability of the whole data target monitoring mechanism of great raising.For example, can expand and send mail and send the built-in bag of note, when checking out problem, as the built-in bag use of alarming mechanism in the checkpoint.

It is exactly very slow when starting script that yet the checkpoint uses script can bring a very big drawback, can just cause system effectiveness low like this.In the present embodiment, the script of checkpoint has used some general caching bags, carries out buffer memory for the script startup engine.Be can be a bit slow so only, carry out the delay of script later on regard to imperceptible travelling speed starting for the first time script engine.

Step S12 manages said checkpoint through separable scheduler module.In the present embodiment, all checkpoints are managed by a uniform dispatching module.Here checking mechanism module and scheduler module have realized de fully, make scheduler module not aim at quality of data platform design, can be used for other any modules.Can let sundry item reuse this dispatching platform like this.

In the present embodiment, time scheduling mainly is that a CRONTAB expresses when to trigger and calls in the scheduler module, here the expression formula rich of CRONTAB.Specifically, a CRONTAB expression formula has at least 6 (also possibility is 7) that the time element of space-separated is arranged.Be followed successively by in order:

Second (0～59)

Minute (0～59)

Hour (0～23)

My god (moon) (0～31, need to consider the fate of this month)

The moon (0～11)

My god (week) (1～71=SUN or SUN, MON, TUE, WED, THU, FRI, SAT)

Time (1970-2099)

A simple example is following:

" 01510? * 6#3 " representative every month the 3rd the morning on Friday 10:15 trigger;

" 00/514,18**? " Representative 2 of every afternoons during the 2:55 with 6 pm during the 6:55 per 5 minutes trigger.

Traditional monitoring trigger mechanism mainly is a Time Triggered, in the present embodiment, triggers except Time Triggered mechanism can also comprise condition.Condition triggers can carry out external trigger, combines such as the ETL flow process with ODS.The interface of an external trigger for example can be set, when flow process of ETL finishes, just can trigger a checkpoint as long as ETL calls this external trigger interface.

Specifically, at first at server deploy quality of data background scheduler, and the database of initialization background scheduler; Then in server deploy quality of data index watchdog routine, and in server deploy note mail alerting service program; Be used for configuration schedules and checkpoint in monitor supervision platform deploy WEB module at last; Be included in the dispatch service of configuration index inspect-type on the index monitoring interface, in the scheduling time of the XML of configuration index inspection on the index monitoring interface and configuration index inspection on the index monitoring interface.

The case of practical implementation described in the present invention is merely preferable case study on implementation of the present invention, is not to be used for limiting practical range of the present invention.Be that all equivalences of doing according to the content of claim of the present invention change and modification, all should be as technological category of the present invention.

Claims

1. the data quality monitoring method based on index set is characterized in that, may further comprise the steps: define some index sets according to data target; Said index set is extracted data target through self-defining SQL and is generated, and adopts XML to be configured; With each index definition is MAP, and it is exactly the index set that needs that these all MAP combine;

Judge through the checkpoint whether the data target in the said index set is normal; Said checkpoint is to be made up of one section script; Said checkpoint can be expanded and send mail and send the built-in bag of note; The general caching bag is used in the checkpoint, and the script startup engine is carried out buffer memory; Manage said checkpoint through separable scheduler module; The trigger mechanism of said scheduler module comprises that Time Triggered and condition trigger.