CN104866619A - Data monitoring method and system for data warehouse - Google Patents
Data monitoring method and system for data warehouse Download PDFInfo
- Publication number
- CN104866619A CN104866619A CN201510312275.3A CN201510312275A CN104866619A CN 104866619 A CN104866619 A CN 104866619A CN 201510312275 A CN201510312275 A CN 201510312275A CN 104866619 A CN104866619 A CN 104866619A
- Authority
- CN
- China
- Prior art keywords
- data
- mode information
- warehouse
- task
- monitor task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Testing And Monitoring For Control Systems (AREA)
Abstract
The application discloses a data monitoring method and system for a data warehouse. The data monitoring method comprises the following steps: configuring and storing acquisition item way information for acquiring data from the data warehouse; configuring and storing contrast judgment way information of the acquired data and existing contrast data, and storing alarm notification information in case of an abnormity; correspondingly binding the data acquisition item way information and the contrast judgment way information with a scheduling task for data processing, and generating a corresponding monitoring task; and at the end of execution of the scheduling task, running a corresponding monitoring task, acquiring corresponding data in the data warehouse according to the corresponding data acquisition item way information by the monitoring task, contrasting the acquired data with the existing contrast data according to a corresponding contrast judgment way, judging a contrast result, and transmitting the alarm notification information under the situation of a contrast result abnormity. Through adoption of the data monitoring method and system, automation of data quality monitoring of the data warehouse can be enhanced.
Description
Technical field
The application relates to field of computer data processing, particularly relates to a kind of data monitoring method and system of data warehouse.
Background technology
Data warehouse (Data Warehouse) is the decision-making process for all ranks of enterprise, provides the strategy set of all types Data support.It is that individual data stores, and creates for analytical presentation and decision support object.For needing the enterprise of business intelligence, provide service guidance flow scheme improvements, Looking Out Time, cost, quality and control.
Data warehouse is the structural data environment of decision support system (DSS) (DSS) and on-line analysis application data source.The problem of data warehouse research and solution obtaining information from database.
Distinguish with database, database is the design towards affairs, and data warehouse is subject-oriented design.Data warehouse deposits in a large number in case at database, in order to further mining data resource, in order to decision-making need and produce.
The data quality problems such as at present, the scale along with data warehouse is more next many, and large data warehouse needs task up to ten thousand every day, and data are imperfect, subregion is incomplete, enumerated value is abnormal are abnormal outstanding, directly have influence on the stable of depot data and data reliability.In prior art, data warehouse technology mainly contains Hadoop, Hive, java etc.
In prior art, do not having under data quality monitoring system situation, data warehouse slip-stick artist and Data Analyst are when determining that whether data are reliable, need manually to write Structured Query Language (SQL) (SQl, StructuredQuery Language) order, data query, draws data result, and carry out comparing, the integrality of verification msg.
But there is following shortcoming in prior art: (1) manually writes SQL query data and comparison, wastes time and energy very much.(2), easily there is error in artificial data inquiry and comparison.(3) data quality monitoring of table up to ten thousand every day cannot manually be completed.
Summary of the invention
In view of this, fundamental purpose of the present invention is to provide a kind of data monitoring system and method for data warehouse, to improve the robotization of the data quality monitoring of data warehouse.
Technical scheme of the present invention is achieved in that
A data monitoring method for data warehouse, comprising:
Configuration store is from the collection item mode information of data warehouse image data;
The contrast judgement mode information of the data that configuration store gathers and existing comparison data, stores alert notice information when occurring abnormal;
By described data acquisition item mode information, described contrast judgement mode information, carry out corresponding binding with the scheduler task for data mart modeling, and generate corresponding monitor task;
After described scheduler task execution terminates, run corresponding monitor task, this monitor task is according to the corresponding data in the data acquisition item mode information acquisition data warehouse of its correspondence, according to the contrast judgement mode of correspondence, institute's image data and existing comparison data are compared, and judge comparison result, send alert notice information when comparison result exception.
Preferably, the method comprises further:
According to gathered data, the image data item mode information of described storage and described contrast judgement mode information are modified.
Preferably, described known comparison data is stored in fabric memory, and described collection item mode information and contrast judgement mode information are stored in relational database.
Preferably, described monitor task, according to the corresponding data in the data acquisition item mode information acquisition data warehouse of its correspondence, specifically comprises:
In data warehouse, broker module is set, described data acquisition item mode information transmission is given described broker module by described monitor task, the monitoring actuator in this broker module notification data warehouse performs the query statement in described data acquisition item mode information, inquires corresponding data and returns to monitor task as image data.
Preferably, it is described after described scheduler task execution terminates, before running corresponding monitor task, comprise further: the running status judging described data warehouse, if the running status of data warehouse is higher than the busy extent of specifying, then delay to run described monitor task, otherwise run described monitor task.
Preferably, described existing comparison data comprises: historical data or dimension table data.
A data monitoring system for data warehouse, comprising:
First configuration module, for the collection item mode information of configuration store from data warehouse image data;
Second configuration module, the contrast judgement mode information of the data gathered for configuration store and existing comparison data, stores alert notice information when occurring abnormal;
Binding module, for by described data acquisition item mode information, described contrast judgement mode information, carries out corresponding binding with the scheduler task for data mart modeling, and generates corresponding monitor task;
Monitoring module, for after described scheduler task execution terminates, run corresponding monitor task, this monitor task is according to the corresponding data in the data acquisition item mode information acquisition data warehouse of its correspondence, according to the contrast judgement mode of correspondence, institute's image data and existing comparison data are compared, and judges comparison result, send alert notice information when comparison result exception.
Preferably, gathered data feedback is given described first configuration module and the second configuration module by described monitoring module, and described first configuration module, according to gathered data, is modified to the image data item mode information of described storage; Described second configuration module, according to gathered data, is modified to the contrast judgement mode information of described storage.
Preferably, described known comparison data is stored in fabric memory; Described collection item mode information is stored in relational database by described first configuration module, and described contrast judgement mode information is stored in relational database by described second configuration module.
Preferably, described existing comparison data comprises: historical data or dimension table data.
Compared with prior art, the present invention can produce the abnormal data occurred in (ETL) process in monitor data warehouse; Automatic operation monitoring task can be realized, improve the robotization of the data quality monitoring of data warehouse, solve wasting time and energy of manual alignment data; Meanwhile, the present invention can also realize alert notice mechanism flexibly, can blame people provide various forms of warning for data minus; The present invention also can control the working time of monitor task, to the consumption of warehouse computational resource during avoiding warehouse high capacity.
The present invention can solve the data quality monitoring of global data warehouse, finds data quality problem and alarm and notification data responsible official, and provides detailed quality monitoring result data, judges data quality problem source for data operation personnel.
Accompanying drawing explanation
Fig. 1 is a kind of process flow diagram of the data monitoring method of data warehouse of the present invention;
Fig. 2 is the one composition schematic diagram of the data monitoring system of data warehouse of the present invention.
Embodiment
Below in conjunction with drawings and the specific embodiments, the present invention is further described in more detail.
Fig. 1 is a kind of process flow diagram of the data monitoring method of data warehouse of the present invention.See Fig. 1, the method mainly comprises:
Step 101, configuration store are from the collection item mode information of data warehouse image data.
In this step, need to be configured storage relevant information according to configuration-direct, a configuration interface be normally provided, by basic configuration-direct information display out, by user therefrom option and installment instruction or directly input corresponding configuration-direct.The described collection item mode information from data warehouse image data comprises: gather the collection subregion of target, data query conditions, also comprise some inquiry indexs such as: the line number of record, the maximal value of data and minimum value, mean value etc.In the process storing described collection item mode information, also need the data query conditions of initial configuration and other parameter described to be converted into the query statement (such as SQL statement etc.) that data warehouse can identify, like this when data query, data warehouse can inquire about qualified data according to these query statements.In typical embodiment, described querying condition can be filled according to the form of HiveSQL.
The contrast judgement mode information of the data that step 102, configuration store gather and existing comparison data, stores alert notice information when occurring abnormal.
In this step, need to be configured storage relevant information according to configuration-direct, a configuration interface be normally provided, by basic configuration-direct information display out, by user therefrom option and installment instruction or directly input corresponding configuration-direct.Described existing comparison data such as mainly comprises historical data and dimension table data etc.Described dimension table is the mapping table of a kind of concrete data content and computer code, such as data content is " man " or " female ", adopt 1 mark " man " in the data, adopt 2 marks " female ", so the corresponding relation of 1 correspondence " man " and 2 correspondences " female " is just stored in dimension table.Utilize dimension table to compare to data and just can judge that exception has appearred in which data.
The contrast judgement mode information of described gathered data and existing comparison data, specifically can comprise: the contrast judgement mode information of the data gathered and historical data and corresponding threshold value thereof, such as alignments is the mean value first calculating data according to a certain dimension (such as certain hour section), if this mean value is greater than a threshold value of specifying, send corresponding warning information, described warning information can be arranged voluntarily;
The contrast judgement mode information of described gathered data and existing comparison data, also specifically can comprise: the contrast judgement mode information of the data gathered and dimension table data, such as alignments is all data of certain dimension of traversal (as sex), judge that whether its concrete data are different from described dimension table data, if its concrete data are not that data in dimension table are as 0 or 1, then send corresponding warning information, described warning information can be arranged voluntarily.
Described alarm notification information can be short message alarm announcement information, also can be mail alarm notification information etc.
Configuration store described in the present invention, refer to the storage control carrying out specific configuration information according to configuration-direct, concrete storage location can be the database of specifying, such as, described collection item mode information and contrast judgement mode information can be stored in relational database such as Mysql database.But described known comparison data is stored in fabric memory such as Hbase database.
Step 103, by described data acquisition item mode information, described contrast judgement mode information, carry out corresponding binding with the scheduler task for data mart modeling, and generate corresponding monitor task.
The scheduler task of described user data processing is the task of data warehouse being carried out to read-write operation, and the execution of this task can cause the change of Data Warehouse.And object of the present invention is exactly the data wanting scheduler task described in intelligent monitoring to change whether occurs exception.In aforementioned two steps, the contrast judgement mode information of the different data acquisition item mode information of configuration store and correspondence thereof can be distinguished, form the contrast judgement mode information of many data acquisition item mode information and correspondence thereof.Each scheduler task can bind one or one or more data acquisition item mode information and corresponding contrast judgement mode information thereof, each binding relationship is a corresponding monitor task again, this monitor task is used for according to the corresponding data in the data acquisition item mode information acquisition data warehouse of its correspondence, and according to the contrast judgement mode of correspondence, institute's image data and existing comparison data are compared, and judge comparison result, send alert notice information when comparison result exception.
Step 104, after described scheduler task performs and terminates, run the monitor task of its correspondence, this monitor task is according to the corresponding data in the data acquisition item mode information acquisition data warehouse of its correspondence, according to the contrast judgement mode of correspondence, institute's image data and existing comparison data are compared, and judge comparison result, send alert notice information when comparison result exception.
Concrete, described monitor task is according to the corresponding data in the data acquisition item mode information acquisition data warehouse of its correspondence, specifically comprise: in data warehouse, broker module is set, described data acquisition item mode information transmission is given described broker module by described monitor task, the monitoring actuator in this broker module notification data warehouse performs the query statement in described data acquisition item mode information, thus inquires corresponding data and return to monitor task as image data.
In an advantageous embodiment, method of the present invention can further include: according to gathered data, modifies to the image data item mode information of described storage and described contrast judgement mode information.
In addition, the present invention can also show that when described monitor task runs task run checks interface, for checking the running status of described monitor task, the log information of described monitor task can also be recorded further, and log information is kept in described Hbase database.
In another preferred embodiment, the present invention can also after described scheduler task execution terminates, before running corresponding monitor task, comprise further: the running status judging described data warehouse, if the running status of data warehouse is higher than the busy extent of specifying, then delay (such as delaying 2 hours) and run described monitor task, otherwise run described monitor task.By above-mentioned monitor task to rule control working time, solve the problem of warehouse cycle of operation load too high.
Corresponding with said method, the invention also discloses a kind of data monitoring system of data warehouse, Fig. 2 is a kind of component relationship figure of the data monitoring system of data warehouse of the present invention.See Fig. 2, this system comprises:
First configuration module 201, for the collection item mode information of configuration store from data warehouse image data;
Second configuration module 202, the contrast judgement mode information of the data gathered for configuration store and existing comparison data, stores alert notice information when occurring abnormal; Described existing comparison data comprises: historical data or dimension table data;
Binding module 203, for by described data acquisition item mode information, described contrast judgement mode information, carries out corresponding binding with the scheduler task for data mart modeling, and generates corresponding monitor task;
Monitoring module 204, for after described scheduler task execution terminates, run corresponding monitor task, this monitor task is according to the corresponding data in the data acquisition item mode information acquisition data warehouse of its correspondence, according to the contrast judgement mode of correspondence, institute's image data and existing comparison data are compared, and judges comparison result, send alert notice information when comparison result exception.
Concrete, described collection item mode information specifically can be stored in relational database such as Mysql database by described first configuration module 201, and described contrast judgement mode information specifically can be stored in relational database as in Mysql database by described second configuration module 202.And described known comparison data is stored in fabric memory as in Hbase database, described monitoring module 204 can read corresponding consistent comparison data from this Hbase database.Simultaneously, described monitoring module 204 can also show that when described monitor task runs task run checks interface, for checking the running status of described monitor task, the log information of described monitor task can also be recorded further, and log information is kept in described Hbase database.
In other embodiment, gathered data feedback can also be given described first configuration module 201 and the second configuration module 202 by described monitoring module 204, described first configuration module 201, according to gathered data, is modified to the image data item mode information of described storage; Described second configuration module 202, according to gathered data, is modified to the contrast judgement mode information of described storage.
Described dispatching system is the system of data warehouse being carried out to read-write operation, wherein run and have timer and scheduler task, the time operation rule of the warehouse broker module in control data warehouse, control the concurrent running of warehouse broker module, by rule working time of the rule control warehouse broker module of contrigger.
Described data warehouse is the core system that data store, store all historical datas, the Agent controlled by dispatching system and supervisory system is run in warehouse environment (HIVE), and draw result of calculation, realize comparing with historical results, return comparison result to described supervisory system.Concrete, described data acquisition item mode information transmission is given described broker module by described monitor task, the monitoring actuator in this broker module notification data warehouse performs the query statement in described data acquisition item mode information, inquires corresponding data and returns to monitor task as image data.The present invention is based on the warehouse broker module of Hive client, realize monitor task and data producer (scheduler task) can share Hive client.
Described Mysql database specifically can store the relevant all configuration informations of data quality monitoring, comprises and gathers item configuration information, regularization term configuration information, reports to the police and notification of contacts configuration information, warehouse metadata information etc.Described Hbase database can be used for storing history detailed data, and the detailed data that warehouse calculates is stored in the medium-term and long-term preservation of Hbase, and the snapshots of web pages configured together is also stored into Hbase and preserves for a long time.
In addition, each functional module in each embodiment of the present invention can be integrated in a processing unit, also can be that the independent physics of modules exists, also can two or more module integrations in a unit.Above-mentioned integrated unit both can adopt the form of hardware to realize, and the form of SFU software functional unit also can be adopted to realize.The functional module of described each embodiment can be positioned at a terminal or network node, or also can be distributed on multiple terminal or network node.
In addition, each embodiment of the present invention can be realized by the data processor performed as computing machine by data processing equipment.Obviously, data processor constitutes the present invention.In addition, program is read out storage medium or memory device (as hard disk and or internal memory) the middle execution by program being installed or copied to data processing equipment by direct by the data processor be usually stored in a storage medium.Therefore, such storage medium also constitutes the present invention.Storage medium can use the recording mode of any type, such as paper storage medium (as paper tape etc.), magnetic storage medium (as floppy disk, hard disk, flash memory etc.), optical storage media (as CD-ROM etc.), magnetic-optical storage medium (as MO etc.) etc.
Therefore the invention also discloses a kind of storage medium, wherein store data processor, this data processor is for performing any one embodiment of said method of the present invention.
In addition, method step of the present invention is except realizing with data processor, can also be realized by hardware, such as, can be realized by logic gate, switch, special IC (ASIC), programmable logic controller (PLC) and embedding microcontroller etc.Therefore this hardware that can realize the method for the invention also can form the present invention.
The foregoing is only preferred embodiment of the present invention, not in order to limit the present invention, within the spirit and principles in the present invention all, any amendment made, equivalent replacement, improvement etc., all should be included within the scope of protection of the invention.
Claims (10)
1. a data monitoring method for data warehouse, is characterized in that, comprising:
Configuration store is from the collection item mode information of data warehouse image data;
The contrast judgement mode information of the data that configuration store gathers and existing comparison data, stores alert notice information when occurring abnormal;
By described data acquisition item mode information, described contrast judgement mode information, carry out corresponding binding with the scheduler task for data mart modeling, and generate corresponding monitor task;
After described scheduler task execution terminates, run corresponding monitor task, this monitor task is according to the corresponding data in the data acquisition item mode information acquisition data warehouse of its correspondence, according to the contrast judgement mode of correspondence, institute's image data and existing comparison data are compared, and judge comparison result, send alert notice information when comparison result exception.
2. method according to claim 1, is characterized in that, the method comprises further:
According to gathered data, the image data item mode information of described storage and described contrast judgement mode information are modified.
3. method according to claim 1, is characterized in that, described known comparison data is stored in fabric memory, and described collection item mode information and contrast judgement mode information are stored in relational database.
4. method according to claim 1, is characterized in that, described monitor task, according to the corresponding data in the data acquisition item mode information acquisition data warehouse of its correspondence, specifically comprises:
In data warehouse, broker module is set, described data acquisition item mode information transmission is given described broker module by described monitor task, the monitoring actuator in this broker module notification data warehouse performs the query statement in described data acquisition item mode information, inquires corresponding data and returns to monitor task as image data.
5. method according to claim 1, it is characterized in that, it is described after described scheduler task execution terminates, before running corresponding monitor task, comprise further: the running status judging described data warehouse, if the running status of data warehouse is higher than the busy extent of specifying, then delays to run described monitor task, otherwise run described monitor task.
6. the method according to any one of claim 1-5, is characterized in that, described existing comparison data comprises: historical data or dimension table data.
7. a data monitoring system for data warehouse, is characterized in that, comprising:
First configuration module, for the collection item mode information of configuration store from data warehouse image data;
Second configuration module, the contrast judgement mode information of the data gathered for configuration store and existing comparison data, stores alert notice information when occurring abnormal;
Binding module, for by described data acquisition item mode information, described contrast judgement mode information, carries out corresponding binding with the scheduler task for data mart modeling, and generates corresponding monitor task;
Monitoring module, for after described scheduler task execution terminates, run corresponding monitor task, this monitor task is according to the corresponding data in the data acquisition item mode information acquisition data warehouse of its correspondence, according to the contrast judgement mode of correspondence, institute's image data and existing comparison data are compared, and judges comparison result, send alert notice information when comparison result exception.
8. system according to claim 7, it is characterized in that, gathered data feedback is given described first configuration module and the second configuration module by described monitoring module, and described first configuration module, according to gathered data, is modified to the image data item mode information of described storage; Described second configuration module, according to gathered data, is modified to the contrast judgement mode information of described storage.
9. system according to claim 7, is characterized in that, described known comparison data is stored in fabric memory; Described collection item mode information is stored in relational database by described first configuration module, and described contrast judgement mode information is stored in relational database by described second configuration module.
10. the system according to any one of claim 7-9, is characterized in that, described existing comparison data comprises: historical data or dimension table data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510312275.3A CN104866619A (en) | 2015-06-09 | 2015-06-09 | Data monitoring method and system for data warehouse |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510312275.3A CN104866619A (en) | 2015-06-09 | 2015-06-09 | Data monitoring method and system for data warehouse |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104866619A true CN104866619A (en) | 2015-08-26 |
Family
ID=53912445
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510312275.3A Pending CN104866619A (en) | 2015-06-09 | 2015-06-09 | Data monitoring method and system for data warehouse |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104866619A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105607983A (en) * | 2015-11-09 | 2016-05-25 | 北京京东尚科信息技术有限公司 | Data exception monitoring method and apparatus |
| CN105868036A (en) * | 2015-12-14 | 2016-08-17 | 乐视网信息技术（北京）股份有限公司 | Exception determination method and apparatus |
| CN105912605A (en) * | 2016-04-05 | 2016-08-31 | Tcl集团股份有限公司 | Statistical method and system for BI report |
| CN106371983A (en) * | 2016-08-31 | 2017-02-01 | 五八同城信息技术有限公司 | Method and device for alarming based on data development |
| CN107025224A (en) * | 2016-01-29 | 2017-08-08 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of monitor task operation |
| CN107748752A (en) * | 2017-09-05 | 2018-03-02 | 新智云数据服务有限公司 | A kind of data processing method and device |
| CN108255661A (en) * | 2016-12-29 | 2018-07-06 | 北京京东尚科信息技术有限公司 | A kind of method and system for realizing Hadoop cluster monitorings |
| CN108628669A (en) * | 2018-04-25 | 2018-10-09 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus of scheduling machine learning algorithm task |
| CN108959309A (en) * | 2017-05-23 | 2018-12-07 | 北京京东尚科信息技术有限公司 | The method and apparatus of data analysis |
| CN109376140A (en) * | 2018-08-24 | 2019-02-22 | 国网吉林省电力有限公司信息通信公司 | A kind of static resource automation collection method, system, equipment and storage medium |
| CN110837458A (en) * | 2019-11-08 | 2020-02-25 | 深圳市彬讯科技有限公司 | Data balance verification method, equipment and storage medium |
| CN112347198A (en) * | 2020-10-30 | 2021-02-09 | 广西电网有限责任公司南宁供电局 | Data rapid processing comparison system and method |
| CN113128943A (en) * | 2019-12-30 | 2021-07-16 | 北京懿医云科技有限公司 | Data quality monitoring method and device, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101105793A (en) * | 2006-07-11 | 2008-01-16 | 阿里巴巴公司 | Data processing method and system of data library |
| CN101576893A (en) * | 2008-05-09 | 2009-11-11 | 北京世纪拓远软件科技发展有限公司 | Method and system for analyzing data quality |
| CN102339288A (en) * | 2010-07-21 | 2012-02-01 | 中国移动通信集团辽宁有限公司 | Method and device for detecting abnormal data of data warehouse |
| CN102609537A (en) * | 2012-02-17 | 2012-07-25 | 广东电网公司电力科学研究院 | Data quality audit method based on database schema |
| CN102708149A (en) * | 2012-04-01 | 2012-10-03 | 河海大学 | Data quality management method and system |
-
2015
- 2015-06-09 CN CN201510312275.3A patent/CN104866619A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101105793A (en) * | 2006-07-11 | 2008-01-16 | 阿里巴巴公司 | Data processing method and system of data library |
| CN101576893A (en) * | 2008-05-09 | 2009-11-11 | 北京世纪拓远软件科技发展有限公司 | Method and system for analyzing data quality |
| CN102339288A (en) * | 2010-07-21 | 2012-02-01 | 中国移动通信集团辽宁有限公司 | Method and device for detecting abnormal data of data warehouse |
| CN102609537A (en) * | 2012-02-17 | 2012-07-25 | 广东电网公司电力科学研究院 | Data quality audit method based on database schema |
| CN102708149A (en) * | 2012-04-01 | 2012-10-03 | 河海大学 | Data quality management method and system |
Non-Patent Citations (1)
| Title |
|---|
| 吴毅敏 等: "数据仓库监控技术研究", 《计算机工程》 * |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105607983B (en) * | 2015-11-09 | 2018-08-10 | 北京京东尚科信息技术有限公司 | Data exception monitoring method and device |
| CN105607983A (en) * | 2015-11-09 | 2016-05-25 | 北京京东尚科信息技术有限公司 | Data exception monitoring method and apparatus |
| CN105868036A (en) * | 2015-12-14 | 2016-08-17 | 乐视网信息技术（北京）股份有限公司 | Exception determination method and apparatus |
| CN107025224B (en) * | 2016-01-29 | 2020-10-16 | 阿里巴巴集团控股有限公司 | Method and equipment for monitoring task operation |
| CN107025224A (en) * | 2016-01-29 | 2017-08-08 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of monitor task operation |
| CN105912605A (en) * | 2016-04-05 | 2016-08-31 | Tcl集团股份有限公司 | Statistical method and system for BI report |
| CN106371983A (en) * | 2016-08-31 | 2017-02-01 | 五八同城信息技术有限公司 | Method and device for alarming based on data development |
| CN108255661A (en) * | 2016-12-29 | 2018-07-06 | 北京京东尚科信息技术有限公司 | A kind of method and system for realizing Hadoop cluster monitorings |
| CN108959309A (en) * | 2017-05-23 | 2018-12-07 | 北京京东尚科信息技术有限公司 | The method and apparatus of data analysis |
| CN108959309B (en) * | 2017-05-23 | 2021-05-25 | 北京京东尚科信息技术有限公司 | Method and device for data analysis |
| CN107748752A (en) * | 2017-09-05 | 2018-03-02 | 新智云数据服务有限公司 | A kind of data processing method and device |
| CN108628669A (en) * | 2018-04-25 | 2018-10-09 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus of scheduling machine learning algorithm task |
| CN109376140A (en) * | 2018-08-24 | 2019-02-22 | 国网吉林省电力有限公司信息通信公司 | A kind of static resource automation collection method, system, equipment and storage medium |
| CN110837458A (en) * | 2019-11-08 | 2020-02-25 | 深圳市彬讯科技有限公司 | Data balance verification method, equipment and storage medium |
| CN110837458B (en) * | 2019-11-08 | 2024-03-29 | 土巴兔集团股份有限公司 | Method, equipment and storage medium for data balance verification |
| CN113128943A (en) * | 2019-12-30 | 2021-07-16 | 北京懿医云科技有限公司 | Data quality monitoring method and device, electronic equipment and storage medium |
| CN113128943B (en) * | 2019-12-30 | 2023-12-05 | 北京懿医云科技有限公司 | Data quality monitoring method, device, electronic equipment and storage medium |
| CN112347198A (en) * | 2020-10-30 | 2021-02-09 | 广西电网有限责任公司南宁供电局 | Data rapid processing comparison system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104866619A (en) | Data monitoring method and system for data warehouse | |
| US20210240736A1 (en) | Method and Apparatus for Monitoring an In-memory Computer System | |
| US10936479B2 (en) | Pluggable fault detection tests for data pipelines | |
| KR102033971B1 (en) | Data quality analysis | |
| US9946989B2 (en) | Management and notification of object model changes | |
| CN103425584B (en) | Based on the large-scale application regression test information processing method of Java bytecode | |
| KR20150132858A (en) | System for metadata management | |
| US10110419B2 (en) | Alarm to event tracing | |
| AU2014216441B2 (en) | Queue monitoring and visualization | |
| CN108681556A (en) | The access method and its system of distributed instruction numeric field data | |
| EP3161641A1 (en) | Methods and apparatuses for automated testing of streaming applications using mapreduce-like middleware | |
| US10223232B2 (en) | System and method for recording the beginning and ending of job level activity in a mainframe computing environment | |
| CN110865806B (en) | Code processing method, device, server and storage medium | |
| Scherbaum et al. | Spline: Spark lineage, not only for the banking industry | |
| US20220083320A1 (en) | Maintenance of computing devices | |
| CN112835779A (en) | Test case determination method and device and computer equipment | |
| US11537963B2 (en) | Systems and methods for decommissioning business intelligence artifacts | |
| CN108140047B (en) | Data processing apparatus and method, and data container structure | |
| CN113934595A (en) | Data analysis method and system, storage medium and electronic terminal | |
| Kumar | Software Engineering for Big Data Systems | |
| CN112347180B (en) | Data pushing method and electronic equipment | |
| CN116185771A (en) | Data processing method, device, electronic equipment and storage medium | |
| CN118245277A (en) | Method, equipment and medium for recovering human resource system data | |
| CN117911153A (en) | Service data processing method, device and equipment based on attribute change | |
| CN117391819A (en) | Data processing method, device, computing equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| EXSB | Decision made by sipo to initiate substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20150826 |
|
| RJ01 | Rejection of invention patent application after publication |