CN107958049A - A kind of quality of data checking and administration system - Google Patents
A kind of quality of data checking and administration system Download PDFInfo
- Publication number
- CN107958049A CN107958049A CN201711212498.8A CN201711212498A CN107958049A CN 107958049 A CN107958049 A CN 107958049A CN 201711212498 A CN201711212498 A CN 201711212498A CN 107958049 A CN107958049 A CN 107958049A
- Authority
- CN
- China
- Prior art keywords
- data
- quality
- task
- rule
- monitoring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a kind of quality of data checking and administration system, it is related to Data management and utilizaition technical field.The system, can to data plan, obtain, processing, storage, shared, maintenances, application, extinction life cycle each stage in may initiation Various types of data quality problems, it is identified, measures, monitoring, a series of management activity such as early warning, realize the global control of the quality of data, it can quickly connect data source at the same time, a variety of quality of data inspection rules are provided, and quality of data report is provided with a variety of visual means, data are carried out continuing monitoring and improve quality problems.And the management level by improving tissue further improves the quality of data, with the use of system, on the basis of pinpointing the problems in time, solve the problems, such as, provide an effective methods and techniques basis, can help the stakeholder of government department and IT departments carry out cooperation, it is ensured that all data it is complete, consistent, accurate and newest.
Description
Technical field
The present invention relates to Data management and utilizaition technical field, more particularly to a kind of quality of data checking and administration system.
Background technology
With cloud computing and the fast development of big data technology, a large amount of business and governments are internal, each branch all successively
Form respective mass data resource.In order to improve the management and control ability of data resource, user is highly desirable from these data
Analysis obtains the tendency information of business development, so as to provide correct decision support.Hair of the reasonability of decision-making to government
Open up significant, the decision-making of high quality depends on the data of high quality.This requires the data in data warehouse meet integrality,
The demands such as correctness, uniformity and reliability, provide believable data environment, as data warehouse for business and government decision-making
Data source, it is also necessary to which the data for ensureing to enter data center are high quality.However, the data of data center are from multiple industry
Integrated in business system, the time that these operation systems are built is different, and hardware environment is different, and database design is not also abided by
According to unified standard.Also tend to there are undesirable database schema design, or lack integrity constraint, or lack correct data
Examine logic so that the business datum of accumulation is integrated there are substantial amounts of quality problems, therefore to these isomeric data resources
When, easily there is " dirty data ", such as missing values, abnormal, inconsistent or repeated data, allow these dirty datas to enter data center
Various management costs are not only increased, the data warehouse more seriously established based on this, analyzes the decision-making gesture of generation
Heavy losses must be brought to government.Therefore various dirty datas should be eliminated, with ensure last data be it is correct, reliable,
It can reflect extension quality data exactly.
At present, the technical solution of quality of data checking and administration system is the Web applications built on the frame for build JavaEE,
The quality of data is established on this basis and veritifies module, is realized external Web service, is checked assembly module, is helped by regulation engine
Help each system to quickly generate to check script and implement the quality of data and check, checked by formulating, implementing the quality of data, each system of exposure
Data quality problem.Each system data quality fluctuation situation and the quality of data rule accounting analysis is persistently monitored, is periodically generated each
System-critical data quality report, grasps system data quality condition.The cleaning assembly and the quality of data provided with reference to system
Issue handling flow provides effective support for each system data increased quality.But this method there are it is following the problem of：
1st, poor expandability, quality of data rule reusability is weak, semi-automated data quality monitoring.
2nd, poor compatibility, the database species of support are less.
3rd, very flexible, it is impossible to self-defining data monitor task.
The content of the invention
It is an object of the invention to provide a kind of quality of data checking and administration system, so as to solve existing in the prior art
Foregoing problems.
To achieve these goals, the technical solution adopted by the present invention is as follows：
A kind of quality of data checking and administration system, including：
Rules administration module, for creating data monitoring rule, the monitoring rules include general foundation class measurement rule
Then and it is user-defined measurement rule, be additionally operable to be managed the monitoring rules, and to the monitoring rules into
Row imports and export operation；
Monitoring management module, including task management submodule, alarm management submodule and supervisor engine submodule, described
Business management submodule is used for the management that Life cycle is carried out to monitor task, including inquiry to monitor task, creates, repaiies
Change, delete, dispatch, terminate, suspend and manually perform, be additionally operable to disposition data source and corresponding data when monitor task creates
Monitoring rules；The alarm management submodule is used for during data monitoring tasks carrying, when discovery data are not inconsistent normally
When, problem data is preserved and issues the user with alarm；The supervisor engine submodule is used for according to data monitoring rule logarithm
Monitor task is performed according to source timing；
Statistical analysis module is used for data the problem of to found in monitor task implementation procedure and carries out statistics and analysis, and
The reason for according to statistical result to producing data quality problem, is analyzed；
System management module is used for data source control, system log and user management；
Interface module is used to provide a series of inside and outside interface, realizes the work compound with other systems.
Preferably, the foundation class measurement rule includes uniqueness, non-NULL, external key, codomain, form, uniformity, repeatability
And/or integrity checking rule, the user-defined measurement rule use self-defined SQL or regular expression.
Preferably, the task management submodule supports multiple data sources, including Oracle, Mysql, SqlServer,
DB2 and Access mainstream relevant databases, while support Excel, XML data file access.
Preferably, the task management submodule creates the monitor task, is implemented as follows：
Each data monitoring rule is converted into a corresponding task, while configures following content：The data of selection
Source, tables of data, field, task names, opening time, end time and cycle, and specific logic rules are added to task
Execute () method in, while configure the schedule time list of good each task.
Preferably, scheduling of the task management submodule to the monitor task, is implemented as follows：
In implementation procedure, the initial time according to specified in timetable, is called in corresponding task using scheduler
Execute () method, after the completion of method execution, can be called, until the knot of the task again after a cycle terminates
Untill the beam time reaches.
Preferably, the supervisor engine submodule performs monitor task to data source timing, carries out as follows real
Apply：
Each data monitoring rule is added in supervisor engine submodule and is performed, supervisor engine submodule utilizes
RESTful frameworks are realized, the implementation procedure of each data monitoring rule are packaged into rest interfaces, data monitoring rule performed
Data source connection and corresponding parameter are provided in journey, obtain the result that data monitoring rule performs.
The beneficial effects of the invention are as follows：Quality of data checking and administration system provided in an embodiment of the present invention, can be to data
Plan, obtain, processing, storage, shared, maintenances, application, extinction life cycle each stage in may initiation all kinds of numbers
According to quality problems, it is identified, measures, monitoring, a series of management activitys such as early warning, realizing the global control of the quality of data,
It can quickly connect data source at the same time, there is provided a variety of quality of data inspection rules, and provide data matter with a variety of visual means
Data are carried out continuing monitoring and improve quality problems by amount report.And the management level by improving tissue is further
The quality of data is improved, with the use of system, on the basis of pinpointing the problems in time, solve the problems, such as, there is provided row has
The methods and techniques basis of effect, can help the stakeholder of government department and IT departments to carry out cooperation, so that generally in depth
Develop Data quality control, it is ensured that all data it is complete, consistent, accurate and newest, regardless of whether data are located at where.
Brief description of the drawings
Fig. 1 is the logical construction schematic diagram of quality of data checking and administration system；
Fig. 2 is the Technical Architecture figure of quality of data checking and administration system；
Fig. 3 is quality of data Rulemaking process interface figure；
Fig. 4 is to create monitor task process interface figure；
Fig. 5 is that process interface figure is detected and monitored to the quality of data；
Fig. 6 is quality of data statistics and analysis process interface figure.
Embodiment
In order to make the purpose , technical scheme and advantage of the present invention be clearer, below in conjunction with attached drawing, to the present invention into
Row is further described.It should be appreciated that the specific embodiments described herein are not used to only to explain the present invention
Limit the present invention.
As shown in Figure 1, an embodiment of the present invention provides a kind of quality of data checking and administration system, including：
Rules administration module, for creating data monitoring rule, the monitoring rules include general foundation class measurement rule
Then and it is user-defined measurement rule, be additionally operable to be managed the monitoring rules, and to the monitoring rules into
Row imports and export operation；
Monitoring management module, including task management submodule, alarm management submodule and supervisor engine submodule, described
Business management submodule is used for the management that Life cycle is carried out to monitor task, including inquiry to monitor task, creates, repaiies
Change, delete, dispatch, terminate, suspend and manually perform, be additionally operable to disposition data source and corresponding data when monitor task creates
Monitoring rules；The alarm management submodule is used for during data monitoring tasks carrying, when discovery data are not inconsistent normally
When, problem data is preserved and issues the user with alarm；The supervisor engine submodule is used for according to data monitoring rule logarithm
Monitor task is performed according to source timing；
Statistical analysis module is used for data the problem of to found in monitor task implementation procedure and carries out statistics and analysis, and
The reason for according to statistical result to producing data quality problem, is analyzed；
System management module is used for data source control, system log and user management；
Interface module is used to provide a series of inside and outside interface, realizes the work compound with other systems.
Wherein, the foundation class measurement rule includes uniqueness, non-NULL, external key, codomain, form, uniformity, repeatability
And/or integrity checking rule, the user-defined measurement rule use self-defined SQL or regular expression.
In rules administration module, user can create new data monitoring rule, can both create general foundation class
Measurement rule, User Defined measurement rule can also be created according to different business demands, then carries out pipe to these rules
Reason, and can be imported and derived operation.Data rule is the key factor of determination data quality, is created in data rule
When building, in addition to the general-purpose attribute of data to be considered in itself, specific business demand and service logic are further accounted for, could be accurate
It was found that data quality problem and improving, foundation class measurement rule, including uniqueness, non-NULL, external key, value are provided in the present invention
Domain, form, uniformity, repeatability, integrality etc. check rule, while provide custom rule, can input self-defined SQL or just
Then expression formula.
Monitoring management module includes the function of task management and alarm management, and contains a data quality monitoring and draw
Hold up.Task management functions provide the management of the Life cycle of monitor task, inquiry, establishment, modification comprising monitor task,
The function such as delete, dispatch, terminate, suspend and manually perform.Data source and corresponding number are needed to configure when monitor task creates
According to contents such as rules, then periodically performed in supervisor engine.During data monitoring tasks carrying, when discovery data are not inconsistent
When normally, such as the problems such as data uniqueness, format error, problem data is preserved and issues the user with alarm, by user
Different modes is selected to handle problem data.
The problem of statistical analysis module finds data quality management data carry out statistics and analysis, according to statistical result
The reason for producing data quality problem, is analyzed, and further lifts the quality of data, authority data quality management, is continuously improved
Data application is horizontal.Data statistic analysis result is classified according to different monitor tasks, for example, having created for task and
Rule carries out filtering classification, facilitates user individually to check the quality of data specifying information of each monitor task.
System management module provides the system-level management functions such as data source control, system log and user management.
Interface module provides a series of inside and outside interface, convenient and other systems work compound.
As shown in Fig. 2, the Technical Architecture of quality of data checking and administration system provided in an embodiment of the present invention uses B/S frameworks
Hierarchical design is carried out, is divided into data Layer, logical layer and presentation layer according to MVC models.Realize that interface is isolated between at all levels, subtract
The close coupling of few system, facilitates later function modification or Function Extension.Utilize the encapsulation of Object-Oriented Programming, succession, more
The characteristic of state, function and actual realization are kept apart, and convenient realized to function is adjusted, and reduction realizes logic change to function
The influence of logical code.
Core of the logical layer as whole Technical Architecture, including reminder announced management, task scheduling, data permission control,
Multi-data source support, quality of data inspection engine, log processing, abnormality processing, data veritification, transaction management and data cleansing etc.
Function.
Multi-data source support is the basis of data monitoring, and in the embodiment of the present invention, the task management submodule can prop up
Multiple data sources, including Oracle, Mysql, SqlServer, DB2 and Access mainstream relevant database are held, is supported at the same time
Excel, XML data file access.
In the present embodiment, the task management submodule creates the monitor task, can carry out reality as follows
Apply：
Each data monitoring rule is converted into a corresponding task, while configures following content：The data of selection
Source, tables of data, field, task names, opening time, end time and cycle, and specific logic rules are added to task
Execute () method in, while configure the schedule time list of good each task.
Scheduling of the task management submodule to the monitor task, can be implemented as follows：
In implementation procedure, the initial time according to specified in timetable, is called in corresponding task using scheduler
Execute () method, after the completion of method execution, can be called, until the knot of the task again after a cycle terminates
Untill the beam time reaches.
In the present embodiment, the supervisor engine submodule performs monitor task to data source timing, can be according to such as lower section
Method is implemented：
Each data monitoring rule is added in supervisor engine submodule and is performed, supervisor engine submodule utilizes
RESTful frameworks are realized, the implementation procedure of each data monitoring rule are packaged into rest interfaces, data monitoring rule performed
Data source connection and corresponding parameter are provided in journey, obtain the result that data monitoring rule performs.
Using system provided in an embodiment of the present invention, the process of quality of data checking and administration is carried out, can be according to following step
Suddenly implemented：
1st, quality of data rule is formulated
New data monitoring rule is created, can both create general foundation class measurement rule, can also be according to different
Business demand creates User Defined measurement rule.Data rule is the key factor of determination data quality, in data rule
During establishment, in addition to the general-purpose attribute of data to be considered in itself, it is also contemplated that specific business demand and service logic, Cai Nengzhun
Really find data quality problem and improve.Its operating process interface is as shown in Figure 3.
2nd, monitor task planning is created
New monitor task is created, user chooses data source and data according to business demand, and monitoring rules are matched somebody with somebody
Put., it is necessary to data source of the content of configuration including selection, tables of data, field, rule, task names, unlatching during establishment
The contents such as time, end time and cycle (frequency).The monitor task newly created is in the state being not carried out.Its operating process circle
Face is as shown in Figure 4.
3rd, the quality of data is detected and monitored
Supervisor engine is the engine of data quality management system, is responsible for execution monitor task and dispatches and produce monitoring knot
Fruit.The time timing that supervisor engine is set by scheduler program according to monitor task performs, and is checked according to the rule of monitor task
Data, the data record not being inconsistent normally is got off.Supervisor engine is the core of data monitoring, when a data monitor task is opened
After beginning execution is placed in supervisor engine, supervisor engine can check at the beginning of the task scheduling between, the end time and perform the cycle,
It can start the task after execution time point is reached and load the data in data source, then read rule and matches somebody with somebody confidence
Breath, for each data rule in monitor task, scans source data and tests or veritify, when data do not meet number one by one
During according to rule, the specifying information of problem data is recorded, and user is notified according to the alarm mode of setting.Its operating process interface
As shown in Figure 5.
4th, quality of data statistics and analysis
The data progress of the problem of to found in data quality monitoring statistics and analysis, according to statistical result to producing data
The reason for quality problems, is analyzed, and quality report is generated, so as to further lift the quality of data, authority data quality pipe
Reason, it is horizontal to be continuously improved data application.Data statistic analysis result can classify according to different monitor tasks, convenient to use
The quality of data specifying information of each monitor task is individually checked at family.Its operating process interface is as shown in Figure 6.
By using above-mentioned technical proposal disclosed by the invention, following beneficial effect has been obtained：The embodiment of the present invention carries
The quality of data checking and administration system of confession, can be to data in plan, acquisition, processing, storage, shared, maintenance, application, extinction
The Various types of data quality problems that may trigger in each stage of life cycle, are identified, measure, monitoring, a system such as early warning
Row management activity, realizes the global control of the quality of data, while can quickly connect data source, there is provided a variety of qualities of data are checked
Rule is looked into, and quality of data report is provided with a variety of visual means, data are carried out continuing monitoring and improve quality problems.And
The quality of data is further improved by the management level for improving tissue, with the use of system, pinpoint the problems in time,
On the basis of solving the problems, such as, there is provided an effective methods and techniques basis, can help government department and IT departments
Stakeholder carry out cooperation so that universal in depth Develop Data quality control, it is ensured that all data it is complete, consistent, accurate
It is really and newest, regardless of whether data are located at where.
The above is only the preferred embodiment of the present invention, it is noted that for the ordinary skill people of the art
For member, various improvements and modifications may be made without departing from the principle of the present invention, these improvements and modifications also should
Depending on protection scope of the present invention.
Claims (6)
- A kind of 1. quality of data checking and administration system, it is characterised in that including：Rules administration module, for creating data monitoring rule, the monitoring rules include general foundation class measurement rule with And user-defined measurement rule, it is additionally operable to be managed the monitoring rules, and lead the monitoring rules Enter and export operation；Monitoring management module, including task management submodule, alarm management submodule and supervisor engine submodule, the task pipe Manage the management that submodule is used to carry out monitor task Life cycle, including inquiry to monitor task, establishment, change, delete Remove, dispatch, terminate, suspend and manually perform, be additionally operable to disposition data source and corresponding data monitoring when monitor task creates Rule；The alarm management submodule is used for during data monitoring tasks carrying, will when finding that data are not inconsistent normally Problem data preserves and issues the user with alarm；The supervisor engine submodule is used to determine data source according to data monitoring rule Shi Zhihang monitor tasks；The problem of statistical analysis module is for found in monitor task implementation procedure data progress statistics and analysis, and according to The reason for statistical result is to producing data quality problem is analyzed；System management module is used for data source control, system log and user management；Interface module is used to provide a series of inside and outside interface, realizes the work compound with other systems.
- 2. quality of data checking and administration system according to claim 1, it is characterised in that the foundation class measurement rule bag Include uniqueness, non-NULL, external key, codomain, form, uniformity, repeatability and/or integrity checking rule, the User Defined Measurement rule use self-defined SQL or regular expression.
- 3. quality of data checking and administration system according to claim 1, it is characterised in that the task management submodule branch Multiple data sources, including Oracle, Mysql, SqlServer, DB2 and Access mainstream relevant database are held, is supported at the same time Excel, XML data file access.
- 4. quality of data checking and administration system according to claim 1, it is characterised in that the task management submodule wound The monitor task is built, is implemented as follows：Each data monitoring rule is converted into a corresponding task, while configures following content：The data source of selection, number It is added to task according to table, field, task names, opening time, end time and cycle, and by specific logic rules In execute () method, while configure the schedule time list of good each task.
- 5. quality of data checking and administration system according to claim 4, it is characterised in that the task management submodule pair The scheduling of the monitor task, is implemented as follows：In implementation procedure, the initial time according to specified in timetable, the execute () in corresponding task is called using scheduler Method, after the completion of method execution, can again be called after a cycle terminates, be reached until the end time of the task Untill.
- 6. quality of data checking and administration system according to claim 1, it is characterised in that the supervisor engine submodule pair Data source timing performs monitor task, is implemented as follows：Each data monitoring rule is added in supervisor engine submodule and is performed, supervisor engine submodule utilizes RESTful Framework is realized, the implementation procedure of each data monitoring rule is packaged into rest interfaces, is carried in data monitoring rule implementation procedure For data source connection and corresponding parameter, the result that data monitoring rule performs is obtained.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711212498.8A CN107958049B (en) | 2017-11-28 | 2017-11-28 | Data quality inspection management system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711212498.8A CN107958049B (en) | 2017-11-28 | 2017-11-28 | Data quality inspection management system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107958049A true CN107958049A (en) | 2018-04-24 |
| CN107958049B CN107958049B (en) | 2021-09-14 |
Family
ID=61959474
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711212498.8A Active CN107958049B (en) | 2017-11-28 | 2017-11-28 | Data quality inspection management system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107958049B (en) |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109299083A (en) * | 2018-10-16 | 2019-02-01 | 全球能源互联网研究院有限公司 | A kind of data governing system |
| CN109993439A (en) * | 2019-04-02 | 2019-07-09 | 山东浪潮云信息技术有限公司 | A kind of quality determining method based on government data |
| CN110543483A (en) * | 2019-08-30 | 2019-12-06 | 北京百分点信息科技有限公司 | Data auditing method and device and electronic equipment |
| CN110704502A (en) * | 2019-11-20 | 2020-01-17 | 中电万维信息技术有限责任公司 | Componentized data quality checking method |
| CN111026804A (en) * | 2019-12-04 | 2020-04-17 | 深圳瑞力网科技有限公司 | Big data analysis intelligent service system based on semantics |
| CN111291990A (en) * | 2020-02-04 | 2020-06-16 | 浙江大华技术股份有限公司 | Quality monitoring processing method and device |
| CN111400288A (en) * | 2019-01-02 | 2020-07-10 | 中国移动通信有限公司研究院 | Data quality inspection method and system |
| CN111475495A (en) * | 2020-03-19 | 2020-07-31 | 深圳市酷开网络科技有限公司 | Mass analysis method, system and storage medium based on big data |
| CN112181957A (en) * | 2020-09-08 | 2021-01-05 | 支付宝(杭州)信息技术有限公司 | Archive data supervision processing method and device and electronic equipment |
| CN112231312A (en) * | 2020-10-29 | 2021-01-15 | 山东超越数控电子股份有限公司 | Data quality verification method based on process |
| CN112256782A (en) * | 2020-10-30 | 2021-01-22 | 内蒙古电力(集团)有限责任公司乌海超高压供电局 | Electric power big data processing system based on Hadoop |
| CN112306997A (en) * | 2019-07-23 | 2021-02-02 | 杭州中软安人网络通信股份有限公司 | Data quality management system |
| CN112948365A (en) * | 2021-03-04 | 2021-06-11 | 浪潮云信息技术股份公司 | Data quality detection method based on intelligent data element matching |
| CN113157676A (en) * | 2021-04-14 | 2021-07-23 | 联通(广东)产业互联网有限公司 | Data quality management method, system, device and storage medium |
| CN115292297A (en) * | 2022-06-29 | 2022-11-04 | 江苏昆山农村商业银行股份有限公司 | Method and system for constructing data quality monitoring rule of data warehouse |
| CN117240505A (en) * | 2023-08-15 | 2023-12-15 | 北京希嘉创智数据技术有限公司 | Early warning processing method and system based on data management platform |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2009046396A1 (en) * | 2007-10-04 | 2009-04-09 | Growers Express, Llc | Crop production, planning, management, tracking and reporting system and method |
| CN101719237A (en) * | 2009-12-09 | 2010-06-02 | 南京联创科技集团股份有限公司 | Data quality monitoring method based on full service indicator rule allocation |
| CN103473672A (en) * | 2013-09-30 | 2013-12-25 | 国家电网公司 | System, method and platform for auditing metadata quality of enterprise-level data center |
| CN104766151A (en) * | 2014-12-29 | 2015-07-08 | 国家电网公司 | Quality management and control method for electricity transaction data warehouses and management and control system thereof |
| CN105471671A (en) * | 2015-11-10 | 2016-04-06 | 国云科技股份有限公司 | Method for customizing monitoring rules of cloud platform resources |
| CN105574082A (en) * | 2015-12-08 | 2016-05-11 | 曙光信息产业(北京)有限公司 | Storm based stream processing method and system |
| CN106407391A (en) * | 2016-09-19 | 2017-02-15 | 北京集奥聚合科技有限公司 | A data quality monitoring method and system |
| CN107066500A (en) * | 2016-12-30 | 2017-08-18 | 江苏瑞中数据股份有限公司 | A kind of electrical network mass data quality indicator method based on PMS models |
| CN107092694A (en) * | 2017-04-25 | 2017-08-25 | 杭州数梦工场科技有限公司 | The inspection task creating method and device of the quality of data |
-
2017
- 2017-11-28 CN CN201711212498.8A patent/CN107958049B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2009046396A1 (en) * | 2007-10-04 | 2009-04-09 | Growers Express, Llc | Crop production, planning, management, tracking and reporting system and method |
| CN101719237A (en) * | 2009-12-09 | 2010-06-02 | 南京联创科技集团股份有限公司 | Data quality monitoring method based on full service indicator rule allocation |
| CN103473672A (en) * | 2013-09-30 | 2013-12-25 | 国家电网公司 | System, method and platform for auditing metadata quality of enterprise-level data center |
| CN104766151A (en) * | 2014-12-29 | 2015-07-08 | 国家电网公司 | Quality management and control method for electricity transaction data warehouses and management and control system thereof |
| CN105471671A (en) * | 2015-11-10 | 2016-04-06 | 国云科技股份有限公司 | Method for customizing monitoring rules of cloud platform resources |
| CN105574082A (en) * | 2015-12-08 | 2016-05-11 | 曙光信息产业(北京)有限公司 | Storm based stream processing method and system |
| CN106407391A (en) * | 2016-09-19 | 2017-02-15 | 北京集奥聚合科技有限公司 | A data quality monitoring method and system |
| CN107066500A (en) * | 2016-12-30 | 2017-08-18 | 江苏瑞中数据股份有限公司 | A kind of electrical network mass data quality indicator method based on PMS models |
| CN107092694A (en) * | 2017-04-25 | 2017-08-25 | 杭州数梦工场科技有限公司 | The inspection task creating method and device of the quality of data |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109299083A (en) * | 2018-10-16 | 2019-02-01 | 全球能源互联网研究院有限公司 | A kind of data governing system |
| CN111400288A (en) * | 2019-01-02 | 2020-07-10 | 中国移动通信有限公司研究院 | Data quality inspection method and system |
| CN109993439A (en) * | 2019-04-02 | 2019-07-09 | 山东浪潮云信息技术有限公司 | A kind of quality determining method based on government data |
| CN112306997A (en) * | 2019-07-23 | 2021-02-02 | 杭州中软安人网络通信股份有限公司 | Data quality management system |
| CN110543483A (en) * | 2019-08-30 | 2019-12-06 | 北京百分点信息科技有限公司 | Data auditing method and device and electronic equipment |
| CN110704502A (en) * | 2019-11-20 | 2020-01-17 | 中电万维信息技术有限责任公司 | Componentized data quality checking method |
| CN111026804A (en) * | 2019-12-04 | 2020-04-17 | 深圳瑞力网科技有限公司 | Big data analysis intelligent service system based on semantics |
| CN111291990B (en) * | 2020-02-04 | 2023-11-07 | 浙江大华技术股份有限公司 | Quality monitoring processing method and device |
| CN111291990A (en) * | 2020-02-04 | 2020-06-16 | 浙江大华技术股份有限公司 | Quality monitoring processing method and device |
| CN111475495A (en) * | 2020-03-19 | 2020-07-31 | 深圳市酷开网络科技有限公司 | Mass analysis method, system and storage medium based on big data |
| CN112181957A (en) * | 2020-09-08 | 2021-01-05 | 支付宝(杭州)信息技术有限公司 | Archive data supervision processing method and device and electronic equipment |
| CN112181957B (en) * | 2020-09-08 | 2024-04-12 | 支付宝(杭州)信息技术有限公司 | File data supervision processing method and device and electronic equipment |
| CN112231312A (en) * | 2020-10-29 | 2021-01-15 | 山东超越数控电子股份有限公司 | Data quality verification method based on process |
| CN112256782B (en) * | 2020-10-30 | 2024-03-29 | 内蒙古电力(集团)有限责任公司乌海超高压供电局 | Hadoop-based power big data processing system |
| CN112256782A (en) * | 2020-10-30 | 2021-01-22 | 内蒙古电力(集团)有限责任公司乌海超高压供电局 | Electric power big data processing system based on Hadoop |
| CN112948365A (en) * | 2021-03-04 | 2021-06-11 | 浪潮云信息技术股份公司 | Data quality detection method based on intelligent data element matching |
| CN113157676A (en) * | 2021-04-14 | 2021-07-23 | 联通(广东)产业互联网有限公司 | Data quality management method, system, device and storage medium |
| CN115292297A (en) * | 2022-06-29 | 2022-11-04 | 江苏昆山农村商业银行股份有限公司 | Method and system for constructing data quality monitoring rule of data warehouse |
| CN115292297B (en) * | 2022-06-29 | 2024-02-02 | 江苏昆山农村商业银行股份有限公司 | Method and system for constructing data quality monitoring rule of data warehouse |
| CN117240505B (en) * | 2023-08-15 | 2024-03-15 | 北京希嘉创智数据技术有限公司 | Early warning processing method and system based on data management platform |
| CN117240505A (en) * | 2023-08-15 | 2023-12-15 | 北京希嘉创智数据技术有限公司 | Early warning processing method and system based on data management platform |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107958049B (en) | 2021-09-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107958049A (en) | A kind of quality of data checking and administration system | |
| EP3513314B1 (en) | System for analysing data relationships to support query execution | |
| CN107844424B (en) | Model-based testing system and method | |
| Buijs | Mapping data sources to xes in a generic way | |
| US10013439B2 (en) | Automatic generation of instantiation rules to determine quality of data migration | |
| US9448915B2 (en) | Modular script designer for next generation testing system | |
| US8510603B2 (en) | Systems and methods providing an exception buffer to facilitate processing of event handler errors | |
| CN107810500A (en) | Data quality analysis | |
| US20140100910A1 (en) | System and Method for Audits with Automated Data Analysis | |
| US20210026753A1 (en) | Methods and systems for estimating process capacity | |
| CN111984709A (en) | Visual big data middle station-resource calling and algorithm | |
| CN108052542B (en) | Multidimensional data analysis method based on presto data | |
| EP3809277A1 (en) | Object-centric user system and graphical user interface | |
| CN106293891A (en) | Multidimensional investment target measure of supervision | |
| CN108897686A (en) | It is complete to record separately automated testing method and device | |
| CN106709026A (en) | Data processing method and data processing system | |
| US7992126B2 (en) | Apparatus and method for quantitatively measuring the balance within a balanced scorecard | |
| CN110427387A (en) | A kind of data consistency detection and device | |
| CN112100984B (en) | Data conversion method and system from EBOM to SBOM | |
| US20170270163A1 (en) | Data Information Framework | |
| CN113506098A (en) | Power plant metadata management system and method based on multi-source data | |
| CN108829578A (en) | A kind of CDR association backfill accuracy automated testing method and system | |
| Blanco et al. | Test adequacy evaluation for the user-database interaction: A specification-based approach | |
| Harrison | Software measurement: a decision-process approach | |
| CN113760681A (en) | Unified SQL (structured query language) -based multi-source heterogeneous data quality verification method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB03 | Change of inventor or designer information |
Inventor after: Zhang Siqi Inventor after: Xu Xuesong Inventor after: Guo Dandan Inventor after: Li Xiaopeng Inventor before: Xu Xuesong Inventor before: Zhang Siqi Inventor before: Guo Dandan Inventor before: Li Xiaopeng |
|
| CB03 | Change of inventor or designer information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |