CN109165212A - Big data real-time monitoring and auditing method - Google Patents
Big data real-time monitoring and auditing method Download PDFInfo
- Publication number
- CN109165212A CN109165212A CN201811007432.XA CN201811007432A CN109165212A CN 109165212 A CN109165212 A CN 109165212A CN 201811007432 A CN201811007432 A CN 201811007432A CN 109165212 A CN109165212 A CN 109165212A
- Authority
- CN
- China
- Prior art keywords
- data
- rule
- scrubbing
- checked
- result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 15
- 238000012544 monitoring process Methods 0.000 title claims abstract description 12
- 238000012550 audit Methods 0.000 claims abstract description 4
- 238000005201 scrubbing Methods 0.000 claims description 16
- 238000012986 modification Methods 0.000 claims description 4
- 230000004048 modification Effects 0.000 claims description 4
- 238000012545 processing Methods 0.000 abstract description 2
- 238000012423 maintenance Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Abstract
The invention provides a method for monitoring and auditing big data in real time, which belongs to the technical field of big data processing. The problem that the existing data monitoring audit is complex and tedious is solved.
Description
Technical field
The present invention relates to a kind of methods that big data processing technique more particularly to big data are monitored in real time and checked.
Background technique
Data monitoring and check be all Data Management Analysis vital task, by data monitoring and check can be timely
It was found that the truth of data, whether data content is reasonable, and whether size is abnormal etc..Existing data auditing is for some
Specific business establishes relevant regulations, and when modification or increase by one being needed to check rule, will pass through the process of exploitation test,
It takes time and effort, flexibility is poor, and maintenance cost is high.And as business datum amount is increasing, existing data auditing process
Need to consume excessive system resource and time.
Summary of the invention
In order to solve the above technical problems, the invention proposes a kind of methods big data real time monitoring and checked, it is intended to
Solve the problems, such as to check in existing data monitoring it is complicated, cumbersome, so that field maintenance person safeguards and uses, and in data
Measure it is increasing in the case where still can efficiently real-time monitoring data quality.
The technical scheme is that
A kind of method big data real time monitoring and checked,
It can simply be configured by interface and check rule, in data scrubbing link audit record stage as a result, will check that result is deposited
It stores up in database, is alerted according to checking that result triggers.
Data scrubbing server can increase according to data volume and be extended as a cluster.
Data auditing rule can not need modification code in interface configurations, and rule totally verifies rule comprising data
And single field checks rule.
Data auditing is completed at the same time in data scrubbing.
Data scrubbing is completed to alert according to result triggering is checked.
Specific implementation step is as follows:
S1 obtains the data information for needing to acquire according to interface, it is clear to be handed down to data by data distribution (manager) node
Manage server.
S2 checks rule by interface configuration data.
S3 records the total line number of file in data scrubbing and does not meet each line number for checking rule, and pass is recorded
It is in type database.
S4, ring ratio set threshold values triggering ring than fluctuation alarm with the total line number of time segment data.
S5 is obtained and is not met each ratio data for checking rule, setting threshold values difference trigger data quality alarm.
The beneficial effects of the invention are as follows
1) check rule by interface configurations, only need to simply select field whether can for it is empty, set value range, whether meet
Regular expression does not need modification code, is conducive to field maintenance person and uses.
2) data scrubbing server can extend easily as a cluster, adapt to the environment of big data, avoid due to number
The situation of system resource deficiency according to being gradually increased for amount.
3) file concrete condition is recorded when data scrubbing, does not need to inquire data auditing in cluster or database,
Can more real-time monitoring data quality, can also save inquiry data bring resource consumption.
Detailed description of the invention
Fig. 1 is workflow schematic diagram of the invention.
Specific embodiment
More detailed elaboration is carried out to the contents of the present invention below:
The method that a kind of big data of the invention is monitored in real time and checked, can simply be configured by interface and check rule,
Data scrubbing link audit record stage as a result, will check result store into database, according to check result triggering alarm.
As shown, specific implementation step is as follows:
S1 obtains the data information for needing to acquire according to interface, is handed down to data scrubbing by data distribution (manager) node
Server.
S2 checks rule by interface configuration data.
S3 records the total line number of file in data scrubbing and does not meet each line number for checking rule, and pass is recorded
It is in type database.
S4, ring ratio set threshold values triggering ring than fluctuation alarm with the total line number of time segment data.
S5 is obtained and is not met each ratio data for checking rule, setting threshold values difference trigger data quality alarm.
Claims (6)
1. a kind of method big data real time monitoring and checked, which is characterized in that
Rule is checked by interface configurations, in data scrubbing link audit record stage as a result, result storage will be checked to database
In, it is alerted according to checking that result triggers.
2. the method according to claim 1, wherein
Data scrubbing server can increase according to data volume and be extended as a cluster.
3. method according to claim 1 or 2, which is characterized in that
Data auditing rule can in interface configurations, do not need modification code, rule comprising data totally verify rule and
Single field checks rule.
4. according to the method described in claim 3, it is characterized in that,
Data auditing is completed at the same time in data scrubbing.
5. as claimed in claim 4, which is characterized in that
Data scrubbing is completed to alert according to result triggering is checked.
6. as claimed in claim 5, which is characterized in that
Specific implementation step is as follows:
S1 obtains the data information for needing to acquire according to interface, is handed down to data scrubbing server by data distributing node；
S2 checks rule by interface configuration data；
S3 records the total line number of file in data scrubbing and does not meet each line number for checking rule, and relationship type is recorded
In database；
S4, ring ratio set threshold values triggering ring than fluctuation alarm with the total line number of time segment data；
S5 is obtained and is not met each ratio data for checking rule, setting threshold values difference trigger data quality alarm.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811007432.XA CN109165212A (en) | 2018-08-31 | 2018-08-31 | Big data real-time monitoring and auditing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811007432.XA CN109165212A (en) | 2018-08-31 | 2018-08-31 | Big data real-time monitoring and auditing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109165212A true CN109165212A (en) | 2019-01-08 |
Family
ID=64893556
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811007432.XA Pending CN109165212A (en) | 2018-08-31 | 2018-08-31 | Big data real-time monitoring and auditing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109165212A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110543483A (en) * | 2019-08-30 | 2019-12-06 | 北京百分点信息科技有限公司 | Data auditing method and device and electronic equipment |
| CN113392099A (en) * | 2021-07-01 | 2021-09-14 | 苏州维众数据技术有限公司 | Automatic data cleaning method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030110172A1 (en) * | 2001-10-24 | 2003-06-12 | Daniel Selman | Data synchronization |
| US7272613B2 (en) * | 2000-10-26 | 2007-09-18 | Intel Corporation | Method and system for managing distributed content and related metadata |
| CN104915756A (en) * | 2015-05-22 | 2015-09-16 | 电信科学技术第五研究所 | Data consistency cloud auditing system and implementation method |
| CN106407216A (en) * | 2015-07-31 | 2017-02-15 | 国网能源研究院 | Clue tracing audition system developed on basis of semantic net construction path and construction method of clue tracing audition system |
| US20170046217A1 (en) * | 2015-08-12 | 2017-02-16 | Avekshaa Technologies Private Ltd | System and method for batch monitoring of performance data |
| CN108268549A (en) * | 2016-12-31 | 2018-07-10 | 中国移动通信集团湖北有限公司 | Data auditing system and method |
-
2018
- 2018-08-31 CN CN201811007432.XA patent/CN109165212A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7272613B2 (en) * | 2000-10-26 | 2007-09-18 | Intel Corporation | Method and system for managing distributed content and related metadata |
| US20030110172A1 (en) * | 2001-10-24 | 2003-06-12 | Daniel Selman | Data synchronization |
| CN104915756A (en) * | 2015-05-22 | 2015-09-16 | 电信科学技术第五研究所 | Data consistency cloud auditing system and implementation method |
| CN106407216A (en) * | 2015-07-31 | 2017-02-15 | 国网能源研究院 | Clue tracing audition system developed on basis of semantic net construction path and construction method of clue tracing audition system |
| US20170046217A1 (en) * | 2015-08-12 | 2017-02-16 | Avekshaa Technologies Private Ltd | System and method for batch monitoring of performance data |
| CN108268549A (en) * | 2016-12-31 | 2018-07-10 | 中国移动通信集团湖北有限公司 | Data auditing system and method |
Non-Patent Citations (2)
| Title |
|---|
| 宋雨等: "基于大数据平台的通信设备故障预警系统研究与实现", 《网络安全技术与应用》 * |
| 谌迅: "大数据资产管理系统的设计与实现", 《软件》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110543483A (en) * | 2019-08-30 | 2019-12-06 | 北京百分点信息科技有限公司 | Data auditing method and device and electronic equipment |
| CN113392099A (en) * | 2021-07-01 | 2021-09-14 | 苏州维众数据技术有限公司 | Automatic data cleaning method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106886485B (en) | System capacity analysis and prediction method and device | |
| CN107810500A (en) | Data quality analysis | |
| CN105868373B (en) | Method and device for processing key data of power business information system | |
| US20150170147A1 (en) | Automated transaction cancellation | |
| CN105631026A (en) | Security data analysis system | |
| JP2017536604A (en) | Using machine learning to identify non-technical losses | |
| CN112001586A (en) | Enterprise networking big data audit risk control architecture based on block chain consensus mechanism | |
| CN108255671A (en) | The monitoring of the application of computer system and aposematic mechanism | |
| CN106022617A (en) | Inspection control system based on marketing multi-system data center | |
| CN104991939A (en) | Transaction data monitoring method and system | |
| CN105872061A (en) | Server cluster management method, device and system | |
| CN105302697A (en) | Running state monitoring method and system of density data model database | |
| CN105574666A (en) | Method and device for evaluating credit level of enterprise based on key data modeling | |
| CN109902919A (en) | Server assets management method, device, equipment and readable storage medium storing program for executing | |
| CN114880405A (en) | Data lake-based data processing method and system | |
| CN109165212A (en) | Big data real-time monitoring and auditing method | |
| CN112883001A (en) | Data processing method, device and medium based on marketing and distribution through data visualization platform | |
| CN114443437A (en) | Alarm root cause output method, apparatus, device, medium, and program product | |
| CN109214649A (en) | A kind of analysis of economic index system based on big data | |
| CN113434575A (en) | Data attribution processing method and device based on data warehouse and storage medium | |
| CN107277143A (en) | A kind of resource matched management method and device | |
| CN104484277B (en) | Process data dynamic analysis device and its application method based on control point | |
| CN110827172A (en) | Wisdom water affairs cloud service platform | |
| CN110910061A (en) | Material management method, material management system, storage medium and electronic equipment | |
| CN115689713A (en) | Abnormal risk data processing method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190108 |