CN111767283B - Data system monitoring method and system - Google Patents
Data system monitoring method and system Download PDFInfo
- Publication number
- CN111767283B CN111767283B CN202010567262.1A CN202010567262A CN111767283B CN 111767283 B CN111767283 B CN 111767283B CN 202010567262 A CN202010567262 A CN 202010567262A CN 111767283 B CN111767283 B CN 111767283B
- Authority
- CN
- China
- Prior art keywords
- auditing
- main body
- query
- data
- field
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 33
- 238000000034 method Methods 0.000 title claims abstract description 31
- 230000009471 action Effects 0.000 claims abstract description 47
- 238000012216 screening Methods 0.000 claims abstract description 13
- 238000013439 planning Methods 0.000 claims description 14
- 238000010276 construction Methods 0.000 claims description 12
- 238000012550 audit Methods 0.000 claims description 11
- 238000012423 maintenance Methods 0.000 abstract description 16
- 238000004519 manufacturing process Methods 0.000 abstract description 5
- 230000009286 beneficial effect Effects 0.000 abstract description 4
- 238000004364 calculation method Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
- G06Q30/0201—Market modelling; Market analysis; Collecting market data
-
- G06Q50/40—
Abstract
The invention discloses a data system monitoring method and a system, which are used for acquiring inquiry results at set time and auditing the inquiry results according to a set auditing algorithm, so that the inquiry results and the auditing results can be automatically acquired by an automatic tool; the main body table fields are generalized and divided through main body categories, so that the main body table fields can be selected quickly, and the situation that system maintenance personnel miss part of the main body table fields to cause data errors or recalculation is avoided; by generalizing the action fields, the method is beneficial to monitoring or screening the data of the main body in different action links in the production flow, improves the fineness of monitoring and ensures the accuracy and the integrity of the statistics main body.
Description
Technical Field
The invention relates to the technical field of data monitoring, in particular to a data system monitoring method and system.
Background
With the development of information technology, the information quantity is increased in an explosive manner, the service data is various, the service data quantity or content is greatly increased, the number of butting systems or butting personnel is increased, the time point of the service data is not fixed, the maintenance of the data system is difficult, the problems of missing data or data maintenance errors and the like are caused, finally, calculation errors such as results and cost are caused, the calculation needs to be repeated, and the statistical data and the statistical period are seriously influenced.
With the increase of service data and types of the system, more time is not required to count the whole data system, only a part of statistics data which is focused on indexes is required, and the statistics period or time point of each service is also various, so that omission can be generated, incomplete data is caused, and new requirements are put forward on the maintenance and statistics data of the data system.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a vulnerability management method based on a database, which monitors data of main bodies in different action links in a production process and improves the monitoring fineness.
The invention discloses a data system monitoring method, which comprises the following steps: respectively establishing a main body category and an action category according to a main body table field and an action field of the data system; establishing a master-slave relationship between the main body category and the main body table field; constructing a data auditing plan table according to the action category, the main body category and the main body table field, wherein the auditing plan table further comprises an auditing algorithm and data query time; acquiring a query field according to the main body type of the data auditing plan table, the main body table field and the action type; acquiring a query result of a data system according to the query field in the data query time; and obtaining the auditing result according to the auditing algorithm and the query result.
Preferably, the invention further comprises a method for generating early warning information: generating early warning information according to the auditing result and the early warning range; generating warning information according to the warning information and the warning information template.
Preferably, the present invention further includes a method for sending alert information to an alert information recipient by one or a combination of the following means: short message push, instant chat tool send, mail send, bulletin push, or PC presentation.
Preferably, the audit plan includes a plurality of action categories, and the query field is constructed by a main body table field and a plurality of action categories.
Preferably, the audit plan further includes a screening time, and the query result of the screening time is obtained according to the query field.
Preferably, the data query time includes a query time point and a query period.
Preferably, the invention further comprises a monitoring method for auditing the planning table: acquiring data query time and audit planning quantity; acquiring the auditing plan number in unit time according to the data query time and the auditing plan number; generating early warning information according to the auditing planning quantity in unit time and a preset early warning range, and sending the early warning information to an early warning receiver.
Preferably, the data system comprises at least one database.
The invention also provides a data system monitoring system, which comprises a first construction module, a second construction module, an acquisition module, a query module and an auditing module, wherein the first construction module is used for respectively establishing a main body category and an action category according to a main body table field and an action field of the data system and for establishing a master-slave relationship between the main body category and the main body table field; the second construction module is used for constructing a data auditing plan table according to the action category, the main body category and the main body table field, and the auditing plan table also comprises an auditing algorithm and data query time; the acquisition module is used for acquiring the query field according to the main body type of the data auditing plan table, the main body table field and the action type; the query module is used for acquiring a query result of the data system according to the query field at the data query time; the auditing module is used for acquiring auditing results according to the auditing algorithm and the query results.
Preferably, the data system monitoring system further comprises an early warning module, wherein the early warning module is used for generating early warning information according to the auditing result and the early warning range and generating warning information according to the early warning information and the warning information template.
Compared with the prior art, the invention has the beneficial effects that:
1. the main body table fields are generalized and divided through main body categories, so that the main body table fields can be selected quickly, and the situation that system maintenance personnel miss part of the main body table fields to cause data errors or recalculation is avoided;
2. by generalizing the action fields, the method is beneficial to monitoring or screening the data of the main body on different action links in the production flow, and improves the monitoring fineness.
Drawings
FIG. 1 is a flow chart of a data system monitoring method of the present invention;
FIG. 2 is a flow chart of a method of monitoring an audit plan;
fig. 3 is a logic block diagram of a data system monitoring system of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention is described in further detail below with reference to the attached drawing figures:
a data system monitoring method, as shown in fig. 1, the method comprising:
step 101: and respectively establishing a main body category and an action category according to the main body table field and the action field of the data system. The main body table fields of the data system are various, the main body table fields are divided into a plurality of main body categories according to the contents of the main body table fields, such as interface categories, storage categories and main body categories, the action categories are divided according to the action fields, and the action categories reflect actions of the main body in the production flow links, such as acquisition, warehousing or inspection of the interface categories.
Step 102: and establishing a master-slave relationship between the main body category and the main body table field. The body table fields are divided under their corresponding body categories, and a secondary catalog form is created, such as a user revenue interface is created under the interface category.
Step 103: and constructing a data auditing plan table according to the action category, the main body category and the main body table field, wherein the auditing plan table further comprises an auditing algorithm and data query time. The main body category, the main body table field, the action category, the auditing algorithm and the data query time are stored under the auditing plan table, so that unified planning and maintenance are facilitated.
Step 104: and acquiring a query field according to the main body type of the data auditing plan table, the main body table field and the action type. The main body table field and the action category group form key elements of the query field, and the query field is constructed according to the selected main body category, main body table field and action category.
Step 105: and in the data query time, acquiring a query result of the data system according to the query field.
Step 106: and obtaining an auditing result according to the auditing algorithm and the query result.
The query result is obtained at a set time, and is checked according to a set checking algorithm, so that the query result and the checking result can be automatically obtained through an automatic tool.
The main body table fields are generalized and divided through main body categories, so that the main body table fields can be selected quickly, the situation that system maintenance personnel miss part of main body table fields to cause data errors or recalculation is avoided, and the accuracy and the integrity of the statistics main body are ensured; by generalizing the action fields, the method is beneficial to monitoring or screening the data of the main body on different action links in the production flow, and improves the monitoring fineness.
The invention can also comprise a method for generating early warning information:
step 111: and generating early warning information according to the auditing result and the early warning range.
Step 112: generating warning information according to the warning information and the warning information template. And different warning information templates can be adopted to generate different warning information according to the fact that the auditing result is in different warning ranges.
The invention may also include a method for sending alert information, step 113: the warning information is sent to a warning information receiver through one or a combination of the following modes: short message pushing, instant chat tool sending, mail sending, bulletin pushing and PC displaying.
The early warning range and the warning information receiver can be stored in the data auditing planning table, so that unified scheduling and planning are facilitated.
The audit plan may include a plurality of action categories through which a query field is constructed. Multiple action categories of the subject are monitored.
The audit plan may further include a screening time, and query results within the screening time are obtained according to the query field.
The data query time may include a query time point and a query period.
Example 1
And counting the users who do not send short messages, use the flow and talk within three months, and sending the short messages to remind a client manager. Type of body: business indexes; body table field: a mobile user; action category: sending short messages, using flow or talking; screening time: within three months; auditing algorithm: a mobile user sending short message volume, using flow volume and call volume with the sum equal to zero; query time: 09:00:00, inquiry cycle: monthly No. 5; warning information receiver: a customer manager; the warning information mode is as follows: and (5) short messages.
The method comprises the steps that the mobile user short message quantity, the mobile user flow and the mobile user call quantity in three months are obtained in month No. 5, the short message quantity, the mobile user flow and the mobile user call quantity are checked, and a checking result is obtained: and the mobile user sends the auditing result to a client manager in a short message mode.
Example 2
And counting the number of the concentrated disassembly machines of the broadband. Type of body: business indexes; a main body table field, a broadband centralized disassembly number; screening time: the previous day; auditing algorithm: the number of the broadband centralized disassembly machines is more than 80; query time: 09:00:00; query period: daily; warning information receiver: a disassembling worker and a broadband responsible person; the warning information mode is as follows: and (5) short messages.
Example 3
And monitoring the integrity of the maintenance of the checking data. Type of body: storing a process; a main body table field, data maintenance; action type: executing; screening time: last month; auditing algorithm: the execution number of the data maintenance is less than 80% of the average value of the execution number of the data maintenance in the past six months; query time: 09:00:00; query period: monthly No. 20; warning information receiver: a data maintainer; the warning information mode is as follows: e, mail.
That is, when the execution number of the previous month data maintenance is less than 80% of the average value of the past six month data maintenance number, a warning mail is sent to the data maintenance personnel. In the implementation, two inquiry fields are constructed, the execution number of the previous month data maintenance and the execution number of the past six month data maintenance are respectively obtained, and the two inquiry results are checked.
Example 4
Revenue conditions are monitored. Type of body: accounting; a body table field, revenue; action type: obtaining or warehousing; screening time: last month; auditing algorithm: the sum of income from last month is larger than a preset value; query time: 11:00:00; query period: monthly No. 7; warning information receiver: finance; the warning information mode is as follows: e, mail.
The invention can also comprise a monitoring method of the audit plan, as shown in fig. 2:
step 201: and acquiring data query time and audit planning quantity.
Step 202: and acquiring the auditing plan number in unit time according to the data query time and the auditing plan number. In a specific embodiment, taking half hour as a calculation reference, obtaining the auditing plan number of each half hour.
Step 203: and generating early warning information according to the auditing planning quantity in unit time and a preset early warning range.
Step 204: and sending the early warning information to an early warning receiver.
In order to avoid excessive auditing planning data in the same time period, auditing is carried out on the auditing planning quantity in unit time, so that the auditing planning of peak-shifting arrangement is facilitated, and the congestion of a data system and the auditing planning is prevented.
The data system can comprise a database or a plurality of databases, the query results are obtained in different databases according to the query fields, and the query results returned by the different databases are audited.
The invention also provides a data system monitoring system, as shown in fig. 3, which comprises a first construction module 1, a second construction module 2, an acquisition module 3, a query module 4 and an auditing module 5, wherein the first construction module 1 is used for respectively establishing a main body category and an action category according to a main body table field and an action field of the data system, and is used for establishing a master-slave relationship between the main body category and a main body table field; the second construction module 2 is used for constructing a data auditing plan table according to the action category, the main body category and the main body table field, and the auditing plan table further comprises an auditing algorithm and data query time; the acquisition module 3 is used for acquiring a query field according to the main body type of the data auditing plan table, the main body table field and the action type; the query module 4 is configured to obtain a query result of the data system according to the query field at the data query time; the auditing module 5 is used for obtaining auditing results according to an auditing algorithm and the query results.
The data system monitoring system provided by the invention can further comprise an early warning module 6, wherein the early warning module 6 is used for generating early warning information according to the auditing result and the early warning range and generating warning information according to the early warning information and the warning information template.
The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method of monitoring a data system, the method comprising:
respectively establishing a main body category and an action category according to a main body table field and an action field of the data system;
establishing a master-slave relationship between the main body category and the main body table field;
constructing a data auditing plan table according to the action category, the main body category and the main body table field, wherein the auditing plan table further comprises an auditing algorithm and data inquiry time;
acquiring a query field according to the main body type of the data auditing plan table, the main body table field and the action type;
acquiring a query result of a data system according to the query field in the data query time;
and obtaining an auditing result according to the auditing algorithm and the query result.
2. The data system monitoring method of claim 1, further comprising the step of generating pre-warning information:
generating early warning information according to the auditing result and the early warning range;
generating warning information according to the warning information and the warning information template.
3. The data system monitoring method of claim 2, further comprising a method of transmitting alert information to an alert information recipient by one or a combination of: short message pushing, instant chat tool sending, mail sending, bulletin pushing and PC displaying.
4. The data system monitoring method of claim 1, wherein the audit plan includes a plurality of action categories, and wherein the query field is constructed from a body table field and the plurality of action categories.
5. The method of claim 1, wherein the audit plan further includes a screening time, and wherein query results within the screening time are obtained based on the query field.
6. The data system monitoring method of claim 1, wherein the data query time comprises a query time point and a query period.
7. The method of any one of claims 1 to 6, further comprising the step of auditing the monitoring method of:
acquiring data query time and audit planning quantity;
acquiring the auditing plan number in unit time according to the data query time and the auditing plan number;
generating early warning information according to the auditing planning quantity in unit time and a preset early warning range,
and sending the early warning information to an early warning receiver.
8. A method of monitoring a data system according to claim 1, wherein the data system comprises at least one database.
9. A data system monitoring system is characterized by comprising a first construction module, a second construction module, an acquisition module, a query module and an auditing module,
the first construction module is used for respectively establishing a main body category and an action category according to main body table fields and action fields of the data system, and is used for establishing a master-slave relationship between the main body category and main body table fields;
the second construction module is used for constructing a data auditing plan table according to the action category, the main body category and the main body table field, and the auditing plan table further comprises an auditing algorithm and data query time;
the acquisition module is used for acquiring a query field according to the main body type of the data auditing plan table, the main body table field and the action type;
the query module is used for acquiring a query result of the data system according to the query field at the data query time;
the auditing module is used for acquiring auditing results according to the auditing algorithm and the query results.
10. The data system monitoring system of claim 9, further comprising an early warning module,
the early warning module is used for generating early warning information according to the auditing result and the early warning range and generating warning information according to the early warning information and the warning information template.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010567262.1A CN111767283B (en) | 2020-06-19 | 2020-06-19 | Data system monitoring method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010567262.1A CN111767283B (en) | 2020-06-19 | 2020-06-19 | Data system monitoring method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111767283A CN111767283A (en) | 2020-10-13 |
| CN111767283B true CN111767283B (en) | 2023-08-18 |
Family
ID=72721394
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010567262.1A Active CN111767283B (en) | 2020-06-19 | 2020-06-19 | Data system monitoring method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111767283B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101415168A (en) * | 2007-10-17 | 2009-04-22 | 华为技术有限公司 | Method and apparatus for checking charging result and configuration data |
| CN101902532A (en) * | 2009-05-27 | 2010-12-01 | 北京汉铭通信有限公司 | Data auditing method and system of telecommunication services |
| EP2804121A2 (en) * | 2013-05-17 | 2014-11-19 | Hitachi Ltd. | Analysis system and health business support method |
| CN104424307A (en) * | 2013-09-04 | 2015-03-18 | 腾讯科技（深圳）有限公司 | Intelligent terminal application classifying method, system and intelligent terminal, |
| CN105468352A (en) * | 2014-09-12 | 2016-04-06 | 中兴通讯股份有限公司 | Query interface generation method and apparatus |
| CN105956103A (en) * | 2016-05-04 | 2016-09-21 | 北京思特奇信息技术股份有限公司 | Automatic amount calculation method and system |
| CN109947746A (en) * | 2017-10-26 | 2019-06-28 | 亿阳信通股份有限公司 | A kind of quality of data management-control method and system based on ETL process |
| CN110311850A (en) * | 2019-07-04 | 2019-10-08 | 北京天融信网络安全技术有限公司 | A kind of network-based data processing method and electronic equipment |
| CN111159171A (en) * | 2019-12-31 | 2020-05-15 | 中国铁塔股份有限公司 | Data auditing method and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070168335A1 (en) * | 2006-01-17 | 2007-07-19 | Moore Dennis B | Deep enterprise search |
-
2020
- 2020-06-19 CN CN202010567262.1A patent/CN111767283B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101415168A (en) * | 2007-10-17 | 2009-04-22 | 华为技术有限公司 | Method and apparatus for checking charging result and configuration data |
| CN101902532A (en) * | 2009-05-27 | 2010-12-01 | 北京汉铭通信有限公司 | Data auditing method and system of telecommunication services |
| EP2804121A2 (en) * | 2013-05-17 | 2014-11-19 | Hitachi Ltd. | Analysis system and health business support method |
| CN104424307A (en) * | 2013-09-04 | 2015-03-18 | 腾讯科技（深圳）有限公司 | Intelligent terminal application classifying method, system and intelligent terminal, |
| CN105468352A (en) * | 2014-09-12 | 2016-04-06 | 中兴通讯股份有限公司 | Query interface generation method and apparatus |
| CN105956103A (en) * | 2016-05-04 | 2016-09-21 | 北京思特奇信息技术股份有限公司 | Automatic amount calculation method and system |
| CN109947746A (en) * | 2017-10-26 | 2019-06-28 | 亿阳信通股份有限公司 | A kind of quality of data management-control method and system based on ETL process |
| CN110311850A (en) * | 2019-07-04 | 2019-10-08 | 北京天融信网络安全技术有限公司 | A kind of network-based data processing method and electronic equipment |
| CN111159171A (en) * | 2019-12-31 | 2020-05-15 | 中国铁塔股份有限公司 | Data auditing method and system |
Non-Patent Citations (1)
| Title |
|---|
| 电力监控信息智能提取与分类处理的系统及方法;曹德发;电工技术(第03期);137-139 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111767283A (en) | 2020-10-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109409633B (en) | Business monitoring and risk early warning system | |
| CN105868373B (en) | Method and device for processing key data of power business information system | |
| CN110599007A (en) | Project progress management system | |
| CN101197876A (en) | Method and system for multi-dimensional analysis of message service data | |
| CN114257636B (en) | Unified message publishing method | |
| CN107329877A (en) | Air ticket business monitoring execution system and method | |
| CN101925039A (en) | Prewarning method and device of billing ticket | |
| SG177868A1 (en) | Digital analytics platform | |
| CN109063066A (en) | Data query method and device, data management system | |
| CN115730909A (en) | Enterprise client information management system | |
| US6338071B1 (en) | Method and system for providing a contract management system using an action-item table | |
| CN111767283B (en) | Data system monitoring method and system | |
| CN110930121A (en) | Industry integration management system based on wisdom travel | |
| CN114124743B (en) | Method and system for executing full-link inspection rule of data application | |
| CN115271514A (en) | Communication enterprise monitoring method and device, electronic equipment and storage medium | |
| CN113986656B (en) | Power grid data safety monitoring system based on data center platform | |
| CN115423361A (en) | Data processing method and device for risk view, storage medium and equipment | |
| CN114118762A (en) | Service monitoring and risk prediction closed-loop method based on power grid marketing management | |
| CN112396349A (en) | Data quality monitoring method based on business entity | |
| CN113723900A (en) | Product life cycle management platform, and system and method for executing and managing changed data | |
| CN113141275A (en) | Intelligent integrated operation and maintenance system based on cloud service platform | |
| CN112258151A (en) | Reconciliation method and device based on pandas, computer equipment and storage medium | |
| CN112965793B (en) | Identification analysis data-oriented data warehouse task scheduling method and system | |
| Ferreira | Telco Voice Traffic: Database Construction and Visualization | |
| CN115423454A (en) | Agricultural wheat information management system based on information cloud platform |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |