CN110096415A - A kind of data monitoring method based on topological relation - Google Patents
A kind of data monitoring method based on topological relation Download PDFInfo
- Publication number
- CN110096415A CN110096415A CN201910400680.9A CN201910400680A CN110096415A CN 110096415 A CN110096415 A CN 110096415A CN 201910400680 A CN201910400680 A CN 201910400680A CN 110096415 A CN110096415 A CN 110096415A
- Authority
- CN
- China
- Prior art keywords
- data
- topological relation
- monitoring
- tables
- monitoring method
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 47
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000003860 storage Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 9
- 238000004590 computer program Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
Data monitoring method disclosed by the invention based on topological relation, it is related to big data technical field, by obtaining the data source for needing to monitor from database, obtain the topological relation of data in multiple tables of data, according to topological relation, write the configuration file of corresponding text formatting, utilize java language, write the code that can load, parse and execute the configuration file, write the MapReduce program for executing the code, the MapReduce program is run in big data platform, obtains corresponding monitoring result, improves the efficiency and accuracy of data monitoring.
Description
Technical field
The invention belongs to big data technical fields, and in particular to a kind of data monitoring method based on topological relation.
Background technique
Big data has influenced all trades and professions at present, is all too busy to get away technology of current information technology enterprise.Work as enterprise
When the data volume and data target and dimension of industry are all continuously increased, the calculating process of business can also become complicated therewith,
Cause data monitoring efficiency lower.There is also complicated dependences for data in complicated calculating process.Once it goes wrong,
It cannot find the problem well and orientation problem.In current dynamic scheduling system, using to tables of data have more than 50, number
According to dependence depth at most up to 10 layers or so, it is not easy to be found, data is caused to be supervised once there are data to go wrong especially
The accuracy of survey is lower.
Summary of the invention
In view of the defects existing in the prior art, the embodiment of the invention provides a kind of data monitoring sides based on topological relation
Method, method includes the following steps:
The data source for needing to monitor is obtained from database, wherein the data source includes multiple tables of data；
The topological relation for obtaining data in the multiple tables of data writes corresponding text lattice according to the topological relation
The configuration file of formula；
Using java language, the code that can load, parse and execute the configuration file is write；
Write the MapReduce program for executing the code；
The MapReduce program is run in big data platform, obtains corresponding monitoring result.
Preferably, the topological relation for obtaining data in the multiple tables of data includes:
Corresponding monitoring sequence, number, execution movement, dependence and monitoring result is arranged in respectively each tables of data.
Preferably, after obtaining monitoring result, the method also includes:
The monitoring result is analyzed, the root data led to the problem of are searched from the data source.
Preferably, after obtaining the root data that lead to the problem of in the data source, the method also includes:
The root data are graphically shown in big data platform.
Data monitoring method provided in an embodiment of the present invention based on topological relation, has the advantages that
It is found the problem by the dependence of monitoring data and orientation problem, improves the efficiency of data monitoring and accurate
Degree.
Detailed description of the invention
Fig. 1 is the flow diagram of the data monitoring method provided in an embodiment of the present invention based on topological relation；
Fig. 2 is the topological relation schematic diagram between the tables of data provided in an embodiment of the present invention for needing to monitor.
Specific embodiment
Specific introduce is made to the present invention below in conjunction with the drawings and specific embodiments.
As shown in Figure 1, the data monitoring method provided in an embodiment of the present invention based on topological relation the following steps are included:
S101 obtains the data source for needing to monitor, wherein the data source includes multiple tables of data from database.
Further, which includes the database of each enterprise or public institution's different service types.
As a specific embodiment, the data source for needing to monitor includes 5 data Table As, B, C, D and E, this 5 numbers
It is as shown in Figure 2 according to the topological relation between table.
S102 obtains the topological relation of data in multiple tables of data, according to topological relation, writes corresponding text formatting
Configuration file.
S103 writes the code that can load, parse and execute the configuration file using java language.
S104 writes the MapReduce program for executing the code.
S105 runs the MapReduce program in big data platform, obtains corresponding monitoring result.
Preferably, the topological relation for obtaining data in multiple tables of data includes:
Corresponding monitoring sequence, number, execution movement, dependence and monitoring result is arranged in respectively each tables of data.
As a specific embodiment, according to 5 data Table As shown in Figure 1, the topological relation between B, C, D and E
The configuration file write is as follows:
Data 1=tables of data A, execution=monitoring data Table A, output=tables of data A monitoring result；
Data 2=tables of data D, execution=monitoring data table D, output=tables of data D monitoring result；
Data 3=tables of data B, execution=monitoring data table B, dependence=tables of data A and tables of data D, output=B monitoring
As a result；
Data 4=tables of data E, execution=monitoring data table E, output=tables of data E monitoring result；
Data 5=tables of data C, execution=monitoring data table C, dependence=tables of data B and tables of data E, output=tables of data C
Monitoring result.
Optionally, after obtaining monitoring result, the method also includes:
The monitoring result is analyzed, the root data led to the problem of are searched from the data source.
As a specific embodiment, by taking data source shown in Fig. 2 as an example, when the data in tables of data B occur it is abnormal,
When missing or vacancy, according to the dependence of tables of data B, monitoring result is searched in tables of data A and tables of data D respectively exception
Node, that is, can determine the root data led to the problem of.
Optionally, after obtaining monitoring result, the method also includes:
The monitoring result is analyzed, the root data led to the problem of are searched from the data source.
Optionally, after obtaining the root data that lead to the problem of in the data source, the method also includes:
The root data are graphically shown in big data platform.
Data monitoring method provided in an embodiment of the present invention based on topological relation needs to supervise by obtaining from database
The data source of survey obtains the topological relation of data in multiple tables of data, according to topological relation, writes matching for corresponding text formatting
File is set, using java language, the code that can load, parse and execute the configuration file is write, writes for executing the generation
The MapReduce program of code, the MapReduce program is run in big data platform, obtains corresponding monitoring result, is improved
The efficiency and accuracy of data monitoring.
It is apparent to those skilled in the art that for convenience and simplicity of description, the system of foregoing description,
The specific work process of device and unit, can refer to corresponding processes in the foregoing method embodiment, and details are not described herein.
Algorithm and display are not inherently related to any particular computer, virtual system, or other device provided herein.
Various general-purpose systems can also be used together with teachings based herein.As described above, it constructs required by this kind of system
Structure be obvious.In addition, the present invention is also not directed to any particular programming language.It should be understood that can use various
Programming language realizes summary of the invention described herein, and the description done above to language-specific is to disclose this hair
Bright preferred forms.
In addition, memory may include the non-volatile memory in computer-readable medium, random access memory
(RAM) and/or the forms such as Nonvolatile memory, such as read-only memory (ROM) or flash memory (flash RAM), memory includes extremely
A few storage chip.
It should be understood by those skilled in the art that, embodiments herein can provide as method, system or computer program
Product.Therefore, complete hardware embodiment, complete software embodiment or reality combining software and hardware aspects can be used in the application
Apply the form of example.Moreover, it wherein includes the computer of computer usable program code that the application, which can be used in one or more,
The computer program implemented in usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) produces
The form of product.
The application is referring to method, the process of equipment (system) and computer program product according to the embodiment of the present application
Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions
The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer journeys
Sequence instruct to general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices processor with
A machine is generated, so that the instruction generation executed by computer or the processor of other programmable data processing devices is used for
Realize the dress for the function of specifying in one or more flows of the flowchart and/or one or more blocks of the block diagram
It sets.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates,
Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or
The function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting
Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or
The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one
The step of function of being specified in a box or multiple boxes.
In a typical configuration, calculating equipment includes one or more processors (CPU), input/output interface, net
Network interface and memory.
Memory may include the non-volatile memory in computer-readable medium, random access memory (RAM) and/
Or the forms such as Nonvolatile memory, such as read-only memory (ROM) or flash memory (flash RAM).Memory is computer-readable Jie
The example of matter.
Computer-readable medium includes permanent and non-permanent, removable and non-removable media can be by any method
Or technology come realize information store.Information can be computer readable instructions, data structure, the module of program or other data.
The example of the storage medium of computer includes, but are not limited to phase change memory (PRAM), static random access memory (SRAM), moves
State random access memory (DRAM), other kinds of random access memory (RAM), read-only memory (ROM), electric erasable
Programmable read only memory (EEPROM), flash memory or other memory techniques, read-only disc read only memory (CD-ROM) (CD-ROM),
Digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape magnetic disk storage or other magnetic storages are set
Standby or any other non-transmission medium, can be used for storage can be accessed by a computing device information.As defined in this article, it counts
Calculation machine readable medium does not include temporary computer readable media (transitory media), such as the data-signal and carrier wave of modulation.
It should also be noted that, the terms "include", "comprise" or its any other variant are intended to nonexcludability
It include so that the process, method, commodity or the equipment that include a series of elements not only include those elements, but also to wrap
Include other elements that are not explicitly listed, or further include for this process, method, commodity or equipment intrinsic want
Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including element
There is also other identical elements in process, method, commodity or equipment.
It will be understood by those skilled in the art that embodiments herein can provide as method, system or computer program product.
Therefore, complete hardware embodiment, complete software embodiment or embodiment combining software and hardware aspects can be used in the application
Form.It is deposited moreover, the application can be used to can be used in the computer that one or more wherein includes computer usable program code
The shape for the computer program product implemented on storage media (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.)
Formula.
The above is only embodiments herein, are not intended to limit this application.To those skilled in the art,
Various changes and changes are possible in this application.It is all within the spirit and principles of the present application made by any modification, equivalent replacement,
Improve etc., it should be included within the scope of the claims of this application.
It should be noted that the above embodiments do not limit the invention in any form, it is all to use equivalent replacement or equivalent change
The mode changed technical solution obtained, falls within the scope of protection of the present invention.
Claims (4)
1. a kind of data monitoring method based on topological relation characterized by comprising
The data source for needing to monitor is obtained from database, wherein the data source includes multiple tables of data；
The topological relation for obtaining data in the multiple tables of data writes corresponding text formatting according to the topological relation
Configuration file；
Using java language, the code that can load, parse and execute the configuration file is write；
Write the MapReduce program for executing the code；
The MapReduce program is run in big data platform, obtains corresponding monitoring result.
2. the data monitoring method according to claim 1 based on topological relation, which is characterized in that obtain the multiple number
Topological relation according to data in table includes:
Corresponding monitoring sequence, number, execution movement, dependence and monitoring result is arranged in respectively each tables of data.
3. the data monitoring method according to claim 1 based on topological relation, which is characterized in that obtaining being monitored
As a result after, the method also includes:
The monitoring result is analyzed, the root data led to the problem of are searched from the data source.
4. the data monitoring method according to claim 3 based on topological relation, which is characterized in that from the data source
After the root data that middle acquisition is led to the problem of, the method also includes:
The root data are graphically shown in big data platform.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910400680.9A CN110096415A (en) | 2019-05-15 | 2019-05-15 | A kind of data monitoring method based on topological relation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910400680.9A CN110096415A (en) | 2019-05-15 | 2019-05-15 | A kind of data monitoring method based on topological relation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110096415A true CN110096415A (en) | 2019-08-06 |
Family
ID=67448030
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910400680.9A Pending CN110096415A (en) | 2019-05-15 | 2019-05-15 | A kind of data monitoring method based on topological relation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110096415A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111177977A (en) * | 2019-12-31 | 2020-05-19 | 新奥数能科技有限公司 | Data model construction method and device of energy system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105468869A (en) * | 2015-12-22 | 2016-04-06 | 上海核工程研究设计院 | Topological relation analysis method of PDS (Plant Design System) model |
| CN108388495A (en) * | 2018-01-10 | 2018-08-10 | 链家网（北京）科技有限公司 | A kind of data monitoring method and system |
| CN109344189A (en) * | 2018-09-19 | 2019-02-15 | 浪潮软件集团有限公司 | Big data calculation method and device based on NiFi |
-
2019
- 2019-05-15 CN CN201910400680.9A patent/CN110096415A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105468869A (en) * | 2015-12-22 | 2016-04-06 | 上海核工程研究设计院 | Topological relation analysis method of PDS (Plant Design System) model |
| CN108388495A (en) * | 2018-01-10 | 2018-08-10 | 链家网（北京）科技有限公司 | A kind of data monitoring method and system |
| CN109344189A (en) * | 2018-09-19 | 2019-02-15 | 浪潮软件集团有限公司 | Big data calculation method and device based on NiFi |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111177977A (en) * | 2019-12-31 | 2020-05-19 | 新奥数能科技有限公司 | Data model construction method and device of energy system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108984652A (en) | A kind of configurable data cleaning system and method | |
| CN105511957A (en) | Method and system for generating work alarm | |
| US8725461B2 (en) | Inferring effects of configuration on performance | |
| US20120042209A1 (en) | System and method for analyzing data | |
| US20160170821A1 (en) | Performance assessment | |
| CN110457182A (en) | A kind of load balancing cluster example operating index monitoring system | |
| CN110471754A (en) | Method for exhibiting data, device, equipment and storage medium in job scheduling | |
| CN109215383A (en) | A kind of vehicle dispatching method | |
| CN109412604A (en) | A kind of data compression method based on language model | |
| CN109240867A (en) | Hard disk failure prediction technique | |
| US9665836B2 (en) | System and method for analyzing data | |
| CN109508791A (en) | Vehicle damage management method | |
| CN110096415A (en) | A kind of data monitoring method based on topological relation | |
| CN104765775B (en) | A kind of log preservation method and device | |
| US8849638B2 (en) | System and method for analyzing data | |
| CN113434320A (en) | Information system fault positioning method, device, equipment and storage medium | |
| JP2016103126A (en) | Method for finding condition of category division of key performance indicator, computer for the purpose and computer program | |
| US20140310069A1 (en) | Coordinated business rules management and mixed integer programming | |
| CN110928941A (en) | Data fragment extraction method and device | |
| CN109086362A (en) | The management method of multimode project | |
| CN109902178A (en) | A kind of multistage file classification method and system | |
| CN108845844A (en) | A kind of state transition method and device | |
| CN110019196A (en) | Data processing method and device | |
| CN110245090A (en) | A kind of interface test method | |
| CN105677652B (en) | A kind of data managing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190806 |
|
| RJ01 | Rejection of invention patent application after publication |