CN110990382A - Data traceability management system for information operation monitoring - Google Patents
Data traceability management system for information operation monitoring Download PDFInfo
- Publication number
- CN110990382A CN110990382A CN201911320010.2A CN201911320010A CN110990382A CN 110990382 A CN110990382 A CN 110990382A CN 201911320010 A CN201911320010 A CN 201911320010A CN 110990382 A CN110990382 A CN 110990382A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- information
- full
- mapping relation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to data tracing, in particular to a data tracing management system for information operation monitoring, which comprises a source system for acquiring data resources and a full-service data center for processing and storing the data resources acquired by the source system, wherein a tracing server is arranged between the source system and the full-service data center and is connected with a data acquisition module for acquiring data resources flowing into the full-service data center from the source system; the technical scheme provided by the invention can effectively overcome the defects that the prior art can not effectively trace the source of the processed data, can not check the authenticity of the data resource and can not quickly know the relationship between the data link and the service.
Description
Technical Field
The invention relates to data tracing, in particular to a data tracing management system for information operation monitoring.
Background
The massive and diversified data assets of the power grid cause difficulty in comprehensively knowing the data types, distribution, links, quantity, association with services and the like, and the overall view of the data assets cannot be mastered. In the aspect of the current data service directory, the data service directory is provided by a file under line, which is not convenient for updating and information synchronization, and has the problems of difficult data resource retrieval and low efficiency of obtaining data resources under line, and needs multi-party coordination and multi-party participation.
Due to the lack of an online data tracing management system, business personnel cannot intuitively and quickly know data sources, data links and the relationship between data and business, cannot quickly find the data, understand the data and use the data, and simultaneously have the problems of repeated data tracing and business tracing, so that the resource utilization efficiency is reduced, the excavation of data value is seriously restricted, and the improvement of data sharing service capability is influenced.
The existing data tracing management system cannot effectively trace the source of the processed data, cannot check the authenticity of data resources, and cannot quickly know the relationship between a data link and data and business.
Disclosure of Invention
Technical problem to be solved
Aiming at the defects in the prior art, the invention provides a data traceability management system for information operation monitoring, which can effectively overcome the defects that the prior art cannot effectively trace the source of processed data, cannot check the authenticity of data resources and cannot quickly know the relationship between a data link and data and service.
(II) technical scheme
In order to achieve the purpose, the invention is realized by the following technical scheme:
a data traceability management system for information operation monitoring comprises a source system for acquiring data resources and a full-service data center for processing and storing the data resources acquired by the source system, wherein a traceability server is arranged between the source system and the full-service data center;
the tracing server is connected with a data acquisition module which is used for acquiring data resources flowing into the full-service data center from the source system, the tracing server is connected with a data source information collection module which is used for collecting source system information corresponding to the data resources flowing into the full-service data center, the tracing server is connected with a Hash algorithm generation module which is used for generating a Hash value, the tracing server is connected with a watermark algorithm generation module which is used for generating a watermark, and the data source information collection module is connected with a data source information fusion module which is used for fusing source system information into a Hash algorithm and a watermark algorithm;
the tracing server is connected with a Hash generating module which is used for generating a corresponding Hash value according to the Hash algorithm generating module, the tracing server is connected with a watermark generating module which is used for generating a corresponding watermark according to the watermark algorithm generating module, and the tracing server is connected with a verification information embedding module which is used for embedding the Hash value and the watermark into the data resource flowing into the full-service data center correspondingly;
the tracing server is connected with a verification information extraction module used for extracting the verification information of the data resources flowing into the full-service data center, and the tracing server is connected with a mapping relation association module used for carrying out full-process tracking on the data resources processed and stored by the full-service data center.
Preferably, the source system information includes an acquisition host MAC and an acquisition device serial number.
Preferably, the mapping relationship association module performs link tracking on the whole process of processing, storing, applying and sharing of the data resource, and embeds the association mapping relationship obtained by link tracking into the verification information.
Preferably, the mapping relation association module is connected to an association mapping relation storage module for storing the association mapping relation obtained by the mapping relation association module performing link tracking, and the association mapping relation storage module is connected to an association mapping relation query module for querying a corresponding association mapping relation of the input data resource.
Preferably, the full-service data center comprises a data buffer area, a Gbase database and an Hbase database, wherein the Gbase database comprises a detail data layer and a motion monitoring data processing area.
Preferably, the source system data collection resource comprises structured data and measurement data, the structured data is extracted to the data buffer by an ETL and OGG manner, and the measurement data is extracted to the Hbase database for storage.
Preferably, the structured data is calculated on a Gbase database at regular time by using a Spark component, and the measured data stored in the Hbase database is extracted into the Gbase database for operation.
Preferably, the structured data comprises ERP, PMS and marketing data, and the measurement data comprises power transmission and transformation state data, power supply voltage and power utilization information data.
Preferably, the operation and supervision data processing area contains a writing script which forms a wide table according to the service logic relationship of data tracing.
(III) advantageous effects
Compared with the prior art, the data traceability management system for information operation monitoring provided by the invention has the following beneficial effects:
1. the data source information collection module collects source system information corresponding to data resources flowing into the full-service data center, the data source information fusion module fuses the source system information into a Hash algorithm and a watermark algorithm, the Hash generation module and the watermark generation module respectively generate a Hash value and a watermark containing the source system information, and the verification information embedding module embeds the Hash value and the watermark into the data resources flowing into the full-service data center correspondingly, so that the processed data can be effectively traced to the source, and the authenticity of the data resources is convenient to check;
2. the verification information extraction module extracts data resource verification information flowing into the full-service data center, the mapping relation association module carries out overall process tracking on data resources processed and stored by the full-service data center, the association mapping relation storage module stores the mapping relation association module and carries out link tracking to obtain an association mapping relation, and the association mapping relation query module is used for inputting data resources to query the corresponding association mapping relation, so that service personnel can quickly know data links and the relation between data and services.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
FIG. 1 is a schematic view of an operation monitoring data processing system according to the present invention;
fig. 2 is a schematic diagram of a data tracing system between the source system and the full-service data center in fig. 1 according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. It is to be understood that the embodiments described are only a few embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
A data traceability management system for information operation monitoring, as shown in fig. 1 and 2, includes a source system for acquiring data resources, and a full-service data center for processing and storing the data resources acquired by the source system, wherein a traceability server is arranged between the source system and the full-service data center;
the tracing server is connected with a data acquisition module for acquiring data resources flowing into the full-service data center from a source system, the tracing server is connected with a data source information acquisition module for acquiring source system information corresponding to the data resources flowing into the full-service data center, the tracing server is connected with a Hash algorithm generation module for generating a Hash value, the tracing server is connected with a watermark algorithm generation module for generating a watermark, and the data source information acquisition module is connected with a data source information fusion module for fusing the source system information into the Hash algorithm and the watermark algorithm;
the tracing server is connected with a Hash generating module which is used for generating a corresponding Hash value according to a Hash algorithm generating module, the tracing server is connected with a watermark generating module which is used for generating a corresponding watermark according to a watermark algorithm generating module, and the tracing server is connected with a verification information embedding module which is used for embedding the Hash value and the watermark into data resources which correspondingly flow into the full-service data center;
the tracing server is connected with a verification information extraction module for extracting the verification information of the data resources flowing into the full-service data center, and the tracing server is connected with a mapping relation association module for carrying out full-process tracking on the data resources processed and stored by the full-service data center.
The source system information comprises an acquisition host MAC and an acquisition equipment serial number.
And the mapping relation association module performs link tracking on the whole process of processing, storing, applying and sharing the data resources and embeds the association mapping relation obtained by the link tracking into the verification information.
The mapping relation association module is connected with an association mapping relation storage module which is used for storing the mapping relation association module to perform link tracking to obtain the association mapping relation, and the association mapping relation storage module is connected with an association mapping relation query module which is used for inputting data resource query corresponding association mapping relation.
The full-service data center comprises a data buffer area, a Gbase database and an Hbase database, wherein the Gbase database comprises a detail data layer and a motion monitoring data processing area.
The source system acquires data resources including structured data and measurement data, the structured data are extracted to a data buffer area through an ETL (extract transform and load) and the measurement data are extracted to an Hbase database for storage.
Structured data are calculated on a Gbase database in a timing mode by a Spark component, and measured data stored in an Hbase database are extracted into the Gbase database to be operated.
The structured data comprises ERP, PMS and marketing data, and the measured data comprises power transmission and transformation state data, power supply voltage and power utilization information data.
The operation monitoring data processing area contains a writing script which forms a wide table according to the service logic relation of data tracing.
The data source information collection module collects source system information corresponding to data resources flowing into the full-service data center, the data source information fusion module fuses the source system information into a Hash algorithm and a watermark algorithm, the Hash generation module and the watermark generation module respectively generate a Hash value and a watermark containing the source system information, and the verification information embedding module embeds the Hash value and the watermark into the data resources flowing into the full-service data center correspondingly, so that the processed data can be effectively traced to the source, and the authenticity of the data resources is convenient to check.
The source system information comprises an acquisition host MAC and an acquisition equipment serial number.
And the mapping relation association module performs link tracking on the whole process of processing, storing, applying and sharing the data resources and embeds the association mapping relation obtained by the link tracking into the verification information.
The verification information extraction module extracts data resource verification information flowing into the full-service data center, the mapping relation association module carries out overall process tracking on data resources processed and stored by the full-service data center, the association mapping relation storage module stores the mapping relation association module and carries out link tracking to obtain an association mapping relation, and the association mapping relation query module is used for inputting data resources to query the corresponding association mapping relation, so that service personnel can quickly know data links and the relation between data and services.
The operation monitoring data processing system of the invention is shown in figure 1:
the full-service data center comprises a data buffer area, a Gbase database and an Hbase database, wherein the Gbase database comprises a detail data layer and a motion monitoring data processing area.
The operation monitoring data processing area contains a writing script which forms a wide table according to the service logic relation of data tracing.
The source system acquires data resources including structured data and measurement data, the structured data are extracted to a data buffer area through an ETL (extract transform and load) and the measurement data are extracted to an Hbase database for storage. Structured data are calculated on a Gbase database in a timing mode by a Spark component, and measured data stored in an Hbase database are extracted into the Gbase database to be operated.
The structured data comprises ERP, PMS and marketing data, and the measured data comprises power transmission and transformation state data, power supply voltage and power utilization information data.
The above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not depart from the spirit and scope of the corresponding technical solutions.
Claims (9)
1. The utility model provides a data management system that traces to source for information operation monitoring which characterized in that: the system comprises a source system for acquiring data resources and a full-service data center for processing and storing the data resources acquired by the source system, wherein a source tracing server is arranged between the source system and the full-service data center;
the tracing server is connected with a data acquisition module which is used for acquiring data resources flowing into the full-service data center from the source system, the tracing server is connected with a data source information collection module which is used for collecting source system information corresponding to the data resources flowing into the full-service data center, the tracing server is connected with a Hash algorithm generation module which is used for generating a Hash value, the tracing server is connected with a watermark algorithm generation module which is used for generating a watermark, and the data source information collection module is connected with a data source information fusion module which is used for fusing source system information into a Hash algorithm and a watermark algorithm;
the tracing server is connected with a Hash generating module which is used for generating a corresponding Hash value according to the Hash algorithm generating module, the tracing server is connected with a watermark generating module which is used for generating a corresponding watermark according to the watermark algorithm generating module, and the tracing server is connected with a verification information embedding module which is used for embedding the Hash value and the watermark into the data resource flowing into the full-service data center correspondingly;
the tracing server is connected with a verification information extraction module used for extracting the verification information of the data resources flowing into the full-service data center, and the tracing server is connected with a mapping relation association module used for carrying out full-process tracking on the data resources processed and stored by the full-service data center.
2. The data traceability management system for information operation monitoring according to claim 1, wherein: the source system information comprises an acquisition host machine MAC and an acquisition equipment serial number.
3. The data traceability management system for information operation monitoring according to claim 1, wherein: the mapping relation association module carries out link tracking on the whole process of processing, storing, applying and sharing of the data resources and embeds the association mapping relation obtained by the link tracking into the verification information.
4. The data traceability management system for information operation monitoring according to claim 3, wherein: the mapping relation association module is connected with an association mapping relation storage module used for storing the association mapping relation obtained by the mapping relation association module through link tracking, and the association mapping relation storage module is connected with an association mapping relation query module used for querying the corresponding association mapping relation of the input data resource.
5. The data traceability management system for information operation monitoring according to claim 1, wherein: the full-service data center comprises a data buffer area, a Gbase database and an Hbase database, wherein the Gbase database comprises a detail data layer and a motion monitoring data processing area.
6. The data traceability management system for information operation monitoring according to claim 5, wherein: the source system acquires data resources including structured data and measurement data, the structured data are extracted to the data buffer area in an ETL and OGG mode, and the measurement data are extracted to the Hbase database for storage.
7. The data traceability management system for information operation monitoring according to claim 6, wherein: and the structured data is calculated on a Gbase database at regular time by adopting a Spark component, and the measured data stored in the Hbase database is extracted into the Gbase database for operation.
8. The data traceability management system for information operation monitoring according to claim 6 or 7, wherein: the structured data comprises ERP, PMS and marketing data, and the measurement data comprises power transmission and transformation state data, power supply voltage and power utilization information data.
9. The data traceability management system for information operation monitoring according to claim 5, wherein: the operation monitoring data processing area comprises a writing script which forms a wide table according to the service logic relation of data tracing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911320010.2A CN110990382A (en) | 2019-12-19 | 2019-12-19 | Data traceability management system for information operation monitoring |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911320010.2A CN110990382A (en) | 2019-12-19 | 2019-12-19 | Data traceability management system for information operation monitoring |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110990382A true CN110990382A (en) | 2020-04-10 |
Family
ID=70065654
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911320010.2A Pending CN110990382A (en) | 2019-12-19 | 2019-12-19 | Data traceability management system for information operation monitoring |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110990382A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111625342A (en) * | 2020-06-07 | 2020-09-04 | 智博云信息科技(广州)有限公司 | Data tracing method, device and server |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104599139A (en) * | 2014-12-31 | 2015-05-06 | 康美药业股份有限公司 | Tracing interaction system and tracing system interaction method |
| CN105184345A (en) * | 2015-10-13 | 2015-12-23 | 北京工业大学 | Digital-watermark-based implicit two-dimensional code |
| CN106874373A (en) * | 2016-12-30 | 2017-06-20 | 江苏瑞中数据股份有限公司 | A kind of implementation method of the electric network data asset management platform based on SOA |
| CN107633413A (en) * | 2017-10-17 | 2018-01-26 | 深圳市裕同包装科技股份有限公司 | A kind of anti-fake traceability system and method |
| CN108804280A (en) * | 2018-04-09 | 2018-11-13 | 广州中国科学院软件应用技术研究所 | A kind of online electronic evidence traceability system of intelligent terminal and method |
| CN109427029A (en) * | 2017-08-21 | 2019-03-05 | 苏宁云商集团股份有限公司 | A kind of method and device generating watermark |
| CN109558709A (en) * | 2018-08-02 | 2019-04-02 | 北京同余科技有限公司 | A kind of safe tracing technology of big data based on Hive |
| CN109740316A (en) * | 2018-12-27 | 2019-05-10 | 北京三未信安科技发展有限公司 | A kind of insertion of dynamic watermark, verification method and system and dynamic watermark processing system |
| CN109754037A (en) * | 2019-01-28 | 2019-05-14 | 成都宇飞信息工程有限责任公司 | A kind of one yard of commodity counterfeit prevention of an object and traceability system based on digital watermarking two dimensional code |
| CN110069937A (en) * | 2019-04-11 | 2019-07-30 | 智链万源(北京)数字科技有限公司 | Data source tracing method, system and device based on block chain |
-
2019
- 2019-12-19 CN CN201911320010.2A patent/CN110990382A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104599139A (en) * | 2014-12-31 | 2015-05-06 | 康美药业股份有限公司 | Tracing interaction system and tracing system interaction method |
| CN105184345A (en) * | 2015-10-13 | 2015-12-23 | 北京工业大学 | Digital-watermark-based implicit two-dimensional code |
| CN106874373A (en) * | 2016-12-30 | 2017-06-20 | 江苏瑞中数据股份有限公司 | A kind of implementation method of the electric network data asset management platform based on SOA |
| CN109427029A (en) * | 2017-08-21 | 2019-03-05 | 苏宁云商集团股份有限公司 | A kind of method and device generating watermark |
| CN107633413A (en) * | 2017-10-17 | 2018-01-26 | 深圳市裕同包装科技股份有限公司 | A kind of anti-fake traceability system and method |
| CN108804280A (en) * | 2018-04-09 | 2018-11-13 | 广州中国科学院软件应用技术研究所 | A kind of online electronic evidence traceability system of intelligent terminal and method |
| CN109558709A (en) * | 2018-08-02 | 2019-04-02 | 北京同余科技有限公司 | A kind of safe tracing technology of big data based on Hive |
| CN109740316A (en) * | 2018-12-27 | 2019-05-10 | 北京三未信安科技发展有限公司 | A kind of insertion of dynamic watermark, verification method and system and dynamic watermark processing system |
| CN109754037A (en) * | 2019-01-28 | 2019-05-14 | 成都宇飞信息工程有限责任公司 | A kind of one yard of commodity counterfeit prevention of an object and traceability system based on digital watermarking two dimensional code |
| CN110069937A (en) * | 2019-04-11 | 2019-07-30 | 智链万源(北京)数字科技有限公司 | Data source tracing method, system and device based on block chain |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111625342A (en) * | 2020-06-07 | 2020-09-04 | 智博云信息科技(广州)有限公司 | Data tracing method, device and server |
| CN111625342B (en) * | 2020-06-07 | 2020-11-17 | 贵州正高科技有限公司 | Data tracing method, device and server |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107402976B (en) | Power grid multi-source data fusion method and system based on multi-element heterogeneous model | |
| CN109635127B (en) | Power equipment portrait knowledge map construction method based on big data technology | |
| CN102750406B (en) | Multi-version management method for model set and difference model-based power grid model | |
| CN105787089A (en) | Power distribution network scheduling foundation data integrating method | |
| CN102254024A (en) | Mass data processing system and method | |
| CN108446396B (en) | Power data processing method based on improved CIM model | |
| CN106446085A (en) | Big data management system | |
| CN104268219A (en) | Management method and system thereof for mass electricity utilization information collection data | |
| CN106845816A (en) | A kind of intelligent electric energy meter life cycle management cloud platform management system | |
| CN109241065B (en) | Intelligent substation cable operation information identification system and identification method thereof | |
| Zieliński | Internet of Everything (IoE) in smart grid | |
| CN108595390B (en) | Method for verifying parameter consistency of SCD and IED of intelligent substation | |
| CN107862459B (en) | Metering equipment state evaluation method and system based on big data | |
| Du et al. | An improved BIRCH clustering algorithm and application in thermal power | |
| CN110990382A (en) | Data traceability management system for information operation monitoring | |
| CN114218291A (en) | Portrait generation method, apparatus, device and storage medium based on target object | |
| CN113934807A (en) | Territorial space planning system and method based on GIS | |
| CN105321331B (en) | The processing method of ammeter information, apparatus and system | |
| CN106127602B (en) | Electricity stealing identification method and device based on reduction outlier algorithm | |
| CN109508354A (en) | A kind of parallel processing system (PPS) | |
| CN112446619B (en) | Power distribution network rush-repair processing method and device | |
| CN111967008B (en) | Code safety automatic scanning method and device based on multi-CPU (Central processing Unit) architecture container cloud | |
| CN105938610A (en) | Illegal electricity utilization judgment and report automatic generation method applied to electricity utilization collection system | |
| CN113868821A (en) | Distribution network loss reduction method based on marketing and distribution big data fusion and terminal | |
| CN113064791A (en) | Scattered label feature extraction method based on real-time monitoring of mass log data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200410 |