CN112699100A - Management and analysis system based on metadata - Google Patents
Management and analysis system based on metadata Download PDFInfo
- Publication number
- CN112699100A CN112699100A CN202011644798.5A CN202011644798A CN112699100A CN 112699100 A CN112699100 A CN 112699100A CN 202011644798 A CN202011644798 A CN 202011644798A CN 112699100 A CN112699100 A CN 112699100A
- Authority
- CN
- China
- Prior art keywords
- metadata
- data
- module
- package
- management
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 51
- 238000013507 mapping Methods 0.000 claims abstract description 29
- 238000012545 processing Methods 0.000 claims abstract description 24
- 238000007726 management method Methods 0.000 claims description 55
- 238000006243 chemical reaction Methods 0.000 claims description 12
- 238000013075 data extraction Methods 0.000 claims description 7
- 238000004140 cleaning Methods 0.000 claims description 6
- 238000012800 visualization Methods 0.000 claims description 6
- 238000007418 data mining Methods 0.000 claims description 5
- 230000009466 transformation Effects 0.000 claims description 4
- 238000012795 verification Methods 0.000 claims description 4
- 230000002457 bidirectional effect Effects 0.000 claims description 3
- 238000013499 data model Methods 0.000 claims description 3
- 238000000034 method Methods 0.000 description 19
- 230000008569 process Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 238000011144 upstream manufacturing Methods 0.000 description 4
- 239000008280 blood Substances 0.000 description 2
- 210000004369 blood Anatomy 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000008676 import Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000003556 assay Methods 0.000 description 1
- 239000000969 carrier Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000035899 viability Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a management analysis system based on metadata, which comprises a service unit and a mapping management unit, wherein the service unit and the mapping management unit are configured by a meta-model management module and comprise a service data table of a source system, a plurality of increment fields are stored in the service data table of the source system, and the mapping management unit is used for creating a mapping relation and a code matching rule of the metadata; the metadata acquisition module is used for acquiring metadata and transmitting the acquired metadata to the metadata processing module, and comprises a template management unit, a metadata database and a warehousing auditing unit; the template management unit is used for providing templates for importing and exporting metadata, the metadata base is configured with a plurality of metadata, and the warehousing auditing unit is used for providing warehousing auditing functions for the metadata which changes; and the metadata processing module is used for processing the service data table of the source system and integrating the service data table into the metadata base. The invention can provide reliable and convenient support for enterprises to establish a metadata management system.
Description
Technical Field
The invention relates to the technical field of metadata management, in particular to a metadata management and analysis system.
Background
Metadata (Metadata), also called intermediary data and relay data, is data (data about data) describing data, and mainly describes and records the definition of a model in a data warehouse, the mapping relationship among layers, the data state of a monitoring data warehouse and the task running state of an ETL. Metadata is generally stored and managed uniformly through a metadata repository, the primary purpose of which is to enable collaborative reconciliation of the design, deployment, operation and management of the data repository. Metadata is an important component of data warehouse management, the metadata management is a core part in an enterprise-level data warehouse, and the metadata is used for driving the development of the data warehouse through the whole life cycle of the data warehouse, so that the data warehouse is automated and visualized.
The basic characteristics of metadata are as follows:
a) once the metadata is established, it can be shared. The structure and integrity of the metadata depend on the value and use environment of the information resources; the development and utilization environment of metadata is often a varied distributed environment; either format cannot fully meet the different needs of different groups;
b) metadata is first a coding scheme. Metadata is an encoding system used to describe digital information resources, especially network information resources, which results in fundamental differences between metadata and conventional data encoding systems; the most important feature and function of metadata is to build a machine understandable framework for digitized information resources.
The data warehouse needs to be managed by means of metadata, because the data processing of the data warehouse is complex, and the access mode of a user is complex, taking DW data of a certain bank as an example: 60 upstream systems, 20 downstream systems, up to 12000 tables in the warehouse, 6000 running ETL tasks, online new versions every month, a lot of data information needing to be maintained, a lot of upstream and downstream systems and frequent change; the maintenance period is long. In addition, the prior art often adopts the three-normal modeling, so the core table of the LDM has a large number of sources and targets, and once the influence analysis is performed, the result is diffused and amplified. Ultimately rendering the assay unusable.
Disclosure of Invention
In view of the above, the present invention provides a metadata-based management analysis system.
In order to solve the technical problems, the invention adopts the technical scheme that: a management analysis system based on metadata comprises a meta-model management module, wherein the meta-model management module is configured with a service unit containing a service data table of a source system and a mapping management unit, the service data table of the source system is stored with a plurality of increment fields, and the mapping management unit is used for creating a mapping relation and a code matching rule of the metadata;
the metadata acquisition module is used for acquiring metadata and transmitting the acquired metadata to the metadata processing module, and comprises a template management unit, a metadata database and a warehousing auditing unit; the template management unit is used for providing templates for importing and exporting metadata, the metadata base is provided with a plurality of metadata, and the warehousing auditing unit is used for providing warehousing auditing functions for the metadata which changes; and
and the metadata processing module is used for processing the service data table of the source system and integrating the service data table into the metadata base.
In the present invention, preferably, the metadata processing module includes a data extraction module, a data cleaning module, a database conversion module and a data loading module, the data extraction module is configured to determine a data source and a source affiliation thereof, and the data cleaning module is configured to process and remove incomplete data, error data and duplicate data; the database conversion module is used for null value processing, data splitting and responsible verification, and the data loading module is used for loading the data of the data buffer area into the database corresponding table.
In the present invention, preferably, the meta-model management module is further configured with a meta-access analysis module, and the meta-access analysis module is configured to provide a meta-data correlation degree, a meta-data access frequency, and a version comparison.
In the present invention, preferably, the increment field is set as a time field or a self-increment field, and the increment field is used for judging newly added data or modified data.
In the present invention, preferably, the data loading module completes data loading in a full-scale manner or an incremental manner.
In the present invention, preferably, the meta-model management module sequentially includes, from bottom to top, an object model layer, a base layer, a resource layer, an analysis layer, and a management layer, where the object model layer is configured to provide a basic structure and corresponding type attributes of a class of a meta-data model, the base layer is configured to provide services for other packages residing in a higher hierarchy, the resource layer is configured to provide bidirectional exchange of data resources as source data or target data, the analysis layer is configured to provide general meta-data, and the management layer is configured to describe a data warehouse information stream and an important event corresponding to the data warehouse information stream.
In the present invention, preferably, the base layer includes a service information packet, a data type packet, an expression packet, a key and index packet, a software deployment packet, and a type mapping packet.
In the present invention, preferably, the resource layer includes an object package, a relational package, a record package, a multidimensional package, and an XML package.
In the present invention, preferably, the analysis layer includes a conversion package, an OLAP analysis package, a data mining package, an information visualization package, and a business term package.
In the present invention, preferably, the metadata base includes DDL, SPQ, Source Data, Excel, XML, and Perl.
In the present invention, preferably, the database correspondence table includes a metadata instance table, a metadata combination relation table, and a metadata dependency relation table.
The invention has the advantages and positive effects that: the invention creates the mapping relation and the code matching rule of the metadata through the mapping management unit in the meta-model management module, and the mapping relation and the code matching rule are matched with the metadata acquisition module, so as to carry out warehousing examination on a plurality of changed parts of the metadata configured in the metadata database, thereby fully utilizing the performance of the metadata database.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings:
FIG. 1 is an architectural topology diagram of a metadata-based management analytics system of the present invention;
FIG. 2 is a diagram of a metadata model architecture for a metadata-based management analysis system in accordance with the present invention;
FIG. 3 is a schematic diagram of an ETL tool based on a metadata management analysis system of the present invention;
FIG. 4 is a diagram illustrating a metadata object relationship of a metadata access analysis module implementing function based on a metadata management analysis system according to the present invention;
fig. 5 is a schematic diagram of the overall structure of a metadata-based management analysis system according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It will be understood that when an element is referred to as being "secured to" another element, it can be directly on the other element or intervening elements may also be present. When a component is referred to as being "connected" to another component, it can be directly connected to the other component or intervening components may also be present. When a component is referred to as being "disposed on" another component, it can be directly on the other component or intervening components may also be present. The terms "vertical," "horizontal," "left," "right," and the like as used herein are for illustrative purposes only.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.
As shown in fig. 5, the present invention provides a management analysis system based on metadata, which includes a meta-model management module, where the meta-model management module configures a service unit containing a service data table of a source system and a mapping management unit, the service data table of the source system stores a plurality of incremental fields, and the mapping management unit is used to create a mapping relationship and a code matching rule of metadata;
the metadata acquisition module is used for acquiring metadata and transmitting the acquired metadata to the metadata processing module, and comprises a template management unit, a metadata database and a warehousing auditing unit; the template management unit is used for providing templates for importing and exporting metadata, the metadata base is provided with a plurality of metadata, and the warehousing auditing unit is used for providing warehousing auditing functions for the metadata which changes; and
and the metadata processing module is used for processing the service data table of the source system and integrating the service data table into the metadata base. Specifically, a physical model is acquired through a database of a service data table of a direct connection source system, the database can be accessed by network connection, and the mapping authority of source data to target data is provided. The template management unit is used for collecting metadata, and the template adopts Excel as a file format. For a general type of template, the metadata of each 'class' in the meta-model corresponds to a Sheet worksheet in Excel, the combination relationship among the metadata is embodied through a combined metadata path, the dependency relationship among the metadata is stored by using a special Sheet, the type of template is suitable for importing and exporting the metadata of the system, and all the meta-models (classes) in the system can be embodied in the template. Users can flexibly customize the corresponding relation between names in the Excel and the meta-model listed in the Sheet according to the requirements, and data validity verification is added to the Excel according to the data type of the attributes of the classes for convenient use.
In this embodiment, the metadata processing module further includes a data extraction module, a data cleaning module, a database conversion module, and a data loading module, where the data extraction module is configured to determine a data source and a source affiliation thereof, and the data cleaning module is configured to process and remove incomplete data, error data, and duplicate data; the database conversion module is used for null value processing, data splitting and responsible verification, and the data loading module is used for loading the data of the data buffer area into the database corresponding table. As shown in fig. 3, since the metadata is collected by extracting data from different metadata carriers through the data extraction module, the collected result needs to be verified after the collection of all data objects and data relationships is completed. The metadata processing module adopts an ETL mode, and firstly extracts, converts and loads the metadata into a target database. The method mainly comprises four modes, namely a trigger mode, an increment field, full synchronization and log comparison, wherein the trigger mode is to establish an insertion trigger, a modification trigger and a deletion trigger on a source system service data table to be extracted according to extraction requirements, when data in the source system service data table changes, the changed data is written into an increment log table by the corresponding trigger, and the increment extraction is to extract the data from the increment log table instead of the source table directly and simultaneously mark or delete the extracted data in the increment log table. The incremental log table does not store all field information of the incremental data, but only stores the name of the service data table of the source system and the updated key word value, acquires a corresponding complete record from the source table, and then correspondingly processes the target database according to the updating operation type.
The full mode adopts a load mode, and the increment mode adopts a MEGRE entering database according to the service rule. View down definitions, including fields and indexes.
The method comprises the steps of obtaining change data through an increment field, utilizing the increment field added in a service data table of a source system, changing the increment field when data in the service data table of the source system is newly added or modified, modifying a timestamp field into corresponding system time, and adding a self-growth field, and further judging which data is newly added and which data is modified.
The process can fully utilize a database engine to realize the expandability, keep the data in the database all the time and avoid the loading and the export of the data, thereby ensuring the efficiency and improving the monitorability of the system; parallel processing optimization can be carried out according to the distribution condition of data, and meanwhile, a disk can be optimized by utilizing the inherent functions of a database; by optimizing the performance of the related database, the efficiency of data processing is greatly improved.
The metadata object relationships are in a three-dimensional mesh structure, see in particular figure 4,
view ontology-view the object itself definition, such as the name of the table, comments, etc. information.
Looking up-looking at the definition of the object to which the object belongs, e.g., the database to which the table belongs.
Look-down-to see the definition of the object that the object contains, e.g., the fields, indexes, etc. that the table contains.
Look ahead — see an upstream information object of an object, e.g., a source table of data for the table.
Look-back-view downstream information objects of an object, such as a target table of data for the table.
View history-View historical change information for an object. Such as the contents of the table in the last version.
View-friendship-view other objects that have a relationship to the object, such as information related to the script of the table.
In this embodiment, the meta-model management module is further configured with a meta-access analysis module, and the meta-access analysis module is configured to provide a meta-data correlation degree, a meta-data access frequency, and a version comparison. Influence analysis: analyzing the influence of one metadata object on a downstream object downwards; blood margin analysis: tracing the data source of an object upwards in the opposite direction to the direction of influence analysis; full-chain analysis: from a certain object, analysis is carried out in both directions of upstream and downstream: and (3) viability analysis: analyzing the frequency of accessing a database object; isolated object analysis: analyzing isolated metadata objects in a data preparation area (SData) and a physical model area (PData); and (3) consistency analysis: periodically analyzing whether the metadata in the meta-model is consistent with the actual situation; and (3) comparing the versions: selecting versions of any two time points for comparison; and (3) quality analysis: the quality of the metadata in the data warehouse is analyzed.
Because the traditional LDM mostly adopts the three-paradigm modeling, the core table of the LDM has a large number of sources and targets, and once influence analysis is carried out, the result is diffused and amplified. The SQL parser module is adopted to analyze the real source and target of the data, so that the analysis result is more reliable, and the analysis result is refined.
In this embodiment, the increment field is further set as a time field or a self-increment field, and the increment field is used to determine newly added data or modified data.
In this embodiment, the data loading module further completes data loading in a full-scale manner or an incremental manner.
In this embodiment, the meta-model management module sequentially includes, from bottom to top, an object model layer, a base layer, a resource layer, an analysis layer, and a management layer, where the object model layer is configured to provide a basic structure and corresponding type attributes of a class of a meta-data model, the base layer is configured to provide services for other packages residing in a higher hierarchy, the resource layer is configured to serve as source data or target data to provide data resource bidirectional exchange, the analysis layer is configured to provide general meta-data, the management layer is configured to describe a data warehouse information stream and an important event corresponding to the data warehouse information stream, and the important event is specifically a conversion execution event, a measurement event, and a request change event. Referring specifically to fig. 1 and 2, the conversion execution event records the details of the most recent ETL process execution, identifying the time when the ETL process started and ended, and can be used to determine some specific information in the data warehouse related to the process execution status. The metric event can maintain some metric criteria for the model element, such as the true size, estimated size, and projected size that can be used to hold a table, can assist in predicting the scale of the system, and in making decisions. A warehouse process object associates an ETL transformation process with an event, and the set of events is used to trigger the execution of the transformation.
Specifically, the object model layer provides a basic structure and corresponding type attributes for describing classes of the metadata model in other packages, and defines concepts, relations and constraints of the metadata model, including a core package, a behavior package, a relation package and an instance package; wherein the core package contains the base classes and associations used by other packages and is independent of other packages; the behavior package is used for describing the behavior characteristics of the classes in other packages and providing a basis for recording a specific behavior request; the relationship package is used for describing how two objects are related to each other, and defining two types of relationships of generalization and association, wherein the generalization refers to the association and hierarchical structure of the objects with universality and specific objects; association refers to defining a specific relationship between two or more class elements; the instance package provides an infrastructure that includes the valued metadata in the exchange process.
In this embodiment, the base layer further includes a service information package, a data type package, an expression package, a key and index package, a software deployment package, and a type mapping package. The basic layer is used for providing a package for providing characteristic services for other packages residing at a higher level, wherein the service information meta-model provides service-oriented information for all the packages; the data type package is used for providing basic structures required for supporting definition of basic data types and construction of the data types; the expression package is used for providing a uniform expression tree format and is used for serving the conversion package; the key and the index packet are used for providing a unified method for identifying, sorting and searching the elements and can be shared by other packets, wherein the index is an element list arranged in sequence, and the key is a set of one or more values and is used for determining a certain record in the database; the software deployment package is used for managing and recording the distribution and connection condition of each software system and recording how to use the software and hardware in the data warehouse; the type mapping package defines the concept of a type system as a data type set, also supports the conversion of data types among the type systems, mainly aims at the mapping which is carried out to meet the difference of the data types among different systems, and can carry out many-to-many mapping.
In this embodiment, the resource layer further includes an object package, a relational package, a record package, a multidimensional package, and an XML package. The resource layer is used for describing a data resource structure which is used as source data or target data in the exchange process taking a package as an intermediary, and because the object model layer can directly establish object-oriented data resource description and is also used for describing the structure of an object-oriented database and the structure of an object-oriented application component, if features and functions which cannot be processed are met, an extension package can be defined to increase the processing capacity; the relational package is used for describing a schema of a relational database, and the record package is used for providing an infrastructure for describing various record-oriented data structures, wherein the record structure, the record examples and the record files are included; the multidimensional package is used for providing general description about the multidimensional database, and comprises data structures such as dimensions, dimension hierarchies, dimension attributes, dimension members, dimension quantities and the like in the multidimensional model; an XML package defines how data sources in a data repository are described using XML documents, and contains generic classes and associations that are used to describe XML data sources.
In this embodiment, further, the analysis layer includes a conversion package, an OLAP analysis package, a data mining package, an information visualization package, and a business term package. The transformation package includes meta-models related to the ETL process that provide generic metadata describing the ETL tools and ETL behavior, in particular associating the ETL process with data sources and data targets, which can be of any type (relational or object oriented), of any granularity (classes, attributes, tables, columns); the ETL process is allowed to be grouped and executed in parallel to improve the execution efficiency, including the loading condition, the behavior, the steps and the like of the ETL process; defining metadata describing a general concept of the OLAP system, providing a step of mapping the metadata in the OLAP system into a physical data source of a data graph; the data mining package is used for constructing a relatively universal representation method for the data mining model; the information visualization package defines metadata supporting information release and information visualization, and provides support for realizing a more complex visualization mechanism; users of a data warehouse need a good understanding of the information contained in the warehouse, as well as the tools provided by the warehouse, business term packages are used to provide entities and relationships that can express business metadata.
In this embodiment, further, the metadata base includes DDL, SPQ, Source Data, Excel, XML, and Perl. And collecting standardized definition mapping through an Excel template file, and defining the LDM mapping of the information items in the template. The metadata base supports the export of an original template, so that a user can conveniently modify data and then import the data, thereby forming a closed loop of the data and ensuring the transferability and easy maintainability of the metadata; the export of the original template of the analysis result supports the export of the result influencing the blood margin analysis according to the original template, thereby facilitating the user to quickly lock the relevant metadata and export and modify the metadata, and supporting the import of the modified metadata to update the metadata.
In this embodiment, further, the database correspondence table includes a metadata instance table, a metadata combination relation table, and a metadata dependency relation table. The data loading module queries a metadata instance table in a database corresponding table and constructs a mapping table according to the metadata instance table, the data loading module queries the metadata instance table in the database corresponding table and acquires data corresponding to three fields of a serial number, a metadata type and a metadata name of a metadata instance, the data loading module performs hash operation on the serial number of the metadata instance to further acquire a hash value corresponding to the serial number of the data instance, and the data loading module stores the serial number of the metadata instance and the corresponding hash value into the mapping table.
The embodiments of the present invention have been described in detail, but the description is only for the preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. All equivalent changes and modifications made within the scope of the present invention should be covered by the present patent.
Claims (10)
1. A metadata-based management analytics system, comprising:
the meta-model management module is configured with a service unit containing a source system service data table and a mapping management unit, the source system service data table stores a plurality of increment fields, and the mapping management unit is used for creating a mapping relation and a code matching rule of the meta-data;
the metadata acquisition module is used for acquiring metadata and transmitting the acquired metadata to the metadata processing module, and comprises a template management unit, a metadata database and a warehousing auditing unit; the template management unit is used for providing templates for importing and exporting metadata, the metadata base is provided with a plurality of metadata, and the warehousing auditing unit is used for providing warehousing auditing functions for the metadata which changes; and
and the metadata processing module is used for processing the service data table of the source system and integrating the service data table into the metadata base.
2. The metadata-based management analysis system according to claim 1, wherein the metadata processing module comprises a data extraction module, a data cleaning module, a database conversion module and a data loading module, the data extraction module is used for determining a data source and a source attribution thereof, and the data cleaning module is used for processing and eliminating incomplete data, error data and repeated data; the database conversion module is used for null value processing, data splitting and responsible verification, and the data loading module is used for loading the data of the data buffer area into the database corresponding table.
3. The system of claim 1, wherein the meta-model management module is further configured with a meta-access analysis module, and the meta-access analysis module is configured to provide meta-data correlation, meta-data access frequency, and version comparison.
4. The metadata-based management analytics system of claim 1, wherein the delta field is set to a time field or a self-growth field.
5. The metadata-based management analysis system of claim 2, wherein the data loading module performs data loading in a full-scale manner or an incremental manner.
6. The metadata-based management and analysis system according to claim 1, wherein the meta-model management module comprises, in order from bottom to top, an object model layer, a base layer, a resource layer, an analysis layer, and a management layer, the object model layer is configured to provide a basic structure of a class of the meta-data model and corresponding type attributes, the base layer is configured to provide services for other packages residing at a higher level, the resource layer is configured to serve as source data or target data to provide bidirectional exchange of data resources, the analysis layer is configured to provide general metadata, and the management layer is configured to describe important events of a data warehouse information flow and a corresponding data warehouse information flow.
7. The metadata-based management analytics system of claim 6, wherein said base layer comprises a service information package, a data type package, an expression package, a key and index package, a software deployment package, and a type mapping package.
8. The metadata-based management analysis system of claim 6, wherein the analysis layer comprises a transformation package, an OLAP analysis package, a data mining package, an information visualization package, and a business term package.
9. The metadata-based management analysis system of claim 1, wherein the metadata repository comprises DDL, SPQ, Source Data, Excel, XML, and Perl.
10. The metadata-based management analysis system of claim 2, wherein the database correspondence table comprises a metadata instance table, a metadata combination relationship table, and a metadata dependency relationship table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011644798.5A CN112699100A (en) | 2020-12-31 | 2020-12-31 | Management and analysis system based on metadata |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011644798.5A CN112699100A (en) | 2020-12-31 | 2020-12-31 | Management and analysis system based on metadata |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112699100A true CN112699100A (en) | 2021-04-23 |
Family
ID=75514281
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011644798.5A Pending CN112699100A (en) | 2020-12-31 | 2020-12-31 | Management and analysis system based on metadata |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112699100A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113377851A (en) * | 2021-06-11 | 2021-09-10 | 上海妙一生物科技有限公司 | Data processing system and method |
| CN113779132A (en) * | 2021-09-15 | 2021-12-10 | 深圳我家云网络科技有限公司 | Data import method and device, computer equipment and storage medium |
| CN114330998A (en) * | 2021-11-25 | 2022-04-12 | 上海欣兆阳信息科技有限公司 | General data model applied to field of big data marketing and construction method thereof |
| CN114648058A (en) * | 2022-03-03 | 2022-06-21 | 昆明理工大学 | Method and device for processing tin-bismuth metal material data based on metadata |
| CN115587087A (en) * | 2022-12-13 | 2023-01-10 | 四川华西集采电子商务有限公司 | Efficient data sharing platform based on data extraction and system modeling |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104199978A (en) * | 2014-09-24 | 2014-12-10 | 普元信息技术股份有限公司 | System and method for realizing metadata cache and analysis based on NoSQL and method |
| CN104239572A (en) * | 2014-09-30 | 2014-12-24 | 普元信息技术股份有限公司 | System and method for achieving metadata analysis based on distributed cache |
| US9158827B1 (en) * | 2012-02-10 | 2015-10-13 | Analytix Data Services, L.L.C. | Enterprise grade metadata and data mapping management application |
| CN105760520A (en) * | 2016-02-26 | 2016-07-13 | 广州品唯软件有限公司 | Data control platform and architecture |
| CN107657052A (en) * | 2017-10-17 | 2018-02-02 | 上海计算机软件技术开发中心 | A kind of data governing system based on metadata management |
| CN110232098A (en) * | 2019-04-22 | 2019-09-13 | 汇通达网络股份有限公司 | A kind of data warehouse administered based on data and genetic connection designs |
| CN111324602A (en) * | 2020-02-21 | 2020-06-23 | 上海软中信息技术有限公司 | Method for realizing financial big data oriented analysis visualization |
| CN111930821A (en) * | 2020-09-08 | 2020-11-13 | 平安国际智慧城市科技股份有限公司 | One-step data exchange method, device, equipment and storage medium |
| CN112148679A (en) * | 2020-09-21 | 2020-12-29 | 江苏锐创软件技术有限公司 | Data interaction method, system, device and storage medium based on multiple data platforms |
-
2020
- 2020-12-31 CN CN202011644798.5A patent/CN112699100A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9158827B1 (en) * | 2012-02-10 | 2015-10-13 | Analytix Data Services, L.L.C. | Enterprise grade metadata and data mapping management application |
| CN104199978A (en) * | 2014-09-24 | 2014-12-10 | 普元信息技术股份有限公司 | System and method for realizing metadata cache and analysis based on NoSQL and method |
| CN104239572A (en) * | 2014-09-30 | 2014-12-24 | 普元信息技术股份有限公司 | System and method for achieving metadata analysis based on distributed cache |
| CN105760520A (en) * | 2016-02-26 | 2016-07-13 | 广州品唯软件有限公司 | Data control platform and architecture |
| CN107657052A (en) * | 2017-10-17 | 2018-02-02 | 上海计算机软件技术开发中心 | A kind of data governing system based on metadata management |
| CN110232098A (en) * | 2019-04-22 | 2019-09-13 | 汇通达网络股份有限公司 | A kind of data warehouse administered based on data and genetic connection designs |
| CN111324602A (en) * | 2020-02-21 | 2020-06-23 | 上海软中信息技术有限公司 | Method for realizing financial big data oriented analysis visualization |
| CN111930821A (en) * | 2020-09-08 | 2020-11-13 | 平安国际智慧城市科技股份有限公司 | One-step data exchange method, device, equipment and storage medium |
| CN112148679A (en) * | 2020-09-21 | 2020-12-29 | 江苏锐创软件技术有限公司 | Data interaction method, system, device and storage medium based on multiple data platforms |
Non-Patent Citations (1)
| Title |
|---|
| 雷启明: "基于CWM的数据仓库元数据管理策略研究", 《商场现代化》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113377851A (en) * | 2021-06-11 | 2021-09-10 | 上海妙一生物科技有限公司 | Data processing system and method |
| CN113779132A (en) * | 2021-09-15 | 2021-12-10 | 深圳我家云网络科技有限公司 | Data import method and device, computer equipment and storage medium |
| CN114330998A (en) * | 2021-11-25 | 2022-04-12 | 上海欣兆阳信息科技有限公司 | General data model applied to field of big data marketing and construction method thereof |
| CN114648058A (en) * | 2022-03-03 | 2022-06-21 | 昆明理工大学 | Method and device for processing tin-bismuth metal material data based on metadata |
| CN115587087A (en) * | 2022-12-13 | 2023-01-10 | 四川华西集采电子商务有限公司 | Efficient data sharing platform based on data extraction and system modeling |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11409764B2 (en) | System for data management in a large scale data repository | |
| US11360950B2 (en) | System for analysing data relationships to support data query execution | |
| US11461294B2 (en) | System for importing data into a data repository | |
| US20200151202A1 (en) | Data relationships storage platform | |
| CN106095862B (en) | Storage method of centralized extensible fusion type multi-dimensional complex structure relation data | |
| CN112699100A (en) | Management and analysis system based on metadata | |
| Gupta et al. | Cloud computing and big data analytics: what is new from databases perspective? | |
| CN109542967B (en) | Smart city data sharing system and method based on XBRL standard | |
| CN112199433A (en) | Data management system for city-level data middling station | |
| CN111813870B (en) | Machine learning algorithm resource sharing method and system based on unified description expression | |
| CN112926852A (en) | Atmospheric ecological environment analysis method based on data fusion | |
| Mesiti et al. | Towards a user-friendly loading system for the analysis of big data in the internet of things | |
| CN114925054A (en) | Meta-model-based metadata management system and method | |
| Peng et al. | Using object deputy model to prepare data for data warehousing | |
| Zgolli et al. | Metadata in data lake ecosystems | |
| Başaran | A comparison of data warehouse design models | |
| Powell | Oracle high performance tuning for 9i and 10g | |
| Huawei Technologies Co., Ltd. | Database Design Fundamentals | |
| Shah | A Novel Framework for Data Acquisition and Retrieval Using Hierarchical Schema Over Structured Big Data | |
| Mokhtari | Advanced Databases | |
| CN118796903A (en) | Metadata management system and method for heterogeneous data sources | |
| Hou | Analysis and research on the difference between data warehouse and database | |
| Govil | AN OVERVIEW OF DATABASE MODELING | |
| Martelli | The temporal dimension in data bases | |
| Singh et al. | Tools Used in Big Data Analytics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210423 |