CN107491476B - Data model conversion and query analysis method suitable for various big data management systems - Google Patents
Data model conversion and query analysis method suitable for various big data management systems Download PDFInfo
- Publication number
- CN107491476B CN107491476B CN201710515379.3A CN201710515379A CN107491476B CN 107491476 B CN107491476 B CN 107491476B CN 201710515379 A CN201710515379 A CN 201710515379A CN 107491476 B CN107491476 B CN 107491476B
- Authority
- CN
- China
- Prior art keywords
- model
- data
- document
- associated document
- association
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Abstract
The invention relates to a data model conversion and query analysis method suitable for various big data management systems. The method comprises the following steps: 1) establishing an association document model which comprises a document set and an association set, wherein the association set is a set formed by the association between the documents; 2) converting different original data models into associated document models according to semantic information of the data; 3) and converting the data in the associated document model into a data structure which can be accepted by the distributed computation programming model, and further converting the associated document model into the distributed computation programming model. And then uniformly querying and analyzing the data from different original data models by using the distributed computing programming model obtained by converting the associated document model. The invention can realize the uniform access, query and analysis of the multi-source heterogeneous data source.
Description
Technical Field
The invention relates to a data model, in particular to a data model conversion and query analysis method suitable for a big data management system to perform management analysis, and belongs to the technical field of big data and databases.
Background
With the continuous popularization of computers, the demands for data management and processing are increasingly urgent, people propose different data models aiming at different data forms and characteristics, and realize the management and analysis of data by a corresponding data management system. More influential data models, such as the E-R model, have dominated the database world for over 40 years since the 70's of the last century. In recent decades, with the penetration of internet and internet of things applications, the generation of large-scale structured, semi-structured, unstructured data has triggered NoSQL movements [ Cattell r. The database world is transformed from the first SQL monopoly scenario to the traditional SQL, NoSQL, NewSQL divide and conquer scenario.
At present, popular data models include a relationship model, a key-value model, a document model, an attribute graph model and the like. The database maps the things in the actual system into a preset data model for storage management. The query and analysis of the data are performed by corresponding operations on the data model. The relational model is based on relational algebra, and the main operations comprise set operation, connection operation, selection operation and projection operation; the main operation of the document model (taking MongDB as an example) is the selection and projection of documents; the main operations of the key-value model (Redis as an example) are put and get, and in addition, it also includes operations for the management of different types of keys and values; the main operations of the attribute graph model (in Titan for example) are graph traversal, selection, and projection.
In the context of big data, it is a distributed computing programming model (programming model for distributed computing) that is closely related to the data model. The currently popular programming models are map/reduce, spark RDD, Pergel, DraydLNQ, and the like. Most of these distributed programming models feature mapping of the problem to be analyzed (e.g., pagerank, rank, k-means, etc.) to the data model specified by the programming model. For example, when performing pagerank operation on a graph structure by using map/reduce, the graph structure is firstly mapped into an array formed by key-values, and then data parallel operation is performed on the array according to a map/reduce programming model.
It can be seen that, the existing big data processing scheme has the defects that different data model databases are independent from each other, and the basic operation of the data model is disjointed with the data analysis. For example, data in one system may be stored in data sources such as MySQL, MongoDB, Titan, Hbase, file system, etc., and how to combine and cooperate the data in these data sources requires a developer to perform a series of ETL (abbreviation of Extract-Transform-Load, process of index data extraction (Extract), Transform (Transform), and Load (Load)) transformation programming; meanwhile, the operation result of the local database on the data cannot be directly used as the input of the distributed analysis tool, when the map/reduce programming model is required to be used for operation, developers need to process the data into a key-value array, and when the Pergel is required to be used, the data needs to be converted into a graph structure. While some solutions already exist to mask the differences of different data models, these solutions tend to take one into account. For example, presto, Spark DataFrame can map any data into a relational model and then perform a unified SQL query, but it cannot provide operations of a non-relational model, and presto cannot perform data analysis.
Based on the above background, the present invention provides a new data model for implementing unified access, query and analysis of multi-source heterogeneous data sources.
Disclosure of Invention
The invention aims to provide a data Model conversion and query analysis method suitable for various big data management systems, which mainly relates to a data Model called as an associated Document Model (LDM), wherein the data Model can represent semantic information of a relation Model, a Document Model, an attribute graph Model and the like, and can Model a plurality of problems in big data analysis and convert data into a format acceptable by an analysis tool, thereby achieving the purpose of uniform query analysis.
The technical scheme adopted by the invention is as follows:
a data model conversion method suitable for various big data management systems comprises the following steps:
1) establishing an association document model which comprises a document set and an association set, wherein the association set is a set formed by the association between the documents;
2) converting different original data models into associated document models according to semantic information of the data;
3) and converting the data in the associated document model into a data structure which can be accepted by the distributed computation programming model, and further converting the associated document model into the distributed computation programming model.
Further, the data from different original data models are uniformly queried and analyzed by using the distributed computing programming model converted from the associated document model.
Further, the documents in the document set of the associated document model are a set formed by a group of attributes, and the attributes are sets formed by the same type of data; each document default contains a main code attribute, and the main code attribute is a global unique identifier; the document set and the association set have a name identifier to describe the semantics of the documents and associations in the set.
Further, the raw data model comprises one or more of a relationship model, a key-value model, a document model, a property graph model, and the like; the distributed computing programming model includes one or more of map/reduce, spark SQL, Pergel, etc.
Further, the distributed computing programming model can accept data structures including arrays, tables, graphs, and the like.
Further, when the data in the associated document model is converted into an array, the data in the associated document model is combined into elements of the array, and which data in the associated document model is used, which data is key and which data is value are described; when data in the associated document model is converted into a table, taking a document set in the associated document model as the table and explaining which type of document in the associated document model is used; when converting data in the associated document model into a graph, documents and associations in the associated document model are combined into a graph structure, and which documents and associations in the associated document model are used are explained.
Further, defining an algorithm of the associated document model based on the operation of the original data model, wherein the algorithm comprises set operation, associated operation and document operation; the set operation comprises a parallel operation, an intersection operation and a difference operation; the correlation operation comprises the steps of establishing correlation, selecting operation and projecting operation; the document operation comprises selection operation and projection operation.
Furthermore, all data sources are connected into a whole through the associated document model to form a network, and the data in the network is uniformly accessed in a URL (uniform resource locator) similar form.
The invention has three using modes: firstly, developing a corresponding query engine according to the operation of LDM; second, use LDM as an ETL (Extract-Transform-Load) tool that interacts with existing analysis tools; thirdly, a distributed computing model is designed according to the LDM, and unified query analysis of data is realized on the basis.
Compared with the prior art, the invention has the advantages that:
1) a data model special for unified query analysis is proposed for the first time. Existing data models are typically query analyzed against a single data model, such as spark RDD, Pergel, mogoDB, and the like. There are also some schemes that want to query different data models in a uniform manner, such as presto, spark dataframe, impala, etc., and these techniques uniformly map different data models into a relational model and query the data models using SQL. It is conceivable that these schemes undoubtedly lose some of the data model operations.
2) The method has the function of providing data for the mainstream distributed computing model. The model can use simple URLs to combine data in the data model together and can combine the data into data structures that are acceptable for distributed computing models, such as arrays, tables, graph structures, and the like.
3) The data sources are connected as a network. The data model provided by the invention can connect data in different databases together through the mapping rule. For example, it can represent the data of tables, documents, nodes, foreign keys, edges, etc. in a database in a uniform manner, so as to connect them into a whole like a network. At the same time, the resources in the network can be accessed uniformly like the URL.
Drawings
FIG. 1 is a flow chart diagram of a data model transformation method.
Fig. 2 is a schematic structural diagram of an LDM model.
FIG. 3 is a schematic diagram of a corresponding query engine developed from LDM's operations.
FIG. 4 is a schematic diagram of using LDM as an ETL tool for interacting with existing analysis tools.
Detailed Description
The present invention will be described in further detail below with reference to specific examples and the accompanying drawings.
FIG. 1 is a flow chart of a data model conversion process applicable to various big data management systems, mainly including: 1) establishing an associated Document Model (Linked Document Model) which comprises a Document set and an associated set, wherein the associated set is a set formed by the association between the documents; 2) converting different original data models into associated document models according to semantic information of the data; 3) and converting the data in the associated document model into a data structure which can be accepted by the distributed computation programming model, and further converting the associated document model into the distributed computation programming model. Then, the distributed computing programming model obtained by the conversion of the associated document model can be used for uniformly inquiring and analyzing the data from different original data models.
The data Model Linked Document Model provided by the invention comprises the following parts: definition of Linked Document Model, conversion rule of Linked Document Model and other models, operation rule of Linked Document Model, and data access mode of Linked Document Model.
1. Linked Document model definition
A document is a collection of a set of attributes, which are collections of the same type of data. Each document contains by default a uniquely identified master code attribute. The main code attribute is similar to the function of an IP address and must be a global unique identifier; the type of other attributes may be arbitrary, including a document, association, custom type, etc. An association is a special document that must contain two attributes (from: main code, to: main code) to represent an association between documents, which refers to a relationship between two pieces of data, such as a knows association between one person document and another person document representing that the first person knows the second person. Both the document set and the association set must have a name identifier to account for the semantics of the documents and associations in the set. The number of attributes in the same type of document or association may be different, which means that { 'id': fff0 ',' name ': blue joe', 'age': 30} can be a member of both person class documents and teacher class documents.
The LDM model is a binary set (document set, association set) composed of a document set and an association set, wherein the association set is a plurality of relationship sets between two types of documents. The general structure of the LDM model is shown in fig. 2. Wherein Documents represents a document set, Links represents an association set, PersonDocument represents a document set such as a person, software document represents a document set such as a software document set, InventLink represents a set of associations such as a person invented software, 1 and 2 represent main codes of unique identifiers of Documents, and attr1 and attr2 … represent attributes of the Documents.
2.LDM conversion rules
LDM is directed to the query and analysis of data, which provides two types of transformation rules: conversion of raw data model to LDM, conversion of LDM to existing programming model requires format conversion.
1) Original data model → LDM
The formal definition of the data model transformation is (G, L, M), where G represents Schema of the global model, i.e. LDM, L represents the local data model (relational model, key-value model, document model, property graph model), and M represents the mapping rule from L to G. The conversion of the original data model into the LDM mainly considers the semantics of the data, and the conversion at the data type level can be determined by developers according to the system requirements. The transformations given below include the raw data models of relational, key-value, document, and property graph models, with the main transformation rules as shown in table 1. The self-defined conversion rule is to extract a data set meeting certain characteristics according to the characteristics of the original data model. For example, data containing Person's keys in the key-value model is extracted to be used as a Person class document set; extracting a peak with a legacy of Person in the attribute graph model as a Person class document; the relation that personid of the Person class document in the document model is equal to personid in the Software document is extracted as a connection set invent.
TABLE 1 conversion rules of raw data model to LDM
| LDM | Relational model | Key-value model | Document model | Attribute graph model |
| Properties | Properties | Key | Properties | Properties |
| Document | Recording | Pair | Document | Vertex point |
| Document collection | Watch (A) | Self-defining | Collection | Self-defining |
| Connection of | External key | Self-defining | Self-defining | Edge |
| Connection set | External key | Self-defining | Self-defining | Self-defining |
It should be noted that in LDM, both the document set and the association set must have a name, so during the conversion process for the foreign key and other custom parts of the relationship model, a name must be provided by the converter as the semantic of the collection element. For example, in the attribute graph model, a node with a stable of 'person' may be used as a person class document in the LDM; it is also possible to let the node containing the attribute 'teacher' be the teacher class document in the LDM, and in fact both classes of documents may correspond to the same node.
In addition, the conversion of the original model to the LDM may not be limited to the above model, and a developer may define conversion rules of other data models to the LDM, such as a file system, a column database, etc., according to requirements.
2) LDM → Programming model
The conversion of LDM into programming models primarily takes into account relationships on the data structure. The data structures accepted by the current popular programming models such as map/reduce, spark SQL, Pergel and the like mainly comprise arrays, tables and graphs. The rules for the conversion of LDM to these three data structures are therefore given below, as shown in Table 2.
TABLE 2 conversion rules of LDM to array, table, graph
3.LDM operation rule
The operation rule of the LDM is defined based on the operation of a relationship model, a key-value model, a document model and an attribute graph model. The method comprises the steps of set operation, connection operation, selection operation and projection operation of a relation model; get operation of the key-value model; selecting a document model and performing projection operation; and traversing the attribute graph model and selecting operation. Algorithms of the LDM model are mainly divided into three categories: the specific operation rules of the set operation, the association operation and the document operation are shown in table 3.
TABLE 3 operational rules of LDM
4. LDM data access rules
Since LDM links databases to a network, we can use a URL-like format to reference data in the network. This URL is of the form:
<datasource>.<document>.<link>.<identity>.<propertyName>
the data source represents a data source, such as MySQL, MongoDB and the like, the document represents a document mapped from the data source to the LDM, the link represents the association mapped from the data source to the LDM, the identity represents the main code of the document, and the propertylame represents the attribute name of the document.
Data can be referenced at different levels, for example, a reference to the name property of a person document in a MySQL database can be expressed as:
MySQL.person.name
applying the gather association of a person document in a MongoDB database can be expressed as:
MongoDB.person.father
the association represents the document set corresponding to the association, and further reference can be carried out, such as
MongoDB.person.father.name
The data corresponding to the data reference URL is actually the result after the relational operation and the projection operation of the LDM. For example, the data represented by mongodb.
The invention has three using modes: firstly, developing a corresponding query engine according to the operation of LDM; second, using LDM as an ETL tool to interact with existing analysis tools; thirdly, a distributed computing model is designed according to the LDM, and unified query analysis of data is realized on the basis.
Example 1: developing corresponding query engine according to LDM operation
As shown in FIG. 3, a developer can compute a custom structured query language based on associated documents, incorporating domain knowledge. In the process of executing the query, the query statement is converted into a series of operations of the associated documents, and finally the associated documents combined by the multi-element databases are operated through the operations. As shown in fig. 3, these multiple databases include relational database tables, Key-value databases, graph databases, document databases, etc., and the data in each database is only an example and has no special meaning, so that the meaning of the data is not explained one by one.
Example 2: using LDM as an ETL tool to interact with existing analysis tools
As shown in FIG. 4, a developer may implement the mapping rules between the associated document and the raw data model and the three data structures. And integrating and converting the data in the metadata model into data of a target type according to the requirements of users, and finally transmitting the data to a series of data analysis tools such as Spark DataFrame, Spark Graphx, TensorFlow and the like in a pipeline or Driver mode for use.
Example 3: designing a distributed computing model according to LDM (laser direct memory access), and realizing unified query analysis of data on the basis
Spark realizes a distributed array RDD, developers can realize a distributed LDM memory management platform, and relevant basic operation and operation interfaces of LDM are provided. Developers using the platform can directly use the interface to write data query and machine learning algorithms.
The above embodiments are only intended to illustrate the technical solution of the present invention and not to limit the same, and a person skilled in the art can modify the technical solution of the present invention or substitute the same without departing from the spirit and scope of the present invention, and the scope of the present invention should be determined by the claims.
Claims (7)
1. A data model conversion method suitable for various big data management systems comprises the following steps:
1) establishing an association document model which comprises a document set and an association set, wherein the association set is a set formed by the association between the documents; the documents in the document set of the associated document model are a set formed by a group of attributes, and the attributes are sets formed by the same type of data; each document default contains a main code attribute, and the main code attribute is a global unique identifier; the document set and the association set have a name identifier to explain the semantics of the documents and the associations in the set;
2) converting different original data models into associated document models according to semantic information of the data;
3) converting the data in the associated document model into a data structure which can be accepted by the distributed computation programming model, and further converting the associated document model into the distributed computation programming model;
the data structure which can be accepted by the distributed computing programming model comprises an array, a table and a graph; when the data in the associated document model is converted into an array, combining the data in the associated document model into elements of the array, and explaining which data in the associated document model is used, which data is key and which data is value; when data in the associated document model is converted into a table, taking a document set in the associated document model as the table and explaining which type of document in the associated document model is used;
when converting data in the associated document model into a graph, documents and associations in the associated document model are combined into a graph structure, and which documents and associations in the associated document model are used are explained.
2. The method of claim 1, wherein data from different raw data models is uniformly queried and analyzed using a distributed computational programming model transformed from an associated document model.
3. The method of claim 1, wherein the raw data model comprises one or more of a relational model, a key-value model, a document model, a property graph model; the distributed computing programming model comprises one or more of map/reduce, spark SQL and Pergel.
4. The method of claim 1, wherein the algorithms associated with the document model are defined based on operations of the raw data model, including set operations, association operations, document operations; the set operation comprises a parallel operation, an intersection operation and a difference operation; the correlation operation comprises the steps of establishing correlation, selecting operation and projecting operation; the document operation comprises selection operation and projection operation.
5. The method of claim 1, wherein the data sources are connected into a whole through the associated document model to form a network, and the data in the network is uniformly accessed by using a form similar to a URL.
6. The method of claim 1, wherein the respective query engines are developed based on operations of the associated document model.
7. The method of claim 1, using a relevance document model as an ETL tool for interacting with existing analysis tools.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710515379.3A CN107491476B (en) | 2017-06-29 | 2017-06-29 | Data model conversion and query analysis method suitable for various big data management systems |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710515379.3A CN107491476B (en) | 2017-06-29 | 2017-06-29 | Data model conversion and query analysis method suitable for various big data management systems |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107491476A CN107491476A (en) | 2017-12-19 |
| CN107491476B true CN107491476B (en) | 2021-01-12 |
Family
ID=60643499
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710515379.3A Active CN107491476B (en) | 2017-06-29 | 2017-06-29 | Data model conversion and query analysis method suitable for various big data management systems |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107491476B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108133000B (en) * | 2017-12-21 | 2021-05-04 | 百度在线网络技术（北京）有限公司 | Metadata storage method and device and server |
| CN108376174A (en) * | 2018-02-27 | 2018-08-07 | 河北中科开元数据科技有限公司 | The method and apparatus for supporting structuring to be merged with unstructured big data |
| CN109241054A (en) * | 2018-08-02 | 2019-01-18 | 成都松米科技有限公司 | A kind of multimodal data library system, implementation method and server |
| CN112306987A (en) * | 2020-01-19 | 2021-02-02 | 深圳新阳蓝光能源科技股份有限公司 | Data management method and device and electronic equipment |
| CN112000851B (en) * | 2020-08-28 | 2023-03-28 | 北京计算机技术及应用研究所 | Key value model, document model and graph model data unified storage method |
| CN112364033B (en) * | 2021-01-13 | 2021-04-13 | 北京云真信科技有限公司 | Data retrieval system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104182454A (en) * | 2014-07-04 | 2014-12-03 | 重庆科技学院 | Multi-source heterogeneous data semantic integration model constructed based on domain ontology and method |
| CN104933095A (en) * | 2015-05-22 | 2015-09-23 | 中国电子科技集团公司第十研究所 | Heterogeneous information universality correlation analysis system and analysis method thereof |
| US20160179979A1 (en) * | 2014-12-22 | 2016-06-23 | Franz, Inc. | Semantic indexing engine |
| CN106227800A (en) * | 2016-07-21 | 2016-12-14 | 中国科学院软件研究所 | The storage method of the big data of a kind of highlights correlations and management system |
-
2017
- 2017-06-29 CN CN201710515379.3A patent/CN107491476B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104182454A (en) * | 2014-07-04 | 2014-12-03 | 重庆科技学院 | Multi-source heterogeneous data semantic integration model constructed based on domain ontology and method |
| US20160179979A1 (en) * | 2014-12-22 | 2016-06-23 | Franz, Inc. | Semantic indexing engine |
| CN104933095A (en) * | 2015-05-22 | 2015-09-23 | 中国电子科技集团公司第十研究所 | Heterogeneous information universality correlation analysis system and analysis method thereof |
| CN106227800A (en) * | 2016-07-21 | 2016-12-14 | 中国科学院软件研究所 | The storage method of the big data of a kind of highlights correlations and management system |
Non-Patent Citations (2)
| Title |
|---|
| 《基于HBase的RDF数据存储方案研究与设计》;王媛媛、吕晓丹等;《信息网络安全》;20160331;第59-63页 * |
| 《大数据环境下的动态知识网络模型及构建方法》;刘剑、许洪波 等;《计算机研究与发展》;20141231;第86-93页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107491476A (en) | 2017-12-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107491476B (en) | Data model conversion and query analysis method suitable for various big data management systems | |
| US10049143B2 (en) | Ontology harmonization and mediation systems and methods | |
| CN107515887B (en) | Interactive query method suitable for various big data management systems | |
| Busse et al. | Federated information systems: Concepts, terminology and architectures | |
| Hartig et al. | Publishing and consuming provenance metadata on the web of linked data | |
| Zhao et al. | Modeling MongoDB with relational model | |
| CN106934062A (en) | A kind of realization method and system of inquiry elasticsearch | |
| US20070061318A1 (en) | System and method of data source agnostic querying | |
| Slepicka et al. | KR2RML: An Alternative Interpretation of R2RML for Heterogenous Sources. | |
| Comyn-Wattiau et al. | Model driven reverse engineering of NoSQL property graph databases: The case of Neo4j | |
| Unbehauen et al. | Knowledge extraction from structured sources | |
| Touma et al. | Supporting data integration tasks with semi-automatic ontology construction | |
| Michel et al. | A generic mapping-based query translation from SPARQL to various target database query languages | |
| Pokorný | Integration of relational and NoSQL databases | |
| Pokorný | Integration of relational and NoSQL databases | |
| Chari | Model composition in a distributed environment | |
| Ramanujam et al. | R2D: A bridge between the semantic web and relational visualization tools | |
| US20050060307A1 (en) | System, method, and service for datatype caching, resolving, and escalating an SQL template with references | |
| Salas et al. | RDB2RDF plugin: relational databases to RDF plugin for eclipse | |
| Ho et al. | An approach of transforming ontologies into relational databases | |
| RU2605387C2 (en) | Method and system for storing graphs data | |
| Kim et al. | Semantic query transformations for increased parallelization in distributed knowledge graph query processing | |
| Comito et al. | XML data integration in OGSA grids | |
| Hosain et al. | An algebraic language for semantic data integration on the hidden web | |
| Su-Cheng et al. | Mapping of extensible markup language-to-ontology representation for effective data integration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |