CN107526790A - A kind of implementation based on the database language technology for realizing data unified standard - Google Patents

A kind of implementation based on the database language technology for realizing data unified standard Download PDF

Info

Publication number
CN107526790A
CN107526790A CN201710696246.0A CN201710696246A CN107526790A CN 107526790 A CN107526790 A CN 107526790A CN 201710696246 A CN201710696246 A CN 201710696246A CN 107526790 A CN107526790 A CN 107526790A
Authority
CN
China
Prior art keywords
data
unified standard
database language
cigql
realizing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201710696246.0A
Other languages
Chinese (zh)
Inventor
王成
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yuan Jianing
Original Assignee
Yuan Jianing
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yuan Jianing filed Critical Yuan Jianing
Priority to CN201710696246.0A priority Critical patent/CN107526790A/en
Publication of CN107526790A publication Critical patent/CN107526790A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • G06F16/258Data format conversion from or to a database

Abstract

The invention discloses a kind of implementation based on the database language technology for realizing data unified standard, including:By data extraction module, source data is extracted according to different data types, and the data after extraction are stored in the way of FIFO in CigQL databases;By data transformation module, to the data after extraction logically, arithmetic and data attribute enter line translation, and Hash tables data, the metadata after conversion be stored in into Index areas, Metedata areas respectively;By data package module, the data for being stored in Index areas and Metedata areas are packaged according to CigQL agreements, form the data of unified standard;By data export module, the data for forming unified standard are exported according to purposes classification, and the data after export are deposited to Data Switch areas, so that rear end is called to data.Implementation provided by the present invention based on the database language technology for realizing data unified standard, converts the data into unified standard, solves the problems, such as that isomeric data is difficult to fusion.

Description

A kind of implementation based on the database language technology for realizing data unified standard
Technical field
The present invention relates to computer information technology field, more particularly to it is a kind of based on the number for realizing data unified standard According to the implementation of storehouse language technology.
Background technology
With the arriving of information age, database develops towards complication and diversification direction.Due between various databases Form and standard be skimble-scamble, so result in the structure of each database, document format data and other data Stock is incompatible the problem of, and database used in each client possesses the data representation of uniqueness, and each should With all corresponding to disparate databases.
In the prior art, although under specific circumstances can be by incompatible data by some special softwares or instrument Storehouse connects, but its workload is huge, consumes sizable time and energy.
Therefore, prior art has yet to be improved and developed.
The content of the invention
The technical problem to be solved in the present invention is, for the drawbacks described above of prior art, there is provided one kind is based on realizing number According to the implementation of the database language technology of unified standard, it is intended to solve the database of different structure and form in the prior art Between can not be compatible the problem of.
The technical proposal for solving the technical problem of the invention is as follows:
A kind of implementation based on the database language technology for realizing data unified standard, wherein, it is described based on realization The implementation of the database language technology of data unified standard comprises the following steps:
A, by data extraction module, source data is extracted according to different data types, and by the number after extraction According to being stored in the way of FIFO in CigQL databases;
B, by data transformation module, to the data after extraction logically, arithmetic and data attribute enter line translation, and will Hash tables data, metadata after conversion are stored in Index areas, Metedata areas respectively;
C, by data package module, the data for being stored in Index areas and Metedata areas are carried out according to CigQL agreements Encapsulation, form the data of unified standard;
D, by data export module, the data for forming unified standard are exported according to purposes classification, and will export Data afterwards are deposited to Data Switch areas, so that rear end is called to data.
Preferably, the implementation based on the database language technology for realizing data unified standard, wherein, it is described Also include step E before step A:
E, module is declared by variable, variable statement is carried out to source data.
Preferably, the implementation based on the database language technology for realizing data unified standard, wherein, it is described The element that variable statement includes is structure, single argument and set of variables.
Preferably, the implementation based on the database language technology for realizing data unified standard, wherein, it is described Single argument includes:Integer, floating number, character, character string, memory block, date, time, moment, logical number and enumerator.
Preferably, the implementation based on the database language technology for realizing data unified standard, wherein, it is described Step A is specifically included:
Data are identified by A1, the data type for obtaining source data first according to the position of data type and attribute;
A2, according to the data attribute identified, data are divided into level one data and secondary data;
A3, level one data and secondary data are extracted respectively, and the data after extraction are saved in CigQL databases In.
Preferably, the implementation based on the database language technology for realizing data unified standard, wherein, it is described Step B is specifically included:
B1, the data after extraction are torn open using conditional statement, Do statement, case statement and skip instruction to data Divide, calculate, restructuring and form are changed;
B2, data are being split, calculate, recombinated and when form is changed, logically, arithmetic and data attribute logarithm According to progress assignment;
B3, after being converted according to assignment to data, Hash tables data and metadata are formed respectively.
Preferably, the implementation based on the database language technology for realizing data unified standard, wherein, it is described CigQL agreements are XML tunnelings.
Preferably, the implementation based on the database language technology for realizing data unified standard, wherein, it is described Step D is specifically included:
After the connection of D1, CigQL data store internal, data export request is sent;
D2, exported and asked according to data, confirm the logical place of export data, by CigQL standards, open data Api interface, derived data write-in buffer area will be treated, derived data export then will be treated in buffer area;
After D3, pending data export, the connection of CigQL data store internals is disconnected.
Preferably, the implementation based on the database language technology for realizing data unified standard, wherein, it is described Data Switch areas are provided with multiple Data Subject subregions, and different types of data are stored in corresponding Data Subject subregions.
Compared with prior art, the reality provided by the present invention based on the database language technology for realizing data unified standard Existing mode, comprises the following steps:A, by data extraction module, source data is extracted according to different data types, and Data after extraction are stored in the way of FIFO in CigQL databases;B, by data transformation module, after extraction Data are logically, arithmetic and data attribute enter line translation, and Hash tables data, the metadata after conversion are stored in respectively Index areas, Metedata areas;C, by data package module, to be stored in the data in Index areas and Metedata areas according to CigQL agreements are packaged, and form the data of unified standard;D, by data export module, the data to forming unified standard Exported according to purposes classification, and the data after export are deposited to Data Switch areas, so that rear end is adjusted to data With the database data of different-format to be converted into the normal data of uniform format, realizes the connection between data in different formats It is dynamic, quickly solve the problems, such as that isomeric data is difficult to fusion.
Brief description of the drawings
Fig. 1 is that a kind of implementation based on the database language technology for realizing data unified standard of the present invention is preferably implemented The flow chart of example.
Fig. 2 is that a kind of implementation based on the database language technology for realizing data unified standard of the present invention is preferably implemented The middle CigQL language of example is formed.
Fig. 3 is that a kind of implementation based on the database language technology for realizing data unified standard of the present invention is another preferably The flow chart of embodiment.
Fig. 4 is that a kind of implementation based on the database language technology for realizing data unified standard of the present invention is preferably implemented First split flow figure of example.
Fig. 5 is that a kind of implementation based on the database language technology for realizing data unified standard of the present invention is preferably implemented Second split flow figure of example.
Fig. 6 is that a kind of implementation based on the database language technology for realizing data unified standard of the present invention is preferably implemented 3rd split flow figure of example.
Embodiment
To make the objects, technical solutions and advantages of the present invention clearer, clear and definite, develop simultaneously embodiment pair referring to the drawings The present invention is further described.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, and do not have to It is of the invention in limiting.
SQL SQL (Structured Query Language) is a kind of advanced, wide variety of non-mistake Journey programming language, either Oracle, Sybase, Informix, SQL Server etc. large-scale Database Systems, still The database development system that Visual Foxpro, PowerBuilder etc. are commonly used on PC, all support sql like language as inquiry Language.Same support sql like language of the invention, this allows more users not changing original situation about being accustomed to using database Under, it more can easily use CigQL.
General interactive query language CigQL (Changed interact generic Query Language) be The database language technology of invention is designed on the basis of SQL, it is allowed to which user is worked in high level data structure, and it is not required User specifies the deposit method to data, it is not required that user understands specific data location mode, therefore, is totally different from bottom The disparate databases system of Rotating fields can also use identical CigQL as data input and the interface of management.
Fig. 1 is that a kind of implementation based on the database language technology for realizing data unified standard of the present invention is preferably implemented Example flow chart, as shown in figure 1, present pre-ferred embodiments provide it is a kind of based on realize data unified standard database language The implementation of speech technology, wherein, the implementation based on the database language technology for realizing data unified standard includes Following steps:
S100, by data extraction module, source data is extracted according to different data types, and by after extraction Data are stored in the way of FIFO in CigQL databases;
S200, by data transformation module, to the data after extraction logically, arithmetic and data attribute enter line translation, And Hash tables data, the metadata after conversion are stored in Index areas, Metedata areas respectively;
S300, by data package module, to being stored in the data in Index areas and Metedata areas according to CigQL agreements It is packaged, forms the data of unified standard;
S400, by data export module, the data for forming unified standard are exported according to purposes classification, and will lead Data after going out are deposited to Data Switch areas, so that rear end is called to data.
When it is implemented, CigQL design is broadly divided into 3 parts:Language is formed, Language Composition Module, abbreviation LCM;Functional module, Functional Composition Module, abbreviation FCM;Structural model, Structure Composition Module, abbreviation SCM.
FIFO, First Input First Output abbreviation, First Input First Output, this is a kind of traditional sequentially to hold Row method, the instruction being introduced into first are completed and retired from office, and and then just perform Article 2 instruction.
Fig. 2 is that a kind of implementation based on the database language technology for realizing data unified standard of the present invention is preferably implemented The middle CigQL language of example is formed, as shown in Fig. 2 LCM is used to business event be described, also referred to as data exchange service Description language, be CigQL basis.According to the needs of service logic, the operation of data exchange mainly includes data pick-up, number According to the operation of four kinds of fundamental types such as conversion, data encapsulation, data export.
Fig. 3 is that a kind of implementation based on the database language technology for realizing data unified standard of the present invention is another preferably The flow chart of embodiment, as shown in figure 3, in the further preferred embodiment of the present invention, also include step before the step S100 S500:
S500, by variable declare module, to source data carry out variable statement.
When it is implemented, during CigQL operations are performed, it is also necessary to pass through the memory cell (variable) of some definition Preserve the data involved by data pick-up, data conversion, data encapsulation and data export.
In further preferred embodiment of the invention, the element that the variable statement includes is structure, single argument and set of variables.
In further preferred embodiment of the invention, the single argument includes:Integer, floating number, character, character string, internal memory Block, date, time, moment, logical number and enumerator.
When it is implemented, in CigQL language, basic variable include simple variable, complicated variable, set of variables and Structure variable.
Fig. 4 is that a kind of implementation based on the database language technology for realizing data unified standard of the present invention is preferably implemented First split flow figure of example, as shown in figure 4, in the further preferred embodiment of the present invention, the step S100 is specifically included:
Data are identified by S110, the data type for obtaining source data first according to the position of data type and attribute;
S120, according to the data attribute identified, data are divided into level one data and secondary data;
S130, level one data and secondary data are extracted respectively, and the data after extraction are saved in CigQL data In storehouse.
When it is implemented, data pick-up is mainly by the link of CigQL database resources, digital independent and connection disconnection group Into, for CigQL provide read resource data ability.To manipulate different types of data resource according to unified approach, it is necessary to The operating method of data resource is packaged, forms unified effective manipulation interface.According to the existence form of data, data are taken out This 4 parts by data resource, Service Source, file resource and Policies Resource are taken to form.
Fig. 5 is that a kind of implementation based on the database language technology for realizing data unified standard of the present invention is preferably implemented Second split flow figure of example, as shown in figure 5, in the further preferred embodiment of the present invention, the step S200 is specifically included:
S210, the data after extraction are carried out using conditional statement, Do statement, case statement and skip instruction to data Split, calculate, restructuring and form are changed;
S220, data are being split, calculate, recombinated and when form is changed, logically, arithmetic and data attribute pair Data carry out assignment;
S230, after being converted according to assignment to data, Hash tables data and metadata are formed respectively.
When it is implemented, data conversion is made up of assignment statement and procedure statement, data fractionation, meter are provided for business datum The ability of calculation, restructuring and form conversion, data calculate and not only support arithmetical operation, also support logical operation.
According to actual environment needs, data conversion is mainly divided to two kinds of forms, the first:Grasped for the variable of statement Make, computational methods are provided to obtain source data;Second:Data after extraction are operated, number is arranged according to operation rule According to.No matter which kind of methods are used, and data conversion does not operate directly to source data.
In further preferred embodiment of the invention, the CigQL agreements are XML tunnelings.
Fig. 6 is that a kind of implementation based on the database language technology for realizing data unified standard of the present invention is preferably implemented 3rd split flow figure of example, as shown in fig. 6, in the further preferred embodiment of the present invention, the step S400 is specifically included:
After the connection of S410, CigQL data store internal, data export request is sent;
S420, exported and asked according to data, confirm the logical place of export data, by CigQL standards, open data Api interface, derived data write-in buffer area will be treated, derived data export then will be treated in buffer area;
After S430, pending data export, the connection of CigQL data store internals is disconnected.
When it is implemented, data export is substantially similar with the composition of data pick-up, data export includes:CigQL resources connect Connect, data are exported and disconnected.
In order to manipulate different types of data resource according to unified approach, it is necessary to be carried out to the method for operating of data resource Encapsulation, form unified effective manipulation interface.The existence form of data includes 4 parts as data pick-up, is several respectively According to resource, Service Source, file resource and Policies Resource.
Preferably, the implementation based on the database language technology for realizing data unified standard, wherein, it is described Data Switch areas are provided with multiple Data Subject subregions, and different types of data are stored in corresponding Data Subject subregions.
In the realization based on the database language technology for realizing data unified standard that present pre-ferred embodiments are provided Mode, it is the implementation that CigQL is extracted to source data and is standardized to data, meanwhile, in whole process In, extract every time and transmission is not encrypted to data, compressed.But data are transmitted in the passage of an encryption. Therefore, there is absolute guarantee in Information Security.
And it is of the invention on the framework of entirety, by the way of SSD, flash memory storage, internal memory calculating and distributed deployment, The problem of caused performance difference of the problem of to evade due to IO.
During the entire process of the present invention, it is still that application produces new data file process either to extract data, all It is built upon on open api interface, all structural models are at least required for following function:
Index management module:It is responsible for updating from data transformation module, data package module and data export module and indexes Information, simultaneously, there is provided the index information that other application service module is asked for.
Share and access interface:It is responsible for all sharing requests of adapter, implements to authorize supervision, unified standard examination and other access Rule is implemented, and finally feeds back desired data.
Inquiry system engine:Search where it is desirable want data, according to asking and the integral frame of data sharing service, Its request may be forwarded to other data application service modules or central integrated services module.
Data lift module:The information Source Description of data, parses number needed for Query Result acquisition based on query engine According to logical path, corresponding application system is connected to accordingly, by api interface to obtain corresponding initial data.Meanwhile if having Necessity, finally lift wanted data fragments.
Data conversion module:After the completion of wanted data lift, if its form is not desired, that is, convert the data into logical Reference format.
In summary, the invention discloses a kind of realization side based on the database language technology for realizing data unified standard Formula, comprise the following steps:A, by data extraction module, source data is extracted according to different data types, and will take out Data after taking are stored in the way of FIFO in CigQL databases;B, by data transformation module, to the data after extraction Logically, arithmetic and data attribute enter line translation, and Hash tables data, the metadata after conversion are stored in into Index respectively Area, Metedata areas;C, by data package module, the data for being stored in Index areas and Metedata areas are assisted according to CigQL View is packaged, and forms the data of unified standard;D, by data export module, to forming the data of unified standard according to purposes Classification is exported, and the data after export are deposited to Data Switch areas, will not so that rear end is called to data Database data with form is converted into the normal data of uniform format, realizes the linkage between data in different formats, quickly Solve the problems, such as that isomeric data is difficult to fusion.
It should be appreciated that the application of the present invention is not limited to above-mentioned citing, for those of ordinary skills, can To be improved or converted according to the above description, all these modifications and variations should all belong to the guarantor of appended claims of the present invention Protect scope.

Claims (9)

1. a kind of implementation based on the database language technology for realizing data unified standard, it is characterised in that described to be based on Realize that the implementation of the database language technology of data unified standard comprises the following steps:
A, by data extraction module, source data is extracted according to different data types, and the data after extraction are pressed It is stored according to FIFO mode in CigQL databases;
B, by data transformation module, to the data after extraction logically, arithmetic and data attribute enter line translation, and will conversion Hash tables data, metadata afterwards is stored in Index areas, Metedata areas respectively;
C, by data package module, the data for being stored in Index areas and Metedata areas are sealed according to CigQL agreements Dress, form the data of unified standard;
D, by data export module, the data for forming unified standard are exported according to purposes classification, and by after export Data are deposited to Data Switch areas, so that rear end is called to data.
2. the implementation according to claim 1 based on the database language technology for realizing data unified standard, it is special Sign is, also includes step E before the step A:
E, module is declared by variable, variable statement is carried out to source data.
3. the implementation according to claim 2 based on the database language technology for realizing data unified standard, it is special Sign is that the element that the variable statement includes is structure, single argument and set of variables.
4. the implementation according to claim 3 based on the database language technology for realizing data unified standard, it is special Sign is that the single argument includes:Integer, floating number, character, character string, memory block, the date, the time, the moment, logical number and Enumerator.
5. the implementation according to claim 1 based on the database language technology for realizing data unified standard, it is special Sign is that the step A is specifically included:
Data are identified by A1, the data type for obtaining source data first according to the position of data type and attribute;
A2, according to the data attribute identified, data are divided into level one data and secondary data;
A3, level one data and secondary data are extracted respectively, and the data after extraction are saved in CigQL databases.
6. the implementation according to claim 1 based on the database language technology for realizing data unified standard, it is special Sign is that the step B is specifically included:
B1, the data after extraction are split using conditional statement, Do statement, case statement and skip instruction to data, Calculate, restructuring and form are changed;
B2, data are being split, calculate, recombinated and when form is changed, logically, arithmetic and data attribute enter to data Row assignment;
B3, after being converted according to assignment to data, Hash tables data and metadata are formed respectively.
7. the implementation according to claim 1 based on the database language technology for realizing data unified standard, it is special Sign is that the CigQL agreements are XML tunnelings.
8. the implementation according to claim 1 based on the database language technology for realizing data unified standard, it is special Sign is that the step D is specifically included:
After the connection of D1, CigQL data store internal, data export request is sent;
D2, exported and asked according to data, confirm the logical place of export data, by CigQL standards, the API of open data connects Mouthful, derived data write-in buffer area will be treated, derived data export then will be treated in buffer area;
After D3, pending data export, the connection of CigQL data store internals is disconnected.
9. the implementation according to claim 1 based on the database language technology for realizing data unified standard, it is special Sign is that the Data Switch areas are provided with multiple Data Subject subregions, and different types of data are stored in accordingly Data Subject subregions.
CN201710696246.0A 2017-08-15 2017-08-15 A kind of implementation based on the database language technology for realizing data unified standard Pending CN107526790A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710696246.0A CN107526790A (en) 2017-08-15 2017-08-15 A kind of implementation based on the database language technology for realizing data unified standard

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710696246.0A CN107526790A (en) 2017-08-15 2017-08-15 A kind of implementation based on the database language technology for realizing data unified standard

Publications (1)

Publication Number Publication Date
CN107526790A true CN107526790A (en) 2017-12-29

Family

ID=60681174

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710696246.0A Pending CN107526790A (en) 2017-08-15 2017-08-15 A kind of implementation based on the database language technology for realizing data unified standard

Country Status (1)

Country Link
CN (1) CN107526790A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108153919A (en) * 2018-02-28 2018-06-12 弘成科技发展有限公司 DBF data export platform and its deriving method
CN108647283A (en) * 2018-05-04 2018-10-12 武汉灵动在线科技有限公司 A kind of configuration of game data is quick to be generated and analytic method
CN110007912A (en) * 2019-03-05 2019-07-12 山东浪潮通软信息科技有限公司 A kind of visual configuration realization system and method for data sharing interface

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102567330A (en) * 2010-12-15 2012-07-11 上海杉达学院 Heterogeneous database integration system
CN103092980A (en) * 2013-01-31 2013-05-08 中国科学院自动化研究所 Method and system of data automatic conversion and storage
CN106372185A (en) * 2016-08-31 2017-02-01 广东京奥信息科技有限公司 Data preprocessing method for heterogeneous data sources

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102567330A (en) * 2010-12-15 2012-07-11 上海杉达学院 Heterogeneous database integration system
CN103092980A (en) * 2013-01-31 2013-05-08 中国科学院自动化研究所 Method and system of data automatic conversion and storage
CN106372185A (en) * 2016-08-31 2017-02-01 广东京奥信息科技有限公司 Data preprocessing method for heterogeneous data sources

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108153919A (en) * 2018-02-28 2018-06-12 弘成科技发展有限公司 DBF data export platform and its deriving method
CN108647283A (en) * 2018-05-04 2018-10-12 武汉灵动在线科技有限公司 A kind of configuration of game data is quick to be generated and analytic method
CN110007912A (en) * 2019-03-05 2019-07-12 山东浪潮通软信息科技有限公司 A kind of visual configuration realization system and method for data sharing interface

Similar Documents

Publication Publication Date Title
CN110837492B (en) Method for providing data service by multi-source data unified SQL
US7469248B2 (en) Common interface to access catalog information from heterogeneous databases
US9223817B2 (en) Virtual repository management
US7526503B2 (en) Interactive schema translation with instance-level mapping
US5966707A (en) Method for managing a plurality of data processes residing in heterogeneous data repositories
CA2318299C (en) Metadata exchange
US20220292092A1 (en) System and method for querying multiple data sources
US11068512B2 (en) Data virtualization using leveraged semantic knowledge in a knowledge graph
CN110032604A (en) Data storage device, transfer device and data bank access method
US10346375B2 (en) In-database parallel analytics
US11886411B2 (en) Data storage using roaring binary-tree format
Li et al. An integration approach of hybrid databases based on SQL in cloud computing environment
US20150154259A1 (en) Sql query on a nosql database
US6549901B1 (en) Using transportable tablespaces for hosting data of multiple users
WO2011111532A1 (en) Database system
CN107977446A (en) A kind of memory grid data load method based on data partition
CN107526790A (en) A kind of implementation based on the database language technology for realizing data unified standard
US7213014B2 (en) Apparatus and method for using a predefined database operation as a data source for a different database operation
US11960616B2 (en) Virtual data sources of data virtualization-based architecture
US11687513B2 (en) Virtual data source manager of data virtualization-based architecture
US11263026B2 (en) Software plugins of data virtualization-based architecture
Arputhamary et al. A review on big data integration
CN113641862A (en) Method and system for integrating multi-source heterogeneous data based on uniform access distribution
US11960488B2 (en) Join queries in data virtualization-based architecture
Engle A Methodology for Evaluating Relational and NoSQL Databases for Small-Scale Storage and Retrieval

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20171229