CN107315743A - A kind of big data conversion method and system based on adapter - Google Patents
A kind of big data conversion method and system based on adapter Download PDFInfo
- Publication number
- CN107315743A CN107315743A CN201610263799.2A CN201610263799A CN107315743A CN 107315743 A CN107315743 A CN 107315743A CN 201610263799 A CN201610263799 A CN 201610263799A CN 107315743 A CN107315743 A CN 107315743A
- Authority
- CN
- China
- Prior art keywords
- data
- isomeric
- isomeric data
- conversion
- adapter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/904—Browsing; Visualisation therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention relates to a kind of big data conversion method and system based on adapter, big data conversion method of the one of which based on adapter comprises the following steps：M IS is received to ask and configure isomeric data to isomeric data dissection process and based on SQL scripts；Caching process is carried out to the isomeric data of dissection process and operation phase configuration is carried out to the isomeric data configured；Conversion process is carried out to the isomeric data Jing Guo caching process and the isomeric data to conversion is stored and carries out operation phase operation to the isomeric data that the operation phase configures；The isomeric data of above-mentioned storage is showed, the method of the present invention can perform batch data conversion script set automatically, preserve conversion script execution result information, and accordingly generate applied analysis report, logic error that isomeric data changed can be reduced using the present invention, exchange efficiency between isomeric data is improved, and reduce the data collection and analysis cost of isomeric data.
Description
Technical field
Turn the present invention relates to data processing field, more particularly to a kind of big data based on adapter
Change method and system.
Background technology
In recent years, with the fast development and popularization and application of computer and information technology, sector application system
The scale of system expands rapidly, and the data produced by sector application are in explosive increase.Easily hundreds of TB are reached
Even industry/enterprise's big data of tens of to hundreds of PB scales is far beyond existing traditional calculating skill
The disposal ability of art and information system, therefore, seeks effective big data treatment technology, ways and means
Have become the active demand of real world.The current total amount of data of Baidu is needed daily more than 1000PB
Web data to be processed reaches 10PB~100PB；The accumulative transaction data amount of Taobao is up to
100PB；Twitter issues more than 200,000,000 message daily, and the amount of posting reaches 80,000,000 to Sina weibo daily
Bar；The telephone logs data of one province of China Mobile are monthly up to 0.5PB~1PB；One provincial capital
Public security bureau's road vehicles monitoring data 3 years reachable 20,000,000,000, total amount 120TB.Believe according to world authority IT
Cease the IDC research reports prediction of consulting analysis company：Whole world data volume coming 10 years are by from 2009
0.8ZB rises to the 35ZB (1ZB=1000EB=1000000PB) of the year two thousand twenty, will increase by 44 times within 10 years,
It is average annual to increase by 40%.And there is so many isomeric data, seamless exchange can not be carried out each other, formed
Data silo, it is impossible to play the application value of data.Now, our facing challenges are how to handle
The unstructured data of magnanimity and uncertain data, and excavate commercial value from these data.
The content of the invention
In view of current technology above shortcomings, the present invention provides a kind of based on the big of adapter
Data transfer device and system, method of the invention can perform batch data conversion set of scripts automatically
Close, preserve conversion script execution result information, and accordingly generate applied analysis report, can using the present invention
To reduce logic error that isomeric data changed, improve exchange efficiency between isomeric data, and
Reduce the data collection and analysis cost of isomeric data.
The present invention's adopts the following technical scheme that：
A kind of big data conversion method based on adapter, comprises the following steps：
Receive M IS request and to isomeric data dissection process and based on SQL scripts to different
Structure data are configured；
Caching process is carried out to the isomeric data of dissection process and the isomeric data configured is entered
The row operation phase configures；
Conversion process is carried out to the isomeric data Jing Guo caching process and the isomeric data to conversion enters
Row storage and the isomeric data progress operation phase operation configured to the operation phase；
The isomeric data of above-mentioned storage is showed.
As the preferred technical solution of the present invention, the reception M IS is asked and to isomery
Data dissection process and include the step of configured based on SQL scripts to isomeric data：
By the data in heterogeneous data source, the interface of standard is formulated；
Set up the wsdl document of data administration tasks and corresponding conversion adapter automatic identification；
Isomeric data is parsed into script and logical mappings file deployment Unified Data Interchange Platform；
Determine isomeric data management role set；
Corresponding SQL scripts are debugged according to isomeric data management role and obtain SQL set of scripts；
Configure the logical mappings file between isomeric data management role and SQL scripts；
The SQL set of scripts and logical mappings file are positioned over data interchange platform.
As the preferred technical solution of the present invention, the isomeric data to dissection process is cached
Handle and include the step of carrying out operation phase configuration to the isomeric data configured；
The isomeric data of dissection process is cached by being adapted to change to be stored in after script is changed；
Based on heterogeneous database exchange platform resolution logic mapped file；
According to performing, action parameter determines to perform the moment heterogeneous database exchange platform and execution is per number
According to library management task items.
As the preferred technical solution of the present invention, the described pair of isomeric data Jing Guo caching process enters
The isomery that row conversion process and isomeric data to conversion are stored and configured to the operation phase
The step of data carry out operation phase operation includes：
To the isomeric data Jing Guo caching process by calling big data transform engine, batch is carried out
Data conversion and isomeric data to conversion is stored；
Heterogeneous database exchange platform finds the database management tasks mapping according to logical mappings file
SQL scripts simultaneously read SQL content for script.
As the preferred technical solution of the present invention, the isomeric data to above-mentioned storage carries out exhibition
Existing step includes：
By calling big data analysis engine to analyze the isomeric data of above-mentioned storage；
Data after by analysis are subjected to Data Frontend displaying.
The another side of the present invention, a kind of big data converting system based on adapter, including be sequentially connected
Request receiving module, isomeric data parsing module, isomeric data buffer process module, adaptation
Device processing module, M IS processing module, isomeric data storage processing module, isomery
Data, services processing module, isomeric data show processing module.
As the preferred technical solution of the present invention, the request receiving module and mobile client,
Web client, pc client are connected.
The invention has the advantages that：
1st, batch data conversion script set can be performed automatically, preserve conversion script execution result information,
And applied analysis report is accordingly generated, the logic mistake that isomeric data is changed can be reduced using the present invention
By mistake, improve isomeric data between exchange efficiency, and reduction isomeric data data collection and analysis into
This.
2nd, for from various isomeric data source datas (including structural data, unstructured data),
Batch conversion is carried out using unified interface specification and consensus standard, passes through the data adapter unit of original creation
Algorithm and transformation rule engine, unattended automatic conversion is carried out to target data and is exported with analysis.
Isomeric data parsing task, convert task, batch signatures task dispatching need to write corresponding script,
Form isomeric data automation conversion set of scripts.
3rd, the data sample and condition code of various heterogeneous data sources are set up, and designs matched conversion
Adapter carries out conversion logic and mapped with transformation rule；By the analysis of batch isomeric data and conversion of automation
Set of scripts and conversion logic are deployed in Unified Data Interchange Platform with transformation rule mapped file.
Brief description of the drawings
Technical scheme in order to illustrate the embodiments of the present invention more clearly, below will be to embodiment
In required for the accompanying drawing that uses be briefly described, it should be apparent that, drawings in the following description
Only some embodiments of the present invention, for those of ordinary skill in the art, are not paying
On the premise of going out creative work, other accompanying drawings can also be obtained according to these accompanying drawings.
The flow that Fig. 1 is embodied for the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, to the technical side in the embodiment of the present invention
Case is clearly and completely described, it is clear that described embodiment is only the present invention one
Divide embodiment, rather than whole embodiments.Based on the embodiment in the present invention, this area is general
The every other embodiment that logical technical staff is obtained under the premise of creative work is not made,
Belong to the scope of protection of the invention.
As shown in figure 1, the present invention provides a kind of big data conversion method based on adapter, bag
Include following steps：
Step S1：Receive M IS request and to isomeric data dissection process and based on SQL
Script is configured to isomeric data, specially step S1a：By the data in heterogeneous data source, system
Accurate interface is calibrated, it is more preferably, unified to formulate mark for the data in each heterogeneous data source
Accurate interface specification, using SOAP standard agreements, converts thereof into the XML data of standard；Step S1b：
The wsdl document of data administration tasks and corresponding conversion adapter automatic identification is set up, further to build
The wsdl document of vertical various data administration tasks and corresponding conversion adapter automatic identification；Step S1c：
Isomeric data is parsed into script and logical mappings file deployment Unified Data Interchange Platform；Step S1d：Really
Determine isomeric data management role set；Step S1e：According to isomeric data management role debugging phase
The SQL scripts answered obtain SQL set of scripts；Configure between isomeric data management role and SQL scripts
Logical mappings file；Step S1f：The SQL set of scripts and logical mappings file are positioned over into data to hand over
Change platform.
Step S2：Caching process is carried out and different to what is configured to the isomeric data of dissection process
Structure data carry out operation phase configuration, are specially：Step S2a：The isomeric data of dissection process is led to
Cross after adaptation conversion script is changed and be stored in caching, then specifically by data conversion keyword certainly
Dynamic adaptation conversion script is stored in slow after being changed；Step S2b：Based on heterogeneous database exchange platform
Resolution logic mapped file；Step S2c：Heterogeneous database exchange platform determines to hold according to action parameter is performed
The row moment simultaneously performs each database management tasks.
Step S3：Conversion process is carried out to the isomeric data Jing Guo caching process and to the different of conversion
Structure data are stored and carry out operation phase operation to the isomeric data that the operation phase configures, specifically
For step S3a：To the isomeric data Jing Guo caching process by calling big data transform engine, enter
Row batch data is changed and the isomeric data of conversion is stored, then turns specifically by definition
Rule is changed, big data transform engine is called, batch data conversion is carried out and to the isomery number of conversion
According to being stored；Step S3b：Heterogeneous database exchange platform finds the data according to logical mappings file
The SQL scripts of library management task items mapping simultaneously read SQL content for script.
Step S4：The step of showing to the isomeric data of above-mentioned storage includes, step S4a：
By calling big data analysis engine to analyze the isomeric data of above-mentioned storage；Step S4b：
Data after by analysis are subjected to Data Frontend displaying, specifically by startup report form showing mould
Block carries out statistical analysis and displaying to the change over condition of isomeric data.
The present invention is expanded on further in specific embodiment presented below
Embodiment 1
In the big data that presently, there are, it is varied that long-term existence data format, it is impossible to carries out data
Exchange, the problem of maintenance and upgrading for data but have high cost, poor efficiency.
Scene 1：Many data have been precipitated by accumulation for many years in business bank, including business datum,
Customer data, risk data, performance data, accounting data etc., the form of these data are varied,
There are Flat files, the same files from core system, autoexec, csv file, pdf
File etc., also has some image files, and both the data including structuring had including non-structured data,
It can not be swapped between data, form data barrier, have impact on operating efficiency.Therefore, in the urgent need to
A kind of technology or instrument so that various isomeric datas can realize Batch conversion, so as to reach data seamless
Exchange the target with data sharing.
Scene 2：The value potentiality of industrial big data application is huge.But, realize that these values also have very
Many work will be done.One is the problem of big data consciousness is set up.In the past, also there are these big datas, but by
In the consciousness without big data, means of numerical analysis is also not enough, many real time datas be dropped or beam height
Pavilion, the potential value of mass data is buried.The problem of also one major issue is data silo.Very
In each isolated island of the data distribution of many industrial enterprises in enterprise, particularly in large multinational company,
Want to extract these data in whole enterprise extremely difficult.
By the method for the present invention just can solve seamless exchange in above-mentioned scene between each isomeric data and
Integrate, it is automatic by the data characteristics code of original creation shown in specific steps step 1 described above to step 4
Identification technology loads various isomeric datas automatically, and exchanges corresponding adapter and enter row format conversion, agreement
The sequence of operations such as conversion, message conversion, generation target data is stored in caching.By calling big data
Analysis engine carries out data analysis and data mining.Front end exhibition is carried out finally by big data visualization technique
Existing analysis result.
The other one side of the present invention, a kind of big data converting system based on adapter, including successively
Connected request receiving module, isomeric data parsing module, isomeric data buffer process module,
Adapter processing module, M IS processing module, isomeric data storage processing module,
Isomeric data service processing module, isomeric data show processing module.
Further, request receiving module and mobile client, Web client, pc client phase
Even, but be not restricted to that above-mentioned client.
In the present invention, convert requests of the request receiving module to receive isomeric data, isomery number
It is used to the dissection process of isomeric data according to parsing module, isomeric data buffer process module is to different
The caching process of structure data, adapter processing module starts to configure transformation rule and script
Data translation tasks, M IS processing module is used to the conversion process of isomeric data, different
Structure data storage processing module is handled to the storage of isomeric data, isomeric data service processing mould
Block is to provide data, services, data analysis service, data switching service, data calculating sum
According to service of excavating etc., and by calling big data analysis engine to carry out data analysis operation；It is different
Structure data exhibiting processing module is shown to isomeric data in Data Frontend.
The foregoing is only a specific embodiment of the invention, but protection scope of the present invention is simultaneously
This is not limited to, any those skilled in the art is in technology model disclosed by the invention
In enclosing, the change or replacement that can be readily occurred in should be all included within the scope of the present invention.
Therefore, protection scope of the present invention should be based on the protection scope of the described claims.
Claims (7)
1. a kind of big data conversion method based on adapter, it is characterised in that including following step
Suddenly：
Receive M IS request and to isomeric data dissection process and based on SQL scripts to different
Structure data are configured；
Caching process is carried out to the isomeric data of dissection process and the isomeric data configured is entered
The row operation phase configures；
Conversion process is carried out to the isomeric data Jing Guo caching process and the isomeric data to conversion enters
Row storage and the isomeric data progress operation phase operation configured to the operation phase；
The isomeric data of above-mentioned storage is showed.
2. a kind of big data conversion method based on adapter according to claim 1, its
It is characterised by, the reception M IS is asked and to isomeric data dissection process and is based on
The step of SQL scripts are configured to isomeric data includes：
By the data in heterogeneous data source, the interface of standard is formulated；
Set up the wsdl document of data administration tasks and corresponding conversion adapter automatic identification；
Isomeric data is parsed into script and logical mappings file deployment Unified Data Interchange Platform；
Determine isomeric data management role set；
Corresponding SQL scripts are debugged according to isomeric data management role and obtain SQL set of scripts；
Configure the logical mappings file between isomeric data management role and SQL scripts；
The SQL set of scripts and logical mappings file are positioned over data interchange platform.
3. a kind of big data conversion method based on adapter according to claim 1, its
It is characterised by, the isomeric data to dissection process carries out caching process and to being configured
The step of isomeric data carries out operation phase configuration includes；
The isomeric data of dissection process is cached by being adapted to change to be stored in after script is changed；
Based on heterogeneous database exchange platform resolution logic mapped file；
Heterogeneous database exchange platform determines to perform the moment and performs each database according to execution action parameter
Management role.
4. a kind of big data conversion method based on adapter according to claim 1, its
It is characterised by, the described pair of isomeric data Jing Guo caching process carries out conversion process and to conversion
Isomeric data is stored and carries out operation phase operation to the isomeric data that the operation phase configures
Step includes：
To the isomeric data Jing Guo caching process by calling big data transform engine, lot number is carried out
According to conversion and the isomeric data of conversion is stored；
Heterogeneous database exchange platform finds the database management tasks mapping according to logical mappings file
SQL scripts simultaneously read SQL content for script.
5. a kind of big data conversion method based on adapter according to claim 1, its
It is characterised by, the step of isomeric data to above-mentioned storage shows includes：
By calling big data analysis engine to analyze the isomeric data of above-mentioned storage；
Data after by analysis are subjected to Data Frontend displaying.
6. a kind of big data converting system based on adapter, it is characterised in that including being sequentially connected
Request receiving module, isomeric data parsing module, isomeric data buffer process module, adapter
Processing module, M IS processing module, isomeric data storage processing module, isomeric data
Service processing module, isomeric data show processing module.
7. a kind of big data converting system based on adapter according to claim 6, it is special
Levy and be, the request receiving module is connected with mobile client, Web client, pc client.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610263799.2A CN107315743A (en) | 2016-04-26 | 2016-04-26 | A kind of big data conversion method and system based on adapter |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610263799.2A CN107315743A (en) | 2016-04-26 | 2016-04-26 | A kind of big data conversion method and system based on adapter |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107315743A true CN107315743A (en) | 2017-11-03 |
Family
ID=60185724
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610263799.2A Pending CN107315743A (en) | 2016-04-26 | 2016-04-26 | A kind of big data conversion method and system based on adapter |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107315743A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107864152A (en) * | 2017-12-08 | 2018-03-30 | 武汉虹信技术服务有限责任公司 | A kind of method that wisdom building multi-protocols adaptation engineering template is rapidly completed using VBA |
| CN109101468A (en) * | 2018-08-02 | 2018-12-28 | 浙江大学 | A kind of execution optimization method of text data conversion script |
| CN111159993A (en) * | 2019-12-25 | 2020-05-15 | 江苏方天电力技术有限公司 | Automatic analysis and report generation system platform for power transformation mobile operation card |
| CN111694874A (en) * | 2020-06-17 | 2020-09-22 | 科技谷（厦门）信息技术有限公司 | User behavior analysis system based on big data platform |
| CN113918634A (en) * | 2021-10-15 | 2022-01-11 | 国家电网有限公司大数据中心 | Data adaptation method, adapter and storage medium for data interaction |
-
2016
- 2016-04-26 CN CN201610263799.2A patent/CN107315743A/en active Pending
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107864152A (en) * | 2017-12-08 | 2018-03-30 | 武汉虹信技术服务有限责任公司 | A kind of method that wisdom building multi-protocols adaptation engineering template is rapidly completed using VBA |
| CN107864152B (en) * | 2017-12-08 | 2020-04-28 | 武汉虹信技术服务有限责任公司 | Method for rapidly completing intelligent building multi-protocol adaptation engineering template by using VBA |
| CN109101468A (en) * | 2018-08-02 | 2018-12-28 | 浙江大学 | A kind of execution optimization method of text data conversion script |
| CN111159993A (en) * | 2019-12-25 | 2020-05-15 | 江苏方天电力技术有限公司 | Automatic analysis and report generation system platform for power transformation mobile operation card |
| CN111159993B (en) * | 2019-12-25 | 2023-09-01 | 江苏方天电力技术有限公司 | Automatic analysis and report generation system platform for power transformation mobile operation card |
| CN111694874A (en) * | 2020-06-17 | 2020-09-22 | 科技谷（厦门）信息技术有限公司 | User behavior analysis system based on big data platform |
| CN113918634A (en) * | 2021-10-15 | 2022-01-11 | 国家电网有限公司大数据中心 | Data adaptation method, adapter and storage medium for data interaction |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107315743A (en) | A kind of big data conversion method and system based on adapter | |
| US20180165346A1 (en) | Multi-dimensional analysis using named filters | |
| CN112651126B (en) | BIM-based construction engineering construction optimization system, method, terminal and storage medium | |
| Jeong et al. | A NoSQL data management infrastructure for bridge monitoring | |
| CN105159951A (en) | Open tourism multi-source heterogeneous data fusion method and system | |
| CN110543571A (en) | knowledge graph construction method and device for water conservancy informatization | |
| CN105786913A (en) | Cloud manufacturing platform oriented ERP integrated database service interface encapsulation system and method | |
| EP3321821A1 (en) | Big data exchange method and device | |
| CN108170832B (en) | Monitoring system and monitoring method for heterogeneous database of industrial big data | |
| CN104486116A (en) | Multidimensional query method and multidimensional query system of flow data | |
| CN108964971A (en) | A kind of data interchange platform | |
| CN103399735A (en) | Method for developing intermediate layer of remote function call interface | |
| CN103870921A (en) | Operation maintenance management system | |
| CN104598235A (en) | Method and system for displaying team working interface of power management system | |
| CN109471957B (en) | Metadata conversion method and device based on uniform tags | |
| CN111062593A (en) | Distribution network operation management method and system, computer equipment and storage medium | |
| CN114547175A (en) | Data processing method, device, storage medium and computer system | |
| CN112488643A (en) | Digital middle platform system for enterprise application | |
| CN107122359B (en) | Data real-time tracking visualization processing method and device | |
| CN115424212B (en) | Electric power operation site violation identification system and application thereof | |
| CN115563341A (en) | Spatial video field for electric power operation violation identification and intelligent data processing system and method thereof | |
| CN105512829A (en) | Web service protocol-based card value multidimensional cost accumulation method | |
| Kazmi et al. | A qos-aware integrated management of iot deployments in smart cities | |
| CN112381457A (en) | Encryption and decryption operation method for mobile terminal equipment applied to production and reporting | |
| CN111563178A (en) | Rule logic diagram comparison method, device, medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20171103 |
|
| WD01 | Invention patent application deemed withdrawn after publication |