CN104899301B - A kind of processing method of multi-source heterogeneous data - Google Patents
A kind of processing method of multi-source heterogeneous data Download PDFInfo
- Publication number
- CN104899301B CN104899301B CN201510316367.9A CN201510316367A CN104899301B CN 104899301 B CN104899301 B CN 104899301B CN 201510316367 A CN201510316367 A CN 201510316367A CN 104899301 B CN104899301 B CN 104899301B
- Authority
- CN
- China
- Prior art keywords
- data
- source heterogeneous
- processing method
- user
- heterogeneous data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention belongs to technical field of data processing, and in particular to a kind of processing method of multi-source heterogeneous data.A kind of processing method of multi-source heterogeneous data, comprising the following steps: (1), data extract；(2), constructing SQL statement data screening；(3), by process, nodal operation realizes that data generate；(4), client is pushed data into according to demand；(5), after needing to merge data according to user, fused data are presented on the display device.The invention discloses a kind of processing methods of multi-source heterogeneous data, have the advantages that 1, data-handling efficiency height；2, data procedures visualize；3, multi-source heterogeneous data seamless access, system can synchronize processing to the data of different-format, different structure.
Description
Technical field
The invention belongs to technical field of data processing, and in particular to a kind of processing method of multi-source heterogeneous data.
Background technique
" when internet is not made clear also, mobile interchange comes, when mobile interchange is not made clear also, big data
Come ".Big data application is deep into already among ordinary life, will be increasingly becoming a part of modern society's infrastructure,
Just as highway, railway, harbour, water power are indispensable as communication network.
What does big data era change? we work, the various scenes in life are quietly changed.American writer
Robert scott uncle is in " Age of Context " book, and " after big data era, next science and technology trend is exactly scene to title
Epoch！Who can occupy scene, can win future！"
By the informatization of many years, in the Chinese government, enterprise, smart city, digital oil field etc. are all built powerful
Database Systems, department at different levels has all successively built up the relevant information system of all kinds of business especially in enterprise, advances
The development of IT application in enterprises；However, due to the diversity of construction, such as more phases time, more IT quotient, multi-format, multiple database, polyphyly
System, more technical methods etc. lead to the polyphyly of data.Constitute " the data disease " of data.Here it is database VS databases；Letter
Breath system VS information system；Information system VS professional software；Professional software VS professional software can not achieve data sharing, occur
" fragmentation of data " and " wide gap ".
Currently, in China, data construction cost makes an investment in trillion yuan according to estimates, for such status, it is impossible to overthrow weight
Come, it is also not possible to abandon these data and not have to.It and is conventionally exactly often to encounter " data disease " problem it is necessary to use people
Work coding, the mode for arranging data by hand develop software (middleware) and do data-interface.But software development is one complicated
Engineering, the development cycle is long, under huge data, realizes that the integrated of system is answered by the way of h coding, manual arrangement data
With with data operation management, work like being built by the way of manual labor and get through the bridge spanning the sea between each island
It measures, it is very huge.
We have invented one kind to follow zero code principle in whole design, and user is not required to the programming skill it is to be understood that complicated
Art, it is only necessary to arrange and combine various functional nodes, the big number of the integration and application to massive multi-source data can be realized
According to the critical software of extraction, arrangement, fusion, analysis and data mining.
Summary of the invention
Goal of the invention: the present invention has made improvements in view of the above-mentioned problems of the prior art, i.e., the invention discloses one kind
The processing method of multi-source heterogeneous data.
A kind of technical solution: processing method of multi-source heterogeneous data, comprising the following steps:
(1), data are extracted；
(2), constructing SQL statement data screening；
(3), by process, nodal operation realizes that data generate；
(4), client is pushed data into according to demand；
(5), after needing to merge data according to user, fused data are presented on the display device.
Further, step (1) the following steps are included:
(11), database is selected according to user demand and inputs the data requirements of user；
(12), data flow is established；
(13), nodeization operates.
From various databases, in face of structuring, semi-structured and unstructured database various data, including government
Data, municipal data and professional superpower data will start to work as desired as long as the demand of proposition.
The utility model has the advantages that being had the advantages that the invention discloses a kind of processing method of multi-source heterogeneous data
1, data-handling efficiency is high；
2, data procedures visualize, and Business Stream and data flow are blended, and are IT technical staff and traditional business field
Expert provides dialogue, cooperation platform, makes cross-cutting mixing together.Meanwhile cured experience, formula, algorithm, classics can be formed
Data analysis process so that research method can inherit, can layout also avoid looking forward to ensure the continuity of data analysing method
The loss for the research method that industry causes because the talent is transferred and promoted；
3, multi-source heterogeneous data seamless access, system can synchronize processing to the data of different-format, different structure, lead to
Cross function and process and support more than ten databases such as Oracle, MySQL, SQL Server, FTP, Excel, Word, TEXT,
The file formats such as GIS, WIS, have opened second development interface, and user can read in data by customizing script as needed；
4, Enterprise Data is integrated, one-touch visioning procedure.System provides Enterprise Data and quickly accesses module, can basis
Data model, the database dictionary of enterprise press professional domain group organization data, provide key search, realize the quick fixed of tables of data
Position, and one-touch visioning procedure is provided, access the data in enterprise-level database.System provides node abundant and method, just
In reconstruction business events flow path；
5, data presentation mode is versatile and flexible, and the dimensions such as report, statistical graph, professional chart board, spatial distribution can be used in user
Display data is spent, inherent connection and rule between mining data；
6, any data intelligence in face of any format Yu arbitrary data library is extracted；
7, Zero-code, workflow editor, according to demand automatic editing process；
8, node type operates, and the extraction of data is completed as played with building blocks, can do data preparation, fusion and visualization.
Detailed description of the invention
Fig. 1 is a kind of flow diagram of the processing method of multi-source heterogeneous data disclosed by the invention；
Fig. 2 is a kind of flow chart of the processing method of multi-source heterogeneous data disclosed by the invention.
Specific embodiment:
Detailed description of specific embodiments of the present invention below.
As depicted in figs. 1 and 2, a kind of processing method of multi-source heterogeneous data, comprising the following steps:
(1), data are extracted；
(2), constructing SQL statement data screening；
(3), by process, nodal operation realizes that data generate；
(4), client is pushed data into according to demand；
(5), after needing to merge data according to user, fused data are presented on the display device.
Further, data extract the following steps are included:
(11), database is selected according to user demand and inputs the data requirements of user；
(12), data flow is established；
(13), nodeization operates.
By taking oil field data as an example: certain oilfield enterprise, since information system is unstable, putaway rule is complicated, in analysis test
The heart has 50,000 casting body flake images, fails to be put in storage in time；Oil-gas reservoir research is carried out using sheet data to scientific research personnel to bring
It is inconvenient.These picture datas are arranged and are put in storage by higher level's goal, by sample lot number in extraction well-name, depth and database
Pairing, standardization name of pictures, reject repeat photo, reject existing database in have photo, typing print reference information, on
Pass more than 10 steps such as photo files；It arranges a thin slice photo and about expends 3 minutes, complete 50,000 photos and take around 300
Multiple working days, data preparation intricate operation, workload are huge.
Using the processing method of multi-source heterogeneous data, the process of Data Analysis Services is constructed, it is only necessary to which 4 hours complete number
According to the task of arrangement (improving working efficiency hundreds times).
Embodiments of the present invention are elaborated above.But present invention is not limited to the embodiments described above,
Technical field those of ordinary skill within the scope of knowledge, can also do without departing from the purpose of the present invention
Various change out.
Claims (1)
1. a kind of processing method of multi-source heterogeneous data, which comprises the following steps:
(1), data are extracted；
(11), database is selected according to user demand and inputs the data requirements of user；
(12), data flow is established；
(13), nodeization operates；
(2), constructing SQL statement data screening；
(3), by process, nodal operation realizes that data generate；
(4), client is pushed data into according to demand；
(5), after needing to merge data according to user, fused data are presented on the display device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510316367.9A CN104899301B (en) | 2015-06-10 | 2015-06-10 | A kind of processing method of multi-source heterogeneous data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510316367.9A CN104899301B (en) | 2015-06-10 | 2015-06-10 | A kind of processing method of multi-source heterogeneous data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104899301A CN104899301A (en) | 2015-09-09 |
| CN104899301B true CN104899301B (en) | 2019-04-16 |
Family
ID=54031963
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510316367.9A Active CN104899301B (en) | 2015-06-10 | 2015-06-10 | A kind of processing method of multi-source heterogeneous data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104899301B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105677710A (en) * | 2015-12-28 | 2016-06-15 | 曙光信息产业(北京)有限公司 | Processing method and system of big data |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103136249A (en) * | 2011-11-30 | 2013-06-05 | 北京航天长峰科技工业集团有限公司 | System and method of multiplex mode isomerous data integration |
| CN104142927A (en) * | 2013-05-07 | 2014-11-12 | 天津冠创科技有限公司 | Multi-source heterogeneous spatial data interoperability platform |
-
2015
- 2015-06-10 CN CN201510316367.9A patent/CN104899301B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103136249A (en) * | 2011-11-30 | 2013-06-05 | 北京航天长峰科技工业集团有限公司 | System and method of multiplex mode isomerous data integration |
| CN104142927A (en) * | 2013-05-07 | 2014-11-12 | 天津冠创科技有限公司 | Multi-source heterogeneous spatial data interoperability platform |
Non-Patent Citations (2)
| Title |
|---|
| 《基于Web Servie的异构关系数据源的集成研究》;孙媛媛;《中国优秀硕士论文全文数据库》;20071231;第18、20、22、23、27、35、36、38页 |
| 《多源异构信息的融合方法及其在应急监控预警中的应用》;梁屹等;《《风险分析和危机反应中的信息技术-中国灾害防御协会风险分析专业委员会第六届年会论文集》》;20140823;第316-321页 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104899301A (en) | 2015-09-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104318402B (en) | Integrated planning and designing information system based on power network GIS platform | |
| CN104252345B (en) | The method and system of complex management object in cloud environment | |
| CN109740159B (en) | Processing method and device for named entity recognition | |
| CN110647632B (en) | Image and text mapping technology based on machine learning | |
| CN113535977B (en) | Knowledge graph fusion method, device and equipment | |
| CN105550375A (en) | Heterogeneous data integrating method and system | |
| CN104182494A (en) | Method and system capable of realizing CMS website construction with PC terminal and mobile terminal | |
| WO2017193471A1 (en) | Digital global sharing platform for preserving dongba ancient texts | |
| CN110046637A (en) | A kind of training method, device and the equipment of contract paragraph marking model | |
| Leetaru | Mining libraries: Lessons learned from 20 years of massive computing on the world’s information | |
| Trepal et al. | Heritage making through community archaeology and the spatial humanities | |
| CN104899301B (en) | A kind of processing method of multi-source heterogeneous data | |
| Morioka | Multiple-policy Character Annotation based on CHISE | |
| CN116097253A (en) | Method and device for constructing multi-level knowledge graph | |
| Jung | Semantic wiki-based knowledge management system by interleaving ontology mapping tool | |
| Penela et al. | miKrow: Semantic intra-enterprise micro-knowledge management system | |
| Foka et al. | Semantically geo-annotating an ancient Greek" travel guide" itineraries, chronotopes, networks, and linked data | |
| Aydinoglu | Modelling, encoding and transforming of open geographic data to examine interoperability between GIS applications | |
| CN114691643A (en) | Data migration method and system applied to domestic substitution | |
| Li et al. | Introducing OpenStreetMap user embeddings: Promising steps toward automated vandalism and community detection | |
| Kaasa et al. | The matter of erasure: making room for utopia at Nonoalco-Tlatelolco, Mexico City | |
| Schwitter | Bridging the offline and the online: Twenty years of offline meeting data of the German-language Wikipedia | |
| CN117575579B (en) | Hydraulic engineering perspective inspection method and related device based on BIM and knowledge graph | |
| CN117556507A (en) | Modulus combining method based on fbx format modulus-first and modulus-last | |
| Zheng et al. | A Storage Method of Online Educational Resources for College Courses Based on Artificial Intelligence Technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |