CN109800090A - A kind of data integrated system and method - Google Patents
A kind of data integrated system and method Download PDFInfo
- Publication number
- CN109800090A CN109800090A CN201811558696.4A CN201811558696A CN109800090A CN 109800090 A CN109800090 A CN 109800090A CN 201811558696 A CN201811558696 A CN 201811558696A CN 109800090 A CN109800090 A CN 109800090A
- Authority
- CN
- China
- Prior art keywords
- data
- data integration
- target
- module
- integration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the present invention provides a kind of data integrated system and method, it include: data integration task module, it is obtained based on Eclipse RCP platform construction, connect multiple data sources, the data integration instruction sent according to control module, pending data is extracted from multiple data sources, and generates corresponding data integration task, so that execution module handles pending data；Execution module is analyzed and processed pending data, the target data for meeting preset condition is integrated target database, so that user terminal transfers target data for executing data integration task；Control module is scheduled control to data integration task module, and to execution module for sending data integration instruction, sends user terminal for target data.The embodiment of the present invention provides visualized operation for amateur programming personnel, avoids complicated programming grammar, ensure that the standard of delivery of data integration scheme is unified, alleviates follow-up maintenance workload, improve the quality of data integration.
Description
Technical field
The present embodiments relate to technical field of data processing more particularly to a kind of data integrated system and methods.
Background technique
In enterprise, due to development time or the difference of developing department, often have multiple isomeries, operate in it is different soft
Information system on hardware platform is run simultaneously, and the data source of these systems independently of one another, is mutually closed.Implementing data sharing
Process in, since data may be from different approach, data content, data format and the quality of data are multifarious,
Sometimes even can encounter data format cannot convert or data conversion format after lose information the problems such as, seriously hinder data and exist
Flowing in each department and each software systems and shared.
In order to which data information that be distributed these, isomery is integrated, need to carry out data integration.Existing data set
At scheme, usually there are the implementation personnel on exploitation basis, according to business needs, carries out Software Coding, realize database and software
Data between system synchronize equal data integration jobs.However, current data integrating method is high to personnel requirement is implemented, can make
At the standard disunity of delivery.The quality of data integration and later maintenance are difficult to ensure, cause resource input larger.
Therefore, a kind of data integrated system and method are needed now to solve the above problems.
Summary of the invention
In view of the problems of the existing technology, the embodiment of the present invention provides a kind of data integrated system and method.
In a first aspect, the embodiment of the invention provides a kind of data integrated systems, comprising: data integration task module is held
Row module and control module, in which:
The data integration task module is obtained based on Eclipse RCP platform construction, connects multiple data sources, according to
The data integration instruction that the control module is sent, extracts pending data, and generate corresponding data from multiple data sources
Integration servers, so that the execution module handles the pending data；
The execution module is analyzed and processed the pending data, incites somebody to action for executing the data integration task
The target data for meeting preset condition integrates target database, so that user terminal transfers the target data；
The control module, for sending the data integration instruction to the data integration task module, and to described
Execution module is scheduled control, sends the user terminal for the target data.
Second aspect, the embodiment of the invention provides a kind of data sets based on data integrated system described in first aspect
At method, comprising:
Obtain the data integration request that user terminal is sent；
Pending data is extracted from multiple data sources according to data integration request, and generates corresponding data integration
Task；
The pending data of the data integration task is analyzed and processed, the target data set of preset condition will be met
At target database is arrived, so that user terminal transfers the target data.
The third aspect, the embodiment of the present invention provides a kind of electronic equipment, including memory, processor and is stored in memory
Computer program that is upper and can running on a processor, is realized when the processor executes described program as second aspect provides
Method the step of.
Fourth aspect, the embodiment of the present invention provide a kind of non-transient computer readable storage medium, are stored thereon with calculating
Machine program is realized as provided by second aspect when the computer program is executed by processor the step of method.
A kind of data integrated system provided in an embodiment of the present invention and method, by being based on Eclipse RCP platform construction
Obtained data integration task module provides visualized operation for amateur programming personnel, and data integration demand is automatically generated
Corresponding data integration task completes data integration job by execution module, avoids complicated programming grammar, reduces terminal and uses
The use difficulty at family ensure that the standard of delivery of data integration scheme is unified, alleviate follow-up maintenance workload, improve data
Integrated quality.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is this hair
Bright some embodiments for those of ordinary skill in the art without creative efforts, can be with root
Other attached drawings are obtained according to these attached drawings.
Fig. 1 is the structural schematic diagram of data integrated system provided in an embodiment of the present invention；
Fig. 2 is the flow diagram of data integrating method provided in an embodiment of the present invention；
Fig. 3 is electronic devices structure schematic diagram provided in an embodiment of the present invention.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art
Every other embodiment obtained without creative efforts, shall fall within the protection scope of the present invention.
The scale and complexity of the computer technology trend data integrated system of continuous development increasingly increase, information content it is fast
Surge length, also higher and higher to requirements such as the performances, scalability and ease for maintenance of system.However, to heterogeneous data source into
When row is integrated, generally requires and the user of no programming experience is learnt a series of if necessary by the language that data converge
It instructs or script guarantees the problem of being just able to achieve the integrated of data, and eliminate the isomerism between integrated data and autonomy
The permission of data is very tired right, and needs prolonged study, and more users are more willing to using some simple mouse
Mark, which pulls, can complete satisfactory data integration or query statement.
Fig. 1 is the structural schematic diagram of data integrated system provided in an embodiment of the present invention, as shown in Figure 1, the present invention is implemented
Example provides a kind of data integrated system, comprising: data integration task module 101, execution module 102 and control module 103,
In:
The data integration task module 101 is obtained based on Eclipse RCP platform construction, connects multiple data sources, root
The data integration instruction sent according to the control module 103, extracts pending data, and generate corresponding from multiple data sources
Data integration task, so that the execution module 102 handles the pending data；
The execution module 102 carries out at analysis the pending data for executing the data integration task
Reason, integrates target database for the target data for meeting preset condition, so that user terminal transfers the target data；
The control module 103, for sending the data integration instruction to the data integration task module 101, and
Control is scheduled to the execution module 102, sends the user terminal for the target data.
In embodiments of the present invention, data integration task module 101 is constructed based on Eclipse RCP platform, will be used
Family interface and service logic separate, and multiple data sources are provided in data integration task module 101, and each data source component is responsible for
Dock a kind of data source, including machine system, relational database, extensible markup language, application program and spreadsheet etc..With
Family can by carrying out visualized operation on the display unit in control module 103, by inputting simple data type information,
Data integration task module 101 is sent by corresponding data integration instruction, to generate corresponding data integration task.Herein
In the process, processing is programmed to data integration servers without user, it is only necessary to which being input to required data association message can
Depending on change window, the plan target that can be integrated by 101 automatically generated data of data integration task module.Then, data integration
The data integration task automatically generated is sent execution module 102 by task module 101, at the same from multiple data sources extract and
The relevant data of data integration task, execution module 102 are analyzed and processed these pending datas, according to preset data
The data with potential quality problems are marked in execution standard, and mistake or duplicate data are deleted, to protect
Quality, accuracy and the integrality of data are demonstrate,proved, finally treated that target data integrates target database, Yong Huke by these
These data are scheduled or be inquired by visualization window.
The embodiment of the present invention is non-special by obtaining data integration task module 101 based on Eclipse RCP platform construction
Industry programming personnel provides visualized operation, and data integration demand is automatically generated corresponding data integration task, passes through execution
Module 102 completes data integration job, avoids complicated programming grammar, reduces the use difficulty of terminal user, ensure that data
The standard of delivery of Integrated Solution is unified, alleviates follow-up maintenance workload, improves the quality of data integration.
On the basis of the above embodiments, the multiple data source includes: host system, relational database, expansible mark
Remember language, application program and spreadsheet.
On the basis of the above embodiments, the control module 103 is also used to obtain the state day of the execution module 102
Will, to carry out data synchronization monitoring to the execution module 102.
In embodiments of the present invention, when data source is updated or is deleted, execution module 102 to target database into
Row data are synchronous, and control module 103 monitors the status log of execution module 102 in real time, and is shown accordingly by visualization window
Synchronizing information.
On the basis of the above embodiments, the control module 103 includes:
Instruction input unit, for providing application programming interface, to obtain the assignment instructions of data integration；
Display unit, for carrying out visualization display to the executive condition of data integration.
Fig. 2 is the flow diagram of data integrating method provided in an embodiment of the present invention, as shown in Fig. 2, the present invention is implemented
Example provides a kind of data integrating method based on above-mentioned data integrated system, comprising:
Step 201, the data integration request that user terminal is sent is obtained；
Step 202, pending data is extracted from multiple data sources according to data integration request, and generated corresponding
Data integration task；
Step 203, the pending data of the data integration task is analyzed and processed, the mesh of preset condition will be met
Data integration is marked to target database, so that user terminal transfers the target data.
In embodiments of the present invention, firstly, by step 201, the data integration request that user terminal is sent is obtained.It needs
Illustrate, the embodiment of the present invention can provide data integration service to enterprises, can also be business by SaaS Apps
Affiliate provides data integration service, and user is based on visualization window, makes a plan to data integration.Getting data set
After request, by step 202, pending data is extracted from multiple data sources according to data integration request, and raw
At corresponding data integration task.According to the data integration demand of user, from host system, relational database, extensible markup
Data to be processed are extracted in the data sources such as language, application program and spreadsheet, and are generated for these data to be processed
Corresponding data integration task.
Finally, being analyzed and processed, meeting pre- to the pending data of the data integration task by step 203
If the target data of condition integrates target database, so that user terminal transfers the target data.In the embodiment of the present invention
In, these pending datas are analyzed and processed, standard are executed according to preset data, to the number with potential quality problems
It is deleted according to being marked, and by mistake or duplicate data, to guarantee the quality, accuracy and integrality of data, most
Afterwards by these treated target data integrates target databases, user can be carried out these data by visualization window
Scheduling or inquiry.
The embodiment of the present invention is given birth to data integration demand by providing visualized operation for amateur programming personnel automatically
At corresponding data integration task, to complete data integration job, complicated programming grammar is avoided, making for terminal user is reduced
With difficulty, it ensure that the standard of delivery of data integration scheme is unified, alleviate follow-up maintenance workload, improve data integration
Quality.
On the basis of the above embodiments, it is carried out at analysis in the pending data to the data integration task
Reason, integrates target database for the target data for meeting preset condition, after transferring the target data for user terminal,
The method also includes:
Data monitoring is carried out to the target database, according to the more new state of corresponding data in data source, to the mesh
Mark database synchronizes.
Fig. 3 is electronic devices structure schematic diagram provided in an embodiment of the present invention, as shown in figure 3, the electronic equipment can wrap
It includes: processor (Processor) 301, communication interface (Communications Interface) 302, memory (Memory)
303 and communication bus 304, wherein processor 301, communication interface 302, memory 303 are completed mutually by communication bus 304
Between communication.Processor 301 can call the logical order in memory 303, to execute following method: obtain user terminal hair
The data integration request sent；Pending data is extracted from multiple data sources according to data integration request, and generates correspondence
Data integration task；The pending data of the data integration task is analyzed and processed, the mesh of preset condition will be met
Data integration is marked to target database, so that user terminal transfers the target data.
In addition, the logical order in above-mentioned memory 303 can be realized by way of SFU software functional unit and conduct
Independent product when selling or using, can store in a computer readable storage medium.Based on this understanding, originally
Substantially the part of the part that contributes to existing technology or the technical solution can be in other words for the technical solution of invention
The form of software product embodies, which is stored in a storage medium, including some instructions to
So that a computer equipment (can be personal computer, server or the network equipment etc.) executes each implementation of the present invention
The all or part of the steps of example the method.And storage medium above-mentioned include: USB flash disk, mobile hard disk, read-only memory (ROM,
Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic or disk etc. it is various
It can store the medium of program code.
The embodiment of the present invention discloses a kind of computer program product, and the computer program product is non-transient including being stored in
Computer program on computer readable storage medium, the computer program include program instruction, when described program instructs quilt
When computer executes, computer is able to carry out method provided by above-mentioned each method embodiment, for example, obtains user terminal
The data integration of transmission is requested；Pending data, and generation pair are extracted from multiple data sources according to data integration request
The data integration task answered；The pending data of the data integration task is analyzed and processed, preset condition will be met
Target data integrates target database, so that user terminal transfers the target data.
The embodiment of the present invention provides a kind of non-transient computer readable storage medium, the non-transient computer readable storage medium
The instruction of matter storage server, the computer instruction make computer execute data integrating method provided by above-described embodiment, such as
It include: the data integration request for obtaining user terminal and sending；According to the data integration request from multiple data sources extract to
Data are handled, and generate corresponding data integration task；The pending data of the data integration task is analyzed and processed,
The target data for meeting preset condition is integrated into target database, so that user terminal transfers the target data.
Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention, rather than its limitations；Although
Present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that: it still may be used
To modify the technical solutions described in the foregoing embodiments or equivalent replacement of some of the technical features；
And these are modified or replaceed, technical solution of various embodiments of the present invention that it does not separate the essence of the corresponding technical solution spirit and
Range.
Claims (8)
1. a kind of data integrated system characterized by comprising data integration task module, execution module and control module,
In:
The data integration task module is obtained based on Eclipse RCP platform construction, multiple data sources is connected, according to described
The data integration instruction that control module is sent, extracts pending data, and generate corresponding data integration from multiple data sources
Task, so that the execution module handles the pending data；
The execution module is analyzed and processed the pending data, will meet for executing the data integration task
The target data of preset condition integrates target database, so that user terminal transfers the target data；
The control module, for sending the data integration instruction to the data integration task module, and to the execution
Module is scheduled control, sends the user terminal for the target data.
2. system according to claim 1, which is characterized in that the multiple data source includes: host system, relation data
Library, extensible markup language, application program and spreadsheet.
3. system according to claim 2, which is characterized in that the control module is also used to obtain the execution module
Status log, to carry out data synchronization monitoring to the execution module.
4. system according to claim 3, which is characterized in that the control module includes:
Instruction input unit, for providing application programming interface, to obtain the assignment instructions of data integration；
Display unit, for carrying out visualization display to the executive condition of data integration.
5. a kind of data integrating method based on the described in any item data integrated systems of Claims 1-4, which is characterized in that
Include:
Obtain the data integration request that user terminal is sent；
Pending data is extracted from multiple data sources according to data integration request, and is generated corresponding data integration and appointed
Business；
The pending data of the data integration task is analyzed and processed, the target data for meeting preset condition is integrated
Target database, so that user terminal transfers the target data.
6. according to the method described in claim 5, it is characterized in that, in the pending data to the data integration task
It is analyzed and processed, the target data for meeting preset condition is integrated into target database, so that user terminal transfers the mesh
After marking data, the method also includes:
Data monitoring is carried out to the target database, according to the more new state of corresponding data in data source, to the number of targets
It is synchronized according to library.
7. a kind of electronic equipment including memory, processor and stores the calculating that can be run on a memory and on a processor
Machine program, which is characterized in that the processor is realized when executing described program such as any one of claim 5 to 6 the method
Step.
8. a kind of non-transient computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer journey
It is realized when sequence is executed by processor such as the step of any one of claim 5 to 6 the method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811558696.4A CN109800090A (en) | 2018-12-19 | 2018-12-19 | A kind of data integrated system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811558696.4A CN109800090A (en) | 2018-12-19 | 2018-12-19 | A kind of data integrated system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109800090A true CN109800090A (en) | 2019-05-24 |
Family
ID=66557315
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811558696.4A Pending CN109800090A (en) | 2018-12-19 | 2018-12-19 | A kind of data integrated system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109800090A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110765297A (en) * | 2019-11-01 | 2020-02-07 | 广东三维家信息科技有限公司 | Picture data management method and device and electronic equipment |
| CN111984740A (en) * | 2020-07-28 | 2020-11-24 | 招联消费金融有限公司 | Business data table processing method and device, computer equipment and storage medium |
| CN113010421A (en) * | 2021-03-16 | 2021-06-22 | 北京奇艺世纪科技有限公司 | Data processing method and device, electronic equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150095303A1 (en) * | 2013-09-27 | 2015-04-02 | Futurewei Technologies, Inc. | Knowledge Graph Generator Enabled by Diagonal Search |
| CN104809151A (en) * | 2015-03-11 | 2015-07-29 | 同济大学 | Multi-dimension based traffic heterogeneous data integrating method |
| CN105278373A (en) * | 2015-10-16 | 2016-01-27 | 中国南方电网有限责任公司电网技术研究中心 | Method for realizing comprehensive information processing system of transformer substation |
| CN105488229A (en) * | 2016-01-20 | 2016-04-13 | 航天科工智慧产业发展有限公司 | Data exchange and integration method applied to heterogeneous data environment |
| US20160283511A1 (en) * | 2015-03-24 | 2016-09-29 | International Business Machines Corporation | Systems and methods for query evaluation over distributed linked data stores |
| CN106203828A (en) * | 2016-07-11 | 2016-12-07 | 浪潮软件集团有限公司 | Data management platform based on data full life cycle management |
| CN107528864A (en) * | 2016-06-20 | 2017-12-29 | 中国科学院微电子研究所 | Heterogeneous network data processing method and system |
-
2018
- 2018-12-19 CN CN201811558696.4A patent/CN109800090A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150095303A1 (en) * | 2013-09-27 | 2015-04-02 | Futurewei Technologies, Inc. | Knowledge Graph Generator Enabled by Diagonal Search |
| CN104809151A (en) * | 2015-03-11 | 2015-07-29 | 同济大学 | Multi-dimension based traffic heterogeneous data integrating method |
| US20160283511A1 (en) * | 2015-03-24 | 2016-09-29 | International Business Machines Corporation | Systems and methods for query evaluation over distributed linked data stores |
| CN105278373A (en) * | 2015-10-16 | 2016-01-27 | 中国南方电网有限责任公司电网技术研究中心 | Method for realizing comprehensive information processing system of transformer substation |
| CN105488229A (en) * | 2016-01-20 | 2016-04-13 | 航天科工智慧产业发展有限公司 | Data exchange and integration method applied to heterogeneous data environment |
| CN107528864A (en) * | 2016-06-20 | 2017-12-29 | 中国科学院微电子研究所 | Heterogeneous network data processing method and system |
| CN106203828A (en) * | 2016-07-11 | 2016-12-07 | 浪潮软件集团有限公司 | Data management platform based on data full life cycle management |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110765297A (en) * | 2019-11-01 | 2020-02-07 | 广东三维家信息科技有限公司 | Picture data management method and device and electronic equipment |
| CN111984740A (en) * | 2020-07-28 | 2020-11-24 | 招联消费金融有限公司 | Business data table processing method and device, computer equipment and storage medium |
| CN111984740B (en) * | 2020-07-28 | 2023-07-25 | 招联消费金融有限公司 | Service data table processing method, device, computer equipment and storage medium |
| CN113010421A (en) * | 2021-03-16 | 2021-06-22 | 北京奇艺世纪科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113010421B (en) * | 2021-03-16 | 2023-09-01 | 北京奇艺世纪科技有限公司 | Data processing method, device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10860905B1 (en) | Long running workflows for document processing using robotic process automation | |
| US10949807B2 (en) | Model building architecture and smart routing of work items | |
| JP2022062036A (en) | Graph generation for distributed event processing system | |
| US20140358844A1 (en) | Workflow controller compatibility | |
| CN111066039A (en) | Microprocessor including enterprise model | |
| US11977470B2 (en) | Monitoring long running workflows for robotic process automation | |
| US11055454B1 (en) | Configuring and deploying Monte Carlo simulation pipelines | |
| US20150193404A1 (en) | Operational transformations proxy for thin clients | |
| CN109800090A (en) | A kind of data integrated system and method | |
| US10249197B2 (en) | Method and system for mission planning via formal verification and supervisory controller synthesis | |
| US20210397447A1 (en) | Automated compliance and testing framework for software development | |
| CN109657177A (en) | The generation method of the page, device, storage medium and computer equipment after upgrading | |
| US20140258250A1 (en) | Flexible Control Framework Featuring Standalone Rule Engine | |
| CN111190892A (en) | Method and device for processing abnormal data in data backfilling | |
| US20150012502A1 (en) | Big Data Centralized Intelligence System | |
| CN109960212B (en) | Task sending method and device | |
| CN111178849A (en) | Linear process engine implementation method, device, equipment and storage medium | |
| US20180189039A1 (en) | Automatic generation of manual coding suggestions for conversion of application programs to off-line environment | |
| US20180189038A1 (en) | Automatic conversion of application program code listing segments for off-line environment | |
| US11226885B1 (en) | Monte Carlo simulation monitoring and optimization | |
| US11741045B2 (en) | Format agnostic database migration | |
| US11797770B2 (en) | Self-improving document classification and splitting for document processing in robotic process automation | |
| US20220100964A1 (en) | Deep learning based document splitter | |
| US20210182284A1 (en) | System and method for data ingestion and workflow generation | |
| US9413684B2 (en) | Provisioning legacy systems network architecture resource communications through a group of servers in a data center |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190524 |