CN108763387A - Big data fusion method, electronic equipment, storage medium and the system of heterogeneous platform - Google Patents
Big data fusion method, electronic equipment, storage medium and the system of heterogeneous platform Download PDFInfo
- Publication number
- CN108763387A CN108763387A CN201810484553.7A CN201810484553A CN108763387A CN 108763387 A CN108763387 A CN 108763387A CN 201810484553 A CN201810484553 A CN 201810484553A CN 108763387 A CN108763387 A CN 108763387A
- Authority
- CN
- China
- Prior art keywords
- data
- source
- delta
- delta data
- destination end
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides the big data fusion method of heterogeneous platform, including step：The fileinfo of user configuration is obtained, fileinfo includes source coding, destination end coding, address, data interaction frequency, data mapping relations；The data of source database are extracted by data synchronization means, obtains and extracts data, and the file system of source and extraction data are subjected to journal formatting storage, generate source data；The variation for monitoring source data in real time, obtains delta data, delta data is sent to message queue component；The subject information for subscribing to message queue component, parses delta data, delta data is converted to the corresponding data of destination end storage format according to configuration file；Delta data is synchronized to target end system according to the data source of destination end.The invention further relates to a kind of electronic equipment, the big data emerging systems of storage medium, heterogeneous platform.The present invention realizes data synchronization and Data Conversion Service between isomorphism or heterogeneous platform rapidly and efficiently.
Description
Technical field
The present invention relates to technical field of data processing more particularly to the big data fusion method of heterogeneous platform, electronic equipment,
Storage medium and system.
Background technology
Continuous development with internet+and big data and large-scale application, all trades and professions number while resource-sharing
Normality is had become according to shared.However the information system of most enterprises with outside melt using customization when data are docked at present
Hair, the implementation cycle of customized development is long, of high cost, and repeated customized development has resulted in the wasting of resources, and as current
The common shared barrier of all trades and professions data.Existing system data transmission sequentially executes source on an equal basis by destination end by source
Data variation, this requires source and destination end Data Structure Design having the same.This mode disposes complexity, needs related special
It could use after industry personnel's long-time Learning Studies, and without transparence, visual operative configuration interface, only support
It is synchronized between the databases such as Oracle, Mysql, does not support the data types such as WepService services, file system, XML
Data synchronize.
Invention content
The present invention is based at least one above-mentioned technical problem, it is proposed that the big data fusion method of heterogeneous platform, electricity
Sub- equipment, storage medium and system, big data quickly, can not be docked efficiently between solving the problems, such as current heterogeneous platform.
In order to achieve the above objectives, the present invention provides the big data fusion method of heterogeneous platform, includes the following steps：
Configuration file is obtained, the fileinfo of user configuration is obtained, the fileinfo includes source coding, destination end volume
Code, address, data interaction frequency, data mapping relations；
Source data are generated, the data of source database are extracted by data synchronization means, obtains and extracts data, it will be described
The file system of source and the extraction data carry out journal formatting storage, generate source data；
Monitored data changes, and monitors the variation of the source data in real time, obtains delta data, the delta data is sent out
It send to message queue component；
Delta data is parsed, the subject information of the message queue component is subscribed to, the delta data is parsed, root
The delta data is converted into the corresponding data of destination end storage format according to the configuration file；
Synchronous delta data, target end system is synchronized to according to the data source of the destination end by the delta data.
Further, the data synchronization means is specially Oracle Golden Gate, and the step generates source number
According to the transaction log for capturing source database in real time specifically by Oracle Golden Gate, by the file system of the source
System and the transaction log carry out journal formatting storage, generate the source data.
Further, the message queue component is specially Kafka, and the step monitored data variation is specially to supervise in real time
The variation of the source data is listened, delta data is obtained, the delta data is sent to Kafka.
Further, step parsing delta data be specially subscribe to Kafka subject informations to the delta data into
Row JSON string de-parsing, the corresponding number of destination end storage format is converted to according to the configuration file by the delta data after parsing
According to.
A kind of electronic equipment, including memory, processor and storage are on a memory and the meter that can run on a processor
Calculation machine program realizes the big data fusion method for stating heterogeneous platform when the processor executes described program.
A kind of computer readable storage medium, is stored thereon with computer program, and the computer program is held by processor
The big data fusion method of above-mentioned heterogeneous platform is realized when row.
The big data emerging system of heterogeneous platform, including：
Profile module：For profile information, the fileinfo include source coding, destination end coding,
Location, data interaction frequency, data mapping relations；
Generate source data module：Data for extracting source database by data synchronization means obtain and extract number
According to by the file system of the source and extraction data progress journal formatting storage, generation source data；
Monitored data changes module：Variation for monitoring the source data in real time obtains delta data, by the change
Change data and is sent to message queue component；
Parse delta data module：Subject information for subscribing to the message queue component, to the delta data into
Row parsing, the corresponding data of destination end storage format are converted to according to the configuration file by the delta data；
Synchronous delta data module：For the delta data to be synchronized to destination end according to the data source of the destination end
System.
Further, the data synchronization means is specially Oracle Golden Gate, the generation source data mould
Block captures the transaction log of source database specifically by Oracle Golden Gate in real time, by the file system of the source
System and the transaction log carry out journal formatting storage, generate the source data.
Further, the message queue component is specially Kafka, and the monitored data variation module is specially to supervise in real time
The variation of the source data is listened, delta data is obtained, the delta data is sent to Kafka.
Further, the parsing delta data module be specially subscribe to Kafka subject informations to the delta data into
Row JSON string de-parsing, the corresponding number of destination end storage format is converted to according to the configuration file by the delta data after parsing
According to.
Compared with prior art, advantage of the invention is that：The present invention provides the big data fusion method of heterogeneous platform, packet
It includes step and obtains configuration file, generate source data, monitored data variation parses delta data, and synchronous delta data is obtained and used
The fileinfo of family configuration, fileinfo include source coding, destination end coding, address, data interaction frequency, data mapping pass
System；The data of source database are extracted by data synchronization means, obtains and extracts data, by the file system of source and extract number
According to journal formatting storage is carried out, source data are generated；The variation for monitoring source data in real time, obtains delta data, will change
Data are sent to message queue component；The subject information for subscribing to message queue component, parses delta data, according to configuration
Delta data is converted to the corresponding data of destination end storage format by file；Delta data is synchronized according to the data source of destination end
To target end system.The invention further relates to a kind of electronic equipment, the big data emerging systems of storage medium, heterogeneous platform.This hair
Delta data is simultaneously synchronized to target end system by bright variation by monitoring source data in real time according to configuration rule, is supported in short-term
Between large-scale data fusion demand, elastic dilatation can be carried out with the demand of business；Deployment is easy, supports Windows,
The kinds of platform such as Linux are served by；By profile information, the big data fusion for meeting user's self-demand is realized
Demand；The present invention can be grasped whenever and wherever possible using C/S application deployment, all configuration process and service status visualization, user
Data fusion operating condition and relevant abnormalities information between platform eliminate most of data review synchronization means command operation at present
Inconvenient drawback；The present invention realizes that the data rapidly and efficiently between isomorphism or heterogeneous platform are synchronous and Data Conversion Service.
Above description is only the general introduction of technical solution of the present invention, in order to better understand the technical means of the present invention,
And can be implemented in accordance with the contents of the specification, below with presently preferred embodiments of the present invention and after coordinating attached drawing to be described in detail such as.
The specific implementation mode of the present invention is shown in detail by following embodiment and its attached drawing.
Description of the drawings
It is described in further detail below in conjunction with the accompanying drawings with embodiments of the present invention.
Fig. 1 is the big data fusion method flow chart of the heterogeneous platform of the present invention；
Fig. 2 is the big data emerging system structural schematic diagram of the heterogeneous platform of the present invention；
Fig. 3 is the big data emerging system integrated stand composition of the heterogeneous platform of the embodiment of the present invention；
Fig. 4 is that the big data emerging system of the heterogeneous platform of the embodiment of the present invention disposes Organization Chart.
Specific implementation mode
Understand to make the objectives, technical solutions, and advantages of the present invention more remove, it is with reference to the accompanying drawings and embodiments, right
The present invention is further elaborated.It should be appreciated that specific embodiment described herein is only used to explain the present invention, not
For limiting the present invention.
The big data fusion method of heterogeneous platform, as shown in Figure 1, including the following steps：
Obtain configuration file, obtain the fileinfo of user configuration, fileinfo include source coding, destination end coding,
Address, data interaction frequency, data mapping relations.
Source data are generated, the data of source database are extracted by data synchronization means, obtains and extracts data, by source
File system and extract data carry out journal formatting storage, generate source data.
Monitored data changes, and monitors the variation of source data in real time, and by supporting relational database system or distribution
Correlation table record information or file system journal acquisition of information delta data, delta data is sent to and is disappeared in data file system
Breath queue element is ranked up buffering and keeps in.
Delta data is parsed, the subject information of message queue component is subscribed to, delta data is parsed, according to configuration text
Delta data is converted to the corresponding data of destination end storage format by part.
Synchronous delta data, is synchronized to target end system, such as relationship type number according to the data source of destination end by delta data
According to library Oracle, Mysql, file system, HDFS, RestFul Web service.Between meeting distinct type data-base, not identical text
Data fusion between part system, as any type of system interface carries out data interaction between Web service or API.Pass through reality
When monitor the variation of source data and delta data be synchronized to target end system according to configuration rule, support the short time extensive
Data fusion demand, can with the demand of business carry out elastic dilatation.
In one embodiment, it is preferred that data synchronization means is specially Oracle Golden Gate, and step generates source
Data capture the transaction log of source database specifically by Oracle Golden Gate in real time, and transaction log is to have been filed on
The file system of source and transaction log are carried out journal formatting by data including DML and DDL, can be filtered according to rule
Storage generates source data.
In one embodiment, it is preferred that message queue component is specially Kafka, and the variation of step monitored data is specially real
When monitor source data variation, obtain delta data, delta data is sent to Kafka.Preferably, step parsing variation number
According to being specially to subscribe to Kafka subject informations and carry out JSON to delta data to go here and there de-parsing, according to configuration file by the change after parsing
Change data and is converted to the corresponding data of destination end storage format.
A kind of electronic equipment, including memory, processor and storage are on a memory and the meter that can run on a processor
Calculation machine program realizes the big data fusion method for stating heterogeneous platform when processor executes program.
A kind of computer readable storage medium is stored thereon with computer program, when computer program is executed by processor
Realize the big data fusion method of above-mentioned heterogeneous platform.
The big data emerging system of heterogeneous platform, is disposed using C/S, and all configuration process and service status visualization are used
Family can grasp data fusion operating condition and relevant abnormalities information between platform whenever and wherever possible, and it is multiple to eliminate most of data at present
The drawback of core synchronization means command operation inconvenience.As shown in figs 2-4, system supports concentrating type Platform deployment mode also to support to divide
Cloth deployment way, respectively as servo, data are logical by DXH LMIS, BEIJING TMS, GD TPL, GD LMIS, JLP in Fig. 4
It crosses message queue component Kafka to be ranked up buffering and keep in, flow configuration is carried out by streaming computing engine and node configures,
Data manipulation is carried out by atomic service and business calculates, realizes the data fusion between different platform, and system deployment is easy, branch
Windows is held, the kinds of platform such as Linux are served by, and system includes：
Profile module profile information, fileinfo include source coding, destination end coding, address, data friendship
Crossing over frequency, data mapping relations.In the present embodiment, profile module provides page configuration service, operating parameter configuration, operation
Log services.
The data that source data module extracts source database by data synchronization means are generated, obtains and extracts data, it will
The file system and extraction data of source carry out journal formatting storage, generate source data.
Monitored data variation module monitors the variation of source data in real time, and by supporting relational database system or distribution
Correlation table record information or file system journal acquisition of information delta data, delta data is sent in formula data file system
Message queue component is ranked up buffering and keeps in.In the present embodiment, monitored data variation module offer file monitoring service,
SQL analysis services, message packing service, message send service.
Parse delta data module subscribe to message queue component subject information, delta data is parsed, according to
It sets file and delta data is converted into the corresponding data of destination end storage format.In the present embodiment, parsing delta data module carries
For message subscribing service, Context resolution service.
Delta data is synchronized to target end system, such as relationship type by synchronous delta data module according to the data source of destination end
Database Oracle, Mysql, file system, HDFS, RestFul Web service.It is different between meeting distinct type data-base
Data fusion between file system, as any type of system interface carries out data interaction between Web service or API.Pass through
Monitored data variation module monitor in real time the variation of source data and by synchronous delta data module by delta data according to matching
Regular and synchronized is set to target end system, short time large-scale data fusion demand is supported, can be carried out with the demand of business
Elastic dilatation.In the present embodiment, synchronous delta data module provides mapping relation service, data dump service.
In one embodiment, as shown in Figure 3, it is preferred that data synchronization means is specially Oracle Golden Gate, raw
Capture the transaction log of source database, the day of trade in real time specifically by Oracle Golden Gate at source data module
Will is has been filed on data, including DML and DDL, can be filtered according to rule, database Database, by the file of source
System and transaction log carry out journal formatting storage, generate source data.
In one embodiment, as shown in Figure 3, it is preferred that message queue component is specially Kafka, monitored data changing pattern
Block is specially the variation for monitoring source data in real time, obtains delta data, delta data is sent to Kafka, passes through general number
The messenger service of data source and data configuration are sent to Kafka according to interface.Preferably, parsing delta data module is specially
It subscribes to Kafka subject informations and JSON string de-parsing is carried out to delta data, turned the delta data after parsing according to configuration file
The corresponding data of destination end storage format are changed to, log services frame is generated, synchronize number of the delta data module according to destination end
Delta data is synchronized to target end system, such as Redis/Kafka, file system, Database according to source.
The present invention provides the big data fusion method of heterogeneous platform, including step obtains configuration file, generates source data,
Monitored data changes, and parses delta data, and synchronous delta data obtains the fileinfo of user configuration, fileinfo includes source
Hold coding, destination end coding, address, data interaction frequency, data mapping relations；Source data are extracted by data synchronization means
The data in library obtain and extract data, by the file system of source and extract data progress journal formatting storage, generate source number
According to；The variation for monitoring source data in real time, obtains delta data, delta data is sent to message queue component；Subscribe to message
The subject information of queue element, parses delta data, and delta data is converted to destination end storage according to configuration file
The corresponding data of format；Delta data is synchronized to target end system according to the data source of destination end.The invention further relates to one kind
The big data emerging system of electronic equipment, storage medium, heterogeneous platform.The present invention by monitoring the variation of source data simultaneously in real time
Delta data is synchronized to target end system according to configuration rule, supports short time large-scale data fusion demand, Neng Gousui
The demand for business carries out elastic dilatation；Deployment is easy, and Windows, the kinds of platform such as Linux is supported to be served by；Pass through use
Family profile information realizes the big data fusion demand for meeting user's self-demand；The present invention is using C/S application deployment, institute
There are a configuration process and service status visualization, user can grasp between platform data fusion operating condition and related different whenever and wherever possible
Normal information eliminates the drawback of most of data review synchronization means command operation inconvenience at present；The present invention realizes isomorphism or different
Data rapidly and efficiently between structure platform are synchronous and Data Conversion Service.
Each technical characteristic of above example can be combined arbitrarily, to keep description succinct, not to above-described embodiment
In each technical characteristic it is all possible combination be all described, as long as however, the combination of these technical characteristics be not present lance
Shield is all considered to be the range of this specification record.
Only several embodiments of the present invention are expressed for above example, the description thereof is more specific and detailed, but can not
Therefore it is construed as limiting the scope of the patent.It should be pointed out that for those of ordinary skill in the art,
Under the premise of not departing from present inventive concept, various modifications and improvements can be made, these are all within the scope of protection of the present invention.
Therefore, the protection domain of patent of the present invention should be determined by the appended claims.
Claims (10)
1. the big data fusion method of heterogeneous platform, it is characterised in that include the following steps：
Obtain configuration file, obtain the fileinfo of user configuration, the fileinfo include source coding, destination end coding,
Address, data interaction frequency, data mapping relations；
Source data are generated, the data of source database are extracted by data synchronization means, obtains and extracts data, by the source
File system and the extraction data carry out journal formatting storage, generate source data；
Monitored data changes, and monitors the variation of the source data in real time, obtains delta data, the delta data is sent to
Message queue component；
Delta data is parsed, the subject information of the message queue component is subscribed to, the delta data is parsed, according to institute
It states configuration file and the delta data is converted into the corresponding data of destination end storage format；
Synchronous delta data, target end system is synchronized to according to the data source of the destination end by the delta data.
2. the big data fusion method of heterogeneous platform as described in claim 1, it is characterised in that：The data synchronization means tool
Body is Oracle Golden Gate, and the step generates source data and caught in real time specifically by Oracle Golden Gate
The file system of the source and the transaction log are carried out journal formatting storage by the transaction log for obtaining source database,
Generate the source data.
3. the big data fusion method of heterogeneous platform as described in claim 1, it is characterised in that：The message queue component tool
Body is Kafka, and the variation of the source data is specially monitored in the step monitored data variation in real time, obtains delta data,
The delta data is sent to Kafka.
4. the big data fusion method of heterogeneous platform as claimed in claim 3, it is characterised in that：The step parsing variation number
According to being specially to subscribe to Kafka subject informations to carry out JSON string de-parsing to the delta data, will be solved according to the configuration file
Delta data after analysis is converted to the corresponding data of destination end storage format.
5. a kind of electronic equipment, including memory, processor and storage are on a memory and the calculating that can run on a processor
Machine program, which is characterized in that the processor realizes the step of any one of claim 1-4 methods when executing described program.
6. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program quilt
The step of any one of claim 1-4 methods are realized when processor executes.
7. the big data emerging system of heterogeneous platform, it is characterised in that including：
Profile module：For profile information, the fileinfo includes source coding, destination end coding, address, number
According to frequency of interaction, data mapping relations；
Generate source data module：Data for extracting source database by data synchronization means obtain and extract data, will
The file system of the source and the extraction data carry out journal formatting storage, generate source data；
Monitored data changes module：Variation for monitoring the source data in real time obtains delta data, by the variation number
According to being sent to message queue component；
Parse delta data module：Subject information for subscribing to the message queue component, solves the delta data
Analysis, the corresponding data of destination end storage format are converted to according to the configuration file by the delta data；
Synchronous delta data module：For the delta data to be synchronized to destination end system according to the data source of the destination end
System.
8. the big data emerging system of heterogeneous platform as claimed in claim 7, it is characterised in that：The data synchronization means tool
Body is Oracle Golden Gate, and the generation source data module is caught in real time specifically by Oracle Golden Gate
The file system of the source and the transaction log are carried out journal formatting storage by the transaction log for obtaining source database,
Generate the source data.
9. the big data emerging system of heterogeneous platform as claimed in claim 7, it is characterised in that：The message queue component tool
Body is Kafka, and the monitored data variation module specially monitors the variation of the source data in real time, obtains delta data,
The delta data is sent to Kafka.
10. the big data emerging system of heterogeneous platform as claimed in claim 9, it is characterised in that：The parsing delta data
Module is specially to subscribe to Kafka subject informations to carry out JSON string de-parsing to the delta data, will according to the configuration file
Delta data after parsing is converted to the corresponding data of destination end storage format.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810484553.7A CN108763387A (en) | 2018-05-20 | 2018-05-20 | Big data fusion method, electronic equipment, storage medium and the system of heterogeneous platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810484553.7A CN108763387A (en) | 2018-05-20 | 2018-05-20 | Big data fusion method, electronic equipment, storage medium and the system of heterogeneous platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108763387A true CN108763387A (en) | 2018-11-06 |
Family
ID=64007166
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810484553.7A Pending CN108763387A (en) | 2018-05-20 | 2018-05-20 | Big data fusion method, electronic equipment, storage medium and the system of heterogeneous platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108763387A (en) |
Cited By (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109660610A (en) * | 2018-12-07 | 2019-04-19 | 北京奇虎科技有限公司 | A kind of data processing method, device, equipment and storage medium |
| CN109815028A (en) * | 2018-12-27 | 2019-05-28 | 北京摩拜科技有限公司 | Data synchronous system, method, apparatus and computer storage medium |
| CN109947464A (en) * | 2019-03-22 | 2019-06-28 | 优信拍（北京）信息科技有限公司 | A kind of configuration update method and device |
| CN109992595A (en) * | 2019-04-11 | 2019-07-09 | 北京启迪区块链科技发展有限公司 | Different database conversion method, apparatus, equipment and storage medium |
| CN110134737A (en) * | 2019-05-20 | 2019-08-16 | 中国铁道科学研究院集团有限公司 | Data variation monitor method and device, electronic equipment and computer readable storage medium |
| CN110309206A (en) * | 2019-07-10 | 2019-10-08 | 中国联合网络通信集团有限公司 | Order information acquisition method and system |
| CN110427426A (en) * | 2019-08-02 | 2019-11-08 | 中国工商银行股份有限公司 | A kind of data synchronizing processing method and device |
| CN110555583A (en) * | 2019-07-02 | 2019-12-10 | 国网浙江省电力有限公司 | method for uniformly processing wide-area operation data of intelligent power grid dispatching control system |
| CN110784419A (en) * | 2019-10-22 | 2020-02-11 | 中国铁道科学研究院集团有限公司电子计算技术研究所 | Method and system for visualizing professional data of railway electric affairs |
| CN111460038A (en) * | 2020-04-07 | 2020-07-28 | 中国建设银行股份有限公司 | Quasi-real-time data synchronization method and device |
| CN111782886A (en) * | 2020-06-28 | 2020-10-16 | 杭州海康威视数字技术股份有限公司 | Method and device for managing metadata |
| CN111813850A (en) * | 2019-04-11 | 2020-10-23 | 百度在线网络技术（北京）有限公司 | Heterogeneous data synchronization method and device, electronic equipment and storage medium |
| CN111984715A (en) * | 2020-08-19 | 2020-11-24 | 银盛支付服务股份有限公司 | Heterogeneous data synchronous processing method and system |
| CN112287021A (en) * | 2020-07-13 | 2021-01-29 | 上海柯林布瑞信息技术有限公司 | Data real-time synchronization parameter generation method, data real-time synchronization parameter synchronization method, data real-time synchronization parameter generation device, data real-time synchronization parameter synchronization device, storage medium and terminal |
| CN112380229A (en) * | 2020-11-16 | 2021-02-19 | 中消云（北京）物联网科技研究院有限公司 | Service data synchronization method and device, nonvolatile storage medium and processor |
| CN112507013A (en) * | 2021-02-07 | 2021-03-16 | 北京工业大数据创新中心有限公司 | Industrial equipment data storage method and device |
| CN112788074A (en) * | 2019-11-07 | 2021-05-11 | 中兴通讯股份有限公司 | Data transmitting method, processing method, receiving method and equipment and storage medium |
| CN113010607A (en) * | 2021-04-06 | 2021-06-22 | 工银科技有限公司 | Method, device, computer system and storage medium for data synchronization between systems |
| CN113220791A (en) * | 2021-06-03 | 2021-08-06 | 西安热工研究院有限公司 | Data cascade synchronization system and method |
| CN113836224A (en) * | 2021-09-07 | 2021-12-24 | 南方电网大数据服务有限公司 | Method and device for processing synchronous files from OGG (one glass solution) to HDFS (Hadoop distributed File System) and computer equipment |
| CN115150466A (en) * | 2022-06-29 | 2022-10-04 | 北京百度网讯科技有限公司 | Method and device for realizing data distribution, electronic equipment and storage medium |
| CN115203336A (en) * | 2022-09-19 | 2022-10-18 | 平安银行股份有限公司 | Database data real-time synchronization method, system, computer terminal and storage medium |
| CN115827777A (en) * | 2022-11-21 | 2023-03-21 | 中国人民财产保险股份有限公司 | Self-adaptive synchronization and difference identification method, device and equipment for multiple data sources |
| CN117194549A (en) * | 2023-11-07 | 2023-12-08 | 上海柯林布瑞信息技术有限公司 | Data transmission method and device based on task data configuration |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102495910A (en) * | 2011-12-28 | 2012-06-13 | 畅捷通信息技术股份有限公司 | Device and method for data timing synchronization of heterogeneous system |
| CN103699580A (en) * | 2013-12-03 | 2014-04-02 | 中铁程科技有限责任公司 | Database synchronization method and database synchronization device |
| CN103984715A (en) * | 2014-05-08 | 2014-08-13 | 武汉库百网络技术有限公司 | Data synchronizing and checking method, device and system of isomerous database |
| CN106557592A (en) * | 2016-12-02 | 2017-04-05 | 中铁程科技有限责任公司 | Method of data synchronization, device and server cluster |
| CN107491558A (en) * | 2017-09-08 | 2017-12-19 | 北京奇艺世纪科技有限公司 | Metadata updates method and device |
| US20180032596A1 (en) * | 2016-07-28 | 2018-02-01 | Invensys Systems, Inc. | Summarization retrieval in a process control environment |
| CN107943979A (en) * | 2017-11-29 | 2018-04-20 | 山东鲁能软件技术有限公司 | The quasi real time synchronous method and device of data between a kind of database |
| US20180150541A1 (en) * | 2016-11-28 | 2018-05-31 | Sap Se | Proxy Views for Extended Monitoring of Database Systems |
-
2018
- 2018-05-20 CN CN201810484553.7A patent/CN108763387A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102495910A (en) * | 2011-12-28 | 2012-06-13 | 畅捷通信息技术股份有限公司 | Device and method for data timing synchronization of heterogeneous system |
| CN103699580A (en) * | 2013-12-03 | 2014-04-02 | 中铁程科技有限责任公司 | Database synchronization method and database synchronization device |
| CN103984715A (en) * | 2014-05-08 | 2014-08-13 | 武汉库百网络技术有限公司 | Data synchronizing and checking method, device and system of isomerous database |
| US20180032596A1 (en) * | 2016-07-28 | 2018-02-01 | Invensys Systems, Inc. | Summarization retrieval in a process control environment |
| US20180150541A1 (en) * | 2016-11-28 | 2018-05-31 | Sap Se | Proxy Views for Extended Monitoring of Database Systems |
| CN106557592A (en) * | 2016-12-02 | 2017-04-05 | 中铁程科技有限责任公司 | Method of data synchronization, device and server cluster |
| CN107491558A (en) * | 2017-09-08 | 2017-12-19 | 北京奇艺世纪科技有限公司 | Metadata updates method and device |
| CN107943979A (en) * | 2017-11-29 | 2018-04-20 | 山东鲁能软件技术有限公司 | The quasi real time synchronous method and device of data between a kind of database |
Cited By (33)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109660610B (en) * | 2018-12-07 | 2023-12-22 | 三六零科技集团有限公司 | Data processing method, device, equipment and storage medium |
| CN109660610A (en) * | 2018-12-07 | 2019-04-19 | 北京奇虎科技有限公司 | A kind of data processing method, device, equipment and storage medium |
| CN109815028A (en) * | 2018-12-27 | 2019-05-28 | 北京摩拜科技有限公司 | Data synchronous system, method, apparatus and computer storage medium |
| CN109815028B (en) * | 2018-12-27 | 2022-02-08 | 汉海信息技术(上海)有限公司 | System, method, apparatus and computer storage medium for data synchronization |
| CN109947464A (en) * | 2019-03-22 | 2019-06-28 | 优信拍（北京）信息科技有限公司 | A kind of configuration update method and device |
| CN109992595A (en) * | 2019-04-11 | 2019-07-09 | 北京启迪区块链科技发展有限公司 | Different database conversion method, apparatus, equipment and storage medium |
| CN111813850A (en) * | 2019-04-11 | 2020-10-23 | 百度在线网络技术（北京）有限公司 | Heterogeneous data synchronization method and device, electronic equipment and storage medium |
| CN110134737B (en) * | 2019-05-20 | 2021-02-26 | 中国铁道科学研究院集团有限公司 | Data change monitoring method and device, electronic equipment and computer readable storage medium |
| CN110134737A (en) * | 2019-05-20 | 2019-08-16 | 中国铁道科学研究院集团有限公司 | Data variation monitor method and device, electronic equipment and computer readable storage medium |
| CN110555583A (en) * | 2019-07-02 | 2019-12-10 | 国网浙江省电力有限公司 | method for uniformly processing wide-area operation data of intelligent power grid dispatching control system |
| CN110309206A (en) * | 2019-07-10 | 2019-10-08 | 中国联合网络通信集团有限公司 | Order information acquisition method and system |
| CN110427426A (en) * | 2019-08-02 | 2019-11-08 | 中国工商银行股份有限公司 | A kind of data synchronizing processing method and device |
| CN110784419B (en) * | 2019-10-22 | 2023-02-28 | 中国铁道科学研究院集团有限公司电子计算技术研究所 | Method and system for visualizing professional railway electric service data |
| CN110784419A (en) * | 2019-10-22 | 2020-02-11 | 中国铁道科学研究院集团有限公司电子计算技术研究所 | Method and system for visualizing professional data of railway electric affairs |
| CN112788074B (en) * | 2019-11-07 | 2024-05-31 | 中兴通讯股份有限公司 | Data transmitting method, processing method, receiving method, apparatus thereof, and storage medium |
| CN112788074A (en) * | 2019-11-07 | 2021-05-11 | 中兴通讯股份有限公司 | Data transmitting method, processing method, receiving method and equipment and storage medium |
| CN111460038A (en) * | 2020-04-07 | 2020-07-28 | 中国建设银行股份有限公司 | Quasi-real-time data synchronization method and device |
| CN111782886A (en) * | 2020-06-28 | 2020-10-16 | 杭州海康威视数字技术股份有限公司 | Method and device for managing metadata |
| CN112287021A (en) * | 2020-07-13 | 2021-01-29 | 上海柯林布瑞信息技术有限公司 | Data real-time synchronization parameter generation method, data real-time synchronization parameter synchronization method, data real-time synchronization parameter generation device, data real-time synchronization parameter synchronization device, storage medium and terminal |
| CN112287021B (en) * | 2020-07-13 | 2024-04-05 | 上海柯林布瑞信息技术有限公司 | Method and device for generating and synchronizing real-time data synchronization parameters, storage medium and terminal |
| CN111984715A (en) * | 2020-08-19 | 2020-11-24 | 银盛支付服务股份有限公司 | Heterogeneous data synchronous processing method and system |
| CN112380229A (en) * | 2020-11-16 | 2021-02-19 | 中消云（北京）物联网科技研究院有限公司 | Service data synchronization method and device, nonvolatile storage medium and processor |
| CN112507013A (en) * | 2021-02-07 | 2021-03-16 | 北京工业大数据创新中心有限公司 | Industrial equipment data storage method and device |
| CN113010607A (en) * | 2021-04-06 | 2021-06-22 | 工银科技有限公司 | Method, device, computer system and storage medium for data synchronization between systems |
| CN113220791B (en) * | 2021-06-03 | 2023-07-28 | 西安热工研究院有限公司 | Data cascading synchronization system and method |
| CN113220791A (en) * | 2021-06-03 | 2021-08-06 | 西安热工研究院有限公司 | Data cascade synchronization system and method |
| CN113836224A (en) * | 2021-09-07 | 2021-12-24 | 南方电网大数据服务有限公司 | Method and device for processing synchronous files from OGG (one glass solution) to HDFS (Hadoop distributed File System) and computer equipment |
| CN115150466A (en) * | 2022-06-29 | 2022-10-04 | 北京百度网讯科技有限公司 | Method and device for realizing data distribution, electronic equipment and storage medium |
| CN115150466B (en) * | 2022-06-29 | 2023-08-15 | 北京百度网讯科技有限公司 | Method and device for realizing data distribution, electronic equipment and storage medium |
| CN115203336A (en) * | 2022-09-19 | 2022-10-18 | 平安银行股份有限公司 | Database data real-time synchronization method, system, computer terminal and storage medium |
| CN115827777A (en) * | 2022-11-21 | 2023-03-21 | 中国人民财产保险股份有限公司 | Self-adaptive synchronization and difference identification method, device and equipment for multiple data sources |
| CN117194549B (en) * | 2023-11-07 | 2024-01-26 | 上海柯林布瑞信息技术有限公司 | Data transmission method and device based on task data configuration |
| CN117194549A (en) * | 2023-11-07 | 2023-12-08 | 上海柯林布瑞信息技术有限公司 | Data transmission method and device based on task data configuration |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108763387A (en) | Big data fusion method, electronic equipment, storage medium and the system of heterogeneous platform | |
| CN110784419B (en) | Method and system for visualizing professional railway electric service data | |
| CN110647512B (en) | Data storage and analysis method, device, equipment and readable medium | |
| US20160012150A1 (en) | System and method for main distribution network graph/model/data integration based on remote access and information interaction | |
| CN104536965B (en) | A kind of data query display systems under the conditions of big data and method | |
| CN106375149A (en) | Auto associating and analyzing cloud computing monitor apparatus and method | |
| CN106254145A (en) | network request tracking processing method and device | |
| CN110502491A (en) | A kind of Log Collect System and its data transmission method, device | |
| CN103412893A (en) | Collecting system and collecting method of logs | |
| CN104361031B (en) | A kind of government data pre-processing system and processing method | |
| CN109086410A (en) | The processing method and system of streaming mass data | |
| CN108133024A (en) | Towards the geographical spatial data service integration method of the dynamic configuration of mobile client | |
| CN109190025A (en) | information monitoring method, device, system and computer readable storage medium | |
| CN106598700A (en) | Second-level high availability realization method of virtual machine based on pacemaker | |
| Leao et al. | Big data processing for power grid event detection | |
| Liu et al. | System anomaly detection in distributed systems through MapReduce-Based log analysis | |
| CN104202328A (en) | GOOSE/SMV (generic object oriented substation event/sampled measured value) message subscribing method, GOOSE/SMV message subscribing configuration module and GOOSE/SMV message subscribing terminal | |
| CN109241510A (en) | A kind of autochart generation system and its implementation based on wechat small routine | |
| CN108712306A (en) | A kind of information system automation inspection platform and method for inspecting | |
| CN107124292A (en) | A kind of information system method of operation incidence relation dynamic creation method | |
| CN109165203A (en) | Large public building energy consumption data based on Hadoop framework stores analysis method | |
| CN106528352A (en) | Fault injection platform for transaction processing type fault-tolerant computer | |
| CN109525422A (en) | A kind of daily record data method for managing and monitoring | |
| Zakarija et al. | Discovering process model from incomplete log using process mining | |
| CN114756301A (en) | Log processing method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181106 |
|
| RJ01 | Rejection of invention patent application after publication |