CN104639585B - A kind of real-time data transmission system and its method of data synchronization - Google Patents
A kind of real-time data transmission system and its method of data synchronization Download PDFInfo
- Publication number
- CN104639585B CN104639585B CN201310566157.6A CN201310566157A CN104639585B CN 104639585 B CN104639585 B CN 104639585B CN 201310566157 A CN201310566157 A CN 201310566157A CN 104639585 B CN104639585 B CN 104639585B
- Authority
- CN
- China
- Prior art keywords
- data
- server
- module
- import
- transmission
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Abstract
The present invention is suitable for data processing field, provides a kind of real-time data transmission system and its method of data synchronization, which comprises data import server when receiving data synchronic command, send request of data to data adapter unit by transmission agency module；Data adapter unit reads data from data source, and the data of reading are sent to data by transmission agency module and import server；Data import server and receive data from transmission agency module, and received data are imported online service module.The present invention reads data from data source by data adapter unit due to unified, the data forwarding for being read data adapter unit by transmission agency module to data import server, therefore, when needing dilatation, only need to increase data adapter unit and/or transmission agency module, so that real-time data transmission system has preferable scalability, so as to realize the synchronization of mass data.
Description
Technical field
The invention belongs to data processing field more particularly to a kind of real-time data transmission systems and its method of data synchronization.
Background technique
Real-time data transmission system (also referred to as real time data transmission platform) has high-performance, real-time, succession, Gao Ke
The features such as by property, high availability, scalability, therefore it is widely used in internet application field, as log collection, data are supervised
Control, advertising feedback, quantum statistics, database synchronization etc..
Existing real-time data transmission system includes TimeTunnel etc..TimeTunnel generally uses 3 servers.?
TimeTunnel is applied to Taobao come when handling data, the data volume that 3 servers are handled daily is about 2.3T, and peak value is about
For the inflow flow of 50Mb per second, the outflow flow of 130Mb.
Existing TimeTunnel is since scalability is restricted, to be difficult to support the processing of mass data.And due to
The peak value outflow flow of every server of TimeTunnel itself is 350Mb, and when TimeTunnel is applied to Taobao,
Every server outflow flow per second is about 130Mb, and therefore, server bandwidth utilization rate is relatively low.In addition TimeTunnel is not
Support the reparation of format data, therefore the reliability of data transmission is lower.
Summary of the invention
The embodiment of the present invention provides a kind of real-time data transmission system and its method of data synchronization, it is intended to solve existing number
Cause to be difficult to the problem of handling mass data since scalability is restricted according to Transmission system.
On the one hand, a kind of real-time data transmission system, including the first Internet data center and the second interconnection netting index are provided
According to center, first Internet data center includes data source and at least one data adapter unit, the second interconnection netting index
It include at least one transmission agency module according to center, at least one data imports server, and imports server with data and connect
The online service module connect；
Wherein the data adapter unit reads data from the data source, and the data of reading are passed through the transmission generation
It manages module forwards to the data and imports server, the data import server and the data received are imported the online clothes
Business module.
On the other hand, the method for data synchronization in a kind of real-time data transmission system is provided, which comprises
Data import server when receiving data synchronic command, are sent by transmission agency module to data adapter unit
Request of data；
Data adapter unit reads data from data source, and the data of reading are sent to data by transmission agency module and are led
Enter server；
Data import server and receive data from transmission agency module, and received data are imported online service module.
In embodiments of the present invention, unified that data are read from data source by data adapter unit, pass through transmission agency module
The data forwarding that data adapter unit is read to data import server, therefore, when needing dilatation, it is only necessary to it is suitable to increase data
Orchestration and/or transmission agency module, so that real-time data transmission system has preferable scalability, so as to realize magnanimity number
According to synchronization.
Detailed description of the invention
It to describe the technical solutions in the embodiments of the present invention more clearly, below will be to embodiment or description of the prior art
Needed in attached drawing be briefly described, it should be apparent that, the accompanying drawings in the following description is only of the invention some
Embodiment for those of ordinary skill in the art without any creative labor, can also be according to these
Attached drawing obtains other attached drawings.
Fig. 1 is the structural block diagram of real-time data transmission system provided in an embodiment of the present invention；
Fig. 2 is the concrete structure diagram that data provided in an embodiment of the present invention import server；
Fig. 3 be another embodiment of the present invention provides real-time data transmission system structural block diagram；
Fig. 4 is the implementation flow chart of the method for data synchronization of real-time data transmission system provided in an embodiment of the present invention；
Fig. 5 be another embodiment of the present invention provides real-time data transmission system method of data synchronization implementation process
Figure.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to the accompanying drawings and embodiments, right
The present invention is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, and
It is not used in the restriction present invention.
In embodiments of the present invention, the data adapter unit of the first Internet data center reads data from data source, and
The data read are sent to the data of the second Internet data center by the transmission agency of the second Internet data center
Server is imported, server is imported by data and imports data to each online service module of the second Internet data center,
So as to which the data in data source to be synchronized to each online service module of the second Internet data center, and work as real time data
When Transmission system data volume to be processed increases, it is only necessary to increase the data adapter unit and/or the of the first Internet data center
The transmission agency of two Internet data centers so that real-time data transmission system can flexible expansion, and then can facilitate
Handle mass data.
In order to illustrate technical solutions according to the invention, the following is a description of specific embodiments.
Fig. 1 shows the structural block diagram of real-time data transmission system provided in an embodiment of the present invention.
The real-time data transmission system includes the first Internet data center (Internet Data Center, IDC) 1 He
Second Internet data center 2.Wherein the first Internet data center 1 is Internet data center as data source, this
All initial data being stored in one Internet data center 1 in the real-time data transmission system.In second internet data
The heart 2 is the Internet data center for needing to obtain data from the data source of the first Internet data center 1.
Such as when the real-time data transmission system is applied to search engine, then first Internet data center 1 is to deposit
The Internet data center of full dose webpage and index data is contained, the second Internet data center 2 is to be stored with index data,
And need to obtain the Internet data center of full dose webpage from the first Internet data center.Wherein full dose webpage refers to that search is drawn
Hold up all webpages searched for and downloaded to according to search key.
Wherein the first Internet data center 1 includes data source 11, and at least one data connecting with data source 11
Adapter (DataAdapter) 12.Wherein the connection between data source 11 and data adapter unit 12 is physical connection or logic
Connection.It can be wired or wireless connection when for physical connection.
All initial data being wherein stored in data source 11 in the real-time data transmission system.As worked as the real time data
When Transmission system is applied to search engine, full dose webpage and index data etc. are stored in the data source 11.
The quantity of data adapter unit 12 can increase or subtract according to real-time data transmission system data volume to be treated
It is few.Specifically, when real-time data transmission system data volume to be treated increases, it is only necessary to increase of data adapter unit
Number, and the information preservation of increased data adapter unit to data is imported into server 22.In this way, making the real-time Data Transmission system
System has preferable parallel expansion, supports smooth quick dilatation.
The quantity of the data adapter unit 12 can also increase or reduce according to the size for the total bandwidth that transmission uses.Specifically
, the size control data adapter unit 12 for the total bandwidth that can be used according to transmission uses number, and by data adapter unit
Information preservation to data can be used to import server 22.Pass through the usage quantity of control data adapter unit in this way, so that it may control
Transmit the total bandwidth used.
The quantity of the data adapter unit 12 can also be adjusted according to the period.Specifically, can be to different time sections
Interior data volume to be treated is for statistical analysis, and according to statistical result to the data adaptation that can be used in different time sections
The quantity of device is configured.
The data adapter unit 12 reads data from data source 11 according to data synchronic command or request of data.
In the present embodiment, when the data in its data source are synchronized to second mutually by 1 active of the first Internet data center
When each online service module in networking data center 2, then data adapter unit 12 is according to data synchronic command from data source 11
Read data；When will be in its data source under activly request of first Internet data center 1 in the second Internet data center 2
When data are synchronized to each online service module in the second Internet data center 2, then data adapter unit 12 is according to the second interconnection
The request of data that network data center 2 is sent reads data from data source 11.
In an alternative embodiment of the invention, which calculates and reads after reading data in data source 11
The error check code for the data got can carry out error checking and correction to the data read according to error check code so as to subsequent.
Wherein the error check code can be cyclic redundancy check code (Cyclic Redundancy Check, CRC) etc..
In an alternative embodiment of the invention, which carries out compression place to reading data from data source 11
Reason.Since data adapter unit 12 has carried out compression processing to the data read, need to reduce through network transmission
Data volume, and then saved bandwidth resources.
Second Internet data center 2 includes at least one transmission agency module (Tproxy) 21, with transmission agency module
21 connection at least one data import server (FastBuild) 22, and with data import server 22 connect it is online
Service module 23.
Wherein the quantity of transmission agency module 21 can increase according to real-time data transmission system data volume to be treated
Add or reduces.
Wherein transmission agency module 21 provides outer net agent functionality, by network and the first Internet data center 1
Data adapter unit 12 connect, allow the second Internet data center 2 by network and the first Internet data center 1 it
Between communicated.
Specifically, the transmission agency module 21 import data to server 22 transmission request of data by forwarded to
Data adapter unit 12, and the data that data adapter unit 12 is read from data source 11, and the number that will be received are received by network
Server 22 is imported according to data are transmitted to.
In an alternative embodiment of the invention, when data adapter unit 12 is to have calculated difference from the data that data source 11 is read
When wrong check code, then the transmission agency module 21 by network receive the data that are read from data source 11 of data adapter unit 12 with
And the error check code of the data, and the error check code of the data received and the data is transmitted to data and imports service
Device 22.
Data import the data adapter unit 12 that the reception transmission agency module 21 of server 22 forwards and read from data source 11
The data arrived, and the data that transmission agency module 22 is forwarded are imported into online service module 23.
In the present embodiment, when the data in its data source 11 are synchronized to second by 1 active of the first Internet data center
When each online service module 23 in Internet data center 2, which imports server 22 directly by transmission agency module 22
The data of forwarding import online service module 23；
When will be in its data source under activly request of first Internet data center 1 in the second Internet data center 2
When data are synchronized to each online service module in the second Internet data center 2, then data import server 22 and are receiving
When data synchronic command, request of data, request data adapter 12 are sent to data adapter unit 12 by transmission agency module 21
Data are read from data source, while receiving the number that data adapter unit 12 is read from data source 11 from transmission agency module 21
According to, and the data received are imported into online service module 23.
In the present embodiment, the data synchronic command that data import that server 22 receives can be Operation Support System
(Operation Support System, OSS) 3 is sent to data and imports server 22.Wherein Operation Support System is one
The service operation and management platform of a synthesis.Operation Support System 3 is sent to the data synchronic command that data import server 22
In include source data file path and target online service module id.Wherein source data file path is used to indicate data importing
The store path of file in the data source requested required for the data synchronic command of server 22, so as to be read from data source
Get the data of data synchronic command request.Target online service module id refers to the requested number of data synchronic command
According to the mark for needing to be synchronized to online service module, so that designation date imports server 22 after getting the data of request,
The data can be imported in the corresponding online service module of target online service module id, so as to by the number in data source
According to being synchronized in corresponding online service module.
The data import in server 22 and are stored with the list of available transmission proxy module and data available adapter list.Number
According to server 22 is imported when receiving data synchronic command, a transmission agency is selected from available transmission proxy module list
Module, and a data adapter unit is selected from data available adapter list, data import transmission of the server 22 to selection
Proxy module sends request of data, which includes the IP information of the data adapter unit of selection, to inform transmission agency mould
Request of data is transmitted to the data adapter unit of selection by block.Transmission agency module is believed according to the IP of data adapter unit in request of data
Request of data is sent to the data adapter unit of selection by breath, with the data adapter unit by selection from data source read requests
Data, and the data forwarding of reading is imported to data by server 22 by transmission agency module.
In an alternative embodiment of the invention, which it is suitable to import the data that the reception transmission agency module 21 of server 22 forwards
The error check code of data and the data that orchestration 12 is read from data source 11, using the error check code pair of the data
Received data carry out error checking and correction, and the successful data of error checking and correction are imported online service module 23.
In the present embodiment, data import server 22 in the data and the data for receiving the forwarding of transmission agency module 21
Error check code when, error checking and correction is carried out to received data using error check code, and by the successful data of error checking and correction
Online service module is imported, to ensure that the accuracy for the data being directed into online service module.
In an alternative embodiment of the invention, data import server 22 in the error check code using the data to received
After data carry out error checking and correction, if error checking and correction fails, verification failure information is recorded.Wherein the verification failure information includes text
Part grade failure information and/or record grade failure information.File-level failure information is recorded when referring to the failure of entire file synchronization
Information.This document grade failure information can be the file path etc. of this document.Record grade failure information refers to the part in file
The information that recording synchronism is recorded when failing.The record grade failure information includes the initial address and length information of record.
In the present embodiment, data import server in the error checking and correction failure to data, record verification failure information,
To provide foundation for subsequent data reparation.
In an alternative embodiment of the invention, data import server 22 after the completion of a synchronous task, according to record
Verification failure information repairs the data of synchronization failure.Detailed process is as follows for it:
If verification failure information be file-level failure information, data import server 22 by transmission agency module 21 to
Data adapter unit 12 sends file acquisition request, and this document acquisition request includes the file path of this document, and data import clothes
Device 22 be engaged in the file for the request that the data adapter unit 12 for receiving the forwarding of transmission agency module 21 is read from data source,
Import the file into online service module 23；
If verification failure information be record grade failure information, data import server 22 by transmission agency module 21 to
Data adapter unit 12 sends record acquisition request, which includes the initial address and length information of record, and counts
The request read from data source according to server 22 is imported in the data adapter unit 12 for receiving the forwarding of transmission agency module 21
Record when, by the record import online service module 23.
In the present embodiment, by being repaired according to the verification failure information of record to the data of synchronization failure, thus
It ensure that the synchronous high reliability of data, while by distinguishing file-level failure information and record grade failure information, thus greatly
The expense for reducing reparation.
In an alternative embodiment of the invention, which imports server 22 in the number for receiving the forwarding of transmission agency module 21
According to, and by the data of forwarding import online service module 23 when, use asynchronous buffer mode.
Specifically, referring to Fig. 2, importing the concrete structure diagram of server 22 for data provided in an embodiment of the present invention.It should
It includes data transmission module 221 that data, which import server 22, and caching 222 and data import submodule 223.Wherein cache
222 can store a plurality of record of file.
Wherein data transmission module 221 is not when caching also writes full, by transmission agency module 21 from data adapter unit
12 obtain data, and the data buffer storage that will acquire is into caching 222.
Specifically, data transmission module 221 passes through transmission agency using concurrent writing mode when caching does not also write full
Module 21 obtains data from data adapter unit 12, and in the data write-in caching 222 that will acquire as unit of recording.
Data import submodule 223 in discovery caching 222 there are when data, and the data in caching are imported online service
In module 23.
Specifically, data import submodule 223 in discovery caching 22 there are when data, concomitantly will as unit of record
Data in caching 222 are directed into online service module 23.
In the present embodiment, data import server and realize the reception of data by using the mode of asynchronous buffer and lead
Enter, to make data transmission efficiency can achieve the upper limit of single machine network interface card bandwidth, so that the band of single machine network interface card adequately be utilized
Width improves the efficiency of data transmission.
Fig. 3 show another embodiment of the present invention provides real-time data transmission system structural block diagram.The real time data
Transmission system and second Internet data center 2 that the difference is that only of real-time data transmission system shown in FIG. 1 further include
Verify server (CheckServer) 24.
The verification server 24 is when receiving data check instruction, respectively from data source 11 and online service module 23
Middle acquisition data, and the data that will acquire compare, to be verified to the data being directed into online service module 23.
If wherein identical as the data got from online service module 23 from the data that data source 11 is got, success is verified,
Otherwise, verification failure.In this manner it is ensured that the data in the data and data source that are synchronized in online service module keep one
It causes.
In the present embodiment, data check instruction is that Operation Support System is sent to verification server 24.
In an alternative embodiment of the invention, the verification server 24 extraction section data from data source 11, and from online
Corresponding data are extracted in service module 23, the data being drawn into are compared, with to being directed into online service module 23
Data verified.Since only extraction section data compare, make data school so as to avoid excessive due to data volume
The expense tested is excessive.
It is worth noting that, above system, included each unit is only divided according to the functional logic, but simultaneously
It is not limited to above-mentioned division, as long as corresponding functions can be realized；In addition, the specific name of each functional unit is also
For the ease of mutually distinguishing, the protection scope being not intended to restrict the invention.
Fig. 4 shows the realization stream of the method for data synchronization in real-time data transmission system provided in an embodiment of the present invention
Journey, wherein real-time data transmission system is as shown in Figures 1 to 3, and details are as follows for this method:
S101, data import server when receiving data synchronic command, by transmission agency module to data adaptation
Device sends request of data.
The data import in server and are stored with the list of available transmission proxy module and data available adapter list.The number
According to importing server timing or when receiving list and pulling instruction, obtain available transmission generation in real-time data transmission system
Module information and available data adapter unit information are managed, and according to the available transmission agency module information of acquisition and available number
According to Adapter Information update its storage the list of available transmission proxy module and data available adapter list, with timing update or
The list of available transmission proxy module and data available adapter list of its storage of person's real-time update.
Specifically, importing server to data by Operation Support System sends data synchronic command.The data, which synchronize, to be referred to
It include source data file path and target online service module id in order.
Wherein data import server when receiving data synchronic command, by transmission agency module to data adapter unit
The step of sending request of data specifically includes:
A1, data import server when receiving data synchronic command, select from available transmission proxy module list
One transmission agency module selects a data adapter unit from data available adapter list.
A2, data import server and send request of data to the transmission agency module of selection, include choosing in the request of data
The IP information for the data adapter unit selected is forwarded to the data adapter unit of selection to inform transmission agency module for request of data.
And the request of data specifies the Data Identification for needing to request.The Data Identification can be file identification, file path etc..
Request of data is forwarded to selection according to the IP information of the data adapter unit in request of data by A3, transmission agency module
Data adapter unit.
S102, data adapter unit reads data from data source, and the data of reading are sent to by transmission agency module
Data import server.
Specifically, after data adapter unit receives request of data from transmission agency module, according to the request of data from data
Data are read in source, and the data of reading are sent to data by transmission agency module and import server.
In an alternative embodiment of the invention, after data adapter unit receives request of data from transmission agency module, from data
Data are read in source, and calculate the error check code of the data of reading, and the error check code of the data of reading and the data is led to
It crosses defeated proxy module and is sent to data importing server.Wherein error check code can be cyclic redundancy check etc..
In an alternative embodiment of the invention, data adapter unit leads to after the data read from data source are carried out compression processing
It crosses transmission agency module and is sent to data importing server or data adapter unit for the data read from data source and the number
According to error check code carry out compression processing after by transmission agency module be sent to data importing server.
S103, data import server and receive data from transmission agency module, and received data are imported online service
Module.
In the present embodiment, general data synchronic command needs synchronous data volume larger, and data transmission is due to by band
Wide influence, the data volume that a request of data can request that need synchronous data volume to want small relative to data synchronic command,
Therefore, when sending request of data to the transmission agency module of selection according to data synchronic command, multiple data be may be generally divided into
Request after transferring the requested data of request of data, retransmits request of data next time to carry out.Wherein transmitting
The requested data of a complete request of data refer to will be after the data importing online service module in requested data source.
It can be a file as data synchronic command needs synchronous data, several numbers are divided into for this document
It is retransmited next after the data in a requested data source of request of data are directed into online service module according to request
Secondary request of data, after the data synchronic command is needed synchronous file synchronization to online service module, one subsynchronous
Business is completed.
In the present embodiment, unified that data are read from data source by data adapter unit, it will be counted by transmission agency module
The data forwarding read according to adapter to data import server, therefore, when needing dilatation, it is only necessary to increase data adapter unit
And/or transmission agency module, so that real-time data transmission system has preferable scalability, so as to realize mass data
It is synchronous.
In an alternative embodiment of the invention, when the number that data adapter unit will be read by transmission agency module from data source
When accordingly and the error check code of the data is sent to data importing server, data import server and connect from transmission agency module
Receive data, and the step of received data are imported into online service module specifically:
Data import server and receive data and the error check code of the data from transmission agency module, and use mistake
Check code carries out error checking and correction to received data, the successful data of error checking and correction is imported online service module, to guarantee
It is directed into the accuracy of the data in online service module.
In an alternative embodiment of the invention, data import server in the error check code using the data to received number
After carrying out error checking and correction, if error checking and correction fails, verification failure information is recorded.Wherein the verification failure information includes file
Grade failure information and/or record grade failure information.
In an alternative embodiment of the invention, this method further include:
Data import server after the completion of a synchronous task, according to the verification failure information of record to synchronization failure
Data are repaired.One of synchronous task completion refers to that data synchronic command specified data synchronously completes.Wherein data
Server is imported after the completion of a synchronous task, the data of synchronization failure are repaired according to the verification failure information of record
Detailed process is as follows:
If verification failure information is file-level failure information, data import server and pass through transmission agency module to data
Adapter sends file acquisition request, and this document acquisition request includes the file path of this document, and data import server and exist
When receiving the file for the request that the data adapter units of transmission agency module forwards is read from data source, import the file into
Online service module；
If verifying failure information is record grade failure information, server is imported in data and passes through transmission agency module to data
Adapter sends record acquisition request, which includes the initial address and length information of record, and data import
Server is in the record for the request that the data adapter unit for receiving transmission agency module forwards is read from data source, by this
Record imports online service module.
In an alternative embodiment of the invention, data import server and receive data from transmission agency module, and will be received
Data import the step of online service module specifically:
B1, data import server data transmission module caching also do not write full when, by transmission agency module from
Data adapter unit obtains data, and the data buffer storage that will acquire is into caching.
Specifically, data transmission module passes through transmission agency mould when caching does not also write full, using concurrent writing mode
Block obtains data from data adapter unit, and in the data write-in caching that will acquire as unit of recording.
The data that B2, data import server import submodule in discovery caching there are when data, by the number in caching
According in importing online service module.
Specifically, data import submodule in discovery caching there are when data, it concomitantly will caching as unit of record
In data be directed into online service module.
The step of wherein step B1 and B2 is asynchronous execution, no sequencing.
Fig. 5 show another embodiment of the present invention provides real-time data transmission system in method of data synchronization realization
Process, wherein real-time data transmission system is as shown in Figures 1 to 3, and wherein S201 to S203 is identical as the S101 to S103 in Fig. 4,
Details are not described herein, and this method is after S203 further include:
S204, verification server is when receiving data check instruction, respectively from data source and online service module
Data are obtained, and the data that will acquire compare, to verify to the data being directed into online service module.
Specifically, importing server to data by Operation Support System sends data check instruction.
If wherein identical as the corresponding data got from online service module from the data that data source is got, school
Success is tested, otherwise, verification failure.In this manner it is ensured that the data in the data and data source that are synchronized in online service module
It is consistent.
In an alternative embodiment of the invention, the verification server extraction section from data source and online service module respectively
Data, and the data being drawn into are compared, to be verified to the data being directed into online service module.
In an alternative embodiment of the invention, this method further includes following step:
When in real-time data transmission system increase perhaps reduce data adapter unit quantity when by increase or reduce
The information of data adapter unit stores to data and imports server, and data import server according to the data adaptation of increase or reduction
The information update data available adapter list of device.
In the present embodiment, when real-time data transmission system needs dilatation, such as since data volume to be treated growth needs
When wanting dilatation, or needing dilatation due to online machine quantity increase, it is suitable that data can be increased in real-time data transmission system
The quantity of orchestration only needs the information by increased data adapter unit to store to data at this time and imports server, and data import clothes
Device be engaged according to the information update data available adapter list of the data adapter unit of increase or reduction, dilatation can be realized.
In an alternative embodiment of the invention, this method further includes following step:
It will increase or reduce when increasing the quantity for perhaps reducing transmission agency module in real-time data transmission system
Transmission agency module information store to data import server, data import server according to increase or reduction data
The information update available transmission proxy module list of adapter.
In an alternative embodiment of the invention, this method further includes following step:
The total bandwidth available for obtaining real-time data transmission system controls the number of data available adapter according to total bandwidth available
Amount, and the information of data available adapter is stored to data and imports server, data import server according to increasing or subtract
The information update data available adapter list of few data adapter unit.
In an alternative embodiment of the invention, this method further includes following step:
Current time is obtained, the period according to belonging to current time controls the quantity of data available adapter, and can
It is stored with the information of data adapter unit to data and imports server, it is suitable according to the data of increase or reduction that data import server
The information update data available adapter list of orchestration.
In the present embodiment, the quantity for presetting period and the period data available adapter, is being obtained in this way
After getting current time, the quantity of data available adapter can be controlled preparatory according to the period belonging to current time
The quantity for the period data available adapter being arranged.
It will appreciated by the skilled person that implement the method for the above embodiments be can be with
Relevant hardware is instructed to complete by program, the program can be stored in a computer-readable storage medium
In, the storage medium, such as ROM/RAM, disk, CD.
In the present embodiment, unified that data are read from data source by data adapter unit, it will be counted by transmission agency module
The data forwarding read according to adapter to data import server, therefore, when needing dilatation, it is only necessary to increase data adapter unit
And/or transmission agency module, so that real-time data transmission system has preferable scalability, so as to realize mass data
It is synchronous.By carrying out error checking and correction to received data every time, and the successful data of error checking and correction are imported into online service module,
To guarantee the accuracy for the data being directed into online service module.By being lost according to the verification failure information of record to synchronous
The data lost are repaired, to ensure that the synchronous high reliability of data, while by distinguishing file-level failure information and note
Grade failure information is recorded, to greatly reduce the expense of reparation.Data import server by using the mode of asynchronous buffer
The reception and importing of data are realized, to make data transmission efficiency can achieve the upper limit of single machine network interface card bandwidth, thus sufficiently
The bandwidth that single machine network interface card is utilized, improve data transmission efficiency.By respectively from data source and online service module
Middle acquisition data, and the data that will acquire compare, to be verified to the data being directed into online service module, from
The data held in the data and data source that can guarantee to be synchronized in online service module are consistent.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all in essence of the invention
Made any modifications, equivalent replacements, and improvements etc., should all be included in the protection scope of the present invention within mind and principle.
Claims (17)
1. a kind of real-time data transmission system, including the first Internet data center and the second Internet data center, feature
It is, first Internet data center includes data source and at least one data adapter unit, second internet data
Center includes at least one transmission agency module, at least one data imports server, and imports server with data and connect
Online service module；
Wherein the data adapter unit reads data from the data source, and the data of reading are passed through the transmission agency mould
Block is forwarded to the data and imports server, and the data import server and the data received are imported the online service mould
Block.
2. real-time data transmission system as described in claim 1, which is characterized in that the data adapter unit is also used to from institute
It states after data source reads data, calculates the error check code of the data read, and by the data read and the number
According to error check code the data be sent to by the transmission agency module import server.
3. real-time data transmission system as claimed in claim 2, which is characterized in that the data import server and are also used to
When receiving the error check code of data and the data, using the data error check code to the data received into
Row verifying, and the successful data of error checking and correction are imported into the online service module.
4. real-time data transmission system as claimed in claim 3, which is characterized in that the data import server and are also used to adopt
After being verified with the error check code of the data to the data received, if authentication failed, record verification failure information,
Described in verification failure information include file-level failure information and/or record grade failure information.
5. real-time data transmission system as claimed in claim 4, which is characterized in that the data import server and are also used to
After the completion of one synchronous task, the data of synchronization failure are repaired according to the verification failure information.
6. real-time data transmission system according to any one of claims 1 to 5, which is characterized in that the data are led
Entering server includes:
Caching, a plurality of record for storage file；
Data transmission module, for being fitted from the data by the transmission agency module when the caching does not also write full
Orchestration obtains data, and the data buffer storage that will acquire is to the caching；
Data import submodule, for, there are when data, the data in the caching being imported described online in the caching
In service module.
7. real-time data transmission system as claimed in claim 6, which is characterized in that the data transmission module is using concurrent
For the data buffer storage that writing mode will acquire to the caching, the data import submodule will be described slow using concurrent reading manner
The data deposited import in the online service module.
8. real-time data transmission system according to any one of claims 1 to 5, which is characterized in that the system is also
Include:
Verify server, for receive data check instruction when, respectively from the data source and the online service mould
Data are obtained in block, and the data that will acquire compare, to carry out to the data being directed into the online service module
Verification.
9. the method for data synchronization in a kind of real-time data transmission system, which is characterized in that the described method includes:
At least one data imports server when receiving data synchronic command, by least one transmission agency module to extremely
A few data adapter unit sends request of data；
At least one data adapter unit reads data from data source, and the data of reading are passed through at least one transmission agency module
It is sent at least one data and imports server；
At least one data imports server and receives data from least one transmission agency module, and received data are imported
Line service module.
10. method of data synchronization as claimed in claim 9, which is characterized in that the data import server and receiving number
When according to synchronic command, sending request of data to data adapter unit by transmission agency module includes:
Data import server when receiving data synchronic command, select from pre-stored available transmission proxy module list
A transmission agency module is selected, a data adapter unit is selected from pre-stored data available adapter list；
Data import server and send request of data to the transmission agency module of selection, include the number of selection in the request of data
According to the IP information of adapter；
Request of data is forwarded to the data of selection according to the IP information of the data adapter unit in request of data by transmission agency module
Adapter.
11. method of data synchronization as claimed in claim 9, which is characterized in that read in the data adapter unit from data source
After data, the method also includes:
Data adapter unit calculates the error check code of the data read, and the error check code of the data is sent to data and is led
Enter server.
12. method of data synchronization as claimed in claim 9, which is characterized in that the data import server from transmission agency
Module receives data, and received data importing online service module is specifically included:
Data import the error check code that server receives data and the data from transmission agency module, and use mistake school
It tests code and error checking and correction is carried out to received data, the successful data of error checking and correction are imported into online service module.
13. method of data synchronization as claimed in claim 12, which is characterized in that import server in the data and use mistake
After check code carries out error checking and correction to received data, the method also includes:
If error checking and correction fails, record verification failure information, the verification failure information includes file-level failure information and/or note
Record grade failure information.
14. method of data synchronization as claimed in claim 13, which is characterized in that the method also includes:
Data import server after the completion of a synchronous task, according to the verification failure information to the data of synchronization failure into
Row is repaired.
15. method of data synchronization as claimed in claim 14, which is characterized in that it is described according to the verification failure information to same
The data of step failure repair
If the verification failure information is file-level failure information, data import server and pass through transmission agency module to data
Adapter sends file acquisition request, and the file acquisition request includes the file path of the file, and data import service
Device is in the file for the request that the data adapter unit for receiving transmission agency module forwards is read from data source, by the text
Part imports online service module；
If verifying failure information is record grade failure information, server is imported in data and passes through transmission agency module to data adaptation
Device sends record acquisition request, and the record acquisition request includes the initial address and length information of record, and data import clothes
Device be engaged in the record for the request that the data adapter unit for receiving transmission agency module forwards is read from data source, it will be described
Record imports online service module.
16. the method for data synchronization as described in claim 9 to 15 any claim, which is characterized in that the data import
Server receives data from transmission agency module, and received data are imported online service module, comprising:
Data import the data transmission module of server when caching does not also write full, by transmission agency module from data adaptation
Device obtains data, and the data buffer storage that will acquire is into caching；
The data that data import server import submodule in discovery caching there are when data, and the data in caching are imported
In line service module.
17. the method for data synchronization as described in claim 9 to 15 any claim, which is characterized in that the method is also wrapped
It includes:
Verification server obtains data from data source and online service module respectively when receiving data check instruction,
And the data that will acquire compare, to be verified to the data being directed into online service module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310566157.6A CN104639585B (en) | 2013-11-13 | 2013-11-13 | A kind of real-time data transmission system and its method of data synchronization |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310566157.6A CN104639585B (en) | 2013-11-13 | 2013-11-13 | A kind of real-time data transmission system and its method of data synchronization |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104639585A CN104639585A (en) | 2015-05-20 |
| CN104639585B true CN104639585B (en) | 2019-01-08 |
Family
ID=53217895
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310566157.6A Active CN104639585B (en) | 2013-11-13 | 2013-11-13 | A kind of real-time data transmission system and its method of data synchronization |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104639585B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106454767A (en) * | 2015-08-05 | 2017-02-22 | 中兴通讯股份有限公司 | Business data synchronization method, device and system |
| CN105550325B (en) * | 2015-12-16 | 2019-05-31 | 北京奇虎科技有限公司 | A kind of data managing method and device |
| CN105608132B (en) * | 2015-12-16 | 2019-04-30 | 北京奇虎科技有限公司 | It is a kind of that the method and apparatus of the customized service for generating chart are provided |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101197694A (en) * | 2006-12-04 | 2008-06-11 | 中兴通讯股份有限公司 | Central statistics and processing system and method for communication system log |
| CN101448020A (en) * | 2008-12-29 | 2009-06-03 | 深圳市迅雷网络技术有限公司 | Data source return method and device |
| CN100531055C (en) * | 2006-04-10 | 2009-08-19 | 华为技术有限公司 | Data synchronous system and its method |
| CN102629929A (en) * | 2012-04-18 | 2012-08-08 | 华为技术有限公司 | Method and system and device for obtaining data |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI439873B (en) * | 2011-08-08 | 2014-06-01 | Dimerco Express Taiwan Corp | Data synchronization method |
-
2013
- 2013-11-13 CN CN201310566157.6A patent/CN104639585B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100531055C (en) * | 2006-04-10 | 2009-08-19 | 华为技术有限公司 | Data synchronous system and its method |
| CN101197694A (en) * | 2006-12-04 | 2008-06-11 | 中兴通讯股份有限公司 | Central statistics and processing system and method for communication system log |
| CN101448020A (en) * | 2008-12-29 | 2009-06-03 | 深圳市迅雷网络技术有限公司 | Data source return method and device |
| CN102629929A (en) * | 2012-04-18 | 2012-08-08 | 华为技术有限公司 | Method and system and device for obtaining data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104639585A (en) | 2015-05-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104410692B (en) | A kind of method and system uploaded for duplicate file | |
| CN106856434A (en) | The method and apparatus of access request conversion | |
| CN103475682B (en) | File transfer method and file transfer equipment | |
| CN108133008A (en) | The processing method of business datum, device, equipment and storage medium in database | |
| CN105447046A (en) | Distributed system data consistency processing method, device and system | |
| CN110457199A (en) | The method and apparatus of performance test | |
| CN106709069A (en) | High-reliability big data logging collection and transmission method | |
| CN108959337A (en) | Big data acquisition methods, device, equipment and storage medium | |
| CN106874334A (en) | A kind of data processing method and device, information handling systems utilizing | |
| CN110011952A (en) | A kind of method, service cluster and the client of data transmission | |
| CN109639768A (en) | A kind of distribution blocked file transmission method and system | |
| CN104639585B (en) | A kind of real-time data transmission system and its method of data synchronization | |
| CN110493302A (en) | A kind of document transmission method, equipment and computer readable storage medium | |
| CN109583896A (en) | Transaction verification method, block chain node and storage medium | |
| CN103634269B (en) | A single sign-on system and a method | |
| CN107395559A (en) | Data processing method and equipment based on redis | |
| CN110502572A (en) | A kind of method and server that data are synchronous | |
| CN104899312B (en) | A kind of data-updating method of multistratum classification data system | |
| CN113326165A (en) | Data processing method and device based on block chain and computer readable storage medium | |
| CN109254921A (en) | Application version verification method, device, computer equipment and storage medium | |
| CN106203179B (en) | A kind of completeness check system and method for pair of file | |
| CN105184559B (en) | A kind of payment system and method | |
| CN103944779B (en) | A kind of WAP service features monitoring method and system | |
| CN106454767A (en) | Business data synchronization method, device and system | |
| CN108494875A (en) | A kind of method and apparatus of feedback resources file |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210922 Address after: 518000 Tencent Building, No. 1 High-tech Zone, Nanshan District, Shenzhen City, Guangdong Province, 35 Floors Patentee after: TENCENT TECHNOLOGY (SHENZHEN) Co.,Ltd. Patentee after: TENCENT CLOUD COMPUTING (BEIJING) Co.,Ltd. Address before: 2, 518044, East 403 room, SEG science and Technology Park, Zhenxing Road, Shenzhen, Guangdong, Futian District Patentee before: TENCENT TECHNOLOGY (SHENZHEN) Co.,Ltd. |
|
| TR01 | Transfer of patent right |