CN104639585A - Real-time data transmission system and data synchronization method thereof - Google Patents
Real-time data transmission system and data synchronization method thereof Download PDFInfo
- Publication number
- CN104639585A CN104639585A CN201310566157.6A CN201310566157A CN104639585A CN 104639585 A CN104639585 A CN 104639585A CN 201310566157 A CN201310566157 A CN 201310566157A CN 104639585 A CN104639585 A CN 104639585A
- Authority
- CN
- China
- Prior art keywords
- data
- importing
- adapter unit
- server
- proxy module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Abstract
The invention is applicable to the field of data processing and provides a real-time data transmission system and a data synchronization method thereof. The method includes: when a data import server receives a data synchronization command, sending a data request to a data adapter through a transmission agent module; through the data adapter, reading data from a data source and sending the read data to the data import server through the transmission agent module; through the data import server, receiving data from the transmission agent module, and importing the received data to an on-line service module. Since the data are read from the data source through the data adapter uniformly and the data read by the data adapter are forwarded to the data import server through the transmission agent module, when capacity expansion is required, only the data adapter and/or the transmission agent module is required to be added, the real-time data transmission system has good expandability, and further synchronization of massive data can be realized.
Description
Technical field
The invention belongs to data processing field, particularly relate to a kind of real-time data transmission system and method for data synchronization thereof.
Background technology
Real-time data transmission system (also referred to as real time data transmission platform) has the features such as high-performance, real-time, succession, high reliability, high availability, extensibility, therefore internet, applications field is widely used in, as log collection, data monitoring, advertising feedback, quantum statistics, database synchronization etc.
Existing real-time data transmission system comprises TimeTunnel etc.TimeTunnel generally adopts 3 station servers.When TimeTunnel being applied to Taobao and carrying out deal with data, the data volume of 3 station server process every day is about 2.3T, and peak value is about the inflow flow of 50Mb per second, the outflow flow of 130Mb.
Existing TimeTunnel is restricted due to autgmentability, thus is difficult to the process supporting mass data.And be 350Mb because the peak value of the every station server of TimeTunnel itself flows out flow, and when TimeTunnel is applied to Taobao, every station server outflow flow per second is about 130Mb, and therefore, server bandwidth utilance is on the low side.TimeTunnel does not support the reparation of formatted data in addition, and therefore the reliability of transfer of data is lower.
Summary of the invention
The embodiment of the present invention provides a kind of real-time data transmission system and method for data synchronization thereof, is intended to solve the problem that existing data transmission system causes because extensibility is restricted being difficult to process mass data.
On the one hand, a kind of real-time data transmission system is provided, comprise the first Internet data center and the second Internet data center, described first Internet data center comprises data source and at least one data adapter unit, described second Internet data center comprises at least one transmission proxy module, at least one data importing server, and the online service module be connected with data importing server;
Wherein said data adapter unit reads data from described data source, and the data of reading are forwarded to described data importing server by described transmission proxy module, and described data importing server is by online service module described in the data importing that receives.
On the other hand, provide the method for data synchronization in a kind of real-time data transmission system, described method comprises:
Data importing server, when receiving data syn-chronization instruction, sends request of data by transmission proxy module to data adapter unit;
Data adapter unit reads data from data source, and the data of reading are sent to data importing server by transmission proxy module;
Data importing server receives data from transmission proxy module, and the data importing online service module that will receive.
In embodiments of the present invention, unification reads data by data adapter unit from data source, the data retransmission read by data adapter unit by transmission proxy module is to data importing server, therefore, when needs dilatation, only need to increase data adapter unit and/or transmission proxy module, make real-time data transmission system have good autgmentability, thus the synchronous of mass data can be realized.
Accompanying drawing explanation
In order to be illustrated more clearly in the technical scheme in the embodiment of the present invention, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
Fig. 1 is the structured flowchart of the real-time data transmission system that the embodiment of the present invention provides;
Fig. 2 is the concrete structure figure of the data importing server that the embodiment of the present invention provides;
Fig. 3 is the structured flowchart of the real-time data transmission system that another embodiment of the present invention provides;
Fig. 4 is the realization flow figure of the method for data synchronization of the real-time data transmission system that the embodiment of the present invention provides;
Fig. 5 is the realization flow figure of the method for data synchronization of the real-time data transmission system that another embodiment of the present invention provides.
Embodiment
In order to make object of the present invention, technical scheme and advantage clearly understand, below in conjunction with drawings and Examples, the present invention is further elaborated.Should be appreciated that specific embodiment described herein only in order to explain the present invention, be not intended to limit the present invention.
In embodiments of the present invention, the data adapter unit of the first Internet data center reads data from data source, and the data read are sent to the data importing server of the second Internet data center by the transmission agency of the second Internet data center, by each online service module of data importing server by data importing to the second Internet data center, thus can by each online service module of the data syn-chronization in data source to the second Internet data center, and when real-time data transmission system data volume to be processed increases, the data adapter unit of increase by first Internet data center and/or the transmission of the second Internet data center is only needed to act on behalf of, thus make real-time data transmission system can flexible expansion, and then conveniently can process mass data.
In order to technical solutions according to the invention are described, be described below by specific embodiment.
Fig. 1 shows the structured flowchart of the real-time data transmission system that the embodiment of the present invention provides.
This real-time data transmission system comprises the first Internet data center (Internet Data Center, IDC) 1 and the second Internet data center 2.Wherein the first Internet data center 1 is the Internet data center as data source, stores all initial data in this real-time data transmission system in this first Internet data center 1.Second Internet data center 2 is the Internet data centers needing to obtain from the data source of the first Internet data center 1 data.
As when this real-time data transmission system is applied to search engine, then this first Internet data center 1 is for storing the Internet data center of full dose webpage and index data, second Internet data center 2 for storing index data, and needs the Internet data center obtaining full dose webpage from the first Internet data center.Wherein full dose webpage refers to that search engine is according to searching for keyword search and all webpages downloaded to.
Wherein the first Internet data center 1 comprises data source 11, and at least one data adapter unit (DataAdapter) 12 be connected with data source 11.Connection wherein between data source 11 and data adapter unit 12 is that physical connection or logic connect.When for physical connection, can be wired or wireless connections.
Wherein store all initial data in this real-time data transmission system in data source 11.As when this real-time data transmission system is applied to search engine, in this data source 11, store full dose webpage and index data etc.
The quantity of data adapter unit 12 can need data volume to be processed to increase according to real-time data transmission system or reduce.Concrete, when real-time data transmission system needs data volume to be processed to increase, only need the number increasing data adapter unit, and the information of the data adapter unit of increase is saved to data importing server 22.Like this, make this real-time data transmission system have good parallel expansion, support level and smooth dilatation fast.
The quantity of this data adapter unit 12 can also increase according to the size of the total bandwidth of transmission use or reduce.Concrete, the use number of the size control data adapter 12 of the total bandwidth that can use according to transmission, and the used information of data adapter unit is saved to data importing server 22.Like this by the usage quantity of control data adapter, just can the total bandwidth of controls transfer use.
The quantity of this data adapter unit 12 can also adjust according to the time period.Concrete, statistical analysis can be carried out to different time sections domestic demand data volume to be processed, and arrange according to the quantity of statistics to data adapter unit operable in different time sections.
This data adapter unit 12 reads data according to data syn-chronization instruction or request of data from data source 11.
In the present embodiment, when the first Internet data center 1 initiatively by the data syn-chronization in its data source to each online service module in the second Internet data center 2 time, then data adapter unit 12 reads data according to data syn-chronization instruction from data source 11; When the first Internet data center 1 under the active request of the second Internet data center 2 by the data syn-chronization in its data source to each online service module in the second Internet data center 2 time, then the request of data that data adapter unit 12 sends according to the second Internet data center 2 reads data from data source 11.
In an alternative embodiment of the invention, this data adapter unit 12, after read data from data source 11, calculates the error check code of the data read, can carry out error checking and correction so that follow-up according to error check code to the data read.Wherein this error check code can be cyclic redundancy check (CRC) code (Cyclic Redundancy Check, CRC) etc.
In an alternative embodiment of the invention, this data adapter unit 12 carries out compression process to reading data from data source 11.Because data adapter unit 12 has carried out compressing process to the data read, thus decrease the data volume needed by Internet Transmission, and then save bandwidth resources.
Second Internet data center 2 comprises at least one transmission proxy module (Tproxy) 21, at least one the data importing server (FastBuild) 22 be connected with transmission proxy module 21, and the online service module 23 be connected with data importing server 22.
The quantity wherein transmitting proxy module 21 can need data volume to be processed to increase according to this real-time data transmission system or reduce.
Wherein transmit proxy module 21 and provide outer net agent functionality, it is connected with the data adapter unit 12 in the first Internet data center 1 by network, and the second Internet data center 2 can be communicated with between the first Internet data center 1 by network.
Concrete, the request of data that data importing server 22 sends by this transmission proxy module 21 by forwarded to data adapter unit 12, and by the data that network reception data adapter unit 12 reads from data source 11, and by the data retransmission that receives to data importing server 22.
In an alternative embodiment of the invention, when data adapter unit 12 be the data read from data source 11 calculated error check code time, the then error check code of the data that read from data source 11 by network reception data adapter unit 12 of this transmission proxy module 21 and these data, and the error check code of the data received and these data is transmitted to data importing server 22.
The data that the data adapter unit 12 that data importing server 22 receives the forwarding of transmission proxy module 21 reads from data source 11, and just transmit the data importing online service module 23 of proxy module 22 forwarding.
In the present embodiment, when the first Internet data center 1 initiatively by the data syn-chronization in its data source 11 to each online service module 23 in the second Internet data center 2 time, this data importing server 22 is directly by data importing online service module 23 that transmission proxy module 22 forwards;
When the first Internet data center 1 under the active request of the second Internet data center 2 by the data syn-chronization in its data source to each online service module in the second Internet data center 2 time, then data importing server 22 is when receiving data syn-chronization instruction, request of data is sent to data adapter unit 12 by transmission proxy module 21, request msg adapter 12 reads data from data source, simultaneously receive from transmission proxy module 21 data that data adapter unit 12 reads from data source 11, and the data importing online service module 23 that will receive.
In the present embodiment, the data syn-chronization instruction that data importing server 22 receives can be that OSS (Operation Support System, OSS) 3 sends to data importing server 22.Wherein OSS is a comprehensive service operation and management platform.OSS 3 sends to the data syn-chronization instruction of data importing server 22 to comprise source data file path and target online service module id.Wherein source data file path is used to indicate the store path of the data source file of the required request of this data syn-chronization instruction of data importing server 22, can read the data of this data syn-chronization instruction request from data source.Target online service module id refers to that data that this data syn-chronization instruction is asked need to be synchronized to the mark of online service module, so that designation data imports server 22 after getting the data of request, can by online service module corresponding for this target online service module id of this data importing, so that by the data syn-chronization in data source in the online service module of correspondence.
The list of available transmission proxy module and the list of data available adapter is stored in this data importing server 22.Data importing server 22 is when receiving data syn-chronization instruction, a transmission proxy module is selected from the list of available transmission proxy module, and a data adapter unit is selected from the list of data available adapter, data importing server 22 sends request of data to the transmission proxy module selected, this request of data comprises the IP information of the data adapter unit of selection, to inform that request of data to be transmitted to the data adapter unit of selection by transmission proxy module.Request of data is sent to the data adapter unit of selection by transmission proxy module according to the IP information of data adapter unit in request of data, with the data of the data adapter unit read requests from data source by selecting, and by transmitting proxy module by the data retransmission of reading to data importing server 22.
In an alternative embodiment of the invention, the error check code of the data that the data adapter unit 12 that this data importing server 22 receives the forwarding of transmission proxy module 21 reads from data source 11 and these data, the error check code of these data is adopted to carry out error checking and correction, by successful for error checking and correction data importing online service module 23 to the data received.
In the present embodiment, data importing server 22 is when receiving the error check code of data and these data transmitting proxy module 21 forwarding, error check code is adopted to carry out error checking and correction to the data received, and by successful for error checking and correction data importing online service module, thus ensure that the accuracy of the data be directed in online service module.
In an alternative embodiment of the invention, data importing server 22 is after adopting the error check code of these data to carry out error checking and correction to the data received, if error checking and correction failure, then and record verification failure information.Wherein this verification failure information comprises file-level failure information and/or record level failure information.The information that file-level failure information records when referring to the failure of whole file synchronization.This file-level failure information can be the file path etc. of this article part.The information that record level failure information records when referring to the part recording synchronism failure in file.This record level failure information comprises initial address and the length information of record.
In the present embodiment, data importing server is when the error checking and correction failure to data, and record verifies failure information, thus provides foundation for follow-up data restore.
In an alternative embodiment of the invention, data importing server 22 is after a synchronous task completes, and the data of verification failure information to synchronization failure according to record are repaired.Its detailed process is as follows:
If verification failure information is file-level failure information, then data importing server 22 sends file acquisition request by transmission proxy module 21 to data adapter unit 12, this file acquisition request comprises the file path of this file, and data importing server 22 is when the file of the request that the data adapter unit 12 receiving the forwarding of transmission proxy module 21 reads from data source, this file is imported online service module 23;
If verification failure information is record level failure information, send record by transmission proxy module 21 to data adapter unit 12 at data importing server 22 and obtain request, this record obtains initial address and the length information that request comprises record, and data importing server 22 is when the record of the request that the data adapter unit 12 receiving the forwarding of transmission proxy module 21 reads from data source, this record is imported online service module 23.
In the present embodiment, by repairing according to the data of verification failure information to synchronization failure of record, thus ensure that the high reliability of data syn-chronization, simultaneously by distinguishing file-level failure information and record level failure information, thus greatly reduce the expense of reparation.
In an alternative embodiment of the invention, this data importing server 22 is receiving the data of transmission proxy module 21 forwarding, and during by the data importing online service module 23 of forwarding, have employed asynchronous buffer mode.
Concrete, referring to Fig. 2, is the concrete structure figure of the data importing server 22 that the embodiment of the present invention provides.This data importing server 22 comprises transfer of data submodule 221, buffer memory 222, and data importing submodule 223.Wherein buffer memory 222 can many records of storage file.
Wherein transfer of data submodule 221 is not when buffer memory is also write full, obtains data by transmission proxy module 21 from data adapter unit 12, and by the data buffer storage of acquisition in buffer memory 222.
Concrete, transfer of data submodule 221, when buffer memory is not also write full, adopts concurrent writing mode to obtain data by transmission proxy module 21 from data adapter unit 12, and the data got is write in buffer memory 222 to be recorded as unit.
When data importing submodule 223 exists data in discovery buffer memory 222, by the data importing online service module 23 in buffer memory.
Concrete, data importing submodule 223 when finding to there are data in buffer memory 22, be recorded as unit concomitantly by the data importing in buffer memory 222 to online service module 23.
In the present embodiment, reception and the importing of data importing server by adopting the mode of asynchronous buffer to realize data, thus make data transmission efficiency can reach the upper limit of unit network interface card bandwidth, thus make use of the bandwidth of unit network interface card fully, improve the efficiency of transfer of data.
Fig. 3 shows the structured flowchart of the real-time data transmission system that another embodiment of the present invention provides.The difference of the real-time data transmission system shown in this real-time data transmission system and Fig. 1 is only that the second Internet data center 2 also comprises validate service device (CheckServer) 24.
This validate service device 24, when receiving data check instruction, obtains data respectively, and the data got is contrasted from data source 11 and online service module 23, to verify the data be directed in online service module 23.If the data wherein got from data source 11 are identical with the data got from online service module 23, then verification succeeds, otherwise, verify unsuccessfully.Like this, can ensure to be synchronized to the data in online service module and the data in data source are consistent.
In the present embodiment, data check instruction is that OSS is sent to validate service device 24.
In an alternative embodiment of the invention, this validate service device 24 extracting part divided data from data source 11, and from online service module 23, extract corresponding data, the data be drawn into are contrasted, to verify the data be directed in online service module 23.Because only extracting part divided data contrasts, thus avoid because data volume is excessive and make the expense of data check excessive.
It should be noted that said system, included unit is carry out dividing according to function logic, but is not limited to above-mentioned division, as long as can realize corresponding function; In addition, the concrete title of each functional unit, also just for the ease of mutual differentiation, is not limited to protection scope of the present invention.
Fig. 4 shows the realization flow of the method for data synchronization in the real-time data transmission system that the embodiment of the present invention provides, and wherein real-time data transmission system is as shown in Figures 1 to 3, and details are as follows for the method:
S101, data importing server, when receiving data syn-chronization instruction, sends request of data by transmission proxy module to data adapter unit.
The list of available transmission proxy module and the list of data available adapter is stored in this data importing server.This data importing server timing or when receiving list and pulling instruction, obtain available transmission proxy module information and available data adapter unit information in real-time data transmission system, and according to the available transmission proxy module information obtained and available data adapter unit information updating its available transmission proxy module list stored and the list of data available adapter, upgrade with timing or real-time update its available transmission proxy module list stored and the list of data available adapter.
Concrete, send data syn-chronization instruction by OSS to data importing server.This data syn-chronization instruction comprises source data file path and target online service module id.
Wherein data importing server is when receiving data syn-chronization instruction, and the step sending request of data to data adapter unit by transmission proxy module specifically comprises:
A1, data importing server, when receiving data syn-chronization instruction, are selected a transmission proxy module, from the list of data available adapter, are selected a data adapter unit from the list of available transmission proxy module.
A2, data importing server send request of data to the transmission proxy module selected, and comprise the IP information of the data adapter unit of selection in this request of data, to inform that request of data is forwarded to the data adapter unit of selection by transmission proxy module.And this request of data specifies the Data Identification needing request.This Data Identification can be file identification, file path etc.
Request of data is forwarded to the data adapter unit of selection by A3, transmission proxy module according to the IP information of the data adapter unit in request of data.
S102, data adapter unit reads data from data source, and the data of reading are sent to data importing server by transmission proxy module.
Concrete, data adapter unit reads data according to this request of data from data source after receiving request of data from transmission proxy module, and the data of reading are sent to data importing server by transmission proxy module.
In an alternative embodiment of the invention, after data adapter unit receives request of data from transmission proxy module, read data from data source, and calculate the error check code of the data read, the data of reading and the error check code of these data are sent to data importing server by defeated proxy module.Wherein error check code can be CRC check code etc.
In an alternative embodiment of the invention, be sent to data importing server by transmission proxy module after the data read from data source are carried out compression process by data adapter unit, or the error check code of the data read from data source and these data is carried out being sent to data importing server by transmission proxy module after compression processes by data adapter unit.
S103, data importing server receives data from transmission proxy module, and the data importing online service module that will receive.
In the present embodiment, general data synchronic command needs synchronous data volume larger, and transfer of data is due to the impact by bandwidth, the data volume that one secondary data request can be asked needs synchronous data volume little relative to data syn-chronization instruction, therefore, when sending request of data according to data syn-chronization instruction to the transmission proxy module selected, generally can be divided into repeatedly request of data to carry out, after transferring the data that a secondary data request asks, then send request of data next time.Wherein refer to after by the data importing online service module in asked data source transferring the data that a secondary data request asks.
It can be a file as data synchronic command needs synchronous data, several times request of data is divided into for this file, after data importing in the data source that a secondary data request is asked to online service module, send request of data next time again, until after this data syn-chronization instruction is needed synchronous file synchronization to online service module, one time namely synchronous task completes.
In the present embodiment, unification reads data by data adapter unit from data source, the data retransmission read by data adapter unit by transmission proxy module is to data importing server, therefore, when needs dilatation, only need to increase data adapter unit and/or transmission proxy module, make real-time data transmission system have good autgmentability, thus the synchronous of mass data can be realized.
In an alternative embodiment of the invention, when the error check code of the data read from data source and these data is sent to data importing server by transmission proxy module by data adapter unit, data importing server receives data from transmission proxy module, and the step of the data importing online service module of reception is specially:
Data importing server receives the error check code of data and these data from transmission proxy module, and adopt error check code to carry out error checking and correction to the data received, by successful for error checking and correction data importing online service module, thus ensure the accuracy of the data be directed in online service module.
In an alternative embodiment of the invention, data importing server is after adopting the error check code of these data to carry out error checking and correction to the data received, if error checking and correction failure, then and record verification failure information.Wherein this verification failure information comprises file-level failure information and/or record level failure information.
In an alternative embodiment of the invention, the method also comprises:
Data importing server is after a synchronous task completes, and the data of verification failure information to synchronization failure according to record are repaired.One of them synchronous task completes and refers to that the data syn-chronization that data syn-chronization instruction is specified completes.Wherein data importing server is after a synchronous task completes, as follows according to the detailed process that the data of verification failure information to synchronization failure of record are repaired:
If verification failure information is file-level failure information, then data importing server sends file acquisition request by transmission proxy module to data adapter unit, this file acquisition request comprises the file path of this file, and data importing server is when the file of the request that the data adapter unit receiving the forwarding of transmission proxy module reads from data source, this file is imported online service module;
If verification failure information is record level failure information, send record by transmission proxy module to data adapter unit at data importing server and obtain request, this record obtains initial address and the length information that request comprises record, and data importing server is when the record of the request that the data adapter unit receiving the forwarding of transmission proxy module reads from data source, this record is imported online service module.
In an alternative embodiment of the invention, data importing server receives data from transmission proxy module, and the step of the data importing online service module of reception is specially:
The transfer of data submodule of B1, data importing server, when buffer memory is not also write full, obtains data by transmission proxy module from data adapter unit, and by the data buffer storage of acquisition in buffer memory.
Concrete, transfer of data submodule, when buffer memory is not also write full, adopts concurrent writing mode to obtain data by transmission proxy module from data adapter unit, and the data got is write in buffer memory to be recorded as unit.
When the data importing submodule of B2, data importing server exists data in discovery buffer memory, by the data importing online service module in buffer memory.
Concrete, data importing submodule when finding to there are data in buffer memory, be recorded as unit concomitantly by the data importing in buffer memory to online service module.
Wherein step B1 and B2 is the step of asynchronous execution, without sequencing.
Fig. 5 shows the realization flow of the method for data synchronization in the real-time data transmission system that another embodiment of the present invention provides, wherein real-time data transmission system as shown in Figures 1 to 3, wherein S201 to S203 is identical with the S101 to S103 in Fig. 4, do not repeat them here, the method also comprises after S203:
S204, validate service device, when receiving data check instruction, obtains data respectively, and the data got is contrasted from data source and online service module, to verify the data be directed in online service module.
Concrete, send data check instruction by OSS to data importing server.
If the data wherein got from data source are identical with the corresponding data got from online service module, then verification succeeds, otherwise, verify unsuccessfully.Like this, can ensure to be synchronized to the data in online service module and the data in data source are consistent.
In an alternative embodiment of the invention, this validate service device is extracting part divided data from data source and online service module respectively, and the data be drawn into is contrasted, to verify the data be directed in online service module.
In an alternative embodiment of the invention, the method also comprises the steps:
When increasing or reduce the quantity of data adapter unit in real-time data transmission system, the information of the data adapter unit increasing or reduce is stored to data importing server, and data importing server is according to the information updating data available adapter list of the data adapter unit increased or reduce.
In the present embodiment, when real-time data transmission system needs dilatation, as needed dilatation owing to needing data volume to be processed to increase, or due to online machine quantity increase need dilatation time, the quantity of data adapter unit can be increased in real-time data transmission system, now only need the information of the data adapter unit of increase to be stored to data importing server, data importing server, according to the information updating data available adapter list of the data adapter unit increased or reduce, can realize dilatation.
In an alternative embodiment of the invention, the method also comprises the steps:
When increasing or reduce the quantity of transmission proxy module in real-time data transmission system, the information of the transmission proxy module increasing or reduce is stored to data importing server, and data importing server is according to the information updating available transmission proxy module list of the data adapter unit increased or reduce.
In an alternative embodiment of the invention, the method also comprises the steps:
Obtain the total bandwidth available of real-time data transmission system, the quantity of data available adapter is controlled according to total bandwidth available, and the information of data available adapter is stored to data importing server, data importing server is according to the information updating data available adapter list of the data adapter unit increased or reduce.
In an alternative embodiment of the invention, the method also comprises the steps:
Obtain current time, the quantity of the time period control data available adapter belonging to current time, and the information of data available adapter is stored to data importing server, data importing server is according to the information updating data available adapter list of the data adapter unit increased or reduce.
In the present embodiment, pre-set the quantity of time period and this time period data available adapter, like this after getting current time, can time period belonging to current time, the quantity of data available adapter is controlled the quantity at this time period data available adapter pre-set.
One of ordinary skill in the art will appreciate that, the all or part of step realized in above-described embodiment method is that the hardware that can carry out instruction relevant by program has come, described program can be stored in a computer read/write memory medium, described storage medium, as ROM/RAM, disk, CD etc.
In the present embodiment, unification reads data by data adapter unit from data source, the data retransmission read by data adapter unit by transmission proxy module is to data importing server, therefore, when needs dilatation, only need to increase data adapter unit and/or transmission proxy module, make real-time data transmission system have good autgmentability, thus the synchronous of mass data can be realized.By carrying out error checking and correction to each data received, and by successful for error checking and correction data importing online service module, thus ensure the accuracy of the data be directed in online service module.By repairing according to the data of verification failure information to synchronization failure of record, thus ensure that the high reliability of data syn-chronization, simultaneously by distinguishing file-level failure information and record level failure information, thus greatly reducing the expense of reparation.Reception and the importing of data importing server by adopting the mode of asynchronous buffer to realize data, thus make data transmission efficiency can reach the upper limit of unit network interface card bandwidth, thus make use of the bandwidth of unit network interface card fully, improve the efficiency of transfer of data.By obtaining data respectively from data source and online service module, and the data got are contrasted, to verify the data be directed in online service module, can ensure calmly to be synchronized to the data in online service module and the data in data source are consistent.
The foregoing is only preferred embodiment of the present invention, not in order to limit the present invention, all any amendments done within the spirit and principles in the present invention, equivalent replacement and improvement etc., all should be included within protection scope of the present invention.
Claims (17)
1. a real-time data transmission system, comprise the first Internet data center and the second Internet data center, it is characterized in that, described first Internet data center comprises data source and at least one data adapter unit, described second Internet data center comprises at least one transmission proxy module, at least one data importing server, and the online service module be connected with data importing server;
Wherein said data adapter unit reads data from described data source, and the data of reading are forwarded to described data importing server by described transmission proxy module, and described data importing server is by online service module described in the data importing that receives.
2. real-time data transmission system as claimed in claim 1, it is characterized in that, described data adapter unit is also for after reading data from described data source, calculate the error check code of the data read, and the error check code of the data read and described data is sent to described data importing server by described transmission proxy module.
3. real-time data transmission system as claimed in claim 2, it is characterized in that, described data importing server is also for when receiving the error check code of data and described data, the error check code of described data is adopted to verify the data received, and online service module described in the successful data importing of error checking and correction.
4. real-time data transmission system as claimed in claim 3, it is characterized in that, described data importing server is also for after adopting the error check code of described data to verify the data received, if authentication failed, record verification failure information, wherein said verification failure information comprises file-level failure information and/or record level failure information.
5. real-time data transmission system as claimed in claim 4, it is characterized in that, described data importing server, also for after a synchronous task completes, is repaired according to the data of described verification failure information to synchronization failure.
6. the real-time data transmission system as described in claim as arbitrary in claim 1 to 5, is characterized in that, described data importing server comprises:
Buffer memory, for many records of storage file;
Transfer of data submodule, for when described buffer memory is not also write full, obtains data by described transmission proxy module from described data adapter unit, and the data buffer storage extremely described buffer memory that will obtain;
Data importing submodule, during for there are data in described buffer memory, by online service module described in the data importing in described buffer memory.
7. real-time data transmission system as claimed in claim 6, it is characterized in that, the data buffer storage extremely described buffer memory that described transfer of data submodule adopts concurrent writing mode to obtain, described data importing submodule adopts in online service module described in the data importing in the just described buffer memory of concurrent reading manner.
8. the real-time data transmission system as described in claim as arbitrary in claim 1 to 5, is characterized in that, described system also comprises:
Validate service device, for when receiving data check instruction, obtaining data respectively, and the data got being contrasted from described data source and described online service module, to verify the data be directed in described online service module.
9. the method for data synchronization in real-time data transmission system, is characterized in that, described method comprises:
Data importing server, when receiving data syn-chronization instruction, sends request of data by transmission proxy module to data adapter unit;
Data adapter unit reads data from data source, and the data of reading are sent to data importing server by transmission proxy module;
Data importing server receives data from transmission proxy module, and the data importing online service module that will receive.
10. method of data synchronization as claimed in claim 9, is characterized in that, described data importing server, when receiving data syn-chronization instruction, sends request of data by transmission proxy module to data adapter unit and comprises:
Data importing server, when receiving data syn-chronization instruction, is selected a transmission proxy module, from the data available adapter list prestored, is selected a data adapter unit from the available transmission proxy module list prestored;
Data importing server sends request of data to the transmission proxy module selected, and comprises the IP information of the data adapter unit of selection in described request of data;
Request of data is forwarded to the data adapter unit of selection by transmission proxy module according to the IP information of the data adapter unit in request of data.
11. method of data synchronization as claimed in claim 9, is characterized in that, read after data from data source at described data adapter unit, described method also comprises:
Data adapter unit calculates the error check code of the data read, and the error check code of described data is sent to data importing server.
12. method of data synchronization as claimed in claim 9, is characterized in that, described data importing server receives data from transmission proxy module, and the data importing online service module of reception is specifically comprised:
Data importing server receives the error check code of data and described data from transmission proxy module, and adopts error check code to carry out error checking and correction, by successful for error checking and correction data importing online service module to the data received.
13. method of data synchronization as claimed in claim 12, is characterized in that, after described data importing server employing error check code carries out error checking and correction to the data received, described method also comprises:
If error checking and correction failure, record verification failure information, described verification failure information comprises file-level failure information and/or record level failure information.
14. method of data synchronization as claimed in claim 13, it is characterized in that, described method also comprises:
Data importing server, after a synchronous task completes, is repaired according to the data of described verification failure information to synchronization failure.
15. method of data synchronization as claimed in claim 14, is characterized in that, describedly carry out reparation according to described verification failure information to the data of synchronization failure and comprise:
If described verification failure information is file-level failure information, then data importing server sends file acquisition request by transmission proxy module to data adapter unit, described file acquisition request comprises the file path of described file, and data importing server is when the file of the request that the data adapter unit receiving the forwarding of transmission proxy module reads from data source, described file is imported online service module;
If verification failure information is record level failure information, send record by transmission proxy module to data adapter unit at data importing server and obtain request, described record obtains initial address and the length information that request comprises record, and data importing server is when the record of the request that the data adapter unit receiving the forwarding of transmission proxy module reads from data source, described record is imported online service module.
Method of data synchronization as described in 16. claims as arbitrary in claim 9 to 15, is characterized in that, described data importing server receives data from transmission proxy module, and the data importing online service module that will receive, comprising:
The transfer of data submodule of data importing server, when buffer memory is not also write full, obtains data by transmission proxy module from data adapter unit, and by the data buffer storage of acquisition in buffer memory;
When the data importing submodule of data importing server exists data in discovery buffer memory, by the data importing online service module in buffer memory.
Method of data synchronization as described in 17. claims as arbitrary in claim 9 to 15, it is characterized in that, described method also comprises:
Validate service device, when receiving data check instruction, obtains data respectively, and the data got is contrasted from data source and online service module, to verify the data be directed in online service module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310566157.6A CN104639585B (en) | 2013-11-13 | 2013-11-13 | A kind of real-time data transmission system and its method of data synchronization |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310566157.6A CN104639585B (en) | 2013-11-13 | 2013-11-13 | A kind of real-time data transmission system and its method of data synchronization |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104639585A true CN104639585A (en) | 2015-05-20 |
| CN104639585B CN104639585B (en) | 2019-01-08 |
Family
ID=53217895
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310566157.6A Active CN104639585B (en) | 2013-11-13 | 2013-11-13 | A kind of real-time data transmission system and its method of data synchronization |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104639585B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105550325A (en) * | 2015-12-16 | 2016-05-04 | 北京奇虎科技有限公司 | Data management method and device |
| CN105608132A (en) * | 2015-12-16 | 2016-05-25 | 北京奇虎科技有限公司 | Method and device for providing self-defined diagram generation service |
| CN106454767A (en) * | 2015-08-05 | 2017-02-22 | 中兴通讯股份有限公司 | Business data synchronization method, device and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101197694A (en) * | 2006-12-04 | 2008-06-11 | 中兴通讯股份有限公司 | Central statistics and processing system and method for communication system log |
| CN101448020A (en) * | 2008-12-29 | 2009-06-03 | 深圳市迅雷网络技术有限公司 | Data source return method and device |
| CN100531055C (en) * | 2006-04-10 | 2009-08-19 | 华为技术有限公司 | Data synchronous system and its method |
| CN102629929A (en) * | 2012-04-18 | 2012-08-08 | 华为技术有限公司 | Method and system and device for obtaining data |
| US20130042023A1 (en) * | 2011-08-08 | 2013-02-14 | Li Jen CHEN | Data synchroniztion method |
-
2013
- 2013-11-13 CN CN201310566157.6A patent/CN104639585B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100531055C (en) * | 2006-04-10 | 2009-08-19 | 华为技术有限公司 | Data synchronous system and its method |
| CN101197694A (en) * | 2006-12-04 | 2008-06-11 | 中兴通讯股份有限公司 | Central statistics and processing system and method for communication system log |
| CN101448020A (en) * | 2008-12-29 | 2009-06-03 | 深圳市迅雷网络技术有限公司 | Data source return method and device |
| US20130042023A1 (en) * | 2011-08-08 | 2013-02-14 | Li Jen CHEN | Data synchroniztion method |
| CN102629929A (en) * | 2012-04-18 | 2012-08-08 | 华为技术有限公司 | Method and system and device for obtaining data |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106454767A (en) * | 2015-08-05 | 2017-02-22 | 中兴通讯股份有限公司 | Business data synchronization method, device and system |
| CN105550325A (en) * | 2015-12-16 | 2016-05-04 | 北京奇虎科技有限公司 | Data management method and device |
| CN105608132A (en) * | 2015-12-16 | 2016-05-25 | 北京奇虎科技有限公司 | Method and device for providing self-defined diagram generation service |
| CN105608132B (en) * | 2015-12-16 | 2019-04-30 | 北京奇虎科技有限公司 | It is a kind of that the method and apparatus of the customized service for generating chart are provided |
| CN105550325B (en) * | 2015-12-16 | 2019-05-31 | 北京奇虎科技有限公司 | A kind of data managing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104639585B (en) | 2019-01-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102476531B1 (en) | Data Synchronization Method and Apparatus, Media, and Electronic Device for Distributed Systems | |
| CN107911249B (en) | Method, device and equipment for sending command line of network equipment | |
| KR20180105675A (en) | Method, apparatus, and system for synchronizing data | |
| US11687488B2 (en) | Directory deletion method and apparatus, and storage server | |
| CN104468843A (en) | File uploading method and device | |
| CN102355682A (en) | Sensor adapting method, sensor adapting device and IOT (Internet Of Things) system | |
| CN103164523A (en) | Inspection method, device and system of data consistency inspection | |
| CN104104717A (en) | Inputting channel data statistical method and device | |
| CN111125175B (en) | Service data query method and device, storage medium and electronic device | |
| CN106339273A (en) | Application program restoration method, terminal and server | |
| CN107066300A (en) | The firmware upgrade method and storage device of a kind of storage device | |
| CN105657781A (en) | Method and device for accessing WiFi network | |
| CN104639585A (en) | Real-time data transmission system and data synchronization method thereof | |
| CN104298675A (en) | Method and device for cache management | |
| CN115080515A (en) | Block chain based system file sharing method and system | |
| CN105573872A (en) | Hardware maintenance method and device of data storage system | |
| CN109783026A (en) | A kind of method and device of automatic configuration server RAID | |
| CN104915376A (en) | Cloud storage file archiving and compressing method | |
| CN102523205A (en) | Determination method and device for content checksum | |
| CN103391246A (en) | Message processing method and device | |
| JP6364727B2 (en) | Information processing system, distributed processing method, and program | |
| CN110765125A (en) | Data storage method and device | |
| CN103167545B (en) | Be correlated with the method for IP cutover and device in a kind of base station | |
| CN103458030B (en) | welding machine control program upgrading system and upgrading method | |
| US10979287B2 (en) | Method and apparatus for receiving a solution when configuration function verification fails in a terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210922 Address after: 518000 Tencent Building, No. 1 High-tech Zone, Nanshan District, Shenzhen City, Guangdong Province, 35 Floors Patentee after: TENCENT TECHNOLOGY (SHENZHEN) Co.,Ltd. Patentee after: TENCENT CLOUD COMPUTING (BEIJING) Co.,Ltd. Address before: 2, 518044, East 403 room, SEG science and Technology Park, Zhenxing Road, Shenzhen, Guangdong, Futian District Patentee before: TENCENT TECHNOLOGY (SHENZHEN) Co.,Ltd. |