CN104468648B - Data processing system and method - Google Patents

Data processing system and method Download PDF

Info

Publication number
CN104468648B
CN104468648B CN201310419703.3A CN201310419703A CN104468648B CN 104468648 B CN104468648 B CN 104468648B CN 201310419703 A CN201310419703 A CN 201310419703A CN 104468648 B CN104468648 B CN 104468648B
Authority
CN
China
Prior art keywords
data
information
distribution
server
configuration information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201310419703.3A
Other languages
Chinese (zh)
Other versions
CN104468648A (en
Inventor
付凯
罗伟
杨帅
杨光
金德强
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Tencent Cloud Computing Beijing Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN201310419703.3A priority Critical patent/CN104468648B/en
Publication of CN104468648A publication Critical patent/CN104468648A/en
Application granted granted Critical
Publication of CN104468648B publication Critical patent/CN104468648B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06—Management of faults, events, alarms or notifications
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/565—Conversion or adaptation of application format or content
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/565—Conversion or adaptation of application format or content
    • H04L67/5651—Reducing the amount or size of exchanged application data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The invention discloses a kind of data processing system and methods, belong to network technique field.The system comprises: upper space in a newspaper, the first data information inputted for acquiring user by human-machine interface module, and the first data information is reported to Receiving Agent server；First data information includes the data to be reported, the Data Identification for unique identification data；Receiving Agent server carries out format to the first data information according to data format configuration information and configures to obtain the second data information for receiving the first data information and obtaining corresponding data format configuration information according to Data Identification；The second data information is sent to corresponding file reception the distribution server；File reception the distribution server obtains corresponding data receiver distribution of configuration information according to Data Identification, and carry out storage or distribution processor to data according to data receiver distribution of configuration information for receiving the second data information.By using technical solution of the present invention, the efficiency of data processing can be effectively improved.

Description

Data processing system and method
Technical field
The present invention relates to network technique field, in particular to a kind of data processing system and method.
Background technique
With the rapid development of the computer and networks communication technology, the growth of enterprise's data explosion formula to be processed, data Amount be raised to PB(1024TB=1PB from TB(1024GB=1TB) rank) grade, EB(1024PB=1EB), even ZB(1024EB= 1ZB) rank, therefore the processing of data is known as people and pays close attention to increasingly closer problem.
In the prior art, each business needs data being transferred to destination from upper space in a newspaper when carrying out data processing, by In business nature difference, the format of data is caused also to be not quite similar, while being also possible to lead to data handling requirements, data transmission Mode and the destination of data transmission may also be different.For example, format and the bank of data caused by educational institution The format of data caused by mechanism is not identical, and corresponding data handling requirements, data transfer mode and data transmit purpose Ground is also different.Under normal conditions, it in data transmission procedure, for the data of each format, requires voluntarily to realize Interface is reported, server is built, transmission channel is established and data distribution receives the actions such as management.
In the implementation of the present invention, the inventor finds that the existing technology has at least the following problems: the number of the prior art According in transmission process, for the data of each format, requires voluntarily to realize and report interface, build server, establish transmission Channel and data distribution receive the actions such as management.And with the increase of the type of the format of data to be treated, data The significantly explosion of amount increases, and the data processing work in data transmission procedure is related to many levels, and framework is complicated, workload Greatly, multiplicity is high, and acquisition territorial scope is wide, and transmission difficulty is big, causes the efficiency of data processing very low.
Summary of the invention
In order to solve problems in the prior art, the embodiment of the invention provides a kind of data processing system and methods.It is described Technical solution is as follows:
On the one hand, a kind of data processing system is provided, the system comprises:
Upper space in a newspaper, the first data information inputted for acquiring user by human-machine interface module, and to the reception generation Reason server reports first data information；First data information includes the data to be reported, for unique identification institute State the Data Identification of data；
The Receiving Agent server, first data information reported for receiving the upper space in a newspaper；According to described Data Identification obtains corresponding data format configuration information, according to the data format configuration information to first data information Format is carried out to configure to obtain the second data information；And the second data letter is sent to corresponding file reception the distribution server Breath, second data information further includes the IP of the file reception the distribution server on the basis of first data information Address；
The file reception the distribution server is obtained for receiving second data information according to the Data Identification Corresponding data receiver distribution of configuration information, and according to the data receiver distribution of configuration information to the data carry out storage or Person's distribution processor.
On the other hand, a kind of data processing method is provided, which comprises
The first data information that upper space in a newspaper acquisition user is inputted by human-machine interface module, and to the Receiving Agent service Device reports first data information；First data information includes the data to be reported, for data described in unique identification Data Identification；
The Receiving Agent server receives first data information that upper space in a newspaper reports, and according to the Data Identification Obtain corresponding data format configuration information；
The Receiving Agent server carries out format to first data information according to the data format configuration information Configuration obtains the second data information；
The Receiving Agent server sends second data information to corresponding file reception the distribution server, described Second data information further includes the IP address of the file reception the distribution server on the basis of first data information；
The file reception the distribution server receives second data information, is obtained according to the Data Identification corresponding Data receiver distribution of configuration information；
The file reception the distribution server stores the data according to the data receiver distribution of configuration information Or distribution processor.
The data processing system and method for the embodiment of the present invention introduce above-mentioned upper space in a newspaper, Receiving Agent server and text Part receives the distribution server can carry out data transmission the data of any format by using above-mentioned technical proposal；It is not required to Interface reported to realize to the data of each format, build server, established transmission channel and data distribution receives management Equal work, so as to which all numbers can be completed using same set of system in the case where the significantly explosion of data volume increases According to transmission, efficiently reduce data transmission complexity, therefore using the embodiment of the present invention technical solution, can be effectively Improve the efficiency of data processing.
And the technical solution of the embodiment of the present invention, when in use, user only needs the user interface in upper space in a newspaper enterprising Row configuration, reports the first data information, user can be needed to pass by Receiving Agent server and file reception the distribution server Defeated data are sent to destination.Very easy to use, data-handling efficiency is very high.
Detailed description of the invention
To describe the technical solutions in the embodiments of the present invention more clearly, make required in being described below to embodiment Attached drawing is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the invention, for For those of ordinary skill in the art, without creative efforts, it can also be obtained according to these attached drawings other Attached drawing.
Fig. 1 is the structural schematic diagram for the data processing system that one embodiment of the invention provides.
Fig. 2 be another embodiment of the present invention provides data processing system structural schematic diagram.
Fig. 3 is the structural schematic diagram for the data processing system that yet another embodiment of the invention provides.
Fig. 4 is a kind of work structure diagram of data processing system provided in an embodiment of the present invention.
Fig. 5 is the flow chart for the data processing method that one embodiment of the invention provides.
Fig. 6 is the signaling diagram of data processing method provided in an embodiment of the present invention.
Specific embodiment
To make the object, technical solutions and advantages of the present invention clearer, below in conjunction with attached drawing to embodiment party of the present invention Formula is described in further detail.
Fig. 1 is the structural schematic diagram for the data processing system that one embodiment of the invention provides.As shown in Figure 1, the present embodiment Data processing system include: upper space in a newspaper 10, Receiving Agent server 11 and file reception the distribution server 12.
Wherein upper space in a newspaper 10 is for acquiring the first data information that user input by human-machine interface module, and to reception generation Reason server 11 reports the first data information；First data information includes the data to be reported, for the unique identification data Data Identification (Data ID)；Using Data ID unique identification data (may include in amount due to the data is a string, Therefore it is referred to as data set), such as data can be indicated with 1xxxxxx, such as 1000001.Receiving Agent server 11 It is connect with upper space in a newspaper 10, Receiving Agent server 11 is for receiving the first data information that space in a newspaper 10 reports, the first data letter Breath include to be reported data, for unique identification data Data Identification for example, for each region, can be correspondingly arranged Space in a newspaper 10 reports the first data information to Receiving Agent server 11 on one group.Such as Shenzhen area can dispose space in a newspaper on one group 10, District of Shanghai can also dispose space in a newspaper 10 on one group.Space in a newspaper may include space in a newspaper 10 at least two on each group.
User can pass through user interface (the User Interface of upper space in a newspaper 10；Abbreviation UI) report the first data to believe Breath, that is to say, that it may include multiple options, such as the data to be reported on UI, for the Data ID of unique identification data, It such as can also include that following information correspond at least one of option data format, data length (such as also refers to want The length of the data reported), timestamp (such as also referring to the time reported), upper space in a newspaper IP address, data to be sent to Purpose geographical location information (such as geographical location information herein can be the city or area to be sent), O&M information (also referring to operator's informaiton) and the destination address to be sent to (such as can be the better address to be sent, it such as can be with At least one of including the corresponding detailed IP address of purpose geographical location information), such as can also include version number.For every A kind of information can correspond to an option, the institute for including in the first data information that user is reported by option selection on UI There is information, wherein the data and Data ID to be reported are the information that must include in the first data information.In the present embodiment, The interface that upper space in a newspaper 10 acquires data can be the interface of any acquisition data in the prior art, in detail can be with reference to correlation The prior art, details are not described herein.
Receiving Agent server 11 is connect with file reception the distribution server 12, and Receiving Agent server 11 is also used to basis Data Identification obtains corresponding data format configuration information, carries out format to the first data information according to data format configuration information Configuration obtains the second data information；And the second data information, the second data are sent to corresponding file reception the distribution server 12 Information further includes the IP address of file reception the distribution server on the basis of the first data information；Such as it at this time can be second Increase field sequence number in data information, which is made of the IP+Data ID+ of Receiving Agent server from ID is increased.Such as If the following table 1 is second data information.Data format configuration information can be pre-configured in Receiving Agent server 11.
Table 1
Version Data ID Sequence number Length Timestamp Upper space in a newspaper IP Area/city Operator Data
File reception the distribution server 12 is used for the second data information that Receiving Agent server 11 is sent, according to data mark Know and obtain corresponding data receiver distribution of configuration information, and according to data receiver distribution of configuration information to data carry out storage or Distribution processor.Data receiver distribution of configuration information can be pre-configured in file reception the distribution server 12.
Optionally, in the present embodiment, when in the first data information that upper space in a newspaper 10 reports only include the data to be reported with It may include addition number and when Data ID, in the data format configuration information that Receiving Agent server 11 is obtained according to Data ID According to format, addition data length, addition timestamp, (such as corresponding timestamp can receive for Receiving Agent server at this time The receiving time of one data information), the purpose geographical location information that the IP address of space in a newspaper 10, addition data to be sent in addition, Add O&M information and the addition destination address to be sent to and data are converted in specified data format at least one Kind；Such as the format of data is key-value pair (Key-Value) format, json format, binary coded format or supports load User's custom coding format that dynamic base mode is realized.The format that user specifies can be one of above-mentioned format.
The data processing system of the present embodiment introduces above-mentioned upper space in a newspaper, Receiving Agent server and file reception distribution Server can carry out data transmission the data of any format by using above-mentioned technical proposal；It does not need to each The data of format report interface to realize, build server, establish transmission channel and data distribution receives the work such as management, thus The transmission of all data can be completed using same set of system, have in the case where the significantly explosion of data volume increases Effect ground reduces the complexity of data transmission, therefore using the technical solution of the present embodiment, can effectively improve data processing Efficiency.
And the technical solution of the present embodiment, when in use, user only needs to match in the user interface of upper space in a newspaper It sets, reports the first data information, what user can be needed to transmit by Receiving Agent server and file reception the distribution server Data are sent to destination.Very easy to use, data-handling efficiency is very high.
Optionally, the Receiving Agent server 11 in above-mentioned embodiment illustrated in fig. 1 is specifically used for when in the second data information Including purpose geographical location information, second is sent to corresponding file reception the distribution server 12 according to purpose geographical location information Data information；Such as when matching in the second data information postponed includes purpose geographical location information, show Receiving Agent service Device 11 knows data whereabouts to be sent, such as when purpose geographical location is Shanghai, at this time Receiving Agent server 11 The second data information is sent to the file reception server 12 of District of Shanghai.When destination address is district in Beijing, receive at this time Proxy server sends the second data information to the file reception server 12 of district in Beijing.Certainly, the purpose geographical location It is provided with file reception the distribution server.When the purposes stated in detail geographical location does not have file reception server 12, at this time Second data information can be sent to the file reception server that its higher level's geographical location is disposed.
Or Receiving Agent server 11 determines that file connects also particularly useful for according to the format of data in the second data information The IP address for receiving the distribution server 12 sends the second data information to corresponding file reception the distribution server 12.This kind of situation Correspond to, for certain some data, user is after reporting, the not specified destination sent, and by Receiving Agent server 11 The purpose geographical location of the data is determined according to the format of data, so that it is determined that the corresponding file in purpose geographical location to be sent connects Receive the IP address of the distribution server 12.
Optionally, the Receiving Agent server 11 in above-mentioned embodiment illustrated in fig. 1 is also used to corresponding file reception Before the distribution server 12 sends the second data information, according to the corresponding data transmission configuration information of Data ID acquisition, and according to Data send the sample rate in configuration information, carry out resampling filter to data.Wherein data send configuration information and are stored in advance in In Receiving Agent server 11.
Optionally, the Receiving Agent server 11 in above-mentioned embodiment illustrated in fig. 1 is also used to corresponding file reception Before the distribution server 12 sends the second data information, corresponding transmission link configuration information is obtained according to Data ID, according to biography Transmission link configuration information obtains corresponding transmission link, and the second data information is sent to the corresponding transmit queue of transmission link In, so that the second data information is sent to file reception the distribution server by the transmit queue of transmission link.Wherein transmission link Configuration information is also stored in advance in Receiving Agent server 11.During carrying out data transmission, each transmission link is all Correspondence establishment has a transmit queue.Such as transmission link therein is virtual Internet data center (Virtual Internet Data Center；VIDC) link.
Still optionally further, on the basis of the technical solution of above-described embodiment, Receiving Agent server 11 is also used to Before sending the second data information to corresponding file reception the distribution server 12, corresponding data are obtained according to Data ID and are passed Defeated configuration information, and according to data transmission of configuration information the second data information of VIDC link pair be encrypted and decrypted processing, Perhaps compression & decompression processing or data check processing.Wherein data transmission of configuration information, which is also stored in advance in, receives generation It manages in server 11.
Processing wherein, which is encrypted and decrypted, to the second data information in VIDC link can effectively guarantee important number According to safe transmission.Carrying out compression & decompression processing to the second data information in VIDC link can be improved bandwidth usage Rate.The integrality of data can effectively be guaranteed by carrying out checking treatment to the second data information in VIDC link.
All the above alternatives can use combinable mode any combination, be formed of the invention optional Technical solution.
Fig. 2 be another embodiment of the present invention provides data processing system structural schematic diagram.As shown in Fig. 2, this implementation Example data processing system it is above-mentioned be embodiment on the basis of, data processing system further includes database 13 and/or file Server 14；As shown in Fig. 2, of the invention to describe by taking data processing system includes database 13 and file server 14 as an example Technical solution.
Database 13 and file server 14 are connect with file reception the distribution server 12 respectively, database 13 and file clothes Device 14 be engaged in for when not including destination address in the second data information, storage file to receive the number that the distribution server 12 is distributed According to.This kind of situation is corresponding, when UI interface reported data of the user by upper space in a newspaper, the data reported is not specified to pass Defeated destination address can store data in database 13 or file service at the reason position of user designated destination at this time In device 14, when user needs in use, voluntarily being gone in database or file server 14 to extract by user.Such as when data are It when the form of data flow, stores data in database 13, when data are the form of file, at this time stores data at this time In file server 14.
Optionally, in the present embodiment, it includes mesh in the second data information that file reception the distribution server 11, which is specifically used for working as, Address when, according to data receiver distribution of configuration information, using preset data distribution strategy, to the corresponding user of destination address Equipment sends data.Such as the corresponding user equipment of destination address include it is multiple, multiple user equipmenies may be constructed one at this time User equipment cluster, is only externally an address, and data distribution strategy that can be preset is a certain into cluster by data distribution On a user equipment.Such as preset data distribution strategy may include poll, timesharing or the distribution of the load balancing of weighting Strategy or load dynamic base or regular expression by way of realize the customized distribution policy of user.Using this kind of side Formula can be by data distribution in the other users equipment into cluster when wherein some user equipment failure.It can be effective Ground avoids Single Point of Faliure, the stability of improve data transfer, thus the efficiency of improve data transfer.
Optionally, in the present embodiment, file reception the distribution server 12 is also used to set to the corresponding user of destination address Before preparation send data, data are backed up according to data receiver distribution of configuration information.However, to ensure that memory space It rationally utilizes, after a backup, can be arranged to the data of backup reasonable validity period, it, can be right in time when data are exceeded the time limit The data exceeded the time limit are deleted.
As shown in Fig. 2, the data processing system of the present embodiment can also include management platform 15.The management platform 15 is used for Resource information and data configuration resource information to all devices in system are managed；The data configuration resource information includes Data format configuration information and data receiver distribution of configuration information；It should be noted that the other configurations in above-described embodiment are believed Breath, such as data send configuration information, transmission link configuration information and data transmission of configuration information and also belong to management platform The 15 data configuration resource informations managed.Manage platform 15 respectively with upper space in a newspaper 10, Receiving Agent server 11 and file It receives the distribution server 12 to be all connected with, such as each equipment space in a newspaper 10 as above, Receiving Agent in management 15 detection system of platform Whether server 11 and file reception the distribution server 12 are abnormal, can specifically be sent by each equipment to management platform 15 Heartbeat message realizes, when within a preset period of time, when management platform 15 does not receive the heartbeat message of certain equipment, it is believed that should Equipment breaks down, and alarm can be issued by managing platform 15 at this time, such as can be wide to the other equipment in data processing system The unit exception is broadcast, stops communicating with, carries out Automatic dispatching replacement if the equipment of the failure has alternate device.Such as Management platform 15, which can issue alarm, specifically can also use instant messaging (such as short message, wechat, the side QQ for management platform 15 Formula) mode notify administrative staff.And managing platform 15 can notify that the other equipment equipment breaks down, and may temporarily cannot Carry out data transmission.
Further, all devices in management platform 15 and system, space in a newspaper 10, Receiving Agent server 11 and text as above Part receives the distribution server 12 and connects.Management platform 15 is specifically used for periodically interacting with Receiving Agent device 11, is connect with synchronizing Receive the data format configuration information in proxy server 11；Or for when data format configuration information updates, to receiving generation It manages device 11 and sends the data format configuration information updated；
Management platform 15 is also used to periodically be handed over file reception the distribution server 12, receives distribution clothes with synchronous documents The data receiver distribution of configuration information being engaged in device 12；Or for when data receiver distribution of configuration information updates, Xiang Wen Part receives the distribution server 12 and sends the data receiver distribution of configuration information updated.
Optionally, the management platform 15 in the present embodiment is also used to reception proxy server 11 and file reception the distribution server 12 unusual condition is detected, and in Receiving Agent device 11 and/or abnormal file reception the distribution server 12, is issued and accused It is alert.
Still optionally further, the management platform 15 of the present embodiment is also used in Receiving Agent device 11 and/or file reception point When sending out 12 exception of server, administrative staff are notified by the way of instant messaging.It is logical for example, by using means such as short message, wechat, QQ Administrative staff are known, so that administrative staff in time carry out abnormal Receiving Agent device 11 and/or file reception the distribution server 12 Processing.
The data processing system of the present embodiment, by using above-mentioned technical proposal, can data to any format into The transmission of row data；It does not need to report interface to the data of each format to realize, builds server, establish transmission channel sum number According to distribution receive management etc. work, so as to the significantly explosion of data volume increase in the case where, using same set of system The transmission that all data can be completed efficiently reduces the complexity of data transmission, therefore uses the technical solution of the present embodiment, The efficiency of data processing can be effectively improved.
And the technical solution of the present embodiment, when in use, user only needs to match in the user interface of upper space in a newspaper It sets, reports the first data information, what user can be needed to transmit by Receiving Agent server and file reception the distribution server Data are sent to destination.Very easy to use, data-handling efficiency is very high.
Fig. 3 is the structural schematic diagram for the data processing system that yet another embodiment of the invention provides.As shown in figure 3, this implementation The data processing system of example may include space in a newspaper 10 at least one set, at least on the basis of the working principle of above-described embodiment One group of Receiving Agent server 11 and at least one set of file reception the distribution server 12；Wherein the setting of upper space in a newspaper 10 of different groups exists Different geographical location, such as one group of Beijing area, one group of District of Shanghai, one group of Shenzhen area, one group of area XX etc..On every group Space in a newspaper 10 includes space in a newspaper 10 at least two；In this way can be on some when 10 failure of space in a newspaper, space in a newspaper 10 can be normal on other Work, prevents Single Point of Faliure from causing system that can not work, ensure that the stability of work, wherein space in a newspaper 10 constitutes one on every group Upper space in a newspaper cluster, outside the upper space in a newspaper 10 from the point of view of, upper space in a newspaper cluster is an entirety, and particularly multiple in space in a newspaper cluster report How end 10 works can go to be arranged according to internal job requirement, such as wherein some failure when, space in a newspaper 10 is temporary in the failure It does not work, maintains work to continue by space in a newspaper 10 on other in upper space in a newspaper cluster.And when all upper spaces in a newspaper 10 are all normal, at this time It can be according to preset rules, such as the reporting policy of the load balancing of timesharing, poll or weighting or by loading dynamic base Or the mode of regular expression realizes the customized reporting policy of user, to distribute the work of each upper space in a newspaper 10.
Similarly as shown in figure 3, in the present embodiment, every group of Receiving Agent server 11 includes at least two Receiving Agent services Device 11 constitutes Receiving Agent cluster.Similarly different geographical locations is arranged in different group Receiving Agent clusters, such as Beijing area one Group Receiving Agent cluster, one group of District of Shanghai Receiving Agent cluster, one group of Receiving Agent cluster in the area XX etc..In Receiving Agent From the point of view of outside cluster, Receiving Agent cluster is an entirety, has an address, and upper space in a newspaper 10 is reported to Receiving Agent cluster Data require no knowledge about which Receiving Agent server be reported to, and are arranged by Receiving Agent cluster according to the working method of setting Which worked by Receiving Agent server.By using Receiving Agent cluster, wherein when some Receiving Agent 11 failure of server, The failure Receiving Agent server 11 does not work temporarily, maintains work by other Receiving Agent servers 11 in Receiving Agent cluster Work continues.And when all Receiving Agent servers 11 are all normal, at this time can according to preset rules, as timesharing, poll or The load balancing of weighting send and receive strategy or load dynamic base or regular expression by way of realize user from Definition sends and receivees strategy, to distribute the work of each Receiving Agent server 11, guarantees that Receiving Agent collection group energy is stablized The data that upper space in a newspaper 10 is reported receive, can steadily to file reception the distribution server 12 send data.It is every to connect Receiving Agent cluster includes at least two Receiving Agent servers 11；In this way can in some Receiving Agent 11 failure of server, Other Receiving Agent servers 11 can work normally, and prevent Single Point of Faliure from causing system that can not work, and ensure that the steady of work It is qualitative.
In addition, similarly, as shown in figure 3, every group of file reception the distribution server 12 of the present embodiment also may include at least Two file reception the distribution servers 12, configuration file receive distribution cluster.If Beijing area a group of file receives distribution cluster, District of Shanghai a group of file receives distribution cluster, and the area XX a group of file receives distribution cluster etc..Distribute in file reception and collects From the point of view of group is external, it is an entirety that file reception distribution cluster, which is one, has an address, Receiving Agent cluster is sent to file The data for receiving distribution cluster require no knowledge about which file reception the distribution server 12 be reported to, and distribute cluster by file reception It is arranged according to the working method of setting and is worked by which file reception the distribution server 12.Distribute by using file reception Collection, wherein when some file reception 12 failure of the distribution server, which does not work temporarily, by File reception distributes the alternative document in cluster and receives the maintenance work continuation of the distribution server 12.And when All Files receive distribution When server 12 is all normal, at this time can according to preset rules, as the load balancing of timesharing, poll or weighting reception and Distribution policy or load dynamic base or regular expression by way of realize the customized reception of user and distribution policy, It distributes the work of each file reception the distribution server 12, guarantees that file reception distribution collection group energy is stable to Receiving Agent collection The data of pocket transmission are received, and can be stablized and are distributed to received data.Per each file reception, distribution cluster includes at least Two file reception the distribution servers 12；In this way can be in some file reception 12 failure of the distribution server, alternative document connects Receiving the distribution server 12 can work normally, and prevents Single Point of Faliure from causing system that can not work, ensure that the stability of work.
Wherein management platform 15 and each Receiving Agent cluster and each file reception distribute cluster communication, can be to reception Agent cluster and file reception distribution cluster issue respectively required configuration information, and can periodically with Receiving Agent cluster and text Part receives distribution cluster interaction, to update newest configuration information to Receiving Agent cluster and file reception distribution cluster.Or The configuration information updated can also be sent to corresponding equipment when the configuration information managed is updated by managing platform 15. It such as manages the data format configuration information managed in platform 15 to update, manages platform 5 at this time to Receiving Agent collection pocket transmission The data format configuration information of update.When the data receiver distribution of configuration information managed in management platform 15 updates, pipe Platform distributes the reception distribution of configuration information that collection pocket transmission updates to file reception.
In the present embodiment, upper space in a newspaper 10 in upper space in a newspaper cluster and the Receiving Agent server 11 in Receiving Agent cluster it Between communication it is identical as above-described embodiment, Receiving Agent server 11 in Receiving Agent cluster and file reception distribute cluster In file receive between the distribution server 12 communication it is also identical with above-described embodiment, above-described embodiment can be referred in detail Record, details are not described herein.
The data processing system of the present embodiment is arranged by the way of cluster at each equipment, can be to avoid Single Point of Faliure The phenomenon that causing system can not work normally, can effectively enhance the stability of system.Effectively improve data processing effect Rate.And the data processing system of the present embodiment can also improve transmission link by way of using cluster at each equipment Reliability, and transmission link support service quality (Quality of Service；QoS), have load balancing and calamity standby, have Effect ground guarantees the timely, stable and reliable of data transmission.
The data processing system of the present embodiment is arranged by the way of cluster at each equipment, not with reported data amount Disconnected to increase, data processing system can be stablized by simple parallel expansion and quickly realize System Expansion, guarantee system just Often operation.
The technical solution of the present embodiment, when in use, user only need to configure in the user interface of upper space in a newspaper, on The first data information is reported, the data that user can be needed to transmit by Receiving Agent server and file reception the distribution server are sent out It send to destination.Very easy to use, data-handling efficiency is very high.The technical solution of the embodiment of the present invention, can be adapted for Acquisition, the transmission of cross-region mass data, whole flow process have complete calamity standby and Load Balance Design, ensure that data acquire Reliable and stable efficient operation, it is near minimum to develop maintenance, realizes intelligent management and control, data-handling efficiency is very It is high.
Fig. 4 is a kind of work structure diagram of data processing system provided in an embodiment of the present invention.As shown in figure 4, root According to the working principle of the data processing system of above-mentioned Fig. 1-embodiment illustrated in fig. 3, data handling procedure is divided into access layer, transmission Layer and distributing layer.Wherein the data acquisition in access layer, which reports, is completed by upper space in a newspaper.The conversion of data access format and data Sampling is completed by Receiving Agent server.The transmission link selection of transport layer, data transmission, data solution/compression, data add solution Close and data check is realized in the Receiving Agent server.Wherein the data receiver of distributing layer, data backup and Data distribution etc. is to receive to complete in the distribution server in file.Wherein system running state information collection is flat by managing Platform is completed.In addition management platform is also responsible for data format configuration, data send configuration, transmission link configuration, data transmission Configuration, data receiver distribution configuration, alarm management, global equipment/Network resource allocation and the configuration of global data format etc.. It only lists the partial function in data transmission system in the present embodiment, in practical application, may be used also according to the demand of data processing With the function that growth data is handled, details are not described herein.
Fig. 5 is the flow chart for the data processing method that one embodiment of the invention provides.The data processing method of the present embodiment Suitable for the data processing system of above-mentioned Fig. 1-Fig. 3 any embodiment.As shown in figure 5, the data processing method of the present embodiment, It can specifically include following steps:
100, the first data information that upper space in a newspaper acquisition user is inputted by human-machine interface module, and to Receiving Agent service Device reports the first data information；
The first data information in the present embodiment includes data to be reported and for data described in unique identification Data ID.Upper space in a newspaper can be realized to Receiving Agent server using DC interface or other interfaces and be reported the first data information.
101, Receiving Agent server receives the first data information that upper space in a newspaper reports, and is obtained and corresponded to according to Data ID Data format configuration information；
102, Receiving Agent server based on data fonnat configuring information configures to obtain to the first data information progress format Second data information；
103, Receiving Agent server sends the second data information to corresponding file reception the distribution server；
The second data information in the present embodiment further includes file reception distribution service on the basis of the first data information The IP address of device；
104, file reception the distribution server receives the second data information, and obtains corresponding data according to Data ID and connect Receive distribution of configuration information；
105, file reception the distribution server is stored or is distributed to data according to data receiver distribution of configuration information Processing.
The data processing method of the present embodiment is realized at data with the data processing system of the implemented embodiment of above-mentioned Fig. 1 The realization mechanism of reason is identical, can refer to the record of above-mentioned related embodiment in detail, details are not described herein.
The data processing method of the present embodiment, by using above-mentioned technical proposal, can data to any format into The transmission of row data；It does not need to report interface to the data of each format to realize, builds server, establish transmission channel sum number According to distribution receive management etc. work, so as to the significantly explosion of data volume increase in the case where, using same set of system The transmission that all data can be completed efficiently reduces the complexity of data transmission, therefore uses the technical solution of the present embodiment, The efficiency of data processing can be effectively improved.
Optionally, on the basis of the technical solution of above-mentioned embodiment illustrated in fig. 5, data format configuration information includes addition Purpose geographical location information that data format, data length, timestamp, upper space in a newspaper IP address, data to be sent to, O&M information and The destination address to be sent to and at least one of specified data format is converted by data；Receiving Agent server root It according to data format configuration information, can be configured to the first data information, such as addition data format, addition data are long Degree, the upper space in a newspaper IP address of addition, adds purpose geographical location information that data to be sent to, adds O&M information addition timestamp With the destination address to be sent to of addition and data can also be converted at least one of specified data format.It is each Kind configuration information can refer to the record of above-mentioned related embodiment, and details are not described herein.And/or the first number that upper space in a newspaper reports It is believed that breath further includes data format, data length, timestamp, upper space in a newspaper IP address, the data purpose geographical location to be sent to letter At least one of breath, O&M information and the destination address to be sent to.
Optionally, the format of data is key assignments (Key-Value) to format, json format, binary system in above-described embodiment Coded format or the user's custom coding format for supporting load dynamic base mode to realize.
Optionally, on the basis of the technical solution of above-mentioned embodiment illustrated in fig. 5, step 103 " Receiving Agent server to Corresponding file reception the distribution server sends the second data information ", it can specifically include the following two kinds situation:
The first situation, when including purpose geographical location information in the second data information, Receiving Agent server is according to mesh Geographical location information to corresponding file reception the distribution server send the second data information；
Second situation, Receiving Agent server determine that file reception is distributed according to the format of data in the second data information The IP address of server sends the second data information to corresponding file reception the distribution server.
Optionally, on the basis of the technical solution of above-mentioned embodiment illustrated in fig. 5, step 102 " Receiving Agent server root Format is carried out to the first data information according to data format configuration information to configure to obtain the second data information " after, step 103 " connects Receive proxy server and send the second data information to corresponding file reception the distribution server " before, at the data of the present embodiment Reason method can also include: Receiving Agent server according to the corresponding data transmission configuration information of DataID acquisition, and according to number According to the sample rate sent in configuration information, resampling filter is carried out to data.
Optionally, on the basis of the technical solution of above-mentioned embodiment illustrated in fig. 5, step 102 " Receiving Agent server root Format is carried out to the first data information according to data format configuration information to configure to obtain the second data information " after, step 103 " connects Receive proxy server and send the second data information to corresponding file reception the distribution server " before, it can also include: to receive generation It manages server and corresponding transmission link configuration information is obtained according to Data ID, obtained according to transmission link configuration information corresponding Transmission link, and the second data information is sent in the corresponding transmit queue of transmission link, for the transmission team of transmission link Second data information is sent to file reception the distribution server by column.Such as transmission link therein is VIDC link.
Still optionally further, in step 102, " Receiving Agent server based on data fonnat configuring information is to the first number at this time Configure to obtain the second data information it is believed that breath carries out format " after, " Receiving Agent server connects step 103 to corresponding file Receive the distribution server and send the second data information " before, it can also include: Receiving Agent server according to Data ID acquisition pair The data transmission of configuration information answered, and according to data transmission of configuration information the second data information of VIDC link pair carry out encryption and Decryption processing or compression & decompression processing or data check processing.
Optionally, on the basis of the technical solution of above-described embodiment, wherein step 105 " file reception the distribution server Storage or distribution processor are carried out to data according to data receiver distribution of configuration information ", it specifically includes: when in the second data information When not including destination address, store data in database or file server.
Or when in the second data information including destination address, file reception the distribution server is distributed according to data receiver Configuration information sends data to the corresponding user equipment of destination address using preset data distribution strategy.
It should be noted that the corresponding user equipment of destination address described in above-described embodiment may include it is multiple, i.e., The corresponding user equipment of destination address may be constructed a user equipment cluster.Wherein preset data distribution strategy includes wheel It askes, the distribution policy of the load balancing of timesharing or weighting or realized by way of load dynamic base or regular expression The customized distribution policy of user.
Optionally, on the basis of the technical solution of above-described embodiment, in step, " file reception the distribution server is to mesh The corresponding user equipment in address send data " before, can also include: file reception the distribution server according to data receiver point Hair configuration information backs up data.
It should be noted that on the basis of the technical solution of above-described embodiment, further, data processing system may be used also To include management platform, manages platform and pipe is carried out to the resource information and data configuration resource information of all devices in system Reason；Data configuration resource information includes data format configuration information and data receiver distribution of configuration information.Such as above-described embodiment In data format configuration information, data receiver distribution of configuration information, data send configuration information, transmission link configuration information, Data transmission of configuration information etc. is all by management platform configuration and to manage, and when first used, issued by management platform To each corresponding server apparatus.
Further, management platform carries out pipe to the resource information and data configuration resource information of all devices in system Reason, can specifically include:
Management platform is periodically interacted with Receiving Agent device, to synchronize the data format in Receiving Agent device with confidence Breath；Or management platform sends the data format updated to Receiving Agent device and matches when data format configuration information is updated Confidence breath；
And management platform is periodically handed over file reception the distribution server, receives the distribution server with synchronous documents In data receiver distribution of configuration information；Or platform is managed when data receiver distribution of configuration information updates, to file It receives the distribution server and sends the data receiver distribution of configuration information updated.
Further, management platform is periodically interacted with Receiving Agent device, can also synchronize the number in Receiving Agent device According to transmission configuration information, transmission link configuration information and data transmission of configuration information；Or management platform is configured in data format When information updates, the data updated are sent to Receiving Agent device and send configuration information, transmission link configuration information and data Transmission of configuration information.
Optionally, on the basis of the technical solution of above-described embodiment, the data processing method of the present embodiment can also wrap Include: management platform detects the unusual condition for receiving proxy server and file reception the distribution server, and in Receiving Agent device And/or when file reception the distribution server exception, alarm is issued.The step does not have time order and function suitable with the step of above-described embodiment Sequence limitation.
Further, management platform, can be using i.e. in Receiving Agent device and/or file reception the distribution server exception The mode of Shi Tongxin notifies administrative staff, so that administrative staff in time repair abnormal server apparatus.
It should be noted that all optional technical solutions in above-described embodiment, it can be any using combinative mode Combination, forms alternative embodiment of the invention, this is no longer going to repeat them.
The data processing method of above-described embodiment realizes data processing, realizes data processing with above-mentioned data processing system Realization mechanism is identical, can refer to the record of above-described embodiment in detail, details are not described herein.
The data processing system of above-described embodiment can be to the data of any format by using above-mentioned technical proposal Carry out data transmission；Do not need to report interface to the data of each format to realize, build server, establish transmission channel and Data distribution receive management etc. work, so as to the significantly explosion of data volume increase in the case where, using same set of system The transmission of all data can be completed in system, efficiently reduces the complexity of data transmission, therefore using the technical side of the present embodiment Case can effectively improve the efficiency of data processing.
And the technical solution of above-described embodiment, when in use, user only needs to carry out in the user interface of upper space in a newspaper Configuration, reports the first data information, user can be needed to transmit by Receiving Agent server and file reception the distribution server Data be sent to destination.Very easy to use, data-handling efficiency is very high.
It should be noted that the data processing method of above-described embodiment also can be applied to data processing system shown in Fig. 3 In system, the record of above-mentioned embodiment illustrated in fig. 3 can be referred in detail, details are not described herein.
Fig. 6 is the signaling diagram of data processing method provided in an embodiment of the present invention.As shown in fig. 6, the data of the present embodiment Processing method further introduces technical solution of the present invention on the basis of above-mentioned Fig. 5 and alternative embodiment in further detail.Such as Shown in Fig. 6, the data processing method of the present embodiment can specifically include following steps:
200, management platform is the user data configuration Data ID to be sent, and generates corresponding data configuration resource letter Breath；
The data resource configuration information of the present embodiment may include data format configuration information, data receiver distribution with confidence Breath, data send configuration information, transmission link configuration information and data transmission of configuration information etc.；Platform is managed in the present embodiment The process for configuring Data ID can be off-line arrangement, which is used for the unique identification data.
201, when first used, management platform is interacted with Receiving Agent server, sends data to Receiving Agent server Fonnat configuring information, data send configuration information, transmission link configuration information and data transmission of configuration information；
202, when first used, management platform is interacted with file reception the distribution server, to file reception the distribution server Send data receiver distribution of configuration information；
203, user selects data, Data ID and the other parameters information to be reported by the interface UI of upper space in a newspaper, on Space in a newspaper acquires user and generates the first data information by the parameter information that the interface UI selects；
Such as other parameters information may include that data format, data length, timestamp, upper space in a newspaper IP address, data are wanted At least one of purpose geographical location information, O&M information and the destination address to be sent to for being sent to.The data can be Key-Value format, json format, binary coded format or the customized volume of user for supporting load dynamic base mode to realize Code format.
204, upper space in a newspaper sends the first data information to Receiving Agent server；
205, Receiving Agent server obtains corresponding data format configuration information according to Data ID, and according to data lattice Formula configuration information carries out format to the first data information and configures to obtain the second data information；
Data format configuration information includes adding data format, data length, timestamp, upper space in a newspaper IP address, data to want The purpose geographical location information, O&M information and the destination address to be sent to that are sent to and specified data is converted by data At least one of format.The second data information in the present embodiment further includes file reception on the basis of the first data information The IP address of the distribution server.
206, Receiving Agent server obtains corresponding transmission link configuration information according to Data ID, according to transmission link Configuration information obtains corresponding VIDC link；
207, Receiving Agent server obtains corresponding data transmission of configuration information according to Data ID；
The data transmission of configuration information includes encryption and decryption process or compression & decompression processing or data school Processing is tested, is that processing is encrypted and decrypted in the second data information of VIDC link pair, can effectively guarantee significant data Safe transmission；Or compression & decompression processing is carried out in the second data information of VIDC link pair, bandwidth availability ratio can be improved； Or it can effectively guarantee the integrality of data in the progress data check processing of the second data information of VIDC link pair.Certainly Above-mentioned three kinds of processing or any two kinds of processing can be carried out simultaneously.
208, Receiving Agent server based on data transmission of configuration information carries out with postponing, by second second data information Data information is sent in the corresponding transmit queue of VIDC link, and Receiving Agent server passes through the corresponding transmission team of VIDC link It arranges to file reception the distribution server and sends the second data information；
209, file reception the distribution server receives the second data information, obtains corresponding data receiver according to Data ID Distribution of configuration information；
210, file reception the distribution server backs up data according to data receiver distribution of configuration information, then adopts With preset data distribution strategy, data are sent to the corresponding user equipment of destination address.
Wherein the corresponding user equipment of destination address may include multiple, i.e. the corresponding user equipment of destination address can be with structure At a user equipment cluster.Wherein preset data distribution strategy includes point of poll, timesharing or the load balancing of weighting Hair is tactful or realizes the customized distribution policy of user by way of load dynamic base or regular expression.
The process of data processing can be completed by above-mentioned process.On the basis of above scheme, management platform is regular It is interacted with Receiving Agent device, to synchronize the data format configuration information in Receiving Agent device, data send configuration information, pass Transmission link configuration information and data transmission of configuration information.Or platform is managed when data format configuration information updates, to Receiving Agent device sends the data format configuration information updated, data send configuration information, transmission link configuration information and data Transmission of configuration information；
Management platform is periodically handed over with file reception the distribution server, is received in the distribution server with synchronous documents Data receiver distribution of configuration information；Or platform is managed when data receiver distribution of configuration information updates, to file reception The distribution server sends the data receiver distribution of configuration information updated.
And management platform can also detect the unusual condition for receiving proxy server and file reception the distribution server, And in Receiving Agent device and/or file reception the distribution server exception, alarm is issued.The step of step and above-described embodiment There is no chronological order limitation.
Further, management platform, can be using i.e. in Receiving Agent device and/or file reception the distribution server exception The mode of Shi Tongxin notifies administrative staff, so that administrative staff in time repair abnormal server apparatus.
The data processing method of the present embodiment, by using above-mentioned technical proposal, can data to any format into The transmission of row data；It does not need to report interface to the data of each format to realize, builds server, establish transmission channel sum number According to distribution receive management etc. work, so as to the significantly explosion of data volume increase in the case where, using same set of system The transmission that all data can be completed efficiently reduces the complexity of data transmission, therefore uses the technical solution of the present embodiment, The efficiency of data processing can be effectively improved.
And the technical solution of the present embodiment, when in use, user only needs to match in the user interface of upper space in a newspaper It sets, reports the first data information, what user can be needed to transmit by Receiving Agent server and file reception the distribution server Data are sent to destination.Very easy to use, data-handling efficiency is very high.
And the technical solution of the present embodiment, it can be used in data transmission system shown in Fig. 3, be suitable for cross-region Acquisition, the transmission of mass data, whole flow process have complete calamity standby and Load Balance Design, ensure that the stabilization of data acquisition Reliable efficient operation, it is near minimum to develop maintenance, realizes that intelligent management and control, data-handling efficiency are very high.
It should be understood that data processing system provided by the above embodiment is in data processing, only with above-mentioned each function The division progress of module can according to need and for example, in practical application by above-mentioned function distribution by different function moulds Block is completed, i.e., the internal structure of system is divided into different functional modules, to complete all or part of function described above Energy.In addition, data processing system provided by the above embodiment and data processing method embodiment belong to same design, it is specific real Existing process is detailed in embodiment of the method, and which is not described herein again.
The serial number of the above embodiments of the invention is only for description, does not represent the advantages or disadvantages of the embodiments.
Those of ordinary skill in the art will appreciate that realizing that all or part of the steps of above-described embodiment can pass through hardware It completes, relevant hardware can also be instructed to complete by program, the program can store in a kind of computer-readable In storage medium, storage medium mentioned above can be read-only memory, disk or CD etc..
The foregoing is merely presently preferred embodiments of the present invention, is not intended to limit the invention, it is all in spirit of the invention and Within principle, any modification, equivalent replacement, improvement and so on be should all be included in the protection scope of the present invention.

Claims (31)

1. a kind of data processing system, which is characterized in that the system comprises；
Upper space in a newspaper, the first data information inputted for acquiring user by human-machine interface module, and to Receiving Agent server Report first data information；First data information includes the data to be reported, for data described in unique identification Data Identification；
The Receiving Agent server, first data information reported for receiving the upper space in a newspaper；According to the data Mark obtains corresponding data format configuration information, is carried out according to the data format configuration information to first data information Format configures to obtain the second data information；And second data information, institute are sent to corresponding file reception the distribution server State the IP address that the second data information further includes the file reception the distribution server on the basis of first data information；
The file reception the distribution server is obtained according to the Data Identification and is corresponded to for receiving second data information Data receiver distribution of configuration information, and the data are stored or are divided according to the data receiver distribution of configuration information Hair processing.
2. system according to claim 1, which is characterized in that the data format configuration information includes addition data lattice Purpose geographical location information that formula, data length, timestamp, the upper space in a newspaper IP address, the data to be sent to, O&M information At least one of specified data format is converted into the destination address to be sent to and by the data；
And/or first data information that the upper space in a newspaper reports further includes data format, data length, timestamp, institute State space in a newspaper IP address, the purpose geographical location information that the data to be sent to, O&M information and the destination address to be sent to At least one of.
3. system according to claim 2, which is characterized in that the format of the data be key-value pair format, json format, Binary coded format or the user's custom coding format for supporting load dynamic base mode to realize.
4. system according to claim 2, which is characterized in that the Receiving Agent server is specifically used for when described the It include the purpose geographical location information in two data informations, according to the purpose geographical location information to the corresponding file It receives the distribution server and sends second data information；
Or the Receiving Agent server, it is determined also particularly useful for the format of the data according to second data information The IP address of the file reception the distribution server sends second data to the corresponding file reception the distribution server Information.
5. system according to claim 4, which is characterized in that the Receiving Agent server is also used to corresponding Before the file reception the distribution server sends second data information, corresponding data are obtained according to the Data Identification Configuration information is sent, and sends the sample rate in configuration information according to the data, resampling filter is carried out to the data.
6. system according to claim 1, which is characterized in that the Receiving Agent server is also used to corresponding Before the file reception the distribution server sends second data information, corresponding transmission is obtained according to the Data Identification Link configuration information obtains corresponding transmission link according to the transmission link configuration information, and by second data information It is sent in the corresponding transmit queue of the transmission link, so that the transmit queue of the transmission link believes second data Breath is sent to the file reception the distribution server.
7. system according to claim 6, which is characterized in that the transmission link is virtual Internet data center chain Road.
8. system according to claim 7, which is characterized in that the Receiving Agent server is also used to corresponding Before the file reception the distribution server sends second data information, corresponding data are obtained according to the Data Identification Transmission of configuration information, and according to the data transmission of configuration information in virtual Internet data center link pair described in second Processing or compression & decompression processing or data check processing is encrypted and decrypted in data information.
9. system according to claim 3, which is characterized in that the system also includes database and/or file services Device；
The database and/or the file server do not include the destination in second data information for working as When location, the data are stored.
10. system according to claim 2, which is characterized in that the file reception the distribution server is specifically used for working as institute When stating in the second data information including the destination address, according to the data receiver distribution of configuration information, using preset number According to distribution policy, the data are sent to the corresponding user equipment of the destination address.
11. system according to claim 10, which is characterized in that the corresponding user equipment of the destination address includes more It is a, the preset data distribution strategy include poll, timesharing or the load balancing of weighting distribution policy or by plus The mode for carrying dynamic base or regular expression realizes the customized distribution policy of user.
12. system according to claim 10, which is characterized in that the file reception the distribution server, be also used to Before the destination address corresponding user equipment transmission data, according to the data receiver distribution of configuration information to described Data are backed up.
13. system according to claim 1, which is characterized in that the system also includes management platforms；
The management platform, for all devices in system resource information and data configuration resource information be managed； The data configuration resource information includes the data format configuration information and the data receiver distribution of configuration information；
Further, the management platform is interacted specifically for regular with the Receiving Agent server, described to synchronize The data format configuration information in Receiving Agent server；Or for being updated in the data format configuration information When, the data format configuration information updated is sent to the Receiving Agent server；
The management platform, be also used to it is regular interacted with the file reception the distribution server, connect with synchronizing the file Receive the data receiver distribution of configuration information in the distribution server；Or for being sent out in the data receiver distribution of configuration information When raw update, the data receiver distribution of configuration information updated is sent to the file reception the distribution server.
14. system according to claim 13, which is characterized in that the management platform is also used to the Receiving Agent Server and the unusual condition of the file reception the distribution server are detected, and in the Receiving Agent server and/or When the file reception the distribution server exception, alarm is issued.
15. system according to claim 14, which is characterized in that the management platform was specifically used in the reception generation When managing server and/or the file reception the distribution server exception, administrative staff are notified by the way of instant messaging.
16. -15 any system according to claim 1, which is characterized in that the system comprises report described at least one set End, at least one set of Receiving Agent server and at least one set of file reception the distribution server；The wherein institute of different groups It states space in a newspaper and different geographical locations is set, upper space in a newspaper described in every group includes at least two upper spaces in a newspaper；It is connect described in every group Receiving proxy server includes at least described two Receiving Agent servers, constitutes Receiving Agent cluster；File reception described in every group The distribution server includes at least two file reception the distribution servers, and configuration file receives distribution cluster.
17. a kind of data processing method, which is characterized in that the method includes；
The first data information that upper space in a newspaper acquisition user is inputted by human-machine interface module, and institute is reported to Receiving Agent server State the first data information；First data information includes the data to be reported, the data mark for data described in unique identification Know；
The Receiving Agent server receives first data information that upper space in a newspaper reports, and is obtained according to the Data Identification Corresponding data format configuration information；
The Receiving Agent server carries out format configuration to first data information according to the data format configuration information Obtain the second data information；
The Receiving Agent server sends second data information to corresponding file reception the distribution server, and described second Data information further includes the IP address of the file reception the distribution server on the basis of first data information；
The file reception the distribution server receives second data information, obtains corresponding data according to the Data Identification Receive distribution of configuration information；
The file reception the distribution server according to the data receiver distribution of configuration information to the data carry out storage or Distribution processor.
18. according to the method for claim 17, which is characterized in that the data format configuration information includes addition data lattice Purpose geographical location information that formula, data length, timestamp, the upper space in a newspaper IP address, the data to be sent to, O&M information At least one of specified data format is converted into the destination address to be sent to and by the data；
And/or first data information that the upper space in a newspaper reports further includes data format, data length, timestamp, institute State space in a newspaper IP address, the purpose geographical location information that the data to be sent to, O&M information and the destination address to be sent to At least one of.
19. according to the method for claim 18, which is characterized in that the format of the data is key-value pair format, json lattice Formula, binary coded format or the user's custom coding format for supporting load dynamic base mode to realize.
20. method described in 8 or 19 according to claim 1, which is characterized in that the Receiving Agent server is to corresponding text Part receives the distribution server and sends second data information, specifically includes:
When including the purpose geographical location information in second data information, the Receiving Agent server is according to the mesh Geographical location information send second data information to the corresponding file reception the distribution server；
Or the format of Receiving Agent server data according to second data information determines that the file connects The IP address for receiving the distribution server sends second data information to the corresponding file reception the distribution server.
21. according to the method for claim 20, which is characterized in that the Receiving Agent server is according to the data format Configuration information carries out format to first data information and configures after obtaining the second data information, the Receiving Agent server Before sending second data information to corresponding file reception the distribution server, the method also includes:
The Receiving Agent server obtains corresponding data according to the Data Identification and sends configuration information, and according to the number According to the sample rate sent in configuration information, resampling filter is carried out to the data.
22. according to the method for claim 17, which is characterized in that the Receiving Agent server is according to the data format Configuration information carries out format to first data information and configures after obtaining the second data information, the Receiving Agent server Before sending second data information to corresponding file reception the distribution server, the method also includes:
The Receiving Agent server obtains corresponding transmission link configuration information according to the Data Identification, according to the transmission Link configuration information obtains corresponding transmission link, and second data information is sent to the corresponding hair of the transmission link It send in queue, so that second data information is sent to the file reception distribution clothes by the transmit queue of the transmission link Business device.
23. according to the method for claim 22, which is characterized in that the transmission link is virtual Internet data center chain Road.
24. according to the method for claim 23, which is characterized in that the Receiving Agent server is according to the data format Configuration information carries out format to first data information and configures after obtaining the second data information, the Receiving Agent server Before sending second data information to corresponding file reception the distribution server, the method also includes:
The Receiving Agent server obtains corresponding data transmission of configuration information according to the Data Identification, and according to the number Place is encrypted and decrypted according to transmission of configuration information second data information described in virtual Internet data center link pair Reason or compression & decompression processing or data check processing.
25. according to the method for claim 19, which is characterized in that the file reception the distribution server is according to the data It receives distribution of configuration information and storage or distribution processor is carried out to the data, specifically include:
When not including the destination address in second data information, database or file clothes are stored the data in It is engaged in device.
26. according to the method for claim 18, which is characterized in that the file reception the distribution server is according to the data It receives distribution of configuration information and storage or distribution processor is carried out to the data, specifically include:
When in second data information including the destination address, the file reception the distribution server is according to the data Distribution of configuration information is received, using preset data distribution strategy, to described in the corresponding user equipment transmission of the destination address Data.
27. according to the method for claim 26, which is characterized in that the corresponding user equipment of the destination address includes more It is a, the preset data distribution strategy include poll, timesharing or the load balancing of weighting distribution policy or by plus The mode for carrying dynamic base or regular expression realizes the customized distribution policy of user.
28. the method according to claim 26 or 27, which is characterized in that the file reception the distribution server is to described Before the corresponding user equipment of destination address sends the data, the method also includes:
The file reception the distribution server backs up the data according to the data receiver distribution of configuration information.
29. according to the method for claim 17, which is characterized in that the method also includes:
Management platform is managed the resource information and data configuration resource information of all devices in system；The data are matched Setting resource information includes the data format configuration information and the data receiver distribution of configuration information；
Further, the management platform carries out pipe to the resource information and data configuration resource information of all devices in system Reason, specifically includes:
The management platform is periodically interacted with the Receiving Agent server, to synchronize in the Receiving Agent server The data format configuration information；Or the management platform is when the data format configuration information updates, to described Receiving Agent server sends the data format configuration information updated；
And the management platform is periodically interacted with the file reception the distribution server, to synchronize the file reception The data receiver distribution of configuration information in the distribution server；Or the management platform distributes configuration in the data receiver When information updates, the data receiver distribution of configuration information updated is sent to the file reception the distribution server.
30. according to the method for claim 29, which is characterized in that the method also includes:
The management platform examines the unusual condition of the Receiving Agent server and the file reception the distribution server It surveys, and in the Receiving Agent server and/or the file reception the distribution server exception, issues alarm.
31. according to the method for claim 30, which is characterized in that in the Receiving Agent server and/or the file When receiving the distribution server exception, alarm is issued, is specifically included:
The management platform is in the Receiving Agent server and/or the file reception the distribution server exception, using i.e. The mode of Shi Tongxin notifies administrative staff.
CN201310419703.3A 2013-09-13 2013-09-13 Data processing system and method Active CN104468648B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201310419703.3A CN104468648B (en) 2013-09-13 2013-09-13 Data processing system and method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201310419703.3A CN104468648B (en) 2013-09-13 2013-09-13 Data processing system and method

Publications (2)

Publication Number Publication Date
CN104468648A CN104468648A (en) 2015-03-25
CN104468648B true CN104468648B (en) 2019-01-29

Family

ID=52914019

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201310419703.3A Active CN104468648B (en) 2013-09-13 2013-09-13 Data processing system and method

Country Status (1)

Country Link
CN (1) CN104468648B (en)

Families Citing this family (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105243078B (en) * 2015-08-28 2018-09-28 北京奇艺世纪科技有限公司 A kind of distribution method of file resource, system and device
CN106612297A (en) * 2015-10-21 2017-05-03 北京国双科技有限公司 Data processing method and device
CN106803831A (en) * 2015-11-26 2017-06-06 中兴通讯股份有限公司 A kind of document distribution system and its management method and device
CN107241371B (en) * 2016-03-29 2020-04-07 中国移动通信集团湖北有限公司 Application data message processing method, mobile terminal and system
CN107544901A (en) * 2016-06-27 2018-01-05 五八同城信息技术有限公司 The data format method of inspection and device
CN106648984A (en) * 2016-12-20 2017-05-10 上海银禾油信息技术有限公司 Data backup method and device
CN108287706B (en) * 2017-05-10 2021-09-14 腾讯科技（深圳）有限公司 Data processing method and device
CN108491364A (en) * 2018-01-25 2018-09-04 苏州麦迪斯顿医疗科技股份有限公司 Medical treatment and nursing paperwork management system
CN110309191B (en) * 2018-04-08 2023-07-04 腾讯科技（深圳）有限公司 Method, device and system for storing business data
CN108900482B (en) * 2018-06-13 2021-04-16 平安科技（深圳）有限公司 Script execution method, server management system, and storage medium
CN109639708B (en) * 2018-12-28 2022-03-18 东莞见达信息技术有限公司 Deep learning data access control method and device
CN109918403A (en) * 2019-02-02 2019-06-21 中国银行股份有限公司 Data verification method, device, computer equipment and storage medium
CN111698159A (en) * 2019-03-15 2020-09-22 顺丰科技有限公司 Service data processing method, device and storage medium
WO2020237409A1 (en) * 2019-05-24 2020-12-03 Intel Corporation Technologies for memory-efficient key-value lookup
CN110471970B (en) * 2019-08-08 2023-04-18 紫光云技术有限公司 Flexible and configurable rule engine data forwarding method
CN110647552B (en) * 2019-08-29 2022-11-11 深圳市新系区块链技术有限公司 Data distribution method and related product
CN110602254B (en) * 2019-10-08 2022-05-20 腾讯科技（深圳）有限公司 Method, device and system for realizing load balance
CN113497783B (en) * 2020-03-19 2024-01-16 北京沃东天骏信息技术有限公司 Method and device for processing data

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101483611A (en) * 2008-01-09 2009-07-15 华为技术有限公司 System, method and apparatus for implementing message service interconnection
CN102546730A (en) * 2010-12-30 2012-07-04 中国移动通信集团公司 Data processing method, device and system
CN102724107A (en) * 2011-03-31 2012-10-10 百度在线网络技术（北京）有限公司 Method for forwarding message and selecting message forwarding device, device and equipment

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8549327B2 (en) * 2008-10-27 2013-10-01 Bank Of America Corporation Background service process for local collection of data in an electronic discovery system

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101483611A (en) * 2008-01-09 2009-07-15 华为技术有限公司 System, method and apparatus for implementing message service interconnection
CN102546730A (en) * 2010-12-30 2012-07-04 中国移动通信集团公司 Data processing method, device and system
CN102724107A (en) * 2011-03-31 2012-10-10 百度在线网络技术（北京）有限公司 Method for forwarding message and selecting message forwarding device, device and equipment

Also Published As

Publication number Publication date
CN104468648A (en) 2015-03-25

Similar Documents

Publication Publication Date Title
CN104468648B (en) Data processing system and method
CN106412113A (en) Energy cloud service system and communication method thereof
CN109618002A (en) A kind of micro services gateway optimization method, device and storage medium
CN106294713A (en) The method of data synchronization resolved based on Incremental Log and data synchronization unit
CN105847139A (en) Data transmission method, apparatus and system
CN102255974A (en) Cloud storage method for cloud computing server
US20130117423A1 (en) Methods for managing data synchronization and devices thereof
US8909189B2 (en) System, method and program product for maintaining deployed response team members synchronized
CN101977236A (en) Large-file multipoint distribution system
CN201323637Y (en) Palmtop operation system
CN104536965A (en) System and method for data query and presentation under big data condition
CN110234140A (en) A kind of communication means, device, entity and storage medium
CN108156040A (en) A kind of central control node in distribution cloud storage system
CN103916256A (en) Network optimization method, device and system
CN102036188B (en) Mail proxy method, equipment and system under multi-node system
CN101467132B (en) Method and system for distributing data processing units in a communication network
EP1872256B1 (en) System and method of waste management
CN106330994A (en) User message publishing method and system
CN106487880B (en) A kind of disaster generation area Transmission system based on Big Dipper short message communication
CN103297477B (en) A kind of data acquisition reporting system and data processing method and proxy server
CN102185717A (en) Service processing equipment, method and system
JP2008225740A (en) Information processor and information processing method
CN202353815U (en) GIS short message power grid service system
CN116893933A (en) Disaster recovery control method and device of edge computing system and related equipment
CN102904739A (en) Method for realizing event transmission and common information model (CIM) server

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right

Effective date of registration: 20190730

Address after: Shenzhen Futian District City, Guangdong province 518000 Zhenxing Road, SEG Science Park 2 East Room 403

Co-patentee after: Tencent cloud computing (Beijing) limited liability company

Patentee after: Tencent Technology (Shenzhen) Co., Ltd.

Address before: Shenzhen Futian District City, Guangdong province 518000 Zhenxing Road, SEG Science Park 2 East Room 403

Patentee before: Tencent Technology (Shenzhen) Co., Ltd.

TR01 Transfer of patent right