CN106027523A

CN106027523A - Data collection method of distributed data system and distributed data system

Info

Publication number: CN106027523A
Application number: CN201610340663.7A
Authority: CN
Inventors: 张锐
Original assignee: Shenzhen Longrise Technology Co Ltd
Current assignee: Shenzhen Longrise Technology Co Ltd
Priority date: 2016-05-20
Filing date: 2016-05-20
Publication date: 2016-10-12
Anticipated expiration: 2036-05-20
Also published as: CN106027523B

Abstract

The invention discloses a data collection method and the distributed data system. The method comprises the steps that a management data center and remote terminals share own shared secret keys with each other, and the management data center receives first random numbers sent by the remote terminals and sends second random numbers generated by the management data center to the remote terminals; the management data center generates first checking values according to the shared secret keys, the first random numbers, the second random numbers and pre-stored first signature information; the management data center sends the first checking values to the remote terminals, so that the remote terminals check the first checking values, and second checking values of the remote terminals are received; and the management data center averagely allocates data collection tasks for colleting target data uploaded by the remote terminals to each auxiliary data center for execution when the first checking value and the second checking value are checked successfully and checking success signals sent by the remote terminals are received. According to the system, the phenomenon that the target data is wrongly sent to illegal terminals or the illegal target data is collected can be avoided, and the information security of the target data can be improved.

Description

The collecting method of distributed data system and distributed data system

Technical field

The present invention relates to distributed system technical field, particularly relate to the data of a kind of distributed data system Acquisition method and distributed data system.

Background technology

In reality, the most all producing substantial amounts of data, some needs to analyze, and some needs storage, All these needs data bulk to be processed to be huge, has similarity, so needing these big Data are analyzed processing, and gather the data needed.

In the prior art, the target data (example using data gather computer that remote terminal is generated it is usually Such as performance data, alarm data, data on flows, business datum etc.) it is acquired, data gather computer is also Row gathers or gathers the target data of remote terminal one by one, and then target data is uploaded to by data gather computer Data center, but, owing to the target data of generation is directly directly uploaded by remote terminal, remote terminal And not carrying out any authentication between data gather computer, data gather computer cannot ensure the target gathered The reliability of Data Source and integrity, remote terminal also cannot ensure to gather the data acquisition of target data The legitimacy of machine, thus be easily caused target data mistake send to illegal terminal or gather illegitimate target data and Cause information security.

Summary of the invention

Present invention is primarily targeted at collecting method and the distribution that a kind of distributed data system is provided Formula data system, it is intended to solve to gather in prior art target data and be easily caused target data mistake and send to non- Method terminal or gather illegitimate target data and cause the technical problem of information security.

For achieving the above object, the collecting method of a kind of distributed data system that the present invention provides, Described distributed data system includes multiple data center being in communication with each other connection and each data center subordinate Remote terminal, data center includes managing data center and assistance data center,

The collecting method of this distributed data system includes:

Respective key of sharing mutually is shared with described remote terminal by management data center, receives remote terminal The first random number sent, and the second random number generated voluntarily is sent to remote terminal；

Management data center is according to sharing key, the first random number, the second random number and the first label prestored Name information, generates the first check value；

First check value is sent to remote terminal by management data center, verifies the first school for remote terminal Test value, and receive remote terminal according to sharing key, the first random number, the second random number and this is long-range The second check value that the second signing messages that terminal prestores generates；

When management data center verifies the first check value and the success of the second check value, and receive remote terminal During the verification pass signal sent, management data center uploads the data of target data by gathering remote terminal Acquisition tasks is averagely allocated to each assistance data center and performs, the corresponding number of one of them remote terminal According to acquisition tasks.

Preferably, the data acquisition that target data uploaded by collection remote terminal is appointed by described management data center Business also includes after being averagely allocated to the step that each assistance data center performs:

Management data center obtains once each assistance data center processing data acquisition at interval of preset duration The hardware load of task, using this hardware load as with reference to hardware load；

Management data center with reference to hardware load according to described, dynamically adjust each assistance data center and performs Data acquisition session, to equalize the hardware load at each assistance data center.

Preferably, the collecting method of described distributed data system also includes:

When managing data center and breaking down, extract during current management data center is run the replacement generated Person's information, and using assistance data center corresponding for replacement person's information as new management data center.

Preferably, described management data center with reference to hardware load, dynamically adjusts each auxiliary according to described The step of the data acquisition session that data center performs includes:

Management data center with reference to hardware load, selects the very big supplementary number that hardware load is the highest according to described According to the minimum assistance data center that center and hardware load are minimum, and by the hardware at very big assistance data center Load is as very big hardware load, using the hardware load at minimum assistance data center as minimum hardware load,

Management data center calculates the load difference of described very big hardware load and minimum hardware load；

If described load difference is more than load threshold value, then management data center is from described very big assistance data The heart reclaims one or more data acquisition session, and is reallocated to by the data acquisition session of recovery minimum auxiliary Help data center.

Preferably, management data center adds up the adjustment that each data acquisition session experience reclaims and reallocates Number of times,

Described management data center reclaims one or more data acquisitions and appoints from described very big assistance data center It is engaged in, and the step that the data acquisition session of recovery is reallocated to minimum assistance data center includes:

Management data center reclaims one or more adjustment number of times less than secondary from described very big assistance data center The data acquisition session of number threshold value, using this data acquisition session as task to be allocated；

Described task to be allocated is reallocated to minimum assistance data center by management data center.

The present invention also provides for a kind of distributed data system, described distributed data system include multiple mutually The data center of communication connection and the remote terminal of each data center subordinate, data center includes managing number According to center and assistance data center,

Described management data center includes:

Sharing module, for mutually sharing respective key of sharing with described remote terminal, receives the most eventually The first random number that end sends, and the second random number generated voluntarily is sent to remote terminal；

Generation module, for according to sharing key, the first random number, the second random number and prestore first Signing messages, generates the first check value；

Receiver module, for sending the first check value to remote terminal, verifies first for remote terminal Check value, and receive remote terminal according to sharing key, the first random number, the second random number and this is remote The second check value that the second signing messages that journey terminal prestores generates；

The equal sub-module of task, for working as verification the first check value and the success of the second check value, and receives remote During the verification pass signal that journey terminal sends, the data acquisition that target data uploaded by collection remote terminal is appointed Business is averagely allocated to each assistance data center and performs, the corresponding data acquisition of one of them remote terminal Task.

Preferably, described management data center also includes

Load obtaining module, for obtaining once each assistance data center processing number at interval of preset duration According to the hardware load of acquisition tasks, using this hardware load as with reference to hardware load；

Dynamically adjustment module, for according to described reference hardware load, dynamically adjusting in each assistance data The data acquisition session that the heart performs, to equalize the hardware load at each assistance data center；

Described assistance data center sends currently processed at interval of preset duration to current management data center The hardware load of data acquisition session.

Preferably, described management data center also includes:

Replacement module, for when managing data center and breaking down, extracts current management data center fortune The replacement person's information generated in row, and using assistance data center corresponding for replacement person's information as new management Data center.

Preferably, described dynamic adjustment module includes:

Screening unit, for according to described reference hardware load, selecting the very big auxiliary that hardware load is the highest Data center and the minimum minimum assistance data center of hardware load, wherein by very big assistance data center Hardware load is as very big hardware load, using the hardware load at minimum assistance data center as minimum hardware Load；

Reallocation unit is used for:

Calculate the load difference of described very big hardware load and minimum hardware load；

When described load difference is more than load threshold value, then reclaim one from described very big assistance data center Or multiple data acquisition session, and the data acquisition session of recovery is reallocated to minimum assistance data center.

Described reallocation unit is additionally operable to:

One or more adjustment number of times data less than frequency threshold value are reclaimed from described very big assistance data center Acquisition tasks, using this data acquisition session as task to be allocated；

Described task to be allocated is reallocated to minimum assistance data center.

The present invention is by being averagely allocated to data acquisition session in each assistance data in management data center Before the heart, between management data center and remote terminal, carry out bidirectional identification verification, in management data center After determining remote terminal legal identity, just allow to receive the target data that remote terminal sends, the most remote Journey terminal, after the legal identity confirming management data center, just allows to send target data, thus prevents The individual of illegal identity or equipment pretend to be management data center to obtain gathering the authority of target data, and The individual or the equipment that prevent illegal identity pretend to be remote terminal to obtain pushing number of targets to management data center According to authority, thus ensure manage data center gather target data source reliability and integrity, also Ensure that remote terminal sends the legitimacy of target data objects, thus occur when avoiding gathering target data Target data mistake is sent to illegal terminal or gathers the phenomenon of illegitimate target data, improves the letter of target data Breath safety.

Accompanying drawing explanation

Fig. 1 is the schematic flow sheet of the collecting method first embodiment of distributed data system of the present invention；

Fig. 2 is the schematic flow sheet of collecting method second embodiment of distributed data system of the present invention；

Fig. 3 is the schematic flow sheet of collecting method the 3rd embodiment of distributed data system of the present invention；

Fig. 4 be distributed data system of the present invention collecting method the 3rd embodiment preferred version in new Management data center the data center broken down is repaired, and safeguard end outgoing event to default The refinement schematic flow sheet of the step of prompting；

Fig. 5 be distributed data system of the present invention collecting method the 4th embodiment in manage in data The heart, according to reference to hardware load, dynamically adjusts the data acquisition session that each assistance data center performs, with Equalize the refinement schematic flow sheet of the step of the hardware load at each assistance data center；

Fig. 6 be distributed data system of the present invention collecting method the 4th embodiment preferred version in manage Reason data center reclaims one or more data acquisition session from very big assistance data center, and by recovery Data acquisition session is reallocated to the refinement schematic flow sheet at minimum assistance data center；

If Fig. 7 is that in Fig. 6, load difference is more than load threshold value, then management data center is from very big assistance data One or more data acquisition session are reclaimed at center, and are reallocated to minimum by the data acquisition session of recovery The refinement schematic flow sheet at assistance data center；

Fig. 8 is the system framework schematic diagram of distributed data system first embodiment of the present invention；

Fig. 9 is that the functional module managing data center in distributed data system the second embodiment of the present invention is shown It is intended to；

Figure 10 is that the functional module managing data center in distributed data system the 3rd embodiment of the present invention is shown It is intended to；

Figure 11 is the merit managing data center in distributed data system the 3rd embodiment preferred version of the present invention Can module diagram；

Figure 12 is the refinement high-level schematic functional block diagram of repair module in Figure 11；

Figure 13 is the thin of dynamic adjustment module in distributed data system the 4th embodiment preferred version of the present invention Change high-level schematic functional block diagram.

The realization of the object of the invention, functional characteristics and advantage will in conjunction with the embodiments, do referring to the drawings further Explanation.

Detailed description of the invention

Should be appreciated that specific embodiment described herein, only in order to explain the present invention, is not used to limit Determine the present invention.

The present invention provides collecting method (the hereinafter referred to as data acquisition side of a kind of distributed data system Method), in the first embodiment of collecting method of the present invention, with reference to Fig. 1 and Fig. 8, distributed data System includes multiple data center being in communication with each other connection and the remote terminal 300 of each data center subordinate, Data center includes managing data center 100 and assistance data center 100, will can start data acquisition the earliest The data center of process is as management data center 100, other data center's conduct in distributed data system Assistance data center 110, this collecting method includes:

Step S10, management data center mutually shares respective key of sharing, receives remotely with remote terminal The first random number that terminal sends, and the second random number generated voluntarily is sent to remote terminal；

The communication structure between each data center in distributed data system can be star-like, net type etc., Preferably, each data center is connected with a network interconnection device 200 (such as switch, hub etc.), Each remote terminal 300 of subordinate is also connected with network interconnection device 200, and the present invention introduces data acquisition and enters Journey carries out the process of data acquisition in order to manage data center 100, and each data center runs one Data acquisition process, when the managing process of a certain data center starts, if finding, other data center does not has Data acquisition process is being run, then this data acquisition process is as master data collection process, thus master data Data center corresponding to collection process is as management data center 100, and other data center is then as supplementary number According to center 110, the data acquisition process that assistance data center 110 is run is as from data acquisition process； Management data center 100 and the major-minor relation of assistance data center 110 simply process, each data herein In the data acquisition session that processes in the heart there is no major-minor point.

Management data center 100 is internal arranges a randomizer, and in management, data center 100 needs During data acquisition session to be distributed, control randomizer and generate a random number, this random number conduct Current random number, i.e. management data center 100 is when the remote terminal 300 of its subordinate changes, and all can give birth to Become a corresponding random number.Alternatively, the randomizer preset is real random number generator, i.e. Randomizer be the random number obtained based on physical event rather than based on software algorithm obtain with Machine number.

Sharing key can be a pair RSA (RSA cryptographic algorithms) public private key pair, manages data center 100 In the initialization procedure of production process, one in above-mentioned RSA public private key pair is respectively generated with remote terminal 300 Individual and each preserve, the public affairs in public private key pair are shared mutually with remote terminal 300 by management data center 100 Key.The key of sharing of remote terminal 300 side is stored in hardware memory own, in management data The heart will be shared key and be saved in server not open area.

In management, data center 100 carries out sharing when sharing of key with remote terminal 300, manages data Center 100 generates the second random number voluntarily and is sent to remote terminal 300, and remote terminal 300 generates voluntarily First random number also is sent to manage data center 100, thus manages data center 100 and remote terminal 300 all obtain the first random number and the second random number.

Step S20, management data center is according to sharing key, the first random number, the second random number and prestoring The first signing messages, generate the first check value；

Step S30, the first check value is sent to remote terminal by management data center, tests for remote terminal Demonstrate,prove the first check value, and receive remote terminal according to sharing key, the first random number, the second random number The second check value with the second signing messages generation that this remote terminal prestores；

First signing messages of management data center 100 has uniqueness, and the i.e. first signing messages can be only One mark manages data center 100, and in like manner, the second signing messages can also uniquely identify remote terminal 300. Management data center 100 according to share the algorithm of key and setting to the first random number, the second random number and First signing messages is encrypted, and generates the first check value.In like manner, remote terminal 300 is close according to sharing The second verification that the second signing messages that key, the first random number, the second random number and source host prestore generates Value；Then management data center 100 sends the first check value, remote terminal 300 to remote terminal 300 Sending the second check value to management data center 100, management data center 100 and remote terminal 300 are equal First check value and the second check value are compared verification.

Step S40, when management data center verifies the first check value and the success of the second check value, and receives During the verification pass signal that remote terminal sends, collection remote terminal is uploaded number of targets by management data center According to data acquisition session be averagely allocated to each assistance data center perform, one of them remote terminal pair Answer a data acquisition session.

When managing data center 100 side and detecting that the first check value and the second check value verify successfully, pipe Reason data center 100 sends verification pass signal to remote terminal 300, in like manner, when remote terminal 300 Side detects when the first check value and the second check value verify successfully, and remote terminal 300 is in management data The heart 100 sends verification pass signal.When the first check value and the second check value verify successfully, and management number When receiving, according to center 100, the verification pass signal that remote terminal 300 sends, manage data center 100 Side judges that between management data center 100 and remote terminal 300, bidirectional identification verifies successfully, then manage number According to center 100, the data acquisition session that target data uploaded by collection remote terminal 300 is averagely allocated to respectively Individual assistance data center 110 performs.

Especially it is emphasized that management data center 100 all carries out bidirectional identification with each remote terminal 300 Checking, has only passed through and has managed the remote terminal 300 of bidirectional identification checking between data center 100 (to set For reliable remote terminal) it is only management data center 100 and gathers the object of target data, i.e. manage data 100, center gathers the target data of reliable remote terminal, an only reliable remote terminal just corresponding number According to acquisition tasks.

Corresponding 300 1 data acquisition session of a remote terminal of management data center 100, will gather each Remote terminal 300 is uploaded the data acquisition session of data and is averagely allocated to each assistance data center 110 and holds OK, mean allocation herein is it is to be understood that distribute average number for each assistance data center 110 According to acquisition tasks, wherein average is that data acquisition session sum is divided by data center's sum；Certainly, flat All distribution can also be alternate manner, as long as substantially realizing being data acquisition session each supplementary number According to center 110, the most one by one data acquisition session is assigned randomly to each assistance data center 110.

In the present embodiment, by data acquisition session being averagely allocated to respectively in management data center 100 Before individual assistance data center 110, carry out two-way between management data center 100 and remote terminal 300 Proof of identity, after management data center 100 determines remote terminal 300 legal identity, just allows to connect Receiving the target data that remote terminal 300 sends, remote terminal 300 is confirming management data center 100 simultaneously Legal identity after, just allow to send target data, thus prevent the individual of illegal identity or equipment from pretending to be Manage data center and obtain gathering the authority of target data, and prevent individual or the equipment of illegal identity Pretend to be remote terminal to obtain pushing to management data center the authority of target data, thus ensure to manage number Gather reliability and the integrity in target data source according to center 100, also ensure that remote terminal 300 Send the legitimacy of target data objects, thus avoid occurring that target data mistake is sent to non-when gathering target data Method terminal or the phenomenon of collection illegitimate target data, improve the information security of target data.

Further, on the basis of collecting method first embodiment of the present invention, data acquisition is proposed Method the second embodiment, with reference to Fig. 2, in a second embodiment, the most also includes:

Step S50, management data center obtains once each assistance data center processing at interval of preset duration The hardware load of data acquisition session, using this hardware load as with reference to hardware load；

Step S60, management data center, according to reference to hardware load, dynamically adjusts each assistance data center The data acquisition session performed, to equalize the hardware load at each assistance data center.

Data center, before performing data acquisition session, is to know which data acquisition session ratio Relatively time consuming, which data acquisition session is fairly simple, and the hardware processing capability of each data center is also It is not quite similar, so each data center (including managing data center 100 and Added Management center) is because of place Hardware load produced by reason data acquisition session can not persistently be kept in balance, so needing dynamic Adjust.

Hardware load can include in cpu load, Memory Load etc. one or more, prioritizing selection CPU Load is as hardware load, and data center is the longest to the process time of data acquisition session, then hardware load The highest, management data center 100 obtains the most each supplementary number at interval of preset duration (such as 3 minutes) According to the hardware load (i.e. with reference to hardware load) at center 110, then management data center 100 is according to each The reference hardware load at assistance data center 110, dynamically adjusts what each assistance data center 110 performed Data acquisition session, wherein the data acquisition session at the assistance data center 110 that hardware load is high is to hardware The assistance data center 110 that load is low is shifted.

In the present embodiment, by management data center 100 data acquisition session is evenly distributed to auxiliary After helping data center 110, then obtain the reference hardware load at each assistance data center 110, finally According to reference to hardware load, dynamically adjust the data acquisition session at each assistance data center 110, with reality The dynamic load balancing at each assistance data center 110 existing, thus make full use of the hard of multiple data center Part resource and the network bandwidth, process the data acquisition session of multiple remote terminal 300 concurrently, thus add The fast processing speed of data acquisition session, data center can gather required target data in time, keep away Exempt from the target data because target data collection is caused not in time and gather incomplete problem.

Further, on the basis of collecting method the second embodiment of the present invention, data acquisition is proposed Method the 3rd embodiment, with reference to Fig. 3, in the third embodiment, collecting method also includes:

Step S70, when managing data center and breaking down, extracts during current management data center is run raw The replacement person's information become, and using assistance data center corresponding for replacement person's information as in new management data The heart.

In data center by during as management data center 100, this management data center 100 preserves Replacement person's information (i.e. the information at replacement data center), when managing data center 100 and breaking down, directly Connect the replacement person's information extracted in current management data center 100, and by auxiliary corresponding for replacement person's information Data center 110 is as new management data center 100, it is to avoid because of management, data center 100 breaks down, Two management data centers 100 occur or affects data acquisition appoint without the situation of management data center 100 The execution of business and distribution.Additionally, it should be noted that, the management data center 100 broken down is being taken The data acquisition session oneself processed, new management data can be discharged when disappearing as management data center 100 The data acquisition session that the release of last management data center 100 is reclaimed at center 100 is auxiliary to be again assigned to Help data center 110.

Preferably, on the basis of collecting method the 3rd embodiment of the present invention, after step S70 also Including:

Step S80, the data center broken down is repaired by new management data center, and to presetting Safeguard that end outgoing event is pointed out.

The new data center to breaking down of the management data center 100 is (in the management data of the most previous The heart) to repair, this reparation includes software reparation and/or hardware reparation, and to default maintenance end output Event prompt, to inform system maintenance personnel's failure condition

Specifically, with reference to Fig. 4, step S80 includes:

Step S81, new management data center uses the correct procedure content the prestored data to breaking down The contents of program at center carries out data reparation；

Step S82, if the data center being not detected by breaking down is sent out based on the program after carrying out data reparation The response message sent, then reset and restart the data center broken down；

Step S83, if the response message that the data center broken down after restarting that resets sends being detected, Then judge that the data center broken down occurs software failure, export the first event prompt；

Step S84, if the response that the data center broken down after restarting that is not detected by resetting sends disappears Breath, then judge that the data center broken down occurs hardware failure, output second event prompting.

The contents of program of data center is the most changeless, is determining that management data center breaks down Afterwards, the contents of program using the correct procedure content the prestored data center to breaking down carries out data Repair, if the data center broken down is after carrying out data reparation based on correct procedure content, it is possible to Current management data center sends response message, i.e. management data center and the data broken down detected The response message that center sends based on the program after carrying out data reparation, then judge in the data broken down Heart generation software failure, exports the first event prompt to maintenance terminal, and this first event prompt is for dimension Protecting the data center that terminal report breaks down is that program software lost efficacy, and prompting maintenance personnel note generation The data center of fault carries out the maintenance of software aspects, thus provides accurate fault for attendant Testing result, and automatically solve the Problem of Failure of the data center's software aspects broken down.

After the data center broken down is carried out software data reparation, if being still not detected by not examining Measure the response message that the data center broken down sends based on the program after carrying out data reparation, now Can speculate that problem on hardware occurs in the data center broken down, then reset and restart fail data node； If the response message that the data center broken down after restarting that resets sends being detected, show to break down Data center's in house software program the most out of joint, then judge that the data center broken down is sent out Raw software failure, exports the first event prompt；If to be still not detected by reset restart after break down The response message that data center sends, i.e. carries out data reparation and the weight that resets to the data center broken down After opening, the data center broken down still can not normally send response message, then judge to break down Data center there is hardware failure, output second event prompting, the prompting of this second event is for maintenance The data center that terminal report breaks down is that hardware of memory device lost efficacy, and prompting maintenance personnel note sending out The data center of raw fault carries out the maintenance of hardware aspect, thus provides accurate event for attendant Barrier testing result, in order to attendant prepares the instrument needed for data center's hardware maintenance and consumptive material in time.

Further, on the basis of collecting method the 3rd embodiment of the present invention, data acquisition is proposed Method the 4th embodiment, with reference to Fig. 5, in the fourth embodiment, step S60 includes:

Step S61, management data center according to reference to hardware load, selects the highest the most auxiliary of hardware load Help data center and the minimum minimum assistance data center of hardware load；

Step S62, management data center reclaims one or more data acquisitions and appoints from very big assistance data center Business, and the data acquisition session of recovery is reallocated to minimum assistance data center.

The hardware load at management each assistance data center 110 of data center 100 comparison, selects hardware and bears Very big assistance data center 110 that lotus is the highest and the minimum minimum assistance data center 110 of hardware load, pipe Reason data center 100 reclaims one or more data acquisition session from very big assistance data center 110, returns The data acquisition session number received determines according to the highest hardware load and minimal hardware load；Then number is managed According to center 100, the data acquisition session of recovery is being distributed to minimum assistance data center 110, thus avoiding Very big assistance data center 110 hardware load is too high and causes processing speed to be crossed slowly, even crash, simultaneously Avoid minimum assistance data center 110 hardware load too small and cause the waste of hardware resource.

Preferably, on the basis of collecting method the 4th embodiment of the present invention, with reference to Fig. 4, by pole The hardware load at big assistance data center 110 is as very big hardware load, by minimum assistance data center 110 Hardware load as minimum hardware load, with reference to Fig. 6,

Step S62 includes:

Step S621, management data center calculates the load difference of very big hardware load and minimum hardware load；

Step S622, if load difference is more than load threshold value, then management data center is from very big assistance data One or more data acquisition session are reclaimed at center, and are reallocated to minimum by the data acquisition session of recovery Assistance data center.

Management data center 100 calculates the load difference of very big hardware load and minimum hardware load, the most negative Lotus difference=very big hardware load-minimum hardware load, presets load threshold value, compares load difference and load threshold Value, if load difference is more than load threshold value, shows hardware load and the pole at very big assistance data center 110 The hardware load difference at little assistance data center 110 is relatively big, and hardware load uneven phenomenon is more serious, institute With management, data center 100 reclaims one or more data acquisition session from very big assistance data center 110, And the data acquisition session of recovery is reallocated to minimum assistance data center 110 with relax hardware load divide The problem that cloth is uneven；If load difference is less than load threshold value, show the hard of very big assistance data center 110 Part load differs less with the hardware load at minimum assistance data center 110, and hardware load distribution is the most equal Weighing apparatus, it is not necessary to carry out hardware load adjustment, thus only when hardware load skewness is more serious, just enter The adjustment of the data acquisition session that row is corresponding, improves accuracy and promptness that data acquisition session adjusts.

Further, on the basis of collecting method the 4th embodiment preferred version of the present invention, propose Collecting method the 5th embodiment, with reference to Fig. 7, in the 5th embodiment, management data center 100 unites Count each data acquisition session experience to reclaim and the adjustment number of times of reallocation,

Step S622 includes:

Step S6221, if load difference is more than load threshold value, then management data center is from very big assistance data One or more adjustment number of times data acquisition session less than frequency threshold value is reclaimed at center, by this data acquisition Task is as task to be allocated；

Step S6222, task to be allocated is reallocated to minimum assistance data center by management data center.

Give each data acquisition session one in management data center 100 and adjust number of times statistical value, When data acquisition session is once reclaimed and reallocates, the adjustment number of times of this data acquisition session is added One, i.e. one data acquisition session is recovered and reallocates once that it adjusts number of times from adding one.

In the present embodiment, when managing data center 100 data collection acquisition tasks, adjustment is only reclaimed Number of times is less than the data acquisition session (task the most to be allocated) of frequency threshold value, then by task to be allocated again It is assigned to minimum assistance data center 110, i.e. the adjustment number of times at a certain data acquisition session and is more than number of times threshold During value, then this data acquisition session will not be recovered reallocation again, until corresponding assistance data center 110 Perform this data acquisition session, thus avoid a data acquisition session repeatedly at assistance data center 110 Between reclaim and reallocate, and cannot be performed in time.

The present invention also provides for a kind of distributed data system, in distributed data system first embodiment, With reference to Fig. 8, distributed data system includes multiple data centers being in communication with each other connection and each data center The remote terminal 300 of subordinate, data center includes managing data center 100 and assistance data center 100, Can be using the data center starting data acquisition process the earliest as managing data center 100, distributed data system In system, other data center is as assistance data center 110,

Management data center 100 includes:

Sharing module 10, for mutually sharing respective key of sharing with remote terminal, receives remote terminal The first random number sent, and the second random number generated voluntarily is sent to remote terminal；

In management, data center 100 carries out sharing when sharing of key, sharing module with remote terminal 300 10 generate the second random number voluntarily and are sent to remote terminal 300, and remote terminal 300 generates first voluntarily Random number is also sent to manage data center 100, thus manages data center 100 and remote terminal 300 is equal Obtain the first random number and the second random number.

Generation module 20, for according to sharing key, the first random number, the second random number and prestore the One signing messages, generates the first check value；

Receiver module 30, for sending the first check value to remote terminal, for remote terminal checking the One check value, and receive remote terminal according to sharing key, the first random number, the second random number and being somebody's turn to do The second check value that the second signing messages that remote terminal prestores generates；

First signing messages of management data center 100 has uniqueness, and the i.e. first signing messages can be only One mark manages data center 100, and in like manner, the second signing messages can also uniquely identify remote terminal 300. The generation module 20 of management data center 100 according to share the algorithm of key and setting to the first random number, Second random number and the first signing messages are encrypted, and generate the first check value.In like manner, remote terminal 300 According to sharing the second signing messages generation that key, the first random number, the second random number and source host prestore The second check value；Then the receiver module 30 of management data center 100 sends the to remote terminal 300 One check value, remote terminal 300 sends the second check value to management data center 100, in management data First check value and the second check value are all compared by the receiver module 30 of the heart 100 and remote terminal 300 Verification.

The equal sub-module of task 40, for working as verification the first check value and the success of the second check value, and receives During the verification pass signal that remote terminal sends, collection remote terminal is uploaded the data acquisition of target data Task is averagely allocated to each assistance data center and performs, the corresponding data acquisition of one of them remote terminal Set task.

When managing data center 100 side and detecting that the first check value and the second check value verify successfully, pipe The equal sub-module of task 40 of reason data center 100 sends to remote terminal 300 and verifies pass signal, in like manner, When remote terminal 300 side detects that the first check value and the second check value verify successfully, remote terminal 300 Verification pass signal is sent to management data center 100.When the first check value and the second check value verify into Merit, and when management data center 100 receives the verification pass signal that remote terminal 300 sends, management Data center 100 side judges that between management data center 100 and remote terminal 300, bidirectional identification verifies into Merit, then collection remote terminal 300 is uploaded number of targets by the equal sub-module of task 40 of management data center 100 According to data acquisition session be averagely allocated to each assistance data center 110 and perform.

Corresponding 300 1 data acquisition session of a remote terminal of management data center 100, task is divided equally The data acquisition session that data uploaded by each remote terminal 300 of collection is averagely allocated to each auxiliary by module 40 Data center 110 performs, and mean allocation herein is it is to be understood that be each assistance data center 110 Distribution average data acquisition session, wherein average is that data acquisition session sum is divided by data center Sum；Certainly, mean allocation can also be alternate manner, as long as substantially realizing data acquisition session equal For to each assistance data center 110, the most one by one data acquisition session being assigned randomly to each Assistance data center 110.

In the present embodiment, data acquisition session is averagely allocated to each by dividing equally module 40 in task Before assistance data center 110, between management data center 100 and remote terminal 300, carry out two-way body Part verification, after management data center 100 determines remote terminal 300 legal identity, just allows to receive The target data that remote terminal 300 sends, remote terminal 300 is confirming management data center 100 simultaneously Legal identity after, just allow to send target data, thus prevent the individual of illegal identity or equipment from pretending to be Manage data center and obtain gathering the authority of target data, and prevent individual or the equipment of illegal identity Pretend to be remote terminal to obtain pushing to management data center the authority of target data, thus ensure to manage number Gather reliability and the integrity in target data source according to center 100, also ensure that remote terminal 300 Send the legitimacy of target data objects, thus avoid occurring that target data mistake is sent to non-when gathering target data Method terminal or the phenomenon of collection illegitimate target data, improve the information security of target data.

Further, on the basis of distributed data system first embodiment of the present invention, propose distributed Data system the second embodiment, with reference to Fig. 9, in a second embodiment, management data center also includes:

Load obtaining module 50, for obtaining once each assistance data center processing at interval of preset duration The hardware load of data acquisition session, using this hardware load as with reference to hardware load；

Dynamically adjustment module 60, for according to reference to hardware load, dynamically adjusting each assistance data center The data acquisition session performed, to equalize the hardware load at each assistance data center.

Hardware load can include in cpu load, Memory Load etc. one or more, prioritizing selection CPU Load is as hardware load, and data center is the longest to the process time of data acquisition session, then hardware load The highest, Load obtaining module 50 obtains the most each assistance data at interval of preset duration (such as 3 minutes) The hardware load (i.e. with reference to hardware load) at center 110, then dynamically adjustment module 60 is auxiliary according to each Help the reference hardware load of data center 110, dynamically adjust the number that each assistance data center 110 performs According to acquisition tasks, wherein the data acquisition session at the assistance data center 110 that hardware load is high is born to hardware The assistance data center 110 that lotus is low is shifted.

Assistance data center 110, sends currently to current management data center 100 at interval of preset duration Process the hardware load of data acquisition session.

In the present embodiment, first data acquisition session is evenly distributed to auxiliary by the equal sub-module of task 40 Data center 110, then Load obtaining module 50 obtains the reference hardware at each assistance data center 110 Load, last dynamically adjustment module 60, according to reference to hardware load, dynamically adjusts each assistance data center The data acquisition session of 110, to realize the dynamic load balancing at each assistance data center 110, thus fills Divide hardware resource and the network bandwidth utilizing multiple data center, process multiple remote terminal 300 concurrently Data acquisition session, thus accelerate the processing speed of data acquisition session, data center can be timely Target data needed for collection, it is to avoid the target data caused not in time because of target data collection gathers not Complete problem.

Further, on the basis of distributed data system the second embodiment of the present invention, propose distributed Data system the 3rd embodiment, with reference to Figure 10, in the third embodiment, with reference to Fig. 7, in management data The heart 100 also includes:

Replacement module 70, for breaking down when management data center, extracts current management data center fortune The replacement person's information generated in row, and using assistance data center corresponding for replacement person's information as new management Data center.

In data center by during as management data center 100, this management data center 100 preserves Replacement person's information (i.e. the information at replacement data center), when managing data center 100 and breaking down, replaces Replacement person's information in die change block 70 extracting directly current management data center 100, and by replacement person's information Corresponding assistance data center 110 is as new management data center 100, it is to avoid because of management data center 100 break down, occur two management data centers 100 or without the management situation of data center 100 shadow Ring execution and the distribution of data acquisition session.Additionally, it should be noted that, in the management data broken down The heart 100 can discharge, when being cancelled as management data center 100, the data acquisition session oneself processed, New management data center 100 reclaim the data acquisition session of last management data center 100 release with It is again assigned to assistance data center 110.

Preferably, with reference to Figure 11, on the basis of distributed data system the 3rd embodiment of the present invention, pipe Reason data center 100 also includes also including:

To default, repair module 80, for repairing the data center broken down, and safeguards that end is defeated Outgoing event is pointed out.

The repair module 80 of new management data center 100 data center (the most previous to breaking down Management data center) repair, this reparation includes software reparation and/or hardware reparation, and to presetting Maintenance end outgoing event prompting, to inform system maintenance personnel's failure condition.

Specifically, with reference to Figure 12, repair module 80 includes:

Data repair unit 81, for when becoming new management data center, use the correct journey prestored The contents of program of the sequence content data center to breaking down carries out data reparation；

Unit 82 repaired by hardware, for repairing based on carrying out data in the data center being not detected by breaking down During the response message that the program after Fu sends, reset and restart the data center broken down；

First Tip element 83, for detecting that the data center broken down after restarting that resets sends Response message time, it is determined that there is software failure in the data center that breaks down, exports the first event prompt；

Second Tip element 84, for sending out in the data center broken down after restarting that is not detected by resetting During the response message sent, it is determined that the data center broken down occurs hardware failure, output second event to carry Show.

The contents of program of data center is the most changeless, is determining that management data center breaks down Afterwards, data are repaired unit 81 and are used the journey of the correct procedure content the prestored data center to breaking down Sequence content carries out data reparation, if the data center broken down is carrying out data based on correct procedure content After reparation, it is possible to send response message, i.e. management data center to current management data center and detect The response message that the data center broken down sends based on the program after carrying out data reparation, then first carry Show that unit 83 judges that the data center broken down occurs software failure, export the first event to maintenance terminal Prompting, this first event prompt is program software for the data center broken down to maintenance terminal report Losing efficacy, prompting maintenance personnel note carrying out the data center broken down the maintenance of software aspects, thus There is provided accurate failure detection result for attendant, and automatically solve in the data that break down The Problem of Failure of heart software aspects.

After the data center broken down is carried out software data reparation, if being still not detected by not examining Measure the response message that the data center broken down sends based on the program after carrying out data reparation, now Can speculate that problem on hardware occurs in the data center broken down, then hardware reparation unit 82 resets and restarts Fail data node；If the response message that the data center broken down after restarting that resets sends being detected, Show that the data center's in house software program broken down is the most out of joint, then the first Tip element 83 judge that the data center broken down occurs software failure, export the first event prompt；If still not examining Measure the response message that the data center broken down after restarting that resets sends, i.e. to the number broken down Carrying out data reparation according to center and after reset restarts, the data center broken down still can not normally send out Send response message, then the second Tip element 84 judges that the data center broken down occurs hardware failure, defeated Going out second event prompting, the prompting of this second event for reporting the data center broken down to maintenance terminal Being that hardware of memory device lost efficacy, prompting maintenance personnel note the data center broken down is carried out hardware side The maintenance in face, thus provide accurate failure detection result for attendant, in order to attendant and Time prepare the instrument needed for data center hardware maintenance and consumptive material.

Further, on the basis of distributed data system the 3rd embodiment of the present invention, propose distributed Data system the 4th embodiment, in the fourth embodiment, with reference to Figure 13, dynamic adjustment module 60 includes:

Screening unit 61, for according to reference to hardware load, selecting the very big supplementary number that hardware load is the highest According to the minimum assistance data center that center and hardware load are minimum；

Reallocation unit 62, for reclaiming one or more data acquisition session from very big assistance data center, And the data acquisition session of recovery is reallocated to minimum assistance data center.

The hardware load at each assistance data center 110 of screening unit 61 comparison, selects hardware load the highest Very big assistance data center 110 and the minimum minimum assistance data center 110 of hardware load, reallocation is single Unit 62 reclaims one or more data acquisition session from very big assistance data center 110, the data acquisition of recovery Set task number determines according to the highest hardware load and minimal hardware load；Then reallocation unit 62 is incited somebody to action back The data acquisition session received is distributing to minimum assistance data center 110, thus avoids in very big assistance data The heart 110 hardware load is too high and causes processing speed to be crossed slowly, even crash, and avoids minimum supplementary number simultaneously Too small and cause the waste of hardware resource according to center 110 hardware load.

Preferably, on the basis of distributed data system the 4th embodiment of the present invention, by very big supplementary number According to the hardware load at center 110 as very big hardware load, by the hardware at minimum assistance data center 110 Load as minimum hardware load,

Reallocation unit 62 is used for:

Calculate the load difference of very big hardware load and minimum hardware load；

When load difference is more than load threshold value, then reclaim one or many from very big assistance data center 110 Individual data acquisition session, and the data acquisition session of recovery is reallocated to minimum assistance data center 110.

Reallocation unit 62 calculates the load difference of very big hardware load and minimum hardware load, i.e. load difference Value=very big minimum hardware load of hardware load, presets load threshold value, compares load difference and load threshold Value, if load difference is more than load threshold value, shows hardware load and the pole at very big assistance data center 110 The hardware load difference at little assistance data center 110 is relatively big, and hardware load uneven phenomenon is more serious, institute One or more data acquisition session is reclaimed from very big assistance data center 110 with reallocation unit 62, and The data acquisition session of recovery is reallocated to minimum assistance data center 110 to relax hardware load distribution Uneven problem；If load difference is less than load threshold value, show the hardware at very big assistance data center 110 Load differs less with the hardware load at minimum assistance data center 110, and hardware load distribution more equalizes, Without carrying out hardware load adjustment, thus only when hardware load skewness is more serious, it is right just to carry out The adjustment of the data acquisition session answered, improves accuracy and promptness that data acquisition session adjusts.

Further, on the basis of distributed data system the 4th embodiment of the present invention is preferred, propose to divide Cloth data system the 5th embodiment, in the 5th embodiment, management data center 100 adds up every number Reclaim and the adjustment number of times of reallocation according to acquisition tasks experience,

Reallocation unit 62 is additionally operable to:

The data acquisition less than frequency threshold value of one or more adjustment number of times is reclaimed from very big assistance data center Task, using this data acquisition session as task to be allocated；

Task to be allocated is reallocated to minimum assistance data center.

In the present embodiment, when managing data center 100 data collection acquisition tasks, unit of reallocating 62 only reclaim the data acquisition session (task the most to be allocated) adjusting number of times less than frequency threshold value, then will treat Distribution task is assigned to minimum assistance data center 110 again, i.e. in the adjustment time of a certain data acquisition session When number is more than frequency threshold value, then this data acquisition session will not be recovered reallocation again, until corresponding auxiliary Data center 110 has performed this data acquisition session, thus avoids a data acquisition session repeatedly auxiliary Help and reclaim between data center 110 and reallocate, and cannot be performed in time.

Through the above description of the embodiments, those skilled in the art is it can be understood that arrive above-mentioned Embodiment method can add the mode of required general hardware platform by software and realize, naturally it is also possible to logical Cross hardware, but a lot of in the case of the former is more preferably embodiment.Based on such understanding, the present invention's The part that prior art is contributed by technical scheme the most in other words can be with the form body of software product Revealing to come, this computer software product is stored in a storage medium (such as ROM/RAM, magnetic disc, light Dish) in, including some instructions with so that a station terminal equipment (can be mobile phone, computer, service Device, air-conditioner, or the network equipment etc.) perform each embodiment of the present invention method.

These are only the preferred embodiments of the present invention, not thereby limit the scope of the claims of the present invention, every Utilize equivalent structure or equivalence flow process conversion that description of the invention and accompanying drawing content made, or directly or Connect and be used in other relevant technical fields, be the most in like manner included in the scope of patent protection of the present invention.

Claims

1. the collecting method of a distributed data system, it is characterized in that, described distributed data system includes multiple data center being in communication with each other connection and the remote terminal of each data center subordinate, and data center includes managing data center and assistance data center

The collecting method of this distributed data system includes:

Respective key of sharing mutually is shared with described remote terminal by management data center, receives the first random number that remote terminal sends, and the second random number generated voluntarily is sent to remote terminal；

Management data center, according to sharing key, the first random number, the second random number and the first signing messages prestored, generates the first check value；

First check value is sent to remote terminal by management data center, verify the first check value for remote terminal, and receive remote terminal according to the second check value sharing the second signing messages generation that key, the first random number, the second random number and this remote terminal prestore；

When management data center verifies the first check value and the success of the second check value, and when receiving the verification pass signal of remote terminal transmission, collection remote terminal is uploaded the data acquisition session of target data and is averagely allocated to the execution of each assistance data center by management data center, the corresponding data acquisition session of one of them remote terminal.

The most such asClaimThe collecting method of the distributed data system described in 1, it is characterised in that described management data center uploads after the data acquisition session of target data is averagely allocated to the step that each assistance data center performs also include gathering remote terminal:

Management data center obtains once the hardware load of each assistance data center processing data acquisition session at interval of preset duration, using this hardware load as with reference to hardware load；

Management data center with reference to hardware load, dynamically adjusts the data acquisition session that each assistance data center performs, to equalize the hardware load at each assistance data center according to described.

The most such asClaimThe collecting method of the distributed data system described in 2, it is characterised in that the collecting method of described distributed data system also includes:

When managing data center and breaking down, extract during current management data center is run the replacement person's information generated, and using assistance data center corresponding for replacement person's information as new management data center.

The most such asClaimThe collecting method of the distributed data system described in 2 or 3, it is characterised in that described management data center is according to described with reference to hardware load, and the step dynamically adjusting the data acquisition session that each assistance data center performs includes:

Management data center is according to described reference hardware load, select the highest very big assistance data center of hardware load and the minimum minimum assistance data center of hardware load, and using the hardware load at very big assistance data center as very big hardware load, using the hardware load at minimum assistance data center as minimum hardware load

If described load difference is more than load threshold value, then management data center reclaims one or more data acquisition session from described very big assistance data center, and the data acquisition session of recovery is reallocated to minimum assistance data center.

The most such asClaimThe collecting method of the distributed data system described in 4, it is characterised in that management data center adds up each data acquisition session experience and reclaims and the adjustment number of times of reallocation,

Described management data center reclaims one or more data acquisition session from described very big assistance data center, and the step that the data acquisition session of recovery is reallocated to minimum assistance data center includes:

Management data center reclaims one or more adjustment number of times data acquisition session less than frequency threshold value from described very big assistance data center, using this data acquisition session as task to be allocated；

6. a distributed data system, it is characterised in that described distributed data system includes multiple data center being in communication with each other connection and the remote terminal of each data center subordinate, data center includes managing data center and assistance data center,

Described management data center includes:

Sharing module, for mutually sharing respective key of sharing with described remote terminal, receives the first random number that remote terminal sends, and the second random number generated voluntarily is sent to remote terminal；

Generation module, for according to sharing key, the first random number, the second random number and the first signing messages prestored, generating the first check value；

Receiver module, for the first check value is sent to remote terminal, verify the first check value for remote terminal, and receive remote terminal according to the second check value sharing the second signing messages generation that key, the first random number, the second random number and this remote terminal prestore；

The equal sub-module of task, for when verification the first check value and the success of the second check value, and when receiving the verification pass signal of remote terminal transmission, the data acquisition session that collection remote terminal is uploaded target data is averagely allocated to the execution of each assistance data center, the corresponding data acquisition session of one of them remote terminal.

The most such asClaimDistributed data system described in 6, it is characterised in that described management data center also includes

Load obtaining module, for obtaining once the hardware load of each assistance data center processing data acquisition session, using this hardware load as with reference to hardware load at interval of preset duration；

Dynamically adjustment module, for according to described reference hardware load, dynamically adjusting the data acquisition session that each assistance data center performs, to equalize the hardware load at each assistance data center；

Described assistance data center sends the hardware load of currently processed data acquisition session at interval of preset duration to current management data center.

The most such asClaimDistributed data system described in 7, it is characterised in that described management data center also includes:

Replacement module, for when managing data center and breaking down, extracts during current management data center is run the replacement person's information generated, and using assistance data center corresponding for replacement person's information as new management data center.

The most such asClaimDistributed data system described in 7 or 8, it is characterised in that described dynamic adjustment module includes:

Screening unit, for according to described reference hardware load, select the highest very big assistance data center of hardware load and the minimum minimum assistance data center of hardware load, wherein using the hardware load at very big assistance data center as very big hardware load, using the hardware load at minimum assistance data center as minimum hardware load；

Reallocation unit is used for:

When described load difference is more than load threshold value, then reclaims one or more data acquisition session from described very big assistance data center, and the data acquisition session of recovery is reallocated to minimum assistance data center.

The most such asClaimDistributed data system described in 9, it is characterised in that management data center adds up each data acquisition session experience and reclaims and the adjustment number of times of reallocation,

Described reallocation unit is additionally operable to:

One or more adjustment number of times data acquisition session less than frequency threshold value is reclaimed, using this data acquisition session as task to be allocated from described very big assistance data center；