CN104794155A

CN104794155A - Data loading method, device and system

Info

Publication number: CN104794155A
Application number: CN201510109465.5A
Authority: CN
Inventors: 曹柏余; 王颖; 曹雪林; 邓宇; 洪春晓
Original assignee: Beijing Qihoo Technology Co Ltd; Qizhi Software Beijing Co Ltd
Current assignee: Beijing Qihoo Technology Co Ltd
Priority date: 2015-03-12
Filing date: 2015-03-12
Publication date: 2015-07-22
Anticipated expiration: 2035-03-12
Also published as: CN104794155B

Abstract

The invention discloses a data loading method, device and system, and relates to the technical field of the internet. The problem that the computer room striding data loading efficiency is low is solved. The method comprises the steps that block data in a transferring device are sent to a main computer room, the main computer room comprises a plurality of sets of loading machines, the number of the sets of the loading machines is equal to the number of auxiliary computer rooms, and the number of the loading machines in each set is equal to the number of secondary servers in the corresponding auxiliary computer room; the block data are written in the corresponding sets of loading machines through the main computer room respectively; the block data in the loading machines are written in the secondary servers corresponding to the loading machines in the auxiliary computer rooms. The data loading method, device and system are mainly applied to computer room striding updating of mass data.

Description

The method of Data import, Apparatus and system

Technical field

The present invention relates to Internet technical field, particularly relate to a kind of method of Data import, Apparatus and system.

Background technology

Based on the development of internet cloud, large data grows receives publicity.Along with the increase of data mining dimension, the increase of covering customer volume, excavate valuable data volume also in continuous increase.In real process, in the face of the valuable data of magnanimity, in order to improve the experience of user and meet the performance of system, data can be deployed to respectively in multiple machine room or multiple database, for bearing the query demand of different user.Such as, usually mass data can be stored in outside database (as MySQL, Hbase), and timing upgrades.

In prior art, for the scene of multimachine room Data import, need data block to be loaded to be landed in the disk of springboard machine, and by springboard machine, the data block of disk is loaded in each machine room respectively.

In the process of above-mentioned Data import, inventor finds: the data load process of springboard machine to multiple machine room is serial implementation, and namely springboard machine carries out Data import to another machine room again after a machine room loads data.When in practical application, machine room quantity is more, because network input and output (Input/Output the is called for short network I/O) interface resource between springboard machine and machine room is limited, therefore, data load process is consuming time longer, the operational efficiency of influential system.

Summary of the invention

In view of this, the present invention proposes a kind of method of Data import, Apparatus and system, fundamental purpose is to solve across the low problem of machine room data loading efficiency.

According to first aspect of the present invention, the invention provides a kind of method of Data import, the method comprises:

Be sent in main equipment room by the data block in transferring equipment, comprise many groups and add carrier aircraft in main equipment room, wherein, the group number adding carrier aircraft is identical with the number from machine room, and often to add the quantity of carrier aircraft identical from the secondary server quantity machine room with corresponding for group;

What data block is written to respectively each group by main equipment room eachly adds in carrier aircraft;

The data block added in carrier aircraft is written in secondary server corresponding with adding carrier aircraft from machine room.

According to second aspect of the present invention, the invention provides a kind of method of Data import, the method, mainly for main equipment room side, comprising:

Main equipment room control server receives the data block that transferring equipment sends;

By the data block received, each group of being written to respectively in main equipment room adds in carrier aircraft main equipment room control server, wherein, comprise many groups in main equipment room and add carrier aircraft, the group number adding carrier aircraft is identical with the number from machine room, and often to add the quantity of carrier aircraft identical from the secondary server quantity machine room with corresponding for group;

Data block is written in secondary server corresponding with adding carrier aircraft from machine room by adding carrier aircraft by main equipment room control server.

According to the 3rd aspect of the present invention, the invention provides a kind of device of Data import, this device can be arranged in main equipment room, comprising:

Receiving element, for receiving the data block that transferring equipment sends;

Writing unit, data block for being received by receiving element each group of being written to respectively in main equipment room adds in carrier aircraft, wherein, comprise many groups in main equipment room and add carrier aircraft, the group number adding carrier aircraft is identical with the number from machine room, and often to add the quantity of carrier aircraft identical from the secondary server quantity machine room with corresponding for group;

Control module, is written to the data block that writing unit writes in secondary server corresponding with adding carrier aircraft from machine room for controlled loading machine.

According to the 4th aspect of the present invention, the invention provides a kind of system of Data import, this system comprises:

Transferring equipment, main equipment room and from machine room, wherein comprise main equipment room control server in main equipment room and many groups add carrier aircraft, from machine room, comprise secondary server, the group number adding carrier aircraft is identical with the number from machine room, and often to add the quantity of carrier aircraft identical from the secondary server quantity machine room with corresponding for group;

Transferring equipment, for being sent to main equipment room control server by data block;

Main equipment room control server, what the data block for transferring equipment being sent was written to each group respectively eachly adds in carrier aircraft;

Add carrier aircraft, the data block for main equipment room control server being write is written in secondary server corresponding with adding carrier aircraft from machine room.

By technique scheme, the method of the Data import that the embodiment of the present invention provides, Apparatus and system, data block in transferring equipment can be sent in main equipment room, by main equipment room, by data block, each group of being written to respectively in main equipment room adds in carrier aircraft, then adds carrier aircraft by each group data block is written to secondary server corresponding with adding carrier aircraft from machine room.With in prior art by transferring equipment order propelling data to different machine room mode compared with, the present invention is responsible for the loading of data by main equipment room, adding carrier aircraft by many groups in main equipment room walks abreast to multiple from the secondary server propelling data machine room, transferring equipment only need send data to main equipment room, because transferring equipment only relates to a data load process, therefore greatly can save the resource occupation of I/O, thus can improve across the data loading efficiency under building environment.

Above-mentioned explanation is only the general introduction of technical solution of the present invention, in order to technological means of the present invention can be better understood, and can be implemented according to the content of instructions, and can become apparent, below especially exemplified by the specific embodiment of the present invention to allow above and other objects of the present invention, feature and advantage.

Accompanying drawing explanation

By reading hereafter detailed description of the preferred embodiment, various other advantage and benefit will become cheer and bright for those of ordinary skill in the art.Accompanying drawing only for illustrating the object of preferred implementation, and does not think limitation of the present invention.And in whole accompanying drawing, represent identical parts by identical reference symbol.In the accompanying drawings:

Fig. 1 shows the process flow diagram of the method for a kind of Data import that the embodiment of the present invention provides;

The transferring equipment that Fig. 2 shows the embodiment of the present invention to be provided sends the schematic diagram of data block in one way;

The transferring equipment that Fig. 3 shows the embodiment of the present invention to be provided sends the schematic diagram of data fragmentation in one way;

The transferring equipment that Fig. 4 shows the embodiment of the present invention to be provided sends the schematic diagram of data fragmentation in another way;

Fig. 5 shows the schematic diagram of the data fragmentation obtained of cutting into slices with a kind of rule that the embodiment of the present invention provides;

Fig. 6 shows the schematic diagram of the data fragmentation obtained of cutting into slices with another kind of rule that the embodiment of the present invention provides;

Fig. 7 shows the schematic diagram of the data fragmentation obtained of cutting into slices with another kind of rule that the embodiment of the present invention provides;

Fig. 8 shows main equipment room that the embodiment of the present invention provides in one way to the schematic diagram adding carrier aircraft write data;

Fig. 9 shows main equipment room that the embodiment of the present invention provides in another way to the schematic diagram adding carrier aircraft write data;

Figure 10 shows main equipment room that the embodiment of the present invention provides in another way to the schematic diagram adding carrier aircraft write data;

Figure 11 shows the process flow diagram of the method for a kind of Data import that the embodiment of the present invention provides;

Figure 12 shows the structural representation of the device of a kind of Data import that the embodiment of the present invention provides;

Figure 13 shows the structural representation of the device of the another kind of Data import that the embodiment of the present invention provides;

Figure 14 shows the schematic diagram of the system of a kind of Data import that the embodiment of the present invention provides.

Embodiment

Below with reference to accompanying drawings exemplary embodiment of the present disclosure is described in further detail.Although show exemplary embodiment of the present disclosure in accompanying drawing, however should be appreciated that can realize the disclosure in a variety of manners and not should limit by the embodiment set forth here.On the contrary, provide these embodiments to be in order to more thoroughly the disclosure can be understood, and complete for the scope of the present disclosure can be conveyed to those skilled in the art.

Based on the development of internet cloud, people can be inquired about by various data system and use the valuable data of magnanimity, in order to improve the experience of user and meet the performance of system, data are deployed in multiple machine room usually respectively, for bearing the query demand of different user.In the face of the renewal of mass data, if to a large amount of machine room more new data one by one, then Data Update overlong time can be made; Meanwhile, mass data order is pushed in the process of a large amount of machine room by transferring equipment, also seriously can take the process resource of transferring equipment, and therefore this mode effect is in actual applications unsatisfactory.In order to solve across the low problem of the data loading efficiency under building environment, embodiments provide a kind of method of Data import, as shown in Figure 1, the method comprises:

101, data block is sent in main equipment room by transferring equipment.

Transferring equipment can sign in target device and safeguard, namely transferring equipment can be regarded as a station server, intrasystem mass data can be landed to local disk by it, then by the data-pushing of disk in each machine room, in actual applications, this transferring equipment can be springboard machine or fort machine; And main equipment room can as the center of data-pushing, namely by main equipment room to other machine rooms (from machine room) propelling data.Include many groups in this main equipment room and add carrier aircraft, the group number wherein adding carrier aircraft is identical with the number from machine room.From machine room, include a large amount of secondary server, wherein secondary server quantity with in corresponding main equipment room often group to add the quantity of carrier aircraft identical.This kind of mode can greatly alleviate transferring equipment excessively taking frequently to the process resource caused during each machine room propelling data.

Further, be sent in the process of main equipment room in transferring equipment by data block, can have different implementations, these implementations can comprise:

1) transferring equipment is sent in main equipment room by landing to the data block of local disk, as shown in Figure 2.

Although the implementation this, data block being sent to main equipment room can reduce the resource overhead of establishment and service data propelling movement process, because the data volume of data block is excessive, still can cause the overlong time that data send.

2) transferring equipment by land to local disk data block section after with individual process to main equipment room serially-transmitted data burst, as shown in Figure 3.

This by data block cut into slices after with the implementation of individual process to main equipment room serially-transmitted data burst, the data volume that single sends can be reduced, thus make the time at every turn sending data shorter, pausing compared to once sending the transmission card that the larger data block of data volume may cause, can send to main equipment room the total data that whole data block comprises faster by the mode repeatedly sending the data fragmentation of small data quantity.

3) transferring equipment by land to local disk data block section after with multiple process to main equipment room transmitted in parallel data fragmentation, as shown in Figure 4.

Based on above-mentioned by data block cut into slices after in the mode of individual process to main equipment room serially-transmitted data burst, this, although data fragmentation is needed to push in establishment and service data the resource that in process, expense is certain with the implementation of multiple process to main equipment room transmitted in parallel, but when not exceeding the process resource of transferring equipment and main equipment room, the speed sending data to main equipment room can greatly be improved.

What 102, data block was written to each group by main equipment room respectively eachly adds in carrier aircraft.

After data block is sent to main equipment room by transferring equipment, what the data block received can be written to each group by main equipment room respectively eachly adds in carrier aircraft, and the result of this step makes each carrier aircraft that adds be written into complete data block.

Further, main equipment room data block is written to respectively each group each add in carrier aircraft time, the implementation with above-mentioned transferring equipment data block being sent to main equipment room is similar, and its implementation can comprise:

A) when main equipment room receive be transferring equipment send data block time, this data block can be added carrier aircraft serial with single thread to each group each and send, add carrier aircraft until each and be written into complete data block.

B) when main equipment room receive be transferring equipment send data block time, this data block can be added carrier aircraft with multiple thread parallel to each group each and send, add carrier aircraft until each and be written into complete data block.

This implementation adding carrier aircraft transmission data block with multiple thread parallel to each group, although need in the resource created and in service data propelling movement thread, expense is certain, but when not exceeding the process resource of main equipment room, the speed to adding carrier aircraft transmission data can greatly be improved.

C) when main equipment room receive be transferring equipment send data block time, obtain data fragmentation after data block can being cut into slices, then data fragmentation is added carrier aircraft serial to each group each successively with single thread send.

This successively to adding in carrier aircraft the mode that writes the less data fragmentation of data volume relative to once writing for the larger data block of data volume to adding in carrier aircraft, can shorten on the whole to adding in carrier aircraft the time writing full block of data.

D) when main equipment room receive be transferring equipment send data block time, obtain data fragmentation after data block can being cut into slices, then data fragmentation is added carrier aircraft serial with multiple thread parallel to each group each send.

This mode sent compared to single thread serial in the mode that multiple thread parallel sends by data fragmentation, when not considering establishment and service data pushes thread overhead resource, can greatly shorten the time sending the data to and add carrier aircraft.

In addition, when main equipment room receive be transferring equipment send data fragmentation time, it can send for single thread serial to the mode adding carrier aircraft transmission data fragmentation or multiple thread parallel sends, similar with above-mentioned implementation c) or d) in practical application, repeats no more herein.

Here it should be noted that, after data block section is obtained data fragmentation, sent to by data fragmentation the mode adding carrier aircraft in fact can shorten the time sending data, when determining in actual applications to send with single thread serial or send with multiple thread parallel, mainly see main equipment room, add the expense situation of the process resources such as carrier aircraft, when processing resource and can meeting multithreading transmission data, data can be realized in the mode of multi-threaded parallel transmission data and push faster.

103, add carrier aircraft data block to be written in secondary server corresponding with adding carrier aircraft from machine room.

By above-mentioned steps, the data received (data block/data fragmentation) are sent to and add after in carrier aircraft by main equipment room, can to from machine room propelling data.Because the number from machine room is identical with the group number adding carrier aircraft in main equipment room, from machine room, number and the every group of secondary server adds the identical and one_to_one corresponding of the number of carrier aircraft, therefore, data block can be pushed in corresponding secondary server by each carrier aircraft that adds of each group, realizes the distribution of data in each machine room.

Add carrier aircraft for one, this adds carrier aircraft can send to corresponding secondary server by whole data block.Certainly, also after data block section, data fragmentation can be sent to corresponding secondary server with individual process serial, or after data block section, data fragmentation be sent to corresponding secondary server with multiple task parallelism.

The method of the Data import that the embodiment of the present invention provides, data block in transferring equipment can be sent in main equipment room, by main equipment room, by data block, each group of being written to respectively in main equipment room adds in carrier aircraft, then adds carrier aircraft by each group data block is written to secondary server corresponding with adding carrier aircraft from machine room.With in prior art by transferring equipment order propelling data to different machine room mode compared with, the present invention is responsible for the loading of data by main equipment room, adding carrier aircraft by many groups in main equipment room walks abreast to multiple from the secondary server propelling data machine room, transferring equipment only need send data to main equipment room, because transferring equipment only relates to a data load process, therefore greatly can save the resource occupation of I/O, thus can improve across the data loading efficiency under building environment.

Understand the method shown in Fig. 1 in order to better, as to the refinement of above-mentioned embodiment and expansion, the embodiment of the present invention is described in detail for the step in Fig. 1.

Because mass data in system can be carried out disk so that the propelling movement of data by transferring equipment, so when the data block after disk is sent to main equipment room by transferring equipment, namely when performing above-mentioned steps 101, not only whole data block can be sent to main equipment room, first data block can also be carried out section and obtain data fragmentation, then data fragmentation be sent to main equipment room.In the process that data block is cut into slices by reality, not only to consider the quantity of data fragmentation, and the rule of section will be considered, namely according to which kind of mode cut into slices.

Although when the data volume of data block is larger, data block section can be obtained mass data burst, be not that the quantity of data fragmentation is more, the data volume of each data fragmentation is less, and it sends to the time of main equipment room to shorten.The transmission of each data fragmentation will take a process, when mass data burst transmitted in parallel, need to take a large amount of process of main equipment room, and create and safeguard that these processes need the process resource of expense main equipment room, so when data block is cut into slices, the number of processes that main equipment room is current must be considered, can not arbitrarily cut into slices.Such as, the quantity of process can being set by webmaster personnel according to the ruuning situation of main equipment room, with reference to current number of processes, data block can be cut to the data fragmentation of respective numbers when data block being cut into slices.

When after the quantity determining data fragmentation, just need to cut into slices to data block.In the process of section, can cut into slices according to different rules, its implementation can comprise:

1) different data fragmentations is divided into according to the mark of every bar data or numbering according to polling mechanism;

2) be divided into different data fragmentations according to the number order of every bar data, wherein, the data number comprised in each data fragmentation can be identical, also can be different.

Such as, when four can be divided into according to the quantity of the current process number determination data fragmentation of main equipment room, data block can be divided into four data fragmentations as shown in Figure 5 according to polling mechanism; And four data fragmentations that data block can be divided into when dividing according to the number order of every bar data as shown in Figure 6, wherein the data number that contains of each data fragmentation is identical; Also data block can be divided into four data fragmentations as shown in Figure 7, wherein the data number that contains of each data fragmentation is different.

Here it should be noted that, data block can be cut into slices according to above-mentioned rule by main equipment room and transferring equipment, wherein, the mode of cutting into slices or the number needing to calculate data is carried out according to the numbering of every bar data, thus can be balanced be the data fragmentation that data volume is suitable by data block cutting, or cut into slices to data block at random, the data fragmentation obtained like this is not of uniform size, if the data volume of data fragmentation is excessive, be then unfavorable for the propelling movement of data.The number calculating data is not then needed according to polling mechanism dividing data burst, its operation is succinct, namely each data is distributed to each data fragmentation corresponding with process successively, until every bar data are assigned with complete, the load relative equilibrium of the corresponding process of each data fragmentation farthest can be ensured like this.

When data block is split after into data fragmentation, according to mode as shown in Figure 4, multiple data fragmentation can be sent to main equipment room with multiple task parallelism.

In step 102 main equipment room, the data received (data block/data fragmentation) are written in each practical implementation added in carrier aircraft of each group respectively, the multiple processes equal with data fragmentation quantity can be set up in main equipment room, wherein each process is responsible for the write operation of a data fragmentation, and what the data fragmentation of correspondence is written to respectively each group by each process eachly adds in carrier aircraft.Because each process is independently when writing data fragmentation, therefore main equipment room need not control each process write data fragmentation whether add carrier aircraft identical, only need to safeguard the normal of each process and run.

Certainly, in order to ensure can stablize when data fragmentation write is added carrier aircraft and fast, therefore carry out write operation different course allocation can to the carrier aircraft that adds of different group.If number of processes is greater than the group number adding carrier aircraft, when needing multiple process to add carrier aircraft write data fragmentation to one group simultaneously, one can be distributed respectively initially add carrier aircraft with reference to multiple process that is numbered that in every group, each adds carrier aircraft, then start order and add carrier aircraft write data fragmentation to follow-up.No matter that different course allocation is carried out when write operation or multiple process add carrier aircraft write data fragmentation to one group being simultaneously that multiple process is distributed one respectively and initially added carrier aircraft to the carrier aircraft that adds of different group, then start order and add carrier aircraft write data fragmentation to follow-up, these two kinds of implementations can avoid too much process to add carrier aircraft write mass data to same simultaneously, improve to adding in carrier aircraft the stability and agility that write data with this.

Eachly add in carrier aircraft to ensure that total data burst can be written to by each process, this is that the different initial mode adding carrier aircraft of different course allocation also needs a kind of testing mechanism, which is used for detecting add carrier aircraft and be written with data fragmentation, which adds carrier aircraft and does not also write data fragmentation.In order to realize testing mechanism, ensure each process can by initial add carrier aircraft before add carrier aircraft write data fragmentation, as the optional embodiment of one, one can be created in main equipment room and add carrier aircraft label table, this add carrier aircraft label table record the group number that adds carrier aircraft and often group add the number information of carrier aircraft, each process can be transferred this when writing data fragmentation and add carrier aircraft label table, whenever one add write data fragmentation in carrier aircraft after just the carrier aircraft that adds of correspondence is marked adding in carrier aircraft label table, when last is added carrier aircraft write terminate after, start traversal and add carrier aircraft label table, determine whether add carrier aircraft all marks, if do not have, then according to the number order added in carrier aircraft label table, write operation is carried out to the carrier aircraft that adds do not marked, until each carrier aircraft numbering that adds added in carrier aircraft label table is labeled, thus complete each process adds write operation from carrier aircraft to correspondence.

Further, set up multiple process in main equipment room before, except the process resource will considering main equipment room, also to consider the load capacity adding carrier aircraft, therefore the maximum number of processes adding carrier aircraft and can bear can be preset according to the performance adding carrier aircraft, the number of processes that main equipment room is set up can not be greater than default number of processes, controls each load pressure adding carrier aircraft with this, avoids adding that carrier aircraft load is excessive causes data to write unsuccessfully and to the infringement adding carrier aircraft.

The process that main equipment room establishes some for the data fragmentation of correspondence is written to respectively each group each add in carrier aircraft time, further can set up many threads respectively under each process, for accelerating the speed writing data fragmentation, wherein a thread is responsible at least one write operation adding carrier aircraft.Write in the process of data fragmentation by thread in reality, for a process, can numerous embodiments be had, comprise:

1) one adds the corresponding thread of carrier aircraft, and many threads that namely a process is identical with adding carrier aircraft number according to its number foundation adding carrier aircraft be responsible for, add carrier aircraft write data fragmentation by many thread parallels to each, improve schedule speed with this.Such as, process A is responsible for loading unit 1, loading unit 2 and load unit 3, load unit 1, load unit 2 and load unit 3 and have three separately and add carrier aircraft, then process A can set up nine thread parallels and adds carrier aircraft write data fragmentation to each group each, as shown in Figure 8.

2) one group adds the corresponding thread of carrier aircraft, namely a process sets up many threads identical with loading unit number according to the group number adding carrier aircraft that it is responsible for, carrier aircraft write data fragmentation is added to each group by many thread parallels, wherein, article one, thread adds carrier aircraft when carrying out write operation for its group of being responsible for, serial can add carrier aircraft to this group and be sequentially written in data fragmentation, the quantity of thread can be reduced thus, reduce and set up and safeguard thread taking resource.Such as, process A is responsible for loading unit 1, loading unit 2 and load unit 3, and therefore process A can set up three thread a ₁, a ₂, a ₃, thread a ₁be responsible for loading unit 1, thread a ₂be responsible for loading unit 2, thread a ₃be responsible for loading unit 3, three thread a ₁, a ₂, a ₃walk abreast to loading unit 1, loading unit 2 and loading unit 3 and write data fragmentation, as shown in Figure 9.

3) many groups add the corresponding thread of carrier aircraft, and the number of threads that namely process is set up is less than the group number adding carrier aircraft that this process is responsible for.Such as, process A is responsible for loading unit 1, loading unit 2 and load unit 3, but set up to reduce and safeguard many threads excessively taking resource, process A can set up a thread a, by thread a serial, successively data fragmentation is write to loading unit 1, loading unit 2 and loading unit 3, as shown in Figure 10.

At main equipment room, data block or data fragmentation write are added in the process of carrier aircraft, owing to writing the circumstance complication of data, whether, as many processes and many thread parallel propelling datas, damaging also may appear in data in propelling movement process simultaneously, therefore need detection data to be successfully written to and add in carrier aircraft.Detection can write to adding carrier aircraft the write state recording every bar data in the process of data by main equipment room, the write state record sheet of called data after write operation terminates, if there are data writes unsuccessfully, then main equipment room is again to adding carrier aircraft writing data blocks or data fragmentation, or again pushes these data according to the mark or numbering writing failed data (unit is bar) in record sheet.

Certainly, detect and also can record the write state of every bar data by adding carrier aircraft and performing at main equipment room in the process of write operation, the write state record sheet of called data after write operation terminates, if there are data writes unsuccessfully, then add carrier aircraft and re-execute write operation to main equipment room request, or again ask to push this data to main equipment room according to the mark or numbering writing failed data (unit is bar) in record sheet.

The write of data unsuccessfully can have many reasons, if because add carrier aircraft or the excessive write failure caused of main equipment room load pressure, so follow-uply again can successfully write these data; If when data there occurs irreversible damage in propelling movement process, push how many times again regardless of follow-up, these data still cannot successfully write.Therefore, main equipment room is when execution re-writes operation, need the rewrite operation carrying out limited number of time, avoid when irreversible damage appears in data, continue push these data cause to main equipment room process resource take and to add carrier aircraft write interface continue take, make other processes cannot write other data.The execution number of times of main equipment room rewrite operation can artificially set, and also can take situation adjustment by main equipment room according to current process resource, when the overall principle is process resource abundance, sets less number of rewrites; When process resource occupation is serious, set more number of rewrites.Such as, if process resource is sufficient, then can sets and rewrite the upper limit once or twice, if process resource occupation is serious, then can set five times or rewrite the upper limit six times, concrete number of rewrites the present embodiment is not restricted.

Because these mass datas need to be deployed in meet the query demand of different user in a large amount of machine room, therefore when adding after carrier aircraft has been written into data in main equipment room, just need to a large amount of from machine room propelling data.Wherein, identical with the group number adding carrier aircraft main equipment room from the number of machine room, have how many will be deployed with how many groups main equipment room from machine room and add carrier aircraft, one group adds carrier aircraft and is responsible for a write operation from machine room.Wherein, not only comprise secondary server from machine room, also comprise three tier server, these data will be stored in from the secondary server and three tier server of machine room.The number of secondary server and every group add the identical and one_to_one corresponding of the number of carrier aircraft, and one adds the write operation that carrier aircraft is responsible for a secondary server; The number of three tier server and one_to_one corresponding identical with the number of secondary server, a secondary server is responsible for the write operation an of three tier server.

Further, consider and need to adding that data that carrier aircraft pushes are more and path that is that push is complicated, continue to duplicate to during secondary server propelling data or omit in order to avoid adding carrier aircraft, as the optional embodiment of one, can before the data block added in carrier aircraft or data fragmentation be written to from the secondary server machine room, first detected by the data mode added in carrier aircraft of secondary server to correspondence, if find, the data added in carrier aircraft change, then secondary server adds carrier aircraft request of loading data to correspondence.

In order to secondary server be allowed to detect add data mode in carrier aircraft, one can be set up load machine data table adding in carrier aircraft, this loading machine data table record has the every bar data adding and write in carrier aircraft, and every bar data all have mark or numbering, whenever add in carrier aircraft carried out Data Update time, this loading machine data table also can by synchronized update; Certainly, also a secondary server tables of data can be set up in secondary server, with the character the same (data are consistent with the corresponding relation of mark or numbering) loading machine data table, secondary server tables of data also records the every bar data write in secondary server, and every bar data also all have mark or numbering.Secondary server tables of data and loading machine data table are compared by secondary server, exist inconsistent in two tables of data if find, namely when secondary server finds that the data added in carrier aircraft have a renewal, secondary server is according to the content loading machine data table, to adding the data of carrier aircraft request write renewal or deleting data voluntarily, thus complete the renewal of data in secondary server.

Further, at secondary server in the process adding carrier aircraft request write data, consider the load capacity of secondary server, if when the data loaded are more, the too fast secondary server that may cause of loading velocity loads data failure, therefore can carry out speed limit to the loading velocity of secondary server.As the optional embodiment of one, can when adding carrier aircraft to secondary server propelling data, data are carried out burst and sets up many processes, multi-process walks abreast and pushes the data fragmentation of small data quantity to secondary server, with this, data are loaded to secondary server and carry out speed limit, shorten the time that secondary server loads data simultaneously as far as possible.

In the data query system of reality, the server in machine room, except the loading/propelling movement carrying out data, be the more important thing is and responded the inquiry request of user, is user's data query.But the propelling movement coverage of data is large, in order to ensure the ageing of data, just need the propelling movement carrying out data when data change immediately, now, the process of propelling data clashes with the process of user's data query possibly, causes the plunder to system resource.Therefore, in order to ensure that Data Update does not affect again the inquiry of user in time simultaneously, background synchronization mechanism can be adopted to the write of secondary server from machine room data, can not response time of influential system current request in the process of write data, real-time query service can be provided for user.Certainly, from machine room, secondary server also can adopt background synchronization mechanism to carry out to three tier server propelling data.

The embodiment of the present invention is except improving data-pushing mode, parallel mode is changed into by original serial mode, the data-pushing time is shortened when data volume is constant, further, have employed principal and subordinate's mechanism to the distribution of machine room, main equipment room is responsible for loading data, is responsible for from main equipment room synchrodata from machine room, and have employed background synchronization mechanism, real-time query service can be provided when not influential system current request response time; Meanwhile, the loading of data and propelling movement add multi-process, multithreading and retry mechanism, shorten the time of Data import and propelling movement while ensureing data consistency.

Above-described embodiment is with transferring equipment, adds carrier aircraft, secondary server and three tier server for the main explanation carried out the Data import across machine room, the present embodiment will be also that master is described the Data import across machine room with main equipment room below, wherein main equipment room comprises main equipment room control server, the method of this Data import as shown in figure 11, comprising:

201, main equipment room control server receives the data block that transferring equipment sends.

Main equipment room control server, as the core component in main equipment room, can receive the various data that transferring equipment sends, comprise:

1) data block that transferring equipment sends to main equipment room is received.

Although this mode directly receiving data block by control server can reduce to create and the resource overhead of service data propelling movement process, because the data volume of data block is excessive, the overlong time of data receiver still can be caused.

2) data fragmentation obtained after section of transferring equipment serial transmission is received with individual process.

This mode receiving data fragmentation with individual process, the data volume that single receives can be reduced, thus make the time at every turn receiving data shorter, pausing compared to once receiving the transmission card that the larger data block of data volume may cause, can receive by the mode repeatedly receiving the data fragmentation of small data quantity the total data that transferring equipment sends faster.

3) data fragmentation obtained after section of transferring equipment transmitted in parallel is received with multiple process.

Based on the above-mentioned mode receiving the data fragmentation obtained after section that transferring equipment serial sends with individual process, although this mode receiving the data fragmentation obtained after section of transferring equipment transmitted in parallel with multiple process needs in the resource created and in service data propelling movement process, expense is certain, but when not exceeding the process resource of main equipment room control server, the speed that main equipment room control server receives data can greatly be improved.

202, by the data block received, each group of being written to respectively in main equipment room adds in carrier aircraft main equipment room control server.

What the data block received was written to each group by main equipment room control server respectively eachly adds in carrier aircraft, and the result of this step makes each carrier aircraft that adds be written into complete data block.

Further, data block is written to the implementation added in carrier aircraft and can comprises by main equipment room control server:

1) the complete data block received is added carrier aircraft serial with single thread to each group each and sends by main equipment room control server, adds carrier aircraft be written into complete data block until each.

2) the complete data block received sends with each carrier aircraft that adds of multiple thread parallel to each group by main equipment room control server, adds carrier aircraft be written into complete data block until each.

3) main equipment room control server obtains data fragmentation after the complete data block received being cut into slices, then data fragmentation is added carrier aircraft serial to each group each successively with single thread sends.

4) main equipment room control server obtains data fragmentation after the complete data block received being cut into slices, then data fragmentation is added carrier aircraft serial with multiple thread parallel to each group each sends.

In addition, when main equipment room control server receive be transferring equipment send data fragmentation time, it equally also can send for single thread serial to the mode adding carrier aircraft and send data fragmentation or multiple thread parallel sends, and repeats no more herein.

203, data block is written in secondary server corresponding with adding carrier aircraft from machine room by adding carrier aircraft by main equipment room control server.

To add after in carrier aircraft when the data received (data block/data fragmentation) are sent to by main equipment room control server, just can to from machine room propelling data.Because the number from machine room is identical with the group number adding carrier aircraft in main equipment room, from machine room, number and the every group of secondary server adds the identical and one_to_one corresponding of the number of carrier aircraft, therefore, main equipment room control server can add carrier aircraft and data block is pushed to and adds in secondary server corresponding to carrier aircraft by controlling each group each, realizes the distribution of data in each machine room.

Certainly, whole data block can be sent to corresponding secondary server by controlled loading machine by main equipment room control server; Also after data block section, data fragmentation can be sent to corresponding secondary server with individual process serial, or after data block section, data fragmentation be sent to corresponding secondary server with multiple task parallelism.

Because mass data in system can be carried out disk so that the propelling movement of data by transferring equipment, so whole data block not only can be sent to main equipment room by transferring equipment, first data block can also be carried out section and obtain data fragmentation, then data fragmentation be sent to main equipment room.In the process that data block is cut into slices by reality, not only to consider the quantity of data fragmentation, and the rule of section will be considered.

When data block is cut into slices by reality, be not that the quantity of data fragmentation is more, the data volume of each data fragmentation is less, and it sends to the time of main equipment room to shorten.Transmission due to each data fragmentation will take a process, when mass data burst transmitted in parallel, main equipment room control server needs to set up a large amount of processes, and create and safeguard that these processes need the process resource of expense main equipment room control server, therefore when determining the quantity of data fragmentation, the number of processes that main equipment room control server can be safeguarded must be considered, can not arbitrarily data block be cut into slices.Now, the quantity of process can be set according to the load condition of main equipment room control server by webmaster personnel, with reference to the main equipment room control server current number of processes that can safeguard, data block is cut to the data fragmentation of respective numbers.

When the quantity according to established data burst is cut into slices to data block, can cut into slices according to different rules, its implementation can comprise:

Here it should be noted that, carry out the mode of cutting into slices according to the numbering of every bar data or need to calculate the number of data, thus can be balanced be the data fragmentation that data volume is suitable by data block cutting; At random data block is cut into slices, obtain the data fragmentation that data volume is not of uniform size, if the data volume of data fragmentation is excessive, be then unfavorable for the propelling movement of data.

Then do not need according to polling mechanism dividing data burst the number calculating data, namely each data is distributed to each data fragmentation corresponding with process successively, until every bar data are assigned with complete.Succinct according to the operation of polling mechanism dividing data burst, farthest can ensure the load relative equilibrium of the corresponding process of each data fragmentation.

Data block cutting can be data fragmentation by transferring equipment in the manner described above, and multiple data fragmentation is sent to main equipment room control server with multiple task parallelism.

Further, the data received (data block/data fragmentation) are written in each practical implementation added in carrier aircraft of each group by main equipment room control server respectively, can set up the multiple processes equal with data fragmentation quantity, wherein each process is responsible for the write operation of a data fragmentation.Because each process is independently when writing data fragmentation, therefore main equipment room control server need not control each process and whether write data to the same carrier aircraft that adds, and only needs to safeguard the normal of each process and runs.

In order to ensure can stablize when data fragmentation write is added carrier aircraft and fast, therefore carry out write operation different course allocation can to the carrier aircraft that adds of different group.If number of processes is greater than the group number adding carrier aircraft, when needing multiple process to add carrier aircraft write data fragmentation to one group simultaneously, one can be distributed respectively initially add carrier aircraft with reference to multiple process that is numbered that in every group, each adds carrier aircraft, then start order and add carrier aircraft write data fragmentation to follow-up.

No matter that different course allocation is carried out when write operation or multiple process add carrier aircraft write data fragmentation to one group being simultaneously that multiple process is distributed one respectively and initially added carrier aircraft to the carrier aircraft that adds of different group, then start order and add carrier aircraft write data fragmentation to follow-up, these two kinds of implementations can avoid too much process to add carrier aircraft write mass data to same simultaneously, improve to adding in carrier aircraft the stability and agility that write data with this.

In order to ensure that each adding in carrier aircraft can be written into whole data, thisly all add carrier aircraft for the different initial mode adding carrier aircraft of different course allocation also needs to detect and whether be written with data.In order to realize testing mechanism, ensure each process can by initial add carrier aircraft before add carrier aircraft write data fragmentation, as the optional embodiment of one, main equipment room control server can create one and add carrier aircraft label table, for record the group number that adds carrier aircraft and often group add the number information of carrier aircraft.Each process is in the process performing write operation, can be transferred this and add carrier aircraft label table, after completing and adding the write operation of carrier aircraft to one, just the carrier aircraft that adds of correspondence is marked adding in carrier aircraft label table, when last is added carrier aircraft write terminate after, start traversal and add carrier aircraft label table, determine whether add carrier aircraft all marks, if do not have, then according to the number order added in carrier aircraft label table, write operation is carried out to the carrier aircraft that adds do not marked, until each carrier aircraft numbering that adds added in carrier aircraft label table is labeled, thus complete each process adds write operation from carrier aircraft to correspondence.

Further, main equipment room control server is before setting up multiple process, except will considering the process resource of self, also to consider the load capacity adding carrier aircraft, the maximum number of processes adding carrier aircraft and can bear is preset according to the performance adding carrier aircraft, the number of processes that main equipment room control server is set up can not be greater than default number of processes, controls each load pressure adding carrier aircraft with this, avoids adding that carrier aircraft load is excessive causes data to write unsuccessfully and to the infringement adding carrier aircraft.

Main equipment room control server carries out write operation for adding carrier aircraft to every group except the process of setting up some, further can also set up many threads respectively under each process, for accelerating the speed writing data fragmentation, wherein a thread is responsible at least one write operation adding carrier aircraft.Write in the process of data fragmentation by thread in reality, can numerous embodiments be had, comprise:

A) one adds the corresponding thread of carrier aircraft.For a process, the number adding carrier aircraft that this process is responsible for according to it sets up many threads identical with adding carrier aircraft number, adds carrier aircraft write data fragmentation, improve schedule speed with this by many thread parallels to each.

B) one group adds the corresponding thread of carrier aircraft, namely a process sets up many threads identical with loading unit number according to the group number adding carrier aircraft that it is responsible for, carrier aircraft write data fragmentation is added to each group by many thread parallels, wherein, article one, thread adds carrier aircraft when carrying out write operation for its group of being responsible for, and serial can add carrier aircraft to this group and be sequentially written in data fragmentation.

C) many groups add the corresponding thread of carrier aircraft, and the number of threads that namely process is set up is less than the group number adding carrier aircraft that this process is responsible for.

Because main equipment room control server is to the circumstance complication added when carrier aircraft performs write operation, as many processes and many thread parallel propelling datas, whether damaging also may appear in data in propelling movement process simultaneously, therefore need detection data to be successfully written to and add in carrier aircraft.Detection can write to adding carrier aircraft the write state recording every bar data in the process of data by main equipment room control server, the write state record sheet of called data after write operation terminates, if there are the data that write is failed, then main equipment room control server is again to adding carrier aircraft writing data blocks or data fragmentation, or again pushes these data according to the mark or numbering writing failed data (unit is bar) in record sheet.

Certainly, detect and also can record the write state of every bar data by adding carrier aircraft and performing in main equipment room control server in the process of write operation, the write state record sheet of called data after write operation terminates, if there are data writes unsuccessfully, then add carrier aircraft and re-execute write operation to the request of main equipment room control server, or again ask to push this data to main equipment room control server according to the mark or numbering writing failed data (unit is bar) in record sheet.

Write due to data unsuccessfully may add carrier aircraft or main equipment room load pressure is excessive causes, also may because data there occurs irreversible damage and cause in propelling movement process.If the former, then follow-uply can successfully write these data again; If the latter, then write how many times again regardless of follow-up, these data still cannot successfully write.Therefore, main equipment room control server needs the number of times controlling to re-write operation, avoid when irreversible damage appears in data, continue push these data cause to main equipment room control server process resource take and to add carrier aircraft write interface continue take, make other processes cannot write other data.

The number of times of main equipment room control server rewrite operation can artificially set, and also can take situation adjustment by main equipment room control server according to current process resource, when principle is process resource abundance, sets less number of rewrites; When process resource occupation is serious, set more number of rewrites.Such as, if process resource is sufficient, then can sets and rewrite the upper limit once or twice, if process resource occupation is serious, then can set five times or rewrite the upper limit six times, concrete number of rewrites the present embodiment is not restricted.

Further, at secondary server in the process adding carrier aircraft request write data, consider the load capacity of secondary server, if when the data loaded are more, the too fast secondary server that may cause of loading velocity loads data failure, therefore can carry out speed limit by main equipment room control server to the loading velocity of secondary server.As the optional embodiment of one, when adding carrier aircraft to secondary server propelling data, data are carried out burst by the carrier aircraft that adds in main equipment room control server main control system room and is set up many processes, multi-process walks abreast and pushes the data fragmentation of small data quantity to secondary server, with this, data are loaded to secondary server and carry out speed limit, shorten the time that secondary server loads data simultaneously as far as possible.

In the data query system of reality, the server in machine room, except the loading/propelling movement carrying out data, be the more important thing is and responded the inquiry request of user, is user's data query.But the propelling movement coverage of data is large, in order to ensure the ageing of data, just need the propelling movement carrying out data when data change immediately, now, the process of propelling data clashes with the process of user's data query possibly, causes the plunder to system resource.Therefore, in order to ensure that Data Update does not affect again the inquiry of user in time simultaneously, main equipment room control server can adopt background synchronization mechanism to write data to from the secondary server in machine room, can not response time of influential system current request in the process of write data, real-time query service can be provided for user.

Further, as the realization to method shown in above-mentioned Fig. 1 and Figure 11, embodiments provide a kind of device of Data import, this device can be arranged in main equipment room or main equipment room control server, also and can have data interaction relation between main equipment room or main equipment room control server independent of main equipment room or main equipment room control server.As shown in figure 12, this device comprises: receiving element 121, writing unit 122 and control module 123, wherein,

Receiving element 121, for receiving the data block that transferring equipment sends;

Writing unit 122, data block for being received by receiving element 121 each group of being written to respectively in main equipment room adds in carrier aircraft, wherein, comprise many groups in main equipment room and add carrier aircraft, the group number adding carrier aircraft is identical with the number from machine room, and often to add the quantity of carrier aircraft identical from the secondary server quantity machine room with corresponding for group;

Control module 123, is written to the data block that writing unit 122 writes in secondary server corresponding with adding carrier aircraft from machine room for controlled loading machine.

Further, receiving element 121, for receiving multiple data fragmentations of transferring equipment transmitted in parallel, multiple data fragmentation carries out section by transferring equipment to data block and obtained.

Further, writing unit 122, for setting up the multiple processes equal with data fragmentation quantity, wherein each process is responsible for the write operation of a data fragmentation, and by process, by the data fragmentation of correspondence, each group of being written to respectively in main equipment room adds in carrier aircraft.

Further, writing unit 122, for before setting up the multiple processes equal with data fragmentation quantity, controls concurrent process number and is not more than preset course number.

Further, writing unit 122, for setting up many threads respectively under each process, wherein, article one, thread is responsible for the write operation that at least one adds carrier aircraft, and that is responsible for separately to many threads by many thread parallels adds data fragmentation corresponding to carrier aircraft write process.

Further, as shown in figure 13, described device also comprises:

Judging unit 124, after each group of being written to respectively in main equipment room of the data block for being received by receiving element 121 at writing unit 122 adding in carrier aircraft, judging whether data block is successfully written to and adding in carrier aircraft;

Writing unit 122, for when judging unit 124 judges that data block writes unsuccessfully, carries out the rewrite operation of limited number of time to data block.

Further, control module 123, for add carrier aircraft by data block the speed be written in the secondary server corresponding with adding carrier aircraft carry out speed limit.

Further, control module 123, adds carrier aircraft for adopting background synchronization mechanism control and data block is written in secondary server corresponding with adding carrier aircraft from machine room.

The device of the Data import that the embodiment of the present invention provides, data block in transferring equipment can be sent in main equipment room, by main equipment room, by data block, each group of being written to respectively in main equipment room adds in carrier aircraft, then adds carrier aircraft by each group data block is written to secondary server corresponding with adding carrier aircraft from machine room.With in prior art by transferring equipment order propelling data to different machine room mode compared with, the present invention is responsible for the loading of data by main equipment room, adding carrier aircraft by many groups in main equipment room walks abreast to multiple from the secondary server propelling data machine room, transferring equipment only need send data to main equipment room, because transferring equipment only relates to a data load process, therefore greatly can save the resource occupation of I/O, thus can improve across the data loading efficiency under building environment.

In addition, the embodiment of the present invention is except improving data-pushing mode, parallel mode is changed into by original serial mode, shorten the data-pushing time when data volume is constant, further, principal and subordinate's mechanism be have employed to the distribution of machine room, main equipment room is responsible for loading data, be responsible for from main equipment room synchrodata from machine room, and have employed background synchronization mechanism, real-time query service can be provided again when not influential system current request response time; Meanwhile, the loading of data and propelling movement add multi-process, multithreading and retry mechanism, shorten the time of Data import and propelling movement while ensureing data consistency.

Further, as to the realization of method shown in above-mentioned Fig. 1 and Figure 11 and the application to above-mentioned Figure 12 and Figure 13 shown device, the embodiment of the present invention additionally provides a kind of system of Data import, as shown in figure 14, this system comprises: transferring equipment 141, main equipment room 142 and from machine room 143, wherein

Comprise main equipment room control server 1421 in main equipment room 142 and many groups add carrier aircraft 1422;

From machine room 143, comprise secondary server 1431, the group number adding carrier aircraft 1422 is identical with from the number of machine room 143, and often to add the quantity of carrier aircraft 1422 identical from secondary server 1431 quantity machine room 143 with corresponding for group;

Transferring equipment 141, for being sent to main equipment room control server 1421 by data block;

Main equipment room control server 1421, what the data block for transferring equipment 141 being sent was written to each group respectively eachly adds in carrier aircraft 1422;

Add carrier aircraft 1422, the data block for main equipment room control server 1421 being write is written in secondary server 1431 corresponding with adding carrier aircraft 1422 from machine room 143.

Further, comprise three tier server 1432 from machine room 143, three tier server 1432 and one_to_one corresponding identical with secondary server 1431 number, secondary server 1431 writes data in corresponding three tier server 1432.

The system of the Data import that the embodiment of the present invention provides, data block in transferring equipment can be sent in main equipment room, by main equipment room, by data block, each group of being written to respectively in main equipment room adds in carrier aircraft, then adds carrier aircraft by each group data block is written to secondary server corresponding with adding carrier aircraft from machine room.With in prior art by transferring equipment order propelling data to different machine room mode compared with, the present invention is responsible for the loading of data by main equipment room, adding carrier aircraft by many groups in main equipment room walks abreast to multiple from the secondary server propelling data machine room, transferring equipment only need send data to main equipment room, because transferring equipment only relates to a data load process, therefore greatly can save the resource occupation of I/O, thus can improve across the data loading efficiency under building environment.

Present embodiment discloses:

A kind of method of A1, Data import, is characterized in that, described method comprises:

Be sent in main equipment room by the data block in transferring equipment, comprise many groups and add carrier aircraft in described main equipment room, wherein, the group number adding carrier aircraft is identical with the number from machine room, and often to add the quantity of carrier aircraft identical from the secondary server quantity machine room with corresponding for group;

What described data block is written to respectively each group by described main equipment room eachly adds in carrier aircraft;

The described data block added in carrier aircraft is written to and adds in secondary server corresponding to carrier aircraft with described from machine room.

A2, method according to claim A1, is characterized in that, is describedly sent in main equipment room by the data block in transferring equipment, comprising:

Described data block is cut into slices, obtains multiple data fragmentation;

Described main equipment room is given by described multiple data fragmentation transmitted in parallel.

A3, method according to claim A2, is characterized in that, described described data block is written to respectively each group by described main equipment room eachly add in carrier aircraft, comprising:

In described main equipment room, set up the multiple processes equal with data fragmentation quantity, wherein each process is responsible for the write operation of a data fragmentation;

What the data fragmentation of correspondence is written to respectively each group by described process eachly adds in carrier aircraft.

A4, method according to claim A3, is characterized in that, set up the multiple processes equal with data fragmentation quantity in described main equipment room before, described method comprises further:

Control concurrent process number by described main equipment room and be not more than preset course number.

A5, method according to claim A3 or A4, is characterized in that, described the data fragmentation of correspondence is written to respectively each group by described process eachly add in carrier aircraft, comprising:

Under each process, set up many threads respectively, wherein, a thread is responsible at least one write operation adding carrier aircraft;

Data fragmentation corresponding to described process is write to the carrier aircraft that adds be responsible for separately by described many thread parallels.

A6, method according to claim A1, is characterized in that, describedly comprises three tier server further from machine room, and the quantity of described three tier server is identical with the quantity of described secondary server;

Described the described data block added in carrier aircraft is written to from machine room with described add in secondary server corresponding to carrier aircraft after, described method comprises further:

Described data block in secondary server is written in corresponding three tier server.

A7, method according to claim A1, is characterized in that, described by described main equipment room described data block to be written to respectively each group each add in carrier aircraft after, described method comprises further:

Judge described data block adds in carrier aircraft described in whether being successfully written to;

If described data block writes unsuccessfully, then described data block is carried out to the rewrite operation of limited number of time.

A8, method according to claim A1, is characterized in that, described the described data block added in carrier aircraft is written to from machine room with described add in secondary server corresponding to carrier aircraft before, described method comprises further:

By described secondary server, the described data mode added in carrier aircraft is detected;

If the data added in carrier aircraft change described in finding, then add data block described in carrier aircraft request of loading by described secondary server to described.

A9, method according to claim A1, it is characterized in that, described method comprises further:

Described data block is loaded to described secondary server and carries out speed limit.

A10, method according to claim A1, it is characterized in that, the described described data block added in carrier aircraft being written to adds in secondary server corresponding to carrier aircraft with described from machine room, comprising:

Background synchronization mechanism is adopted described data block to be written in described secondary server.

A kind of method of B11, Data import, is characterized in that, described method comprises:

Described main equipment room control server each group of the described data block received being written to respectively in main equipment room adds in carrier aircraft, wherein, comprise many groups in described main equipment room and add carrier aircraft, the group number adding carrier aircraft is identical with the number from machine room, and often to add the quantity of carrier aircraft identical from the secondary server quantity machine room with corresponding for group;

Described main equipment room control server adds carrier aircraft and described data block is written to and adds in secondary server corresponding to carrier aircraft with described from machine room described in passing through.

B12, method according to claim B11, is characterized in that, described main equipment room control server receives the data block that transferring equipment sends, and comprising:

Described main equipment room control server receives multiple data fragmentations of described transferring equipment transmitted in parallel, and described multiple data fragmentation carries out section by described transferring equipment to described data block and obtained.

B13, method according to claim B12, is characterized in that, described main equipment room control server each group of the described data block received being written to respectively in main equipment room adds in carrier aircraft, comprising:

Described main equipment room control server sets up the multiple processes equal with data fragmentation quantity, and wherein each process is responsible for the write operation of a data fragmentation;

By described process, by the data fragmentation of correspondence, each group of being written to respectively in main equipment room adds in carrier aircraft described main equipment room control server.

B14, method according to claim B13, is characterized in that, before described main equipment room control server sets up the multiple processes equal with data fragmentation quantity, described method comprises further:

Described main equipment room control server controls concurrent process number and is not more than preset course number.

B15, method according to claim B13 or B14, it is characterized in that, by described process, by the data fragmentation of correspondence, each group of being written to respectively in main equipment room adds in carrier aircraft described main equipment room control server, comprising:

Described main equipment room control server sets up many threads respectively under each process, and wherein, a thread is responsible at least one write operation adding carrier aircraft;

The carrier aircraft that adds that described main equipment room control server is responsible for described many threads separately by described many thread parallels writes data fragmentation corresponding to described process.

B16, method according to claim B11, is characterized in that, each group that the described data block received is written in main equipment room respectively in described main equipment room control server add in carrier aircraft after, described method comprises further:

Described main equipment room control server judges described data block adds in carrier aircraft described in whether being successfully written to;

If described data block writes unsuccessfully, then described main equipment room control server carries out the rewrite operation of limited number of time to described data block.

B17, method according to claim B11, is characterized in that, described main equipment room control server pass through described in add carrier aircraft and described data block is written to adds in secondary server corresponding to carrier aircraft with described from machine room, comprising:

Described data block to be written to and to carry out speed limit to the described carrier aircraft that adds with the described speed added in secondary server corresponding to carrier aircraft by described main equipment room control server.

B18, method according to claim B11, is characterized in that, described main equipment room control server pass through described in add carrier aircraft and described data block is written to adds in secondary server corresponding to carrier aircraft with described from machine room, comprising:

Described main equipment room control server adopt background synchronization mechanism pass through described in add carrier aircraft described data block be written in described secondary server.

The device of C19, a kind of Data import, is characterized in that, described device comprises:

Writing unit, described data block for being received by described receiving element each group of being written to respectively in main equipment room adds in carrier aircraft, wherein, comprise many groups in described main equipment room and add carrier aircraft, the group number adding carrier aircraft is identical with the number from machine room, and often to add the quantity of carrier aircraft identical from the secondary server quantity machine room with corresponding for group;

Control module, to be written to the described data block that said write unit writes to add in secondary server corresponding to carrier aircraft with described from machine room for adding carrier aircraft described in controlling.

C20, device according to claim C19, it is characterized in that, described receiving element, for receiving multiple data fragmentations of described transferring equipment transmitted in parallel, described multiple data fragmentation carries out section by described transferring equipment to described data block and obtained.

C21, device according to claim C20, it is characterized in that, said write unit, for setting up the multiple processes equal with data fragmentation quantity, wherein each process is responsible for the write operation of a data fragmentation, and by described process, by the data fragmentation of correspondence, each group of being written to respectively in main equipment room adds in carrier aircraft.

C22, device according to claim C21, is characterized in that, said write unit, and for before setting up the multiple processes equal with data fragmentation quantity, control concurrent process number is not more than preset course number.

C23, device according to claim C21 or C22, it is characterized in that, said write unit, for setting up many threads respectively under each process, wherein, article one, thread is responsible for the write operation that at least one adds carrier aircraft, and the carrier aircraft that adds be responsible for separately to described many threads by described many thread parallels writes data fragmentation corresponding to described process.

C24, device according to claim C19, it is characterized in that, described device comprises further:

Judging unit, after each group of being written to respectively in main equipment room of the described data block for being received by described receiving element at said write unit adds in carrier aircraft, judges described data block adds in carrier aircraft described in whether being successfully written to;

Said write unit, for when described judging unit judges that described data block writes unsuccessfully, carries out the rewrite operation of limited number of time to described data block.

C25, device according to claim C19, is characterized in that, described control module, for described data block to be written to and to carry out speed limit to the described carrier aircraft that adds with the described speed added in secondary server corresponding to carrier aircraft.

C26, device according to claim C19, is characterized in that, described control module, to add carrier aircraft and described data block to be written to for adopting and to add in secondary server corresponding to carrier aircraft with described from machine room described in background synchronization mechanism control.

The system of D27, a kind of Data import, is characterized in that, described system comprises:

Transferring equipment, main equipment room and from machine room, comprise main equipment room control server in wherein said main equipment room and many groups add carrier aircraft, describedly from machine room, comprise secondary server, the group number adding carrier aircraft is identical with the number from machine room, and often to add the quantity of carrier aircraft identical from the secondary server quantity machine room with corresponding for group;

Described transferring equipment, for being sent to described main equipment room control server by data block;

Described main equipment room control server, what the described data block for described transferring equipment being sent was written to each group respectively eachly adds in carrier aircraft;

Describedly add carrier aircraft, add in secondary server corresponding to carrier aircraft with described from machine room for the described data block of described main equipment room control server write is written to.

D28, system according to claim D27, is characterized in that, described transferring equipment, for cutting into slices to described data block, obtaining multiple data fragmentation, and give described main equipment room control server by described multiple data fragmentation transmitted in parallel.

D29, system according to claim D28, it is characterized in that, described main equipment room control server, for setting up the multiple processes equal with data fragmentation quantity, wherein each process is responsible for the write operation of a data fragmentation, and what the data fragmentation of correspondence is written to respectively each group by described process eachly adds in carrier aircraft.

D30, system according to claim D29, is characterized in that, described main equipment room control server, and for before setting up the multiple processes equal with data fragmentation quantity, control concurrent process number is not more than preset course number.

D31, system according to claim D29 or D30, it is characterized in that, described main equipment room control server, for setting up many threads respectively under each process, wherein, article one, thread is responsible for the write operation that at least one adds carrier aircraft, writes data fragmentation corresponding to described process by described many thread parallels to the carrier aircraft that adds be responsible for separately.

D32, system according to claim D27, is characterized in that, describedly comprises three tier server further from machine room, and the quantity of described three tier server is identical with the quantity of described secondary server;

Described secondary server, for after the described data block of write, is written to described data block in corresponding three tier server.

D33, system according to claim D27, it is characterized in that, described main equipment room control server, for described data block is written to respectively each group each add in carrier aircraft after, judge described data block adds in carrier aircraft described in whether being successfully written to, if judge, described data block writes unsuccessfully, then described data block is carried out to the rewrite operation of limited number of time.

D34, system according to claim D27, it is characterized in that, described secondary server, for before the described data block of write, the described data mode added in carrier aircraft is detected, if the data added in carrier aircraft change described in finding, then add data block described in carrier aircraft request of loading to described.

D35, system according to claim D27, is characterized in that, described main equipment room control server, carrying out speed limit for loading described data block to described secondary server.

D36, system according to claim D27, is characterized in that, described main equipment room control server, adds carrier aircraft described data block be written in described secondary server for adopting described in background synchronization mechanism control.

In the above-described embodiments, the description of each embodiment is all emphasized particularly on different fields, in certain embodiment, there is no the part described in detail, can see the associated description of other embodiments.

Be understandable that, the correlated characteristic in said method and device can reference mutually.In addition, " first ", " second " in above-described embodiment etc. are for distinguishing each embodiment, and do not represent the quality of each embodiment.

Those skilled in the art can be well understood to, and for convenience and simplicity of description, the system of foregoing description, the specific works process of device and unit, with reference to the corresponding process in preceding method embodiment, can not repeat them here.

Intrinsic not relevant to any certain computer, virtual system or miscellaneous equipment with display at this algorithm provided.Various general-purpose system also can with use based on together with this teaching.According to description above, the structure constructed required by this type systematic is apparent.In addition, the present invention is not also for any certain programmed language.It should be understood that and various programming language can be utilized to realize content of the present invention described here, and the description done language-specific is above to disclose preferred forms of the present invention.

In instructions provided herein, describe a large amount of detail.But can understand, embodiments of the invention can be put into practice when not having these details.In some instances, be not shown specifically known method, structure and technology, so that not fuzzy understanding of this description.

Similarly, be to be understood that, in order to simplify the disclosure and to help to understand in each inventive aspect one or more, in the description above to exemplary embodiment of the present invention, each feature of the present invention is grouped together in single embodiment, figure or the description to it sometimes.But, the method for the disclosure should be construed to the following intention of reflection: namely the present invention for required protection requires feature more more than the feature clearly recorded in each claim.Or rather, as claims below reflect, all features of disclosed single embodiment before inventive aspect is to be less than.Therefore, the claims following embodiment are incorporated to this embodiment thus clearly, and wherein each claim itself is as independent embodiment of the present invention.

Those skilled in the art are appreciated that and adaptively can change the module in the equipment in embodiment and they are arranged in one or more equipment different from this embodiment.Module in embodiment or unit or assembly can be combined into a module or unit or assembly, and multiple submodule or subelement or sub-component can be put them in addition.Except at least some in such feature and/or process or unit be mutually repel except, any combination can be adopted to combine all processes of all features disclosed in this instructions (comprising adjoint claim, summary and accompanying drawing) and so disclosed any method or equipment or unit.Unless expressly stated otherwise, each feature disclosed in this instructions (comprising adjoint claim, summary and accompanying drawing) can by providing identical, alternative features that is equivalent or similar object replaces.

In addition, those skilled in the art can understand, although embodiments more described herein to comprise in other embodiment some included feature instead of further feature, the combination of the feature of different embodiment means and to be within scope of the present invention and to form different embodiments.Such as, in the following claims, the one of any of embodiment required for protection can use with arbitrary array mode.

All parts embodiment of the present invention with hardware implementing, or can realize with the software module run on one or more processor, or realizes with their combination.It will be understood by those of skill in the art that the some or all functions of the some or all parts in the denomination of invention (as determined the device of website internal chaining grade) that microprocessor or digital signal processor (DSP) can be used in practice to realize according to the embodiment of the present invention.The present invention can also be embodied as part or all equipment for performing method as described herein or device program (such as, computer program and computer program).Realizing program of the present invention and can store on a computer-readable medium like this, or the form of one or more signal can be had.Such signal can be downloaded from internet website and obtain, or provides on carrier signal, or provides with any other form.

The present invention will be described instead of limit the invention to it should be noted above-described embodiment, and those skilled in the art can design alternative embodiment when not departing from the scope of claims.In the claims, any reference symbol between bracket should be configured to limitations on claims.Word " comprises " not to be got rid of existence and does not arrange element in the claims or step.Word "a" or "an" before being positioned at element is not got rid of and be there is multiple such element.The present invention can by means of including the hardware of some different elements and realizing by means of the computing machine of suitably programming.In the unit claim listing some devices, several in these devices can be carry out imbody by same hardware branch.Word first, second and third-class use do not represent any order.Can be title by these word explanations.

Claims

1. a method for Data import, is characterized in that, described method comprises:

2. method according to claim 1, is characterized in that, is describedly sent in main equipment room by the data block in transferring equipment, comprising:

Described data block is cut into slices, obtains multiple data fragmentation;

3. method according to claim 2, is characterized in that, described described data block is written to respectively each group by described main equipment room eachly add in carrier aircraft, comprising:

4. method according to claim 3, is characterized in that, set up the multiple processes equal with data fragmentation quantity in described main equipment room before, described method comprises further:

5. the method according to claim 3 or 4, is characterized in that, described the data fragmentation of correspondence is written to respectively each group by described process eachly add in carrier aircraft, comprising:

6. a method for Data import, is characterized in that, described method comprises:

7. method according to claim 6, is characterized in that, described main equipment room control server receives the data block that transferring equipment sends, and comprising:

8. method according to claim 7, is characterized in that, described main equipment room control server each group of the described data block received being written to respectively in main equipment room adds in carrier aircraft, comprising:

9. a device for Data import, is characterized in that, described device comprises:

10. a system for Data import, is characterized in that, described system comprises: