CN105991769B

CN105991769B - Data migration method and device, edge node server

Info

Publication number: CN105991769B
Application number: CN201610224612.8A
Authority: CN
Inventors: 葛明雷; 吴林峰; 康凯
Original assignee: Shanghai Supreme Being Joins Information Technology Share Co Ltd
Current assignee: Shanghai Supreme Being Joins Information Technology Share Co Ltd
Priority date: 2016-04-12
Filing date: 2016-04-12
Publication date: 2019-04-16
Anticipated expiration: 2036-04-12
Also published as: CN105991769A

Abstract

Data migration method and device, edge node server, which comprises be in the data migration task in unfinished state from being obtained from preset database and be inserted into Data Migration list；The data migration task for being in unfinished state is obtained from data migration task list, and judges that the corresponding resource file of the data migration task of the unfinished state whether there is；When it is present, the modification time of corresponding resource file is revised as modification time of corresponding resource file when according to preset second caching mapping relations storage；When it be not present, the edge node server where corresponding resource file is determined according to default second caching mapping relations by the second load balancer, corresponding resource file is grabbed from identified edge node server and is stored.Above-mentioned scheme can improve the efficiency of Data Migration in edge cluster dilatation, save resource.

Description

Data migration method and device, edge node server

Technical field

The present invention relates to content distributing network fields, more particularly to a kind of data migration method and device, fringe node Server.

Background technique

Content distributing network (Content Delivery Network, CDN), by placing node serve everywhere in network Device constructs one layer of intelligent virtual network on existing Internet basic, the content of website is published to closest to user's Network " edge ", allows user to obtain required content nearby, to alleviate the situation of network congestion, improves the response speed of website Degree.Technically comprehensively solve causes user to visit due to network bandwidth is small, user's amount of access is big, network point distribution is unequal Ask the response speed of website slow problem.

The service cluster that edge cluster is made of more edge of table nodes and load balancer.Wherein, fringe node is responsible for Storing data and the data cached to client offer, load balancer then realize edge according to preset load-balancing algorithm The load balancing of node server.

With the increase of the data volume cached in edge node server, amount of access increases, in order to improve the sound of website Speed is answered, needs to carry out dilatation to edge cluster, i.e., increases new edge node server in edge cluster.But right During edge cluster carries out dilatation, the caching mapping relations between original resource file and edge node server occur Variation, it is therefore desirable to which cache resource file is migrated.

But existing data migration method, at edge when cluster dilatation, edge node server is directly stored from upper layer Data are grabbed in device, not only increase the load and bandwidth of upper layer memory, and the Data Migration time is longer, there is efficiency The problem of low and wasting of resources.

Summary of the invention

The embodiment of the present invention solves the problems, such as be that the efficiency of Data Migration how is improved in edge cluster dilatation, save money Source.

To solve the above problems, the embodiment of the invention provides a kind of data migration method, for increasing in edge cluster Resource file is migrated when fringing node serve, which comprises be obtained from from preset database be in The data migration task of unfinished state is simultaneously inserted into Data Migration list；Wherein, the data migration task includes corresponding to Resource file corresponding URL when according to preset first caching mapping relations storage information, the first caching mapping Relationship in the edge cluster by increasing between edge node server back edge node server and the resource file cached Caching mapping relations；The data migration task for being in unfinished state is obtained from the data migration task list, and is sentenced The corresponding resource file of data migration task of the unfinished state of breaking whether there is；When determining that corresponding resource file exists When, the modification time of the corresponding resource file grabbed is revised as corresponding resource file according to preset second caching Modification time when mapping relations store；Wherein, the second caching mapping relations are that the edge cluster increases fringe node Edge node server and the caching mapping relations of all resource files cached before server；When determining corresponding money In the absence of source file, corresponding resource file institute is determined according to default second caching mapping relations by the second load balancer Edge node server, corresponding resource file is grabbed from identified edge node server and is stored, and by institute The modification time of the corresponding resource file of crawl is revised as corresponding resource file according to the second caching mapping relations Modification time when storage；Wherein, the second caching mapping relations are stored in second load balancer.

Optionally, described that data migration task and the insertion being in unfinished state are obtained from from preset database Into Data Migration list, comprising: obtain the data migration task for being in unfinished state from the database；Based on described The data migration task of unfinished state determines the information of corresponding resource file；Institute is sent to the first load-balanced server The URL acquisition request information of determining resource file, so that first load balancer returns to identified resource file URL information；Wherein, the first caching mapping relations are stored in first load balancer；From the URL returned In parse corresponding IP address, and judge whether parsed IP address consistent with the IP address of itself；As the IP of itself When address determines consistent with the IP address parsed, the data migration task of the unfinished state is added to the data It migrates in task list.

Optionally, the method also includes: by the status maintenance of the data migration task in the data migration task list It is changed to complete；Corresponding state is sent to data migration server and updates request, so that the data migration server is by institute The status modifier of data migration task described in database is stated to complete.

Optionally, before being stored the resource file grabbed, the method also includes: to the resource grabbed File verify and determination is verified.

Optionally, the method also includes: be obtained from from the database be in completion status Data Migration appoint Business, and delete task as data and be inserted into data deletion task list；It is deleted in task list from the data and obtains number According to the task of deletion, and determine the information of corresponding resource file；Determine that corresponding resource file exists by the first load balancer The information of edge node server where when being stored according to the first caching mapping relations；In first load balancer It is stored with the first caching mapping relations；When determining itself is identified edge node server, deleted from the data Task is deleted except the data are deleted in task list；It, will when determining itself not is identified edge node server Corresponding resource file is deleted；State is sent to data migration server and updates request, so that the data migration server The state of the data migration task of corresponding completion status is set as deleting successfully.

Optionally, described that the data migration task being in completion status is obtained from from database, and deleted as data It is deleted in task list except task is inserted into data, comprising: obtain the Data Migration for being in completion status from the database Task, and determine the information of resource file corresponding with acquired data migration task；Pass through second load balancer Determine the letter of edge node server of the corresponding resource file where when according to preset second caching mapping relations storage Breath；When determining itself is identified edge node server, acquired data migration task is deleted as data and is appointed Business is inserted into the data and deletes in task list.

The embodiment of the invention also provides a kind of data migration devices, for increasing fringe node service in edge cluster When resource file is migrated, described device includes: first acquisition unit, is in suitable for being obtained from from preset database In unfinished state data migration task and be inserted into Data Migration list；Wherein, the data migration task includes pair The information of the resource file answered corresponding URL when according to preset first caching mapping relations storage, first caching reflect Penetrate relationship by increase in the edge cluster edge node server back edge node server and the resource file that caches it Between caching mapping relations；First judging unit is suitable for obtaining from the data migration task list in unfinished state Data migration task, and judge that the corresponding resource file of the data migration task of the unfinished state whether there is；Data Migration units are suitable in the presence of determining corresponding resource file, and the modification time of grabbed corresponding resource file is modified For modification time of corresponding resource file when according to preset second caching mapping relations storage；Wherein, described second is slow The edge node server before mapping relations increase edge node server by the edge cluster is deposited to own with what is cached The caching mapping relations of resource file；In the absence of determining corresponding resource file, by the second load balancer according to pre- If the second caching mapping relations determine the edge node server where corresponding resource file, taken from identified fringe node Corresponding resource file is grabbed in business device and is stored, and the modification time of the corresponding resource file grabbed is revised as corresponding to Resource file according to it is described second caching mapping relations storage when modification time；Wherein, second load balancer In be stored with it is described second caching mapping relations.

Optionally, the first acquisition unit is suitable for obtaining the Data Migration for being in unfinished state from the database Task；Based on the data migration task of the unfinished state, the information of corresponding resource file is determined；To the first load balancing Server sends the URL acquisition request information of identified resource file, so that first load balancer returns to institute really The information of the URL of fixed resource file；Wherein, the first caching mapping relations are stored in first load balancer； Parse corresponding IP address from the URL returned, and judge parsed IP address and the IP address of itself whether one It causes；When the IP address of itself determines consistent with the IP address parsed, by the data migration task of the unfinished state It is added in the data migration task list.

Optionally, described device further include: state updating unit, by the Data Migration in the data migration task list The status modifier of task is to complete；State updates request unit, is suitable for sending corresponding state update to data migration server Request, so that the status modifier of data migration task described in the database is to complete by the data migration server.

Optionally, the data migration unit is suitable for verifying to the resource file grabbed, and determines that verifying is logical The out-of-date resource file that will be grabbed stores.

Optionally, described device further include: second acquisition unit is in suitable for being obtained from from the database in completion The data migration task of state, and delete task as data and be inserted into data deletion task list；First determination unit is fitted Data deletion task is obtained in task list in deleting from the data, and determines the information of corresponding resource file；Second really Order member, suitable for determining corresponding resource file according to the first caching mapping relations storage by the first load balancer When where edge node server information；The first caching mapping relations are stored in first load balancer； Data delete unit, suitable for deleting task list from the data when determining itself is identified edge node server Data described in middle deletion delete task；When determining itself not is identified edge node server, by corresponding resource File is deleted；State update request is sent to data migration server to described, so that the data migration server will be right The state of the data migration task for the completion status answered is set as deleting successfully.

Optionally, the second acquisition unit is suitable for obtaining the Data Migration in completion status from the database and appoint Business, and determine the information of resource file corresponding with acquired data migration task；It is true by second load balancer The information of edge node server of the fixed corresponding resource file where when according to preset second caching mapping relations storage； When determining itself is identified edge node server, acquired data migration task is deleted into task as data and is inserted Enter to the data and deletes in task list.

The embodiment of the invention also provides a kind of edge node servers, including above-mentioned data migration device.

Compared with prior art, technical solution of the present invention has the advantage that

Above-mentioned scheme is established corresponding Data Migration for corresponding resource file and is appointed in edge cluster dilation process Business obtains the data migration task of itself by edge node server and executes acquired data migration task, thus real The resource file in edge cluster is now buffered in corresponding fringe node service according to the first caching mapping relations after dilatation On device, since above-mentioned process only needs to migrate part resource file, rather than grabbed again from the memory of upper layer Corresponding resource file can save the time of Data Migration, and can mitigate the load and bandwidth resources of upper layer memory, because This, can be improved the efficiency of Data Migration, and save resource.

Detailed description of the invention

Fig. 1 is the flow chart of one of embodiment of the present invention data migration method；

Fig. 2 is the composed structure schematic diagram of the edge cluster in the embodiment of the present invention；

Fig. 3 is the flow chart of the data access request processing method of edge cluster shown in Fig. 2；

Fig. 4 is the flow chart of the acquisition methods of one of embodiment of the present invention data migration task；

Fig. 5 is the flow chart of the processing method of one of embodiment of the present invention data migration task；

Fig. 6 is the flow chart for the acquisition methods that one of embodiment of the present invention data delete task；

Fig. 7 is the flow chart for the processing method that one of embodiment of the present invention data delete task；

Fig. 8 is the structural schematic diagram of one of embodiment of the present invention data migration device.

Specific embodiment

Just as described in the background art, existing data migration method, at edge when cluster dilatation, edge node server is straight Connect and grab data from the memory of upper layer, not only increase the load and bandwidth of upper layer memory, and the Data Migration time compared with The problem of length, there is inefficiency and the wastings of resources.

To solve the above-mentioned problems in the prior art, technical solution used in the embodiment of the present invention passes through in advance by thing Part is associated with the movement of corresponding execution, corresponding dynamic when meeting corresponding trigger condition, to obtain the event automatically Make group and execute, the efficiency of server O&M can be improved, promotes the usage experience of user.

To make the above purposes, features and advantages of the invention more obvious and understandable, with reference to the accompanying drawing to the present invention Specific embodiment be described in detail.

Fig. 1 shows the flow chart of one of embodiment of the present invention data migration method.It is shown in Figure 1, having During body is implemented, the data migration method in the embodiment of the present invention may include following step:

Step S101: it is obtained from the data migration task being in unfinished state from preset database and is inserted into In Data Migration list.

In specific implementation, the data migration task includes that corresponding resource file is reflected according to preset first caching The information of corresponding URL when relationship storage is penetrated, the first caching mapping relations are to increase fringe node in the edge cluster Caching mapping relations between server back edge node server and the resource file cached.

Step S102: the data migration task for being in unfinished state is obtained from data migration task list, and is judged The corresponding resource file of the data migration task of the unfinished state whether there is.

In specific implementation, before the second caching mapping relations are edge cluster increase edge node server Edge node server and the caching mapping relations of all resource files that are cached of edge cluster.

Step S103:, will be right in the presence of the corresponding resource file of the data migration task for determining the unfinished state The modification time for the resource file answered is revised as corresponding resource file when according to preset second caching mapping relations storage Modification time.

In specific implementation, in the presence of the corresponding resource file of the data migration task for determining the unfinished state, Show corresponding resource file according to first caching mapping relations storage when where edge node server with according to second The edge node server of this node when edge node server where when caching mapping relations storage is equal, at this point, this several The corresponding data of data migration task, which are considered as, to be migrated successfully, at this point it is possible to which the modification time of the corresponding resource file is modified For preset modification time.

Step S104: in the absence of determining corresponding resource file, by the second load balancer according to default second Caching mapping relations determine the edge node server where corresponding resource file, from identified edge node server It grabs corresponding resource file and stores, and the modification time of the corresponding resource file grabbed is revised as corresponding resource Modification time of file when according to the second caching mapping relations storage.

In specific implementation, the second caching mapping relations are stored in second load balancer, i.e., at edge Mapping relations are cached to the fringe node before edge cluster dilatation according to described second by the second load balancer before cluster dilatation Server is scheduled.

Above-mentioned scheme is established corresponding Data Migration for corresponding resource file and is appointed in edge cluster dilation process Business obtains the data migration task of itself by edge node server and executes acquired data migration task, thus real The resource file in edge cluster is now buffered in corresponding fringe node service according to the first caching mapping relations after dilatation On device, since above-mentioned process only needs to migrate part resource file, rather than grabbed again from the memory of upper layer Corresponding resource file can save the time of Data Migration, and can mitigate the load and bandwidth resources of upper layer memory, because This, can be improved the efficiency of Data Migration, and save resource, promote the usage experience of user.

It does below in conjunction with processing method of the Fig. 2 to Fig. 7 to one of embodiment of the present invention data access request into one Walk detailed introduction.

For ease of description, detailed introduction is done to the edge cluster in present invention implementation below in conjunction with Fig. 2.

Shown in Figure 2, in specific implementation, the edge cluster in the embodiment of the present invention may include multiple edge sections Point server, the first load balancer 102 and the second load balancer 103.

In specific implementation, the first caching mapping relations, the second load balancer are stored in the first load balancer 102 The second caching mapping relations are stored in 103.Wherein, the first caching mapping relations are that the edge cluster increases fringe node clothes Caching mapping relations after business device, that is, dilatation between all resource files of all edge node server and caching, second Caching mapping relations be all edge node servers in the leading edge cluster for increase edge node server namely dilatation and The caching mapping relations between all resource files cached.

Continuing with referring to fig. 2, edge cluster has three edge of table node server 101a before dilatation, newly increases after dilatation Edge node server be three edge of table node server 101b.Wherein, in the first load balancer 102 installation edge cluster Six edge of table node servers in a platform on, i.e., from three edge of table node server 101a and three edge of table node servers 101b chooses first load balancer 102 of installation.First load balancer 102 is according to the first caching mapping relations tune It spends six edge of table node servers and returns to requested resource file to client.Second load balancer 103 is installed only at expansion In edge node server before appearance, i.e. in one in three edge of table node server 101a, according to second caching Mapping relations dispatch three edge of table node server 101a and return to requested resource file to client.

Meanwhile before edge cluster dilatation, edge cluster uses second port pair corresponding with the second load balancer The resource file access request of client C is monitored, such as 8080 ports, the resource file access request sent to client It is monitored；After edge cluster dilatation, the first load balancer 102 has been increased in edge cluster newly, and be provided with the first load The corresponding first port of balanced device 102, such as 80 ports, are monitored with the resource file access request sent to client.

It is to be herein pointed out all edges before edge cluster dilatation and completing Data Migration, i.e., after dilatation Before node server is according to corresponding resource file in the first caching mapping relations caching edge cluster, work as user When resource file corresponding by client request, the part resource file in edge cluster may be slow according to described first Mapping relations are deposited to migrate to corresponding edge node server, and another part resource file then may be still according to original Second caching mapping relations are stored in the edge node server before dilatation.At this point, when first port listens to client hair When the resource file access request sent, the first load balancer will be according to all sides after the first caching mapping relations scheduling dilatation Edge node server may result in the fringe node that the access request of client is dispatched to no requested resource file On server, leads to not return to requested resource file to client, affect the usage experience of user.

To solve the problem above-mentioned, to be provided with corresponding data access in all edge node servers in edge cluster Processing unit is requested, to realize the edge where the resource file access request of client to be dispatched to requested resource file On node server, it is ensured that the normal access of the resource cached in edge cluster promotes the usage experience of user, please specifically join See Fig. 3.

It is shown in Figure 3, in specific implementation, the data access request method in the embodiment of the present invention may include as Under step:

Step S301: the resource file access request of client is monitored by first port.

It in specific implementation, can be first by first port when user sends corresponding resource file by client The resource file access request sent to client is monitored.Wherein, in an embodiment of the present invention, the first port is 80 ports.

Step S302: by preset first load balancer according to it is described first caching mapping relations determine it is requested Edge node server where resource file.

In specific implementation, when listening to the resource file access request of client transmission by first port, by institute The resource file access request of the client received is sent to the edge node server for being equipped with the first load balancer, with The edge node server for being equipped with the first load balancer is asked according to the first caching mapping relations return The information of the corresponding uniform resource locator of the resource file asked (Uniform Resource Loc101ator, URL).Data Access request processing unit can carry out obtained URLs information when getting the URL information of the first load balancer return Parsing, the fringe node service where when determining that requested resource file is stored according to the first caching mapping relations Device, and the resource file access request of client is sent to according to fringe node determined by the first caching mapping relations Server.

Step S303: judge requested resource file in the fringe node determined according to the first caching mapping relations It whether there is in server device；When the judgment result is yes, step S304 is executed, conversely, can then execute step S305.

In specific implementation, it is requested according to edge node server determined by the first caching mapping relations receiving Resource file URL information when, the URL of received URL and all resource files itself cached can be carried out It compares one by one, to determine whether there is requested resource file.

Step S304: institute is returned to client from the edge node server determined according to the first caching mapping relations The resource file of request.

In specific implementation, it is determining according to there are institutes in edge node server determined by the first caching mapping relations When the resource file of request, from directly being returned to client according to edge node server determined by the first caching mapping relations Requested resource file.

S305: Xiang Suoshu client of step returns to the address information of preset second load balancer, so that the visitor The resource file access request is sent to second load balancer by preset second port by family end.

In specific implementation, when requested resource file is according to edge determined by the first caching mapping relations In the absence of in node server, when showing that requested resource file is not yet migrated to according to the first caching mapping relations storage In the edge node server at place.At this point, can be to visitor according to the edge node server that the first caching mapping relations determine Family end returns to 302, i.e., the information of the URL of second load balancer is returned to client, so that client will be described Resource file access request is sent to second load balancer by second port.Wherein, in an embodiment of the present invention, The second port is 8080 ports.

Step S306: requested resource text is determined according to the second caching mapping relations by the second load balancer Edge node server where part.

In specific implementation, when listening to the resource file access request of client by second port, data access The resource file access request of client is sent to the fringe node clothes for being equipped with the second load balancer by request processing unit Business device, so that the second load balancer can return to institute to data request processing device according to the second caching mapping relations The information of the corresponding URL of the resource file of request.Data access request processing unit is getting the return of the second load balancer URL information when, obtained URLs information can be parsed, to determine requested resource file according to described second Cache the information of the edge node server where when mapping relations store.

Step S307: judge to whether there is according in edge node server determined by the second caching mapping relations Requested resource file；When the judgment result is yes, step S308 can be executed；Conversely, can then execute step S309.

In specific implementation, it is requested according to edge node server determined by the second caching mapping relations receiving Resource file URL information when, received URL can be buffered in itself with according to the second caching mapping relations The URL of all resource files is compared, to determine according in edge node server determined by the second caching mapping relations With the presence or absence of requested resource file.

Step S308: from being returned according to edge node server determined by the second caching mapping relations to client Requested resource file.

In specific implementation, it is determining according to there are institutes in edge node server determined by the second caching mapping relations When the resource file of request, from directly being returned to client end according to edge node server determined by the second caching mapping relations Return requested resource file.

Step S309: as the upper node layer according to edge node server determined by the second caching mapping relations Server returns to requested resource file to client.

In specific implementation, when requested resource file is according to edge determined by the second caching mapping relations In node server still in the absence of, show that client requested resource file is not present in edge cluster.At this point, pressing Corresponding return code 302 can be returned to client according to the edge node server that the second caching mapping relations determine, i.e., to visitor Family end sends the URL of the upper node layer server according to edge node server determined by the second caching mapping relations Information so that client by the resource file access request be sent to the upper node layer server i.e. upper layer store In server, as other edge clusters where upper layer storage server determines requested resource file, and by identified Other edge clusters return to requested resource file to client, to realize the successful access of resource file.

It, can be in edge collection by above-mentioned introduction it is found that data access request processing method in the embodiment of the present invention During group's dilatation, the resource file access request of client is accurately dispatched to the side where requested resource file On edge node server, so as to provide normal data access service for user, the usage experience of user is promoted.

Edge cluster described in Fig. 2 when carrying out Data Migration, edge node server can using existing method from Corresponding resource file is directly grabbed in the storage server of upper layer to be stored.But it is directly grabbed from the storage server of upper layer It fetches evidence, the problem of following three aspect will be present:

First, directly grabbing whole resource files from the memory of upper layer, more storage resource will be expended in this way, needed It to be the problem of upper layer storage server is equipped with corresponding bandwidth resources, and there is the wastes of bandwidth resources.

Second, directly grabbing corresponding resource file from the storage server of upper layer, that is to say all to grab originally The resource file taken is grabbed from upper layer memory again again, this will expend longer time, and accordingly, there exist data migration efficiencies Low problem.

Second, needing when edge node server is when grabbing corresponding resource file and storing by the clothes of edge cluster Business is closed and is restarted, and to close the second load balancer, and the edge cluster after restarting is made to pass through the first load balancing Device dispatches the resource file access request that client is sent.During above-mentioned, the interruption of service certainly will be will cause, so that The resource file access request of client will be unable to dispatch during this period, affect the usage experience of user.

For this purpose, before carrying out Data Migration, being generated and edge cluster the embodiment of the invention provides a kind of data migration method The interior one-to-one data migration task of each resource file is simultaneously stored in preset database.Wherein, each resource text The corresponding data migration task of part includes the letter of the URL when resource file is stored according to preset first caching mapping relations The information of breath and corresponding execution status of task.Each fringe node service when carrying out Data Migration, in edge cluster Device directly obtains itself corresponding data migration task from database, and by executing corresponding data migration task, crawl Corresponding resource file, to realize that all resource files in edge cluster are deposited according to the first caching mapping relations Storage improves data migration efficiency, and promote the usage experience of user so as to save the bandwidth of upper layer node server, has Body refers to Fig. 4 to Fig. 7.

For ease of description, data migration task how will be obtained below, how to execute acquired data migration task With three aspects of deletion of redundant data, the data migration method in the embodiment of the present invention is further described in detail.

Fig. 4 shows the flow chart of the acquisition methods of one of embodiment of the present invention data migration task.Refer to Fig. 4 Shown, in specific implementation, the acquisition methods of the data migration task in the embodiment of the present invention may include following step:

Step S401: the data migration task for being in unfinished state is obtained from database.

In specific implementation, the edge node server of this node can be sent according to the preset time cycle to database Job enquiry request to obtain the data migration task for being in unfinished state in database, and is stored in the memory of itself. Wherein, the preset time cycle can be configured according to the actual needs, such as be set as 3 hours etc., and the present invention exists This is with no restrictions.

Step S402: resource file corresponding with acquired data migration task is requested to the first load balancer The information of URL.

It in specific implementation, can be from corresponding data migration task when getting corresponding data migration task It determines the information of corresponding resource file, and sends the GET for obtaining the URL of identified resource file to the first load balancer Request, to send the corresponding URL request of identified resource file to the first load balancer.First load balancer is receiving When to the GET request, according to the first caching mapping relations stored be calculated determined by respective file in edge collection The information of corresponding URL after group's dilatation.

Step S403: the information of corresponding IP address is parsed from requested URL information.

In specific implementation, edge node server, can be with when receiving the URL information of the first load balancer return The URL information returned is parsed, with the corresponding resource file of determination according to the first caching mapping relations storage When where node server IP address information.

Step S404: whether the parsed IP address of judgement is consistent with the IP address of itself；When the judgment result is yes, Step S405 can be executed；Conversely, then can restart to execute from step S401.

Step S405: acquired data migration task is added in the data migration task list of itself.

In specific implementation, the edge node server of this node can by the IP address parsed and the IP of itself Location is compared, and when determining that the two is consistent, illustrates corresponding resource file according to the first caching mapping relations storage when institute Edge node server be exactly this node, at this point, the edge node server of this node can by the Data Migration appoint Business is added in the Data Migration list of itself, to pass through the data migration task executed in Data Migration list, from edge collection Corresponding resource file is grabbed in edge node server in group before dilatation.

Conversely, the edge node server of this node is determining that parsed IP address and itself IP address are inconsistent When, illustrate that edge node server of the corresponding resource file where when according to the first caching mapping relations storage is not this section Point.At this point, edge node server can delete the data migration task.

In specific implementation, when obtaining the Data Migration of this node by data migration task acquisition methods shown in Fig. 4 Task is simultaneously added to after the data migration task list of itself, can be by executing the number in the data migration task list It according to migration task, is stored to grab corresponding resource file, to realize that corresponding resource file is mapped according to the first caching The storage of relationship, specifically refers to Fig. 5.

Fig. 5 shows the flow chart of one of present invention implementation data migration task processing method.As shown in figure 5, this The processing method of data migration task in inventive embodiments may include following step:

Step S501: the data migration task for being in unfinished state is obtained from the Data Migration list of itself.

In specific implementation, the edge node server of this node can be obtained from Data Migration list in different ways Corresponding data migration task is handled.For example, the edge node server of this node can by the way of single thread, one Secondary one data migration task in unfinished state of the acquisition from Data Migration list is handled；Alternatively, can also adopt The mode of multithreading once obtains a plurality of data migration task in unfinished state simultaneously and is handled, with Data Migration Improve efficiency.

Step S502: judge that the acquired corresponding resource file of data migration task whether there is itself；Work as judgement It as a result is that when being, step S503 and S504 can be executed；Conversely, can then execute step S505.

In specific implementation, the edge node server of this node can be according to the acquired number in unfinished state According to the information of migration task, determine the information of corresponding resource file, i.e., the information of the URL of corresponding resource file, and by institute Determine that the URL for the resource file that the URL of corresponding resource file is stored with itself is compared, to determine acquired data The corresponding resource file of migration task whether there is itself.

Step S503: the modification time of corresponding resource file is revised as depositing according to preset second caching mapping relations Modification time when storage.

In specific implementation, the edge node server of this node is determining the acquired corresponding money of data migration task Source file in the presence of itself, illustrate data according to first caching mapping relations storage when where edge node server with Edge node server where when storing according to the second caching mapping relations is the edge node server of this node.Cause This, will be without grabbing the resource file from other edge node servers, which has been realized according to preset First caching mapping relations are stored, that is, are considered as Data Migration success.At this point, determining acquired data migration task pair The resource file answered can be executed directly in the presence of itself and be revised as taking from upper layer for the first time by the modification time of the resource file The modification time of the resource file is grabbed in business device node, that is, father's memory node, that is, is realized according to the second caching mapping relations storage When modification time.

Step S504: the data migration task is deleted from Data Migration list, is sent to data migration server Task status updates request, so that data migration task corresponding in database has been updated to by the data migration server At.

In specific implementation, when corresponding resource file is considered as and migrates successfully, the edge node server of this node will The data migration task is deleted from Data Migration list, and is sent task status to data migration server and updated request letter Breath, so that the data migration server can set the task status of the corresponding data migration task stored in database It is set to success, to avoid repeating for identical data migration task, and is convenient for holding to the data migration task in database Market condition is controlled.

Step S505: determine corresponding resource file according to the second caching mapping relations by the second load balancer The information of URL when storage.

In specific implementation, the edge node server of this node is determining the acquired corresponding money of data migration task Source file illustrates edge node server of the data where when according to the first caching mapping relations storage in the absence of itself Different from according to the edge node server where when the second caching mapping relations storage, therefore, it is necessary to from according to preset the The resource file is grabbed in edge node server where when two caching mapping relations storages.

At this point, the edge node server of this node can send the resource file according to the second load balancer The information of corresponding URL when two caching mapping relations storages, to be solved from the corresponding URL information that the second load balancer returns The information of the IP address of corresponding edge node server is precipitated, namely determines that the resource file is reflected according to second caching Penetrate the information of the IP address of the edge node server where when relationship stores.

Step S506: corresponding resource file is grabbed from the corresponding edge node server of identified URL.

In specific implementation, the corresponding URL that the second load balancer returns is started with corresponding IP address, the IP Location is the IP address of the edge node server where when the resource file is stored according to the second caching mapping relations.Therefore, The edge node server of this node is when receiving the corresponding URL of second load balancer return, by being returned URL is parsed, and the fringe node service where when the resource file is stored according to the second caching mapping relations can be obtained The IP address of device, and grab the resource file from the corresponding edge node server of the IP address and store.

Step S507: verifying the resource file grabbed, and judges to verify whether by storing grabbed money Source file.

In specific implementation, reliable in order to ensure the resource file that is grabbed when grabbing corresponding resource file Property when, the edge node server of this node can first verify the resource file grabbed, such as execute MD5 verifying, And when being verified, the resource file grabbed is stored, to complete the Data Migration of the resource file, realizing should The storage that resource file caches mapping relations according to first.

When being stored the resource file grabbed, step S503 and S504 can be then executed, by the resource The modification time of file is revised as grabbing the modification time of the resource file from top service device node for the first time, and to by data It migrates corresponding data migration task in task list to delete, while will be counted accordingly in database by data migration server Success is updated to according to the state of migration task.

By above-mentioned introduction it is found that obtaining itself corresponding Data Migration from database by edge node server Task simultaneously executes, and can realize all resource files in edge cluster from the edge node server before dilatation to dilatation The migration of edge node server afterwards.In this way, it on the one hand may not need and grabbed again from the memory node of upper layer Resource file can realize the Data Migration of resource file, therefore, can save the bandwidth of upper layer memory node；Another party Face, due to only needing to carry out the migration of part resource file, without will be already in edge cluster from upper layer memory node In all resource files grab again again one time, therefore, the time of Data Migration can be saved, improve the effect of Data Migration Rate.Meanwhile at the end of above-mentioned Data Migration, i.e., all resource files in realization edge cluster are mapped according to the first caching When relationship stores, without restarting to edge cluster, therefore, it is possible to provide normal data access demand for user, promoted The usage experience of user.

In specific implementation, when carrying out Data Migration completion, due to the edge node server before edge cluster dilatation In be still stored with the duplicate data in part, the duplicate data in the part are to be stored in this according to the second caching mapping relations In the edge node server of node, the resource file of the part is stored in other when according to the first caching mapping relations storage The edge node server of node.Therefore, in order to save resource, the duplicate data in the part can be deleted, to save Storage resource specifically refers to Fig. 6 and Fig. 7.

Fig. 6 shows the flow chart that one of embodiment of the present invention data delete the acquisition methods of task.Refer to Fig. 6 Shown, in specific implementation, it may include following step that the data in the embodiment of the present invention, which delete the acquisition methods of task:

Step S601: the data migration task for being in completion status is obtained from database.

In specific implementation, edge node server can send job enquiry to database according to the preset time cycle Request to obtain the data migration task for being in completion status in database, and is stored in the memory of itself.Wherein, described The preset time cycle can be configured according to the actual needs, and the present invention is herein with no restrictions.

Step S602: resource file corresponding with acquired data migration task is requested to the second load balancer The information of URL.

It in specific implementation, can be from correspondence when getting the data migration task for being in completion status in database Data migration task in determine the information of corresponding resource file, and the GET request of URL is sent to the second load balancer, To request the information of the URL of identified corresponding resource file to the second load balancer.Second load balancer is receiving When to the GET request, the information of corresponding URL is calculated according to the second caching mapping relations stored, and return to The edge node server of this node.

Step S603: the information of corresponding IP address is parsed from requested URL information.

In specific implementation, the edge node server of this node is in the URL letter for receiving the return of the second load balancer When breath, the URL information that can be returned is parsed, and is closed with the corresponding resource file of determination according to the second caching mapping The information of the IP address of node server where when system's storage.

Step S604: whether the parsed IP address of judgement is consistent with the IP address of itself；When the judgment result is yes, Step S605 can be executed；Conversely, then can restart to execute from step S601.

Step S605: the data that acquired data migration task is inserted into itself are deleted in task list.

In specific implementation, the edge node server of this node can by the IP address parsed and the IP of itself Location is compared；When determining that the two is consistent, illustrate corresponding resource file according to the second caching mapping relations storage when institute Edge node server be this node, the resource file according to first caching mapping relations storage when where edge section Point server is not this node, thus is needed the resource file from this knot removal.At this point, edge node server can incite somebody to action The data migration task for being in completion status is deleted task as data and is added in the Data Migration list of itself, to pass through The data executed in data delete list delete task, delete extra resource file.

Conversely, the edge node server of this node is determining that parsed IP address and itself IP address are inconsistent When, illustrate that edge node server of the corresponding resource file where when according to the second caching mapping relations storage is not this section Therefore point should be stored in other nodes in the data migration task of completion status, it is therefore not necessary to data deletion is executed, because The data migration task that this can be in completion status by this edge node server is deleted.

Fig. 7 shows the flow chart of one of embodiment of the present invention data-erasure method.It is shown in Figure 7, having During body is implemented, the data-erasure method in the embodiment of the present invention may include following step:

Step S701: the data in unfinished state are obtained from the data delete list of itself and delete task.

In specific implementation, the edge node server of this node can be in different ways from the data delete list of itself It is middle obtain corresponding data and delete task handled.For example, edge node server can be by the way of single thread, once Data deletion task is obtained from data delete list to be handled, and can also be adopted the mode of multithreading, once be obtained simultaneously It takes a plurality of data to delete task to be handled, be improved efficiency with what data were deleted, and then improve the efficiency of Data Migration.

Step S702: to the first load balancer request resource file corresponding with acquired data deletion task The information of URL.

In specific implementation, the edge node server of this node is when the data that processing obtains delete task, Ke Yigen Determine the information of corresponding resource file according to the information that acquired data delete task, and to the request pair of the first load balancer The information of the resource file answered corresponding URL when according to the first caching mapping relations storage.

Step S703: the information of corresponding IP address is parsed from requested URL information.

In specific implementation, the corresponding URL that the first load balancer returns is started with corresponding IP address, the IP Location is the IP address of the edge node server where when the resource file is stored according to the first caching mapping relations.Therefore, The edge node server of this node is when receiving the corresponding URL of second load balancer return, by being returned URL is parsed, and the fringe node service where when the resource file is stored according to the first caching mapping relations can be obtained The IP address of device may thereby determine that the side for parsing to obtain where when the resource file is stored according to the first caching mapping relations Whether the IP address of the edge node server of the IP address and this node of edge node server is consistent.

Step S704: whether the parsed IP address of judgement is consistent with the IP address of itself；When the judgment result is no, Step S705 can be executed；Conversely, then can directly execute step S706.

Step S705: corresponding resource file is deleted.

In specific implementation, where when determining the resource file parsed according to the first caching mapping relations storage Edge node server IP address and this node edge node server IP address it is inconsistent when, then show corresponding Edge node server of the resource file where when according to the first caching mapping relations storage is the edge of other nodes Node server, therefore, the edge node server of this node can be by the corresponding resource file stored from this node It deletes.

Step S706: data deletion task is deleted from data delete list, and is sent out to data migration server Task status is sent to update request, so that the data migration server is by the data in completion status corresponding in database The state of migration task is updated to delete successfully.

In specific implementation, where when determining the resource file parsed according to the first caching mapping relations storage Edge node server IP address it is consistent with the IP address of the edge node server of this node when, then show corresponding money Edge node server of the source file where when according to the first caching mapping relations storage is the fringe node of this node Server needs to retain corresponding file.At this point, being considered as data deletes task execution success, data can be deleted into task It is deleted from data delete list, corresponding data are deleted task execution and are completed.

In specific implementation, where when determining the resource file parsed according to the first caching mapping relations storage Edge node server IP address and this node edge node server IP address it is inconsistent when, then show corresponding Edge node server of the resource file where when according to the first caching mapping relations storage is not the edge of this node Node server at this point it is possible to delete corresponding resource file, and can delete data when resource file is deleted successfully Task is deleted from data delete list, and corresponding data are deleted task execution and completed.

Meanwhile the fringe node service of this node is when determining that corresponding data are deleted task execution and completed, it can should Data are deleted task and are deleted from the data delete list of itself, and send task status update to data migration server and ask Ask so that the data migration server by the state of the data migration task in completion status corresponding in database more New is to delete successfully, to avoid repeating for identical data migration task, and convenient for the data migration task in database Executive condition controlled.

The corresponding data migration device of data migration method in the embodiment of the present invention will be done below further details of It introduces.

Fig. 8 shows the structure of one of embodiment of the present invention data migration device.It is shown in Figure 8, specific In implementation, the data migration device 800 in the embodiment of the present invention may include first acquisition unit 801, the first judging unit 802 With data migration unit 803, in which:

First acquisition unit 801, suitable for being obtained from the Data Migration being in unfinished state from preset database Task is simultaneously inserted into Data Migration list；Wherein, the data migration task includes corresponding resource file according to default The first caching mapping relations storage when corresponding URL information, the first caching mapping relations are in the edge cluster Increase the caching mapping relations between edge node server back edge node server and the resource file cached.

In specific implementation, the first acquisition unit 801, which is suitable for obtaining from the database, is in unfinished state Data migration task；Based on the data migration task of the unfinished state, the information of corresponding resource file is determined；To One load-balanced server sends the URL acquisition request information of identified resource file, so that first load balancing Device returns to the information of the URL of identified resource file；Wherein, first caching is stored in first load balancer Mapping relations；Corresponding IP address is parsed from the URL returned, and with judging parsed IP address and the IP of itself Whether location is consistent；When the IP address of itself determines consistent with the IP address parsed, by the data of the unfinished state Migration task is added in data migration task list.

First judging unit 802, suitable for obtaining the Data Migration for being in unfinished state from data migration task list Task, and judge that the corresponding resource file of the data migration task of the unfinished state whether there is；Wherein, described second is slow The edge node server before mapping relations increase edge node server by the edge cluster is deposited to own with what is cached The caching mapping relations of resource file.

Data migration unit 803 is suitable in the presence of determining corresponding resource file, by the modification of corresponding resource file Time modification is modification time of corresponding resource file when according to preset second caching mapping relations storage；When determining pair In the absence of the resource file answered, corresponding resource is determined according to default second caching mapping relations by the second load balancer Edge node server where file grabs corresponding resource file and is stored from identified edge node server, And the modification time of corresponding resource file is revised as corresponding resource file and is deposited according to the second caching mapping relations Modification time when storage；Wherein, the second caching mapping relations are stored in second load balancer.

In specific implementation, the data migration unit 803 is suitable for verifying to the resource file grabbed, and really Crawl resource file is stored when being verified surely.

In specific implementation, the data migration device 800 in the embodiment of the present invention can also include state updating unit 804 Request unit 805 is updated with state, in which:

State updating unit 804, the status modifier by the data migration task in the data migration task list have been At.

State updates request unit 805, is suitable for sending corresponding state to data migration server updating and request, so that The status modifier of data migration task described in the database is to complete by the data migration server.

In specific implementation, the data migration device 800 in the embodiment of the present invention can also include second acquisition unit 806:

Second acquisition unit 806 suitable for being obtained from the data migration task being in completion status from database, and is made Task is deleted for data to be inserted into data deletion task list.

In specific implementation, the second acquisition unit 806 is suitable for being obtained from the database in completion status Data migration task, and determine the information of resource file corresponding with acquired data migration task；It is negative by described second It carries balanced device and determines that fringe node of the corresponding resource file where when according to preset second caching mapping relations storage takes The information of business device；When determining itself is identified edge node server, using acquired data migration task as number Data are inserted into according to deletion task to delete in task list.

First determination unit 807 is suitable for deleting acquisition data in task list from data and deletes task, and determination is corresponding The information of resource file.

Second determination unit 808, suitable for determining corresponding resource file according to institute by first load balancer State the information of the edge node server where when the first caching mapping relations store.

Data delete unit 809, suitable for being deleted from the data when determining itself is identified edge node server Task is deleted except the data are deleted in task list；It, will when determining itself not is identified edge node server Corresponding resource file is deleted；State update request is sent to data migration server to described, so that Data Migration service The state of the data migration task of corresponding completion status is set as deleting successfully by device.

Those of ordinary skill in the art will appreciate that all or part of the steps in the various methods of above-described embodiment is can It is completed with instructing relevant hardware by program, which can store in computer readable storage medium, and storage is situated between Matter may include: ROM, RAM, disk or CD etc..

The method and system of the embodiment of the present invention are had been described in detail above, the present invention is not limited thereto.Any Field technical staff can make various changes or modifications without departing from the spirit and scope of the present invention, therefore guarantor of the invention Shield range should be defined by the scope defined by the claims..

Claims

1. a kind of data migration method migrates resource file when for increasing fringe node service in edge cluster, It is characterised by comprising:

It is obtained from the data migration task being in unfinished state from preset database and is inserted into data migration task In list；Wherein, the data migration task includes that corresponding resource file is deposited according to preset first caching mapping relations The information of corresponding URL when storage, the first caching mapping relations are after increasing edge node server in the edge cluster Caching mapping relations between edge node server and the resource file cached；

The data migration task for being in unfinished state is obtained from the data migration task list, and judges described do not complete The corresponding resource file of the data migration task of state whether there is；

In the presence of determining corresponding resource file, the modification time of corresponding resource file is revised as corresponding resource file Modification time when according to preset second caching mapping relations storage；Wherein, the second caching mapping relations are described The caching of edge node server and all resource files cached before edge cluster increase edge node server reflects Penetrate relationship；

It is true according to the second caching mapping relations by the second load balancer in the absence of determining corresponding resource file Edge node server where fixed corresponding resource file, grabs corresponding resource from identified edge node server File simultaneously stores, and the modification time of the corresponding resource file grabbed is revised as corresponding resource file according to described Modification time when the second caching mapping relations storage；Wherein, second caching is stored in second load balancer Mapping relations.

2. data migration method according to claim 1, which is characterized in that described to obtain itself from preset database Data migration task in unfinished state is simultaneously inserted into data migration task list, comprising:

The data migration task for being in unfinished state is obtained from the database；

Based on the data migration task of the unfinished state, the information of corresponding resource file is determined；

The URL acquisition request information of identified resource file is sent to the first load balancer, so that first load Balanced device returns to the information of the URL of identified resource file；Wherein, described first is stored in first load balancer Cache mapping relations；

Corresponding IP address is parsed from the URL returned, and judges that parsed IP address and the IP address of itself are It is no consistent；

When the IP address of itself determines consistent with the IP address parsed, by the data migration task of the unfinished state It is added in the data migration task list.

3. data migration method according to claim 1, which is characterized in that further include:

It is to complete by the status modifier of the data migration task in the data migration task list；

Corresponding state is sent to data migration server and updates request, so that the data migration server is by the data The status modifier of data migration task described in library is to complete.

4. data migration method according to claim 1, which is characterized in that stored by the resource file grabbed Before, further includes:

The resource file grabbed verify and determination is verified.

5. data migration method according to claim 1, which is characterized in that further include:

It is obtained from the data migration task being in completion status from the database, and deletes task as data and is inserted into Data are deleted in task list；

It is deleted from the data and obtains data deletion task in task list, and determine the information of corresponding resource file；

Where determining corresponding resource file when according to the first caching mapping relations storage by the first load balancer Edge node server information；The first caching mapping relations are stored in first load balancer；

When determining itself is identified edge node server, is deleted from the data and delete the number in task list According to the task of deletion；When determining itself not is identified edge node server, corresponding resource file is deleted；

State is sent to data migration server and updates request, so that the data migration server is by corresponding completion status The state of data migration task be set as deleting successfully.

6. data migration method according to claim 5, which is characterized in that described be obtained from from the database is in In the data migration task of completion status, and task is deleted as data and is inserted into data deletion task list, comprising:

The data migration task for being in completion status, and determining and acquired data migration task are obtained from the database The information of corresponding resource file；

Determine corresponding resource file when according to the second caching mapping relations storage by second load balancer The information of the edge node server at place；

When determining itself is identified edge node server, acquired data migration task is deleted as data and is appointed Business is inserted into the data and deletes in task list.

7. a kind of data migration device migrates resource file when for increasing fringe node service in edge cluster, It is characterised by comprising:

First acquisition unit, suitable for being obtained from the data migration task being in unfinished state from preset database and inserting Enter into data migration task list；Wherein, the data migration task includes corresponding resource file according to preset The information of corresponding URL when one caching mapping relations storage, the first caching mapping relations are to increase in the edge cluster Caching mapping relations between edge node server back edge node server and the resource file cached；

First judging unit is appointed suitable for obtaining the Data Migration in unfinished state from the data migration task list Business, and judge that the corresponding resource file of the data migration task of the unfinished state whether there is；

Data migration unit is suitable in the presence of determining corresponding resource file, the modification time of corresponding resource file is repaired It is changed to modification time of corresponding resource file when according to preset second caching mapping relations storage, wherein described second Caching mapping relations are by the edge node server before edge cluster increase edge node server and the institute cached There are the caching mapping relations of resource file；

It is true according to default second caching mapping relations by the second load balancer in the absence of determining corresponding resource file Edge node server where fixed corresponding resource file, grabs corresponding resource from identified edge node server File simultaneously stores, and the modification time of the corresponding resource file grabbed is revised as corresponding resource file according to described Modification time when the second caching mapping relations storage；Wherein, second caching is stored in second load balancer Mapping relations.

8. data migration device according to claim 7, which is characterized in that the first acquisition unit is suitable for from the number The data migration task of unfinished state is according to acquisition in library；Based on the data migration task of the unfinished state, determine The information of corresponding resource file；The URL acquisition request information of identified resource file is sent to the first load balancer, with So that first load balancer returns to the information of the URL of identified resource file；Wherein, first load balancer In be stored with it is described first caching mapping relations；Corresponding IP address is parsed from the URL returned, and judges to be parsed IP address it is whether consistent with the IP address of itself；It, will when the IP address of itself determines consistent with the IP address parsed The data migration task of the unfinished state is added in the data migration task list.

9. data migration device according to claim 7, which is characterized in that further include:

The status modifier of data migration task in the data migration task list is to complete by state updating unit；

State updates request unit, is suitable for sending corresponding state to data migration server updating and request, so that the number The status modifier of data migration task described in the database is completed according to migration server.

10. data migration device according to claim 7, which is characterized in that the data migration unit is suitable for institute The resource file of crawl is verified, and is determined and stored the resource file grabbed when being verified.

11. data migration device according to claim 7, which is characterized in that further include:

Second acquisition unit, suitable for being obtained from the data migration task being in completion status from database, and as data Deletion task is inserted into data and deletes in task list；

First determination unit is suitable for deleting the data deletion task that obtains in task list, and determining corresponding money from the data The information of source file；

Second determination unit, suitable for determining that corresponding resource file is reflected according to first caching by the first load balancer Penetrate the information of the edge node server where when relationship stores；First caching is stored in first load balancer Mapping relations；

Data delete unit, suitable for deleting task from the data when determining itself is identified edge node server The data are deleted in list deletes task；It, will be corresponding when determining itself not is identified edge node server Resource file is deleted；State is sent to data migration server and updates request, so that the data migration server will correspond to The state of data migration task of completion status be set as deleting successfully.

12. data migration device according to claim 11, which is characterized in that the second acquisition unit is suitable for from described The data migration task for being in completion status is obtained in database, and determines resource corresponding with acquired data migration task The information of file；Determine that corresponding resource file is closed according to preset second caching mapping by second load balancer The information of edge node server where when system's storage；When determining itself is identified edge node server, by institute The data migration task of acquisition is deleted task as data and is inserted into the data deletion task list.

13. a kind of edge node server, which is characterized in that including the described in any item Data Migration dresses of claim 7-12 It sets.