CN115994125A

CN115994125A - Stored data processing method and related device

Info

Publication number: CN115994125A
Application number: CN202111222188.0A
Authority: CN
Inventors: 李陆彪; 杨绍鹏
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-10-20
Filing date: 2021-10-20
Publication date: 2023-04-21

Abstract

The application discloses a processing method of stored data and a related device. Determining a cold standby system corresponding to the stored data based on a storage environment of a data source end, so as to obtain cold standby data corresponding to the stored data through the cold standby system; then transmitting the cold standby data from the data source end to the migration target end; controlling a migration target end to process the cold standby data to obtain processed data; and writing the processing data into the migration target according to a preset rule, wherein the preset rule is determined based on the storage environment of the migration target. Therefore, in the efficient migration process of the storage number crossing the clusters, the cold standby data is used as a data source for migration, so that the influence of the network state between the data source end and the migration target end on the data migration is avoided, the method can be adapted to different storage systems, and the efficiency of the migration of the storage data crossing the clusters is improved.

Description

Stored data processing method and related device

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a method and an apparatus for processing stored data.

Background

With the rapid development of internet technology, people have increasingly high requirements on data content, so that corresponding storage systems are required to provide services. The cross-cluster data migration of a storage system is different from the intra-cluster data migration, and the clusters are usually not communicated with a network before the clusters, or have high network delay and poor quality. And the scene of the cross-cluster data migration is generally that the data of the whole service are migrated together, the data volume is large, the migration time is required to be short, and the service operation and maintenance are very challenging.

Generally, in the cross-cluster data migration process, data is traversed from a data migration source end, and the traversed data is written into a data migration target end.

However, the whole process of traversing the data is heavy, complex, inflexible and low in multiplexing rate of migration tools. For cross-region and even cross-continent data migration tasks, the whole process takes quite long time due to high network time delay, and the efficiency of cross-cluster storage data migration is affected.

Disclosure of Invention

In view of this, the present application provides a method for processing storage data, which can effectively improve the efficiency of data migration across clusters.

The first aspect of the present application provides a method for processing stored data, which may be applied to a system or a program including a processing function of stored data in a terminal device, and specifically includes:

determining a cold standby system corresponding to storage data based on a storage environment configured by a data source end, so as to acquire the cold standby data corresponding to the storage data through the cold standby system;

transmitting the cold standby data from the data source end to a migration target end, wherein a storage cluster corresponding to the data source end is different from a storage cluster corresponding to the migration target end;

controlling the migration target end to process the cold standby data to obtain processed data;

And writing the processing data into the migration target according to a preset rule, wherein the preset rule is determined based on a storage environment configured by the migration target.

Optionally, in some possible implementations of the present application, the transmitting the cold standby data from the data source end to the migration destination end includes:

acquiring network transmission parameters between the data source end and the migration target end;

compressing the cold standby data based on the network transmission parameters to obtain compressed data;

dividing the compressed data according to a preset dividing granularity to obtain a plurality of divided data;

and transmitting the segmentation data from the data source end to the migration target end through a concurrent channel.

Optionally, in some possible implementations of the present application, the method further includes:

monitoring the network transmission parameters between the data source end and the migration target end;

if the network transmission parameters meet the transmission conditions, the cold standby data are segmented according to the preset segmentation granularity to obtain a plurality of transmission data;

and transmitting the transmission data from the data source end to the migration target end through the concurrent channel.

Optionally, in some possible implementations of the present application, the transmitting the segmentation data from the cold standby system to the migration target includes:

if the network transmission parameters indicate that the network between the data source end and the migration target end is abnormal, transmitting the segmentation data from the data source end to a transfer cluster;

and outputting the segmentation data to the migration target through the transfer cluster.

Optionally, in some possible implementations of the present application, writing the processing data to the migration target according to a preset rule includes:

comparing the cluster distribution of the data source end with the cluster distribution of the migration target end;

if the cluster distribution of the data source end is the same as the cluster distribution of the migration target end, determining that the preset rule is a physical replacement mode;

writing the processing data into the migration target end in the physical replacement mode;

or;

if the cluster distribution of the data source end is the same as the cluster distribution of the migration target end, determining a write-in interface of the migration target end based on the preset rule;

and adapting the processing data with the migration target terminal based on the writing interface so as to write the processing data into the migration target terminal.

Optionally, in some possible implementations of the present application, the determining, based on a storage environment configured by a data source side, a cold standby system corresponding to storage data, so as to obtain, by using the cold standby system, cold standby data corresponding to the storage data includes:

acquiring a storage environment configured by the data source end in response to a bill of lading operation in a target interface, and acquiring the storage environment configured by the migration target end based on the bill of lading operation;

determining a cold standby system corresponding to the stored data according to a storage environment configured by the data source end, so as to obtain the cold standby data corresponding to the stored data through the cold standby system;

the writing the processing data into the migration target according to a preset rule includes:

determining the preset rule based on the storage environment configured by the data source end and the storage environment configured by the migration target end;

and writing the processing data into the migration target terminal according to the preset rule.

Optionally, in some possible implementations of the present application, the stored data includes a first file and a second file, and the method further includes:

acquiring the first file through the cold standby system;

The first file is transmitted from the data source end to the migration target end, and the second file is obtained through the cold standby system;

the migration target end is controlled to process the first file to obtain first file data, and the second file is transmitted to the migration target end from the data source end;

the first file data is written into the migration target end according to the preset rule, and the migration target end is controlled to process the second file to obtain second file data;

and writing the second file data into the migration target terminal according to the preset rule.

A second aspect of the present application provides a processing apparatus for storing data, including:

the system comprises an acquisition unit, a storage unit and a control unit, wherein the acquisition unit is used for determining a cold standby system corresponding to storage data based on a storage environment configured by a data source end so as to acquire the cold standby data corresponding to the storage data through the cold standby system;

the transmission unit is used for transmitting the cold standby data from the data source end to the migration target end, and the storage cluster corresponding to the data source end is different from the storage cluster corresponding to the migration target end;

the processing unit is used for controlling the migration target end to process the cold standby data to obtain processed data;

The processing unit is further configured to write the processing data into the migration target according to a preset rule, where the preset rule is determined based on a storage environment configured by the migration target.

Optionally, in some possible implementation manners of the present application, the transmission unit is specifically configured to obtain a network transmission parameter between the data source end and the migration destination end;

the transmission unit is specifically configured to compress the cold standby data based on the network transmission parameter to obtain compressed data;

the transmission unit is specifically configured to segment the compressed data according to a preset segmentation granularity, so as to obtain a plurality of segmentation data;

the transmission unit is specifically configured to transmit the sliced data from the data source end to the migration destination end through a concurrent channel.

Optionally, in some possible implementations of the present application, the transmission unit is specifically configured to monitor the network transmission parameter between the data source end and the migration destination end;

the transmission unit is specifically configured to segment the cold standby data according to the preset segmentation granularity if the network transmission parameter meets a transmission condition, so as to obtain a plurality of transmission data;

The transmission unit is specifically configured to transmit the transmission data from the data source end to the migration destination end through the concurrency channel.

Optionally, in some possible implementation manners of the present application, the transmission unit is specifically configured to transmit the split data from the data source end to a relay cluster if the network transmission parameter indicates that a network between the data source end and the migration target end is abnormal;

the transmission unit is specifically configured to transmit the segmentation data to the migration target through the transit cluster.

Optionally, in some possible implementation manners of the present application, the processing unit is specifically configured to compare a cluster distribution of the data source end with a cluster distribution of the migration target end;

the processing unit is specifically configured to determine that the preset rule is a physical replacement mode if the cluster distribution of the data source end is the same as the cluster distribution of the migration target end;

the processing unit is specifically configured to write the processing data into the migration target end in the physical replacement manner;

or;

the processing unit is specifically configured to determine a write interface of the migration target based on the preset rule if the cluster distribution of the data source end is the same as the cluster distribution of the migration target end;

The processing unit is specifically configured to adapt the processing data to the migration target based on the write interface, so as to write the processing data into the migration target.

Optionally, in some possible implementation manners of the present application, the obtaining unit is specifically configured to obtain a storage environment configured by the data source end in response to a bill of lading operation in a target interface, and obtain the storage environment configured by the migration target end based on the bill of lading operation;

the acquisition unit is specifically configured to determine a cold standby system corresponding to the stored data according to a storage environment configured by the data source end, so as to acquire the cold standby data corresponding to the stored data through the cold standby system;

the processing unit is specifically configured to determine the preset rule based on the storage environment configured by the data source end and the storage environment configured by the migration target end;

the processing unit is specifically configured to write the processing data into the migration target according to the preset rule.

Optionally, in some possible implementations of the present application, the processing unit is specifically configured to obtain the first file through the cold standby system;

The processing unit is specifically configured to obtain the second file through the cold standby system while transmitting the first file from the data source end to the migration target end;

the processing unit is specifically configured to control the migration target end to process the first file to obtain first file data, and simultaneously transmit the second file from the data source end to the migration target end;

the processing unit is specifically configured to control the migration target end to process the second file to obtain second file data while writing the first file data into the migration target end according to the preset rule;

the processing unit is specifically configured to write the second file data into the migration target according to the preset rule.

A third aspect of the present application provides a computer device comprising: a memory, a processor, and a bus system; the memory is used for storing program codes; the processor is configured to execute the method for processing stored data according to the first aspect or any one of the first aspects according to instructions in the program code.

A fourth aspect of the present application provides a computer readable storage medium having instructions stored therein which, when run on a computer, cause the computer to perform the method of processing stored data of the first aspect or any one of the first aspects described above.

According to one aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device performs the processing method of the stored data provided in the above-mentioned first aspect or various alternative implementations of the first aspect.

From the above technical solutions, the embodiments of the present application have the following advantages:

determining a cold standby system corresponding to the stored data based on a storage environment configured by a data source end, so as to obtain cold standby data corresponding to the stored data through the cold standby system; then transmitting the cold standby data from a data source end to a migration target end, wherein the storage clusters corresponding to the data source end are different from the storage clusters corresponding to the migration target end; controlling a migration target end to process the cold standby data to obtain processed data; and writing the processing data into the migration target according to a preset rule, wherein the preset rule is determined based on a storage environment configured by the migration target. Therefore, in the efficient migration process of the storage number crossing the clusters, the cold standby data is used as a data source for migration, so that the influence of the network state between the data source end and the migration target end on the data migration is avoided, the method can be adapted to different storage systems, and the efficiency of the migration of the storage data crossing the clusters is improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present application, and that other drawings may be obtained according to the provided drawings without inventive effort to a person skilled in the art.

FIG. 1 is a network architecture diagram of the operation of a processing system storing data;

FIG. 2 is a flow chart of a process for storing data according to an embodiment of the present application;

FIG. 3 is a flowchart of a method for processing stored data according to an embodiment of the present disclosure;

fig. 4 is a schematic view of a scenario of a method for processing stored data according to an embodiment of the present application;

fig. 5 is a schematic view of a scenario of another method for processing stored data according to an embodiment of the present application;

fig. 6 is a schematic view of a scenario of another method for processing stored data according to an embodiment of the present application;

fig. 7 is a schematic view of a scenario of another method for processing stored data according to an embodiment of the present application;

fig. 8 is a schematic view of a scenario of another method for processing stored data according to an embodiment of the present application;

Fig. 9 is a schematic view of a scenario of another method for processing stored data according to an embodiment of the present application;

FIG. 10 is a flowchart of another method for processing stored data according to an embodiment of the present disclosure;

FIG. 11 is a schematic structural diagram of a processing device for storing data according to an embodiment of the present application;

fig. 12 is a schematic structural diagram of a terminal device according to an embodiment of the present application;

fig. 13 is a schematic structural diagram of a server according to an embodiment of the present application.

Detailed Description

The embodiment of the application provides a processing method of stored data and a related device, which can be applied to a system or a program containing a processing function of the stored data in terminal equipment, and a cold standby system corresponding to the stored data is determined based on a storage environment configured by a data source end so as to acquire the cold standby data corresponding to the stored data through the cold standby system; then transmitting the cold standby data from a data source end to a migration target end, wherein the storage clusters corresponding to the data source end are different from the storage clusters corresponding to the migration target end; controlling a migration target end to process the cold standby data to obtain processed data; and writing the processing data into the migration target according to a preset rule, wherein the preset rule is determined based on a storage environment configured by the migration target. Therefore, in the efficient migration process of the storage number crossing the clusters, the cold standby data is used as a data source for migration, so that the influence of the network state between the data source end and the migration target end on the data migration is avoided, the method can be adapted to different storage systems, and the efficiency of the migration of the storage data crossing the clusters is improved.

The terms "first," "second," "third," "fourth" and the like in the description and in the claims of this application and in the above-described figures, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that embodiments of the present application described herein may be capable of operation in sequences other than those illustrated or described herein, for example. Furthermore, the terms "comprises," "comprising," and "includes" and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus.

First, some terms that may appear in the embodiments of the present application will be explained.

Cold standby (cold backup): also referred to as offline backup, refers to a full backup of a database that is performed under conditions where the database is closed and cannot be updated, and may be designated for recovery.

Clustering: a clustered system is a computer system that is connected by a set of loosely-integrated computer software and/or hardware to perform computing work in a highly-tight, coordinated manner.

Dump: at a particular moment, the contents of the entire storage device or a portion of the storage device are recorded in another storage device.

It should be understood that the method for processing the stored data provided in the present application may be applied to a system or a program including a processing function of the stored data in a terminal device, for example, a database management application, and specifically, the system for processing the stored data may be operated in a network architecture shown in fig. 1, and as shown in fig. 1, the system for processing the stored data is a network architecture diagram operated by the system for processing the stored data, as shown in the figure, the system for processing the stored data may provide a processing procedure of the stored data with a plurality of information sources, that is, initiation of data migration is performed through a target operation on a terminal side, so that data between servers across clusters may be shared; it will be appreciated that various terminal devices are shown in fig. 1, the terminal devices may be computer devices, in an actual scenario, there may be more or less terminal devices participating in the process of storing data, and the specific number and types are not limited herein, and in addition, one server is shown in fig. 1, but in an actual scenario, there may also be a plurality of servers participating in the specific number of servers depending on the actual scenario.

In this embodiment, the server may be an independent physical server, or may be a server cluster or a distributed system formed by a plurality of physical servers, or may be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal may be, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc. The terminals and servers may be directly or indirectly connected by wired or wireless communication, and the terminals and servers may be connected to form a blockchain network, which is not limited herein.

It will be appreciated that the above processing system for storing data may be implemented in a personal mobile terminal, for example: the application can be used as a database management application, can also be run on a server, and can also be used as a third party device to provide processing of stored data so as to obtain a processing result of the stored data of the information source; the specific processing system for storing data may be in a program form, or may be operated as a system component in the device, or may be used as a cloud service program, where a specific operation mode is determined according to an actual scenario, and is not limited herein.

Therefore, for the storage system, the realization of an efficient cross-cluster data migration scheme has the following difficulties that one data source machine is multiple, the data types are miscellaneous, and the management and statistics are difficult; secondly, under the condition of poor network quality, the data migration efficiency of the recording level is low; thirdly, no existing tool exists, and the difficulty of cross-system data migration is high; fourth, the whole process is complex, and manual operation or script operation is very easy to make mistakes, and the solution will be described around the four difficulties in the following.

In order to solve the above-mentioned problems, the present application proposes a method for processing storage data, where the method is applied to a flow frame of processing storage data shown in fig. 2, as shown in fig. 2, which is a flow frame diagram of processing storage data provided in an embodiment of the present application, by performing a data processing operation on a terminal, a cluster 1 (source data end) obtains immediate backup data, and after performing compression segmentation on the backup data, the backup data is transmitted to a cluster 2 (migration target end), and after performing processing adaptation on the backup data by the cluster 2, the backup data can be written into the cluster 2, thereby implementing cross-cluster migration of the storage data.

The flow framework can be compatible with a plurality of existing cloud backup systems to acquire data sources, supports global rapid trans-continent migration of data, and supports trans-system data migration, such as migration of data to mysql and redis; the method further supports the page bill to complete the whole flow, reduces the operation and maintenance cost, and is explained in the following step description.

It can be understood that the method provided in the present application may be a program writing, so as to be used as a processing logic in a hardware system, or may be used as a processing device for storing data, and the processing logic is implemented in an integrated or external manner. As an implementation manner, the processing device of the stored data determines a cold standby system corresponding to the stored data based on a storage environment configured by a data source end, so as to obtain the cold standby data corresponding to the stored data through the cold standby system; then transmitting the cold standby data from a data source end to a migration target end, wherein the storage clusters corresponding to the data source end are different from the storage clusters corresponding to the migration target end; controlling a migration target end to process the cold standby data to obtain processed data; and writing the processing data into the migration target according to a preset rule, wherein the preset rule is determined based on a storage environment configured by the migration target. Therefore, in the efficient migration process of the storage number crossing the clusters, the cold standby data is used as a data source for migration, so that the influence of the network state between the data source end and the migration target end on the data migration is avoided, the method can be adapted to different storage systems, and the efficiency of the migration of the storage data crossing the clusters is improved.

With reference to the foregoing flowchart architecture, a method for processing stored data in the present application will be described, and referring to fig. 3, fig. 3 is a flowchart of a method for processing stored data provided in an embodiment of the present application, where the method for managing stored data may be executed by a participant, and the embodiment of the present application at least includes the following steps:

301. and determining a cold standby system corresponding to the stored data based on the storage environment configured by the data source end so as to acquire the cold standby data corresponding to the stored data through the cold standby system.

It will be appreciated that data migration is a very important and irreplaceable operational function for a storage system. The embodiment can be applied to a KV database TcaplusDB, wherein data migration in the daily operation process of the TcaplusDB can be used in the embodiment, such as global trans-continent migration data, and data of a production environment are copied and migrated to a data analysis system and the like.

Specifically, since the distributed storage system may be deployed in multiple regions and multiple environments, the situations need to be strongly compatible when the data of the cold standby is acquired, so that the cold standby is ensured to be successfully performed, and the cold standby system corresponding to the stored data needs to be determined based on the storage environment configured by the data source end. In one possible scenario, for a distributed storage system deployed in a corporate intranet, an owned cold standby system may be used, or an industry-wide cold standby system cluster may be used. For a distributed storage system deployed in a cloud environment, a cold standby system owned by the cloud environment is generally preferentially selected, for example, the distributed storage system deployed in the AWS may select AWS S3 as the cold standby system. For environments without a cold standby system, it is recommended to dump cold standby data in real time directly from the relevant storage nodes.

Therefore, the embodiment can be automatically compatible with the following four methods for acquiring the cold standby, wherein one method is compatible with an industry open source cold standby system, such as a cluster; second, compatible with customized cold standby systems, such as AWS S3; third, it is compatible with own (custom) cold standby systems; in addition, if the cold standby is not available, the cold standby is performed on the storage node in real time, so that the suitability of cold data acquisition in the data migration process is improved.

302. And transmitting the cold standby data from the data source end to the migration target end.

In this embodiment, the storage clusters corresponding to the data source end are different from the storage clusters corresponding to the migration target end, that is, this embodiment may act on the data migration process across clusters, and the specific cluster spans may be different geographic locations, for example, cross-intercontinental data migration; different network states are possible, such as data migration across a mask scenario, with specific cluster distribution depending on the actual scenario.

Specifically, when the cold standby data is transmitted, the cold standby data can be compressed and segmented according to a real-time network state, namely, network transmission parameters, such as transmission rate, network speed, network delay and the like, between a data source end and a migration target end are firstly obtained; then compressing the cold standby data based on the network transmission parameters to obtain compressed data, for example, the slower the transmission rate is, the smaller the cold standby data is compressed; dividing the compressed data according to a preset dividing granularity to obtain a plurality of divided data; and transmitting the segmentation data from the data source end to the migration target end through the concurrency channel.

In one possible scenario, the cold standby data may be optionally compressed and split into substantially identical file blocks after the cold standby is acquired. And if the transmission efficiency of the subsequent cold-standby files such as cross-continent transmission is low, selecting an algorithm for efficiently compressing the files as much as possible. File compression is straightforward using some common tools, such as lzo, tar.

It can be appreciated that, because the sizes of the files are different, in order to improve the subsequent transmission efficiency, the files may be split into file blocks with approximately the same size, for example, all the files may be split into file blocks with a size of 512MB, where the specific size may be dependent on the specific situation.

In addition, if the subsequent cold standby file is transmitted very efficiently, even without major transmission, then no compression may be selected. Monitoring network transmission parameters between a data source end and a migration target end; if the network transmission parameters meet the transmission conditions, the cold standby data are segmented according to a preset segmentation granularity to obtain a plurality of transmission data; and then transmitting the transmission data from the data source end to the migration target end through the concurrent channel, thereby improving the processing efficiency of the cold standby data.

In a possible scenario, the acquisition of the backup data is performed in real time, as shown in fig. 4, and fig. 4 is a schematic diagram of a scenario of a processing method of storage data according to an embodiment of the present application; the figure shows that daily cold standby execution control is performed, so that the integrity and matching degree of cold standby data are guaranteed.

In another possible scenario, the network of the source and destination of the data migration may be disconnected (e.g., the network is masked) or the communication quality may be very poor, since the cold-standby file needs to be transferred once. While transferring the cold-standby file is a time-consuming big head in the data migration task, especially for the transcontinent data migration scenario, efficient transfer of the cold-standby file is a great weight in improving efficiency. Such a transfer task may be accomplished using a relay cluster for scenarios where the network of the source and target of the data migration is not feasible. If the network transmission parameters indicate that the network between the data source end and the migration target end is abnormal, transmitting the segmentation data from the data source end to the transfer cluster; and then the segmentation data is transmitted to the migration target end through the transfer cluster.

It can be understood that the network communication quality of the transfer cluster and the source and target is better than that of the direct communication of the source and target. If the transfer cluster is not available, or the network quality of the source end and the target end is acceptable, the cold standby file can be directly transmitted from the data migration source end to the data migration target end. The cold standby file may be transferred directly using standard tools, such as scp.

In this embodiment, pipelining may also be used to transfer the cold standby file. That is, the cold standby file is transmitted while the cold standby file is generated. Parallelizing cold standby file generation and transmission; in addition, the file transmission efficiency can be improved by adjusting the network transceiving buffer zone of the operating system, for example, in a linux system, the net.core.wmem_max and the net.core.rmem_max can be increased to more than 20MB in a/etc/sysctl.conf file; furthermore, in order to fully utilize the network bandwidth, multiple channels should be used to transmit files concurrently, and because one file is subjected to multiple segmentation, even if the number of the cold standby files is smaller, the transmission efficiency can be improved by several times.

According to the embodiment, the cold standby is directly used as a data migration source, and the cold standby data file is remotely copied to the same region of the target storage system, so that the feasibility of a migration process is ensured; and the target end mode can be imported through various data; data may also be migrated to a variety of different storage systems; and supports the page bill of lading to complete the entire process, which effect can be described in connection with the embodiments described below.

303. And controlling the migration target end to process the cold standby data to obtain processed data.

In this embodiment, since the storage system or cluster distribution of the migration target may be different from that of the source data, the data processing is required.

It can be understood that compression and segmentation may be performed during the transmission of the cold standby data, and at this time, the compression and segmentation may need to be performed correspondingly, and the data writing process may be performed in combination with a specific storage system or cluster distribution.

304. And writing the processed data into the migration target terminal according to a preset rule.

In this embodiment, the preset rule is determined based on the storage environment configured by the migration target, that is, the difference between the storage system (storage environment) of the data source and the storage system (storage environment) of the migration target.

Specifically, after the cold standby file is transmitted to the data migration target, there may be various methods for applying the data in the cold standby file to the data migration target, and the target supports various storage systems. Aiming at different differential preset rules, a brand new environment physical replacement cold standby file at a target end can be adopted to realize data import; or using the target end storage system api to write the data into the target end;

In a possible scenario, for assuming that the data migration target is a brand new storage system, deployment details such as routing distribution of the source storage system can be completely copied, namely, firstly comparing cluster distribution of the data source with cluster distribution of the migration target; if the cluster distribution of the data source end is the same as that of the migration target end, determining that the preset rule is a physical replacement mode; and then writing the processed data into the migration target end in a physical replacement mode. Therefore, the cold standby file can be directly placed into the data migration target end storage system, and the cold standby file can be directly used, so that time and labor are saved. A schematic diagram of a specific physical replacement cold standby file importing scheme is shown in fig. 5, and fig. 5 is a schematic diagram of a scenario of another processing method of stored data provided in an embodiment of the present application; the figure shows that the compressed and split cold standby files are firstly merged and decompressed, and then physical replacement of corresponding cluster nodes (deployment details such as routing distribution of a storage system) is directly carried out, so that efficient data writing is realized.

In addition, the scenario where the data migration destination is an existing storage system may be the same storage system as the source, or may be a different storage system. The scheme for importing data to the target end is to use a tool to read all data in the cold standby file, and then write the processed data (if required) to the target storage system through the api of the target storage system. Fig. 6 is a schematic diagram of a target-end api data importing scheme, and fig. 6 is a schematic diagram of a scenario of another processing method of stored data according to an embodiment of the present application; if the cluster distribution of the data source end is the same as that of the migration target end, determining a write-in interface of the migration target end based on a preset rule; and then, adapting the processing data with the migration target terminal based on the writing interface so as to write the processing data into the migration target terminal, thereby realizing the writing adapting process of the multi-storage system.

In a possible embodiment, the whole data migration process can support pipeline operation, parallelism can be fully utilized, operation time is shortened, and efficiency is improved. Under the condition of limited server resources, the server and network resources can be efficiently utilized, and each link can be operated in parallel. For example, for a scenario in which the stored data includes a first file and a second file, the first file may be acquired first through a backup cooling system; then, the first file is transmitted from the data source end to the migration target end, and a second file is acquired through the cold standby system; the migration target end is controlled to process the first file to obtain first file data, and meanwhile, a second file is transmitted from the data source end to the migration target end; further, the first file data is written into the migration target end according to a preset rule, and the migration target end is controlled to process the second file to obtain second file data; and finally, writing the second file data into the migration target terminal according to a preset rule.

In another possible scenario, as shown in fig. 7, fig. 7 is a schematic view of a scenario of another processing method for storing data according to an embodiment of the present application; there are five tables (files) A, B, C, D, E shown to do data migration. First, the table a obtains the cold backup from the cold backup center, and then when the table a performs cold backup transmission, the table B can obtain the cold backup from the cold backup center. Then, table a performs data import at the target end, table B may perform a backup transmission, table C may acquire a backup from the backup center, and so on. Of course, the operating units of the pipeline here are server-specific. For example, if the server resources are enough, the table a and the table B may be used together to perform a cold standby acquisition in the cold standby center, and then may also be used together to perform a cold standby data transmission, that is, the operation units of the pipeline may operate together with several tables, where the specific number of files depends on the actual scenario, and is not limited herein.

It will be appreciated that data migration is a tedious and complex task, even though all the processes are clear, and if done purely manually or by running script operations is not only time consuming and labor consuming, but can also be very error prone. The embodiment uses the page bill of lading to complete the entire task. Since the source and destination of the migration data across the cluster are usually two completely isolated systems, tasks need to be generated in the two systems respectively, the source constructs the data, and the destination imports the data. Of course, if the source end and the target end of the data migration are the same system, the whole process can be completed directly on the page bill of the system. The method comprises the steps that a storage environment configured by a data source end can be obtained in response to a bill of lading operation in a target interface, and the storage environment configured by a migration target end is obtained based on the bill of lading operation; then determining a cold standby system corresponding to the stored data according to a storage environment configured by the data source end, so as to obtain cold standby data corresponding to the stored data through the cold standby system; for the subsequent writing process, a preset rule can be determined based on the storage environment configured by the data source end and the storage environment configured by the migration target end; and then writing the processed data into the migration target terminal according to a preset rule.

In one possible scenario, a schematic diagram of completing an entire data migration task by using a page system bill of lading is shown in fig. 8, and fig. 8 is a schematic diagram of a scenario of another processing method for storing data according to an embodiment of the present application; and triggering the system adaptation process of the source end and the target end of the data migration through the bill of lading operation in the browser page, thereby automatically executing the data migration process in the embodiment. Specifically, the bill of lading operation may be displayed by using an interface shown in fig. 9, and fig. 9 is a schematic view of a scenario of another processing method of stored data according to an embodiment of the present application; the figure shows an interface of the browser, and related personnel can select a data source end and a migration target end (cluster selection) of data migration, and then click on a migration bill of lading A1, so that the data migration process in the above embodiment can be automatically performed, which is not described herein.

According to the embodiment, the cold standby system corresponding to the stored data is determined based on the storage environment configured by the data source end, so that the cold standby data corresponding to the stored data is obtained through the cold standby system; then transmitting the cold standby data from a data source end to a migration target end, wherein the storage clusters corresponding to the data source end are different from the storage clusters corresponding to the migration target end; controlling a migration target end to process the cold standby data to obtain processed data; and writing the processing data into the migration target according to a preset rule, wherein the preset rule is determined based on a storage environment configured by the migration target. Therefore, in the efficient migration process of the storage number crossing the clusters, the cold standby data is used as a data source for migration, so that the influence of the network state between the data source end and the migration target end on the data migration is avoided, the method can be adapted to different storage systems, and the efficiency of the migration of the storage data crossing the clusters is improved.

Therefore, after the embodiment is used, a storage system administrator can be liberated from complex and heavy data migration tasks, and the whole function can be completed directly on the page bill. The method can be applied to large complex scenes such as global transcontinent migration data; the method can also be applied to data migration across a storage system, so that the data can be repeatedly utilized in multiple scenes; data may also be replicated and migrated from one environment to another for analysis of localization problems.

The above-described functions may play an important role in a plurality of scenes, for example, in a game scene, in which a process of data migration is required when a player who needs to perform cross-zone clothing performs a battle, and the scene will be described below. Referring to fig. 10, fig. 10 is a flowchart of another processing method of stored data according to an embodiment of the present application, where the embodiment of the present application at least includes the following steps:

1001. and obtaining user information corresponding to the target user in the first game server.

In this embodiment, the first game server may be a common server of a player, and because some activities of the cross-regional suit (such as global match) occur in the game, the cross-regional suit migration of the game data needs to be performed, so as to ensure smooth and normal running of the game service.

1002. And calling the cold standby data corresponding to the user information.

In this embodiment, the backup data corresponding to the user information may be the backup data updated automatically last time, for example, the backup data updated last 12 hours or updated after the target user logs in last time, and the specific manner depends on the actual scenario.

1003. And transmitting the cold standby data to the second game server.

In this embodiment, the transmission process of the backup data is referred to step 303 in the embodiment shown in fig. 3, which is not described herein.

1004. And providing game services for the target user based on the second game server.

In this embodiment, the process of writing the backup data into the second game server is referred to as step 304 in the embodiment shown in fig. 3, which is not described herein.

After the cold standby data is written into the second game server, the second game server can be used for the player to log in and provide the same game service as the first server.

It can be understood that, if the game activity is finished, the method of this embodiment may also be used to migrate the game data of the target user (after the second game server is performed) to the first server, where the specific migration times are determined according to the actual scenario, which is not described herein.

Through the embodiment, the normal running of the cross-regional service game is ensured, the cross-regional service game can be adapted to game operation systems in different areas, and the efficiency and the effectiveness of game data migration are ensured.

In order to better implement the above-described aspects of the embodiments of the present application, the following also provides related devices for implementing the above-described aspects. Referring to fig. 11, fig. 11 is a schematic structural diagram of a processing device for storing data according to an embodiment of the present application, where a processing device 1100 for storing data includes:

an obtaining unit 1101, configured to determine a cold standby system corresponding to storage data based on a storage environment configured by a data source, so as to obtain cold standby data corresponding to the storage data through the cold standby system;

a transmission unit 1102, configured to transmit the cold standby data from the data source end to a migration target end, where a storage cluster corresponding to the data source end is different from a storage cluster corresponding to the migration target end;

a processing unit 1103, configured to control the migration target end to process the cold standby data to obtain processed data;

the processing unit 1103 is further configured to write the processing data into the migration target according to a preset rule, where the preset rule is determined based on a storage environment configured by the migration target.

Optionally, in some possible implementations of the present application, the transmission unit 1102 is specifically configured to obtain a network transmission parameter between the data source end and the migration destination end;

The transmission unit 1102 is specifically configured to compress the backup data based on the network transmission parameter to obtain compressed data;

the transmission unit 1102 is specifically configured to segment the compressed data according to a preset segmentation granularity, so as to obtain a plurality of segmentation data;

the transmission unit 1102 is specifically configured to transmit the sliced data from the data source end to the migration destination end through a concurrent channel.

Optionally, in some possible implementations of the present application, the transmission unit 1102 is specifically configured to monitor the network transmission parameter between the data source end and the migration destination end;

the transmission unit 1102 is specifically configured to segment the backup data according to the preset segmentation granularity to obtain a plurality of transmission data if the network transmission parameter meets a transmission condition;

the transmitting unit 1102 is specifically configured to transmit the transmission data from the data source end to the migration destination end through the concurrent channel.

Optionally, in some possible implementations of the present application, the transmission unit 1102 is specifically configured to transmit the split data from the data source end to a transit cluster if the network transmission parameter indicates that a network between the data source end and the migration target end is abnormal;

The transmission unit 1102 is specifically configured to transmit the cut data to the migration target through the transit cluster.

Optionally, in some possible implementations of the present application, the processing unit 1103 is specifically configured to compare a cluster distribution of the data source end with a cluster distribution of the migration target end;

the processing unit 1103 is specifically configured to determine that the preset rule is a physical replacement mode if the cluster distribution of the data source end is the same as the cluster distribution of the migration target end;

the processing unit 1103 is specifically configured to write the processing data into the migration target by adopting the physical replacement manner;

or;

the processing unit 1103 is specifically configured to determine a write interface of the migration target based on the preset rule if the cluster distribution of the data source end is the same as the cluster distribution of the migration target end;

the processing unit 1103 is specifically configured to adapt the processing data with the migration target based on the write interface, so as to write the processing data into the migration target.

Optionally, in some possible implementations of the present application, the obtaining unit 1101 is specifically configured to obtain the storage environment configured by the data source end in response to a bill of lading operation in a target interface, and obtain the storage environment configured by the migration target end based on the bill of lading operation;

The obtaining unit 1101 is specifically configured to determine a cold standby system corresponding to the stored data according to a storage environment configured by the data source end, so as to obtain cold standby data corresponding to the stored data through the cold standby system;

the processing unit 1103 is specifically configured to determine the preset rule based on the storage environment configured by the data source end and the storage environment configured by the migration target end;

the processing unit 1103 is specifically configured to write the processing data into the migration target according to the preset rule.

Optionally, in some possible implementations of the present application, the processing unit 1103 is specifically configured to obtain the first file through the cold standby system;

the processing unit 1103 is specifically configured to obtain the second file through the cold standby system while transmitting the first file from the data source end to the migration target end;

the processing unit 1103 is specifically configured to control the migration target to process the first file to obtain first file data, and simultaneously transmit the second file from the data source to the migration target;

the processing unit 1103 is specifically configured to control the migration target to process the second file to obtain second file data while writing the first file data into the migration target according to the preset rule;

The processing unit 1103 is specifically configured to write the second file data into the migration target according to the preset rule.

The embodiment of the present application further provides a terminal device, as shown in fig. 12, which is a schematic structural diagram of another terminal device provided in the embodiment of the present application, for convenience of explanation, only a portion related to the embodiment of the present application is shown, and specific technical details are not disclosed, and please refer to a method portion of the embodiment of the present application. The terminal may be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (personal digital assistant, PDA), a point of sale (POS), a vehicle-mounted computer, and the like, taking the terminal as an example of the mobile phone:

Fig. 12 is a block diagram showing a part of the structure of a mobile phone related to a terminal provided in an embodiment of the present application. Referring to fig. 12, the mobile phone includes: radio Frequency (RF) circuitry 1210, memory 1220, input unit 1230, display unit 1240, sensor 1250, audio circuitry 1260, wireless fidelity (wireless fidelity, wiFi) module 1270, processor 1280, and power supply 1290. Those skilled in the art will appreciate that the handset configuration shown in fig. 12 is not limiting of the handset and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.

The following describes the components of the mobile phone in detail with reference to fig. 12:

the RF circuit 1210 may be used for receiving and transmitting signals during a message or a call, and in particular, after receiving downlink information of a base station, the signal is processed by the processor 1280; in addition, the data of the design uplink is sent to the base station. Typically, RF circuitry 1210 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (low noise amplifier, LNA), a duplexer, and the like. In addition, RF circuitry 1210 may also communicate with networks and other devices through wireless communication. The wireless communications may use any communication standard or protocol including, but not limited to, global system for mobile communications (global system of mobile communication, GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), wideband code division multiple access (wideband code division multiple access, WCDMA), long term evolution (long term evolution, LTE), email, short message service (short messaging service, SMS), and the like.

Memory 1220 may be used to store software programs and modules, and processor 1280 may perform various functional applications and data processing for the cellular phone by executing the software programs and modules stored in memory 1220. The memory 1220 may mainly include a storage program area that may store an operating system, application programs required for at least one function (such as a sound playing function, an image playing function, etc.), and a storage data area; the storage data area may store data (such as audio data, phonebook, etc.) created according to the use of the handset, etc. In addition, memory 1220 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device.

The input unit 1230 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the mobile phone. In particular, the input unit 1230 may include a touch panel 1231 and other input devices 1232. The touch panel 1231, also referred to as a touch screen, may collect touch operations thereon or thereabout (e.g., operations of a user using any suitable object or accessory such as a finger, a stylus, etc. on the touch panel 1231 or thereabout, and spaced touch operations within a certain range on the touch panel 1231) and drive the corresponding connection device according to a predetermined program. Alternatively, the touch panel 1231 may include two parts, a touch detection device and a touch controller. The touch detection device detects the touch azimuth of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor 1280, and can receive commands from the processor 1280 and execute them. In addition, the touch panel 1231 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. The input unit 1230 may include other input devices 1232 in addition to the touch panel 1231. In particular, other input devices 1232 may include, but are not limited to, one or more of a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, mouse, joystick, etc.

The display unit 1240 may be used to display information input by a user or information provided to the user and various menus of the mobile phone. The display unit 1240 may include a display panel 1241, and alternatively, the display panel 1241 may be configured in the form of a liquid crystal display (liquid crystal display, LCD), an organic light-emitting diode (OLED), or the like. Further, the touch panel 1231 may overlay the display panel 1241, and when the touch panel 1231 detects a touch operation thereon or nearby, the touch operation is transmitted to the processor 1280 to determine the type of touch event, and then the processor 1280 provides a corresponding visual output on the display panel 1241 according to the type of touch event. Although in fig. 12, the touch panel 1231 and the display panel 1241 are two separate components to implement the input and input functions of the mobile phone, in some embodiments, the touch panel 1231 may be integrated with the display panel 1241 to implement the input and output functions of the mobile phone.

The handset can also include at least one sensor 1250, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1241 according to the brightness of ambient light, and the proximity sensor may turn off the display panel 1241 and/or the backlight when the mobile phone moves to the ear. As one of the motion sensors, the accelerometer sensor can detect the acceleration in all directions (generally three axes), and can detect the gravity and direction when stationary, and can be used for applications of recognizing the gesture of a mobile phone (such as horizontal and vertical screen switching, related games, magnetometer gesture calibration), vibration recognition related functions (such as pedometer and knocking), and the like; other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that may also be configured with the handset are not described in detail herein.

Audio circuitry 1260, speaker 1261, microphone 1262 may provide an audio interface between the user and the handset. Audio circuit 1260 may transmit the received electrical signal after audio data conversion to speaker 1261, where the electrical signal is converted to a sound signal by speaker 1261 for output; on the other hand, microphone 1262 converts the collected sound signals into electrical signals, which are received by audio circuit 1260 and converted into audio data, which are processed by audio data output processor 1280 for transmission to, for example, another cell phone via RF circuit 1210, or which are output to memory 1220 for further processing.

WiFi belongs to a short-distance wireless transmission technology, and a mobile phone can help a user to send and receive emails, browse webpages, access streaming media and the like through a WiFi module 1270, so that wireless broadband Internet access is provided for the user. Although fig. 12 shows the WiFi module 1270, it is understood that it does not belong to the necessary constitution of the mobile phone, and can be omitted entirely as required within the scope of not changing the essence of the invention.

Processor 1280 is a control center of the mobile phone, connects various parts of the entire mobile phone using various interfaces and lines, and performs various functions and processes data of the mobile phone by running or executing software programs and/or modules stored in memory 1220 and calling data stored in memory 1220, thereby performing overall monitoring of the mobile phone. In the alternative, processor 1280 may include one or more processing units; alternatively, the processor 1280 may integrate an application processor and a modem processor, wherein the application processor primarily processes operating systems, user interfaces, application programs, etc., and the modem processor primarily processes wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 1280.

The handset further includes a power supply 1290 (e.g., a battery) for powering the various components, optionally in logical communication with the processor 1280 through a power management system so as to perform charge, discharge, and power management functions via the power management system.

Although not shown, the mobile phone may further include a camera, a bluetooth module, etc., which will not be described herein.

In the embodiment of the present application, the processor 1280 included in the terminal further has a function of performing each step of the page processing method as described above.

The present embodiment also provides a server, please refer to fig. 13, fig. 13 is a schematic structural diagram of a server provided in the present embodiment, where the server 1300 may have a relatively large difference due to different configurations or performances, and may include one or more central processing units (central processing units, CPU) 1322 (e.g. one or more processors) and a memory 1332, one or more storage media 1330 (e.g. one or more mass storage devices) storing application programs 1342 or data 1344. Wherein the memory 1332 and storage medium 1330 may be transitory or persistent. The program stored on the storage medium 1330 may include one or more modules (not shown), each of which may include a series of instruction operations on a server. Further, the central processor 1322 may be configured to communicate with the storage medium 1330, and execute a series of instruction operations in the storage medium 1330 on the server 1300.

The server 1300 may also include one or more power supplies 1326, one or more wired or wireless network interfaces 1350, one or more input/output interfaces 1358, and/or one or more operating systems 1341, such as Windows server (tm), mac OS XTM, unixTM, linuxTM, freeBSDTM, and so forth.

The steps performed by the management apparatus in the above-described embodiments may be based on the server structure shown in fig. 13.

Further provided in embodiments of the present application is a computer readable storage medium having stored therein processing instructions for storing data, which when executed on a computer, cause the computer to perform the steps performed by the processing device for storing data in the method described in the embodiments shown in the foregoing fig. 3 to 10.

There is also provided in an embodiment of the present application a computer program product comprising processing instructions for storing data, which when run on a computer causes the computer to perform the steps performed by the processing means for storing data in the method described in the embodiment shown in the foregoing figures 3 to 10.

The embodiment of the application also provides a processing system for storing data, which may include the processing device for storing data in the embodiment described in fig. 11, or the terminal device in the embodiment described in fig. 12, or the server described in fig. 13.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.

In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or all or part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a processing device storing data, or a network device, etc.) to perform all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The above embodiments are merely for illustrating the technical solution of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the corresponding technical solutions.

Claims

1. A method of processing stored data, comprising:

2. The method of claim 1, wherein the transmitting the cold standby data from the data source to the migration target comprises:

3. The method according to claim 2, wherein the method further comprises:

4. The method according to claim 2, wherein said transmitting the segmentation data from the cold standby system to the migration target comprises:

5. The method according to claim 1, wherein writing the processing data to the migration destination according to a preset rule comprises:

or;

6. The method according to claim 1, wherein the determining, based on the storage environment configured by the data source, a cold standby system corresponding to the storage data to obtain, by the cold standby system, the cold standby data corresponding to the storage data includes:

7. The method of any of claims 1-6, wherein the stored data comprises a first file and a second file, the method further comprising:

acquiring the first file through the cold standby system;

8. A processing apparatus for storing data, comprising:

9. A computer device, the computer device comprising a processor and a memory:

The memory is used for storing program codes; the processor is configured to execute the method for processing stored data according to any one of claims 1 to 7 according to instructions in the program code.

10. A computer program product comprising computer programs/instructions which, when executed by a processor, implement the steps of the method of processing stored data according to any one of the preceding claims 1 to 7.